Skip to content

Instantly share code, notes, and snippets.

@michaeldorner

michaeldorner/cached_parallel_requests.py

Created February 22, 2023 13:09
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save michaeldorner/7ca9ae497d55bdc4b2e9114a2ce0f1bc to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save michaeldorner/7ca9ae497d55bdc4b2e9114a2ce0f1bc to your computer and use it in GitHub Desktop.
Download ZIP

Revisions

  1. michaeldorner created this gist Feb 22, 2023.
    34 changes: 34 additions & 0 deletions cached_parallel_requests.py
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,34 @@
    import bz2

    import orjson
    from requests.adapters import HTTPAdapter, Retry
    from requests_cache import CachedSession, SerializerPipeline, Stage, serializers
    from requests_futures.sessions import FuturesSession

    SERIALIZER = SerializerPipeline([
    serializers.preconf.orjson_preconf_stage,
    orjson,
    Stage(dumps=bz2.compress, loads=bz2.decompress),
    ], is_binary=True)

    API_TOKEN = ''
    NUM_WORKERS = 10

    http_session = CachedSession('http_cache', serializer=SERIALIZER, backend='filesystem')
    http_session.headers.update({
    'User-Agent':'hamster_bth/ 1.0',
    'Accept': 'application/vnd.github+json',
    'Authorization': f'Bearer {API_TOKEN}',
    })

    retries = Retry(total=5,
    connect=5,
    backoff_factor=2,
    status_forcelist=[500, 501, 502, 503, 504],
    raise_on_status=False)

    http_session.mount('https://', HTTPAdapter(max_retries=retries))
    http_session.mount('http://', HTTPAdapter(max_retries=retries))

    with FuturesSession(max_workers=NUM_WORKERS, session=http_session) as future_session:
    http_session.get('http://httpbin.org/get')