Skip to content

Instantly share code, notes, and snippets.

@IAlwaysBeCoding

IAlwaysBeCoding/synchronous.py

Last active September 21, 2018 04:52
Show Gist options
  • Save IAlwaysBeCoding/2dbfa1a301a4da60e677ad1f6ed0c354 to your computer and use it in GitHub Desktop.
Save IAlwaysBeCoding/2dbfa1a301a4da60e677ad1f6ed0c354 to your computer and use it in GitHub Desktop.
Download ZIP

Revisions

  1. IAlwaysBeCoding revised this gist Sep 21, 2018. No changes.
  2. IAlwaysBeCoding revised this gist Sep 21, 2018. No changes.
  3. IAlwaysBeCoding revised this gist Sep 21, 2018. No changes.
  4. IAlwaysBeCoding revised this gist Sep 21, 2018. 1 changed file with 2 additions and 1 deletion.
    3 changes: 2 additions & 1 deletion synchronous.py
    View file
    Original file line number Diff line number Diff line change
    @@ -1,3 +1,4 @@

    from scrapy import Request

    class SynchronousSpiderMiddleware(object):
    @@ -24,4 +25,4 @@ def process_spider_output(self, response, result, spider):
    yield request

    if step == 'parse':
    yield listing_request
    yield listing_request
  5. IAlwaysBeCoding renamed this gist Sep 21, 2018. 1 changed file with 0 additions and 0 deletions.
    0 gistfile1.txt → synchronous.py
    View file
    File renamed without changes.
  6. IAlwaysBeCoding created this gist Sep 21, 2018.
    27 changes: 27 additions & 0 deletions gistfile1.txt
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,27 @@
    from scrapy import Request

    class SynchronousSpiderMiddleware(object):


    def process_spider_output(self, response, result, spider):

    listing_request = None
    for request in result:
    step = request.meta.get('step', None)

    if step == 'parse':
    if not listing_request:
    listing_request = request

    listings = listing_request.meta.get('listings', [])
    listings.append(request)
    continue

    elif step == 'result':

    listings = request.meta.get('listings', [])
    request = listings.pop(0)
    yield request

    if step == 'parse':
    yield listing_request