Skip to content

Instantly share code, notes, and snippets.

@arslandevs

arslandevs/infinite_scroll_start_requests.py

Created May 4, 2024 15:23
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save arslandevs/a5f00eb133ac42ca112bd0bdbe3a780e to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save arslandevs/a5f00eb133ac42ca112bd0bdbe3a780e to your computer and use it in GitHub Desktop.
Download ZIP

Revisions

  1. arslandevs created this gist May 4, 2024.
    12 changes: 12 additions & 0 deletions infinite_scroll_start_requests.py
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,12 @@
    def start_requests(self):
    yield scrapy.Request(
    url="http://quotes.toscrape.com/scroll",
    meta={
    "playwright": True,
    "playwright_page_methods": [
    PageMethod("wait_for_selector", "div.quote")
    ],
    "playwright_include_page": True
    },
    errback=self.close_page
    )