Skip to content

Instantly share code, notes, and snippets.

@arslandevs

arslandevs/infinite_scroll_start_requests.py

Created May 4, 2024 15:23
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save arslandevs/a5f00eb133ac42ca112bd0bdbe3a780e to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save arslandevs/a5f00eb133ac42ca112bd0bdbe3a780e to your computer and use it in GitHub Desktop.
Download ZIP
Raw
infinite_scroll_start_requests.py
def start_requests(self):
yield scrapy.Request(
url="http://quotes.toscrape.com/scroll",
meta={
"playwright": True,
"playwright_page_methods": [
PageMethod("wait_for_selector", "div.quote")
],
"playwright_include_page": True
},
errback=self.close_page
)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment