Skip to content

Instantly share code, notes, and snippets.

@let-robots-reign

let-robots-reign/get_total_pages.py

Created July 12, 2019 08:51
Show Gist options
  • Save let-robots-reign/918c4a09f8f5dfb6c47aaa4f22ae0345 to your computer and use it in GitHub Desktop.
Save let-robots-reign/918c4a09f8f5dfb6c47aaa4f22ae0345 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
get_total_pages.py
def get_total_pages(html):
soup = BeautifulSoup(html, "lxml")
try:
pages = soup.find("div", class_="pagination-pages clearfix")
if pages is not None:
pages.find_all("a", class_="pagination-page")[-1].get("href")
total_pages = int(pages.split("=")[1].split("&")[0])
else:
catalog = soup.find("div", class_="js-catalog_serp").find_all("div", class_="item_table")
if catalog is not None:
total_pages = 1
else:
raise Exception
except Exception as e:
with open("logs.txt", "a", encoding="utf8") as file:
file.write(str(e) + " avito get_total_pages\n")
sys.exit(0)
return total_pages
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment