Skip to content

Instantly share code, notes, and snippets.

@let-robots-reign

let-robots-reign/get_total_pages.py

Created July 12, 2019 08:51
Show Gist options
  • Save let-robots-reign/918c4a09f8f5dfb6c47aaa4f22ae0345 to your computer and use it in GitHub Desktop.
Save let-robots-reign/918c4a09f8f5dfb6c47aaa4f22ae0345 to your computer and use it in GitHub Desktop.
Download ZIP

Revisions

  1. let-robots-reign renamed this gist Jul 12, 2019. 1 changed file with 0 additions and 0 deletions.
    0 get_total_pages → get_total_pages.py
    View file
    File renamed without changes.
  2. let-robots-reign created this gist Jul 12, 2019.
    18 changes: 18 additions & 0 deletions get_total_pages
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,18 @@
    def get_total_pages(html):
    soup = BeautifulSoup(html, "lxml")
    try:
    pages = soup.find("div", class_="pagination-pages clearfix")
    if pages is not None:
    pages.find_all("a", class_="pagination-page")[-1].get("href")
    total_pages = int(pages.split("=")[1].split("&")[0])
    else:
    catalog = soup.find("div", class_="js-catalog_serp").find_all("div", class_="item_table")
    if catalog is not None:
    total_pages = 1
    else:
    raise Exception
    except Exception as e:
    with open("logs.txt", "a", encoding="utf8") as file:
    file.write(str(e) + " avito get_total_pages\n")
    sys.exit(0)
    return total_pages