Skip to content

Instantly share code, notes, and snippets.

@BouncingBison

BouncingBison/All of Adidas U.S. Products

Forked from 4bpb/All of Adidas U.S. Products
Created June 3, 2017 17:45
Show Gist options
  • Save BouncingBison/2a80bb11ac066fda2d05cb745871c263 to your computer and use it in GitHub Desktop.
Save BouncingBison/2a80bb11ac066fda2d05cb745871c263 to your computer and use it in GitHub Desktop.
Download ZIP
This script will parse code to get all of the products on adidas US
Raw
All of Adidas U.S. Products
from bs4 import BeautifulSoup
import urllib.request
import re
import urllib.parse
import time
headers = {"User-Agent" : "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36",
"Accept-Language" : "en-US,en;q=0.8"}
url = 'http://www.adidas.com/on/demandware.static/-/Sites-adidas-US-Library/en_US/v/sitemap/product/adidas-US-en-us-product.xml'
values = {'s':'search',
'submit':'search'}
data = urllib.parse.urlencode(values)
data = data.encode('utf-8')
req = urllib.request.Request(url, data, headers=headers)
resp = urllib.request.urlopen(req)
respData = resp.read()
rawdata = re.findall(r'<loc>(.*?)</loc>', str(respData))
for Product_list in rawdata:
print(Product_list)
time.sleep(500)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment