Skip to content

Instantly share code, notes, and snippets.

@pkmishra

pkmishra/settings.py

Last active December 15, 2015 03:19
Show Gist options
  • Save pkmishra/5193538 to your computer and use it in GitHub Desktop.
Save pkmishra/5193538 to your computer and use it in GitHub Desktop.
Download ZIP

Revisions

  1. pkmishra revised this gist Mar 19, 2013. 1 changed file with 2 additions and 2 deletions.
    4 changes: 2 additions & 2 deletions settings.py
    View file
    Original file line number Diff line number Diff line change
    @@ -6,8 +6,8 @@
    'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_3) AppleWebKit/534.55.3 (KHTML, like Gecko) Version/5.1.3 Safari/534.53.10'
    ]
    DOWNLOADER_MIDDLEWARES = {
    'winestores.middlewares.RandomUserAgentMiddleware': 400,
    'winestores.middlewares.ProxyMiddleware': 410,
    'myproject.middlewares.RandomUserAgentMiddleware': 400,
    'myproject.middlewares.ProxyMiddleware': 410,
    'scrapy.contrib.downloadermiddleware.useragent.UserAgentMiddleware': None,
    # Disable compression middleware, so the actual HTML pages are cached
    }
  2. pkmishra created this gist Mar 19, 2013.
    13 changes: 13 additions & 0 deletions settings.py
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,13 @@
    # More comprehensive list can be found at
    # http://techpatterns.com/forums/about304.html
    USER_AGENT_LIST = [
    'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.7 (KHTML, like Gecko) Chrome/16.0.912.36 Safari/535.7',
    'Mozilla/5.0 (Windows NT 6.2; Win64; x64; rv:16.0) Gecko/16.0 Firefox/16.0',
    'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_3) AppleWebKit/534.55.3 (KHTML, like Gecko) Version/5.1.3 Safari/534.53.10'
    ]
    DOWNLOADER_MIDDLEWARES = {
    'winestores.middlewares.RandomUserAgentMiddleware': 400,
    'winestores.middlewares.ProxyMiddleware': 410,
    'scrapy.contrib.downloadermiddleware.useragent.UserAgentMiddleware': None,
    # Disable compression middleware, so the actual HTML pages are cached
    }