sublimecoder · September 19, 2023 14:37 · May 30, 2014 · May 29, 2014 · May 29, 2014
diff --git a/bot-blocking-expression b/bot-blocking-expression
@@ -1 +1 @@
-^$|EasouSpider|Add Catalog|PaperLiBot|Spiceworks|ZumBot|RU_Bot|Wget|Java/1.7.0_25|Slurp|FunWebProducts|80legs|Aboundex|AcoiRobot|Acoon Robot|AhrefsBot|aihit|AlkalineBOT|AnzwersCrawl|Arachnoidea|ArchitextSpider|archive|Autonomy Spider|Baiduspider|BecomeBot|benderthewebrobot|BlackWidow|Bork-edition|Bot mailto:[email protected]|botje|catchbot|changedetection|Charlotte|ChinaClaw|commoncrawl|ConveraCrawler|Covario|crawler|curl|Custo|data mining development project|DigExt|DISCo|discobot|discoveryengine|DOC|DoCoMo|DotBot|Download Demon|Download Ninja|eCatch|EirGrabber|EmailSiphon|EmailWolf|eurobot|Exabot|Express WebPictures|ExtractorPro|EyeNetIE|Ezooms|Fetch|Fetch API|filterdb|findfiles|findlinks|FlashGet|flightdeckreports|FollowSite Bot|Gaisbot|genieBot|GetRight|GetWeb!|gigablast|Gigabot|Go-Ahead-Got-It|Go!Zilla|GrabNet|Grafula|GT::WWW|hailoo|heritrix|HMView|houxou|HTTP::Lite|HTTrack|ia_archiver|IBM EVV|id-search|IDBot|Image Stripper|Image Sucker|Indy Library|InterGET|Internet Ninja|internetmemory|ISC Systems iRc Search 2.1|Jakarta|JetCar|JOC Web Spider|k2spider|larbin|larbin|LeechFTP|libghttp|libwww|libwww-perl|linko|LinkWalker|lwp-trivial|Mass Downloader|metadatalabs|MFC_Tear_Sample|Microsoft URL Control|MIDown tool|Missigua|Missigua Locator|Mister PiX|MJ12bot|MOREnet|MSIECrawler|msnbot|naver|Navroad|NearSite|Net Vampire|NetAnts|NetSpider|NetZIP|NextGenSearchBot|NPBot|Nutch|Octopus|Offline Explorer|Offline Navigator|omni-explorer|PageGrabber|panscient|panscient.com|Papa Foto|pavuk|pcBrowser|PECL::HTTP|PHP/|PHPCrawl|picsearch|pipl|pmoz|PredictYourBabySearchToolbar|RealDownload|Referrer Karma|ReGet|reverseget|rogerbot|ScoutJet|SearchBot|seexie|seoprofiler|Servage Robot|SeznamBot|shopwiki|sindice|sistrix|SiteSnagger|SiteSnagger|smart.apnoti.com|SmartDownload|Snoopy|Sosospider|spbot|suggybot|SuperBot|SuperHTTP|SuperPagesUrlVerifyBot|Surfbot|SurveyBot|SurveyBot|swebot|Synapse|Tagoobot|tAkeOut|Teleport|Teleport Pro|TeleportPro|TweetmemeBot|TwengaBot|twiceler|UbiCrawler|uptimerobot|URI::Fetch|urllib|User-Agent|VoidEYE|VoilaBot|WBSearchBot|Web Image Collector|Web Sucker|WebAuto|WebCopier|WebCopier|WebFetch|WebGo IS|WebLeacher|WebReaper|WebSauger|Website eXtractor|Website Quester|WebStripper|WebStripper|WebWhacker|WebZIP|WebZIP|Wells Search II|WEP Search|Widow|winHTTP|WWWOFFLE|Xaldon WebSpider|Xenu|yacybot|yandex|YandexBot|YandexImages|yBot|YesupBot|YodaoBot|yolinkBot|youdao|Zao|Zealbot|Zeus|ZyBORG
+^$|EasouSpider|Add Catalog|PaperLiBot|Spiceworks|ZumBot|RU_Bot|Wget|Java/1.7.0_25|Slurp|FunWebProducts|80legs|Aboundex|AcoiRobot|Acoon Robot|AhrefsBot|aihit|AlkalineBOT|AnzwersCrawl|Arachnoidea|ArchitextSpider|archive|Autonomy Spider|Baiduspider|BecomeBot|benderthewebrobot|BlackWidow|Bork-edition|Bot mailto:[email protected]|botje|catchbot|changedetection|Charlotte|ChinaClaw|commoncrawl|ConveraCrawler|Covario|crawler|curl|Custo|data mining development project|DigExt|DISCo|discobot|discoveryengine|DOC|DoCoMo|DotBot|Download Demon|Download Ninja|eCatch|EirGrabber|EmailSiphon|EmailWolf|eurobot|Exabot|Express WebPictures|ExtractorPro|EyeNetIE|Ezooms|Fetch|Fetch API|filterdb|findfiles|findlinks|FlashGet|flightdeckreports|FollowSite Bot|Gaisbot|genieBot|GetRight|GetWeb!|gigablast|Gigabot|Go-Ahead-Got-It|Go!Zilla|GrabNet|Grafula|GT::WWW|hailoo|heritrix|HMView|houxou|HTTP::Lite|HTTrack|ia_archiver|IBM EVV|id-search|IDBot|Image Stripper|Image Sucker|Indy Library|InterGET|Internet Ninja|internetmemory|ISC Systems iRc Search 2.1|JetCar|JOC Web Spider|k2spider|larbin|larbin|LeechFTP|libghttp|libwww|libwww-perl|linko|LinkWalker|lwp-trivial|Mass Downloader|metadatalabs|MFC_Tear_Sample|Microsoft URL Control|MIDown tool|Missigua|Missigua Locator|Mister PiX|MJ12bot|MOREnet|MSIECrawler|msnbot|naver|Navroad|NearSite|Net Vampire|NetAnts|NetSpider|NetZIP|NextGenSearchBot|NPBot|Nutch|Octopus|Offline Explorer|Offline Navigator|omni-explorer|PageGrabber|panscient|panscient.com|Papa Foto|pavuk|pcBrowser|PECL::HTTP|PHP/|PHPCrawl|picsearch|pipl|pmoz|PredictYourBabySearchToolbar|RealDownload|Referrer Karma|ReGet|reverseget|rogerbot|ScoutJet|SearchBot|seexie|seoprofiler|Servage Robot|SeznamBot|shopwiki|sindice|sistrix|SiteSnagger|SiteSnagger|smart.apnoti.com|SmartDownload|Snoopy|Sosospider|spbot|suggybot|SuperBot|SuperHTTP|SuperPagesUrlVerifyBot|Surfbot|SurveyBot|SurveyBot|swebot|Synapse|Tagoobot|tAkeOut|Teleport|Teleport Pro|TeleportPro|TweetmemeBot|TwengaBot|twiceler|UbiCrawler|uptimerobot|URI::Fetch|urllib|User-Agent|VoidEYE|VoilaBot|WBSearchBot|Web Image Collector|Web Sucker|WebAuto|WebCopier|WebCopier|WebFetch|WebGo IS|WebLeacher|WebReaper|WebSauger|Website eXtractor|Website Quester|WebStripper|WebStripper|WebWhacker|WebZIP|WebZIP|Wells Search II|WEP Search|Widow|winHTTP|WWWOFFLE|Xaldon WebSpider|Xenu|yacybot|yandex|YandexBot|YandexImages|yBot|YesupBot|YodaoBot|yolinkBot|youdao|Zao|Zealbot|Zeus|ZyBORG
diff --git a/gistfile1.txt → bot-blocking-expression b/gistfile1.txt → bot-blocking-expression
diff --git a/gistfile1.txt b/gistfile1.txt
@@ -0,0 +1 @@
+^$|EasouSpider|Add Catalog|PaperLiBot|Spiceworks|ZumBot|RU_Bot|Wget|Java/1.7.0_25|Slurp|FunWebProducts|80legs|Aboundex|AcoiRobot|Acoon Robot|AhrefsBot|aihit|AlkalineBOT|AnzwersCrawl|Arachnoidea|ArchitextSpider|archive|Autonomy Spider|Baiduspider|BecomeBot|benderthewebrobot|BlackWidow|Bork-edition|Bot mailto:[email protected]|botje|catchbot|changedetection|Charlotte|ChinaClaw|commoncrawl|ConveraCrawler|Covario|crawler|curl|Custo|data mining development project|DigExt|DISCo|discobot|discoveryengine|DOC|DoCoMo|DotBot|Download Demon|Download Ninja|eCatch|EirGrabber|EmailSiphon|EmailWolf|eurobot|Exabot|Express WebPictures|ExtractorPro|EyeNetIE|Ezooms|Fetch|Fetch API|filterdb|findfiles|findlinks|FlashGet|flightdeckreports|FollowSite Bot|Gaisbot|genieBot|GetRight|GetWeb!|gigablast|Gigabot|Go-Ahead-Got-It|Go!Zilla|GrabNet|Grafula|GT::WWW|hailoo|heritrix|HMView|houxou|HTTP::Lite|HTTrack|ia_archiver|IBM EVV|id-search|IDBot|Image Stripper|Image Sucker|Indy Library|InterGET|Internet Ninja|internetmemory|ISC Systems iRc Search 2.1|Jakarta|JetCar|JOC Web Spider|k2spider|larbin|larbin|LeechFTP|libghttp|libwww|libwww-perl|linko|LinkWalker|lwp-trivial|Mass Downloader|metadatalabs|MFC_Tear_Sample|Microsoft URL Control|MIDown tool|Missigua|Missigua Locator|Mister PiX|MJ12bot|MOREnet|MSIECrawler|msnbot|naver|Navroad|NearSite|Net Vampire|NetAnts|NetSpider|NetZIP|NextGenSearchBot|NPBot|Nutch|Octopus|Offline Explorer|Offline Navigator|omni-explorer|PageGrabber|panscient|panscient.com|Papa Foto|pavuk|pcBrowser|PECL::HTTP|PHP/|PHPCrawl|picsearch|pipl|pmoz|PredictYourBabySearchToolbar|RealDownload|Referrer Karma|ReGet|reverseget|rogerbot|ScoutJet|SearchBot|seexie|seoprofiler|Servage Robot|SeznamBot|shopwiki|sindice|sistrix|SiteSnagger|SiteSnagger|smart.apnoti.com|SmartDownload|Snoopy|Sosospider|spbot|suggybot|SuperBot|SuperHTTP|SuperPagesUrlVerifyBot|Surfbot|SurveyBot|SurveyBot|swebot|Synapse|Tagoobot|tAkeOut|Teleport|Teleport Pro|TeleportPro|TweetmemeBot|TwengaBot|twiceler|UbiCrawler|uptimerobot|URI::Fetch|urllib|User-Agent|VoidEYE|VoilaBot|WBSearchBot|Web Image Collector|Web Sucker|WebAuto|WebCopier|WebCopier|WebFetch|WebGo IS|WebLeacher|WebReaper|WebSauger|Website eXtractor|Website Quester|WebStripper|WebStripper|WebWhacker|WebZIP|WebZIP|Wells Search II|WEP Search|Widow|winHTTP|WWWOFFLE|Xaldon WebSpider|Xenu|yacybot|yandex|YandexBot|YandexImages|yBot|YesupBot|YodaoBot|yolinkBot|youdao|Zao|Zealbot|Zeus|ZyBORG
Original file line number	Diff line number	Diff line change
		@@ -1 +1 @@
		^$\|EasouSpider\|Add Catalog\|PaperLiBot\|Spiceworks\|ZumBot\|RU_Bot\|Wget\|Java/1.7.0_25\|Slurp\|FunWebProducts\|80legs\|Aboundex\|AcoiRobot\|Acoon Robot\|AhrefsBot\|aihit\|AlkalineBOT\|AnzwersCrawl\|Arachnoidea\|ArchitextSpider\|archive\|Autonomy Spider\|Baiduspider\|BecomeBot\|benderthewebrobot\|BlackWidow\|Bork-edition\|Bot mailto:[email protected]\|botje\|catchbot\|changedetection\|Charlotte\|ChinaClaw\|commoncrawl\|ConveraCrawler\|Covario\|crawler\|curl\|Custo\|data mining development project\|DigExt\|DISCo\|discobot\|discoveryengine\|DOC\|DoCoMo\|DotBot\|Download Demon\|Download Ninja\|eCatch\|EirGrabber\|EmailSiphon\|EmailWolf\|eurobot\|Exabot\|Express WebPictures\|ExtractorPro\|EyeNetIE\|Ezooms\|Fetch\|Fetch API\|filterdb\|findfiles\|findlinks\|FlashGet\|flightdeckreports\|FollowSite Bot\|Gaisbot\|genieBot\|GetRight\|GetWeb!\|gigablast\|Gigabot\|Go-Ahead-Got-It\|Go!Zilla\|GrabNet\|Grafula\|GT::WWW\|hailoo\|heritrix\|HMView\|houxou\|HTTP::Lite\|HTTrack\|ia_archiver\|IBM EVV\|id-search\|IDBot\|Image Stripper\|Image Sucker\|Indy Library\|InterGET\|Internet Ninja\|internetmemory\|ISC Systems iRc Search 2.1\|Jakarta\|JetCar\|JOC Web Spider\|k2spider\|larbin\|larbin\|LeechFTP\|libghttp\|libwww\|libwww-perl\|linko\|LinkWalker\|lwp-trivial\|Mass Downloader\|metadatalabs\|MFC_Tear_Sample\|Microsoft URL Control\|MIDown tool\|Missigua\|Missigua Locator\|Mister PiX\|MJ12bot\|MOREnet\|MSIECrawler\|msnbot\|naver\|Navroad\|NearSite\|Net Vampire\|NetAnts\|NetSpider\|NetZIP\|NextGenSearchBot\|NPBot\|Nutch\|Octopus\|Offline Explorer\|Offline Navigator\|omni-explorer\|PageGrabber\|panscient\|panscient.com\|Papa Foto\|pavuk\|pcBrowser\|PECL::HTTP\|PHP/\|PHPCrawl\|picsearch\|pipl\|pmoz\|PredictYourBabySearchToolbar\|RealDownload\|Referrer Karma\|ReGet\|reverseget\|rogerbot\|ScoutJet\|SearchBot\|seexie\|seoprofiler\|Servage Robot\|SeznamBot\|shopwiki\|sindice\|sistrix\|SiteSnagger\|SiteSnagger\|smart.apnoti.com\|SmartDownload\|Snoopy\|Sosospider\|spbot\|suggybot\|SuperBot\|SuperHTTP\|SuperPagesUrlVerifyBot\|Surfbot\|SurveyBot\|SurveyBot\|swebot\|Synapse\|Tagoobot\|tAkeOut\|Teleport\|Teleport Pro\|TeleportPro\|TweetmemeBot\|TwengaBot\|twiceler\|UbiCrawler\|uptimerobot\|URI::Fetch\|urllib\|User-Agent\|VoidEYE\|VoilaBot\|WBSearchBot\|Web Image Collector\|Web Sucker\|WebAuto\|WebCopier\|WebCopier\|WebFetch\|WebGo IS\|WebLeacher\|WebReaper\|WebSauger\|Website eXtractor\|Website Quester\|WebStripper\|WebStripper\|WebWhacker\|WebZIP\|WebZIP\|Wells Search II\|WEP Search\|Widow\|winHTTP\|WWWOFFLE\|Xaldon WebSpider\|Xenu\|yacybot\|yandex\|YandexBot\|YandexImages\|yBot\|YesupBot\|YodaoBot\|yolinkBot\|youdao\|Zao\|Zealbot\|Zeus\|ZyBORG
		^$\|EasouSpider\|Add Catalog\|PaperLiBot\|Spiceworks\|ZumBot\|RU_Bot\|Wget\|Java/1.7.0_25\|Slurp\|FunWebProducts\|80legs\|Aboundex\|AcoiRobot\|Acoon Robot\|AhrefsBot\|aihit\|AlkalineBOT\|AnzwersCrawl\|Arachnoidea\|ArchitextSpider\|archive\|Autonomy Spider\|Baiduspider\|BecomeBot\|benderthewebrobot\|BlackWidow\|Bork-edition\|Bot mailto:[email protected]\|botje\|catchbot\|changedetection\|Charlotte\|ChinaClaw\|commoncrawl\|ConveraCrawler\|Covario\|crawler\|curl\|Custo\|data mining development project\|DigExt\|DISCo\|discobot\|discoveryengine\|DOC\|DoCoMo\|DotBot\|Download Demon\|Download Ninja\|eCatch\|EirGrabber\|EmailSiphon\|EmailWolf\|eurobot\|Exabot\|Express WebPictures\|ExtractorPro\|EyeNetIE\|Ezooms\|Fetch\|Fetch API\|filterdb\|findfiles\|findlinks\|FlashGet\|flightdeckreports\|FollowSite Bot\|Gaisbot\|genieBot\|GetRight\|GetWeb!\|gigablast\|Gigabot\|Go-Ahead-Got-It\|Go!Zilla\|GrabNet\|Grafula\|GT::WWW\|hailoo\|heritrix\|HMView\|houxou\|HTTP::Lite\|HTTrack\|ia_archiver\|IBM EVV\|id-search\|IDBot\|Image Stripper\|Image Sucker\|Indy Library\|InterGET\|Internet Ninja\|internetmemory\|ISC Systems iRc Search 2.1\|JetCar\|JOC Web Spider\|k2spider\|larbin\|larbin\|LeechFTP\|libghttp\|libwww\|libwww-perl\|linko\|LinkWalker\|lwp-trivial\|Mass Downloader\|metadatalabs\|MFC_Tear_Sample\|Microsoft URL Control\|MIDown tool\|Missigua\|Missigua Locator\|Mister PiX\|MJ12bot\|MOREnet\|MSIECrawler\|msnbot\|naver\|Navroad\|NearSite\|Net Vampire\|NetAnts\|NetSpider\|NetZIP\|NextGenSearchBot\|NPBot\|Nutch\|Octopus\|Offline Explorer\|Offline Navigator\|omni-explorer\|PageGrabber\|panscient\|panscient.com\|Papa Foto\|pavuk\|pcBrowser\|PECL::HTTP\|PHP/\|PHPCrawl\|picsearch\|pipl\|pmoz\|PredictYourBabySearchToolbar\|RealDownload\|Referrer Karma\|ReGet\|reverseget\|rogerbot\|ScoutJet\|SearchBot\|seexie\|seoprofiler\|Servage Robot\|SeznamBot\|shopwiki\|sindice\|sistrix\|SiteSnagger\|SiteSnagger\|smart.apnoti.com\|SmartDownload\|Snoopy\|Sosospider\|spbot\|suggybot\|SuperBot\|SuperHTTP\|SuperPagesUrlVerifyBot\|Surfbot\|SurveyBot\|SurveyBot\|swebot\|Synapse\|Tagoobot\|tAkeOut\|Teleport\|Teleport Pro\|TeleportPro\|TweetmemeBot\|TwengaBot\|twiceler\|UbiCrawler\|uptimerobot\|URI::Fetch\|urllib\|User-Agent\|VoidEYE\|VoilaBot\|WBSearchBot\|Web Image Collector\|Web Sucker\|WebAuto\|WebCopier\|WebCopier\|WebFetch\|WebGo IS\|WebLeacher\|WebReaper\|WebSauger\|Website eXtractor\|Website Quester\|WebStripper\|WebStripper\|WebWhacker\|WebZIP\|WebZIP\|Wells Search II\|WEP Search\|Widow\|winHTTP\|WWWOFFLE\|Xaldon WebSpider\|Xenu\|yacybot\|yandex\|YandexBot\|YandexImages\|yBot\|YesupBot\|YodaoBot\|yolinkBot\|youdao\|Zao\|Zealbot\|Zeus\|ZyBORG
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1 @@
		^$\|EasouSpider\|Add Catalog\|PaperLiBot\|Spiceworks\|ZumBot\|RU_Bot\|Wget\|Java/1.7.0_25\|Slurp\|FunWebProducts\|80legs\|Aboundex\|AcoiRobot\|Acoon Robot\|AhrefsBot\|aihit\|AlkalineBOT\|AnzwersCrawl\|Arachnoidea\|ArchitextSpider\|archive\|Autonomy Spider\|Baiduspider\|BecomeBot\|benderthewebrobot\|BlackWidow\|Bork-edition\|Bot mailto:[email protected]\|botje\|catchbot\|changedetection\|Charlotte\|ChinaClaw\|commoncrawl\|ConveraCrawler\|Covario\|crawler\|curl\|Custo\|data mining development project\|DigExt\|DISCo\|discobot\|discoveryengine\|DOC\|DoCoMo\|DotBot\|Download Demon\|Download Ninja\|eCatch\|EirGrabber\|EmailSiphon\|EmailWolf\|eurobot\|Exabot\|Express WebPictures\|ExtractorPro\|EyeNetIE\|Ezooms\|Fetch\|Fetch API\|filterdb\|findfiles\|findlinks\|FlashGet\|flightdeckreports\|FollowSite Bot\|Gaisbot\|genieBot\|GetRight\|GetWeb!\|gigablast\|Gigabot\|Go-Ahead-Got-It\|Go!Zilla\|GrabNet\|Grafula\|GT::WWW\|hailoo\|heritrix\|HMView\|houxou\|HTTP::Lite\|HTTrack\|ia_archiver\|IBM EVV\|id-search\|IDBot\|Image Stripper\|Image Sucker\|Indy Library\|InterGET\|Internet Ninja\|internetmemory\|ISC Systems iRc Search 2.1\|Jakarta\|JetCar\|JOC Web Spider\|k2spider\|larbin\|larbin\|LeechFTP\|libghttp\|libwww\|libwww-perl\|linko\|LinkWalker\|lwp-trivial\|Mass Downloader\|metadatalabs\|MFC_Tear_Sample\|Microsoft URL Control\|MIDown tool\|Missigua\|Missigua Locator\|Mister PiX\|MJ12bot\|MOREnet\|MSIECrawler\|msnbot\|naver\|Navroad\|NearSite\|Net Vampire\|NetAnts\|NetSpider\|NetZIP\|NextGenSearchBot\|NPBot\|Nutch\|Octopus\|Offline Explorer\|Offline Navigator\|omni-explorer\|PageGrabber\|panscient\|panscient.com\|Papa Foto\|pavuk\|pcBrowser\|PECL::HTTP\|PHP/\|PHPCrawl\|picsearch\|pipl\|pmoz\|PredictYourBabySearchToolbar\|RealDownload\|Referrer Karma\|ReGet\|reverseget\|rogerbot\|ScoutJet\|SearchBot\|seexie\|seoprofiler\|Servage Robot\|SeznamBot\|shopwiki\|sindice\|sistrix\|SiteSnagger\|SiteSnagger\|smart.apnoti.com\|SmartDownload\|Snoopy\|Sosospider\|spbot\|suggybot\|SuperBot\|SuperHTTP\|SuperPagesUrlVerifyBot\|Surfbot\|SurveyBot\|SurveyBot\|swebot\|Synapse\|Tagoobot\|tAkeOut\|Teleport\|Teleport Pro\|TeleportPro\|TweetmemeBot\|TwengaBot\|twiceler\|UbiCrawler\|uptimerobot\|URI::Fetch\|urllib\|User-Agent\|VoidEYE\|VoilaBot\|WBSearchBot\|Web Image Collector\|Web Sucker\|WebAuto\|WebCopier\|WebCopier\|WebFetch\|WebGo IS\|WebLeacher\|WebReaper\|WebSauger\|Website eXtractor\|Website Quester\|WebStripper\|WebStripper\|WebWhacker\|WebZIP\|WebZIP\|Wells Search II\|WEP Search\|Widow\|winHTTP\|WWWOFFLE\|Xaldon WebSpider\|Xenu\|yacybot\|yandex\|YandexBot\|YandexImages\|yBot\|YesupBot\|YodaoBot\|yolinkBot\|youdao\|Zao\|Zealbot\|Zeus\|ZyBORG