Skip to content

Instantly share code, notes, and snippets.

@ihfbib

ihfbib/facebook_crawler.sh

Forked from landsman/facebook_crawler.sh
Created February 19, 2024 01:10
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save ihfbib/09db36b176f796c03d3c653663ce404f to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save ihfbib/09db36b176f796c03d3c653663ce404f to your computer and use it in GitHub Desktop.
Download ZIP
Cloudflare IP address whitelist for Google Crawlers, Google Ads: https://developers.google.com/search/docs/crawling-indexing/verifying-googlebot
Raw
facebook_crawler.sh
#!/bin/bash
#
# README: run this script like this: sh facebook_crawler.sh > fb_ips.csv
#
# doc: https://developers.facebook.com/docs/sharing/webmasters/crawler/
#
# Run the whois command and store the output in a variable
whois_output=$(whois -h whois.radb.net -- '-i origin AS32934')
# Use grep to extract lines starting with "route"
routes=$(echo "$whois_output" | grep '^route')
# Use grep and regular expression to extract IPv4 addresses
ipv4only=$(echo "$routes" | grep -Eo '[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}')
# Print the formatted IPv4 addresses
echo "$ipv4only"
Raw
google_ads_crawlers.csv
We can make this file beautiful and searchable if this error is corrected: No commas found in this CSV file in line 0.
192.178.17.0/27
209.85.238.0/27
209.85.238.128/27
209.85.238.160/27
209.85.238.192/27
209.85.238.224/27
209.85.238.32/27
209.85.238.64/27
209.85.238.96/27
66.249.87.0/27
66.249.87.128/27
66.249.87.160/27
66.249.87.192/27
66.249.87.224/27
66.249.87.32/27
66.249.87.64/27
66.249.87.96/27
66.249.89.0/27
66.249.89.128/27
66.249.89.160/27
66.249.89.192/27
66.249.89.224/27
66.249.89.32/27
66.249.89.64/27
66.249.89.96/27
66.249.90.0/27
66.249.90.128/27
66.249.90.160/27
66.249.90.192/27
66.249.90.224/27
66.249.90.32/27
66.249.90.64/27
66.249.90.96/27
66.249.91.0/27
66.249.91.128/27
66.249.91.160/27
66.249.91.192/27
66.249.91.224/27
66.249.91.32/27
66.249.91.64/27
66.249.91.96/27
66.249.92.0/27
66.249.92.128/27
66.249.92.160/27
66.249.92.192/27
66.249.92.32/27
66.249.92.64/27
66.249.92.96/27
72.14.199.0/27
72.14.199.128/27
72.14.199.160/27
72.14.199.192/27
72.14.199.224/27
72.14.199.32/27
72.14.199.64/27
72.14.199.96/27
74.125.148.0/27
74.125.148.128/27
74.125.148.160/27
74.125.148.192/27
74.125.148.224/27
74.125.148.32/27
74.125.148.64/27
74.125.148.96/27
74.125.149.0/27
74.125.149.128/27
74.125.149.160/27
74.125.149.192/27
74.125.149.224/27
74.125.149.32/27
74.125.149.64/27
74.125.149.96/27
74.125.150.0/27
74.125.150.32/27
74.125.150.64/27
74.125.151.0/27
74.125.151.128/27
74.125.151.160/27
74.125.151.192/27
74.125.151.224/27
74.125.151.32/27
74.125.151.64/27
74.125.151.96/27
74.125.216.0/27
74.125.216.128/27
74.125.216.160/27
74.125.216.192/27
74.125.216.224/27
74.125.216.32/27
74.125.216.64/27
74.125.216.96/27
74.125.217.0/27
74.125.217.128/27
74.125.217.32/27
74.125.217.64/27
74.125.217.96/27
74.125.218.0/27
74.125.218.128/27
74.125.218.160/27
74.125.218.192/27
74.125.218.32/27
74.125.218.64/27
74.125.218.96/27
74.125.219.0/27
Raw
google_bot_crawlers.csv
We can make this file beautiful and searchable if this error is corrected: No commas found in this CSV file in line 0.
192.178.5.0/27
34.100.182.96/28
34.101.50.144/28
34.118.254.0/28
34.118.66.0/28
34.126.178.96/28
34.146.150.144/28
34.147.110.144/28
34.151.74.144/28
34.152.50.64/28
34.154.114.144/28
34.155.98.32/28
34.165.18.176/28
34.175.160.64/28
34.176.130.16/28
34.22.85.0/27
34.64.82.64/28
34.65.242.112/28
34.80.50.80/28
34.88.194.0/28
34.89.10.80/28
34.89.198.80/28
34.96.162.48/28
35.247.243.240/28
66.249.64.0/27
66.249.64.128/27
66.249.64.160/27
66.249.64.192/27
66.249.64.224/27
66.249.64.32/27
66.249.64.64/27
66.249.64.96/27
66.249.65.0/27
66.249.65.128/27
66.249.65.160/27
66.249.65.192/27
66.249.65.224/27
66.249.65.32/27
66.249.65.64/27
66.249.65.96/27
66.249.66.0/27
66.249.66.128/27
66.249.66.160/27
66.249.66.192/27
66.249.66.32/27
66.249.66.64/27
66.249.66.96/27
66.249.68.0/27
66.249.68.32/27
66.249.68.64/27
66.249.69.0/27
66.249.69.128/27
66.249.69.160/27
66.249.69.192/27
66.249.69.224/27
66.249.69.32/27
66.249.69.64/27
66.249.69.96/27
66.249.70.0/27
66.249.70.128/27
66.249.70.160/27
66.249.70.192/27
66.249.70.224/27
66.249.70.32/27
66.249.70.64/27
66.249.70.96/27
66.249.71.0/27
66.249.71.128/27
66.249.71.160/27
66.249.71.192/27
66.249.71.224/27
66.249.71.32/27
66.249.71.64/27
66.249.71.96/27
66.249.72.0/27
66.249.72.128/27
66.249.72.160/27
66.249.72.192/27
66.249.72.224/27
66.249.72.32/27
66.249.72.64/27
66.249.72.96/27
66.249.73.0/27
66.249.73.128/27
66.249.73.160/27
66.249.73.192/27
66.249.73.224/27
66.249.73.32/27
66.249.73.64/27
66.249.73.96/27
66.249.74.0/27
66.249.74.128/27
66.249.74.32/27
66.249.74.64/27
66.249.74.96/27
66.249.75.0/27
66.249.75.128/27
66.249.75.160/27
66.249.75.192/27
66.249.75.224/27
66.249.75.32/27
66.249.75.64/27
66.249.75.96/27
66.249.76.0/27
66.249.76.128/27
66.249.76.160/27
66.249.76.192/27
66.249.76.224/27
66.249.76.32/27
66.249.76.64/27
66.249.76.96/27
66.249.77.0/27
66.249.77.128/27
66.249.77.160/27
66.249.77.192/27
66.249.77.32/27
66.249.77.64/27
66.249.77.96/27
66.249.78.0/27
66.249.79.0/27
66.249.79.128/27
66.249.79.160/27
66.249.79.192/27
66.249.79.224/27
66.249.79.32/27
66.249.79.64/27
66.249.79.96/27
Raw
parse_ips.js
// copy-paste here file from https://developers.google.com/search/docs/crawling-indexing/verifying-googlebot
const crawlers = {};
const prefixes = crawlers.prefixes;
prefixes.map(i => {
if (i.ipv4Prefix) {
console.log(i.ipv4Prefix);
}
});
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment