Skip to content

Instantly share code, notes, and snippets.

@erikhansen

erikhansen/load_all_sitemap_urls.sh

Last active May 20, 2024 03:21
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save erikhansen/c1b88dce68c8cd76b042c1e1b82b0703 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save erikhansen/c1b88dce68c8cd76b042c1e1b82b0703 to your computer and use it in GitHub Desktop.
Download ZIP

Revisions

  1. erikhansen revised this gist Mar 1, 2019. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion load_all_sitemap_urls.sh
    View file
    Original file line number Diff line number Diff line change
    @@ -2,7 +2,7 @@
    # This script crawls all urls in a /sitemap.xml file and loads them, effectively priming the cache
    # Usage: ./warm_cache.sh www.example.com

    time wget --quiet http://$1/sitemap.xml --output-document - | \
    time wget --quiet https://$1/sitemap.xml --output-document - | \
    egrep -o "https?://[^<]+" | \
    grep $1 | \
    grep -v "jpg" | \
  2. erikhansen created this gist Mar 1, 2019.
    9 changes: 9 additions & 0 deletions load_all_sitemap_urls.sh
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,9 @@
    #!/bin/bash
    # This script crawls all urls in a /sitemap.xml file and loads them, effectively priming the cache
    # Usage: ./warm_cache.sh www.example.com

    time wget --quiet http://$1/sitemap.xml --output-document - | \
    egrep -o "https?://[^<]+" | \
    grep $1 | \
    grep -v "jpg" | \
    xargs -i -d '\n' curl --output /dev/null --silent --write-out '%{http_code} %{time_total}ms %{url_effective} \n' {}