Skip to content

Instantly share code, notes, and snippets.

@vancetran

vancetran/wget-static.md

Last active January 5, 2020 10:26
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save vancetran/c6201bf99829fbe4b20c to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save vancetran/c6201bf99829fbe4b20c to your computer and use it in GitHub Desktop.
Download ZIP

Revisions

  1. vancetran revised this gist Jun 15, 2018. 1 changed file with 23 additions and 1 deletion.
    24 changes: 23 additions & 1 deletion wget-static.md
    View file
    Original file line number Diff line number Diff line change
    @@ -10,4 +10,26 @@ wget -P /destination/ -mpck --user-agent="" -e robots=off --random-wait -E http:
    [source](http://stackoverflow.com/questions/18516411/wget-mirror-html-only)
    ```sh
    wget -P /destination/ -mpck --user-agent="" -e robots=off --random-wait -R gif,jpg,jpeg,png,pdf,ppt,GIF,JPG,JPEG,PNG,PDF,PPT -E http://example.com/
    ```
    ```


    # Wget fun at work


    ## with a 30 second wait between requests…

    wget -mkEpnp --wait=30 -U "Mozilla/5.0 (X11; U; Linux; en-US; rv:1.9.1.16) Gecko/20110929 Firefox/3.5.16" https://join-mosaic.squarespace.com

    ## Basics

    http://www.linuxjournal.com/content/downloading-entire-web-site-wget

    ## Wait x seconds between requests

    https://wiki.hackzine.org/scripts/wget-grab-website.html

    ## Another

    https://swsblog.stanford.edu/blog/creating-static-copy-website

    wget -mpck -e robots=off --wait 5 -E https://join-mosaic.squarespace.com
  2. vancetran revised this gist Apr 28, 2015. 1 changed file with 2 additions and 2 deletions.
    4 changes: 2 additions & 2 deletions wget-static.md
    View file
    Original file line number Diff line number Diff line change
    @@ -6,8 +6,8 @@
    wget -P /destination/ -mpck --user-agent="" -e robots=off --random-wait -E http://example.com/
    ```

    ## Without Images
    ## Without Images, PPT, PDF
    [source](http://stackoverflow.com/questions/18516411/wget-mirror-html-only)
    ```sh
    wget -P /destination/ -mpck --user-agent="" -e robots=off --random-wait -R gif,jpg,jpeg,png,pdf -E http://example.com/
    wget -P /destination/ -mpck --user-agent="" -e robots=off --random-wait -R gif,jpg,jpeg,png,pdf,ppt,GIF,JPG,JPEG,PNG,PDF,PPT -E http://example.com/
    ```
  3. vancetran revised this gist Apr 28, 2015. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion wget-static.md
    View file
    Original file line number Diff line number Diff line change
    @@ -9,5 +9,5 @@ wget -P /destination/ -mpck --user-agent="" -e robots=off --random-wait -E http:
    ## Without Images
    [source](http://stackoverflow.com/questions/18516411/wget-mirror-html-only)
    ```sh
    wget -P /destination/ -mpck --user-agent="" -e robots=off --random-wait -R gif,jpg,png,pdf -E http://example.com/
    wget -P /destination/ -mpck --user-agent="" -e robots=off --random-wait -R gif,jpg,jpeg,png,pdf -E http://example.com/
    ```
  4. vancetran revised this gist Apr 28, 2015. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion wget-static.md
    View file
    Original file line number Diff line number Diff line change
    @@ -9,5 +9,5 @@ wget -P /destination/ -mpck --user-agent="" -e robots=off --random-wait -E http:
    ## Without Images
    [source](http://stackoverflow.com/questions/18516411/wget-mirror-html-only)
    ```sh
    wget -P /destination/ -mpck --user-agent="" -e robots=off --random-wait -R gif,jpg,pdf -E http://example.com/
    wget -P /destination/ -mpck --user-agent="" -e robots=off --random-wait -R gif,jpg,png,pdf -E http://example.com/
    ```
  5. vancetran revised this gist Apr 28, 2015. 1 changed file with 2 additions and 2 deletions.
    4 changes: 2 additions & 2 deletions wget-static.md
    View file
    Original file line number Diff line number Diff line change
    @@ -3,11 +3,11 @@
    ## Make a static copy of a dynamic site, including images
    [via Stanford](https://swsblog.stanford.edu/blog/creating-static-copy-website)
    ```sh
    wget -P /destination/ -mpck --user-agent="" -e robots=off --random-wait -E htt://example.com/
    wget -P /destination/ -mpck --user-agent="" -e robots=off --random-wait -E http://example.com/
    ```

    ## Without Images
    [source](http://stackoverflow.com/questions/18516411/wget-mirror-html-only)
    ```sh
    wget -P /destination/ -mpck --user-agent="" -e robots=off --random-wait -R gif,jpg,pdf -E htt://example.com/
    wget -P /destination/ -mpck --user-agent="" -e robots=off --random-wait -R gif,jpg,pdf -E http://example.com/
    ```
  6. vancetran revised this gist Apr 28, 2015. 1 changed file with 8 additions and 2 deletions.
    10 changes: 8 additions & 2 deletions wget-static.md
    View file
    Original file line number Diff line number Diff line change
    @@ -1,7 +1,13 @@
    # Make a static copy of a dynamic site
    # [Wget recipes](http://www.labnol.org/software/wget-command-examples/28750/)

    ## Make a static copy of a dynamic site, including images
    [via Stanford](https://swsblog.stanford.edu/blog/creating-static-copy-website)
    ```sh
    wget -P /destination/ -mpck --user-agent="" -e robots=off --random-wait -E htt://example.com/
    ```

    ## Without Images
    [source](http://stackoverflow.com/questions/18516411/wget-mirror-html-only)
    ```sh
    wget -P /path/to/destination/directory/ -mpck --user-agent="" -e robots=off --wait 1 -E https://www.example.com/
    wget -P /destination/ -mpck --user-agent="" -e robots=off --random-wait -R gif,jpg,pdf -E htt://example.com/
    ```
  7. vancetran renamed this gist Sep 10, 2014. 1 changed file with 0 additions and 0 deletions.
    0 wget-static → wget-static.md
    View file
    File renamed without changes.
  8. vancetran renamed this gist Sep 10, 2014. 1 changed file with 0 additions and 0 deletions.
    0 gistfile1.md → wget-static
    View file
    File renamed without changes.
  9. vancetran revised this gist Sep 10, 2014. No changes.
  10. vancetran created this gist Aug 7, 2014.
    7 changes: 7 additions & 0 deletions gistfile1.md
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,7 @@
    # Make a static copy of a dynamic site

    [via Stanford](https://swsblog.stanford.edu/blog/creating-static-copy-website)

    ```sh
    wget -P /path/to/destination/directory/ -mpck --user-agent="" -e robots=off --wait 1 -E https://www.example.com/
    ```