Skip to content

Instantly share code, notes, and snippets.

View daviddabney's full-sized avatar

111110000022222299999 daviddabney

View GitHub Profile
@daviddabney
daviddabney / get-image-urls.js
Created September 25, 2024 17:12 — forked from tobek/get-image-urls.js
Save images from chrome inspector/dev tools network tab
/* open up chrome dev tools (Menu > More tools > Developer tools)
* go to network tab, refresh the page, wait for images to load (on some sites you may have to scroll down to the images for them to start loading)
* right click/ctrl click on any entry in the network log, select Copy > Copy All as HAR
* open up JS console and enter: var har = [paste]
* (pasting could take a while if there's a lot of requests)
* paste the following JS code into the console
* copy the output, paste into a text file
* open up a terminal in same directory as text file, then: wget -i [that file]
*/
@daviddabney
daviddabney / ffmpeg-i-have-used.md
Created February 27, 2021 05:07 — forked from rch850/ffmpeg-i-have-used.md
ffmpeg command history
@daviddabney
daviddabney / ffmpeg.md
Created February 27, 2021 05:06 — forked from steven2358/ffmpeg.md
FFmpeg cheat sheet
@daviddabney
daviddabney / example.zsh
Created June 21, 2019 21:17
Recursively download website with wget
# add function to .zshrc
function download-web() {
wget -r -nH --no-parent --reject='index.html*' "$@" ;
}
# then run
download-web http://example.com/dl/
@daviddabney
daviddabney / wget.sh
Created September 29, 2018 19:44 — forked from Dammmien/wget.sh
wget cheat sheet
# POST a JSON file and redirect output to stdout
wget -q -O - --header="Content-Type:application/json" --post-file=foo.json http://127.0.0.1
# Download a complete website
wget -m -r -linf -k -p -q -E -e robots=off http://127.0.0.1
# But it may be sufficient
wget -mpk http://127.0.0.1
# Download all images of a website

Using wget to scrape a site

You can easily scrape (or download) a site with a CLI tool called wget. It's available for Linux, Mac and Windows.

Installation

I recommend using Homebrew, especially if you're on a Mac, to install it.

brew install wget

Scraping a site

@daviddabney
daviddabney / _verify-repair-permissions-disk.md
Created August 7, 2018 05:24 — forked from bzerangue/_verify-repair-permissions-disk.md
Mac OS X Utilities via Terminal: (Verify and Repair: Disk Permissions AND Disk / Software Update / TimeMachine)

Verify and Repair Disk Permissions via Terminal (Mac OS X)

Verify Permissions

diskutil verifyPermissions /

Repair Permissions

diskutil repairPermissions /

List files modified in the last hour - find command shell find linux unix

Use find to list files modified within the last hour:

$ find . -mtime -1
the . is the search path

-mtime time parameter

@daviddabney
daviddabney / wget-static.md
Created September 29, 2017 21:26 — forked from vancetran/wget-static.md
Wget recipes

Make a static copy of a dynamic site, including images

via Stanford

wget -P /destination/ -mpck --user-agent="" -e robots=off --random-wait -E http://example.com/

Without Images, PPT, PDF

source

@daviddabney
daviddabney / wget
Created September 29, 2017 21:14 — forked from jasperf/wget
Wget examples
#Spider Websites with Wget – 20 Practical Examples
Wget is extremely powerful, but like with most other command line programs, the plethora of options it supports can be intimidating to new users. Thus what we have here are a collection of wget commands that you can use to accomplish common tasks from downloading single files to mirroring entire websites. It will help if you can read through the wget manual but for the busy souls, these commands are ready to execute.
1. Download a single file from the Internet
wget http://example.com/file.iso
2. Download a file but save it locally under a different name
wget ‐‐output-document=filename.html example.com