benneic · October 16, 2019 01:38 · Oct 16, 2019 · Oct 16, 2019 · Oct 1, 2019 · Sep 19, 2019
diff --git a/bash.md b/bash.md
@@ -94,5 +94,5 @@ cat access.log | awk '$11 !~ /google|bing|yahoo|yandex|mywebsite.com/' | awk '{p
 
 ### list user agents
 ```bash
-cat access.log | awk '{print $12 $13}' | tr -d '"' | sort | uniq -c | sort -rn | head -1000
+cat access.log | awk '{print $14}' | tr -d '"' | sort | uniq -c | sort -rn | head -1000
 ```
diff --git a/bash.md b/bash.md
@@ -40,4 +40,59 @@ ps aux --sort=start_time | grep 'celery worker' | grep 'celery' | awk '{print $2
 ### Delete all git branches on remote which have been merged
 ```bash
 git branch -r --merged | grep -v master | sed 's/origin\///' | xargs -n 1 git push --delete origin
+```
+
+## Nginx (Apache format) log files
+
+### top 20 URLs from the last 5000 hits
+```bash
+tail -5000 access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -20
+tail -5000 access.log | awk '{freq[$7]++} END {for (x in freq) {print freq[x], x}}' | sort -rn | head -20
+```
+
+### top 20 URLS excluding POST data from the last 5000 hits
+```bash
+tail -5000 access.log | awk -F"[ ?]" '{print $7}' | sort | uniq -c | sort -rn | head -20
+tail -5000 access.log | awk -F"[ ?]" '{freq[$7]++} END {for (x in freq) {print freq[x], x}}' | sort -rn | head -20
+```
+
+### top 20 IPs from the last 5000 hits
+```bash
+tail -5000 access.log | awk '{print $1}' | sort | uniq -c | sort -rn | head -20
+tail -5000 access.log | awk '{freq[$1]++} END {for (x in freq) {print freq[x], x}}' | sort -rn | head -20
+```
+
+### top 20 URLs requested from a certain ip from the last 5000 hits
+```bash
+IP=1.2.3.4
+tail -5000 access.log | grep $IP | awk '{print $7}' | sort | uniq -c | sort -rn | head -20
+tail -5000 access.log | awk -v ip=$IP ' $1 ~ ip {freq[$7]++} END {for (x in freq) {print freq[x], x}}' | sort -rn | head -20
+```
+
+### top 20 URLS requested from a certain ip excluding, excluding POST data, from the last 5000 hits
+```bash
+IP=1.2.3.4
+tail -5000 access.log | fgrep $IP | awk -F "[ ?]" '{print $7}' | sort | uniq -c | sort -rn | head -20
+tail -5000 access.log | awk -F"[ ?]" -v ip=$IP ' $1 ~ ip {freq[$7]++} END {for (x in freq) {print freq[x], x}}' | sort -rn | head -20
+```
+
+### sum of data (in MB) transferred 
+```bash
+cat access.log | awk '{sum+=$10} END {print sum/1048576}'
+```
+
+### top 20 referrers from the last 5000 hits
+```bash
+tail -5000 access.log | awk '{print $11}' | tr -d '"' | sort | uniq -c | sort -rn | head -20
+tail -5000 access.log | awk '{freq[$11]++} END {for (x in freq) {print freq[x], x}}' | tr -d '"' | sort -rn | head -20
+```
+
+### list of bad referers
+```bash
+cat access.log | awk '$11 !~ /google|bing|yahoo|yandex|mywebsite.com/' | awk '{print $11}' | tr -d '"' | sort | uniq -c | sort -rn | head -1000
+```
+
+### list user agents
+```bash
+cat access.log | awk '{print $12 $13}' | tr -d '"' | sort | uniq -c | sort -rn | head -1000
 ```
diff --git a/bash.md b/bash.md
@@ -13,13 +13,20 @@ The pattern is a glob pattern, so ${file%.*} would also work.
 The '%%' operator removes the largest matching suffix, and is interchangeable in the example above.
 As the pattern is fixed, ${file%%.*}.html would turn a.b.htm into a.html though.
 
-### Delete files mathing a pattern recursively
-You know those .DS_Store files that people commit to your source tree, that annoys the crap out of me.
-This will delete all that it can find.
+### Delete files matching a pattern recursively
+#### Them .DS_Store files
+You know those .DS_Store files that someone commited to a source tree instead of the .gitignore? (That annoys the $#!+ out of me). This will delete all that it can find.
 ```sh
 find . -type f -name .DS_Store -exec rm {} +
 ```
 
+#### Delete files older than X days, minutes etc
+Want to delete old log files after 7 days (`-ctime +7`) or perhaps some csv files uploaded to your ftp after 30 minutes (`-cmin +30`)? Well it is `find` to the rescue again.
+```sh
+find /home/sftp -type f -name "*.csv" -ctime +2 -exec rm -f {} \;
+```
+More info on find and time here: https://www.gnu.org/software/findutils/manual/html_mono/find.html#Time
+
 ## Sed, Awk & Xargs
 
 ### Want to kill a bunch of processes you dont like?

diff --git a/bash.md b/bash.md
@@ -20,12 +20,17 @@ This will delete all that it can find.
 find . -type f -name .DS_Store -exec rm {} +
 ```
 
-## Awk
+## Sed, Awk & Xargs
 
 ### Want to kill a bunch of processes you dont like?
 
 Kill those pesky Python Celery process that just seems to go off on their own and never come back.
 Do it by using `grep` to get out what you are looking for from the `ps` command then grab the 2nd 'column' of values using `awk` and pipe it to the `kill` command using `xargs` which will run it for each value that was piped to it.
 ```
 ps aux --sort=start_time | grep 'celery worker' | grep 'celery' | awk '{print $2}' | xargs kill -TERM
+```
+
+### Delete all git branches on remote which have been merged
+```bash
+git branch -r --merged | grep -v master | sed 's/origin\///' | xargs -n 1 git push --delete origin
 ```
diff --git a/bash.md b/bash.md
@@ -18,4 +18,14 @@ You know those .DS_Store files that people commit to your source tree, that anno
 This will delete all that it can find.
 ```sh
 find . -type f -name .DS_Store -exec rm {} +
+```
+
+## Awk
+
+### Want to kill a bunch of processes you dont like?
+
+Kill those pesky Python Celery process that just seems to go off on their own and never come back.
+Do it by using `grep` to get out what you are looking for from the `ps` command then grab the 2nd 'column' of values using `awk` and pipe it to the `kill` command using `xargs` which will run it for each value that was piped to it.
+```
+ps aux --sort=start_time | grep 'celery worker' | grep 'celery' | awk '{print $2}' | xargs kill -TERM
 ```
diff --git a/bash.md b/bash.md
@@ -12,3 +12,10 @@ The one used here, '%', removes the smallest matching suffix from the value of t
 The pattern is a glob pattern, so ${file%.*} would also work.
 The '%%' operator removes the largest matching suffix, and is interchangeable in the example above.
 As the pattern is fixed, ${file%%.*}.html would turn a.b.htm into a.html though.
+
+### Delete files mathing a pattern recursively
+You know those .DS_Store files that people commit to your source tree, that annoys the crap out of me.
+This will delete all that it can find.
+```sh
+find . -type f -name .DS_Store -exec rm {} +
+```
diff --git a/Amazing-CLI-Archive.md b/Amazing-CLI-Archive.md
@@ -1 +1 @@
-reverse-i-search: A command line history archive (because my memory just aint that good)
+My personal **reverse-i-search**: A command line history lookup (because my memory just aint that good)
diff --git a/Amazing-CLI-Archive.md b/Amazing-CLI-Archive.md
@@ -0,0 +1 @@
+reverse-i-search: A command line history archive (because my memory just aint that good)
diff --git a/AAA-bash-files.md → bash.md b/AAA-bash-files.md → bash.md
@@ -1,4 +1,9 @@
+# Bash
+
+## Files
+
 ### Rename a group of files with one command
+
 ```sh
 for file in *.png; do mv "$file" "${file%.png}_512x512.png"; done
 ```

diff --git a/exiftool.md b/exiftool.md
@@ -1,12 +1,15 @@
-# Using Exiftool on the command line cheat sheet
+# Exiftool
+Using Exiftool on the command line cheat sheet
+
 Install on a Mac with `brew install exiftool`
 
 ## Shift timestamps 
 
 ### Shift from one point in time to another
 
 This adds 1 hour to all images in a directory - like say you forgot to adjust your camera for daylight savings or you went on a holiday to another timezone...
-```
+
+```sh
 exiftool -AllDates+=1 dir
 ```
 The AllDates flag shifts the values of DateTimeOriginal, CreateDate and ModifyDate.

diff --git a/image-magick.md → image-magik.md b/image-magick.md → image-magik.md
@@ -1,4 +1,6 @@
-# Using Image Magick on the command line cheat sheet
+# Image Magick
+Command line cheat sheet
+
 Install on a Mac with `brew install imagemagick`
 
 ### Resize one file by a percentage

diff --git a/networking.md b/networking.md
@@ -0,0 +1,25 @@
+# Networking
+
+## Forgot how to find listening ports again?
+
+```
+netstat -an | grep -i "listen"
+```
+List all listening ports and their processes IDs.
+
+```
+lsof -iTCP -sTCP:LISTEN
+```
+This will return every TCP connection with the status LISTEN. This reveals all the open TCP ports on your box. It also lists the processes associated with those open ports. This is a significant upgrade over netstat, which lists PIDs at most.
+
+```
+sudo lsof -i -u^$(whoami)
+```
+Returns all connections not owned by the currently logged-in user. The caret ^ is used for negation. Anything matching the text after the caret will be removed from the results. 
+
+```
+lsof -nP -iTCP@localhost:513
+```
+Lists all the TCP connections with the hostname localhost and the port 513. It will also run lsof without connecting names to IP addresses and ports, making the command run noticeably faster.
+
+
diff --git a/reverse-i-search b/reverse-i-search
@@ -1 +0,0 @@
-One day I will write something here.

diff --git a/bash-files.md → AAA-bash-files.md b/bash-files.md → AAA-bash-files.md
diff --git a/bash-files.md b/bash-files.md
@@ -0,0 +1,9 @@
+### Rename a group of files with one command
+```sh
+for file in *.png; do mv "$file" "${file%.png}_512x512.png"; done
+```
+Bash has an extensive set of variable expansion options.
+The one used here, '%', removes the smallest matching suffix from the value of the variable. 
+The pattern is a glob pattern, so ${file%.*} would also work.
+The '%%' operator removes the largest matching suffix, and is interchangeable in the example above.
+As the pattern is fixed, ${file%%.*}.html would turn a.b.htm into a.html though.
diff --git a/exiftool.md b/exiftool.md
@@ -0,0 +1,35 @@
+# Using Exiftool on the command line cheat sheet
+Install on a Mac with `brew install exiftool`
+
+## Shift timestamps 
+
+### Shift from one point in time to another
+
+This adds 1 hour to all images in a directory - like say you forgot to adjust your camera for daylight savings or you went on a holiday to another timezone...
+```
+exiftool -AllDates+=1 dir
+```
+The AllDates flag shifts the values of DateTimeOriginal, CreateDate and ModifyDate.
+
+```
+exiftool -AllDates+=1:30 -if '$make eq "Canon"' dir
+```
+Shifts forward by 1 hour and 30 minutes for all Canon images in a directory.
+
+### Shifting by timezone
+Change timezone of all timestamps to Australian Eastern Standard Time
+```
+
+```
+
+More info on timeshifts here: https://sno.phy.queensu.ca/~phil/exiftool/Shift.html
+
+## Geo Coordinates
+
+### Add GPS Position
+Add latitude and longitude coordinates (for somewhere in Sydney) to all images in a directory 
+```
+exiftool -XMP:GPSLongitude=151.271305 -XMP:GPSLatitude=-33.891035 -GPSLongitudeRef=East -GPSLatitudeRef=South .
+```
+
+
diff --git a/image-magick.md b/image-magick.md
@@ -0,0 +1,12 @@
+# Using Image Magick on the command line cheat sheet
+Install on a Mac with `brew install imagemagick`
+
+### Resize one file by a percentage
+```
+convert airline.png -resize 50%  airline_half.png
+```
+
+### Generate new thumbnail files with new suffix in filename 
+```
+convert orig-*.png -thumbnail 20x20 -set filename:fname '%t_20x20' +adjoin '%[filename:fname].png'
+```
diff --git a/reverse-i-search b/reverse-i-search
@@ -0,0 +1 @@
+One day I will write something here.
Original file line number	Diff line number	Diff line change
		@@ -1 +1 @@
		reverse-i-search: A command line history archive (because my memory just aint that good)
		My personal reverse-i-search: A command line history lookup (because my memory just aint that good)
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1 @@
		reverse-i-search: A command line history archive (because my memory just aint that good)