# Oneliners

## Bash

- Basic `for` loop to iterate over lines in a file:

```shell
for pkg in $(cat pkgs.txt); do sudo apt purge "$pkg" -y; done
```

- More complex `for` loop using `if` statement to control actions, clean up output etc:

```shell
for node in $(cat nodes.txt); do echo "Node: ${node}"; ssh -q -t "$node" 'if [[ $(lsblk | grep -i lvm) ]]; then sudo apt install mdadm -y; fi'; done
```

- Checking **very** busy log files for their contents without having your console hang (as opposed to using `tail -f`):

```shell
watch -n 0.5 sudo tail /var/log/named/queries.log
```

- Alternative conditional logic in `for` loop iterating over array variable:

```shell
# Declare 'nodes' variable separately or prepend to loop and separate with semicolon
for node in "${nodes[@]}"; do ping -c 2 -W 0.1 "$node" > /dev/null && echo "OK: ${node}" || echo "NOT OK: ${node}"; done
```

- Use `while` loop to iterate over lines in a file:
  - Avoids calls to `cat` as is the case with the `for` loop example

```shell
# Using 'madison' command rather than 'policy' seems to be slightly faster
while read pkg; do if [[ $(apt-cache madison "$pkg") ]]; then echo "OK: ${pkg} exists in some repo"; else echo "NOT OK: ${pkg} doesn't exist in any repo"; fi; done < pkgs.txt
```

- Match lines into an array:

```shell
types=($(grep -oE 'pattern' input.txt))
```

- Grab block of text between two patterns:

```shell
sed -n '/pattern1/,/pattern2/p' input.txt
```

- Just see octal permissions for a file or directory:

```shell
stat -c '%a' /etc/passwd
```

- Grab last character from string:

```shell
last_character=${string_variable:-1}
```

- Parse file list from output of `grep` into subsequent commands:

```shell
grep -rl '\-\- MARK \-\-' /var/log/* | while read line; do echo "Working with file '${line}'"; grep MARK "$line" | tail -n1; done
```

- Include lines before and after a `grep` match:

```shell
grep -B 3 -A 3 -i "hv_fcopy" /var/log/messages
```

- Find all unique directories in listed directories that contain files modified 10 minutes ago since the command was ran:

```shell
ls | xargs -I {} find {} -type f -mmin -10 | cut -d "/" -f2 | sort -u
```

- Find all files in the current directories that were modified at least a minute ago, are larger than 500MB, and long list them:

```shell
find . -type f -mmin -1 -size +500M -exec ls -lsh {} \;
```

- Find all files in the current directories that were modified at least a day ago, are larger than 2GB, and empty their contents:

```shell
find . -type f -mtime -1 -size +2G -exec bash -c 'echo > {}' \;
```

- Run arbitrary command against a list of directories:

```shell
ls | xargs -I {} git -C {} pull
```

- Place the following somewhere in your shell script for easier step-by-step debugging; move ahead with Enter key:

```shell
set -x
trap read debug
```

- Change timezone interactively:

```shell
dpkg-reconfigure tzdata
```

- Search binary file that looks like text while ignoring case:

```shell
grep -ai "end:" /var/log/syslog
```

- Count time, calls, and errors for each system call when performing a directory listing:

```shell
strace -c ls test/
```

- Add to script to determine which line number the execution is at:

```shell
echo "DEBUG: ${LINENO}"
```

- Remove duplicated lines from a file without messing up the order:

```shell
awk '!visited[$0]++' your_file > deduplicated_file
```

- Run local script on a remote endpoint using SSH:

```shell
ssh -q <username>@<endpoint> "sudo bash -s" < local_script.sh
```

- Create new directory and change right into it:

```shell
# Oneliner
mkdir new_directory && cd $_

# Alias
# https://unix.stackexchange.com/a/9124
mkcd () {
  mkdir "$1"
  cd "$1"
}
```

- Recall argument to last used command:

```shell
$_
!$
Alt + .
# https://stackoverflow.com/a/3371711
!:1
!:1-2
```

- Get SSH key fingerprint:

```shell
# SHA-256
ssh-keygen -lf ~/.ssh/id_rsa.pub

# MD5
ssh-keygen -E md5 -lf ~/.ssh/id_rsa.pub
```

- Find broken symbolic links in current directory:

```shell
find . -xtype l
```

- Bulk fix relative symbolic links:

```shell
find . -lname '<relative-to-source target>*' -exec sh -c 'ln -sfn "<new relative-to-source target>/$(basename $0)" $0' {} \;
```

- Run remote script on remote endpoint using SSH:

```shell
ssh -q <username>@<endpoint> './location/to/script'
```

- Run local script on remote endpoint using SSH:

```shell
ssh -q <username>@<endpoint> 'sudo bash -s' < ./location/to/local/script
```

- Create ISO from directory without truncating long names (`-l`) and by not replacing hyphens with underscores `-iso-level 4`:

```shell
genisoimage -o data.iso -iso-level 4 -R -l data/
```

- List ISO file contents without having to mount it:

```shell
isoinfo -l -i data.iso
```

- Simple colouring for log files, both static and running output:

```shell
# https://automationrhapsody.com/coloured-log-files-linux/
cat test.log | perl -pe 's/^\[\*\].*/\e[0;36m$&\e[0m/g; s/^\[\+\].*/\e[0;32m$&\e[0m/g; s/^\[\!\].*/\e[0;31m$&\e[0m/g'
```

- In situations such as [these](https://github.com/Azure/azure-cli/pull/13435) a Bash environment variable will suppress warnings that clog output:

```shell
export PYTHONWARNINGS='ignore'
```

- Remove last column in string based on delimiter:

```console
$ string='my_underscored_string_12345'
$ echo "$string" | rev | cut -d '_' -f 2- | rev
my_underscored_string
```

- Prefix aliased command with backslash to avoid triggering alias:

```console
$ halt -p
REALLY!? -p
$ alias halt
alias halt='echo "REALLY!?"'
$ \halt -p
Connection to example.com closed by remote host.
```

- Pretty print CSV files:

```shell
# https://www.stefaanlippens.net/pretty-csv.html
function pretty_csv {
    perl -pe 's/((?<=,)|(?<=^)),/ ,/g;' "$@" | column -t -s, | less  -F -S -X -K
}

$ pretty_csv data.csv
$ pretty_csv < data.csv
$ sort data.csv | pretty_csv
```

- Pretty print TSV files:

```shell
function pretty_tsv {
    perl -pe 's/((?<=\t)|(?<=^))\t/ \t/g;' "$@" | column -t -s $'\t' | less  -F -S -X -K
}

$ pretty_tsv data.tsv
$ pretty_tsv < data.tsv
$ sort data.tsv | pretty_tsv
```

- Diff two files and save unified output to file:

```shell
diff -u file1 file2 > files.diff
```

- Show build information for cloud-based image:

```console
$ cat /etc/cloud/build.info 
build_name: server
serial: 20201211.1
```

- Show top disk usage and exclude certain directories under root:

```shell
du -Sh / --exclude=/{proc,sys,dev,var} | sort -rh | head -n 10
```

- It's possible to use the built-in `:` as a short-hand for an infinite loop:

```shell
while :; do "looping"; done
```

- Re-execute a shell to 'unsource' variables and aliases:

```shell
exec /bin/bash
```

- Use binary version of `time` instead of shell built-in to get access to more information:

```shell
# https://stackoverflow.com/questions/9006596/is-the-unix-time-command-accurate-enough-for-benchmarks
$(which time) --verbose echo "test"
```

- Use `perf stat` to easily perform repeated executions of a command and measure it in various ways:

```shell
perf stat --null --repeat 5 --table echo "test"
```

- Change nested key value in an array of JSON objects:

```text
.parameters.vmObjects.value |= map(if .vmName == "router" then .moduleSnapshot = "fixed" else . end)
```

- Use indirect references to use dynamic variable names:

```shell
for host in "${hosts[@]}"; do
  declare "parent_disk_${host}=$parent_disk"
done

for host in "${hosts[@]}"; do
  parent_disk="parent_disk_${host}"
  echo "${!parent_disk}"
done
```

- Test terminal's colors:

```shell
msgcat --color=test
```

- Bulk rename files in place:

```shell
find . -type f -name '<file name>' -execdir mv {} "description.txt" \;
```

- Encode with Base64 on a single line:

```shell
echo "text" | base64 -w 0
```

- Convert PEM to single-line:

```shell
awk 'NF {sub(/\r/, ""); printf "%s\\n",$0;}' combined.pem
```

- Install requirements for Poetry using existing `requirements.txt`:

```shell
cat requirements.txt | xargs poetry add
```

- Direct standard output to a file in a directory that might not yet exist:

```shell
echo "something" | install -D /dev/stdin directory/file.txt
```

- Delete files older than 1 year:

```shell
find /the/dir/to/start/in -type f -mtime +365 -ls -exec rm -f -- {} \;
```

- View permissions as a tree

```shell
 # -p - permissions
 # -u - username/userid
 # -f - full path
 # -i - don't print indentation lines
 # -d - print directories only
tree -pufid
```

- Bulk uninstall `pip` packages according to a wildcard

```shell
pip freeze | grep "azure*" | xargs -n 1 pip uninstall -y
```

- Show transaction history for a package:

```shell
dnf history list <package>
```

- Show information about specific transaction in history:
  - More on `history`: https://www.putorius.net/dnf-history.html

```shell
dnf history info <transaction ID>
```

- List SystemD timers:

```shell
systemctl list-timers
```

- Show execution of service tied to a timer of an identical name:

```shell
journalctl -u name.timer
journalctl -u name.service
```

- Copy remote directory to local system:

```shell
scp -rCp <user>@<remote>:<remote path> <local path>
# OR (faster)
rsync -azvhP <user>@<remote>:<remote path> <local path>
```

- Overwrite existing directory with contents from another:

```shell
rsync -av --delete ~/new/ ~/old
```

- Count number of installed kernels:

```console
$ sudo dnf list --installed kernel-core* | tail -n +2 | wc -l
9
```

- Increase number of installed kernels in `/etc/dnf/dnf.conf`:

```text
...
installonly_limit=10
...
```

- Pin specific kernel version:

```shell
$ sudo dnf install python3-dnf-plugins-extras-versionlock
$ # List kernel packages
$ rpm -qa kernel
kernel-6.0.18-300.fc37.x86_64
kernel-6.1.7-200.fc37.x86_64
kernel-6.1.8-200.fc37.x86_64
$ sudo dnf versionlock add kernel-6.0.18-300.fc37.x86_64
Last metadata expiration check: 3:51:11 ago on E 30 jaan  2023 15:47:21.
Adding versionlock on: kernel-0:6.0.18-300.fc37.*
$ # Remove pin
$ sudo dnf versionlock delete kernel-6.0.18-300.fc37.x86_64
...
```

- Undo ad hoc changes made to a SystemD service (e.g. `systemd-resolved`):

```console
$ systemctl revert systemd-resolved.service
Removed "/etc/systemd/system/systemd-resolved.service.d/override.conf".
Removed "/etc/systemd/system/systemd-resolved.service.d".
$ systemctl restart systemd-resolved.service
```

- Back up a file using [brace expansion](https://www.gnu.org/software/bash/manual/html_node/Brace-Expansion.html):
  - Equivalent to `cp /etc/ssh/sshd_config /etc/ssh/sshd_config.bak`

```shell
cp /etc/ssh/sshd_config{,.bak}
```

- Restore a backed up file:
  - Equivalent to `cp /etc/ssh/sshd_config.bak /etc/ssh/sshd_config`

```shell
cp /etc/ssh/sshd_config{.bak,}
```

## Vim

- Write file opened as 'readonly':

```text
:w !sudo tee "%"
```

- Visual editing can be used to delete any number of lines:

```text
Shift+V from point of cursor and press D key to delete
```

- Search and replace:

```text
:%s/search/replace/g 
```

- Disable search highlight in Vim:

```text
:noh
```

- Move line on cursor up or down:

```text
# One line up
:m -2
# One line down
:m +1
```

## Git

- Push existing repository to new remote:

```shell
git remote add <name of new remote> <HTTPS or SSH URL>
git push <name of new remote> master
```

- Pretty-print branch graph:

```shell
git log --all --decorate --oneline --graph
```

- Move `master` back X amount of commits:

```shell
git checkout master
git reset --hard <old_commit_id>
git push -f origin master
```

- Replay changes on `master` to some other branch:

```shell
# Beware of blindly accepting any incoming changes in favor of your own
# https://demisx.github.io/git/rebase/2015/07/02/git-rebase-keep-my-branch-changes.html
git checkout master
git pull
git checkout different_branch
git rebase -Xtheirs master
git push --force
```

- Show changed files in specified commit hash:

```shell
# https://stackoverflow.com/questions/49853177/how-to-see-which-files-were-changed-in-last-commit
git diff-tree --no-commit-id --name-only <commit hash>
```

- Create patch file from diff:

```shell
git diff file.json > file.patch
```

- Create patch file from commit:

```shell
git show <commit hash> > commit.patch
```

- Apply patch file:

```shell
git apply commit.patch
```

- Bulk create patch files from individual files when running `git diff` in a repository:

```shell
https://raymii.org/s/tutorials/Bash_bits_split_a_file_in_blocks_and_do_something_with_each_block.html
OLDIFS=$IFS; IFS=';' blocks=$(git diff | sed -n '/diff/,/(diff|$)/ {/diff / s/^/\;/; p}'); for block in ${blocks#;}; do echo "$block" > $(echo "$block" | head -n 1 | rev | cut -d "/" -f 1 | rev).patch; done; IFS=$OLDIFS
```

- Show diff of stashed hunk:

```shell
git stash show -p [stash@{N}]
```

- Bulk create separate stashes of every changed file with a message equaling the filename:

```shell
git status -s | cut -d " " -f 3 | xargs -I {} git stash push {} -m "{}"
```

- Pop every entry from the stash back to the working tree:

```shell
git stash list | cut -d ":" -f 1 | xargs -I {} git stash pop
```

- Move unpushed commits to a new branch:

```shell
# Pull latest changes from 'origin/master' if haven't already
# https://stackoverflow.com/a/46726955
git checkout -b new_branch
git checkout master
git reset --hard origin/master
```

- Copy commit to current branch:

```shell
git cherry-pick <commit hash>
```

- Undo pushed commit that nobody has yet pulled:

```shell
git reset HEAD^ --hard
git push --force origin
```

- View history of specific function in file:

```shell
git log -L :<function>:<file>
```

- Speed up Git for larger repositories:

```shell
git config feature.manyFiles 1
```

- Search through history for a specific word:

```shell
git rev-list --all | ( while read revision; do git grep -F 'word' "$revision"; done; )
```

- Delete remote branch:

```shell
git push origin --delete branch/name
```

- Bulk reset author of multiple (unpushed) commits (e.g. 9):

```shell
# Set correct user name and email prior to this
git rebase --onto HEAD~9 --exec "git commit --amend --reset-author --no-edit" HEAD~9
```

- Re-order commits:

```shell
git rebase --interactive
# Oldest commit will be at the top
# Move commit down with 'ddp'
# Move commit up with 'ddkP'
```

- Search for 'something' in a commit message:

```shell
git log --all -i --grep='something'
```

- Search for 'something' through all commits' contents:

```shell
git grep 'something' $(git rev-list --all)
```

## Docker/Podman

- Parse lines from an arguments file to separate parameters for building:

```shell
# https://ilhicas.com/2018/11/03/docker-build-with-build-arg-wit-multiple-arguments.html
podman build -t foo $(while IFS= read -r line; do args+="--build-arg ${line} "; done < <(cat .arg); echo "$args"; unset args) .
```

- Remove all 'exited' containers:

```shell
podman rm $(podman ps --all -q -f status=exited)
```

- Build and run container based on Dockerfile in current context:

```shell
podman build -t foo . && podman run --rm -it foo
```

- Prune everything that shouldn't exist anymore without any confirmation:

```shell
podman system prune -a -f
```

- Remove all images except `latest`:

```shell
podman images | grep -v "latest" | tail -n +2 | awk '{ print $3 }' | xargs --no-run-if-empty podman rmi
```

- Possible improvement when executing `RUN` within a Dockerfile:
  - Benefit is that when a specific line fails, then the error message is much more concise as opposed to the standard method of using ampersands

```text
RUN set -eu; \
    python3 -m venv venv; \
    venv/bin/pip install -r requirements.txt; \
    venv/bin/pip install -r requirements-dev.txt; \
    echo 'Venv creation + requirements installation: OK';
```

- Remove dangling `<none>:<none>` images:

```shell
docker rmi $(docker images -f "dangling=true" -q)
```

## Ansible

- Run role against arbitrary host:

```shell
# https://stackoverflow.com/a/38384205
# Note the comma after the IP or FQDN
# https://groups.google.com/d/msg/ansible-project/G_9JRGp5jGE/PTBZdgDb5OEJ
# Additional hosts can be added by supplying '-i' parameter with more arguments (comma at the end only if count == 1)
ansible-playbook -i '<IP or FQDN>,' -u '<user name>' --extra-vars 'ansible_winrm_server_cert_validation=ignore ansible_connection=winrm ansible_winrm_transport=credssp ansible_password=<password>' --tags '<tag value for a role>' playbook.yml
```

- Run ad-hoc command against arbitrary host:

```shell
# Replace the final 'all' with a more precise host pattern if you passed more than one IP or FQDN to the initial list (comma at the end only if count == 1)
ansible -i '<IP or FQDN>,' -u '<user name>' --extra-vars 'ansible_winrm_server_cert_validation=ignore ansible_connection=winrm ansible_winrm_transport=credssp ansible_password=<password>' -m 'win_shell' -a 'ipconfig' 'all'
```

- Add timing information to playbook execution output:

```console
# https://docs.ansible.com/ansible/latest/plugins/callback/profile_tasks.html
$ cat ansible.cfg
[defaults]
callback_whitelist = profile_tasks
```

- Make verbose output more readable by using YAML instead of JSON:

```shell
ANSIBLE_STDOUT_CALLBACK='yaml'
```

- Debug variables without running entire playbook:

```shell
ansible -m debug <host> -a "var=hostvars[inventory_hostname].<variable>"
```

## Grafana/Promtail

- Test simple journal scraping configuration:

```console
$ cat /etc/promtail/config.yml
server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /var/log/positions.yaml

clients:
  - url: https://<token>@<host>/loki/api/v1/push

scrape_configs:
  - job_name: journal
    journal:
      max_age: 12h
      labels:
        job: systemd-journal
        static_label: label-value
    relabel_configs:
      - source_labels: ['__journal__systemd_unit']
        target_label: 'unit'

$ promtail --dry-run --config.file /etc/promtail/config.yml
2021-05-20T03:48:54     {job="systemd-journal", static_label="label-value"}     Stopped target Default.
2021-05-20T03:48:54     {job="systemd-journal", static_label="label-value"}     Stopped target Basic System.
2021-05-20T03:48:54     {job="systemd-journal", static_label="label-value"}     Stopped target Timers.
2021-05-20T03:48:54     {job="systemd-journal", static_label="label-value"}     Stopped target Paths.
2021-05-20T03:48:54     {job="systemd-journal", static_label="label-value"}     Stopped target Sockets.
```

## PowerShell

- Get today's date and time where time is set to midnight:

```powershell
[DateTime]::Today
```

- Show list of logged on users:

```powershell
query user /server:$SERVER
```

- Log off user by specifying session ID:

```powershell
logoff <session ID>
```

- Reload a local module:

```powershell
Import-Module -Name .\module.psd1 -Force
```

- Pretty-print minified JSON:

```powershell
$String | ConvertFrom-Json | ConvertTo-Json -Depth 100
```

- Convert from Base64:

```powershell
[Text.Encoding]::Utf8.GetString([Convert]::FromBase64String($String))
```

- Convert string to boolean:

```powershell
[System.Convert]::ToBoolean($String)
```

- Resolve FQDN:

```powershell
[System.Net.Dns]::GetHostByName($FQDN)
```

## Windows

- Colored prompt with Git branch for terminal (e.g. ConEmu):

```text
PS1='${debian_chroot:+($debian_chroot)}\[\033[01;35m\] \w\[\033[01;33m\]$(__git_ps1)\[\033[01;m\] > '
```

## Miscellaneous

- Query in [JMESPath website](https://jmespath.org/): `[?tags.currently-used == 'False' || tags.currently_used == 'False'].name`
  - Same query in `jpterm`: `[?tags."currently-used" == 'False' || tags."currently_used" == 'False']`
  - Same query in console: `"[?tags.\"currently-used\" == 'False' || tags.\"currently_used\" == 'False'].name"`
- Convert x265 MP4 to x264: `ffmpeg -i input.mp4 -c:v libx264 -crf 20 -c:a copy output.mp4`
- Create Windows install media:
  - Download [WoeUSB](https://github.com/WoeUSB/WoeUSB)
  - Download [Windows 10 ISO](https://www.microsoft.com/en-us/software-download/windows10ISO)
  - Find out USB's disk: `sudo fdisk --list`
  - Wipe disk entirely and burn ISO: `sudo ./woeusb-5.2.4.bash --target-filesystem NTFS --device Win10_22H2_EnglishInternational_x64.iso /dev/sda`