------------------------------------------------------------------------------------------ #CLI shortcut keystrokes(linux&MAC) Ctrl+L: Clear the screen. This is similar to running the “clear” command. Ctrl+C: Interrupt (kill) the current foreground process running in in the terminal. This sends the SIGINT signal to the process Ctrl+Z: Suspend the current foreground process running in bash. This sends the SIGTSTP signal to the process. To return the process to the foreground later, use the fg process_name command. Ctrl+D: Close the bash shell.This is similar to running the exit command Ctrl+L: Clear the screen. This is similar to running the “clear” command. Ctrl+S: Stop all output to the screen. This is particularly useful when running commands with a lot of long, verbose output, but you don’t want to stop the command itself with Ctrl+C. Ctrl+Q: Resume output to the screen after stopping it with Ctrl+S. Ctrl+A or Home: Go to the beginning of the line. Ctrl+E or End: Go to the end of the line. Alt+B: Go left (back) one word. Alt+F: Go right (forward) one word. Ctrl + U to clear up to the beginning Ctrl + U to recall a command by searching history Up Arrow to recall the previous command from history Down Arrow to go to next command in the command history #multiple virtual consoles CTRL + ALT + F1-F8 ------------------------------------------------------------------------------------------------------------------------------------------------- #An absolute path is a complete path to a resource, beginning at the filesystem's root #The filesystem root is represented as a single forward slash: / #Relative path /home/damon #if you're already in your home directory Music/Rock #the rest of the path relative to where you are now ------------------------------------------------------------------------------------------------------------------------------------------------- #the tilde character "~" represents the current, logged-on user's home directory $ cp /etc/ssh/sshd_config ~ $ cd ~ #The single dot represents the present working directory, or where you are right now. $ sudo cp /etc/ssh/sshd_config . #Double dots represent the parent directory, or the directory immediately above the current one in the filesystem $ cd .. ------------------------------------------------------------------------------------------------------------------------------------------------- $ bash --version ------------------------------------------------------------------------------------------ /etc #configuration files on the system /etc/hosts #maps hostnames to IP addresses /etc/skel #files copied to each user's home /usr #user by user and supersuser accounts.Stores application programs /usr/bin #executables used by users, in user's PATH statement /usr/sbin #executables used by superusers /usr/local #applications which are not part of linux #applications installed after initial linux installation,in user's PATH statement #administrative applications installed after initial linux installation /usr/local/bin /usr/doc #the default location for application documentation is in a directory named for the application /var #stores log files, mails and other data /var/spool #mail & printing files /var/tmp #data that should persist across reboot /var/log #log files /var/log/anaconda.log - While installing Linux, all installation related messages are stored in this log file /var/log/audit/ - This subdirectory contains logs information stored by the Linux audit daemon (auditd) /var/log/auth.log - user logins,such as password prompts /var/log/boot.log - Contains information that are logged when the system boots /var/log/cron - Whenever cron daemon (or anacron) starts a cron job, it logs the information about the cron job in this file /var/log/debug #debugging information from the Ubuntu system and applications /var/log/daemon.log - display server, SSH sessions, printing services, bluetooth /var/log/dmesg - Contains kernel ring buffer. This file is overwritten when the system is rebooted. /var/log/faillog -failed user login attempts. ssh(used for remote login), su(to switch users), at, cron (both used for scheduling tasks) use PAM modules for authentication /var/log/kern.log - Contains information logged by the kernel. Helpful to troubleshoot a custom-built kernel. /var/log/lastlog #last logins /var/log/messages #the main system log,the messages that are logged during system startup /var/log/sa/ - Contains the daily sar files that are collected by the sysstat package. /var/log/samba/ - Contains log information stored by samba, which is used to connect Windows to Linux. /var/log/setroubleshoot/ - SELinux uses setroubleshootd (SE Trouble Shoot Daemon) to notify about issues in the security context of files and logs those information in this log file. /var/log/secure #failed login attempts, failed SSH login attempts, successful SSH logins /var/log/syslog /var/log/user.log - Contains information about all user level logs. /var/log/Xorg.x.log - Log messages from the X server to this file. /var/log/Xorg.0.log /var/log/yum.log - Contains information that are logged when a package is installed using yum #binary log files,cannot be read with a normal text editor,special command-line tools are used to display the relevant information in human-readable format /var/log/btmp - This file contains information about failed login attempts /var/log/utmp - The utmp file allows one to discover information about who is currently using the system. /var/log/wtmp - The wtmp file records all logins and logouts. #This subdirectory contains additional logs from the mail server. #For example, sendmail stores the collected mail statistics in /var/log/mail/statistics file /var/log/mail/ /var/log/maillog - Logs information from the mail server that is running on the system. /var/log/mail.log - Logs information from the mail server that is running on the system. #- Contains the apache web server access_log and error_log and related virtual hosts logs if set up to log here. /var/log/httpd/ /var/log/apache2 /var/log/apache2/error.log /var/log/mysql.log #All printer and printing related log messages /var/log/cups /var/log/spooler /bin #binaries run during system startup /sbin #administrative binaries run by superusers /root #home for superuser /home #user home dirs /boot #files run by boot loader and kernel /dev #peripheral access files /proc #virtual dir contains system info /tmp #stores temporary files /etc/grub.conf or /boot/grub/grub.conf #customize the behavior and appearance of the boot menu /dev/shm,/dev/shmfs #also known as tmpfs,df -h,file system which keeps all files in virtual memory,no files created on hard drive /etc/passwd #UID/GID stored, UID/GUID zero(0) for root, all UID/GID <1000 privileged access, all other users UID/GID>1000 #list group members /etc/group #secondary group memberships, awk -F':' '/sudo/{print $4}' /etc/group grep '^sudo' /etc/group grep -w sudo /etc/group ------------------------------------------------------------------------------------------ #Bard The command [ -f /var/run/reboot-required ] && sudo reboot -f first checks if the file /var/run/reboot-required exists. If it does, the command then runs the sudo reboot -f command, which forcefully reboots the system. The file /var/run/reboot-required is created by some Linux distributions when a system update requires a reboot. This is to ensure that the system is properly updated before it is restarted. The sudo reboot -f command forcefully reboots the system, even if there are unsaved changes in open files. This is useful if the system is unresponsive or if you need to reboot it immediately. Some of the best uses cases for the command [ -f /var/run/reboot-required ] && sudo reboot -f are: After a system update that requires a reboot. If the system is unresponsive. If you need to reboot the system immediately. It is important to note that forcefully rebooting the system can sometimes cause data loss. Therefore, you should only use this command if you are sure that there are no unsaved changes in open files. ------------------------------------------------------------------------------------------ #ChatGPT This command is used in a Linux shell environment and is composed of two parts connected by the && operator. The overall purpose of this command is to check if a specific file exists and, if it does, initiate a system reboot with administrative privileges. Let's break down each part of the command: [ -f /var/run/reboot-required ]: This is a conditional test using the [ (test) command. The -f flag checks if the specified file (/var/run/reboot-required) exists. This file is often used on Ubuntu-based systems to indicate that a system reboot is required due to software updates or configuration changes. &&: This is the logical "AND" operator. It's used to execute the command on the right only if the command on the left (the test in this case) evaluates as true or successful. sudo reboot -f: If the previous test [ -f /var/run/reboot-required ] is successful (meaning the file exists), this command is executed. The sudo command is used to execute the subsequent command with superuser (root) privileges. reboot is a command used to restart the system, and the -f flag forces an immediate reboot without further confirmation. Best Use Cases: Automated Reboot after Updates: This command is commonly used in automated scripts or scheduled tasks, particularly after software updates have been applied to a Linux system. If the file /var/run/reboot-required exists, it suggests that important updates have been installed that require a system restart for them to take effect. By using this command, you can ensure that the system automatically reboots when necessary after updates. Post-Configuration Reboot: In situations where configuration changes are applied that necessitate a reboot, this command can be used to automatically restart the system. For example, if you've made changes that affect kernel modules or critical system services, an automated reboot might be necessary to apply those changes. Server Maintenance and Deployment Scripts: When performing maintenance tasks on servers or during deployment procedures, administrators might use this command to ensure that a system is properly restarted after specific actions are taken. Scripted System Checks: This command can be incorporated into system health-check scripts that are run periodically to ensure the system is in a proper state. If the script detects that a reboot is required, it can automatically initiate one. It's important to exercise caution when using commands that trigger a system reboot, especially in automated scenarios. Rebooting a system without proper planning can lead to unexpected downtime and disruption of services. Always thoroughly test any scripts or commands in a controlled environment before deploying them in production systems. ------------------------------------------------------------------------------------------ "&&" #where the second command is executed only if the exit status of the preceding command is 0 (zero exit code) #The right side of && is evaluated if the exit status of the left side is zero (i.e. true) "||" #If the exit status of the first command is not 0 (non-zero exit code), then execute the second command #The right side of || is evaluated if the left side exit status is non-zero (i.e. false) #Opposite to &&, the statement after the || operator will only be executed if the exit status of the test command is false [ -f /etc/resolv.conf ] && echo "$FILE exist." || echo "$FILE does not exist." ------------------------------------------------------------------------------------------ #Syntax for Command Substitution #The old-style uses backticks (also called backquotes) to wrap the command being substituted #The new style begins with a dollar sign and wraps the rest of the command in parenthesis $ rpm -ql $(rpm -qa | grep httpd) $(command) #Command Substitution (list) #Group commands in a subshell: ( ) { list; } #Group commands in the current shell: { } [[ expression ]] #Test - return the binary result of an expression: [[ ]] command substitution; output from pwd works as the argument for echo command $ echo `pwd` Command Substitution # `backquotes` also known as `backticks` KERNEL_VERSION=`uname -r` #(parentheses) KERNEL_VERSION=$( uname -r ) $ echo $KERNEL_VERSION 4.15.0-29-generic #Command substitution in Bash #execute a command and substitute it with its standard output,command executes within a subshell $ var_date=$(date) && echo $var_date $ due_date="01-01" $ echo "Status as of $(date +%m-%d-%Y) : The delivery is due on ${due_date}-2022" echo "${myvar:-bash}" #check the variable, $myvar is set or unset. If $myvar is unset, then the string ‘bash’ will print echo "${myvar:=bash}" #set the value, ‘bash’ to $myvar and print ‘bash’ to the terminal if $myvar is unset echo "${myvar:+python}" #print, ‘python’ to the terminal if $myvar is set before $ mystr="Bangladesh" $ echo "${mystr:0:6}" #six characters from $mystr starting from position 0 to 6 $ echo "${mystr:6}" #all characters from $mystr, starting from position 6 to the end $ echo "${#mystr}" #count and print the total number of characters of $mystr The process substitution >(command) will be replaced by a file name. ------------------------------------------------------------------------------------------ { Commands; } : Commands execute in current shell ( Commands ) : Commands will execute in a subshell $ VAR="1"; { VAR="2"; echo "Inside group: VAR=$VAR"; }; echo "Outside: VAR=$VAR" Inside group: VAR=2 Outside: VAR=2 #the variable is changed in a subshell. Therefore, the changes will not affect the outer shell $ VAR="1"; ( VAR="2"; echo "Inside group: VAR=$VAR" ); echo "Outside: VAR=$VAR" Inside group: VAR=2 Outside: VAR=1 $ pwd; { cd /etc; pwd; }; pwd #current shell $ pwd; ( cd /etc; pwd ); pwd #subshell ------------------------------------------------------------------------------------------ #Placing a list of commands between curly braces causes the list to be executed in the current shell context. #No subshell is created. $ { ls && v=3; } > tmp $ echo "$v" 3 #pipe creates a new subshell $ { ls a* && v=3; } | grep my $ echo "$v" #does not print variable v touch abc{1,2,3,4} echo ${month[3]} echo {10..0} mkdir /some/dir/{a,b,c,d} mkdir -p -v /home/jon/tmp/{dir1,anotherdir,similardir} #Apply a command on files with different names ------------------------------------------------------------------------------------------ #Parameter Expansion #specify a variable within {} to protect it against expansion. #useful when the characters immediately following it can be interpreted as part of the variable name. $ price=5 $ echo "${price}USD" ------------------------------------------------------------------------------------------ # avoid dot slash "./" when running executable scripts, binary file #OPTION 1 export PATH=$PATH:/path/to/directory #OPTION 2 ln -s /path_to_script/myscript /usr/bin/myscript #OPTION 3 bash -c /Users/you/myscript.sh #OPTION 4 , add the line below into "~/.bashrc." export PATH=/path_to_folder_containing_executable/:$PATH ------------------------------------------------------------------------------------------ #Hosts File Windows 10 - "C:\Windows\System32\drivers\etc\hosts" Linux - "/etc/hosts" Mac OS X - "/private/etc/hosts" -------------------------------------------------------------------------------------------------------------------- #Reading file content from command line $ while read line; do echo $line; done < company.txt -------------------------------------------------------------------------------------------------------------------- mount -t tmpfs tmpfs /mnt/tmp #in-memory filesystem,tmpfs is a temporary filesystem that only exists in RAM -------------------------------------------------------------------------------------------------------------------- #which is an external binary, located at /usr/bin/which which steps through the $PATH environment variable and checks for the existence of a file which modprobe #command is built in to shell, with the -v option how shell invokes the command specified as its option command -v modprobe -------------------------------------------------------------------------------------------------------------------- #ls -ai command $ ls .. # double dot(..) represents the parent directory $ sudo rm .. rm: cannot remove '..': Is a directory $ sudo rm -rf .. rm: refusing to remove '.' or '..' directory: skipping '..' $ ls . #single dot(.) represents the current directory $ sudo rm . rm: cannot remove '.': Is a directory $ sudo rm -rf . rm: refusing to remove '.' or '..' directory: skipping '.' -------------------------------------------------------------------------------------------------------------------- #list all failed SSH logins grep "Failed password" /var/log/auth.log egrep "Failed|Failure" /var/log/auth.log cat /var/log/auth.log | grep "Failed password" grep "Failed password" /var/log/auth.log | awk '{print $11}' | uniq -c | sort -nr # the number of failed attempts of each IP address, -------------------------------------------------------------------------------------------------------------------- tar -czf - ./Documents/ | (pv -p --timer --rate --bytes > backup.tgz) #Monitor tar progress pv -p history.log | wc #Count number of lines, words, bytes pv -p /etc/hosts | wc pv history.log | zip>$HOME/Documents/history.zip #history.log and show progress pv origin-cdn.cyberciti.org_access.log > /tmp/origin-cdn-access.log #copy a file and show progress pv origin-cdn.cyberciti.org_access.log > /dev/null #show progress pv -cN rawlogfile origin-cdn.cyberciti.org_access.log | gzip | pv -cN gziplogfile > access.log.gz #see progress of both pipes (pv -n backup.tar.gz | tar xzf - -C path/to/data ) 2>&1 | dialog --gauge "Running tar, please wait..." 10 70 0 #extract tar ball and show progress using the dialog command tar -czf - ./Documents/ | (pv -n > backup.tgz) 2>&1 | dialog --gauge "Progress" 10 70 nc -l -v -w 30 -p 2000 > /tmp/data.bin #create a network port 2000 nc -l -v -w 30 -p 2000 > /tmp/data.bin #Open another terminal -------------------------------------------------------------------------------------------------------------------- timeout 10s tail -f /var/log/pacman.log #terminate after 10 seconds timeout 3.2s dmesg -w #display all messages from the kernel ring buffer, only 3.2 seconds timeout 3m ping 127.0.0.1 #ping command after 3 minutes timeout 2d ping 127.0.0.1 #ping command after 2 days timeout --foreground 2m test.sh #By default runs in background, run it foreground timeout -k 20 10 tail -f /var/log/pacman.log #if the command is still running even after the time out, send a kill signal timeout -s 9 3s ping 127.0.0.1 #use 9 as SIGKILL kill -l #list all acceptable signal timeout -k 5s 3m sh test.sh #let the script run for 3 minutes, if it does not exit, kill after 5 seconds -------------------------------------------------------------------------------------------------------------------- /home/user01/test.file can also be denoted by ~/test.file #The tilde (~) is a Linux "shortcut" to denote a user's home directory. cd ~ #change into user's home directory -------------------------------------------------------------------------------------------------------------------- w #show who is logged on and what they are doing. w user #print info for a specific user w -i #display IP address instead of hostname for from field. w -o #print blank space for idle times less than one minute -------------------------------------------------------------------------------------------------------------------- # stops after 50secs, stdout seen skaffold dev & sleep 50s; kill $! -------------------------------------------------------------------------------------------------------------------- $ ssh root@192.168.1.1 'bash -s' < script.sh #Execute the local script.sh on the remote server ssh $ ssh root@192.168.1.1 'uptime; df -h' $ ssh root@192.168.1.1 'free -m | cat /proc/loadavg' $ ssh root@192.168.1.1 << EOF uname -a lscpu | grep "^CPU(s)" grep -i memtotal /proc/meminfo EOF -------------------------------------------------------------------------------------------------------------------- #setting system locale to en_US.utf8 localectl set-locale LC_CTYPE=en_US.utf8 localectl status ---------------------------------------------------------------------------------------------------- ip -o -4 addr list enp0s8 | awk '{print $4}' | cut -d/ -f1 # print IP with a given interface ip -o -6 addr list enp0s8 | awk '{print $4}' | cut -d/ -f1 ifconfig | awk '/192.168.18.84/ {print $1}' RS="\n\n" # print interface with a given IP ---------------------------------------------------------------------------------------------------- #-n do not output the trailing newline echo -n $PASSWORD | faas-cli login --username admin --password-stdin ---------------------------------------------------------------------------------------------------- cat verifyURL.sh #!/bin/bash url="https://gist.github.com/githubfoam/d313d580a92d84123b841ebbd4f255a6" if wget $url >/dev/null 2>&1 ; then echo "Url : $url ...is online" else echo "Url : $url ...is not online" fi #if link exists #--head avoid downloading the file contents #--fail make the exit status nonzero on a failed request #--silent avoid status or errors url="https://www.katacoda.com/courses/kubernetes/launch-single-node-cluster" if curl --output /dev/null --silent --head --fail "$url"; then echo "URL exists: $url" else echo "URL does not exist: $url" fi time curl -I http://mydomain.com | grep HTTP # get the header for the page, and time the process curl -I "www.google.com" 2>&1 | awk '/HTTP\// {print $2}' # see only the HTTP status code curl -I "https://www.google.com" 2>&1 | grep -w "200\|301" # see if a given website is up or down curl -w "\n" http://localhost:8080/hello # avoid your terminal printing a '%' or put both result and next command prompt on the same line lynx -head -dump http://www.google.com #Check Whether a Website is up or down lynx -head -dump http://www.google.com 2>&1 | awk '/HTTP\// {print $2}' # see only the HTTP status code #SSL certificate problem curl -kfsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg curl https://curl.se/ca/cacert.pem -o /etc/pki/ca-trust/source/anchors/curl-cacert-updated.pem && update-ca-trust wget --no-check-certificate http://curl.haxx.se/ca/cacert.pem ---------------------------------------------------------------------------------------------------- $ time ./a.out real 0m5.279s #real is the wall-clock time,Elapsed real time, real time, wall-clock time, or wall time is the actual time taken from the start of a computer program to the end. In other words, it is the difference between the time at which a task finishes and the time at which the task started. user 0m1.915s #user is the time spent executing the user’s program, sys 0m0.006s #sys is the time spent on system tasks required by the program ---------------------------------------------------------------------------------------------------- #linux benchmark sysbench --test=cpu run sysbench --test=cpu help sysbench --test=cpu --cpu-max-prime=20000 run sysbench --test=memory run sysbench --test=memory help sysbench --test=fileio help sysbench --test=fileio --file-test-mode=seqwr run sysbench --test=fileio --file-total-size=100G cleanup ---------------------------------------------------------------------------------------------------- # generate load $ yes > /dev/null & # maximum number of processes available to a single user. $ ulimit -u # The limit is set with the -S option $ ulimit -S -u 500 #inspect the load figures on the server before and after the stress test uptime sudo stress --cpu 100 --timeout 300 uptime $ sudo (yum/apt-get) install stress ## Stress using CPU-bound task stress -c 4 ## Stress using IO-bound task stress -i 2 # a load average of four is imposed on the system by specifying two CPU-bound processes, one I/O-bound process, and one memory allocator process stress -c 2 -i 1 -m 1 --vm-bytes 128M -t 10s $ sudo apt-get install stress-ng stress-ng --cpu 4 --io 4 --vm 1 --vm-bytes 1G --timeout 60s --metrics-brief stress-ng --hdd 5 --hdd-ops 100000 stress --cpu 4 --io 3 --vm 2 --vm-bytes 256M --timeout 35s # The restriction can be made permanent by configuring the nproc value /etc/security/limits.conf # initiate a fork bomb ":(){ :|:& };:" # run ./$0& twice #!/bin/sh ./$0& ./$0& ---------------------------------------------------------------------------------------------------- # Listing all currently known events: perf list # CPU counter statistics for the specified command: perf stat command # CPU counter statistics for the specified PID, until Ctrl-C: perf stat -p PID # Sample on-CPU functions for the specified command, at 99 Hertz: perf record -F 99 command # Trace all block device (disk I/O) requests with stack traces, until Ctrl-C: perf record -e block:block_rq_insert -ag # Add a tracepoint for the kernel tcp_sendmsg() function entry ("--add" is optional): perf probe --add tcp_sendmsg # Trace system calls by process, showing a summary refreshing every 2 seconds: perf top -e raw_syscalls:sys_enter -ns comm # Show perf.data with a column for sample count: perf report -n ---------------------------------------------------------------------------------------------------- #ls command ltrace -c ls ltrace -p ltrace -l /lib/libselinux.so.1 id -Z #execute the id -Z command and show the calls made to the libselinux.so module ltrace -o foobar.log ./foobar #edirect output of ltrace to a file ltrace -e malloc ./foobar #filter and display only calls to a certain library function #ls command strace ls strace -v #-v verbose option that can provide additional information on each system call strace -s 80 -f ./program #print the first 80 characters of every string strace -i ls # print instruction pointer at the time of system call strace -r ls # display a relative timestamp upon entry to each system call strace -t ls #each line in strace output to start with clock time strace -T ls #show time spent in system calls strace -c ls #print a summary strace -p 3569 #If a process is already running, you can trace it by simply passing its PID strace -p `pidof rsyslogd` strace -p $(pgrep rsyslogd) #monitor process without knowing its PID, but name #if there are multiple processes to be traced at once (e.g. all instances of an running apache httpd) strace $( pgrep httpd | sed 's/^/-p/' ) strace -c -p 3569 Summary of Linux Process strace -c ls Counting number of sys calls strace -o output.txt ls strace -e trace=network -o OUTFILE php -q test2.php # write the output of strace to a file or redirect the output strace -e trace=network php -q test2.php 2> test2debug #redirect the output strace -e trace=open,stat,read,write ls strace -e trace=mprotect,brk ifconfig eth0 # trace mprotect or brk system calls strace -e trace=network ifconfig eth0 #Trace all the network related system calls strace -e trace=network #Monitoring the network strace -e trace=memory Monitoring memory calls strace -e open ls display only a specific system call, use the strace -e option strace -f -eopen /usr/sbin/sshd 2>&1 | grep ssh shows the three config files that OpenSSH’s sshd reads as it starts,strace sends its output to STDERR by default strace -e trace=file -p 1234 #See all file activity,Monitoring file activity strace -e trace=desc -p 1234 strace -P /etc/cups -p 2261 #track specific paths, use 1 or more times the -P parameter, following by the path strace -f -o strace_acroread.txt acroread #follow system calls if a process fork strace -q -e trace=process df -h trace all system calls involving process management. strace -q -e trace=file df -h trace all system calls that take a filename as an argument strace -f -e execve ./script.sh #check what commands are exactly being executed by a script by using strace strace -f -e execve bash x.sh $ strace e execve bash -c true $ strace -ve execve bash -c true $ strace -e execve bash -c /bin/true $ strace -o OUT -ff -e execve bash -c "/bin/true ; /bin/false" $ grep execve OUT* OUT.29328:execve("/usr/bin/bash", ["bash", "-c", "/bin/true"], 0x7ffc75ace798 /* 25 vars */) = 0 OUT.29328:execve("/bin/true", ["/bin/true"], 0x55bf673522c0 /* 25 vars */) = 0 OUT.29336:execve("/usr/bin/bash", ["bash", "-c", "/bin/true ; /bin/false"], 0x7ffe1b316638 /* 25 vars */) = 0 OUT.29337:execve("/bin/true", ["/bin/true"], 0x55aba17c92c0 /* 25 vars */) = 0 OUT.29338:execve("/bin/false", ["/bin/false"], 0x55aba17c92c0 /* 25 vars */) = 0 # Under Linux, fork is a special case of the more general clone system call, which you observed in the strace log. #The child runs a part of the shell script. The child process is called a subshell. strace -f -o bash-mystery-1.strace bash -c 'v=15; (echo $v)' strace -f -o bash-mystery-2.strace bash -c 'v=15; bash x.sh' man 2 clone #create a child process grep clone bash-mystery-2.strace # filter child process strace -e open,read,write cat /etc/HOSTNAME strace -e open,read,write cat /etc/HOSTNAME > /dev/null strace -e file cat /etc/HOSTNAME #Execute Strace on a Running Linux Process Using Option -p ps -C firefox-bin #PID 126 sudo strace -p 126 -o firefox_trace.txt #display the following error when your user id does not match the user id of the given process. pidof sshd #PID 126 strace -p 126 #sleep.sh, endless loop #! /bin/bash while : do sleep 10 & echo "Sleeping for 4 seconds.." sleep 4 done $ sh sleep.sh & # run in the background $ pstree -p #see sleep.sh child/parent processes $ pgrep sleep | sed 's/^/-p/' $ pidof sleep $ sudo strace -c -fp PID # attach to parent process of the sleep.sh $ strace -c -fp $( pgrep sleep | sed 's/^/-p/' ) # another terminal, monitor multipe=le child processes of sleep.sh $ strace $( pgrep sleep | sed 's/^/-p/' ) # another terminal, monitor multipe=le child processes of sleep.sh man 3 stat #access the documentation. stat is the system call that gets a file's status man 2 execve $ grep openat trace.log ---------------------------------------------------------------------------------------------------- PATH=/data/myscripts:$PATH #add directory /data/myscripts to the beginning of the $PATH environment variable PATH=$PATH:/data/myscripts #add that directory to the end of the path echo 'export PATH=$PATH:/new/directory' >> ~/.zshrc source ~/.zshr ---------------------------------------------------------------------------------------------------- echo "$(pwd)" #interpret backslash escapes echo -e "Tecmint \nis \na \ncommunity \nof \nLinux \nNerds" echo -e "Tecmint \bis \ba \bcommunity \bof \bLinux \bNerds" echo -e "Tecmint \tis \ta \tcommunity \tof \tLinux \tNerds" echo -e "\vTecmint \vis \va \vcommunity \vof \vLinux \vNerds" echo -e "\n\vTecmint \n\vis \n\va \n\vcommunity \n\vof \n\vLinux \n\vNerds" echo -e "Geeks \bfor \bGeeks" -e here enables the interpretation of backslash escapes,\b : it removes all the spaces in between the text echo -e "Geeks \cfor Geeks" \c : suppress trailing new line with backspace interpretor ‘-e‘ to continue without emitting new line. echo -e "Geeks \nfor \nGeeks" \n : this option creates new line from where it is used. echo -e "Geeks \tfor \tGeeks" \t : this option is used to create horizontal tab spaces. echo -e "Geeks \vfor \vGeeks \v : this option is used to create vertical tab spaces. echo -e "# SNMP version 2c community\nrocommunity monsvronly 192.168.58.8" >> /etc/snmp/snmpd.conf echo * #print all files/folders, similar to ls command ---------------------------------------------------------------------------------------------------- #sh calls the program sh as interpreter and the -c flag means execute the following command as interpreted by this program #sh -c spawns a non-login, non-interactive session of sh (dash in Ubuntu). $ sudo sh -c "echo 0" #In Ubuntu, sh is usually symlinked to /bin/dash, meaning that if you execute a command with sh -c the dash shell will be used to execute the command instead of bash $ readlink -e $(which sh) /usr/bin/dash sudo sh -c 'ls -hal /root/ > /root/test.out' #The redirection of the output is performed by sudo. sudo sh -c "echo foo > ~root/out.txt" sudo echo "foo" | sudo dd of=/root/test2.out sudo su -c 'echo "foo" > ~root/test4.out' sudo su -c 'echo "foobarr" | tee ~root/test5.out' sudo su -c 'echo "append foobarr " | tee -a ~root/test5.out' sudo ls /root/out.txt $ sudo bash -c "echo 0" $ readlink -e $(which bash) /usr/bin/bash #the features that are specific to interactive shells only (by default), e.g. history expansion, source-ing of ~/.bashrc and /etc/bash.bashrc etc will not be available in this session as it is non-interactive #simulate an interactive sessions behavior (almost), by using the -i option sh -ic 'ls -hal /root/ > /root/test.out' #the features that are specific to login shells only (by default) e.g. source-ing of ~/.profile (given ~/.bash_profile and ~/.bash_login do not exist) and /etc/profile will not be done as the shell is a non-login shell #simulate login-shells behavior using the -l option sh -lc 'ls -hal /root/ > /root/test.out' #simulate both login and interactive sessions sh -lic 'ls -hal /root/ > /root/test.out' ---------------------------------------------------------------------------------------------------- brew install #install a package (or Formula in Homebrew vocabulary) brew uninstall brew upgrade brew update brew list --versions brew search brew info brew cleanup --dry-run #see what formulas Homebrew would delete without actually deleting #Homebrew-Cask extends Homebrew and allows you to install large binary files via a command-line tool brew tap caskroom/cask #make Cask available by adding it as a tap #https://brew.sh/ /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)" #Install Homebrew ---------------------------------------------------------------------------------------------------- bash -c "$(curl -sL https://raw.githubusercontent.com/ilikenwf/apt-fast/master/quick-install.sh)" #remote script install curl https://raw.githubusercontent.com/golang/dep/v0.5.1/install.sh | sh #remote script install ---------------------------------------------------------------------------------------------------- ln -s $PWD/istioctl /usr/local/bin/istioctl #create symbolic link ---------------------------------------------------------------------------------------------------- hwclock --utc --systohc # set the hardware clock to match system (software) clock on a Linux-only computer, The --utc option tells it to use UTC, which is appropriate for a Linux-only system, --systohc sets the hardware clock based on the current value of the software clock ---------------------------------------------------------------------------------------------------- logger shutting down to add network card #create a log file entry noting that you’re manually shutting down the system to add a new network card prior to using shutdown, The logger utility can be used to create a one-time log fle entry that you specify ---------------------------------------------------------------------------------------------------- rsyslogd -v #Check if rsyslog is installed rsyslogd -f /etc/rsyslog.conf -N1 #Check Rsyslog Configuration for Errors semanage port -l| grep syslog #view the current SELinux ports settings for rsyslog semanage port -a -t syslogd_port_t -p udp 541 #add a UDP port 541 to SELinux #Verifying the Configuration #On the client server logger "Test message from the system `hostname`" nc -u 192.168.59.12 514 #verify connectivity to remote rsyslog server TCP port 50514 telnet 192.168.59.12 50514 #verify connectivity to remote rsyslog server TCP port 50514 #On the Centralized rsyslog server tail /var/log/messages tail –f /var/log/messages netstat –pnltu # Confirm That the Syslog Server Is Listening on Port 514 ss -tunlp | grep 514 ss -4altunp | grep 514 ss -tulnp | grep "rsyslog" nc -ulp 514 #cannot telnet to UDP port 514, use netcat #add the necessary UDP and/or TCP firewall rules to allow incoming syslog traffic # systemctl restart rsyslog # firewall-cmd --add-port=514/udp --permanent # firewall-cmd --add-port=514/tcp --permanent # firewall-cmd --reload ---------------------------------------------------------------------------------------------------- username must contain fewer than 32 characters and start with a letter may consist of letters, numbers, and certain symbols ---------------------------------------------------------------------------------------------------- Deleting a group does not delete all the accounts associated with the group Groups may have passwords, these are not account login passwords ---------------------------------------------------------------------------------------------------- /etc/modprobe.d #all the modules and other files, except for the optional configuration files stat /lib/modules/`uname -r` #modprobe searches the module directory more /proc/modules #a text list of the modules that the system has loaded lsmod #list of currently loaded device drivers lsmod | wc -l #total loaded Linux kernel modules lsmod | grep nvidia #see if Linux kernel drivers (modules) named nvidia loaded or not lsmod | egrep -i 'nvidia|e1000e|kvm_intel' #Search for multiple Linux device driver modules modinfo e1000 # information about specific driver ---------------------------------------------------------------------------------------------------- #KVM $ egrep -c '(vmx|svm)' /proc/cpuinfo | echo "virtualization is supported" | echo "virtualization is not supported" $ egrep -c '(vmx|svm)' /proc/cpuinfo && echo "virtualization is supported" || echo "virtualization is not supported" grep -i vmx /proc/cpuinfo #check if the CPU supports virtualization lsmod | grep kvm #check if the kvm kernel module is loaded $ grep -c ^processor /proc/cpuinfo #check that your server has (at least) 8 CPU cores To run KVM, you need a processor that supports hardware virtualization. Intel and AMD both have developed extensions for their processors, deemed respectively Intel VT-x (code name Vanderpool) and AMD-V (code name Pacifica) #If 0 it means that your CPU doesn't support hardware virtualization. #If 1 or more it does - but you still need to make sure that virtualization is enabled in the BIOS. $egrep -c '(vmx|svm)' /proc/cpuinfo $egrep -q 'vmx|svm' /proc/cpuinfo && echo yes || echo no #To use VM drivers, verify that your system has virtualization support enabled #If the above command outputs “no” #If you are running within a VM, your hypervisor does not allow nested virtualization. You will need to use the None (bare-metal) driver #If you are running on a physical machine, ensure that your BIOS has hardware virtualization enabled $cat /sys/hypervisor/properties/capabilities #if it is enabled or not from xen $kvm-ok #If you see You can still run virtual machines, but it'll be much slower without the KVM extensions INFO: Your CPU does not support KVM extensions KVM acceleration can NOT be used $egrep -c ' lm ' /proc/cpuinfo #If 0 is printed, it means that your CPU is not 64-bit. If 1 or higher it is 64-bit $ uname -m x86_64 #By default dhcpd based network bridge configured by libvirtd $ brctl show $ virsh net-list #All VMs (guest machine) only have network access to other VMs on the same server. #A private network 192.168.122.0/24 created $ virsh net-dumpxml default virt-install --name=linuxconfig-vm \ --vcpus=1 \ --memory=2048 \ --cdrom=/media/sanchez/KARNAK/linux_distributions/CentOS-Stream-8-x86_64-20210617-dvd1.iso \ --disk size=5 \ --os-variant=centos-stream8 ---------------------------------------------------------------------------------------------------- wc -mlw file1.txt file2.txt #Count words, characters, and lines in multiple files ls -l *.pdf | wc -l #Count a Certain Type of Files in a Directory wc -m yourTextFile # count the total number of characters wc -w yourTextFile #count the number of words $ wc -c file1.txt #the number of characters in a file $ wc -l file1.txt #the number of lines in a file $ head -5 .bash_history | wc -w # the number of words in the first 5 lines of the file # filecount=$(ls | wc -l) # echo $filecount ---------------------------------------------------------------------------------------------------- # terminal1 $ ls > pipe2 $ mkfifo pipe5 -m700 $ ls -l > pipe5 $ rm pipe5 $rm dir1 #remove a non-empty directory # terminal2 $ ls -lart pipe2 # list hidden files in the current directory prw-rw-r-- 1 vagrant vagrant 0 Feb 25 15:33 pipe2 $ ls -lart pipe5 prwx------ 1 vagrant vagrant 0 Feb 25 20:52 pipe5 $ cat < pipe5 total 15828 prw-rw-r-- 1 vagrant vagrant 0 Feb 25 15:33 pipe2 prw-rw-r-- 1 vagrant vagrant 0 Feb 25 20:51 pipe4 prwx------ 1 vagrant vagrant 0 Feb 25 20:52 pipe5 -rw-rw-r-- 1 vagrant vagrant 16207833 Jan 22 22:02 terraform_0.12.20_linux_amd64.zip ---------------------------------------------------------------------------------------------------- pushd #stores a directory path in the directory stack,adds directory paths onto a directory stack (history), allows you to navigate back to any directory in history pushd +2 #use the directory index in the form pushd +# or pushd -# to add directories to the stack and move into popd #removes the top directory path from the same stack popd +1 #remove a directory from the directory stack inded use popd +# or popd -# dirs #display directories in the directory stack (or history) dirs -v pushd $(pwd) && cd /opt popd ---------------------------------------------------------------------------------------------------- #from windows to linux copy problem fix $ make Makefile:21: *** missing separator. Stop. $ perl -pi -e 's/^ */\t/' Makefile # unix/windows file editing "/bin/bash^M: bad interpreter: No such file or directory" fix: sed -i -e 's/\r$//' build_all.sh ---------------------------------------------------------------------------------------------------- # ls do not provide creation time but change time $ type ll ll is aliased to `ls -alF' # -a option shows all hidden files and directories (Those who start with .") #the -F classify the results in files and folders,makes it more visual when a lot of files and directories with different extensions exist ls -alF ls -i About-TecMint #inode The Bash shell feature that is used for matching or expanding specific types of patterns is called globbing $ ls -l ????.txt #files whose names are four characters long $ ls -l foot????.doc # files whose names are 8 characters long, first 4 characters are f, o, o and t and extension is doc $ ls -l best.??? #all files with the name ‘test’ having any extension of three characters long $ ls –lt #lists files in long listing format, and sorts files based on modification time, newest first $ ls –lth $ ls –ltr #list down files /folders sorted with modified time, -r reverse order $ ls -ltr | grep "`date | awk '{print $2" "$3}'`" #todays date $ ls -ltr | grep "$(date +"%b %e")" $ ls -ltr | grep "Feb 18" #current date "Mar 22" # list files on specific dates $ ls -ltr | awk '$6 == "Feb" && $7 >=15 && $7 <= 31 {print $6,$7,$8,$9}' # list files after Feb 15t $ ls -ltr . | awk '$6 == "Feb" && $7 >=15 && $7 <= 31 {print $6,$7,$8,$9}' # list files after Feb 15t on specific directory $ ls -lrt /var/log | awk '{ total += $5 }; END { print total }' # sum of file sizes $ ls -FaGl /var/log/apt | printf "%'d\n" $(awk '{SUM+=$4}END{print SUM}') #total size $ ls -FaGl /var/log/apt | sudo tee /dev/stderr | printf "%'d\n" $(awk '{SUM+=$4}END{print SUM}') #list directory contents and total size $ ls -laUR /var/log/apt | grep -e "^\-" | tr -s " " | cut -d " " -f5 | awk '{sum+=$1} END {print sum}' #only sum up file sizes not the directory itself # total size, sum of files listed $ sumcol() > { > awk "{sum+=\$$1} END {print sum}" > } $ ls -lrt /var/log/apt/ | sumcol 5 $ ls -l | grep 'Mar 22 12:27' | tr -s ' ' | cut -d ' ' -f9 | xargs rm -rf #delete files on specific date #Listing of files in directory based on last modification time of file’s status information, or the 'ctime' #list that file first whose any status information like: owner, group, permissions, size etc has been recently changed. $ ls –lct #List Files Based on Last Modification Time $ ls –ltu #Listing of files in directory based on last access time, i.e. based on time the file was last accessed, not modified. #Sorting Ouptut of ls -l based on Date #based on 6th field month wise, then based on 7th field which is date, numerically ls -l | sort -k6M -k7n ls -l | head -n 10 | sort -k6 ls -l | head -n 10| sort -k6M -k7n #based on 6th field month wise, then based on 7th field which is date ls -lt --time=birth #sorted by creation/birth date time ls -l --time=creation #sorted by creation/birth date time $ ls -l *.pl #all files of ‘pl’ extension $ ls -l [p-s]* #all files and folders whose name contains p or q or r or s $ ls -l [1-5]* #all files and folders whose name starts with any digit from 1 to 5 $ ls -l {?????.sh,*st.txt} #files whose names are 5 characters long and the extension is ‘sh’ or the last two characters of the files are ‘st’ and the extension is ‘txt’ $ rm {*.doc,*.docx} #delete all files whose extensions are ‘doc’ or ‘docx’ $ ls a*+(.bash|.sh) #filenames which are starting with character ‘a’ and has the extension ‘bash’ or ‘sh’ ls -alt #list files in last modifed date order use the -t flag which is for 'time last modified'. ls -altr #list files in last modifed date order use the -t flag which is for 'time last modified', reverse order ---------------------------------------------------------------------------------------------------- echo test > >(cat) #the output of echo would be redirected to the file that serves as the input to cat, and cat would produce the contents of that file on standard output echo foo | cat -n echo foo > >(cat -n) # emulate pipe above The process substitution >(command) will be replaced by a file name. This file name corresponds to a file that is connected to the standard input of the "command" inside the substitution $ cat .profile | while read line; do ((counter1++)); done $ echo $counter1 $ while read line; do ((count++)); done < <(cat ~/.profile) $ echo $count 101 ---------------------------------------------------------------------------------------------------- #Network Troubleshooting Step 1: Check if your interface is configured $ ifconfig sudo resolvconf -u Step 2: Setting up your interface check if the drivers are loaded $ dmesg | grep -3 -i eth configure the interface ifconfig eth0 128.42.14.176 netmask 255.255.255.0 up Assign a Broadcast to Network Interface ifconfig eth0 netmask 255.255.255.224 If the loopback interface is not up ifconfig lo up now be able to ping your own machine $ ping -c 3 127.0.0.1 Step 3: Check if you can ping the gateway ping the DNS server Step 6: Setting up routing # route add -net netmask eth0 # route add default gw eth0 setup the loopback route if it's missing # route add -host 127.0.0.1 lo #Adding and removing routes ip ro add 10.0.0.0/16 via 192.0.2.253 ip ro del 10.0.0.0/16 via 192.0.2.253 ip ro ip -6 ro #IPv6 ip route get 8.8.8.8 | sed -n '/src/{s/.*src *\([^ ]*\).*/\1/p;q}' # IP reaching the public internet, multiple IPs interfaces ip ro get $dst_ip from $src_ip #Check routing path ip ro get 192.0.0.0 $ ip route show default via 10.0.2.2 dev eth0 proto dhcp metric 100 10.0.2.0/24 dev eth0 proto kernel scope link src 10.0.2.15 metric 100 192.168.18.0/24 dev eth1 proto kernel scope link src 192.168.18.9 metric 101 traffic to anywhere else should be forwarded through eth0 to the gateway at 10.0.2.2 traffic to 10.0.2.2 (the gateway to the public Internet) should be forwarded directly to its destination through eth0 traffic to anywhere within 192.168.18.0/24 (the local area network) should be forwarded directly to its destination through eth1 $ ip route get to 192.168.18.12 from 192.168.18.9 iif eth1 $ ip route get to 192.168.18.9 from 192.168.18.12 iif eth1 $ route -n route add dest_ip_address gateway_address #Add a route to a destination through a gateway route add subnet_ip_address/24 gateway_address #Add a route to a /24 subnet through a gateway route -t add dest_ip_address/24 gateway_address #Run in test mode (does not do anything, just print) route flush #Remove all routes route delete dest_ip_address/24 #Delete a specific route Step 7: Name resolution 3 files: /etc/host.conf, /etc/hosts, /etc/resolv.conf /etc/host.conf: order hosts,bind multi on /etc/hosts: 127.0.0.1 localhost loopback this.host.name /etc/resolv.conf: domain yourdept.yourcompany.com search yourdept.yourcompany.com yourcompany.com nameserver ---------------------------------------------------------------------------------------------------- # ping a host ping 192.168.0.2 #show routing table without resolving domain names netstat -nr netstat -r -n # The flag U indicates that route is up and G indicates that it is gateway netstat -alun | grep 161 # show informations about errors/collisions netstat -ni # show statistics about your network card netstat -i -I em0 netstat -a netstat -at netstat -s netstat -au netstat -l netstat -lu netstat -lt netstat -tulpn netstat -plan netstat -plan | grep ":80" netstat -lntp | grep ':8080.*java' > /dev/null && command netstat -pan -A inet,inet6 | grep -v ESTABLISHED #determine which ports are listening for connections from the network netstat -tlnw #Use the -l option of the netstat command to display only listening server sockets: netstat -plnS #Scan for Open SCTP Ports netstat -nl -A inet,inet6 | grep 2500 #Scan for Open SCTP Ports netstat -pant | grep -Ei 'apache|:80|:443' netstat -tunlp | grep ":80 " List all TCP sockets and related PIDs netstat -antp netstat -anp List all UDP sockets and related PIDs netstat -anup # find out on which port a program is running netstat -ap | grep ssh #If there is an IP address instead, then the port is open only on that specific interface #For listening ports, if the source address is 0.0.0.0, it is listening on all available interfaces #The Recv-Q and Send-Q fields show the number of bytes pending acknowledgment in either direction #the PID/Program name field shows the process ID and the name of the process responsible for the listening port or connection netstat -anptu #number of established connection netstat -an|grep ESTABLISHED|awk '{print $5}'|awk -F: '{print $1}'|sort|uniq -c|awk '{ printf("%s\t%s\t",$2,$1); for (i = 0; i < $1; i++) {printf("*")}; print ""}' #see that the Nessus server is up and running netstat -n | grep tcp netstat -tap | grep LISTEN netstat -pltn | grep 8834 ---------------------------------------------------------------------------------------------------- #ss is the socket statistics command that replaces netstat ss -tr #netstat -t ss -ntr #see port numbers ss -an |grep LISTEN #netstat -an |grep LISTEN ss -an | grep 2500 #show SCTP open ports ss -tlw # list open ports in the listening state ss -plno -A tcp,udp,sctp #The UNCONN state shows the ports in UDP listening mode ---------------------------------------------------------------------------------------------------- # find route to example.com traceroute www.example.com #find route to example.com using tcptraceroute (which uses tcp to discover path) tcpdraceroute www.example.com # The maximum number of hops can be adjusted with the -m flag. traceroute -m 255 obiwan.scrye.net # adjust the size of the packet that is sent to each hop by giving the integer after the hostname traceroute google.com 70 Specify Gateway sudo traceroute -g 10.0.2.2 yahoo.com traceroute -g 192.5.146.4 -g 10.3.0.5 35.0.0.0 #shows the path of a packet that goes from istanbul to sanfrancisco through the hosts cairo and paris #The -I option makes traceroute send ICMP ECHO probes to the host sanfrancisco #The -i options sets the source address to the IP address configured on the interface qe0 traceroute -g cairo -g paris -i qe0 -q 1 -I sanfrancisco ip r / ip route #gateway / router ip r | grep default #default gateway route -n # The flag U indicates that route is up and G indicates that it is gateway route -nee netstat -r -n # The flag U indicates that route is up and G indicates that it is gateway routel #list routes routel | grep default #default gateway Specify Source Interface sudo traceroute -i eth0 yahoo.com Autonomous Systems traceroute -A yahoo.com tracepath yahoo.com tracepath -n yahoo.com tracepath -b yahoo.com sets the initial packet length tracepath -l 28 yahoo.com set maximum hops (or maximum TTLs) to max_hops tracepath -m 5 yahoo.com set the initial destination port to use tracepath -p 8081 yahoo.com hostname -f #show fully qualified domain name (FQDN) hostname #shortname hostname -I #the primary/first IP address hostname -I | awk '{print $1}' # get primary/first IP IP address of server hostname -I | awk '{print $2}' # get second IP IP address of server real-time view of the current state of your system $ htop $ timedatectl $ timedatectl list-timezones $ sudo timedatectl set-timezone 'Africa/Lubumbashi' # Enable NTP synchronization timedatectl set-ntp true timedatectl status # show connected sockets sockstat -c # show listening sockets and processes sockstat -l # show arp table arp -a #Show current arp tabl arp -na ip neighbour #Show neighbors (ARP table) arp -d 192.168.0.2 # delete a record from arp table sudo arp -a -d #Clear the entire cach arp -s 192.168.0.2 00:10:b5:99:bf:c4 #add a static record in arp table # listen on em0 network interface and sniff packets that pass via em0 $ sudo arp -i eth0 find out reachability of an IP on the local Ethernet with arping i.e send ARP request 192.168.1.1: $ sudo arping -I eth0 -c 3 192.168.18.12 $ sudo arping -I eth1 -c 3 192.168.18.12 Find duplicate IP $ sudo arping -D -I eth1 -c 3 192.168.18.12 ping -I eth0 8.8.8.8 #ping 8.8.8.8 using eth0 as a source interface ping google.com ping -6 hostname/IPv6 #request IPv6 or IPv4 address ping -4 hostname/IPv4 #The default interval between each ping request is set to one second. You can increase or decrease that time using the –i switch #decrease the ping interval, use values lower than 1 ping -i 0.5 google.com ping -s 1000 google.com #use -s to increase the packet size from the default value of 56 (84) bytes. ping -f hostname-IP #use ping flood to test your network performance under heavy load ping -c 2 google.com #Limit Number of Ping Packets ping -w 25 google.com #stop printing ping results after 25 seconds ping -c 10 -q google.com #The letter “q” in this command stands for “quiet” output. ping -D google.com #Add Timestamp Before Each Line in ping Output $ file * 20:30: empty file1: ASCII text file2: ASCII text $ file raw_file.txt #if a text file contains ASCII or Unicode $ file -b symbolic_test1.txt symbolic link to test1.txt ----------------------------------------------------------------------------------------------------- # parse html with curl curl -s https://miloserdov.org/ | grep -E -o '

.*

' | sed 's/

//' | sed 's/<\/h3>//' curl http://test.com | sed -rn 's@(^.*
)(.*)(
)@\2@p' cat - > file.html << EOF

Diplo - Justin Bieber - Skrillex

Where Are U Now

toto

tata

EOF cat file.html | tr -d '\n' | sed -e "s/<\/div>/<\/div>\n/g" | sed -n 's/^.*class="artist">\([^<]*\)<\/p> *

\([^<]*\)<.*$/artist : \1\ntitle : \2\n/p' cat file.html | grep -A2 -E -m 1 '

' cat file.html | grep -A2 -E -m 1 '
' | tail -n1 cat file.html | grep -A2 -E -m 1 '
' | tail -n2 | head -n1 cat file.html | grep -A2 -E -m 1 '
' | tail -n2 | head -n1 | sed 's/<[^>]*>//g' ----------------------------------------------------------------------------------------------------- Check what ethernet devices exist currently # ls -al /sys/class/net # ls -Rl list hidden files and the contents of all subdirectories ls -aR /home/username # ls -pu see eth* devices # ls -al /sys/class/net Get the PCI address of the NIC # lspci | grep Mellanox # lspci | grep Eth iw distinguishes between wireless LAN hardware devices (the physical layer, referred to as phy) and the network interface configured to use that hardware (e.g. wlan0, similar to an Ethernet eth0 interface). To see the list of devices, and interfaces for each device #iw dev **MAC_ADDRESS 08:00:27:e3:b0:01 $ ip link show eth1 3: eth1: mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000 link/ether 08:00:27:e3:b0:01 brd ff:ff:ff:ff:ff:ff $ ip addr show eth1 3: eth1: mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 08:00:27:e3:b0:01 brd ff:ff:ff:ff:ff:ff inet 192.168.1.253/24 brd 192.168.1.255 scope global noprefixroute eth1 valid_lft forever preferred_lft forever inet6 fe80::a00:27ff:fee3:b001/64 scope link valid_lft forever preferred_lft forever #temporarily set the IP address ifconfig eth0 192.168.8.185 ifconfig eth0 192.168.8.185 netmask 255.255.255.0 up #temporarily change the MAC address ifconfig eth0 down hw ether AA:BB:CC:DD:EE:FF && ifconfig eth0 up ifconfig eth0 netmask 255.255.255.0 ifconfig eth0 broadcast 192.168.70.255 ip addr show -> List IP address of the server ip addr show eth0 ip addr | grep inet6 #check that your server supports IPV6 ip addr show eth1 | grep "inet " ip addr add 10.132.1.1/24 dev eth1 -> Add a new IP4 address ip addr show eth1 -> confrm that the new address is available on the interface ip link set eth2 down -> bring an interface down ip link set eth2 up ip -s link->view basic network statistics on all interfaces ip -s link ls eth0 ->see the statistics for the eth0 interface ip -s -s link ls eth0 ->see additional info ss -t ->show established TCP connections ss -u ->show established UDP connections ss -A tcp ss -x ss -ltn ->see which ports are listening for connections ss -nt ss -ltn ss -ua ss -a -A udp ss -lun ->udp ss -s->prints out the statistics #Install sysstat package # /etc/default/sysstat ENABLED="true" #sudo service sysstat restar vmstat 1 99999 ->the system statistics every second, for the number of times specifed (99999 in this instance) vmstat –a 1 99 ->show memory usage information vmstat -a -S M 1 9 -> reformat in Mega Bytes vmstat 1 99999 ->gather information for disks and other block devices vmstat -d -w iostat 1 9 ->CPU information and disk information for all devices iostat -d -p sda 1 9-> show information for device sda with disk statistics sar -u 1 30 -> display CPU statistics every second for 30 seconds sar -r 1 30 -> display memoru statistics every second for 30 seconds sar -b 1 30 -> display block device statistics every second for 30 seconds #===================================================================== # curl the binay version info from latest download and later wget the file with $curl -s https://api.github.com/repos/prometheus/prometheus/releases/latest \ | grep browser_download_url \ | grep linux-amd64 \ | cut -d '"' -f 4 \ | wget -qi - $ ls prometheus-2.37.0.linux-amd64.tar.gz $tar xvf prometheus*.tar.gz $cd prometheus*/ #specify directory and rename the file wget --output-document="/home/my_new_file_name" http://someurl #add the appropriate BeeGFS repositories wget -o /etc/yum.repos.d/beegfs-rhel7.repo http://www.beegfs.com/release/beegfs_2015.03/dists/beegfs-rhel7.repo wget -q https://www.virtualbox.org/download/oracle_vbox.asc -O- | apt-key add - wget example.com/big.file.iso #start download and stop download ctrl+c key pair wget -c example.com/big.file.iso #resume download wget ‐‐continue example.com/big.file.iso #Resume an interrupted download previously started by wget itself wget ‐‐continue ‐‐timestamping wordpress.org/latest.zip #Download a file but only if the version on server is newer than your local copy wget ‐‐page-requisites ‐‐span-hosts ‐‐convert-links ‐‐adjust-extension http://example.com/dir/file #Download a web page with all assets - like stylesheets and inline images - that are required to properly display the web page offline. wget -q http://somesite.com/TheFile.jpeg #-q: Turn off wget's output wget http://example.com/images/{1..20}.jpg # Download a list of sequentially numbered files from a server wget -m -r -linf -k -p -q -E -e robots=off http://127.0.0.1 # Download a complete website wget ‐‐mirror ‐‐domains=abc.com,files.abc.com,docs.abc.com ‐‐accept=pdf http://abc.com/ #Download the PDF documents from a website through recursion but stay within specific domains. wget ‐‐execute robots=off ‐‐recursive ‐‐no-parent ‐‐continue ‐‐no-clobber http://example.com/ #Download an entire website including all the linked pages and files wget ‐‐level=1 ‐‐recursive ‐‐no-parent ‐‐accept mp3,MP3 http://example.com/mp3/ #Download all the MP3 files from a sub-directory wget --recursive --no-clobber --page-requisites --html-extension --convert-links --restrict-file-names=windows --domains some-site.com --no-parent www.some-site.com #Download Entire Website wget ‐‐recursive ‐‐no-clobber ‐‐no-parent ‐‐exclude-directories /forums,/support http://example.com #Download all files from a website but exclude a few directories wget --reject=png www.some-site.com #Reject file types while downloading wget -r -A .pdf http://some-site.com/ #Download all PDF files from a website wget -r -H --convert-links --level=NUMBER --user-agent=AGENT URL #Download With Wget Recursively,declare a user agent such as Mozilla (wget –user-agent=AGENT) wget -e https_proxy=xx.xx.xx.xx:8080 https://example.com/ #use proxy server with wget wget -S --spider http://www.uniqlo.com/ #Only Header Information ##if link exists url="https://www.katacoda.com/courses/kubernetes/launch-single-node-cluster" if wget --spider "$url" 2>/dev/null; then #2> /dev/null silences wget's stderr output echo "URL exists: $url" else echo echo "URL does not exist: $url" fi #connect to a remote server,start download on the remote server,disconnect from the remote server,let it run on the background $ nohup wget -q url & wget -i file.txt #Read download URLs from a file,useful in a shell script. #one liner if condition wget --spider http://192.168.50.15/${distribution}_${codename}_oscap_report.html 2>/dev/null && echo "link exists" || echo "link does not exist" wget --spider -S "www.magesh.co.in" 2>&1 | awk '/HTTP\// {print $2}' #see only the HTTP status code wget --spider -o wget.log -e robots=off --wait 1 -r -p http://www.mysite.com/ #crawl a website and generate a log file of any broken links wget --spider https://example.com/filename.zip 2>&1 | grep Length #file download size without downloading the actual file wget ‐‐spider ‐‐server-response http://example.com/file.iso #Find the size of a file without downloading it (look for ContentLength in the response, the size is in bytes) wget ‐‐output-document - ‐‐quiet google.com/humans.txt #Download a file and display the content on the screen without saving it locally wget ‐‐server-response ‐‐spider http://www.labnol.org/ #the last modified date of a web page (check the LastModified tag in the HTTP header) wget ‐‐output-file=logfile.txt ‐‐recursive ‐‐spider http://example.com #Check the links on your website to ensure that they are working. The spider option will not save the pages locally. wget ‐‐limit-rate=20k ‐‐wait=60 ‐‐random-wait ‐‐mirror example.com # limited the download bandwidth rate to 20 KB/s and the wget utility will wait anywhere between 30s and 90 seconds before retrieving the next resource. wget -O index.html --certificate=OK.crt --private-key=OK.key https://example.com/ #Client SSL Certificate wget -q -O - --header="Content-Type:application/json" --post-file=foo.json http://127.0.0.1 # POST a JSON file and redirect output to stdout wget -O wget.zip http://ftp.gnu.org/gnu/wget/wget-1.15.tar.gz #Download file with different name wget -o download.log http://ftp.gnu.org/gnu/wget/wget-1.15.tar.gz #redirect the wget command logs to a log file using ‘-o‘ switch. wget ‐‐output-document=filename.html example.com #Download a file but save it locally under a different name wget ‐‐directory-prefix=folder/subfolder example.com #Download a file and save it in a specific folder wget -r -l inf -A .png,.jpg,.jpeg,.gif -nd https://jekyllrb.com # Download all images of a website wget -r --level=1 -H --timeout=1 -nd -N -np --accept=mp3 -e robots=off -i musicblogs.txt #take a text file of your favourite music blogs and download any new MP3 files wget --ftp-user=User --ftp-password=Mir URL # FTP download wget http://ftp.gnu.org/gnu/wget/wget-1.15.tar.gz ftp://ftp.gnu.org/gnu/wget/wget-1.14.tar.gz.sig #Download multiple file with http and ftp protocol wget -i /wget/urls.txt #Read URL’s from a file wget -Q10m -i download-list.txt #Setting Download Quota wget -c http://ftp.gnu.org/gnu/wget/wget-1.15.tar.gz #Resume download wget -b /wget/log.txt http://ftp.gnu.org/gnu/wget/wget-1.15.tar.gz #Download files in background wget -b -c --tries=NUMBER URL #number of tries (wget –tries=NUMBER), continue partial download (wget -c) wget -b --limit-rate=SPEED -np -N -m -nd --accept=mp3 --wait=SECONDS http://www.uniqlo.com/ #no parent to ensure you only download a sub-directory (wget -np),update only changed files (wget -N), mirror a site (wget -m), ensure no new directories are created (wget -nd), accept only certain extensions (wget –accept=LIST) wget -c --limit-rate=100k /wget/log.txt http://ftp.gnu.org/gnu/wget/wget-1.15.tar.gz #Limit download speed wget --http-user=username --http-password=password http://some-network.net/some-file.txt #Options –http-user=username, –http-password=password wget --ftp-user=username --ftp-password=password ftp://some-network.net/some-file.txt #–ftp-user=username, –ftp-password=password wget --tries=75 http://ftp.gnu.org/gnu/wget/wget-1.15.tar.gz #Increase Retry Attempts. wget ‐‐refer=http://google.com ‐‐user-agent="Mozilla/5.0 Firefox/4.0.1" http://nytimes.com #Wget can be used for downloading content from sites that are behind a login screen or ones that check for the HTTP referer and the User-Agent strings of the bot to prevent screen scraping. wget ‐‐cookies=on ‐‐save-cookies cookies.txt ‐‐keep-session-cookies ‐‐post-data 'user=labnol&password=123' http://example.com/login.php_ _wget ‐‐cookies=on ‐‐load-cookies cookies.txt ‐‐keep-session-cookies http://example.com/paywall #Fetch pages that are behind a login page. You need to replace user and password with the actual form fields while the URL should point to the Form Submit (action) page. wget ‐‐span-hosts ‐‐level=inf ‐‐recursive dmoz.org # wget -r --level=inf -p -k -E --span-hosts --domains=domainA,domainB http://www.domainA #download an entire site (domain A) when its resources are on another domain, (domain B) wget --page-requisites --convert-links --adjust-extension --span-hosts --domains domainA,domainB domainA # wget --recursive --level=inf --page-requisites --convert-links --html-extension -rH -DdomainA,domainB domainA # wget --recursive --level=inf --page-requisites --convert-links --adjust-extension --span-hosts --domains=domainA,domainB domainA # #===================================================================== sudo permission denied The redirection to a file is handled by bash. It does therefore not inherit permissions granted by sudo. "sudo tee" for writing to a file as root. lsblk -f -> when used with the -f option, it prints file system type on partitions sudo file -sL /dev/sdb1 -> file system type on partitions lsblk -f lsblk -l lsblk --scsi lsblk -o name,type,fstype,label,partlabel,model,mountpoint,size lsblk --json | jq -c '.blockdevices[]|[.name,.size]' #/etc/fstab explained, Each field can be separated by another either by spaces or tabs First field – The block device,reference a block device is by using its LABEL or UUID (Universal Unique IDentifier) $ lsblk -d -fs /dev/sdb1 # get UUID Second field – The mountpoint Third field – The filesystem type Fourth field – Mount options, use the default set of mount options we specify default as a value Fifth field – Should the filesystem be dumped?, either 0 or 1,used by the dump backup program (if installed) Sixth field – Fsck order;fsck utility, should check filesystems on boot; value of 1 must always be used for the root filesystem if not root filesystem,for all the others, value of 2 If not provided it defaults to 0 # generate traces of the i/o traffic on block devices "sudo blktrace -d /dev/sda -o - | blkparse -i -" #writing ISO usb bootable sudo umount /dev/sdX sudo dd if=/path/to/ubuntu.iso of=/dev/sdX bs=4M && sync sdx-> lslbk command sync-> sync bit is important as dd can return before the write operation finishes. # mount all file systems on /etc/fstab mount -a mount -fav cat /proc/mounts # format linux swap partition mkswap -------------------------------------------------------------------------------------------------------------------- #search man pages man -k . #List all available man pages on the system and provide a short description man -f virt-manager # Keyword must be the exact name of a man page man -K date -> display a list of all manual pages containing the keyword "date" man -wK word ->list out all manual files with some word. man -wK --regex "ipv4.*listen" man --regex ".*network$" #search through all man pages on a system for the specified keyword and output the location of any man pages that contain that keyword man -S 1 --regex "^virsh" man yum | less +/"install @" #search "install @" pattern in man yum page man -P 'less -p install ' yum #search "install" pattern in man yum page -------------------------------------------------------------------------------------------------------------------- # Clone / Compile specific kernel sudo git clone git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git linux -> latest stable kernel to "linux" directory git tag -l | grep v4.9.1 -> find specific kernel version git checkout -b kernel490 v4.9.1 -> switch to kernel with custom name "kernel490" #tail -f vs less +F #tail -f only reads the end of the file #tail -f doesn't keep the whole file in memory. tail -f /var/log/syslog #less +F reads the whole file #less +F impractical for very large files. #less +F can highlight, search, navigate through file #hit Ctrl-c to go to “normal” less mode (as if you had opened the file without the +F flag) #the search with /foo,next or previous occurrence with n or N, up and down with j and k,, create marks with m #hit F to go back to watching mode less +F /var/log/syslog less -n +F #read only the end of the file tail -100f /var/log/messages | grep -V ACPI | grep -i ata #real-time monitoring,tailing 100 lines from the end tail -f /var/log/nginx/access.log /var/log/nginx/access.log #watching multiple files #top 10 internet protocol addresses hitting a webserver from the access log cat /var/log/nginx/access.log | cut -f 1 -d ' ' | sort | uniq -c | sort -hr | head -n 10 multitail /var/log/auth.log /var/log/kern.log #shows the contents of log files horizontally multitail -s 2 /var/log/auth.log /var/log/kern.log #view the contents of log files vertically in two columns multitail -s 2 /var/log/syslog, /var/log/kern.log, /var/log/daemon.log and var/log/messages multitail -s 2 -ci green /var/log/auth.log -ci blue /var/log/kern.log lnav /var/log/syslog /var/log/messages lnav a.zip b.zip -------------------------------------------------------------------------------------------------------------------- #crontab Display scheduled jobs for the specified user crontab -l -u vagrant crontab -l # Display Cron Table ls -la /etc/cron* sudo crontab -u user -e # crontab -e #when running as a non-root user sudo crontab -e #the root user's crontab #If the /etc/cron.allow file exists, then users must be listed in it in order to be allowed to run the crontab command #If the /etc/cron.allow file does not exist but the /etc/cron.deny file does, #then users must not be listed in the /etc/cron.deny file in order to run crontab #If the /etc/cron.allow file exists, then users must be listed in it in order to be allowed to run the crontab command /etc/cron.deny # If a blank cron.deny file has been created, cron only available to root or users in cron.allow. # Delete All Cron Jobs crontab -r crontab -r -i #the command prompt to confirm # All scripts in each directory are run as root #Cron jobs may not run with the environment, in particular the PATH, that you expect. Try using full paths to files and programs #The "%" character is used as newline delimiter in cron commands. If you need to pass that character into a script, you need to escape it as "\%". #anacron uses the run‑parts command and /etc/cron.hourly, /etc/cron.weekly, and /etc/cron.monthly directories. #anacron itself is invoked from the /etc/crontab file #user crontabs /etc/cron.hourly/, /etc/cron.daily/, /etc/cron.weekly/, and /etc/cron.monthly/ ls -la /etc/cron.daily/ #View daily cron jobs crontab -l > backup_cron.txt #Backup All Cron Jobs /etc/crontab #not recommended that you add anything,this could cause a problem if the /etc/crontab file is affected by updates /etc/cron.d #not be affected by updates, several people might look after a server, then the directory /etc/cron.d is probably the best place to install crontabs /etc/cron.d #These files also have username fields #the files inside /etc/cron.d chown root:root /etc/cron.d/* chmod go-wx /etc/cron.d/* chmod -x /etc/cron.d/* $ cat /etc/cron.allow barak $ sudo systemctl restart cron $ cat /etc/cron.d/barak_job */1 * * * * barak echo "Nightly Backup Successful: $(date) " > /tmp/test1_job.log */1 * * * * /usr/bin/free -m | awk '{ if($1 == "Mem:" ) print $3}' | awk '{ if ( $1 > 140 ) print $0; else print "less" }' >> /tmp/memo.log #troubleshooting cron $ cat /etc/cron.d/barak_job */1 * * * * barak echo "Nightly Backup Successful: $(date) " &> /tmp/test1_job.log #redirect stdout and stderr to a file. */1 * * * * barak echo "Nightly Backup Successful: $(date) " > /tmp/test1_job.log2 2>&1 #redirect stdout and stderr to a file. #php specific php /bla/bla/something.php >> /var/logs/somelog-for-stdout.log #the only difference from the syntax of the user crontabs is that the line specifies the user to run the job as 00 01 * * * rusty /home/rusty/rusty-list-files.sh #run Rusty's command script as user rusty from his home directory. /usr/bin/php /home/username/public_html/cron.php #Execute PHP script: mysqldump -u root -pPASSWORD database > /root/db.sql #MySQL dump: /usr/bin/wget --spider "http://www.domain.com/cron.php" #Access URL: $ cat /etc/cron.allow barak $ sudo systemctl restart cron/crond $ cat /etc/cron.d/barak_job */1 * * * * barak echo "Nightly Backup Successful: $(date)" >> /tmp/mybackup.log $ crontab -u barak -l #*/1 * * * * barak echo "Nightly Backup Successful: $(date) runs" >> /tmp/barak_job.log $ sudo tail -f /var/log/syslog | grep --color=auto CRON crontab -e @hourly echo "Nightly Backup Successful: $(date)" >> /tmp/mybackup.log #"-u borg" is used take the identity of the borg user # cat /etc/cron.daily/borgbackup_check #!/bin/bash sudo -u borg borg check /borgbackup >> /var/log/borgbackup.log #once every 5 minutes cat | sudo tee /etc/cron.d/cron-mrtg << EOF */5 * * * * env LANG=C /usr/bin/mrtg /etc/mrtg.cfg EOF #verify cat /etc/cron.d/cron-mrtg crontab -l cat | sudo tee /etc/cron.d/sysinfo << EOF #once every 5 minutes */5 * * * * /bin/bash /home/vagrant/sysinfo_func.sh EOF #verify cat /etc/cron.d/cron-mrtg * * * * * /bin/date >> /tmp/cron_output #This will append the current date to a log file every minute. * * * * * /usr/bin/php /var/www/domain.com/backup.php > /dev/null 2>&1 #run a script but keep it running in the background at specific time 00 15 * * 4 sh /root/test.sh 35 21 * * 7 /bin/date >> /tmp/cron_output every 5 minutes */5 * * * * mycommand an hourly cron job but run at minute 15 instead (i.e. 00:15, 01:15, 02:15 etc.): 15 * * * * [command] once a day, at 2:30am: 30 2 * * * [command] once a month, on the second day of the month at midnight (i.e. January 2nd 12:00am, February 2nd 12:00am etc.): 0 0 2 * * [command] on Mondays, every hour (i.e. 24 times in one day, but only on Mondays): 0 * * * 1 [command] three times every hour, at minutes 0, 10 and 20: 0,10,20 * * * * [command] # Stop download Mon-Fri, 6am 0 6 * * 1,2,3,4,5 root virsh shutdown download */5 * * * * /path/to/some-script.sh #every 5 minutes @reboot /scripts/script.sh #tasks to execute on system reboot @hourly /scripts/script.sh #execute on an hourly. 0 * * * */scripts/script.sh #execute on an hourly. @daily /scripts/script.sh # execute on a daily basis. 0 2 * * * /scripts/script.sh # executes the task in the second minute of every day.# @weekly /bin/script.sh #execute on a weekly basis 0 0 4 * sun /bin/script.sh #execute on a weekly basis @monthly /scripts/script.sh #execute on a monthly basis 0 0 1 * * /scripts/script.sh #execution of a task in the first minute of the month @yearly /scripts/script.sh #schedule tasks on a yearly basis. @yearly /scripts/script.sh #executes the task in the fifth minute of every year. * * * * * sleep 15; /scripts/script.sh #schedule a cron to execute after every 15 Seconds 0 4,17 * * mon,tue /scripts/script.sh #execute twice on Monday and Tuesday 0 17 * * mon,wed /script/script.sh #run each Monday and Wednesday at 5 PM 0 7,17 * * * /scripts/script.sh #execute at 7 AM and 5 PM daily 0 5 * * mon /scripts/script.sh #execute the task on every Monday at 5 AM 0 */6 * * * /scripts/script.sh #run a script for 6 hours interval 0 8-10 * * * /scripts/script.sh # run every hour between 08-10AM 0 2 * * sat [ $(date +%d) -le 06 ] && /script/script.sh #execute on first Saturday of every month 0 12 1-7 * * [ "$(date '+\%a')" = "Mon" ] && echo "It's Monday" #on the first Monday of every month * * * feb,jun,sep * /script/script.sh #run tasks in Feb, June and September months -------------------------------------------------------------------------------------------------------------------- tail -n 1 /usr/share/dict/words #limit the number of lines to show tail -c 24 /usr/share/dict/words #limit the number of bytes to show tail /usr/share/dict/words /usr/share/dict/french #show the last ten lines of multiple files tail -n 5 file_1 file_2 tail -n +N #the lines starting from line number N tailf -F /var/log/syslog #monitoring the log file even at its rotation tail -q /usr/share/dict/words /usr/share/dict/french #suppress the header line pass the -q option ls -t /etc | tail -n 5 #show the five files or folders modified the longest time ago #brctl show -> Bridge connections -------------------------------------------------------------------------------------------------------------------- #LVM pvdisplay pvck pvs lvscan lvdisplay lvmdiskscan vgchange vgscan -a y e4defrag -cv /path/to/myfiles (defrag folder ) $ sudo pvcreate /dev/sdb $ sudo pvs PV VG Fmt Attr PSize PFree /dev/sda2 centos lvm2 a-- <63.00g 4.00m /dev/sdb vg_iscsi lvm2 a-- <30.00g 0 $ sudo pvdisplay --- Physical volume --- PV Name /dev/sdb VG Name vg_iscsi PV Size 30.00 GiB / not usable 4.00 MiB Allocatable yes (but full) PE Size 4.00 MiB Total PE 7679 Free PE 0 Allocated PE 7679 PV UUID hG93NW-gvRB-njUP-pgj8-omRF-YzFe-rTMWOz --- Physical volume --- PV Name /dev/sda2 VG Name centos PV Size <63.00 GiB / not usable 3.00 MiB Allocatable yes PE Size 4.00 MiB Total PE 16127 Free PE 1 Allocated PE 16126 PV UUID rFHI2D-fvZw-Mf2P-gKTC-ZTwt-vdiY-TEQc14 $ sudo vgcreate vg_iscsi /dev/sdb $ sudo vgdisplay --- Volume group --- VG Name vg_iscsi System ID Format lvm2 Metadata Areas 1 Metadata Sequence No 2 VG Access read/write VG Status resizable MAX LV 0 Cur LV 1 Open LV 0 Max PV 0 Cur PV 1 Act PV 1 VG Size <30.00 GiB PE Size 4.00 MiB Total PE 7679 Alloc PE / Size 7679 / <30.00 GiB Free PE / Size 0 / 0 VG UUID j63noX-S9I0-5Gp0-3FPg-IZ23-oZNK-6qpb7X $ sudo lvcreate -l 100%FREE -n lv_iscsi vg_iscsi [vagrant@vg-suricata-30 ~]$ sudo lvscan ACTIVE '/dev/vg_iscsi/lv_iscsi' [<30.00 GiB] inherit ACTIVE '/dev/centos/swap' [2.00 GiB] inherit ACTIVE '/dev/centos/home' [<20.01 GiB] inherit ACTIVE '/dev/centos/root' [40.98 GiB] inherit $ sudo lvdisplay --- Logical volume --- LV Path /dev/vg_iscsi/lv_iscsi LV Name lv_iscsi VG Name vg_iscsi LV UUID exEdIG-s2bK-vFEa-fD3X-dplu-q2W3-1rTXsE LV Write Access read/write LV Creation host, time vg-suricata-30, 2019-12-18 12:35:56 +0000 LV Status available # open 0 LV Size <30.00 GiB Current LE 7679 Segments 1 Allocation inherit Read ahead sectors auto - currently set to 8192 Block device 253:3 $ sudo vgremove vg_iscsi $ sudo pvremove /dev/sdb -------------------------------------------------------------------------------------------------------------------- #troubleshooting prometheus #Shutting down Prometheus lsof -n -i4TCP:9090 #find a process (Prometheus) listening on port 9090 pgrep -f prometheus curl -X POST :9090/-/quit curl -X POST http://localhost:9090/-/quit # (when the --web.enable-lifecycle flag is enabled) #Prometheus Reload curl -i -XPOST localhost:9090/-/reload killall -HUP prometheus ./promtool check config prometheus.yml #check if prometheus.yml is valid journalctl --boot | grep prometheus #reload Prometheus config file kill -HUP 9783 -------------------------------------------------------------------------------------------------------------------- #list open files lsof #list open files owned by user1 lsof -u user1 #list open file via tcp lsof -i TCP:1-1024 lsof -i TCP:80 PID 27808 lsof -Pan -p 27808 -i lsof -p 2 # troubleshooting #1 find all the opened files and processes along with the one who opened them # lsof –p PID Count number of files & processes # lsof -p 4271 | wc -l Check the currently opened log file lsof –p | grep log Find out port number used by daemon # lsof -i -P |grep 4271 # find out what running processes are associated with each open port on Linux netstat -nlp|grep 9000 sudo ss -lptn 'sport = :80' sudo netstat -nlp | grep :80 sudo lsof -n -i :80 | grep LISTEN fuser 3306/tcp fuser 80/tcp ss -tanp | grep 6379 fuser -v -n tcp 22 sudo netstat -ltnp | grep -w ':80' netstat -tulpn | grep :80 netstat -tulpn ls -l /proc/1138/exe sudo ss -tulpn sudo ss -tulpn | grep :3306 fuser 7000/tcp ls -l /proc/3813/exe man transmission whatis transmission # find out current working directory of a process pid 3813 ls -l /proc/3813/cwd pwdx 3813 # Find Out Owner Of a Process on Linux cat /proc/3813/environ grep --color -w -a USER /proc/3813/environ lsof -i :80 | grep LISTEN # The file /etc/services is used to map port numbers and protocols to service names grep port /etc/services grep 443 /etc/services #Start a Linux Process or Command in Background $ tar -czf home.tar.gz . $ tar -tvf home.tar.gz # list the contents of a .tar file $ bg $ jobs OR $ tar -czf home.tar.gz . & $ jobs #Keep Linux Processes Running After Exiting Terminal $ sudo rsync Templates/* /var/www/html/files/ & $ jobs $ disown -h %1 $ jobs OR $ nohup tar -czf iso.tar.gz Templates/* & $ jobs #Detach a Linux Processes From Controlling Terminal firefox /dev/null & count & # count command running on the background jobs fg bg fg %# #Replace the # with serial number of the job,bring any job in the foreground fg %2 #bring job 2 into the foreground jobs -l count 2> /dev/null & $ tail -f temp.log #Placing a Foreground Job into the Background,suspend the job with a Ctrl-Z, ^Z[1]+ Stopped tail -f temp.log $ bg # bg to place the suspended job in the background $ jobs # list the jobs in the background $ jobs -l # list job in the background, process id 105231 [1]+ 105231 Running $ fg 1 # bring job #1 in the foreground from the background, process id 105231 sudo rsync $ fg 1 #type ctrl+z to send the job #1 to the background, process id 105231 sudo rsync ^Z [1]+ Stopped $ jobs -l # list the job #1 in the background which is stopped, process id 105231 [1]+ 105231 Stopped $ bg 1 # run the job 1 in the background again, process id 105231 [1]+ sudo rsync $ jobs -l # list the job #1 in the background, process id 105231 [1]+ 105231 Running ----------------------------------------------------------------------------------------------------- 2 is the file descriptor of stderr the integer file descriptors associated with the streams stdin, stdout, and stderr are 0, 1, and 2, respectively. a number 0 = standard out (i.e. STDIN) a number 1 = standard out (i.e. STDOUT) a number 2 = standard error (i.e. STDERR) if a number isn't explicitly given, then number 1 is assumed by the shell (bash) ">" send to as a whole completed file, overwriting target if exists "echo test > file.txt" is equivalent to "echo test 1> file.txt" echo test 2> file.txt #redirect stderr to file.txt "/dev/null" is the null device it takes any input you want and throws it away. It can be used to suppress any output "2>/dev/null" Redirect STDERR to /dev/null (nothing shows up on console) The general form of this one is "M>/dev/null", where "M" is a file descriptor number. This will redirect the file descriptor, "M", to "/dev/null". ">&" is the syntax to redirect a stream to another file descriptor "echo test 1>&2" equivalent to "echo test >&2" "2>&1" The general form of this one is "M>&N", where "M" & "N" are file descriptor numbers. It combines the output of file descriptors "M" and "N" into a single stream. "&" indicates that what follows and precedes is a file descriptor, and not a filename. "2>&-" closing a file descriptor used with redirection The general form of this one is "M>&-", where "M" is a file descriptor number. This will close output for whichever file descriptor is referenced, i.e. "M" "|&" Redirect STDERR and STDOUT to STDIN This is just an abbreviation for "2>&1 |" "&>/dev/null" Redirect both STDERR & STDOUT to /dev/null (nothing shows up on console) This is just an abbreviation for >/dev/null 2>&1. It redirects file descriptor 2 (STDERR) and descriptor 1 (STDOUT) to /dev/null ">/dev/null" Redirect STDOUT to /dev/null (only STDERR shows on console) This is just an abbreviation for 1>/dev/null. It redirects file descriptor 1 (STDOUT) to /dev/null. "command > /dev/null 2>&1 &" #Run command in the background, discard stdout and stderr "command >> /path/to/log 2>&1 &" #Run command and append stdout and stderr to a log file ./command >/dev/null 2>&1 #Hide standard and error outputs Hide standard output ./command >/dev/null sends 2 (stderr) into 1 (stdout), and sends stdout to file.log command > file.log 2>&1 Hide standard and error outputs and release terminal (run the command in background) ./command >/dev/null 2>&1 & prevent standard output and error output, redirecting them both to /dev/null script > /dev/null 2>&1 ----------------------------------------------------------------------------------------------------- #Identify processes using files, directories, or sockets.Who is Using a File or Directory $ fuser . $ fuser -v ./ Check Processes Using TCP/UDP Sockets fuser -v -n tcp 5000 the processes that are using my 'home' directory $ fuser ~ $ fuser ~ -v check for the root directory $ fuser / $ fuser / -v $ fuser -v /home/ismail $ fuser -v -m /home/ismail/.bashrc $ fuser -v -n tcp 8080 $ fuser -v -n udp 53 kill this TCP listener, you can use option -k $ fuser -i -k 8080/tcp shows all processes at the (local) TELNET port $ fuser telnet/tcp list signals $ fuser -l STOP a process $ fuser -i -k STOP [FILE/DIRECTORY] kills all processes accessing the file system /home $ fuser -km /home $ who #logged-in users,List Connected Users #all the currently logged in users, the time of login and their host machine's IP address. #review the contents of the /var/log/utmp file $ who -H $ who -r run-level 5 2018-07-14 17:16 $ runlevel ->‘N’ indicates that the runlevel has not been changed since the system was booted. And, "number" is the current runlevel The last run level, and the current run level. $ runlevel md5sum ubuntu-6.10-desktop-i386.iso sha256sum ubuntu-9.10-dvd-i386.iso echo "$(cat kubectl.sha256) kubectl" | sha256sum --check # Validate the kubectl binary against the checksum file sha1sum filename #view the SHA-1 of a file sha1sum -c DNi70074.bio.sha1 #read SHA1 sums from the FILEs and check sha1sum --check --ignore-missing DNi70074.bio.sha1 # Do not fail or report status for missing files $ wget https://github.com/coreruleset/coreruleset/archive/refs/tags/v3.3.2.zip $ echo -n '88f336ba32a89922cade11a4b8e986f2e46a97cf v3.3.2.zip' | sha1sum -c - v3.3.2.zip: OK $ sha1sum v3.3.2.zip 88f336ba32a89922cade11a4b8e986f2e46a97cf v3.3.2.zip #watch is used to run any designated command at regular intervals. watch -n 5 "ls -l | wc l" -------------------------------------------------------------------------------------------------------------------- # detect driver hardware problems dmesg | more The output of dmesg is maintained in the log file /var/log/dmesg cat /var/log/dmesg | less data from /dev/kmsg use syslog # dmesg -S # limit the output to only error and warnings dmesg --level=err,warn # dmesg produce timestamps dmesg --level=err -T dmesg -T | grep -i eth0 dmesg --level=err,warn -T | grep -i eth0 # limit dmesg's output only to userspace messages dmesg -u # timestmaps along with decode facility and levels in dmesg command output dmesg -Tx Supported log levels (priorities): emerg - system is unusable alert - action must be taken immediately crit - critical conditions err - error conditions warn - warning conditions notice - normal but significant condition info - informational debug - debug-level messages dmesg -TL -f kern dmesg -TL -f daemon Supported log facilities: kern - kernel messages user - random user-level messages mail - mail system daemon - system daemons auth - security/authorization messages syslog - messages generated internally by syslogd lpr - line printer subsystem news - network news subsystem # verify vt-d is ON "dmesg | grep Virtualization" # dmesg | grep -i memory # dmesg | grep -i dma # dmesg | grep -i usb # dmesg | grep -E "memory|dma|usb|tty" # dmesg | grep -E "sda|dma" Clear dmesg logs # dmesg -C # dmesg -c Display colored messages # dmesg -L Monitor real time dmesg logs # dmesg --follow # dmesg -Tx --follow # watch "dmesg | tail 7-20" Display raw message buffer # dmesg -r #virtual machine check $ dmesg |grep -i hypervisor -------------------------------------------------------------------------------------------------------------------- $ dmidecode -s system-manufacturer -------------------------------------------------------------------------------------------------------------------- #32x 64x query uname –m arch #linux version lsb_release -a cat /etc/issue cat /etc/os-release cat /etc/lsb-release cat /etc/*-release cat /proc/version hostnamectl set-hostname server1 hostnamectl set-hostname --pretty "Web dev test environment" hostnamectl set-hostname --static webdev-test-env $ hostname --ip-address $ hostname --all-ip-addresses dnsdomainname -f #create file w multiple lines echo \ "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu \ $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null sudo ls -hal /tmp/ | sudo tee /tmp/test.out > /dev/null #The redirect to /dev/null is needed to stop tee from outputting to the screen sudo ls -hal /tmp/ | sudo bash -c "cat >> /tmp/test.out" #To append instead of overwriting the output file METHOD1 sudo ls -hal /tmp/ | sudo tee /tmp/test.out #To append instead of overwriting the output file METHOD2 sudo ls -hal /tmp/ | sudo tee --append /tmp/test.out #To append instead of overwriting the output file METHOD2 # append text with non-root user echo "deb http://research.cs.wisc.edu/htcondor/ubuntu/8.8/bionic bionic contrib" |sudo tee -a /etc/apt/sources.list #do not see the contents of the website (may receive an access denied or ‘bad bot’ message) curl -A 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.140 Safari/537.36' URL curl --compressed URL curl URL | iconv -f windows-1251 -t UTF-8 #convert from the windows-1251 encoding to the UTF-8 encoding curl -u username:password URL #websites require a username and password to view their content,specify other authentication methods using --ntlm | --digest #showing only headers, HTML code not displayed curl -s -I -A 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.140 Safari/537.36' https://www.acrylicwifi.com/AcrylicWifi/UpdateCheckerFree.php?download | grep -i '^location' curl -s -v http://www.paterva.com/web7/downloadPaths41.php -d 'fileType=exe&os=Windows' 2>&1 | grep -i 'Location:' #save cookies,data transfer using the POST method, the --data option is used curl --cookie-jar cookies.txt http://forum.ru-board.com/misc.cgi --data 'action=dologin&inmembername=f123gh4t6&inpassword=111222333&ref=http%3A%2F%2Fforum.ru-board.com%2Fmisc.cgi%3Faction%3Dlogout' #get information from a page that only registered users have access to, specify the path to the file with previously saved cookies curl -b cookies.txt 'http://forum.ru-board.com/topic.cgi?forum=35&topic=80699&start=3040' | iconv -f windows-1251 -t UTF-8 #extracting entire archive curl -L -o ds.tar.gz https://downloads.dockerslim.com/releases/1.37.3/dist_linux.tar.gz tar -xvf ds.tar.gz -C /usr/local/bin tar xvzf tkn_0.10.0_Darwin_x86_64.tar.gz -C /usr/local/bin tkn tar xjvf backup.tbz tar -zxvf backup.tar.gz tar -xf file_name.tar.gz --directory /target/directory #ssh and tar to make secure backups. Make a backup via encrypted file transfer tar --create --directory /home/joe/tmp/ --file - *| \ ssh raspberrypi "tar --directory /home/joe \ --verbose --list --file -" wget --no-check-certificate https://www.cacti.net/downloads/cacti-latest.tar.gz tar -zxvf cacti-latest.tar.gz mv cacti-1* /opt/cacti (OR tar -xf cacti-latest.tar.gz --directory /opt/cacti) tar -zxvf /tmp/onos-1.12.0.tar.gz --strip-components 1 --directory /opt --one-top-level=onos tar xvf mysql-5.7.23-linux-glibc2.12-x86_64.tar.gz --one-top-level=mysql57 --strip-components 1 tar zxvf ugly_name.tgz --one-top-level=pretty_name #extract .xz file unxz tor-browser-linux32-5.5.4_en-US.tar.xz tar xvf tor-browser-linux32-5.5.4_en-US.tar #extract .bz2 file bzip2 -dk FileZilla_3.29.0_x86_64-linux-gnu.tar.bz2 tar xvf FileZilla_3.29.0_x86_64-linux-gnu.tar #extract .zip file unzip terraform_0.11.7_linux_amd64.zip -d terraform #extract .rar file unrar e extract.rar r # create user home directory backup tar cvf filename.tar /home/vagrant/ # show which files were changed tar dvf filename.tar # update the changed files tar uvf filename.tar # make smaller backup gzip filename.tar #format a USB storage device with FAT32 file system mkfs –t vfat # mount -o loop,offset=$((10860003 * 512)) disk.img /mnt #find out the USB device mount point fdisk -l #unmount the drive,you can’t format a mounted drive. sudo umount /dev/sdb1 sudo mkfs.vfat /dev/sdb1 sudo mkfs.ntfs /dev/sdb1 mkfs.ext4 mkfs.ntfs #Set label name to USB drives sudo mkfs.vfat /dev/sdb1 -n sk ========================================================================================================== #Create a New Sudo User(CentOS) adduser username passwd username usermod -aG wheel username #add user to the wheel group.By default, on CentOS, members of the wheel group have sudo privileges su - username # switch to the new user account #verify if user is sudoer sudo -l -U userjohndoe #list user's privileges or check a specific command sudo --validate / sudo -v #update the user's cached credentials, authenticating the user if necessary sudo --list #print the list of allowed and forbidden commands for the user who is executing the sudo command groups #verify if user is sudoer, member of wheel group sudo whoami # returns root ---------------------------------------------------------------------------------------------------- #Create a New Sudo User (ubuntu) sudo adduser barak #create new user sudo adduser barak sudo #Add the user to sudo group usermod -aG sudo barak #Add the user to sudo group id barak #verify sudo group groups newuser #verify sudo group su - barak #Verify Sudo Access $ ls /root ls: cannot open directory '/root': Permission denied sudo ls /root ---------------------------------------------------------------------------------------------------- echo $HOME $USER sudo bash -c 'echo $HOME $USER' sudo -H bash -c 'echo $HOME $USER' ---------------------------------------------------------------------------------------------------- cmd: sudo sh -c "go build ." problem: bash: line 1: go: command not found fix: # indicates that the go command is not in the system's PATH when running the command with sudo. #This is because sudo does not preserve the user's environment variables, including the PATH variable. sudo env "PATH=$PATH" sh -c "go build ." ---------------------------------------------------------------------------------------------------- When you use the "su otheruser" command without the hyphen (-), you switch to the specified user's account, but you inherit the current environment variables and working directory of the original user. This can lead to potential issues if you need the new user's environment to be set up completely. Example Use Cases: If you only need to quickly execute a single command as "otheruser" but do not require the environment to be exactly the same, you might use "su otheruser" to avoid the overhead of starting a new shell session. ┌──(vagrant㉿vg-kali-02)-[~] └─$ su barak Password: ┌──(barak㉿vg-kali-02)-[/home/vagrant] └─$ echo $HOME /home/barak ┌──(barak㉿vg-kali-02)-[/home/vagrant] └─$ pwd /home/vagrant ┌──(barak㉿vg-kali-02)-[/home/vagrant] └─$ exit exit ┌──(vagrant㉿vg-kali-02)-[~] └─$ ---------------------------------------------------------------------------------------------------- When you use the "su - otheruser" command with the hyphen (-), you not only switch to the specified user's account but also start a new shell session with the target user's environment variables and home directory. This ensures a clean environment as if you had logged in directly as that user. In general, you should use su - otheruser if you need to change the shell environment. This is especially important if you are going to be running commands that require specific permissions or settings that are not available in the default shell environmen Example Use Cases: Suppose you are the "originaluser" and you want to run a script that requires specific environment variables and settings of "otheruser." In this case, you would use "su - otheruser" to ensure the script runs with the correct environment. ┌──(vagrant㉿vg-kali-02)-[~] └─$ su - barak Password: ┌──(barak㉿vg-kali-02)-[~] └─$ echo $HOME /home/barak ┌──(barak㉿vg-kali-02)-[~] └─$ pwd /home/barak ┌──(barak㉿vg-kali-02)-[~] └─$ exit logout ┌──(vagrant㉿vg-kali-02)-[~] └─$ ---------------------------------------------------------------------------------------------------- #-H flag makes sudo assume root's home directory as HOME instead of the current user's home directory sudo -H #sudo user echo "stack ALL=(ALL) NOPASSWD: ALL" |sudo tee -a /etc/sudoers #allow a user aaron to run all commands using sudo without a password, open the sudoers file $ sudo visudo aaron ALL=(ALL) NOPASSWD: ALL %sys ALL=(ALL) NOPASSWD: ALL #all member of the sys group will run all commands using sudo without a password alf ALL=(ALL) NOPASSWD: ALL #permit a user to run a given command (/bin/kill) using sudo without a password %sys ALL=(ALL) NOPASSWD: /bin/kill, /bin/rm #the sys group to run the commands: /bin/kill, /bin/rm using sudo without a password #su vs sudo #"sudo" asks for your password,"su" asks for the password for the user whom you are switching to #sudo lets you issue commands as another user without changing your identity,entry in /etc/sudoers to execute these restricted permissions #without entering the root password #su keeps the environment of the old/original user even after the switch to root #creates a new environment (as dictated by the ~/.bashrc of the root user), #similar to the case when you explicitly log in as root user from the log-in screen. "su -" "su -l" #pass more arguments "su -c" #su [target-user] -c [command-to-run] a command that you want to run after switching to the target user. su -c '/home/annie/annie-script.sh' annie #While logged in as user dave, run the annie-script.sh as user annie su -c 'echo I am $(whoami)' #Without specifying a target user,switch into root #The password prompt is not preferable, during scripting #disable the password prompt when user dave is executing scripts as user annie.dave uses su without having to input annie‘s password. #/etc/pam.d/su,add the following lines right after the line "auth sufficient pam_rootok.so" auth [success=ignore default=1] pam_succeed_if.so user = annie #rule checks if the target user is annie auth sufficient pam_succeed_if.so use_uid user = dave #rule to check if the current user is dave su -c /home/annie/annie-script.sh annie #run by dave auth sufficient pam_rootok.so auth [success=ignore default=1] pam_succeed_if.so user = otheruser auth sufficient pam_succeed_if.so use_uid user ingroup somegroup #/etc/sudoers echo 'dave ALL=(annie) /home/annie/annie-script.sh' | EDITOR='tee -a' visudo #The rule grants dave the permission to execute the script annie-script.sh as user annie on any hosts sudo -u annie /home/annie/annie-script.sh #while logged in as dave sudo -u root /home/annie/annie-script.sh #Sorry, user dave is not allowed to execute '/home/annie/annie-script.sh' as root "sudo -s" or "sudo -i" #mimic "su" or "su -l" "sudo -s or sudo -i" #temporarily become a user with root privileges #/etc/sudoers echo 'dave ALL=(ALL) /home/annie/annie-script.sh' | EDITOR='tee -a' #The rule grants dave to execute the script annie-script.sh as any users sudo -u root /home/annie/annie-script.sh #while logged in as dave #The password prompt is not preferable, during scripting #/etc/sudoers dave ALL=(ALL) NOPASSWD: /home/annie/annie-script.sh' | EDITOR='tee -a' # switching to root using sudo -i (or sudo su) cancels auditing/logging # when a sudo command is executed, the original username and the command are logged "sudo su" "sudo -i" su is equivalent to sudo -i gives you the root environment, i.e. your ~/.bashrc is ignored. simulates a login into the root account Your working directory will be /root will read root's .profile "sudo -s" gives you the user's environment, so your ~/.bashrc is respected. launches a shell as root doesn't change your working directory "sudo bash" #runs bash as a super user sudo -E #The -E (preserve environment) option indicates to the security policy that the user wishes to preserve their existing environment variables. ========================================================================================================== mount -l lshw -short sudo lshw -class disk sudo lshw -short -class disk lshw -class processor file -Ls dmesg denyhosts vmstat w uptime ps free iostat pmap paste uname sudo mkdir chown ptree pkill killall kill -KILL PID kill –TERM [PID] #means terminate.the default signal sent by kill,equivalent to kill kill –HUP [PID] #reset or restart the process all signals $ kill -l stop and restart process $ kill -1 13980 1 SIGHUP 9 SIGKILL stop process without letting gracefully 15 SIGTERM stop process ------------------------------------------------------------------------------------------------------------------ $ touch mylog $ ls -lai mylog 3145761 -rw-rw-r-- 1 vagrant vagrant 0 Mar 27 21:01 mylog update the access time of existing file $ touch -c mylog Change file access time - 'a' of existing file $ touch -a mylog Change the modified time '-m' of existing file $ touch -m mylog $ touch -am mylog Set a specific access/modify time instead of current time, specify the datetime in format [[CC]YY]MMDDhhmm[.ss] $ touch -c -t 201603051015 mylog $ touch -c -d '5 Jan 2009' mylog $ touch -c -d '20:30' myfile Use the timestamp of another file as reference $ touch myfile -r mylog use the timestamps of 'apl' for 'apl.c' $ touch apl.c -r apl $ ls -lai mylog 3145761 -rw-rw-r-- 1 vagrant vagrant 0 Mar 27 21:06 mylog $ stat mylog File: 'mylog' Size: 0 Blocks: 0 IO Block: 4096 regular empty file Device: fc00h/64512d Inode: 3145761 Links: 1 Access: (0664/-rw-rw-r--) Uid: ( 1000/ vagrant) Gid: ( 1000/ vagrant) Access: 2019-03-27 21:07:54.953000000 +0000 Modify: 2019-03-27 21:07:54.953000000 +0000 Change: 2019-03-27 21:07:54.953000000 +0000 Birth: - force touch to not create any new file $ touch -c newfile $ ls -l newfile ls: cannot access 'newfile': No such file or directory $ stat newfile stat: cannot stat 'newfile': No such file or directory ------------------------------------------------------------------------------------------------------------------ # create symbolic link ln -s test1.txt symbolic_test1.txt stat symbolic_test1.txt touch -c -d '5 Jun 2001' -h symbolic_test1.txt stat symbolic_test1.txt # detect symbolic link touch a.xt ln -s a.txt b.txt stat b.txt file b.txt ln -s source.file softlink.file ls -lia #does not share the same inode number and permissions of original file rm source.file stat softlink.file #No such file or directory # create hard link ln source.file hardlink.file ls -lia #shares the same inodes number and permissions of original file rm source.file stat hardlink.file echo "hola el mundo" >>source.file #change the content of either of the files, the change will be reflected cat hardlink.file cat source.file lscpiu stat file.txt #standard output activities for each available processor mpstat mpstat -P ALL 2 2 ---------------------------------------------------------------------------------------------------- df –h # view the amount of free disk space df -HT #Display File System Type df -hT /home df -t ext3 #display a certain file system type use the ‘-t‘ option df -x ext3 #Exclude Certain File System Type df -k #usage in 1024-byte blocks, use the option ‘-k‘ df -m # MB (MegaByte) df -i # view number of inodes in the system df -kl # get a detail description on disk space usage --------------------------------------------------------------------------------------------------- du -sh /* # list directory sizes under root / disk du -sh /* | sort -h du -m /some/path | sort -nr | head -n 20 #sorted list containing the 20 biggest dirs for each in $(ls) ; do du -hs "$each" ; done du --threshold=1M -h | sort -h #includes hidden dot folders (folders which start with .). du -h | sort -h 3 du -bch #-b gives you the file size instead of disk usage, and -c gives a total at the end du -ch | tail -1 du -sh /some/dir #the summary of a grand total disk usage size of an directory use the option “-s” du -sh /var/* |grep G du -ah /home/tecmint # displays the disk usage of all the files and directories du -kh /home/tecmint #the disk usage of a directory tree with its subtress in Kilobyte blocks. Use the “-k” (displays size in 1024 bytes units). du -kh /home/tecmint #Megabytes (MB) du -ch /home/tecmint #The “-c” flag provides a grand total usage disk space at the last line du -ah --exclude="*.txt" /home/tecmint du -ha --time /home/tecmint #the disk usage based on modification of time, use the flag “–time” #The problem with du is that it adds up the size of the directory nodes as well,not to sum up only the file sizes. # total size of files in a directory du -h -c directory #listing path du -h -c directory|tail -1 #only total size $ du -sh /var/log/apt #only total size and listing #du prints actual disk usage rounded up to a multiple of (usually) 4 KB instead of logical file size $ for i in {0..9}; do echo -n $i > $i.txt; done #create files $ ls *.txt 0.txt 1.txt 2.txt 3.txt 4.txt 5.txt 6.txt 7.txt 8.txt 9.txt $ du -ch *.txt | tail -1 40K total $ ls -FaGl *.txt | printf "%'d\n" $(awk '{SUM+=$4}END{print SUM}') 10 du /var/* -shc --exclude=lib #--exclude to exclude any directory du /var/ -h --exclude=lib --max-depth=1 #first-level sub-directories in the /var/ directory. $ du -ch /var/log/apt | tail -1 | cut -f 1 $ du -ac --bytes /var/log/apt $ du -ac --bytes /var/log/apt | grep "log$" | awk '{ print; total += $1 }; END { print "total lobsters: ", total, " Bytes" }' $ du -ac --bytes /var/log/apt | grep "log$" | awk '{ print; total += $1 }; END { print "total lobsters: ", total/1024, " KB" }' $ du -ac --bytes /var/log/apt | grep "log$" | awk '{ print; total += $1 }; END { print "total lobsters: " total/1024/1024 " MB" }' $ du /var/log/apt/*.log | awk '{ print; total += $1 }; END { print "total size: ",total }' ---------------------------------------------------------------------------------------------------- $ dir /var/log/apt #list directory contents $ dir /var/log/apt | tee >( awk '{ total += $4 }; END { print total }' ) #list directory contents and total size $ dir /var/log/apt | awk '{ print; total += $4 }; END { print "total size: ",total }' #total size ---------------------------------------------------------------------------------------------------- #List all running processes containing the string stuff cat /proc/cpuinfo grep "physical id" /proc/cpuinfo | wc -l cat /proc/meminfo grep MemTotal /proc/meminfo | awk '{FS=":"}{print $2 }' | awk '{print $1/1024/1024}' cat /proc/zoneinfo cat /proc/mounts cat /etc/issue # missing files $ sudo ls -lai /lost+found/ # errors about a corrupt superblock on the drive $ e2fsck -b 8193 $list files sorted by size ls –lSr ls -il #get pid of my_app my_app & $! pidof lighttpd -> Find PID of A Program/Command pidof -s php5-cgi pidof -x fetch-data -> Get pids of scripts pidof -o 4242 lighttpd -> ignore or omit processes,useful to ignore calling shell or shell script or specific pid. ------------------------------------------------------------------------------------------ # history,The ! prefix is used to access previous commands. !$ - last argument from previous command(last command) !:1 # last command's 1st argument !:2 # last command's 2nd argument !:1-2 # last command's 1st and 2nd argument !^ - first argument (after the program/built-in/script) from previous command !* - all arguments from previous command !! - previous command (often pronounced "bang bang") !n - command number n from history !pattern - most recent command matching pattern !!:s/find/replace - last command, substitute find with replace !3:2 #take the second argument from the third command in the history !-5:3 #take the third argument from the fifth last command in the history,Using a minus sign to traverse from the last command of the history $ echo 'one' 'two' one two $ !$ 'two' $ !^ 'one' $ !* 'one' 'two' $ !! echo 'one' 'two' one two $ ls /tmp && cd !* ------------------------------------------------------------------------------------------ $ curl -kL http://localhost/banana #curl -IL http://localhost HTTP/1.1 200 OK Server: nginx/1.10.2 curl -Is http://www.google.com | head -n 1 #check whether a web site is up, and what status message the web server is showing curl -sSf http://example.org > /dev/null curl -XGET 'localhost:9200/?pretty' curl -X PUT "http://127.0.0.1:9200/mytest_index" #sending data with POST and PUT requests curl -d "param1=value1¶m2=value2" -X POST http://localhost:3000/data curl -d "param1=value1¶m2=value2" -H "Content-Type: application/x-www-form-urlencoded" -X POST http://localhost:3000/data curl -d "@data.txt" -X POST http://localhost:3000/data curl -d '{"key1":"value1", "key2":"value2"}' -H "Content-Type: application/json" -X POST http://localhost:3000/data curl -d "@data.json" -X POST http://localhost:3000/data # check if apache is running curl -sf http://webserver/check_url # process was holding a particular port open ss -tp state listening sport = :80 | grep httpd # check a particular process id lsof -p 1036 -P | grep 'TCP \*:80' $ echo "The process id is" $$ $ echo "The process id is" $$$$ # check what process is listening $ sudo fuser -n tcp 22 22/tcp: 1088 14324 14354 echo $SHELL -> determine current shell type cat /proc/cpuinfo | grep 'vmx\|svm' -> VT-x/AMD-v virtualization is enabled in BIOS # troubleshooting nginx journalctl -u nginx.service tail -n 50 /var/log/nginx/error.log tail -n 50 //var/log/nginx/access.log nginx -t sudo ss -tulpn # Verify that port 80 or 443 curl -I http://10.21.136.13 curl http://10.21.136.13 dig +short localhost @8.8.8.8 ------------------------------------------------------------------------------------------------ #/etc/systemd/journald.conf #SystemMaxUse=100M cat /etc/systemd/journald.conf | grep SystemMaxUse journalctl --vacuum-size=100M sudo usermod -a -G systemd-journal $USER # add the current user to the systemd-journal group # process the data further with text processing tools like grep, awk, or sed, or redirect the output to a file journalctl --no-pager #print its output directly to the standard output instead of using a pager by including the --no-pager flag journalctl -o json -n 10 --no-pager #change the format to format like JSON journalctl -o json-pretty -n 10 --no-pager cp /etc/systemd/journald.conf{,.orig} cp a.txt{,.$(date +%Y%m%d-%H%M)} #clone with date info sudo mkdir -p /var/log/journal ls -l /var/log/journal/3a0d751560f045428773cbf4c1769a5c/ sudo cp /etc/systemd/journald.conf{,.orig} sed -i 's/#Storage.*/Storage=persistent/' /etc/systemd/journald.conf # set "Storage" type to "persistent sudo vi /etc/systemd/journald.conf Storage=persistent systemctl restart systemd-journald.service journalctl --flush # move the journal log files from /run/log/journal to /var/log/journal #The options prefixed with "Runtime" apply to the journal files when stored on a volatile in-memory file system, #more specifically /run/log/journal journalctl --vacuum-files=2 # have 10 archived journal files and want to reduce these down to 2 journalctl --verify journalctl | head -1 #What time range do I have logs for? journalctl -F _SYSTEMD_UNIT #What systemd services do I have logs for? journalctl -F _COMM journalctl -F _EXE journalctl -F _CMDLINE #What users do the services that logged something run as (swap _UID/-u with _GID/-g for groups)? journalctl -F _UID | xargs -n1 id -nu journalctl -F _UID | xargs -n1 id -ng #provide test input for journalctl logger -p err 'something erronous happened' systemd-cat -p info echo 'something informational happened' #What selector fields are there? Show up to 8 values each for f in $(sudo journalctl --fields); do echo ===========$f; sudo journalctl -F $f; done | grep -A8 ======== #remove all entries journalctl --rotate journalctl --vacuum-time=1s journalctl -m --vacuum-time=1s #-m flag, it merges all journals and then clean them up #remove all entries find /var/log/journal -name "*.journal" | xargs sudo rm systemctl restart systemd-journald #remove all entries rm -rf /run/log/journal/* journalctl --rotate --vacuum-size=500M #rotate journal files and remove archived journal files until the disk space they use is under 500M #Rotating is a way of marking the current active log files as an archive and create a fresh logfile from this moment # The flush switch asks the journal daemon to flush any log data stored #in /run/log/journal/ into /var/log/journal/, if persistent storage is enabled. #Manual delete,removes all archived journal log files until the last second,clears everything journalctl --flush --rotate #applies to only archived log files only, not on active journal files journalctl --vacuum-time=1s #Manual delete,clears all archived journal log files and retains the last 400MB files journalctl --flush --rotate journalctl --vacuum-size=400M #Manual delete,only the last 2 journal files are kept and everything else is removed journalctl --flush --rotate journalctl --vacuum-files=2 journalctl -b ->all of the journal entries that have been collected since the most recent reboot journalctl --list-boots #list of boot numbers, their IDs, and the timestamps of the first and last message pertaining to the boot journalctl --boot=ID _SYSTEMD_UNIT=foo journalctl -b -1 -> see the journal from the previous boot,use boot number to pick specific boot journalctl -k -b -1 -> Shows kernel logs for the current boot. $ journalctl --list-boots #list boot id -1 340f8a96d40749f8b2530cc76810d62d Tue 2022-01-18 14:33:46 +03—Tue 2022-01-18 15:14:42 +03 0 75c35ddeb4274787ad78d1092bf9743a Tue 2022-01-18 23:08:10 +03—Wed 2022-01-19 09:25:35 +03 $ journalctl -b 75c35ddeb4274787ad78d1092bf9743a #use boot id journalctl --since "2015-01-10 17:15:00" journalctl -S "2020-91-12 07:00:00" journalctl -S -1d #The “d” stands for “day”, and the “-1” means one day in the past journalctl -S -1h journalctl --since "2015-06-26 23:15:00" --until "2015-06-26 23:20:00" journalctl -S "2020-91-12 07:00:00" -U "2020-91-12 07:15:00" journalctl --since yesterday journalctl -S yesterday journalctl --since yesterday --until now journalctl --since today journalctl -S -2d -U today #everything from two days ago up until the start of today journalctl --since 09:00 --until "1 hour ago" journalctl --since '1h ago' --until '10 min ago' #syslog log levels i.e. "emerg" (0), "alert" (1), "crit" (2), "err" (3), "warning" (4), "notice" (5), "info" (6), "debug" (7) journalctl -p 0 journalctl -p 0..2 # logs for a range between emerg(0) and critical(2) journalctl -f -p warning # show me warnings journalctl -p err # show all errors journalctl -xp info journalctl -xu sshd journalctl -fxu httpd.service journalctl -fxu sshd.service -p debug journalctl -fx journalctl -xn journalctl /dev/sda -> displays logs related to the /dev/sda file system. journalctl /sbin/sshd #logs from the sshd binary journalctl -n20 _EXE=/usr/sbin/sshd journalctl /usr/bin/bash journalctl -u nginx.service -> see all of the logs from an Nginx unit on our system journalctl -u nginx.service --since today journalctl -b -u docker -o json journalctl -u docker.service --since "2016-10-13 22:00" journalctl _SYSTEMD_UNIT=sshd.service journalctl -u sshd.service journalctl -u sshd.service -x #logs with more details journalctl _PID=8088 journalctl -b _SYSTEMD_UNIT=foo _PID=number #logs for systemd-units that match foo and the PID number #all messages from the foo service process with the PID plus all messages from the foo1 service journalctl -b _SYSTEMD_UNIT=foo _PID=number + _SYSTEMD_UNIT=foo1 journalctl -b _SYSTEMD_UNIT=foo _SYSTEMD_UNIT=foo1 #shows logs matching a systemd-unit foo or a systemd-unit foo1 #Filter logs based on user id -u www-data 33 journalctl _UID=33 --since today journalctl -k ->Kernel messages, those usually found in dmesg output journalctl _TRANSPORT=kernel journalctl -n 20 ->see with a number after the -n journalctl -n 10 -o short-full #Changing the Display Format journalctl -n 10 -o verbose journalctl -n 10 -o json journalctl -n 10 -o json-pretty journalctl -n 10 -o cat #see the log entry messages, without time stamps or other metadata journalctl --disk-usage #using persistent storage then the below output shows the amount of disk used #removes archived journal files until the disk space they use falls below the specified size #(specified with the usual "K", "M", "G", "T" suffixes), journalctl --vacuum-size=1G journalctl --vacuum-time=1weeks #clear all messages older than one week journalctl --vacuum-time=2d #Retain only the past two days journalctl -f -> continuously prints log messages, similar to tail -f journalctl -u mysql.service -f journalctl -f -e -p err docker --since today # -e implies -n1000 #who ran sudo in the past week, what commandline, what PWD and user? journalctl --since '1 week ago' _COMM=sudo -o json \ | jq -r '(.__REALTIME_TIMESTAMP|tonumber|(./1e6)|todate) + "\t" + ._CMDLINE + "\t" + .MESSAGE' \ | column -ts $'\t' #How many ssh auth errors today? journalctl -o cat -p err -u ssh --since today | wc -l #filter specific error journalctl -o cat -p err | grep "tx hang" #executables have been logging errors at a loglevel lower than error in the past month? journalctl --since -1month -p 7..4 -o json | jq -r 'select (.MESSAGE | contains("error")) | ._EXE' | sort -u #show error logs for a particular version of a service journactl -p err /opt/fooservice/9e76/bin/fooservice #Filter by start and end dates and particular PIDs journalctl _SYSTEMD_UNIT=docker --since '2018–11–01 14:00' --until '2018–11–13 14:00' _PID=123 _PID=456 ------------------------------------------------------------------------------------------------ Job for autofs.service failed because a configured resource limit was exceeded. See "systemctl status autofs.service" and "journalctl -xe" for details. systemctl start autofs systemctl is-active autofs systemctl is-active autofs >/dev/null 2>&1 && echo YES || echo NO ps -aux | grep -i autofs | grep -v grep #grep command was shown in the output, remove this distraction is to add another pipe to grep -v grep kill -9 `ps -ef | grep '[k]eyword' | awk '{print $2}'` # get the pid from ps command ps -aux | grep dockerd | grep -v grep | awk '{print $2}' # get the pid from ps command ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ $ ps aux | grep root #list services running as root ps aux | grep 3813 ps -eo pid,user,group,args,etime,lstart | grep '[3]813' ps aux | grep '[1]616' ps -eo pid,user,group,args,etime,lstart | grep '[1]616' ps aux | grep stuff The init process, with process ID 1, which does nothing but wait around for its child processes to die. Usually started for /etc/inittab $ ps -ef| grep init # see the name of the process $ sudo ps 1088 14324 14354 # CPU time, page faults of child processes ps -Sla $ ps -lu vagrant memory information long format $ ps -lma signal format $ ps -sx controlling terminal $ ps --tty 1 -s #print a process tree ps -ejH ps axjf list of command line arguments pstree -a show PIDS for each process name pstree -p sort processes with the same ancestor by PID instead of by name,numeric sort pstree -n pstree -np find out the owner of a process in parenthesis pstree -u pstree -u vagrant pstree -unp vagrant highlight the current process and its ancestors pstree -h highlight the specified process pstree -H 60093 find ID of a process owned by a specific user $ pgrep -u vagrant sshd $ pgrep -u vagrant -d: list process names $ pgrep -u vagrant -l $ pgrep -u vagrant -a count of matching processes $ pgrep -c -u vagrant top -> Checking the Priority of Running Processes ps -o pid,comm,nice -p 594 -> Checking the Priority of Running Processes ps -o pid,comm,nice,pri -p $(pidof snmpd) ps -fl -C "perl test.pl" -> The “NI” column in the ps command output indicates the current nice value (i.e priority) of a process. ps -p 2053 -o comm= #NI – is the nice value, which is a user-space concept #PRI – is the process’s actual priority, as seen by the Linux kernel ps -o ni $(pidof snmpd) #Total number of priorities = 140 #Real time priority range(PR or PRI): 0 to 99 #User space priority range: 100 to 139 ps -o ni,pri $(pidof snmpd) #PR = 20 + NI #PR = 20 + (-20 to + 19) #PR = (20 + -20) to (20 + 19) #PR = 0 to 39 (100 to 139 user space priority range) ps -o pid,comm,nice,pri -p $(pidof snmpd) cat /proc/$(pidof snmpd)/stat | awk '{print "priority " $18 " nice " $19}' ps u $(pgrep snmpd) #ps with headers #The NI column shows the scheduling priority or niceness of each process #ranges from -20 to 19, with -20 being the most favorable or highest priority for scheduling #19 being the least favorable or lowest priority ps -e -o uid,pid,ppid,pri,ni,cmd | { head -5 ; grep snmpd; } #ps with headers # priority levels between -20 and 19 nice -10 perl test.pl -> test.pl is launched with a nice value of 10 when the process is started nice --10 perl test.pl -> Launch a Program with High Priority nice #Checking default niceness,the default is 0 # 0 for none, 1 for real-time, 2 for best-exertion, 3 for inactive ionice -c 3 -p 1 #PID as 1 to be an idle I/O process ionice -c 2 bash #run ‘bash’ as a best-effort program ionice -p 3467 #examine the class and priority used by PID 3467 ionice -c 1 -n 3 -p 3467 #set the I/O scheduling class to Idle,takes longer,no longer performance degradation # for pid in $(pidof rsync); do ionice -c 3 -p $pid; done ----------------------------------------------------------------------------------------------------- renice -n -19 -p 3534 -> Change the Priority of a Running Process #adding more virtual runtime to the process #The OS thinks that the process has taken more virtual runtime time than other processes in the run queue. #in the next cycle, the CPU gives less time to the process #The process finishes late as it’s getting less time “on CPU” renice +10 PID #The OS thinks that the process hasn’t got enough “on CPU” time than other processes in the run queue #in the next cycle, the CPU gives more “on CPU” time to that process as compared to other processes in the run queue. renice -10 PID /etc/security/limits.conf -> set the default nice value of a particular user or group $ pidof rsync $ renice +10 2395 2395 (process ID) old priority 0, new priority 10 ----------------------------------------------------------------------------------------------------- #Process affinity is the scheduler property that helps to bind or unbind the process so that the process will run only with the allotted CPU #Processor affinity, or CPU pinning or “cache affinity”, enables the binding and unbinding of a process or a thread to a central processing unit (CPU) or a range of CPUs #According to the taskset command man pages, value f means "any CPU." $ pidof rsync $ taskset -p 2395 #use the PID to get CPU affinity, returns the current CPU affinity in a hexadecimal bit mask format pid 2395's current affinity mask: f $ taskset -cp 2395 #get the CPU range of a process pid 2395's current affinity list: 0-3 $ taskset -c 0 vlc #start the VLC program on CPU core ID 0 taskset 0xa gedit #launch gedit with CPU affinity 0xa. #If the server gets a reboot or the process is restarted, the PID changes taskset -p 0x11 9030 #assign a process to cores 0 and 4 taskset -cp 0,4 9030 #assign a process to cores 0 and 4 taskset -cp 1 9030 # bound the process 9030 to run only on CPU 1, configuration is not permanent ----------------------------------------------------------------------------------------------------- gpg --verify gnupg-2.2.3.tar.bz2.sig gnupg-2.2.3.tar.bz2 -> check the signature of the file gnupg-2.2.3.tar.bz2 ----------------------------------------------------------------------------------------------------- systemd-analyze #the actual boot time of the machine systemd-analyze blame #see how long every program and service takes to start up systemd-analyze critical-chain # print out the results in a chain of events style systemd-analyze critical-chain ntp.service networking.service systemd-analyze plot > boot_analysis.svg xviewer boot_analysis.svg systemd-analyze time -H tecmint@192.168.56.5 #view information from a remote host over ssh systemd-analyze blame -H tecmint@192.168.56.5 systemd-cgtop #top control groups by their resource usage such as tasks, CPU, Memory, Input, and Output ----------------------------------------------------------------------------------------------------- PARSING JSON FILE sudo apt-get install -y jq curl -s 'https://api.github.com/users/lambda' | jq -r '.name' grep -w \"key_name\" /vagrant/test.json |tail -1 | cut -d\" -f4 grep -w \"author\" /vagrant/test.json |tail -1 | cut -d\" -f4 $ FOOBAZ="tester" $ jq -n --arg foobaz "$FOOBAZ" '{"foobaz":$foobaz}' > test1.json $ cat test1.json export $(jq -r '@sh "FOO=\(.foo) BAZ=\(.baz)"') #fill environment variables from JSON object keys (e.g. $FOO from jq query ".foo") echo '{ "foo": 123, "bar": 456 }' | jq '.foo' #print out the foo property apod_url=$(curl -s https://api.nasa.gov/planetary/apod?api_key=DEMO_KEY | jq -r '.hdurl') #get the URL of the current Astronomy Picture of the Day (APOD) echo '{ "Version Number": "1.2.3" }' | jq '."Version Number"' #if a property has a spaces or weird characters echo '[1,2,3]' | jq '.[]' #how iteration works echo '[ {"id": 1}, {"id": 2} ]' | jq '.[].id' #access a property on each item echo '{ "a": 1, "b": 2 }' | jq '.[]' #the value of each key/value pair ---------------------------------------------------------------------------------------------------- bootstrap.sh parted /dev/sdb mklabel msdos parted /dev/sdb mkpart primary 512 100% mkfs.xfs /dev/sdb1 mkdir /mnt/disk mount /mnt/disk # Format the /dev/sdb partition with XFS filesystem and with a GPT partition table sudo parted -s /dev/sdb mklabel gpt mkpart primary xfs sudo mkfs.xfs /dev/sdb -f sudo blkid -o value -s TYPE /dev/sdb # list disk UUIDs ls -l /dev/disk/by-id $ fdisk -v $ sudo fdisk -l $ sudo fdisk -l /dev/sda1 ----------------------------------------------------------------------------------------------------- cut -c3 -> print the character from each 3rd line as a new line of output. cut -c2,7 -> Display the 2nd and 7th character from each line of text cut -c-4 -> Display the first four characters from each line of text cut -c13- -> Print the characters from thirteenth position to the end. cut -d' ' -f4 -> Given a sentence, identify and display its fourth word. Assume that the space (' ') is the only delimiter between words. cut -d' ' -f1-3 -> Given a sentence, identify and display its first three words. Assume that the space (' ') is the only delimiter between words. cut -f 1-3 -> Given a tab delimited file with several columns (tsv format) print the first three fields. cut -f2- -> Given a tab delimited file with several columns (tsv format) print the fields from second fields to last field. ----------------------------------------------------------------------------------------------------- uniq -ci -> count the number of times each line repeats itself (only consider consecutive repetions).compare consecutive lines in a case insensitive manner uniq -u -> display only those lines which are not followed or preceded by identical replications Given a text file, count the number of times each line repeats itself (only consider consecutive repetions). Display the count and the line, separated by a space. uniq -ci | cut -c7- ----------------------------------------------------------------------------------------------------- head -v -n 3 file1 # list the names of the files before outputting their content to the terminal head -n2 -q file1 file2 #Use the -n option to print the first n lines from a file head -n 20 -> Display the first lines of an input file. head -n-2 example.txt # skips the last 2 lines and prints the remaining lines. head -n10 filename | tail -5 #prints the lines between numbers 5 and 10 head -c5 example.txt #prints the first 5 bytes from the file. head -c-7 example.txt #skip printing last 7 bytes. head -c20 -> Display the first characters of an input file. head -n 22 | tail -n +12 -> Display the lines (from line number 12 to 22, both inclusive) of a given text file # print the lines between 5 and 10, both inclusive cat filename | head | tail -6 ----------------------------------------------------------------------------------------------------- tail -n 20 | tail -n +12 -> Display the last lines of an input file. tail -c 20 -> Display the last characters of an input file ----------------------------------------------------------------------------------------------------- echo BigcapsSmallCaps | tr [:lower:] [:upper:] # convert string into lower case, capital case etc tr '()' '[]' -> In a given fragment of text, replace all parentheses with box brackets tr -d [:lower:] -> In a given fragment of text, delete all the lowercase characters tr -d "[:space:]" < raw_file.txt #remove all whitespace characters from the file echo -e " \t A \tB\tC \t " | tr -d "[:blank:]" #deletes any space or tabulation character tr -s ' ' -> In a given fragment of text, replace all sequences of multiple spaces with just one space distribution=$(lsb_release --id | cut -f2 | tr [:upper:] [:lower:]) #all big caps to small caps ----------------------------------------------------------------------------------------------------- sort -> Given a text file, order the lines in lexicographical order. sort -r -> Given a text file, order the lines in reverse lexicographical order sort -n -> the lines reordered in numerically ascending order sort -nr -> The text file, with lines re-ordered in descending order (numerically). given a file of text,in TSV (tab-separated) format.Rearrange the rows of the table in descending order of the values sort -t$'\t' -rnk2 given a file of tab separated weather data (TSV). There is no header column in this data file.Sort the data in ascending order sort -nk2 -t$'\t' given a file of pipe-delimited weather data (TSV). There is no header column in this data file. sort -nrk2 -t$'|' ----------------------------------------------------------------------------------------------------- paste -s -> Given a CSV file where each row contains the name of a city and its state separated by a comma.replace the newlines in the file with tabs paste - - - -> given a CSV file where each row contains the name of a city and its state separated by a comma, restructure the file in such a way, that three consecutive rows are folded into one, and separated by tab. paste -s -d ";" -> given a CSV file where each row contains the name of a city and its state separated by a comma.replace the newlines in the file with semicolon paste - - - -d ";" -> given a CSV file where each row contains the name of a city and its state separated by a comma. restructure the file so that three consecutive rows are folded into one line and are separated by semicolons ----------------------------------------------------------------------------------------------------- #Detect exploitation attempts of the vulnerability in uncompressed files in the Linux logs directory /var/log and all its subdirectories egrep -I -i -r '\$(\{|%7B)jndi:(ldap[s]?|rmi|dns|nis|iiop|corba|nds|http):/[^\n]+' /var/log find /var/log/ -type f -exec sh -c "cat {} | sed -e 's/\${lower://'g | tr -d '}' | egrep -I -i 'jndi:(ldap[s]?|rmi|dns|nis|iiop|corba|nds|http):'" \; find /var/log/ -name '*.gz' -type f -exec sh -c "zcat {} | sed -e 's/\${lower://'g | tr -d '}' | egrep -i 'jndi:(ldap[s]?|rmi|dns|nis|iiop|corba|nds|http):'" \; #searches for exploitation attempts in compressed files in folder /var/log and all sub folders find /var/log -name \*.gz -print0 | xargs -0 zgrep -E -i '\$(\{|%7B)jndi:(ldap[s]?|rmi|dns|nis|iiop|corba|nds|http):/[^\n]+' #files starting at the current directory (.) and that up to a maximum of 1 level of subdirectories find . -maxdepth 2 -type f -name file.txt | xargs -I{} cat {} > ./total_file.txt #Get total size of a list of files perl -le 'map { $sum += -s } @ARGV; print $sum' -- *.pdf #Size of all non-hidden PDF files in current directory. #list files between 1st Dec 2021 and 1st Jan 2022 and total size of each file find . -type f -newermt 2012-01-01 ! -newermt 2022-01-01 -exec du -sh {} \; find . -type f -newermt 2012-01-01 ! -newermt 2022-01-01 -exec ls -lt {} \; | sort -k6M -k7n #sorting month & date based find . -name 'flibble*' -ctime +90 -exec du -sh {} \; find . -type f -mmin -5 -print0 | xargs -0 /bin/ls -ltr #which files was modified in last 5 minutes find . -type f -mmin -5 -exec ls -ltr {} + find . -mmin -5 -exec ls -ltrd {} + #not limiting to files #list files between 1st Dec 2021 and 1st Jan 2022 and grand total size of each found files, not sum of total sizes "find . -name "*.tar" -type f -newermt 2012-01-01 ! -newermt 2022-01-01 -exec du -sch {} +" find . "*.tar" -type f -newermt 2012-01-01 ! -newermt 2022-01-01 -exec du -sch {} + | tail -1 #only total find . -name "*.tar" -type f -newermt 2012-01-01 ! -newermt 2022-01-01 -exec du -sch {} + | tail -1 | awk '{print $1}' $ find . -size +2G #search for all files greater than 2 Gigabytes $ find . -size -10k #search for all files with less than 10 Kilobytes $ find . -size +10M -size -20M #search for files greater than 10MB but smaller than 20MB $ sudo find /var -size +5M -exec ls -sh {} + #search for files in /etc directory which are greater than 5MB and print file size $ find . -type f -exec ls -s {} + | sort -n -r | head -3 #Find first 3 largest files located in a in a current directory recursively $ find /etc/ -type f -exec ls -s {} + | sort -n | head -3 #Find first 3 smallest files located in a in a current directory recursively $ find . -type f -size 0b # search for empty files $ find . -type f -empty # search for empty files $ sudo find /var/log -name \*.log -size +1M -exec ls -lrt {} \; # find files larger than 1M,`M' for Megabytes (units of 1048576 bytes) $ sudo find /var/log -name \*.log -size +1M -exec ls -lrt {} \; | wc -l #get count $ sudo find /var/log -name \*.log -size +1M -exec ls -lrt {} \; | awk '{ total += $5 }; END { print total }' # get total size, column 5(size) of ls command, #total size of all found files $ sudo find /var/log -name \*.log -size +1M -exec ls -l {} \; | awk '{ sum += $5} END \ {hum[1024^3]="Gb"; hum[1024^2]="Mb"; hum[1024]="Kb"; for (x=1024^3; x>=1024; x/=1024) { if (sum>=x) { printf "%.2f %s\n",sum/x,hum[x]; break; } } if (sum<1024) print "1kb"; }' $ find /var/log/apt -type f -name "*.dat" -size +100M #list files larger than 100M $ find /var/log/apt -iname *.log -print0 | xargs -r0 du -csh | tail -n 1; # -iname case insensitive $ find /var/log/apt -iname *.log -exec ls -lh {} \; $ find /var/log/apt -name *.log -size +10c -print0 | du -c --files0-from=- | awk 'END{print $1}' $ find /var/log/apt -name *.log -size +10c -print0 | du -ch --files0-from=- | awk 'END{print $1}' $ find /var/log/apt -name *.log -size +10c -print0 | du -ch --files0-from=- --total -s|tail -1 #xargs pipe "|" calls du command many times $ find /var/log/apt -name *.log -type f -exec ls -s \; | awk '{sum+=$1;} END {print sum/1000;}' #excludes all directories du -ch /var/log/apt | tail -1 | cut -f 1 $ (find /var/log/apt -name *.log -size +10c -printf '%s+'; echo 0 ) | bc $ ( find /var/log/apt -name *.log -size +10c -printf 's+=%s\n'; echo s ) | bc $ find /var/log/apt -name *.log -size +10c -printf '%s\n' | jq -s add $ find /var/log/apt -name *.log -size +10c -exec stat -c%s '{}' + | jq -s add find . -name "*.tar" -type f -newermt 2012-01-01 ! -newermt 2022-01-01 -print0 | xargs -0 du -c --block-size=human-readable find . -name 'flibble*' -ctime +90 -print0 > filenames && du -shc --files0-from=filenames du -c `find . -name 'flibble*' -ctime +90` | tail -1 find . -name 'flibble*' -ctime +90 -printf "%s\n" |perl -lnE '$sum += $_} END {say $sum' find . -name 'flibble*' -ctime +90 -printf "%s\t%p\n" |perl -apE '$sum += $F[0]} END {say $sum' echo "$(( ($(find . -name 'flibble*' -ctime +90 -type f -printf '%k+' )0)/1024/1024 )) GB" #-mtime +7 means older than 8 days (age rounded to integer number of days greater than 7). log_history=13 && find /opt/freeswitch/var/log/freeswitch -type f -mtime +$log_history -delete #Delete old/rotated log files # if tomcat directory exists,delete logs log_history=13 && [[ -d /var/log/tomcat7 ]] && find /var/log/tomcat7 -type f -mtime +$log_history -delete #Delete FreeSWITCH wav/opus recordings older than 13 days history=13 && find /var/freeswitch/meetings/ -name "*.wav" -mtime +$history -delete # find all files, SUID bit enabled find / -perm -4000 -exec ls -l {} \; find /usr/bin/ -perm -4000 -exec ls -l {} \; find /bin/ -perm -4000 -exec ls -l {} \; find / -xdev -perm -4000 2>/dev/null #-perm denotes that we will search for the permissions that follow: #-u=s denotes that we will look for files which are owned by the root user #-type states the type of file we are looking for #f denotes a regular file, excluding directories and special files find / -perm -u=s -type f 2>/dev/null find / -uid 0 –perm -4000 -print #find all programs whose SetUID is set to run as root find / -perm -2000 -exec ls -l {} \; # find all files, SGID bit enabled find /lib/modules/`uname -r` -type f -name '*quota_v*.ko*' #counts files recursively in all subfolders in the specified folder find /data -type f | wc -l #counts files in the current dir, not recursively find /data -maxdepth 1 -type f | wc -l #counts folders recursively in all subfolders in the specified folder find /data -type d | wc -l find -type f -exec md5sum -t {} \; | cut -d ' ' -f 1 | sort | md5sum #compute checksum find . -type f -newermt 2012-02-01 ! -newermt 2022-01-01 #between 1st Dec 2021 and 1st Jan 2022 find . -type f -newermt 2012-02-01 ! -newermt 2022-01-01 -ls #list files between 1st Dec 2021 and 1st Jan 2022 find . -type f -newermt 2012-02-01 ! -newermt 2022-01-01 -exec echo {} \; #test before delete find . -type f -newermt 2012-02-01 ! -newermt 2022-01-01 -exec rm -rf {} \; #delete between 1st Dec 2021 and 1st Jan 2022 find . -type f -newermt 2012-01-01 ! -newermt 2022-01-01 -exec ls -l {} \; #never put the -delete action at the first position #If the -delete action is at the first position, during its evaluation, it deletes the given directory and everything in it #the -delete action implies the -depth option #The -depth option asks the find command to search each directory’s contents before the directory itself. # -delete as the first option, it starts deletion from each directory tree’s very bottom $ find test -delete -type d -name '.git' # the test directory has been deleted $ ls test ls: cannot access 'test': No such file or directory #the -delete action cannot delete a non-empty directory recursively, can only delete files and empty directories find test -depth -type d -name '.git' -exec rm -r '{}' \; #remove all .git directories find test -type d -name '.git' | xargs rm -r #remove all .git directories find ~/Downloads/ -empty -type d -delete #delete all empty directories find /path/ -empty -type d | wc -l ## count empty dirs only ## find /path/to/dir/ -type d -empty -print0 | xargs -0 -I {} /bin/rmdir "{}" #find and delete all empty directories find /path/to/dir -type d -empty -print0 -exec rmdir -v "{}" \; #find and delete all empty directories,slow due to -exec $ sudo find /var -type d -empty -mtime +50 $ sudo find /var -type d -empty -mtime +5 -exec sh -c 'du -sch' sh {} + #-exec with an external command, it fills each found file in the ‘{}’ placeholder find test -name 'whatever.txt' -exec rm {} \; #remove all whatever.txt files find test -name 'whatever.txt' | xargs rm #remove all whatever.txt files find ~/Downloads/ -empty -type -f -delete #delete all empty files find /path/ -empty -type f | wc -l ## count empty files only ## find /path/to/dir/ -type f -empty -print0 | xargs -0 -I {} /bin/rm "{}" #delete all empty files find /path/to/dir/ -type f -empty -print0 -exec rm -v "{}" \; #delete all empty files,slow due to -exec find / -name .DS_Store -delete #-delete will perform better because it doesn't have to spawn an external process for each and every matched file find / -name ".DS_Store" -exec rm {} \; #recommended because -delete does not exist in all versions of find find / -iname "*~" -exec rm -i {} \; # gives an interactive delete find / -name .DS_Store -exec rm {} + #The command termination + instead of \; highly optimizes the exec clause by not running the rm command for each and every .DS_Store present on the file system find / -name .DS_Store -print0 | xargs -0 rm #avoiding the overhead of spawning an external process for each matched file find . -type f -newermt 2012-01-01 ! -newermt 2022-01-01 -exec du -sh {} \;#list files between 1st Dec 2021 and 1st Jan 2022 and total size #-delete does not delete empty directories $ find /path/to/dir/ -type d -name ".TemporaryItems" -delete find: cannot delete ‘./.TemporaryItems’: Directory not empty $ find /path/to/dir/ -type d -name ".TemporaryItems" -exec rm -rv "{}" + find /home -group ftpusers # list file owned by a user or group find /data/project -group ftpusers -name "*.c" # list file owned by all *.c file belongs to a group called “ftpusers find /data/project -group ftpusers -name "*.c" # list file owned by all *.c file belongs to a group called “ftpusers ,case insensitive find $HOME -name "*.mp4" -group pedro -ls #list file in ls command format pass the -ls find /var -user pedro find /var/www -user pedro -name "*.pl" # find all *.pl (perl files) file belongs to a user find / -type f -user bonnie -o -user clyde #find files by users bonnie and clyde find / -type d -user vivek -o -user wendy #find dirs by users bonnie and clyde ---------------------------------------------------------------------------------------------------- #delete multiple folders in a folder which have files in them Run the following command to list the folders that will be affected by the deletion command The output of the find command will display a list of folder names. Carefully review this list to make sure that it includes only the folders you intend to delete find . -type d Once you're satisfied with the list and have made any necessary modifications to exclude specific folders, you can run the original deletion command: find . -type d -exec rm -r {} \; Use the xargs command to pass the output of the find command to the rm command: find . -type d | xargs rm -rf find . -type d -exec rm -r {} \; find .: Searches for files and directories in the current directory and its subdirectories. -type d: Specifies that the search should only include directories (folders). -exec rm -r {} \;: Executes the rm -r command on each found directory. The {} is a placeholder for the found directory name, and \; indicates the end of the -exec command. Be cautious when using this command, as it will delete all directories, including their contents, within the specified directory. find . -type d -exec rm -rf {} \; use the -f option with rm to suppress confirmation prompts and force deletion. However, be cautious with the -f option, as it will delete without asking for confirmation. ---------------------------------------------------------------------------------------------------- find test -type d -name '.git' # list git directories find . -type d -newermt 2012-02-01 ! -newermt 2022-01-01 -ls #list directories between 1st Dec 2021 and 1st Jan 2022 find . -type d -newermt 2012-03-22 ! -newermt 2022-03-24 -exec echo {} \; #test before delete find . -type d -newermt 2012-02-01 ! -newermt 2022-01-01 -exec rm -rf {} \; #delete directories between 1st Dec 2021 and 1st Jan 2022 find /dir/ -type f -newerXY 'yyyy-mm-dd' The letters X and Y can be any of the following letters: a – The access time of the file reference B – The birth time of the file reference c – The inode status change time of reference m – The modification time of the file reference t – reference is interpreted directly as a time find . -type f -newerat 2017-09-25 ! -newerat 2017-09-26 #all files accessed on the 25/Sep/2017 find /home/you -iname "*.c" -atime 30 -type f #all *.c file accessed exactly 30 days ago find /home/you -iname "*.c" -atime -30 -type f #all *.c file accessed 30 days ago, not older than 30 days find /home/you -iname "*.c" -atime -30 -type f -ls find /home/you -iname "*.c" -atime +30 -type f #all *.c file accessed more than 30 days ago, older than 30 days find /home/you -iname "*.c" -atime +30 -type f -ls ----------------------------------------------------------------------------------------------------- #Users of the bash shell need to use an explicit path in order to run the external time command and #not the shell builtin variant. On systemwhere time is installed in /usr/bin, #Real is wall clock time - time from start to finish of the call. This is all elapsed time including #time slices used by other processes and time the process spends blocked (for example if it is waiting for I/O to complete). #User is the amount of CPU time spent in user-mode code (outside the kernel) within the process. #This is only actual CPU time used in executing the process. Other processes and time the process #spends blocked do not count towards this figure. #Sys is the amount of CPU time spent in the kernel within the process. This means executing CPU time #spent in system calls within the kernel, as opposed to library code, which is still running in user-space $ /usr/bin/time -o out.txt sudo find /var/log -name '*log' -ctime +1 -exec du -sh {} \; $ cat out.txt 0.01user 0.03system 0:00.06elapsed 76%CPU (0avgtext+0avgdata 8756maxresident)k 0inputs+0outputs (0major+2771minor)pagefaults 0swaps #the -exec action runs the specified command on the selected files, but the command line is built by appending each selected file name at the end $ /usr/bin/time -o outplus.txt sudo find /var/log -name '*log' -ctime +1 -exec du -sh {} +; $ cat outplus.txt 0.00user 0.01system 0:00.02elapsed 73%CPU (0avgtext+0avgdata 8804maxresident)k 0inputs+0outputs (0major+1091minor)pagefaults 0swaps $ /usr/bin/time -o out.txt sudo find /var/log -name '*log' -ctime +1 -exec du -sh {} \; $ /usr/bin/time -f "\t%C [Command details],\t%K [Total memory usage],\t%k [Number of signals process received]" ping -c 2 howtoforge.com ----------------------------------------------------------------------------------------------------- # displays without comments egrep -v "^#|^$" /etc/zabbix/zabbix_server.conf ----------------------------------------------------------------------------------------------------- #r = recursive i.e, search subdirectories within the current directory #n = to print the line numbers to stdout #i = case insensitive search grep -rni "string" * grep -rni "apache /etc/cron.d" #string search current and subfolders $ grep -rl "900990" . ./.crs-setup.conf.swp ./crs/crs-setup.conf ./crs-setup.conf # displays the comments grep ^# /etc/resolv.conf # displays without comments grep ^[^#] /etc/resolv.confprint directory/file structure in the form of a tree grep ^[^\;] /etc/resolv.conffind . | sed -e "s/[^-][^\/]*\// |/g" -e "s/|\([^ ]\)/|-\1/" grep -v "^#" /etc/zabbix/zabbix_server.conf | grep -v "^$"sed '' quote.txt -> display the contents of the file # search multiple strings, words grep 'string1' filename | grep 'string2' #search two strings in one line grep -n 'string1' filename | grep 'string2' #search two strings in one line and print line numbers grep 'string1.*string2\|string2.*string1' filename #search two strings in one line grep -n 'string1.*string2\|string2.*string1' filename #search two strings in one line and print line numbers grep -E "string1(?.*)string2" file #search two strings in one line grep -nE "string1(?.*)string2" file #search two strings in one line and print line numbers #Grep for Multiple Strings grep 'wordA*'\''wordB' *.py ### Search all python files for 'wordA' or 'wordB' grep 'word*' *.txt ### Search all text files grep 'word1\|word2\|word3' /path/to/file grep 'warning\|error\|critical' /var/log/messages grep -e 'warning\|error\|critical' /var/log/messages egrep -wi --color 'warning|error|critical' /var/log/messages #-i (ignore case) egrep -wi --color 'foo|bar' /etc/*.conf egrep -Rwi --color 'foo|bar' /etc/ #including sub-directories egrep -w 'warning|error|critical' /var/log/messages grep -w 'warning\|error\|critical' /var/log/messages egrep -ne 'null|three' #search multiple string and output line numbers grep -o "0x[^']*" file.txt # matching text starting with "0x" grep "zip$" #filters the lines that end in zip grep -r --include "*.jar" JndiLookup.class / #Detect the presence of Log4j grep --color regex filename #Highlight grep --color ksh /etc/shells grep -o regex filename #Only The Matches, Not The Lines egrep "v{2}" filename #Match a character “v” two times egrep 'co{1,2}l' filename #match both “col” and “cool” words egrep 'c{3,}' filename #match any row of at least three letters ‘c’ grep "[[:digit:]]\{2\}[ -]\?[[:digit:]]\{10\}" filename #match mobile number format 91-1234567890 egrep '[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}' file #match an IP address,All three dots need to be escaped $ grep '^[P-R]' list.txt #lines from list.txt file that starts with P or Q or R $ grep '[^A-C]' list.txt #lines from list.txt file that starts with A or B or C $ grep [!P-R] list.txt #from list.txt file that starts with ‘P’ or Q or R $ grep [!4-8] list.txt #lines from list.txt file that starts with any digit from 4 to 8. $ grep a$ list.txt #lines from list.txt file that ends with ‘a’ $ grep 50$ list.txt #lines from list.txt file that end with the number 50 grep -i "boar" /etc/passwd #Perform a case-insensitive search for the word ‘bar’ grep "Gnome Display Manager" /etc/passwd #If the search string includes spaces, enclose it in single or double quotation marks #the string “linux” will match only if it occurs at the very beginning of a line grep '^linux' file.txt #The ^ (caret) symbol grep 'linux$' file.txt #lines end with linux string grep '^linux$' file.txt #lines contain only linux string grep '^\.[0-9]' filename #lines starting with a dot and digit grep '^..$' filename #lines with two characters #The . (period) symbol is a meta-character that matches any single character grep 'kan..roo' file.txt #match anything that begins with “kan” then has two characters and ends with the string “roo” grep 'acce[np]t' file.txt #find the lines that contain “accept” or “accent” grep 'co[^l]a' file.txt #match any combination of strings starting with “co” followed by any letter except “l” followed by “la”, such as “coca”, “cobalt” and so on grep '^[A-Z]' file.txt #matches each line that starts with a capital letter grep 's*right' #match “right”, “sright” “ssright” and so on grep -E '^[A-Z].*[.,]$' file.txt #matches all lines that starts with capital letter and ends with either period or comma grep 'b\?right' file.txt #match both “bright” and “right”. The ? character is escaped with a backslash because we’re using basic regular expressions grep -E 'b?right' file.txt grep -E 's+right' file.txt #match “sright” and “ssright”, but not “right” grep -E '[[:digit:]]{3,9}' file.txt #matches all integers that have between 3 and 9 digits grep 'word' filename grep 'word' file1 file2 file3 grep -i "boar" /etc/passwd #Perform a case-insensitive search for the word ‘bar’ "grep -R 'httpd' ." #Look for all files in the current directory and in all of its subdirectories # search for a keyword in text files within a folder and its subfolders. grep -r "keyword" /path/to/directory grep -r "keyword" --include "*.txt" /path/to/directory # search for a keyword in a specific type of file grep -r -i "keyword" /path/to/directory #case-sensitive when searching for keywords grep -r "192.168.1.5" /etc/ #search recursively i.e. read all files under each directory for a string “192.168.1.5” grep -c 'nixcraft' frontpage.md #display the total number of times that the string ‘nixcraft’ appears in a file named frontpage.md #Grep NOT #-v flag to print inverts the match; that is, it matches only those lines that do not contain the given word grep -v -c -e "that" -> find out how many lines that does not match the pattern grep -v Sales employee.txt #all the lines except those that contains the keyword “Sales” grep -w "the" -> Output only those lines that contain the word 'the'. grep -iw "the" -> Output only those lines that contain the word 'the'. The search should NOT be case sensitive. grep -viwe "that" -> Only display those lines that do NOT contain the word 'that'. grep -Eiw "th(e|ose|en|at)" < /dev/stdin -> display all those lines which contain any of the following words "the,that,then,those" .The search should not be sensitive to case. Display only those lines of an input file, which contain the required words. grep '\([0-9]\) *\1' -> Given an input file, with N credit card numbers,grep out and output only those credit card numbers which have two or more consecutive occurences of the same digit (which may be separated by a space, if they are in different segments). Assume that the credit card numbers will have 4 space separated segments with 4 digits each #top 10 IP addresses in the log file. grep -E -o "([0-9]{1,3}[\.]){3}[0-9]{1,3}" access.log | uniq -ci | sort -nr | head -n10 ifconfig -a | grep -E -o "([0-9]{1,3}[\.]){3}[0-9]{1,3}" | awk 'ORS=NR%2?" , ":"\n"' ip addr show eth1 | grep inet | awk '{ print $2; }' | sed 's/\/.*$//' ifconfig -a | grep -E -o "([0-9]{1,3}[\.]){3}[0-9]{1,3}" #list process binary path and permissions ps aux | awk '{print $11}' | xargs -r ls -la 2>/dev/null |awk '!x[$0]++' ps -elf | grep autofs | grep -v grep | awk '{print $4}' | xargs kill -9 ######group expressions,grep -E option is for extended regexp,three expressions are functionally equivalent grep "\(grouping\)" file.txt #use parentheses without using extended regular expressions, escape with the backslash grep -E "(grouping)" file.txt egrep "(grouping)" file.txt grep -E "(GPL|General Public License)" GPL-3 #find either GPL or General Public License in the text grep -E "(copy)?right" GPL-3 #matches copyright and right by putting copy in an optional group grep -E '(fear)?less' file.txt #matches both “fearless” and “less”. The ? quantifier makes the (fear) group optional grep -E "free[^[:space:]]+" GPL-3 # matches the string free plus one or more characters that are not white space characters grep -E "[AEIOUaeiou]{3}" GPL-3 #find all of the lines in the GPL-3 file that contain triple-vowels grep -E "[[:alpha:]]{16,20}" GPL-3 #match any words that have between 16 and 20 characters grep -e pattern1 -e pattern2 filename #Grep OR Using grep -e, egrep 'Tech|Sales' employee.txt #Grep OR Using egrep grep 'Tech\|Sales' employee.txt #Grep OR Using \|,grep either Tech or Sales from the employee.txt file grep 'fatal\|error\|critical' /var/log/nginx/error.log grep -E 'fatal|error|critical' /var/log/nginx/error.log # use the extended regular expression, then the operator | should not be escaped grep -E 'Tech|Sales' employee.txt #Grep OR Using -E #Grep AND grep Manager employee.txt | grep Sales #all the lines that contain both “Manager” and “Sales” in the same line grep -E 'Dev.*Tech' employee.txt #all the lines that contain both “Dev” and “Tech” in it (in the same order). grep -E 'Manager.*Sales|Sales.*Manager' employee.txt #all the lines that contain both “Manager” and “Sales” in it (in any order) ----------------------------------------------------------------------------------------------------- user@host: $ cat< file.txt $ > 1 line $ > other line $ > n line $ > EOF user@host: ----------------------------------------------------------------------------------------------------- # append text cat< outputs to a file >> appends to a file < reads input < print.sh #!/bin/bash echo \$PWD echo $PWD EOF ----------------------------------------------------------------------------------------------------- # Pass multi-line string to a pipe in Bash $ cat </dev/null # my config file foo=bar EOF ----------------------------------------------------------------------------------------------------- echo -e " Home Directory: $HOME \n hello world 1 \n hello world 2 \n line n... \n " > file.txt ----------------------------------------------------------------------------------------------------- echo write something to file.txt | cat > file.txt cat >file.txt <<< Write something here # see the line numbers cat -n song.txt # shows at the end of line and also in space showing ‘$‘ if there is any gap between paragraphs # useful to squeeze multiple lines in a single line. cat -e test # all output will be redirected in a newly created file cat test test1 test2 > test3 # Sorting Contents of Multiple Files in a Single File cat test test1 test2 test3 | sort > test4 # Display Multiple Files at Once cat test; cat test1; cat test2 # append your text to the end of the file cat >> ~/.bashrc < /sys/devices/system/cpu/cpu1/cpufreq/scaling_governor < WinXP.img # join smaller files into a larger one cat partfilename* > outputfilename video.avi.01 video.avi.02 video.avi.03 cat video.avi.* > video1.avi $ cat file1 1. Asia: 2. Africa: 3. Europe: 4. North America: tac tacexample.txt #print files in reverse $ tac file1 4. North America: 3. Europe: 2. Africa: 1. Asia: $ cat file2 1. India 2. Nigeria 3. The Netherlands 4. The US $ join file1 file2 1. Asia: India 2. Africa: Nigeria 3. Europe: The Netherlands 4. North America: The US join -1 1 -2 1 -e 'empty' /tmp/in /tmp/out | tr " " "\t" #join the two files,the first column in each file as index # create a new file “new.txt” that is a concatenation of “file1.txt” and “file2.txt” cat file1.txt file2.txt > new.txt format text and convert it to a different width $ fmt --width=20 test.txt ------------------------------------------------------------------------------------------ g++ -v g++ temp.cpp # run ./a.out ------------------------------------------------------------------------------------------ # Installing software from source tar xvzf package.tar.gz tar xvjf package.tar.bz2 cd package ./configure make make install # Cleaning up make clean make uninstall # "make clean" runs as expected even if you do have a file named clean. # There are two reasons to use a phony target: to avoid a conflict with a file of the same name, and to improve performance. .PHONY: clean clean: rm -rf *.o # case1 $ cat make hello : hello.o gcc -Wall hello.o -o hello $ cat hello.c #include int main(void) { printf("\n Hello World!!!\n"); return 0; } $ cp make mon-makefile $ ls hello.c make mon-makefile make -C makefile-test1/ hello make -f mon-makefile make -s # case2 $ cat file2.h void add(int a, int b, void (*f)(int)); $ cat file2.c #include #include"file2.h" void add(int a, int b, void(*f)(int)) { int c = a+b; f(c); } $ cat file1.c #include #include"file2.h" void callback (int result) { printf("\n Result is : [%d] \n", result); } int main(void) { int a=0,b=0; printf("\n Enter two numbers to add: "); scanf("%d %d",&a, &b); add(a,b,callback); return 0; } $ cat makefile file : file1.o file2.o gcc -Wall file2.o file1.o -o file file1.o : file1.c file2.h gcc -c -Wall file1.c -o file1.o file2.o : file2.c file2.h gcc -c -Wall file2.c -o file2.o $ cp make mon-makefile $ ls file1.c file2.c file2.h makefile mon_makefile ------------------------------------------------------------------------------------------ # linux system management top sar vmstat iostat free ps tcpdump iptraf nestat # /Proc file system - Various Kernel Statistics cat /proc/cpuinfo cat /proc/meminfo cat /proc/zoneinfo cat /proc/mounts ------------------------- ----------------------------------------------------------------- $ uptime #"system load averages" that show the running thread (task) demand on the system as an average number of running plus waiting threads. #show three averages, for 1, 5, and 15 minutes #If the averages are 0.0, system is idle #If the 1 minute average is higher than the 5 or 15 minute averages, then load is increasing #If the 1 minute average is lower than the 5 or 15 minute averages, then load is decreasing. ps -eL h -o state | egrep "R|D" | wc -l #The instantaneous number of such tasks #Linux load average,the instantaneous load of a system the number of tasks (processes and threads) that are willing to run at a given time t #either in state R or D, either actually running or blocked on some resource (CPU, IO, ...) waiting for an opportunity to run ------------------------------------------------------------------------------------------ list user vagrant's full command line of processes $ top -c -u vagrant ignore idle processes $ top -i -u vagrant updated with 5 secs intervals, including child processes $ top -u vagrant -c -d 5 -S #determine which Plaso processes are running top -p `ps -ef | grep log2timeline.py | grep python | awk '{ print $2 }' | tr '\n' ',' | sed 's/,$//'` #interactive top command #If you have a multi-core CPU, press "1" to change the display and see individual statistics for each CPU #press “t” to swap the CPU displays to simple ASCII graphs that show the percentage of usage for each CPU #Pressing “c” toggles the COMMAND column between displaying the process name and the full command line #Press “u” to see the processes for a single user. You’ll be prompted for the name or UID #Press I to see only active tasks. #press “r” to change the nice value (priority) for a process #Press “k” to kill a process. You’ll then be prompted for the process ID you want to kill ------------------------------------------------------------------------------------------ The load averages shown by these tools is read /proc/loadavg file cat /proc/loadavg ------------------------------------------------------------------------------------------ # same inodes ls -ldi /. /.. 2 drwxr-xr-x 24 root root 4096 Feb 21 20:28 /. 2 drwxr-xr-x 24 root root 4096 Feb 21 20:28 /.. # different inodes ls -ldi /home/vagrant/. /home/vagrant/.. 3145730 drwxr-xr-x 7 vagrant vagrant 4096 Feb 22 10:14 /home/vagrant/. 3145729 drwxr-xr-x 3 root root 4096 Aug 24 08:48 /home/vagrant/.. ------------------------------------------------------------------------------------------ #ChatGPT Both .bashrc and .profile files are used for setting environment variables and defining user-specific configurations on Linux, but they have some differences in their purpose and usage. Here are some of the differences between .bashrc and .profile files: File Location: .bashrc is a Bash shell-specific initialization file that is located in the user's home directory. On the other hand, .profile is a generic shell initialization file that is also located in the user's home directory. Shell Usage: .bashrc is loaded for Bash shell instances only, while .profile is loaded for every shell instance (including Bash, Korn shell, Bourne shell, etc.) that is started. Invocation: .bashrc is invoked for non-login shells, while .profile is invoked for login shells. Content: .bashrc is typically used for setting Bash shell-specific configurations such as aliases, environment variables, and command prompt settings, while .profile is used for setting environment variables and other configurations that should be applied to all shells. Loading order: When a Bash shell instance is started, it first loads the /etc/profile file, which is a system-wide shell initialization file. After that, it loads the user's ~/.bash_profile or ~/.bash_login or ~/.profile file (in that order) if it exists. If none of these files exist, it loads the ~/.bashrc file. In summary, .bashrc is used for Bash shell-specific configurations, while .profile is used for general shell configurations that should apply to all shells. .bashrc is loaded for non-login shells, while .profile is loaded for login shells. ------------------------------------------------------------------------------------------ # The user file-creation mode mask (umask) /etc/profile ~/.bashrc ~/.bash_profile # By default 0022 (022) 0002 (002) files 666 directories 777 The default umask 002 used for normal user directory permissions 775 file permissions 664 The default umask for the root user is 022 directory permissions 755 file permissions 644 base permissions directory permissions (rwxrwxrwx) 0777 file permissions (rw-rw-rw) 0666 # No other user can read or write your data umask 077 # when you share data with other users in the same group # Members of your group can create and modify data files # those outside your group can read data file, but cannot modify it. umask 022 # exclude users who are not group members umask 007 # The octal umasks are calculated via the bitwise AND of the unary complement of the argument using bitwise NOT Octal value : Permission 0 : read, write and execute 1 : read and write 2 : read and execute 3 : read only 4 : write and execute 5 : write only 6 : execute only 7 : no permissions ------------------------------------------------------------------------------------------ # testing webpages $ telnet control01 80 #test website availability and get the response code GET /index.html HTTP/1.1 Host: control01 GET /index.html HTTP/1.1 Host: control01 If-modified-since: Sun, 24 Feb 2019 12:59:37 GMT GET /telnet-send-get-head-http-request HTTP/1.1 HOST: control01 HEAD / HTTP/1.1 Host: control01 ------------------------------------------------------------------------------------------ # delete files containing special chars $ cat>>"-f"< test > EOF $ ls -f test test2 videos $ ls -li total 260 3145770 -rw-rw-r-- 1 vagrant vagrant 5 Mar 5 17:02 -f find / -name wget 2>/dev/null find /home/vagrant -name file1 find /home/vagrant -user root find /home/vagrant -group root find . -inum 3145770 -delete find . -inum 3145770 -exec rm -i {} \; ls -il {file-name} find . -inum [inode] -exec rm -i {} \; ------------------------------------------------------------------------------------------ # steganography, attaching a .rar file to a .jpg etc. cat pic.jpg file.rar > result.jpg ------------------------------------------------------------------------------------------ # view used IRQs cat /proc/interrupts #determine the IRQ number associated with the Ethernet driver $ grep enp0s3 /proc/interrupts 19: 24006 IO-APIC-fasteoi enp0s3 $ grep eth0 /proc/interrupts 19: 13247 IO-APIC 19-fasteoi eth0 # troubleshoot network card etc. hardware conflicts, what addresses are used or free, or move conflicting hardware to free resource # listing IRQs currently being used.not listed IRQs are considered free.used when devices alert CPU to take action cat /proc/interrupts # listing used DMA channel.when devicess access memory directly without going through CPU cat /proc/dma # listing I/O ports currently being used.any range not listed is free and can be used by other devices. # devices have unique I/O addresses cat /proc/ioports ------------------------------------------------------------------------------------------ # view default shell for each user $ cat /etc/passwd # There are seven fields in the /etc/passwd file # username, UID, GID, comment, home directory, command # add an asterisk at the beginning of the password field in the /etc/passwd file, that user will not be able to log in /etc/passwd /etc/adduser.conf #Set DHOME variable for new default home directory of users. # useradd tecmint # useradd -m tecmint #create user with default home dir # passwd tecmint create admin user sudo useradd vagrant -s /bin/bash -g sudo -m modify a existing user to admin user sudo usermod -aG sudo vagrant # add user to a group, add vagrant to sudo group create a user ‘anusha‘ with a home directory ‘/data/projects‘. # useradd -d /data/projects anusha create a user ‘navin‘ with custom userid ‘999‘. # useradd -u 999 navin add a user ‘tarunika‘ with a specific UID and GID # useradd -u 1000 -g 500 tarunika add a user ‘tecmint‘ into multiple groups like admins, webadmin and developer # useradd -G admins,webadmin,developers tecmint # id tarunika uid=995(tarunika) gid=1001(vboxadd) groups=1001(vboxadd) create a user ‘aparna‘ with account expiry date i.e. 27th April 2014 in YYYY-MM-DD useradd -e 2014-03-27 aparna useradd -r -d /opt/gvm -c "GVM User" -s /bin/bash gvm #Create a system account,subordinate uid/gid feature is disabled,mail directory is not created chage -l aparna #verify the age of account chage -d 0 user #Minimum number of days between password change make changes to "/etc/shadow" file as root user #Minimum number of days between password change chage -M 7 aparna # aparna account’s password must be changed at least once every seven days sudo useradd test-user-0 && sudo passwd -d test-user-0 #create user without password awk -F: '($2 == "") {print}' /etc/shadow # verify users without password passwd -l test-user-0 #Lock all empty password account cat /etc/passwd | cut -d : -f 1 | awk '{ system("sudo passwd -S " $0) }' #list locked users sudo passwd -S -a | grep " L " | cut -d " " -f1 #list locked users, ubuntu sudo passwd -S -a | cut -d " " -f1-2 | grep "L$" #list locked users, ubuntu passwd -l user1 #lock the password grep user1 /etc/shadow #two exclamation mark (!!) before the encrypted password which means that the password has been locked passwd -u user1 #unlock the password passwd -S user1 #see the lock status of the user #A user whose account has a "non-interactive shell" (/bin/false, /sbin/nologin) can't log in interactively #prevents SSH command execution if the user has SSH keys on the system #users can't get a shell prompt to run commands # users may be able to log in to do read/send email (via POP/IMAP & SMTP AUTH) #/etc/security/access.conf #service accounts often don't have a "proper" login shell, /usr/sbin/nologin as login shell (or /bin/false) #service accounts have user IDs in the low range, e.g. < 1000 or so. Except for UID 0 #service accounts are typically locked #not possible to login (for traditional /etc/passwd and /etc/shadow) #achieved by setting the password hash to arbitrary values such as * or x) useradd testuser --shell=/sbin/nologin usermod testuser -s /sbin/nologin usermod -s /sbin/nologin test4 #Disable User Account With nologin or false Shells usermod testuser -s /bin/false #exit as soon as the user logs in, and return exit status 0 (false),do not receive any kind of message like they would with /sbin/nologin adduser subversion --shell=/bin/false --no-create-home # directed to the /sbin/nologin shell and receive the following message This account is currently not available. usermod testuser -s /bin/bash #enable shell by setting a shell #list non-interactive shell users, service accounts, aka technical accounts $ awk -F: '{if( $7~"/sbin/nologin" || $7~"/bin/false") print "service account..:",$1,"shell..:" $7}' /etc/passwd #The users that have nologin defined as their default shell often have higher privileges,unable to log in directly #limit the damage that a breach of system could suffer /sbin/nologin #Limiting system's attack surface /bin/false set a account password expiry date i.e. 45 days on a user ‘tecmint’ # useradd -e 2014-04-27 -f 45 tecmint insert that user’s full name, Manis Khurana # useradd -c "Manis Khurana" mansi add a user ‘tecmint‘ without login shell # useradd -s /sbin/nologin tecmint create a user ‘ravi‘ with home directory ‘/var/www/tecmint‘, default shell /bin/bash and adds extra information about user # useradd -m -d /var/www/ravi -s /bin/bash -c "TecMint Owner" -U ravi # useradd -m -d /var/www/tarunika -s /bin/zsh -c "TecMint Technical Writer" -u 1000 -g 1000 tarunika disabling login shell to a user called ‘avishek‘ # useradd -m -d /var/www/avishek -s /usr/sbin/nologin -c "TecMint Sr. Technical Writer" -u 1019 avishek useradd librenms -d /opt/librenms -M -r -s "$(which bash)" tecmint:x:504:504:tecmint:/home/tecmint:/bin/bash Username: User login name used to login into system. It should be between 1 to 32 charcters long. Password: User password (or x character) stored in /etc/shadow file in encrypted format. User ID (UID): Every user must have a User ID (UID) User Identification Number. By default UID 0 is reserved for root user and UID’s ranging from 1-99 are reserved for other predefined accounts. Further UID’s ranging from 100-999 are reserved for system accounts and groups. Group ID (GID): The primary Group ID (GID) Group Identification Number stored in /etc/group file. User Info: This field is optional and allow you to define extra information about the user. For example, user full name. This field is filled by ‘finger’ command. Home Directory: The absolute location of user’s home directory. Shell: The absolute location of a user’s shell i.e. /bin/bash. userdel member2 # deletes the account but leaves the user’s home directory intact userdel -f member2 # the -f option forces account deletion and fle removal under some circumstances userdel -r member2 # delete user account member2, the -r option to userdel causes it to delete the user’s home directory and mail spoo #disable root user sudo passwd -l root #lock the password for the root user sudo usermod -L root #lock the password for the root user sudo passwd -d root # remove the password of the account,can not unlock a locked account if it doesn't have any password #Enable Root Login sudo passwd root #set a new password for the account sudo passwd -u root #unlock the root user,unlock the password su root usermod -L testuser # lock user account / disable user account cat /etc/shadow | grep testuser # verify locked user account, exclamation mark "!" sudo passwd -S testuser #verify locked user account passwd --status testuser #verify locked user account usermod -U member1 # unlock user account passwd -u user1 #unlock the password # configure user default shell usermod -s /bin/bash member1 # add the user geek to the group sudo usermod -a -G sudo geek useradd -ms /bin/bash newuser #creates a home directory for the user and ensures that bash is the default shell useradd -rm -d /home/ubuntu -s /bin/bash -g root -G sudo -u 1001 ubuntu ------------------------------------------------------------------------------------------ #use useradd instead of its interactive wrapper adduser # create new user, new group with the same name sudo adduser sdn --system --group make sure username is added to the group libvirtd $ sudo adduser `id -un` libvirtd $ sudo adduser $(id -un) libvirtd ------------------------------------------------------------------------------------------ #edit the managers entry in /etc/group to make sally, tom, and dale members of the group managers (GID 501),the #order of users in the comma-separated user list is unimportant managers:x:501:dale,sally,tom cat /etc/group cat /etc/gshadow cut -d: -f1 /etc/group | sort $ groupadd admins #add new group $ sudo groupdel sshusers $ useradd -G admins member1 sudo usermod -aG docker ${USER} #Add current logged in user to the docker group. sudo usermod -aG docker $(whoami) #Add current logged in user to the docker group. gpasswd -a devops1 sudo #Adding user devops1 to group sudo gpasswd -d devops1 sudo #Removing user devops1 from group sudo #list groups on Linux,/etc/group file $ getent group $ getent group sudo # sudo group has ID of 27 and has ubuntu,barak,packer as members sudo:x:27:ubuntu,barak,packer $ cat /etc/group | awk -F: '{print $1}' $ cat /etc/group | cut -d: -f1 $ groups # list the groups for the current logged user $ groups barak #get a list of groups a specific user is in whoami grep "^$(whoami):" /etc/subuid grep "^$(whoami):" /etc/subgid id -u print only the effective user ID id -Gn id vagrant # The newgrp command allows a user to override the current primary group. # newgrp can be handy when you are working in a directory where all files must have the same group ownership newgrp microk8s #When you run the command, the system places you in a new shell and changes the name of your real group to the group specified with the Group parameter #If you’re listed as member of the group and the group has a password assigned, then you won’t be prompted #If there is no group password set, and the user is not listed as a member of the group, the user will be denied access #you are not a member of the developers group and not a root, then you’ll be prompted for a group password if your account doesn’t have a password assigned #If you’re root, then no prompted is presented. newgrp developers newgrp - developers #log in to the group developers sudo groupadd developers #create group developers sudo usermod -a -G developers richard sudo gpasswd developers #system account vs user account #System users will be created with no aging information in /etc/shadow #numeric identifiers are chosen in the SYS_UID_MIN–SYS_UID_MAX range defined in /etc/login.defs useradd --system appuser chage -l appuser useradd --system --shell=/usr/sbin/nologin #create a system user (without home directory and login shell) useradd -r subversion #-r, --system create a system account adduser -r -s /bin/nologin subversion #create system account -s /sbin/nologin to disable any login shell adduser --system --no-create-home --group yourusername adduser subversion --system --group adduser -r -s /bin/nologin subversion $ sudo groupadd --system --gid 1002 appuser $ cat /etc/group | grep appuser appuser:x:1002: $ sudo useradd --no-log-init --system --uid 1001 --gid 1002 appuser $ cat /etc/passwd | grep appuser appuser:x:1001:1002::/home/appuser:/bin/sh # list a group's members $ sudo lid -g sales $ cut -d: -f1,4 /etc/passwd | grep $(getent group sales | cut -d: -f3) | cut -d: -f1 id username grep "docker" /etc/group grep -i --color 'root' /etc/group getent group -> List all groups cat /etc/group -> List all groups getent group vboxusers groups -> View the Groups a User Account groupmod -g 3000 foo -> assign a new GID to group called foo ------------------------------------------------------------------------------------------ The command you've provided is a Linux shell command that utilizes the cp command to copy files and directories. Let's break down the components of the command: cp: This is the command itself, which stands for "copy." It's used to copy files or directories from one location to another. -vrbi: These are options or flags that modify how the cp command behaves: -v: Stands for "verbose." When this option is used, the cp command will provide more detailed output, showing the names of files as they are copied. -r: Stands for "recursive." This option is used when copying directories. It tells the cp command to copy not only the specified directory but also its contents and subdirectories. -b: Stands for "backup." With this option, the cp command will create backup copies of files that already exist in the destination directory before overwriting them. -i: Stands for "interactive." When this option is used, the cp command will prompt you for confirmation before overwriting files in the destination directory. This is a safety measure to prevent accidental data loss. /etc/skel/.: This is the source directory or file. In this case, it's the .skel directory located in the /etc directory. The /etc/skel directory is often used as a template for creating user home directories. ~/: This is the destination directory. The tilde (~) is a shorthand notation for the user's home directory. So, this part of the command specifies that the copied content should be placed in the user's home directory. The purpose of this command is to copy the contents of the /etc/skel directory (which often contains default settings and configurations for new user accounts) into the user's home directory. It uses various options like -v, -r, -b, and -i to control the copying process and handle situations where files already exist in the destination directory. To automatically answer "yes" to the interactive prompts and proceed with copying without manual confirmation, you can use the yes command in combination with the cp command. The yes command repeatedly outputs the letter "y" (or any other specified character) in response to prompts. Here's an example of how you could use it: sh yes | cp -vrbi /etc/skel/. ~/ In this example, the yes command keeps providing "y" responses to the interactive prompts, effectively saying "yes" to each one. This allows the cp command to copy files without requiring manual confirmation. ------------------------------------------------------------------------------------------ #Files in /etc/skel are copied from this directory to new users’ home directories by certain account-creation tools #The files should be in all new users’ home directories should reside in /etc/skel. “skeleton” directory is defined in /etc/default/useradd file. # ls -lart /etc/skel # ls -ldi /etc/skel 33554552 drwxr-xr-x. 2 root root 62 Mar 24 2018 /etc/skel ------------------------------------------------------------------------------------------ valid login shells # cat /etc/shells #list all available shells # chsh -l chsh -s /bin/ksh #Regular user can change their shell to the ksh $ ps -p $$ PID TTY TIME CMD 56017 pts/0 00:00:00 bash grep tecmint /etc/passwd #view default login shell usermod --shell /bin/bash tecmint #change its login shell from /bin/sh to /bin/bash grep tecmint /etc/passwd #verify /bin/bash grep tecmint /etc/passwd #view default login shell chsh --shell /bin/bash tecmint #change its login shell from /bin/sh to /bin/bash grep tecmint /etc/passwd #verify /bin/bash $ sudo chsh -s /bin/ksh UserName #The superuser (root) changes the login shell for any account chsh -s /bin/ksh vagrant #set default login shell to /bin/ksh for vagrant user #open the /etc/passwd file, change manually vi /etc/passwd chsh vagrant -s /bin/rbash #change user vagrant's shell su - vagrant #login to activate shell whereis ksh grep --color ksh /etc/shells cat /etc/shells echo $SHELL ~ [23]$ ps -p $$ PID TTY TIME CMD 193052 pts/1 00:00:00 ksh ~ [27]$ ps -hp $$ | awk '{print $5}' -ksh ~ [28]$ echo $0 -ksh ~ [29]$ printf "%s\n" $0 -ksh ~ [30]$ readlink /proc/$$/exe /usr/bin/ksh93 awk -F: '/vagrant/ { print $7}' /etc/passwd #list user's shell type sudo ps -ef | egrep 'tty|pts' #Listing all shell types used by users $ w -h | awk '{print $2}' | xargs -L1 pgrep -oat verify user vagrant's bash # cat /etc/passwd | grep vagrant # echo $SHELL Reading Library editor $ cat /etc/inputrc keyboard bindings $ bind -v ------------------------------------------------------------------------------------------ semicolon ";" multiple commands on the same line backaslash "\" run commands longer than one line press tab key or twice ESC key, command completion "./" run a command from pwd ------------------------------------------------------------------------------------------ history file size setting $ cat /etc/profile $ echo $HISTSIZE $ echo $HISTFILE $ fc -l #Linux Command History with date and time, temporary HISTTIMEFORMAT="%d/%m/%y %H:%M " HISTTIMEFORMAT="%d/%m/%y %T " export HISTSIZE=0 #Disable the usage of history using HISTSIZE echo $HISTSIZE echo $HISTFILE export HISTCONTROL=ignoredups #Eliminate the continuous repeated entry from history using HISTCONTROL export HISTIGNORE="pwd:ls:ls -ltr:" #Ignore specific commands from the history using HISTIGNORE export HISTCONTROL=erasedups #Erase duplicates across the whole history using HISTCONTROL export HISTCONTROL=ignorespace #Force history not to remember a particular command using HISTCONTROL # service httpd stop [Note that there is a space at the beginning of service,to ignore this command from history] history -c #Clear all the previous history # !ps #Execute previous command that starts with a specific word # !4 #Execute a specific command from history # !-1 #execute the second last command # !! #run the last executed command, or press CTRL+P # !dconf #re-run the last command with the keyword ‘dconf’ in it #lastb #shows users that failed to login,review the /var/log/btmp file (containing failed login attempts) #the login history of users last logins last -R #review the contents of the /var/log/wtmp binary file last | grep sysadmin last -f /var/log/btmp #Use the last command to view the btmp file last mark #pass the user name last pts/0 #pass the tty last mark root pts/0 #specify multiple usernames and ttys last -p 2020-01-15 #find out who logged into the system on a specific date last -s 2020-02-13 -u 2020-02-18 #the -s (--since) and -t (--until) option to tell last to display the lines since or until the specified time last -F #y default, last doesn’t show the seconds and the year. Use the -F, --fulltimes option last -25 #last 25 logins last -i #IP address last -d #DNS address #the system last rebooted last reboot ------------------------------------------------------------------------------------------ echo $PATH view all the env variables $ export -p $ set Set an Environment Variable $ export MYAPP=1 holds the list of all directories that are searched by the shell when you type a command name $PATH system-wide $ cat /etc/profile single user $ cat .bash_profile # add PATH vi .bash_profile export PATH=$PATH:$HOME/Downloads/terraform system-wide prompt setting $ cat /etc/bashrc $ echo $PS1 ------------------------------------------------------------------------------------------ Display current libraries from the cache # ldconfig -p | head -5 Display libraries from every directory ldconfig -v | head # cat /etc/ld.so.conf ------------------------------------------------------------------------------------------ # number the lines in a file nl alphaservices | tee alphabetservices $ cat file1 1. Asia: 2. Africa: 3. Europe: 4. North America: Display the contents of file.txt in octal format (one byte per integer) $ od -b file1 0000000 061 056 040 101 163 151 141 072 012 062 056 040 101 146 162 151 0000020 143 141 072 012 063 056 040 105 165 162 157 160 145 072 012 064 0000040 056 040 116 157 162 164 150 040 101 155 145 162 151 143 141 072 0000060 012 0000061 Display the contents of file.txt in ASCII (character) format, with byte offsets displayed as hexadecimal. $ od -Ax -c file1 000000 1 . A s i a : \n 2 . A f r i 000010 c a : \n 3 . E u r o p e : \n 4 000020 . N o r t h A m e r i c a : 000030 \n 000031 ------------------------------------------------------------------------------------------ echo -n "hola" | od -A n -t x1 | sed 's/ *//g' | tr -d '\n' >> engineID_hex.txt #strips spaces remove newlines ------------------------------------------------------------------------------------------ # merge all files in the directory and split ls | xargs cat | tee file1 | split -5 # printing pr -h "title" file1 list mounted file systems $ cat /etc/mtab split -l 4 index.txt split_file #Split file based on number of lines split index.txt -l 4 --verbose split -l 4 -a 4 index.txt #Change in suffix length. By default, the suffix length is 2 split -l 4 -d index.txt #change the split files suffix to numeric split -l 4 index.txt split_index_ # create split output files with index suffix, split -l 4 -e index.txt #Avoid zero-sized split files split -n 3 index.txt #Create n chunks output files split -n 2 index.txt #Split the file into two files of equal length #split the file index.txt into separate files called indexaa, indexab, …..with each file containing 16 bytes of data split -b 16 index.txt index split -b=1M -d file.txt file --additional-suffix=.txt split -b 10M -d system.log system_split.log ~$ cat test.txt | wc -l 40 ~$ split --numeric-suffixes=2 --additional-suffix=.txt -l 22 test.txt file $ ls -lai file* 2139 -rw-rw-r-- 1 vagrant vagrant 660 Mar 21 11:01 file02.txt 5482 -rw-rw-r-- 1 vagrant vagrant 660 Mar 21 11:01 file03.txt 5483 -rw-rw-r-- 1 vagrant vagrant 660 Mar 21 11:01 file04.txt 18296 -rw-rw-r-- 1 vagrant vagrant 220 Mar 21 11:01 file05.txt $ cat file04.txt | wc -l 12 $ cat file05.txt | wc -l 4 ----------------------------------------------------------------------------------------------------- u stands for user. g stands for group. o stands for others. a stands for all. same output: chmod -R 755 /var/www/html #change the permissions of all files and subdirectories under the /var/www/html directory to 755 chmod +x somefile (Based on umask value) chmod a+x somefile, chmod ugo+x somefile (Without considering umask value), add the execute permission for everyone chmod 644 a.txt stat -c %a a.txt # verify permission granted is 644 ------------------------------------------------------------------------------------------ Applying SUID Permission Numerically # chmod 4755 /bin/ping Removing SUID by Numerically chmod 0755 /bin/ping Applying SUID Permission to ping binary file Alphabetically # chmod u+s /bin/ping Removing SUID Permission chmod u-s /bin/ping ------------------------------------------------------------------------------------------ # Applying SGID Permission chmod g+s /database/ chmod 2775 database/ # Remove SGID Alphabetically chmod g-s /database/ chmod 0775 /database ------------------------------------------------------------------------------------------ a sticky bit is now in place and only root, file or directory owners can rename and delete files # chmod +t /var/share/ # ls -ld /var/share/ drwxrwxrwt. 2 root root 4096 Mar 5 11:21 /var/share/ chmod 0777 somefile (octal) chmod 777 somefile (decimal) chmod 0710 mydir ; ls -ld mydir chmod 00710 mydir ; ls -ld mydir ------------------------------------------------------------------------------------------ # quota settings sudo apt update ; sudo apt install quota -y quota --version find /lib/modules/`uname -r` -type f -name '*quota_v*.ko*' sudo mount -o remount / cat /proc/mounts | grep ' / sudo quotacheck -ugm / sudo quotaon -v / sudo setquota -u member1 200M 240M 0 0 / sudo quota -vs member1 sudo setquota -t 864000 864000 / sudo repquota -s / ------------------------------------------------------------------------------------------ # ldd (Unix) ldd (List Dynamic Dependencies) ldd /bin/ls # display unused direct dependencies ldd -u /bin/ping # more information ldd -v /bin/ping ------------------------------------------------------------------------------------------ # rename root $ head -2 /etc/passwd root:x:0:0:root:/root:/bin/nologin rootmon:x:0:0:root:/root:/bin/bash $ sudo passwd rootmon $ su - rootmon # pwd /root #gain a root shell is by adding a new user to /etc/passwd who has the UID 0 #any user with UID 0 is effectively root #root2 is the username #WVLY0mgH0RtUI is the encrypted password we want him to have. Unencrypted, the password is mrcake, in this case #0:0 means the user id and group id are both 0 #root is a comment field #/root is the home directory #/bin/bash is the default shell root2:WVLY0mgH0RtUI:0:0:root:/root:/bin/bash #Administrative databases in Unix,getent – get entries from administrative database passwd – can be used to confirm usernames, userids, home directories and full names of your users group – all the information about Unix groups known to your system services – all the Unix services configured on your system networks – networking information – what networks your system belongs to protocols – everything your system knows about network protocols $ getent hosts # /etc/hosts file $ getent hosts vg-ubuntu-01 double-check which IPs this hostname points to $ getent networks #check the network and IP address of your system $ getent services 20 #Use “services” with the port number to find the service name and its protocol #List Users(system and normal users) on Linux using the /etc/passwd File, normal user has a real login shell and a home directory. awk -F: '{ print $1}' /etc/passwd cat /etc/passwd | awk -F: '{print $1}' awk -F: '{ print $1}' /etc/passwd | wc -l # get the # of users cut -d: -f1 /etc/passwd cat /etc/passwd | cut -d: -f1 getent passwd # list users getent passwd | awk -F ":" '{print $1}' getent passwd | cut -d: -f1 getent passwd # equivalent to cat /etc/passwd getent passwd rahul #details for a particular user getent passwd 0 #find a username by UID $ cut -d":" -f1 /etc/passwd #list all users #list normal user names awk -F: '{if($3 >= 1000 && $3 < 2**16-2) print $1}' /etc/passwd awk -F: '{if(($3 >= 500)&&($3 <65534)) print $1}' /etc/passwd awk -F: '{if(!(( $2 == "!!")||($2 == "*"))) print $1}' /etc/shadow grep -E ":[0-9]{4,6}:[0-9]{4,6}:" /etc/passwd | cut -d: -f1 $ getent passwd | awk 'NR==FNR { if ($1 ~ /^UID_(MIN|MAX)$/) m[$1] = $2; next } { split ($0, a, /:/); if (a[3] >= m["UID_MIN"] && a[3] <= m["UID_MAX"] && a[7] !~ /(false|nologin)$/) print a[1] }' /etc/login.defs - $ getent passwd | \ nologin|false)> grep -vE '(nologin|false)$' | \ : -v mi> awk -F: -v min=`awk '/^UID_MIN/ {print $2}' /etc/login.defs` \ X/ {p> -v max=`awk '/^UID_MAX/ {print $2}' /etc/login.defs` \ $3 >= > '{if(($3 >= min)&&($3 <= max)) print $1}' | \ t -u> sort -u grep -E '^UID_MIN|^UID_MAX' /etc/login.defs #Each user has a numeric user ID called UID. If not specified automatically selected from the /etc/login.defs getent passwd {1000..60000} #list all normal users depending on UID_MIN/UID_MAX in /etc/login.defs eval getent passwd {$(awk '/^UID_MIN/ {print $2}' /etc/login.defs)..$(awk '/^UID_MAX/ {print $2}' /etc/login.defs)} | cut -d: -f1 # generic,UID_MIN and UID_MIN values may be different, eval getent passwd {$(awk '/^UID_MIN/ {print $2}' /etc/login.defs)..$(awk '/^UID_MAX/ {print $2}' /etc/login.defs)} awk -F ":" '{print $5}' /etc/passwd #print the fifth field getent passwd $UID| awk -F ":" '{print $5}' GECOS fields (which stands for "General Electric Comprehensive Operating System") username:password:userid:groupid:gecos:home-dir:shell GECOS are divided as: :FullName,RoomAddress,WorkPhone,HomePhone,Others: sally:x:0:529:Sally Jones:/home/myhome:/bin/passwd #might be used on, a Samba fle server or a POP mail server to enable users to change their passwords via SSH without granting login shell access. ------------------------------------------------------------------------------------------ # enable the root account sudo passwd root ------------------------------------------------------------------------------------------ # send an email from command line mail -s “Hello world” you@youremailid.com echo “This will go into the body of the mail.” | mail -s “Hello world” you@youremailid.com df -h | mail -s “disk space report” calvin@cnh.com ------------------------------------------------------------------------------------------ # check a file system for errors? fsck fsck.ext3 fsck.nfs fsck.ext2 fsck.vfat fsck.reiserfs fsck.msdos In order to run fsck on the root partition, the root partition must be mounted as readonly ------------------------------------------------------------------------------------------ # list of drives that are mounted at boot /etc/fstab # runs as a daemon and typically has PID 1 # change the default runlevel upon boot up. /etc/inittab chkconfig --list #list of all runlevels and services used by them chkconfig vnicen.sh off #Ensure that the vnicen does not start upon reboot chkconfig --list | grep vnicen chkconfig --level 2345 sshd on #enable sshd startup syslogd # daemon is responsible for tracking events on the system # set which window man-ager you want to use when logging in to X from that account # edit in your home directory to change which window manager you want to use ~/.xinitrc ------------------------------------------------------------------------------------------ find the number of processing units (CPU) available on a system nproc nproc --all echo "Threads/core: $(nproc --all)" lscpu #the number of physical CPU cores lscpu | egrep 'Model name|Socket|Thread|NUMA|CPU\(s\)' lscpu -p grep 'model name' /proc/cpuinfo | wc -l grep 'cpu cores' /proc/cpuinfo | uniq echo "CPU threads: $(grep -c processor /proc/cpuinfo)" cat /proc/cpuinfo grep -c ^processor /proc/cpuinfo cat /proc/cpuinfo | grep 'core id' #get the actual number of cores getconf _NPROCESSORS_ONLN && echo "Number of CPU/cores online at $HOSTNAME: $(getconf _NPROCESSORS_ONLN)" ------------------------------------------------------------------------------------------ $ seq -s";" -w 1 12 01;02;03;04;05;06;07;08;09;10;11;12 # write dummy lines into a file $ seq -s ' ' 23 > file && cat file # read dummy lines from a file $ awk '(NR % 6 == 1) {print; for(i=1; i<6 && getline ; i++) { print }; printf "\n"}' RS=' ' ORS=' ' file 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 # Create a dummy file echo -e "1\n2\n3\n4" > testfile.txt $ echo "3997e1" > ids.txt #write $ echo "45697676107" >> ids.txt #append ------------------------------------------------------------------------------------------ protection from inadvertently overwriting files when copying ~/.bashrc alias cp='cp -i' ------------------------------------------------------------------------------------------ # list tree structure of files and folders $ ls -R .: create_folders_files.sh folder_1 folder_2 folder_3 ./folder_1: file_1.txt file_2.txt file_3.txt file_4.txt ./folder_2: file_1.txt file_2.txt file_3.txt file_4.txt ./folder_3: file_1.txt file_2.txt file_3.txt file_4.txt $ find . . ./folder_2 ./folder_2/file_1.txt ./folder_2/file_2.txt ./folder_2/file_3.txt ./folder_2/file_4.txt ./create_folders_files.sh ./folder_1 ./folder_1/file_1.txt ./folder_1/file_2.txt ./folder_1/file_3.txt ./folder_1/file_4.txt ./folder_3 ./folder_3/file_1.txt ./folder_3/file_2.txt ./folder_3/file_3.txt ./folder_3/file_4.txt $ sudo tree -d /var/log/ --du -sch /var/log/ ├── [4.0K] dist-upgrade ├── [4.0K] fsck ├── [4.0K] lxd ├── [4.0K] apt └── [4.0K] unattended-upgrades $ sudo tree /var/log/ --du -h $ sudo tree -a /var/log #display hidden files $ tree -daC $ tree -f #view the full path for each directory and file $ sudo tree -f -L 3 $ sudo tree -f -P cata* #only list files that match cata*, so files such as Catalina.sh, catalina.bat, etc $ tree -P "*.log" $ sudo tree -f -I *log /var/log #-I option,display all the files that do not match the specified pattern $ sudo tree -d -I *log /var/log $ tree -I "*.log" $ sudo tree -f --prune #prune empty directories from the output $ sudo tree -f -p #-p which prints the file type and permissions for each file $ sudo tree -f -pug #print the username,the group name $ sudo tree -f -pugs #print the size of each file in bytes along with the name using the -s option $ sudo tree -f -pugh #human-readable format, use the -h flag $ sudo tree -f -pug -h -D #display the date of the last modification time for each sub-directory or file $ tree -d -L 3 # the depth of directory tree in output tree -vr #sort the files from Z-A $ tree -L 2 tree -J #the output is in JSON format $ sudo tree -o direc_tree.txt ------------------------------------------------------------------------------------------ ipcs (InterProcess Communication System) provides a report on the semaphore, shared memory & message queue ipcs -u ipcs -m ------------------------------------------------------------------------------------------ nslookup github.com nslookup 140.82.118.4 nslookup -query=mx github.com nslookup -query=ns github.com nslookup -query=any github.com nslookup -query=soa github.com nslookup -query=soa port=54 github.com nslookup -debug github.com ------------------------------------------------------------------------------------------ The command shell interprets the && as the logical AND.the second command will be executed only when the first one has been succcefully executed A double ampersand && in Bash means AND and can be used to separate a list of commands to be run sequentially. Commands separated by a double ampersand && are to be run synchronously, with each one running only if the last did not fail (a fail is interpreted as returning a non-zero return status). && AND – execute both, return true of both succeed ; sequential execution, return status is that of the last in the list $ mkdir /workspace ; mkdir /entrypoint mkdir: cannot create directory ‘/workspace’: Permission denied mkdir: cannot create directory ‘/entrypoint’: Permission denied $ mkdir /workspace && mkdir /entrypoint mkdir: cannot create directory ‘/workspace’: Permission denied ------------------------------------------------------------------------------------------ mkdir -p first/second/third #If the first and second directories do not exist, mkdir creates these directories mkdir -m a=rwx first/second/third #set the file modes, i.e. permissions, etc ------------------------------------------------------------------------------------------ disable and stop service $ sudo systemctl disable --now zabbix-server.service enable and start service $ sudo systemctl enable --now zabbix-server.service # enable and start the service chkconfig tgtd on #configure it to start Automatically while system start-up chkconfig --list tgtd #verify that the run level configured correctly for the tgtd service chkconfig --list #shows SysV services only and does not include native systemd services. chkconfig | grep snmpd systemctl list-units --all systemctl list-unit-files systemctl list-units --all --state=inactive systemctl list-units --type=service #only active service units systemctl cat sshd.service #Displaying a Unit File sudo systemctl edit nginx.service sudo systemctl edit --full nginx.service sudo systemctl daemon-reload systemctl list-unit-files --type=target systemctl list-units --type=target systemctl list-dependencies sshd.service #Displaying Dependencies systemctl show sshd.service #Checking Unit Properties systemctl show sshd.service -p Conflicts #display a single property,pass the -p flag with the property name sudo systemctl mask nginx.service #the ability to mark a unit as completely unstartable, automatically or manually, by linking it to /dev/null. This is called masking the unit ------------------------------------------------------------------------------------------ ls -laZ ~/.ssh # change the security context to system_u:object_r:usr_t:s0 chcon -R -v system_u:object_r:usr_t:s0 ~/.ssh/ ------------------------------------------------------------------------------------------ problem: never edit directly '/etc/sudoers' file $ sudo visudo >>> /etc/sudoers: syntax error near line 28 <<< sudo: parse error in /etc/sudoers near line 28 sudo: no valid sudoers sources found, quitting fix: pkexec visudo #includedir /etc/sudoers -> #includedir /etc/sudoers.d #change last line sudo mount /dev/sda1 /mnt #mount the installed Ubuntu system's root filesystem sudo visudo -f /mnt/etc/sudoers #edit the installed system's sudoers file pkexec visudo -f /etc/sudoers.d/filename #edit configuration files ------------------------------------------------------------------------------------------ problem: sleep: invalid time interval `2\r' fix: sudo cat test.sh | sudo tr -d '\r' | sudo tee test2.sh ------------------------------------------------------------------------------------------ $ diff 1.txt 2.txt # display the differences in the files by comparing the files line by line $ diff -c 1.txt 2.txt $ diff 1.txt 2.txt -u $ diff 1.txt 2.txt -i $ diff 1.txt 2.txt --color $ diff 1.txt 2.txt -s $diff -i test_file_1.txt test_file_2.txt #ignoring the case sensitivity $diff -y -W 60 test_file_1.txt test_file_2.txt #view the difference,The “-W” indicates the width between the content of two files $diff -q test_file_1.txt test_file_2.txt #“-q” option with the “diff” command gives you output in one line $diff -u test_file_1.txt test_file_2.txt # #write the difference between two files into a file diff a.txt b.txt|grep ">"|cut -c 3- > foo.txt #-q, --brief report only when files differ #-s, --report-identical-files report when two files are the same diff -sq /tmp/file1 /tmp/file2 diff chap1.back chap1 #compare two files #If two lines differ only in the number of spaces and tabs between words, the diff -w command considers them to be the same. #compare two files compare two files while ignoring differences in the amount of white space diff -w prog.c.bak prog.c #create a file containing commands that the ed command can use to reconstruct one file from another #creates a file named new.to.old.ed that contains the ed subcommands to change chap2 #back into the version of the text found in chap2.old diff -e chap2 chap2.old >new.to.old.ed # in parentheses add 1,$p to the end of the editing commands sent to the ed editor. #The 1,$p causes the ed command to write the file to standard output after editing it #then piped to the ed command (| ed), and the editor reads it as standard input #The - flag causes the ed command not to display the file size and other extra information #because it would be mixed with the text of chap2.old. (cat new.to.old.ed ; echo '1,$p') | ed - chap2 >chap2.old #compare two text files containing UTF-8 characters and show the differences diff -W filecodeset=UTF-8,pgmcodeset=IBM-1047 myUtf8File01 myUtf8File02 #compare two text files containing EBCDIC characters and show the differences diff -B myMisTaggedFile01 myMisTaggedFile02 ------------------------------------------------------------------------------------------ Local to Remote: rsync [OPTION]... -e ssh [SRC]... [USER@]HOST:DEST Remote to Local: rsync [OPTION]... -e ssh [USER@]HOST:SRC... [DEST] -v, –verbose – Using this option in the rsync command gives the status about transferred files. -vv – Usually, this option is used to get information about the skipped files during file transfer. -q, –quiet – This option simply suppress non-error messages. rsync -av --rsync-path="rsync --log-file=/tmp/rlog" source/ destination/ #enable error log for rsync #rsync exits with a non-zero code when the transfer fails,write details to log files. rsync -avz -e ssh root@example.com:/ /mybackup/ > /var/log/rsync.log 2>&1 Problem: rsync: failed to set times on "some_dir: Operation not permitted (1) mkstemp "some_file" failed: Permission denied (13) Fix: if the user is 'abc' then the destination directory should be lrwxrwxrwx 1 abc abc 34 Jul 18 14:05 Destination_directory chown -R abc:abc Destination_directory rsync -aEim --delete /path/to/remote/ /path/to/local/ # rsync output to stdout with the -i flag #Only files that rsync has fully successfully transferred are removed. rsync -r -z -c --remove-source-files /home/pi/queue root@server.mine.com:/home/foobar rsync -avz source destination #preserve permissions, ownership, and timestamp #When the trailing slash "/" is omitted the source directory will be copied inside the destination directory #transfer the local directory to the directory on a remote machine $ rsync -avz -e "ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null" --progress /home/filerunner/dir1 vg-ubuntu-02:/tmp $ ls /tmp/dir1 a.txt #When the source directory has a trailing slash "/", rsync will copy only the contents of the source directory to the destination directory $ rsync -avz -e "ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null" --progress /home/filerunner/dir1/ vg-ubuntu-02:/tmp $ ls /tmp a.txt #dry run mode, rsync -azhv -e "ssh -p 2212" --dry-run /home/bob/test_219 #find out if the files are in sync, without actually doing a sync dry run mode #-c, --checksum - skip based on checksum, not mod-time & size #-r, --recursive - recurse into directories #-n, --dry-run - perform a trial run with no changes made #does not show anything if server2 has more files then server1 rsync -n -avrc /abc/home/sample1/* server2:/abc/home/sample2/ #dry-run mode verify rsync -avzrch --progress --exclude=lost+found/ -e "ssh -i /home/vagrant/privatekey" --delete user@server:/mnt/files/ /mnt/disk1/ > /tmp/rsync_out 2>&1 #find out if the files are in sync, without actually doing a sync dry run mode #--delete is needed to show if a file exists on server 2 but not server 1 rsync -n -avr --size-only --delete /abc/home/sample1/ server2:/abc/home/sample2/ #find out if the files are in sync, without actually doing a sync dry run mode #Without --dry-run, it will automatically copy files of different sizes #if the sizes are identical, checksum them and copy if different #The delete option will remove files from the target if not present on the source rsync -cr --delete --dry-run source/ target/ > output_file 2>&1 & #--size-only skip files that match in size, no checksum rsync -n -avr --size-only --delete /abc/home/sample1/ server2:/abc/home/sample2/ rsync -a -e "ssh -p 3322" /home/linuxize/images/ user@12.12.12.12:/var/www/images/ #if SSH is listening on port 3322 #transfer a single file /opt/file.zip from the local system to the /var/www/ directory on the remote system with IP 12.12.12.12 #If the file exists on the remote server it is overwritten rsync -a /opt/file.zip user@12.12.12.12:/var/www/ #save the file under a different name rsync -a /opt/file.zip user@12.12.12.12:/var/www/file2.zip #transfer data from a remote to a local machine rsync -a user@12.12.12.12:/var/www/file.zip /opt/ #synchronize the local and remote directory rsync -a /home/linuxize/images/ user@12.12.12.12:/var/www/images/ #use the --delete option if you want to synchronize the local and remote directory #delete files in the destination directory if they don’t exist in the source directory. rsync -a --delete /home/linuxize/images/ user@12.12.12.12:/var/www/images/ #the “-r” option for “recursive” and the “-a” option for “all” (otherwise non-regular files will be skipped) #copy the “/etc” folder to the “/etc_backup” of the remote server #with the “devconnected” username to server 192.168.178.35/24 rsync -ar /etc devconnected@192.168.178.35:/etc_backup #Similarly,copy the content of the “/etc/ directory rather than the directory itself rsync -ar /etc/* devconnected@192.168.178.35:/etc_backup/ # taggged with the current date rsync -ar /etc/* devconnected@192.168.178.35:/etc_backup/etc_$(date "+%F") #from local to remote server with private key rsync -auvz -e "ssh -i private-key-file" source destination #Using rsync With SSH and Private Key rsync -auvz -e "ssh -i /home/yourUserName/.ssh/yourUserName-rsync-key" junk.txt yourUserName@calypso.nrel.colostate.edu rsync -avzhe ssh backup.tar.gz root@192.168.0.141:/backups/ rsync -avzhe ssh --progress /root/rpmpkgs root@192.168.0.141:/root/rpmpkgs #from remote to local server rsync -avzh root@192.168.0.141:/root/rpmpkgs /tmp/myrpms rsync -avze ssh --include 'R*' --exclude '*' root@192.168.0.141:/var/lib/rpm/ /root/rpm #exclude lost+found dir rsync --archive --no-compress --delete-before --info=progress2 --human-readable --exclude=lost+found/ /mnt/backup/ /mnt/backup-2/ #run rsycn on the background rsync -avze ssh --include 'R*' --exclude '*' root@192.168.0.141:/var/lib/rpm/ /root/rpm > rsync.out 2>&1 & tail -f rsync.out ------------------------------------------------------------------------------------------ #copy the “/etc” directory to a backup server located at 192.168.178.35 in the “/etc_backup” folder scp -r /etc devconnected@192.168.178.35:/etc_backup/ # taggged with the current date scp -r /etc devconnected@192.168.178.35:/etc_backup/etc_$(date "+%F") scp your_username@remotehost.edu:foobar.txt /some/local/directory-> Copy the file "foobar.txt" from a remote host to the local host scp file.txt remote_username@10.10.0.2:/remote/directory/newfilename.txt # save the file under a different name,Omitting the filename from the destination location copies the file with the original name. scp foobar.txt your_username@remotehost.edu:/some/remote/directory -> Copy the file "foobar.txt" from the local host to a remote host scp your_username@rh1.edu:/some/remote/directory/foobar.txt your_username@rh2.edu:/some/remote/directory/ ->Copy the file "foobar.txt" from remote host "rh1.edu" to remote host "rh2.edu" scp -P 2322 file.txt remote_username@10.10.0.2:/remote/directory #the remote host is listening on a port other than the default 22 scp -r /local/directory remote_username@10.10.0.2:/remote/directory #copy a directory from a local to remote system,use the -r flag for recursive # don’t have to log in to one of the servers to transfer files from one to another remote machine. #copy the file /files/file.txt from the remote host host1.com to the directory /files on the remote host host2.com scp user1@host1.com:/files/file.txt user2@host2.com:/files scp -3 user1@host1.com:/files/file.txt user2@host2.com:/files #route the traffic through the local host (machine on which the command is issued), use the -3 option ------------------------------------------------------------------------------------------ #cp interpret main as a directory to place scala,doesn't exist, cp will throw an error. cp -av /home/jake/transit/scalaProjects/scalaML/src/main/scala -t /home/jake/project/__workspace/scalaProjects/scalaML/src/main/ #Copy Directory Content Recursively cp -R bashdir bashdir-bck cp -R /etc/* /etc_backup cp -R /etc/* /home/* /backup_folder # copy the “/etc” directory and “/home” directory. #copies the folder Misc and all its contents (the -r, or "recursive," option indicates the contents as well as the folder or file itself) into the folder /media/clh/4388-D5FE cp -r Misc /media/clh/4388-D5FE # #copy over only the new files,use the "update" and "verbose" options cp -ruv Misc /media/clh/4388-D5FE # a file called test1.py, which is the original, and another called test1.py.~1~, which is the backup file. cp --force --backup=numbered test1.py test1.py cp /home/usr/dir/{file1,file2,file3,file4} /home/usr/destination/ #copy multiple files cp -rp /copying/from/{folder1/,folder2/,folder3/} path/to/folder #p is for copying the folder permission cp /home/usr/dir/file{1..4} /tmp #if the all the files have the same prefix but different endings ------------------------------------------------------------------------------------------ #brace expansions $ sudo cp ~/a.txt{,.orig} #backup of file in the current directory, copy shortcut $ ls a.txt a.txt.orig $ echo g{et,ot,it}em getem gotem gitem $ echo {00..8..2} #loop with increase 2 00 02 04 06 08 $ echo {D..T..4} #loop with increase 4 D H L P T $ mv error.log{,.OLD} #expands to "mv error.log error.log.OLD") ------------------------------------------------------------------------------------------ #rename multiple files to change the extensions $ mv *.txt *.tsv mv: target 'd.tsv': Not a directory #rename all the files with a .txt extension to .fg extension in the given directory find . -type f -name "*.txt" -exec sh -c 'mv "$1" "${1%.txt}.fg"' _ {} \; #bash script for file in /path/to/directory/*.txt; do mv "$file" "${file%.txt}.fg" done # one liner version of the bash script above find /path/to/directory -type f -name "*.txt" -exec bash -c 'mv "$0" "${0%.txt}.fg"' {} \; ------------------------------------------------------------------------------------------ cmp file1.txt file2.txt #cmp command reports the byte and line number if a difference is found cmp --silent file1 file2 && echo 'SUCCESS: Files Are Identical' || echo 'Files Are Different' cmp --silent $old $new || echo "files are different" #if= defines the source drive and of= defines the file or location where data saved # dd if=/dev/sda of=/dev/sdb # dd if=/dev/sda of=/home/username/sdadisk.img #create an .img archive of the /dev/sda drive and save it to the home directory # dd if=/dev/sda2 of=/home/username/partition2.img bs=4096 #if= takes the image to restore, and of= takes the target write the image # dd if=sdadisk.img of=/dev/sdb create a compressed image of a remote drive using SSH and save the resulting archive to local machine # ssh username@54.98.132.10 "dd if=/dev/sda | gzip -1 -" | dd of=backup.gz dd if=text.ascii of=text.ebcdic conv=ebcdic #convert an ASCII text file to EBCDIC #convert the variable-length record ASCII file /etc/passwd to a file of 132-byte fixed-length EBCDIC records dd if=/etc/passwd cbs=132 conv=ebcdic of=/tmp/passwd.ebcdic #convert the 132-byte-per-record EBCDIC file to variable-length ASCII lines in lowercase dd if=/tmp/passwd.ebcdic cbs=132 conv=ascii of=/tmp/passwd.ascii #copy blocks from a tape with 1KB blocks to another tape using 2KB blocks dd if=/dev/rmt0 ibs=1024 obs=2048 of=/dev/rmt1 ls -l | dd conv=ucase #displays a long listing of the current directory in uppercase. dd if=/dev/zero of=/dev/sda1 #Wiping disks with dd,writing zeros # dd if=/dev/urandom of=/dev/sda1 #Wiping disks with dd,writing random characters # dd if=/dev/urandom | pv | dd of=/dev/sda1 #Monitoring dd operations,Pipe Viewer (sudo apt install pv on Ubuntu) #create a large file of random conten $ dd if=/dev/urandom of=/tmp/file1 count=1K bs=1MB #copy file1 to file2, and append different characters to each file $ cp /tmp/file1 /tmp/file2 $ echo 1 >> /tmp/file1 $ echo 2 >> /tmp/file2 #use the time command to measure the time taken $ time cmp -s /tmp/file1 /tmp/file2 $ time diff -sq /tmp/file1 /tmp/file2 $ time sha1sum /tmp/file1 $ time sha1sum /tmp/file2 #Empty File Content by Redirecting to Null # > access.log # : > access.log #: is a shell built-in command # true > access.log # cat /dev/null > access.log # cp /dev/null access.log # dd if=/dev/null of=access.log # echo "" > access.log # echo > access.log # echo -n "" > access.log #use the flag -n which tells echo to not output the trailing newline # truncate -s 0 access.log dd if=/dev/urandom of=test.file bs=1M count=100 ; time diff -q test.file test.copy && echo diff true || echo diff false ; \ time cmp -s test.file test.copy && echo cmp true || echo vagrant@lampstack-01:~$ echo "file" > file1.txt vagrant@lampstack-01:~$ cp file1.txt file2.txt vagrant@lampstack-01:~$ cmp file1.txt file2.txt #cmp command reports the byte and line number if a difference is found vagrant@lampstack-01:~$ sudo cmp file1.txt file2.txt vagrant@lampstack-01:~$ echo "identical file1" >> file1.txt vagrant@lampstack-01:~$ cat file1.txt file identical file1 vagrant@lampstack-01:~$ cat file2.txt file vagrant@lampstack-01:~$ cmp file1.txt file2.txt #cmp command reports the byte and line number if a difference is found cmp: EOF on file2.txt after byte 5, line 1 vagrant@lampstack-01:~$ diff file1.txt file2.txt 2d1 < identical file1 ------------------------- ----------------------------------------------------------------- # approval trick $ yes | sudo yum install puppet # user has to type 'y' for each query $ yes | rm -ri test ------------------------- ----------------------------------------------------------------- #oracle java download wget --no-cookies \ --no-check-certificate \ --header "Cookie: oraclelicense=accept-securebackup-cookie" \ "https://download.oracle.com/otn-pub/java/jdk/13.0.1+9/cec27d702aa74d5a8630c65ae61e4305/jdk-13.0.1_linux-x64_bin.tar.gz" \ -O jdk-7-linux-x64.tar.gz curl --silent example.com | sha256sum curl --silent --output - example.com | sha256sum #the o flag is redundant since the output is piped to bash (for execution) - not to a file #-L in case the page has moved curl will redirect the request to the new address #-o output to a file instead of stdout (usually the screen) curl -LO http://example.com/ curl -LO -H "Cookie: oraclelicense=accept-securebackup-cookie" \ https://download.oracle.com/otn-pub/java/jdk/13.0.1+9/cec27d702aa74d5a8630c65ae61e4305/jdk-13.0.1_linux-x64_bin.tar.gz ------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------ hostnamectl set-hostname vg-checkmk-client echo "172.28.128.15 vg-checkmk-client.local vg-checkmk-client" |sudo tee -a /etc/hosts echo "nameserver 8.8.8.8" |sudo tee -a /etc/resolv.conf problem: #append text to a file when using sudo sudo echo "172.28.128.15 vg-checkmk-client.local vg-checkmk-client" >> /etc/hosts -bash: /etc/hosts: Permission denied fix: #running bash/sh shell with root privileges and redirection took place in that shell session $ sudo sh -c 'echo "172.28.128.15 vg-checkmk-client.local vg-checkmk-client" >> /etc/hosts' $ sudo sh -c 'echo "JAVA_HOME=/usr/lib/jvm/java-6-sun" >> /etc/profile' $ sudo bash -c 'echo "JAVA_HOME=/usr/lib/jvm/java-6-sun" >> /etc/profile' $ sudo tee -a /etc/profile.d/java.sh << 'EOF' # configures JAVA JAVA_HOME=/usr/lib/jvm/java-8-oracle export JAVA_HOME export PATH=$PATH:$JAVA_HOME/bin EOF $ sudo sed -i '$a something' /etc/config_file ------------------------------------------------------------------------------------------ #File Creation Times vagrant@lampstack-01:/tmp/nexus$ df -h Filesystem Size Used Avail Use% Mounted on udev 205M 0 205M 0% /dev tmpfs 48M 7.8M 41M 17% /run /dev/mapper/vagrant--vg-root 62G 4.3G 55G 8% / tmpfs 240M 0 240M 0% /dev/shm tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs 240M 0 240M 0% /sys/fs/cgroup vagrant 420G 375G 46G 90% /vagrant tmpfs 48M 0 48M 0% /run/user/1000 vagrant@lampstack-01:/tmp/nexus$ ls -i Dockerfile 3808483 Dockerfile vagrant@lampstack-01:/tmp/nexus$ sudo debugfs -R 'stat <3808483>' /dev/mapper/vagrant--vg-root debugfs 1.44.6 (5-Mar-2019) vagrant@lampstack-01:/tmp/nexus$ vagrant@lampstack-01:/tmp/nexus$ sudo debugfs -R 'stat <3808483>' /dev/mapper/vagrant--vg-root | grep crtime debugfs 1.44.6 (5-Mar-2019) crtime: 0x5e4f0f92:b6a11330 -- Thu Feb 20 23:00:34 2020 ------------------------------------------------------------------------------------------ #how to view gz gunzip files zcat test2.txt.gz #see the content of the compressed file zcat test.txt.gz test2.txt.gz #see the content of the compressed files,multiple inputs zcat test2.txt #uncompress files that have the correct magic number whether they have a .gz suffix or not zcat test2.txt.gz | more zcat test2.txt.gz | less zmore test2.txt.gz zless test2.txt.gz zless -f test2.txt #display file contents in output whether or not the file is compressed # search string in multiple .gz file zgrep -i -H "pattern match" /somedir/filename*.gz zgrep -i -H "pattern match" /somedir/filename*.gz | grep "pattern 2nd" find /somedir/ -name "log.202011917*.gz" -exec zgrep "somestring" \{\} \; find /somedir/ -name 'log.202011917' -print0 | xargs -0 zgrep "somestring" #prints file path inc. folder ------------------------------------------------------------------------------------------ #Hosts File Windows 10 - "C:\Windows\System32\drivers\etc\hosts" Linux - "/etc/hosts" Mac OS X - "/private/etc/hosts" ------------------------------------------------------------------------------------------ type -a ll #if command is an alias or not type -p dash ------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------ echo $? #get the exit status of the previously executed command, http://www.tldp.org/LDP/abs/html/exitcodes.html echo $? # Expands to the exit status (exit code) of the most recently executed foreground pipeline,return the exit status of last command exits with a status code 0 #success exits with a status code 1 #failure command && echo "success: $?" || echo "fail: $?" #test if the command failed if [ $? -eq 0 ]; then echo "success: $?"; fi $ echo "hola el mundo" > file.txt $ cat file.txt && echo "success: $?" || echo "fail: $?" hola el mundo success: 0 $ cat filedoesnotexist.txt && echo "success: $?" || echo "fail: $?" cat: filedoesnotexist.txt: No such file or directory fail: 1 cat 'doesnotexist.txt' 2>/dev/null || exit 0 #suppress exit status cat file.txt || exit 0 #suppress the error silently $ cat filedoesnotexist.txt && echo "success: $?" || echo "fail: $?" cat: filedoesnotexist.txt: No such file or directory fail: 1 $ cat filedoesnotexist.txt || true && echo $? cat: filedoesnotexist.txt: No such file or directory 0 $ fslint /tmp #lists the duplicate files $ rdfind /tmp #delete the duplicates,remove the newer files. $ rdfind -dryrun true /tmp #only report the changes $ rdfind -deleteduplicates true /tmp $ cksum *.html #computes checksums for files $ find . -name "*.html" -exec cksum {} \; #search files by name or type and run the cksum command. $ find . -name "not_existing_file" $ echo $? ----------------------------------------------------------------------------------------------------- bash -x # runs the script with tracing of each command executed bash -x -c ls -lai #run a command in BASH, use -c option test -x #tests whether has execute permissions for the current user ------------------------------------------------------------------------------------------ #print variable export BRANCH_NAME="main" echo "BRANCH_NAME is..: $BRANCH_NAME" ------------------------------------------------------------------------------------------ yum/apt install chrony systemctl stop chronyd chronyd -q 'pool pool.ntp.org iburst' systemctl start chronyd chronyc tracking #verify systemctl restart chronyd ; watch chronyc tracking #realtime witnessing chronyc sources chronyc sources -v chronyc ------------------------------------------------------------------------------------------ #concatenate strings export strservice="libvirtd" echo "${strservice} is still running." echo $(ls) echo "The date is $(date)" echo `pwd` echo $((3 + (4**3 /2))) #Direct calculation in the shell with echo and $(( ) ------------------------------------------------------------------------------------------ #nfs server exportfs -arv df -h cat /etc/exports # /mnt/nfs_share client_IP_1 (re,sync,no_subtree_check) echo "$NFS_DIR 192.168.50.7(rw,sync,no_subtree_check)" | sudo tee -a /etc/exports echo "$NFS_DIR 192.168.50.8(rw,sync,no_subtree_check)" | sudo tee -a /etc/exports #nfs client mount -t nfs NFS_SERVER_IP:NFS_SERVER_DIR NFS_CLIENT_DIR mount -t nfs 10.20.20.8:/mnt/backups /mnt/backups umount /mnt/backups mount | grep nfs showmount -e ------------------------------------------------------------------------------------------ #execute shell command produced using echo and sed echo "mv /server/today/logfile1 /nfs/logs/ && gzip /nfs/logs/logfile1" | bash # bash -c "$(echo "mv /server/today/logfile1 /nfs/logs/ && gzip /nfs/logs/logfile1")" #pass it as an argument to a shell eval "$(echo "mv /server/today/logfile1 /nfs/logs/ && gzip /nfs/logs/logfile1")" #use the bash built-in eval ------------------------------------------------------------------------------------------ $ mycommand="wc -l department.txt" $ eval $mycommand ------------------------------------------------------------------------------------------ $ cat /proc/sys/net/ipv4/ip_local_port_range ------------------------------------------------------------------------------------------ #redirection is done by the shell which doesn't has write permission sudo echo 1 > /proc/sys/net/ipv4/ip_forward bash: /proc/sys/net/ipv4/ip_forward: Permission denied echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward #resolution-1 $ sudo sh -c 'echo 1 > /proc/sys/net/ipv4/ip_forward' #resolution-2 $ sudo -s #resoluton-3 # echo 1 > /proc/sys/net/ipv4/ip_forward $ cat testscript.sh #resolution-4 #!/bin/sh echo 1 > /proc/sys/net/ipv4/ip_forward sudo echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward > /dev/null #same $ sudo tee /proc/sys/net/ipv4/ip_forward > /dev/null << EOF #same 1 EOF ------------------------------------------------------------------------------------------ #source tar build, no version typing tar zfx rrdtool.tar.gz cd rrdtool-* ------------------------------------------------------------------------------------------ echo 1 > /proc/sys/kernel/sysrq;echo f > /proc/sysrq-trigger;echo 0 > /proc/sys/kernel/sysrq #Trigger Out Of Memory (OOM) killer without reboot ------------------------------------------------------------------------------------------ ls -la .mozilla/firefox #user preferences,browser cookies,bookmarks,browser cache content etc. ------------------------------------------------------------------------------------------ #troubleshooting memory problems #Suddenly killed tasks are often the result of the system running out of memory, when the so-called Out-of-memory (OOM) killer steps in $ grep -i -r 'out of memory' /var/log/ #search the logs for messages of out of memory alerts $ free -m #megabytes $ free -m |head -n 2 |tail -n 1 |awk '{free=($4); print free}' watch -n 5 -d '/bin/free -m' $ top $ top -i -u vagrant #ignore idle processes $ top -n 1 -o RES | grep kvm $ uptime #"system load averages" that show the running thread (task) demand on the system as an average number of running plus waiting threads. #show three averages, for 1, 5, and 15 minutes #If the averages are 0.0, system is idle #If the 1 minute average is higher than the 5 or 15 minute averages, then load is increasing #If the 1 minute average is lower than the 5 or 15 minute averages, then load is decreasing. $ ps -aylC $APACHE |grep "$APACHE" |awk '{print $8'} |sort -n |tail -n 1 $ ps -eL h -o state | egrep "R|D" | wc -l #The instantaneous number of such tasks #Linux load average,the instantaneous load of a system the number of tasks (processes and threads) #that are willing to run at a given time t #either in state R or D, either actually running or blocked on some resource (CPU, IO, ...) waiting for an opportunity to run $ cat /proc/sys/vm/swappiness #The Linux kernel moves out pages which are not active or being used at the moment to swap space on the disk. This process is known as swappiness. #turn off swaping by changing t he value in /proc/sys/vm/swappiness to 0.The value ranges from 0 to 100 whe re 100 means aggressive swapping vmstat –a 1 99 #show memory usage information vmstat -n 1 # If 'si' and 'so' (stands for swapin and swapout) fields are always 0, then the system is currently not swapping $ grep DirectMap /proc/meminfo $ cat /proc/meminfo #Relevant fields from /proc/meminfo to match them against the output of free -k MemTotal — Total amount of physical RAM, in kilobytes. MemFree — The amount of physical RAM, in kilobytes, left unused by the system. Buffers — The amount of physical RAM, in kilobytes, used for file buffers. Cached — The amount of physical RAM, in kilobytes, used as cache memory. SwapCached — The amount of swap, in kilobytes, used as cache memory. Active — The total amount of buffer or page cache memory, in kilobytes, that is in active use. This is memory that has been recently used and is usually not reclaimed for other purposes. Inactive — The total amount of buffer or page cache memory, in kilobytes, that are free and available. This is memory that has not been recently used and can be reclaimed for other purposes. HighTotal and HighFree — The total and free amount of memory, in kilobytes, that is not directly mapped into kernel space. The HighTotal value can vary based on the type of kernel used. LowTotal and LowFree — The total and free amount of memory, in kilobytes, that is directly mapped into kernel space. The LowTotal value can vary based on the type of kernel used. SwapTotal — The total amount of swap available, in kilobytes. SwapFree — The total amount of swap free, in kilobytes. Dirty — The total amount of memory, in kilobytes, waiting to be written back to the disk. Writeback — The total amount of memory, in kilobytes, actively being written back to the disk. Mapped — The total amount of memory, in kilobytes, which have been used to map devices, files, or libraries using the mmap command. Slab — The total amount of memory, in kilobytes, used by the kernel to cache data structures for its own use. Committed_AS — The total amount of memory, in kilobytes, estimated to complete the workload. This value represents the worst case scenario value, and also includes swap memory. PageTables — The total amount of memory, in kilobytes, dedicated to the lowest page table level. VMallocTotal — The total amount of memory, in kilobytes, of total allocated virtual address space. VMallocUsed — The total amount of memory, in kilobytes, of used virtual address space. VMallocChunk — The largest contiguous block of memory, in kilobytes, of available virtual address space. HugePages_Total — The total number of hugepages for the system. The number is derived by dividing Hugepagesize by the megabytes set aside for hugepages specified in /proc/sys/vm/hugetlb_pool. This statistic only appears on the x86, Itanium, and AMD64 architectures. HugePages_Free — The total number of hugepages available for the system. This statistic only appears on the x86, Itanium, and AMD64 architectures. Hugepagesize — The size for each hugepages unit in kilobytes. By default, the value is 4096 KB on uniprocessor kernels for 32 bit architectures. For SMP, hugemem kernels, and AMD64, the default is 2048 KB. For Itanium architectures, the default is 262144 KB. This statistic only appears on the x86, Itanium, and AMD64 architectures. Matching output of free -k to /proc/meminfo free output coresponding /proc/meminfo fields Mem: total MemTotal Mem: used MemTotal - MemFree Mem: free MemFree Mem: shared (can be ignored nowadays. It has no meaning.) N/A Mem: buffers Buffers Mem: cached Cached -/+ buffers/cache: used MemTotal - (MemFree + Buffers + Cached) -/+ buffers/cache: free MemFree + Buffers + Cached Swap: total SwapTotal Swap: used SwapTotal - SwapFree Swap: free SwapFree ------------------------------------------------------------------------------------------ cat newpass.txt | chpasswd #update passwords in batch mode echo 'ubuntuser:ubuntupassword' | sudo chpasswd $cat > pass.txt user1:user1_password user2:user2_password user3:user3_password $chpasswd < file_name.txt CURRENT_USERS=$(who) #Assign Output of a Linux Command to a Variable ------------------------------------------------------------------------------------------ ls -1 #list one file per line,vertically using the -1 switch ------------------------------------------------------------------------------------------ sudo !! #Re-Run Last Executed Command as Root User, “sudo” followed by a space and two exclamation points. ------------------------------------------------------------------------------------------ #If set, sudo will only run when the user is logged in to a real tty #sudo can only be run from a login session and not via other means such as cron(8) or cgi-bin scripts #enable sudo within scripts sudo /usr/sbin/visudo $ cat /etc/sudoers Defaults:username !requiretty #Disable requiretty to particular user #sudoers file allows both artbristol and bob to execute /path/to/program as root from a script #artbristol needs no password whereas bob must have to enter a password artbristol ALL = (root) NOPASSWD: /path/to/program bob ALL = (root) /path/to/program Defaults!/path/to/program !requiretty #allows artbristol to run /path/to/program --option in a script, but not /path/to/program with other arguments. Cmnd_Alias MYPROGRAM = /path/to/program --option artbristol ALL = (root) /path/to/program artbristol ALL = (root) NOPASSWD: MYPROGRAM Defaults!MYPROGRAM !requiretty ------------------------------------------------------------------------------------------ #This allows the source (the sending host) to specify the route, loosely or strictly, ignoring the routing tables of some or all of the routers #source-based routing should be disabled. /sbin/sysctl -w net.ipv4.conf.all.accept_source_route=0 #drop packets with the SSR or LSR option set #Disabling the forwarding of packets should also be done in conjunction with the above when possible (disabling forwarding may interfere with virtualization) /sbin/sysctl -w net.ipv4.conf.all.forwarding=0 /sbin/sysctl -w net.ipv6.conf.all.forwarding=0 #These commands disable forwarding of all multicast packets on all interfaces /sbin/sysctl -w net.ipv4.conf.all.mc_forwarding=0 /sbin/sysctl -w net.ipv6.conf.all.mc_forwarding=0 #Accepting ICMP redirects has few legitimate uses. Disable the acceptance and sending of ICMP redirected packets unless specifically required /sbin/sysctl -w net.ipv4.conf.all.accept_redirects=0 /sbin/sysctl -w net.ipv6.conf.all.accept_redirects=0 #disables acceptance of secure ICMP redirected packets on all interfaces /sbin/sysctl -w net.ipv4.conf.all.secure_redirects=0 # disables acceptance of all IPv4 ICMP redirected packets on all interfaces: /sbin/sysctl -w net.ipv4.conf.all.send_redirects=0 #automatically disable sending of ICMP requests whenever you add a new interface /sbin/sysctl -w net.ipv4.conf.default.send_redirects=0 #make these settings persistent across reboots, modify the /etc/sysctl.conf file #disable acceptance of all IPv4 ICMP redirected packets on all interfaces, #open the /etc/sysctl.conf file with an editor running as the root user and add a line as follow net.ipv4.conf.all.send_redirects=0 ------------------------------------------------------------------------------------------ #Reverse Path Forwarding is used to prevent packets that arrived through one interface from leaving through a different interface #unless you know that it is required, it is best enabled as it prevents users spoofing IP addresses from local subnets and reduces the opportunity for DDoS attacks. #permanent changes can be made by adding lines to the /etc/sysctl.conf # make a temporary global change sysctl -w net.ipv4.conf.default.rp_filter=integer sysctl -w net.ipv4.conf.all.rp_filter=integer ------------------------------------------------------------------------------------------ #Enabling Packet Forwarding #enable packets arriving from outside of a system to be forwarded to another external host #change the line which reads net.ipv4.ip_forward = 0 in the /etc/sysctl.conf file to the following net.ipv4.ip_forward = 1 /sbin/sysctl -p #load the changes from the /etc/sysctl.conf file /sbin/sysctl net.ipv4.ip_forward #check if IP forwarding is turned on,If it returns 1, then IP forwarding is enabled /sbin/sysctl -w net.ipv4.ip_forward=1 #If it returns 0, turn it on manually ------------------------------------------------------------------------------------------ #check public IP,private (viewable within an internal network) or public (can be seen by other machines on the Internet) #3rd party web-sites $ wget -qO- http://ipecho.net/plain | xargs echo $ curl ifconfig.co $ curl ifconfig.me $ curl icanhazip.com $ curl -4 icanhazip.com $ curl -6 icanhazip.com $ curl ident.me $ curl checkip.dyndns.org $ curl api.ipify.org $ curl ipinfo.io/ip $ curl checkip.amazonaws.com ------------------------------------------------------------------------------------------ #The Internet Assigned Numbers Authority (IANA) has assigned several address ranges to be used by private networks Class A: 10.0.0.0 to 10.255.255.255 Class B: 172.16.0.0 to 172.31.255.255 Class C: 192.168.0.0 to 192.168.255.255 Class A: 10.0.0.0/8 (255.0.0.0) Class B: 172.16.0.0/12 (255.240.0.0) Class C: 192.168.0.0/16 (255.255.0.0) ------------------------------------------------------------------------------------------ #Running GUI applications as root sudo /usr/bin/etherape ------------------------------------------------------------------------------------------ chage -l ubuntu #last password change date ------------------------------------------------------------------------------------------ #read first, and if a match is found the connection is allowed and the search is stopped. #If no allowed match if found, the hosts.deny file is read /etc/hosts.allow #If a match is found the connection is refused - otherwise it is allowed #/etc/hosts.deny ------------------------------------------------------------------------------------------ # print server's IP on the wellcome page echo $(ifconfig eth0 | grep 'inet addr' | awk -F: '{ print $2 }' | awk '{ print $1 }') >> /var/www/html/index.html ------------------------------------------------------------------------------------------ getfacl -a a.txt #file access control list of a file or directory. getfacl -t a.txt getfacl -n file #numeric user and group IDs getfacl -d a.txt #the default access control list of a file or directory. getfacl -R directory # the ACLs of all files and directories recursively (sub-directories) getfacl -L -R directory #follow symbolic links to directories. The default behavior is to follow symbolic link arguments and skip symbolic links encountered in subdirectories getfacl -P -R directory #do not follow symbolic links to directories,skip symbolic link arguments setfacl -m u:deepak:rw a.txt #grant read and write permission setfacl -b a.txt #remove all extended ACL entries,remove all entries setfacl -x u:deepak a.txt #remove user setfacl -x g:linux file #remove group setfacl -m g:linux:rw -R directory #remove group recursively (sub-directories) setfacl -k file #remove the default access control list setfacl --test -x g:linux -R dir1 #The ACLs are not modified in test mode. It only displays the changes that will take place setfacl -dm "user:my_user:r--" /path/to/directory #Add a default entry to grant access to the user my_user on all newly created files within a directory getfacl file1 | setfacl --set-file=- file2 #copy the ACL of one file to another ------------------------------------------------------------------------------------------ for i in *linux*; do rm $i; done #delete all the files in the current directory that contains the word “linux” cat linux.txt | grep n #list the entries that start has the character ‘n’ cat linux.txt | grep ^a #ist the entries that start with the character ‘a’ echo "shutdown now" | at -m 18:00 #shut down the system at 6 pm today vim -R #open a file in read-only mode vim +/ #search for a specific Employee ID in a file ------------------------------------------------------------------------------------------ cd $mydir && python3 -m http.server 8888 #Share a file quickly using a web server podman run --rm -v .:/usr/share/nginx/html:ro,Z -p 30080:80 -d nginx # Expose a web directory using containers ------------------------------------------------------------------------------------------ $ sudo shutdown --halt 22:00 #--halt option stops the operating system $ sudo shutdown --halt +5 # halt the system after a five-minute delay $ sudo shutdown --halt +5 “Attention. The system is going down in five minutes.” $ sudo shutdown -c #Cancel a timed shutdown $ sudo shutdown -r now #restarting the system $ sudo systemctl reboot #restarting the system ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ #how to delete file with inode find . -inum 1847 -ls find . -inum 1847 -exec rm {} \; find . -inum 782263 -exec rm -i {} \; #how to delete directory/folder with inode find . -inum 393232 -delete ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------