monitor.sh script notes

• Use the following script to gather snap-shots of network statistics at intervals, while the networking issue occurs. Please check that enough disk space exists if the scripts are going to be run for a long time.
  • To create the script, copy the lines "#!/bin/bash ... # monitor.sh begins here" through to the end of the line "# monitor.sh ends here" and paste them into a file called monitor.sh. Then chmod +x monitor.sh to allow the script to be invoked.
  • When the test is finished, stop the script then run the following command to compress the data:

tar cvzf net_stats_$(hostname)-$(date +"%Y-%m-%d-%H-%M-%S").tgz *network_stats_*
and upload the compressed *.tgz file onto the case.

information gathering

• Confirm network throughput between systems using iperf3, netperf, or another tool that does not use encryption or generate disk I/O. Try the same tests between a particular system and others on the network.

• if performance is degraded for a specific host, provide the output from iperf3 or netperf

increase the buffer size on both client and server to 4 MiB:
#  sysctl -w net.core.wmem_max=4194304
#  sysctl -w net.core.rmem_max=4194304

execute on the server machine:
# iperf3 -f M -s

execute on the client machine:
# iperf3 -l 1M -w 4M -f M -t 60 -c SERVER-IP

• gather the following before and after information gathering

# netstat -s
# ss -noemitaup
# cat /proc/net/snmp

packet captures

• Perform a packet capture using tcpdump on both ends while the test is under way. Be sure to provide the IP addresses of the machines relative to each pcap.

• example: # tcpdump -s 0 -i {INTERFACE} -w {FILEPATH} [filter expression]

  • sigkill once complete C^+c

pcap paramaters

• -s: is an option for snaplen. It tells tcpdump the amount of information (bytes) it should capture for each packet. Tcpdump from RHEL 6 and newer, captures 65535 bytes of data from each packet by default. On previous versions (RHEL 5 and older) tcpdumpcaptures 68 bytes by default. 68 bytes is often not sufficient to perform any sort of troubleshooting or diagnosis. Please refer to the tcpdump(8) man page and search for "snaplen" if you are unsure what the default is on your system. If you want to capture all the data from network packets, run the tcpdump command with the "-s 0" option as shown above. In the situation where network traffic is huge or there is need for capturing traffic for a prolonged period of time, this might not produce the best result.

When capturing packets, performance is also important; if there is shortage of resources tcpdump may not capture all packets. To tune tcpdump, it is recommended to limit snaplen to the smallest number that will capture all the important protocol information required for later analysis.

The amount of bytes which are going to be needed is issue dependent. For instance if you are troubleshooting NFS v3, a snaplen of 256 is usually sufficient unless you're troubleshooting READDIR/READDIRPLUS issues, in which case you will probably want to use a snaplen of 0. For NFS v4, you will typically use a snaplen of 0.

• For networks that have multiple components such as virtual machines, please reproduce the issue and perform a packet capture at each device between the VM and the remote host (if possible, simultaneously). For example:

   eth0 ---- vnet0 --- br0 --- eth0 ------ switch ------ eth1
[-- VM --][------- hypervisor -------][-- physical --][-- remote --]

On VM:
# tcpdump -i eth0 -w /tmp/vm_$(hostname)-$(date +"%Y-%m-%d-%H-%M-%S").pcap

On Hypervisor:
# tcpdump -i vnet0 -w /tmp/vnet_$(hostname)-$(date +"%Y-%m-%d-%H-%M-%S").pcap
# tcpdump -i br0 -w /tmp/br0_$(hostname)-$(date +"%Y-%m-%d-%H-%M-%S").pcap

On switch:
mirror switch port and perform a packet capture

On remote host:
# tcpdump -i eth1 -w /tmp/eth1_remote.pcap