This will be a quick post to address the VCAP5-DCA objective of using esxtop/resxtop and vscsiStats to identify storage performance issues on an ESXi host environment. I covered both of these topics in other articles in the course of covering the VCAP-DCA blueprint, so rather than cover the same ground again here I will provide links to those articles.
Using ESXTOP to Identify Storage Performance Issues
I’ve covered using esxtop/resxtop to investigate storage performance issues here, and here. As discussed in those articles, some of the metrics to pay attention to are:
- CMDS/s – This is the total amount of commands per second, which includes IOPS and other SCSI commands (e.g. reservations and locks). Generally speaking CMDS/s = IOPS unless there are a lot of other SCSI operations/metadata operations such as reservations.
- DAVG/cmd – This is the average response time in milliseconds per command being sent to the storage device.
- KAVG/cmd – This is the amount of time the command spends in the VMKernel.
- GAVG/cmd – This is the response time as experienced by the Guest OS. This is calculated by adding together the DAVG and the KAVG values.
As a general rule DAVG/cmd, KAVG/cmd and GAVG/cmd should not exceed 10 milliseconds (ms) for sustained lengths of time.
There are also the following throughput metrics to be aware of:
- CMDS/s – As discussed above
- READS/s – Number of read commands issued per second
- WRITES/s – Number of write commands issued per second
- MBREAD/s – Megabytes read per second
- MBWRTN/s – Megabytes written per second
The sum of reads and writes equals IOPS, which is the the most common benchmark when monitoring and troubleshooting storage performance. These metrics can be monitored at the HBA or Virtual Machine level. More information is available in the following VMware articles:
- https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1008205
- https://blogs.vmware.com/kb/2014/02/using-esxtop-identify-storage-performance-issues-vsphere-esxi-environment-multiple-versions.html
Using vscsiStats to Identify Storage Performance Issues
You can use the vscsiStats tool to gather storage performance data for VMFS and NFS datastores, as discussed in detail in this article.