esxtop and resxtop

by admin

These two tools are used to monitor and gather performance data from an ESXi host.

  • esxtop. This gives real time CPU, memory, disk and network data for hosts and virtual machines. You can run esxtop from a direct connection to a host’s CLI
  • resxtop. This is a remote version of esxtop. It is included as part of vCLI and is present on the vMA (vSphere Management Assistant). resxtop has three modes of operation including Interactive, Batch, and Replay.

Example resxtop output

 7:30:37pm up  1:38, 206 worlds, 0 VMs, 0 vCPUs; CPU load average: 0.01, 0.01, 0.00
PCPU USED(%): 0.9 1.2 0.5 0.6 AVG: 0.8
PCPU UTIL(%): 1.0 1.6 0.6 0.7 AVG: 1.0

      ID      GID NAME             NWLD   %USED    %RUN    %SYS   %WAIT %VMWAIT    %RDY   %IDLE  %OVRLP   %CSTP  %MLMTD  %SWPWT
       1        1 idle                4  397.21  400.00    0.00    0.00       -  400.00    0.00    0.94    0.00    0.00    0.00
     817      817 hostd.2861         13    1.61    1.70    0.01 1298.25       -    1.33    0.00    0.00    0.00    0.00    0.00
       8        8 helper             82    0.28    0.28    0.00 8200.00       -    0.09    0.00    0.01    0.00    0.00    0.00
     995      995 vpxa.3060          18    0.24    0.24    0.00 1800.00       -    0.35    0.00    0.00    0.00    0.00    0.00
     639      639 vmkiscsid.2670      2    0.05    0.05    0.00  200.00       -    0.01    0.00    0.00    0.00    0.00    0.00
     607      607 vmsyslogd.2625      3    0.02    0.03    0.00  299.87       -    0.37    0.00    0.01    0.00    0.00    0.00
       2        2 system              9    0.01    0.01    0.00  900.00       -    0.02    0.00    0.00    0.00    0.00    0.00
       9        9 drivers            11    0.01    0.01    0.00 1100.00       -    0.01    0.00    0.00    0.00    0.00    0.00
    1029     1029 vmware-usbarbit     2    0.01    0.01    0.00  200.00       -    0.01    0.00    0.00    0.00    0.00    0.00
     859      859 dcbd.2906           1    0.01    0.01    0.00  100.00       -    0.00    0.00    0.00    0.00    0.00    0.00
     702      702 storageRM.2745      2    0.01    0.01    0.00  200.00       -    0.00    0.00    0.00    0.00    0.00    0.00

Most of the examples you will see here will be ran from a vMA. So long as fastpass is configured correctly, there is no need to authenticate each time the tool is run.

How to Configure esxtop/resxtop custom profiles

There are a lot of options available to you when using esxtop. When running the tool, pressing ‘h’ will show you what options are available:

Interactive commands are:

fF      Add or remove fields
oO      Change the order of displayed fields
s       Set the delay in seconds between updates
#       Set the number of instances to display
W       Write configuration file ~/.esxtop50rc
e       Expand/Rollup Cpu Statistics
V       View only VM instances
L       Change the length of the NAME field
l       Limit display to a single group

Sort by:
        U:%USED         R:%RDY          N:GID
Switch display:
        c:cpu           i:interrupt     m:memory        n:network
        d:disk adapter  u:disk device   v:disk VM       p:power mgmt

So, as an example, you can press ‘d’ to access stats around the disk adapters:

7:44:11pm up  1:51, 205 worlds, 0 VMs, 0 vCPUs; CPU load average: 0.01, 0.01, 0.01

 ADAPTR PATH                 NPTH   CMDS/s  READS/s WRITES/s MBREAD/s MBWRTN/s DAVG/cmd KAVG/cmd GAVG/cmd QAVG/cmd
 vmhba0 -                       0     0.00     0.00     0.00     0.00     0.00     0.00     0.00     0.00     0.00
 vmhba1 -                       3     0.00     0.00     0.00     0.00     0.00     0.00     0.00     0.00     0.00
vmhba32 -                       1     0.00     0.00     0.00     0.00     0.00     0.00     0.00     0.00     0.00
vmhba33 -                       2     0.00     0.00     0.00     0.00     0.00     0.00     0.00     0.00     0.00

From that screen you can press ‘f’ to select what fields to wish to view:

Current Field order: ABCdEfGhijkl

* A:  ADAPTR = Adapter Name
* B:  PATH = Path Name
* C:  NPATHS = Num Paths
  D:  QSTATS = Queue Stats
* E:  IOSTATS = I/O Stats
  F:  RESVSTATS = Reserve Stats
* G:  LATSTATS/cmd = Overall Latency Stats (ms)
  H:  LATSTATS/rd = Read Latency Stats (ms)
  I:  LATSTATS/wr = Write Latency Stats (ms)
  J:  ERRSTATS/s = Error Stats
  K:  PAESTATS/s = PAE Stats
  L:  SPLTSTATS/s = SPLIT Stats

Once you have set it up to display the values you are interested in you can press ‘W’ to save the config to a file. You can accept the default or type your own path as necessary.

Save config file to (Default to : /home/vi-admin/.esxtop50rc): /home/vi-admin/.esxtopdiskadrc

Next time you need to view those specific stats in resxtop you can open it, referring to the file you saved:

vi-admin@vma:~[192.168.88.134]> resxtop -c .esxtopdiskadrc

 8:16:58pm up  2:24, 206 worlds, 0 VMs, 0 vCPUs; CPU load average: 0.01, 0.00, 0.00

 ADAPTR PATH                   CMDS/s  READS/s WRITES/s MBREAD/s MBWRTN/s
 vmhba0 -                        0.00     0.00     0.00     0.00     0.00
 vmhba1 -                        0.00     0.00     0.00     0.00     0.00
vmhba32 -                        0.00     0.00     0.00     0.00     0.00
vmhba33 -                        0.00     0.00     0.00     0.00     0.00

Determine use cases for and apply esxtop/resxtop Interactive, Batch and Replay modes

esxtop/resxtop has three modes in which it can be run:

  • Interactive. This is the default mode, and the one used in the example above. By default statistics are collected at a 5 second interval, although this can be changed.
  • Batch. This mode is used for collecting statistics for a period of time, for later analysis. Once the stats have been collected, they can be analysed using Excel, Perfmon or ESXplot amongst other tools.
  • Replay. This mode allows you to replay data collected using the vm-support tool. You cannot replay data collected in batch mode interactively.

Collecting Data using Batch Mode

There are a few options to set when doing a batch mode collection – the various resxtop options are shown here:

vi-admin@vma:~> resxtop --help
usage: resxtop [-h] [-v] [-b] [-s] [-a] [-c config file] [-d delay] [-n iterations]
               [--server server-name [--vihost host-name]] [--portnumber socket-port] [--username user-name]
              -h prints this help menu.
              -v prints version.
              -b enables batch mode.
              -s enables secure mode.
              -a show all statistics.
              -c sets the esxtop configuration file, which by default is .esxtop50rc
              -d sets the delay between updates in seconds.
              -n runs resxtop for only n iterations.
              --server      remote server name.
              --vihost      esx host name, if --server specifies vc server.
              --portnumber  socket port, default is 443.
              --username    user name on the remote server.

The example below, ran from a vMA, will collect stats at 5 second intervals, 20 times. The -a switch indicates that you want to collect all stats:

vi-admin@vma:~[192.168.88.134]> resxtop -b -a -d 5 -n 20 > output.csv

Rather than collect all stats, you can specify a configuration file such as the one I used earlier. Batch mode will then only capture the stats specified in the file.

vi-admin@vma:~[192.168.88.134]> resxtop -b -a -d 5 -n 20 -c .esxtopdiskadrc > output2.csv

Be aware that the size of the capture files can grow quickly!

-rw------- 1 vi-admin root 2.6M Oct 24 14:49 output.csv

With this in mind, it’s possible to compress the output file as it is collected:

vi-admin@vma:~[192.168.88.134]> resxtop -b -a -d 5 -n 20 | gzip -9c > output.csv.gz

The zipped file is significantly smaller than the unzipped version:

-rw------- 1 vi-admin root 114K Oct 24 15:01 output.csv.gz

Replaying Performance Data using ESXTOP

You can also capture performance data using the vm-support tool. vm-support is a tool most commonly used to capture log and configuration data to send to VMware, however it can also be used to capture performance data. There are a number of options that can be specified when running vm-support:

~ # vm-support -h
Usage: vm-support [options]

Options:
  -h, --help            show this help message and exit
  -g GROUPS, --groups=GROUPS
                        Gather data from listed groups
  -a MANIFESTS, --manifests=MANIFESTS
                        Gather from listed manifests
  -e EXCLUDEMANIFESTS, --excludemanifests=EXCLUDEMANIFESTS
                        Exclude the listed manifests
  --listmanifests       List available manifests
  -G, --listgroups      List available manifest groups
  -t, --listtags        List available manifest tags
  -p, --performance     Gather performance data
  -d DURATION, --duration=DURATION
                        Duration of performance monitoring (in seconds)
  -i INTERVAL, --interval=INTERVAL
                        Interval between performance snapshots
  -v VM, --vm=VM        Gather detailed information about this specific VM (ie
                        --vm )
  -V, --listvms         List currently registered VMs
  -w WORKINGDIR, --workingdir=WORKINGDIR
                        Directory to create .tgz in
  -D, --dryrun          Prints out the data that would have been gathered
  -s, --stream          stream data to stdout
  -q, --quiet           Output only the location of the bundle
  -E ERRORFILE, --errorfile=ERRORFILE
                        Prints (non-fatal) errors to specified file (overrides
                        --quiet and --stream)
  --loglevel=LOGLEVEL   Set logging to specified level: 0-50 (0=most verbose)
  --version             Display the vm-support version
  -l, --listfiles       List all files gathered by vm-support
~ #

To capture performance data only, the -p switch is used. As with batch mode for esxtop we have to set the length of the capture and the interval. The following command will capture performance statistics for 60 seconds at 10 second intervals, writing the capture to a VMFS datastore:

/var/log #  vm-support -p -d 60 -i 5 -w /vmfs/volumes/datastore1/
15:45:47: Creating /vmfs/volumes/datastore1/esx-esxi1.vmlab.local-2012-08-10--15.45.tgz
15:48:39: Done.
Please attach this file when submitting an incident report.
To file a support incident, go to http://www.vmware.com/support/sr/sr_login.jsp
To see the files collected, run: tar -tzf '/vmfs/volumes/datastore1/esx-esxi1.vmlab.local-2012-08-10--15.45.tgz'

The collected data will be in a zipped file, in order to save space. To work with it we will need to extract the contents as suggested in the commands output:

tar -tzf '/vmfs/volumes/datastore1/esx-esxi1.vmlab.local-2012-08-10--15.45.tgz

Once extracted it may be necessary to run the reconstruct.sh script which can be found in the extracted contents. This will be necessary if you receive an error stating ‘all vm-support snapshots have been used’

Once the files have been extracted, you can run esxtop in replay mode by running the following:

/var/log # esxtop -R esx-esxi1.vmlab.local-2012-08-10--15.23

The data will be replayed in esxtop:

 3:25:41pm up  1:05, 294 worlds, 0 VMs, 0 vCPUs; CPU load average: 0.14, 0.07, 0.03

 ADAPTR PATH                 NPTH   CMDS/s  READS/s WRITES/s MBREAD/s MBWRTN/s DAVG/cmd KAVG/cmd GAVG/cmd QAVG/cmd
 vmhba0 -                       0     0.00     0.00     0.00     0.00     0.00     0.00     0.00     0.00     0.00
 vmhba1 -                       3    14.52     4.25     8.20     0.26     1.37     1.18     0.01     1.19     0.29
vmhba32 -                       1     0.49     0.00     0.00     0.00     0.00     0.28     0.02     0.30     0.01
vmhba33 -                       2     2.17     1.38     0.00     0.10     0.00     9.06     0.01     9.07     0.00
*** Read stats from esx-esxi1.vmlab.local-2012-08-10--15.23/snapshot-3/commands/vsi_traverse_-s.txt ***

Notice the last line of the output, which indicates that esxtop is being run in replay mode, with the performance stats being read from a file.


Keep up to date with new posts on Buildvirtual.net - Follow us on Twitter:
Be Sociable, Share!

Leave a Comment

*

{ 1 trackback }

Previous post:

Next post: