Using esxtop/resxtop to Identify Network Performance Problems

When using esxtop or resxtop to troubleshoot a perceived networking issue, look out for dropped packets. Dropped transmit or received packets indicate that either the destination for the packets is too busy to receive them, or the network is too busy to properly send them.

Firstly the end device/VM will attempt to buffer the traffic until it can be processed. If this queue fills, then the traffic is queued on the vSwitch. Finally, if the vSwitch queue is exhausted, then the packets will be dropped. You can check for dropped packets by looking at the %DRPTX and %DRPRX metrics in resxtop:

1:47:38pm up 5:11, 290 worlds, 0 VMs, 0 vCPUs; CPU load average: 0.01, 0.01, 0.01

   PORT-ID              USED-BY  TEAM-PNIC DNAME              PKTTX/s  MbTX/s    PKTRX/s  MbRX/s %DRPTX %DRPRX
  16777217           Management        n/a vSwitch0              0.00    0.00       0.00    0.00   0.00   0.00
  16777218               vmnic0          - vSwitch0              2.53    0.21       7.77    0.01   0.00   0.00
  16777219               vmnic1          - vSwitch0              0.00    0.00       0.00    0.00   0.00   0.00
  16777220                 vmk0     vmnic0 vSwitch0              2.53    0.21       2.72    0.00   0.00   0.00
  16777221                 vmk3     vmnic1 vSwitch0              0.00    0.00       0.00    0.00   0.00   0.00
  33554433           Management        n/a vSwitch1              0.00    0.00       0.00    0.00   0.00   0.00
  33554434               vmnic4          - vSwitch1              0.00    0.00       0.00    0.00   0.00   0.00
  33554435               vmnic5          - vSwitch1              0.00    0.00       0.00    0.00   0.00   0.00
  33554436                 vmk1     vmnic4 vSwitch1              0.00    0.00       0.00    0.00   0.00   0.00
  50331649           Management        n/a vSwitch3              0.00    0.00       0.00    0.00   0.00   0.00
  50331650               vmnic6          - vSwitch3              0.00    0.00       0.00    0.00   0.00   0.00
  50331651               vmnic7          - vSwitch3              0.00    0.00       0.00    0.00   0.00   0.00
  67108865           Management        n/a DvsPortset-0          0.00    0.00       0.00    0.00   0.00   0.00
  67108866               vmnic8          - DvsPortset-0          0.00    0.00       0.00    0.00   0.00   0.00
  67108867               vmnic9          - DvsPortset-0          0.00    0.00       0.00    0.00   0.00   0.00

Dropped packets can occur for a number of reasons. Possible causes include high guest CPU utilisation, having incorrect network drivers installed in the guest OS. It could also be that there isn’t sufficient uplink capacity in the vSwitch to cope with demand. If this is the case, consider migrating VMs to alternate hosts to reduce load, or add additional uplinks to the affected vSwitch.

You can also look at implementing network I/O control, to help make better use of existing host resources. The other resxtop network metrics to be aware of are:

• MbTX/s – Amount of data transmitted in Mbps
• MbRX/s – Amount of data received in Mbps
• PKTTX/s – Average number of packets transmitted per second in the sampling interval
• PKTRX/s – Average number of packets received per second in the sampling interval