Utilize vSphere CLI Commands to Troubleshoot ESXi Network Configurations

by admin

To troubleshoot networking configurations from the ESXi command line, ESXCLI is the tool to use. This post will focus on looking at the options available using the network namespace of esxcli in order to display a hosts network configuration. I’ll also have a quick look at a couple of other useful CLI tools, such as netcat, vmkping and tcpdump.

There are a number of options available when running ‘esxcli’ in terms of network settings:

~ # esxcli network
Usage: esxcli network {cmd} [cmd options]

Available Namespaces:
  fence                 Commands to list fence information
  firewall              A set of commands for firewall related operations
  ip                    Operations that can be performed on vmknics
  vswitch               Commands to list and manipulate Virtual Switches on an ESX host.
  nic                   Operations having to do with the configuration of Network Interface Card and getting and updating the NIC

We’ll go through some of the options that these namespaces offer.

Listing vSwitch Configuration

You can list the vSwitches configured on a ESXi host by running:

~ # esxcli network vswitch standard list
   Name: vSwitch0
   Class: etherswitch
   Num Ports: 128
   Used Ports: 5
   Configured Ports: 128
   MTU: 1500
   CDP Status: both
   Beacon Enabled: false
   Beacon Interval: 1
   Beacon Threshold: 3
   Beacon Required By:
   Uplinks: vmnic1, vmnic0
   Portgroups: vMotion, Management Network

To list distributed vSwitches instead, swap ‘standard’ for ‘dvs’ in the command. Following on from the output above you can also dig down to look at the Policy, Portgroup settings for the vSwitch:

~ # esxcli network vswitch standard
Usage: esxcli network vswitch standard {cmd} [cmd options]

Available Namespaces:
  policy                Commands to manipulate network policy settings governing the given virtual switch.
  portgroup             Commands to list and manipulate Port Groups on an ESX host.
  uplink                Commands to add and remove uplink on given virtual switch.

For example, to display the failover settings for vSwitch0, the following command can be run:

~ # esxcli network vswitch standard  policy failover get -v vSwitch0
   Load Balancing: srcport
   Network Failure Detection: link
   Notify Switches: true
   Failback: true
   Active Adapters: vmnic0, vmnic1
   Standby Adapters:
   Unused Adapters:

Listing VMKernel Interfaces

To list the VMkernel ports on a host you can run:

~ # esxcli network ip interface list
   Name: vmk0
   MAC Address: 00:0c:29:3c:c3:4e
   Enabled: true
   Portset: vSwitch0
   Portgroup: Management Network
   VDS Name: N/A
   VDS Port: N/A
   VDS Connection: -1
   MTU: 1500
   TSO MSS: 65535
   Port ID: 33554438

The command will display the interface name, MAC address, and which vSwitch and Portgroup it belongs to. To list the IP address configuration for the VMkernel ports:

~ # esxcli network ip interface ipv4 get
Name  IPv4 Address   IPv4 Netmask   IPv4 Broadcast  Address Type  DHCP DNS
----  -------------  -------------  --------------  ------------  --------
vmk0   STATIC           false
vmk1     STATIC           false
vmk2    STATIC           false
vmk3     STATIC           false
vmk4    STATIC           false

Listing Connections and Neighbors

To list established connections on your host you can run:

~ # esxcli network ip connection list
Proto  Recv Q  Send Q  Local Address      Foreign Address      State        World ID  World Name
-----  ------  ------  -----------------  -------------------  -----------  --------  ---------------
tcp         0       0      ESTABLISHED      5225  hostd-worker
tcp         0       0       ESTABLISHED      4791  rhttpproxy-work
tcp         0       0      ESTABLISHED      5215  rhttpproxy-work
tcp         0       0        ESTABLISHED      8061  python
tcp         0       0         TIME_WAIT           0
tcp         0       0         TIME_WAIT           0
tcp         0       0  ESTABLISHED      4629  busybox

This is the equivalent of running ‘netstat’ on a Windows machine. To list the host’s ARP cache (or neighbors table) you can run:

~ # esxcli network ip neighbor list
Neighbor       Mac Address        Vmknic    Expiry  State
-------------  -----------------  ------  --------  -----      00:0c:29:65:b5:b8  vmk1     851 sec    cc:33:bb:6a:0d:da  vmk0    1079 sec  00:0c:29:cb:85:86  vmk0    1003 sec  00:0c:29:de:21:10  vmk0    1191 sec

This can be useful when troubleshooting connectivity, for example, when a host is failing to connect to another over the vMotion network.

You can list the host’s routing table by running:

~ # esxcli network ip route ipv4 list
Network      Netmask        Gateway        Interface  Source
-----------  -------------  -------------  ---------  ------
default  vmk0       MANUAL        vmk1       MANUAL        vmk4       MANUAL        vmk2       MANUAL    vmk0       MANUAL        vmk0       MANUAL

Troubleshooting Network Connectivity Using Netcat

Netcat can be used to test connectivity to and from your ESXi host. You can view the options available by running:

~ # nc -h
usage: nc [-46DdhklnrStUuvzC] [-i interval] [-p source_port]
          [-s source_ip_address] [-T ToS] [-w timeout] [-X proxy_version]
          [-x proxy_address[:port]] [hostname] [port[s]]
        Command Summary:
                -4              Use IPv4
                -6              Use IPv6
                -D              Enable the debug socket option
                -d              Detach from stdin
                -h              This help text
                -i secs         Delay interval for lines sent, ports scanned
                -k              Keep inbound sockets open for multiple connects
                -l              Listen mode, for inbound connects
                -n              Suppress name/port resolutions
                -p port         Specify local port for remote connects
                -r              Randomize remote ports
                -s addr         Local source address
                -T ToS          Set IP Type of Service
                -C              Send CRLF as line-ending
                -t              Answer TELNET negotiation
                -U              Use UNIX domain socket
                -u              UDP mode
                -v              Verbose
                -w secs         Timeout for connects and final net reads
                -X proto        Proxy protocol: "4", "5" (SOCKS) or "connect"
                -x addr[:port]  Specify proxy address and port
                -z              Zero-I/O mode [used for scanning]
        Port numbers can be individual or ranges: lo-hi [inclusive]

The basic syntax is:

# nc -z dest-ip dest-port

For example, to test connectivity on to port 80 you could run:

~ # nc -z 80
Connection to 80 port [tcp/http] succeeded!

You can also use netcat to test a range of ports on a remote host:

~ # nc -w 1 -z 80-85
Connection to 80 port [tcp/http] succeeded!

Netcat will report back with the ports it has found to be open within the specified range.

Troubleshooting network connectivity with ping and vmkping

You can test connectivity to remote ESXi host using the ping and vmkping utilities. Using vmkping to test connectivity via vMotion interfaces is a common practice. For example:

~ # vmkping
PING ( 56 data bytes
64 bytes from icmp_seq=0 ttl=64 time=0.585 ms
64 bytes from icmp_seq=1 ttl=64 time=0.341 ms
64 bytes from icmp_seq=2 ttl=64 time=0.234 ms

--- ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.234/0.387/0.585 ms

If you use vmkping with the ‘-D’ switch, you can test the host’s IP stack as the command will automatically test configured IP addresses:

~ # vmkping -D
PING ( 56 data bytes
64 bytes from icmp_seq=0 ttl=64 time=0.028 ms
64 bytes from icmp_seq=1 ttl=64 time=0.044 ms
64 bytes from icmp_seq=2 ttl=64 time=0.040 ms

--- ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.028/0.037/0.044 ms

PING ( 56 data bytes
64 bytes from icmp_seq=0 ttl=64 time=0.152 ms
64 bytes from icmp_seq=1 ttl=64 time=0.043 ms
64 bytes from icmp_seq=2 ttl=64 time=0.036 ms

--- ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.036/0.077/0.152 ms

Troubleshooting SSL port connectivity with openssl

You can use the open ssl client present on an ESXi host to test connectivity to an ssl port – for example to vCenter or to another host. To do so:

~ # openssl s_client -connect

The output will also contain details about the certificate, which can be useful when troubleshooting certificate problems.

Capturing Traffic with tcpdump-uw

You can learn more about capturing traffic using tcpdump here. I’ll cover a couple of examples here however.
To display packets on interface vmk0 you can run:

~ # tcpdump-uw -i vmk0

To output the traffic capture to a file you can run:

# tcpdump-uw -i vmk0 -s 1514 -w /vmfs/volumes/ESXi1-Local/traffic.pcap
tcpdump-uw: listening on vmk0, link-type EN10MB (Ethernet), capture size 1514 bytes
tcpdump-uw: pcap_loop: recvfrom: Interrupted system call
118 packets captured
118 packets received by filter
0 packets dropped by kernel

You can then open the resulting .pcap file in a tool such as Wireshark, for analysis:


Viewing Physical NIC Configuration

You can list the physical NICs installed in the host by using:

~ # esxcli network nic list
Name    PCI Device     Driver  Link  Speed  Duplex  MAC Address         MTU  Description
------  -------------  ------  ----  -----  ------  -----------------  ----  -------------------------------------------------------
vmnic0  0000:003:00.0  tg3     Up     1000  Full    74:46:a0:fe:43:74  1500  Broadcom Corporation NetXtreme BCM5717 Gigabit Ethernet
vmnic1  0000:003:00.1  tg3     Up     1000  Full    74:46:a0:fe:43:75  1500  Broadcom Corporation NetXtreme BCM5717 Gigabit Ethernet

Using the ‘esxcli network nic’ namespace you can also bring interfaces up and down (which is useful for testing), and you can view interface statistics.

~ # esxcli network nic
Usage: esxcli network nic {cmd} [cmd options]

Available Namespaces:
  vlan                  Get VLAN information for a NIC
  stats                 Get packet statistics for a NIC

Available Commands:
  down                  Bring down the specified network device.
  get                   Get the generic configuration of a network device
  list                  This command will list the Physical NICs currently installed and loaded on the system.
  set                   Set the general options for the specified ethernet device.
  up                    Bring up the specified network device.

An example of how to display interface statistics is shown here:

~ # esxcli network nic stats get -n vmnic0
NIC statistics for vmnic0
   Packets received: 28718
   Packets sent: 20288
   Bytes received: 4159681
   Bytes sent: 4810909
   Receive packets dropped: 42
   Transmit packets dropped: 0
   Total receive errors: 0
   Receive length errors: 0
   Receive over errors: 0
   Receive CRC errors: 0
   Receive frame errors: 0
   Receive FIFO errors: 0
   Receive missed errors: 0
   Total transmit errors: 0
   Transmit aborted errors: 0
   Transmit carrier errors: 0
   Transmit FIFO errors: 0
   Transmit heartbeat errors: 0
   Transmit window errors: 0
Keep up to date with new posts on Buildvirtual.net - Follow us on Twitter:
Be Sociable, Share!

Leave a Comment


Previous post:

Next post: