To troubleshoot networking configurations from the ESXi command line, ESXCLI is the tool to use. This post will focus on looking at the options available using the network namespace of esxcli in order to display a hosts network configuration. I’ll also have a quick look at a couple of other useful CLI tools, such as netcat, vmkping and tcpdump.
There are a number of options available when running ‘esxcli’ in terms of network settings:
~ # esxcli network
Usage: esxcli network {cmd} [cmd options]
Available Namespaces:
fence Commands to list fence information
firewall A set of commands for firewall related operations
ip Operations that can be performed on vmknics
vswitch Commands to list and manipulate Virtual Switches on an ESX host.
nic Operations having to do with the configuration of Network Interface Card and getting and updating the NIC
settings.
We’ll go through some of the options that these namespaces offer.
Listing vSwitch Configuration
You can list the vSwitches configured on a ESXi host by running:
~ # esxcli network vswitch standard list vSwitch0 Name: vSwitch0 Class: etherswitch Num Ports: 128 Used Ports: 5 Configured Ports: 128 MTU: 1500 CDP Status: both Beacon Enabled: false Beacon Interval: 1 Beacon Threshold: 3 Beacon Required By: Uplinks: vmnic1, vmnic0 Portgroups: vMotion, Management Network
To list distributed vSwitches instead, swap ‘standard’ for ‘dvs’ in the command. Following on from the output above you can also dig down to look at the Policy, Portgroup settings for the vSwitch:
~ # esxcli network vswitch standard
Usage: esxcli network vswitch standard {cmd} [cmd options]
Available Namespaces:
policy Commands to manipulate network policy settings governing the given virtual switch.
portgroup Commands to list and manipulate Port Groups on an ESX host.
uplink Commands to add and remove uplink on given virtual switch.
For example, to display the failover settings for vSwitch0, the following command can be run:
~ # esxcli network vswitch standard policy failover get -v vSwitch0 Load Balancing: srcport Network Failure Detection: link Notify Switches: true Failback: true Active Adapters: vmnic0, vmnic1 Standby Adapters: Unused Adapters:
Listing VMKernel Interfaces
To list the VMkernel ports on a host you can run:
~ # esxcli network ip interface list vmk0 Name: vmk0 MAC Address: 00:0c:29:3c:c3:4e Enabled: true Portset: vSwitch0 Portgroup: Management Network VDS Name: N/A VDS UUID: N/A VDS Port: N/A VDS Connection: -1 MTU: 1500 TSO MSS: 65535 Port ID: 33554438
The command will display the interface name, MAC address, and which vSwitch and Portgroup it belongs to. To list the IP address configuration for the VMkernel ports:
~ # esxcli network ip interface ipv4 get Name IPv4 Address IPv4 Netmask IPv4 Broadcast Address Type DHCP DNS ---- ------------- ------------- -------------- ------------ -------- vmk0 192.168.0.240 255.255.255.0 192.168.0.255 STATIC false vmk1 10.10.0.10 255.255.255.0 10.10.0.255 STATIC false vmk2 172.16.0.251 255.255.255.0 172.16.0.255 STATIC false vmk3 10.10.0.12 255.255.255.0 10.10.0.255 STATIC false vmk4 10.20.20.1 255.255.255.0 10.20.20.255 STATIC false
Listing Connections and Neighbors
To list established connections on your host you can run:
~ # esxcli network ip connection list Proto Recv Q Send Q Local Address Foreign Address State World ID World Name ----- ------ ------ ----------------- ------------------- ----------- -------- --------------- tcp 0 0 127.0.0.1:8307 127.0.0.1:56160 ESTABLISHED 5225 hostd-worker tcp 0 0 127.0.0.1:56160 127.0.0.1:8307 ESTABLISHED 4791 rhttpproxy-work tcp 0 0 127.0.0.1:443 127.0.0.1:62376 ESTABLISHED 5215 rhttpproxy-work tcp 0 0 127.0.0.1:62376 127.0.0.1:443 ESTABLISHED 8061 python tcp 0 0 127.0.0.1:51014 127.0.0.1:80 TIME_WAIT 0 tcp 0 0 127.0.0.1:59345 127.0.0.1:80 TIME_WAIT 0 tcp 0 0 192.168.0.235:22 192.168.0.212:51558 ESTABLISHED 4629 busybox
This is the equivalent of running ‘netstat’ on a Windows machine. To list the host’s ARP cache (or neighbors table) you can run:
~ # esxcli network ip neighbor list Neighbor Mac Address Vmknic Expiry State ------------- ----------------- ------ -------- ----- 10.10.0.1 00:0c:29:65:b5:b8 vmk1 851 sec 192.168.0.1 cc:33:bb:6a:0d:da vmk0 1079 sec 192.168.0.238 00:0c:29:cb:85:86 vmk0 1003 sec 192.168.0.239 00:0c:29:de:21:10 vmk0 1191 sec
This can be useful when troubleshooting connectivity, for example, when a host is failing to connect to another over the vMotion network.
You can list the host’s routing table by running:
~ # esxcli network ip route ipv4 list Network Netmask Gateway Interface Source ----------- ------------- ------------- --------- ------ default 0.0.0.0 192.168.0.239 vmk0 MANUAL 10.10.0.0 255.255.255.0 0.0.0.0 vmk1 MANUAL 10.20.20.0 255.255.255.0 0.0.0.0 vmk4 MANUAL 172.16.0.0 255.255.255.0 0.0.0.0 vmk2 MANUAL 184.45.34.0 255.255.255.0 192.168.0.1 vmk0 MANUAL 192.168.0.0 255.255.255.0 0.0.0.0 vmk0 MANUAL
Troubleshooting Network Connectivity Using Netcat
Netcat can be used to test connectivity to and from your ESXi host. You can view the options available by running:
~ # nc -h
usage: nc [-46DdhklnrStUuvzC] [-i interval] [-p source_port]
[-s source_ip_address] [-T ToS] [-w timeout] [-X proxy_version]
[-x proxy_address[:port]] [hostname] [port[s]]
Command Summary:
-4 Use IPv4
-6 Use IPv6
-D Enable the debug socket option
-d Detach from stdin
-h This help text
-i secs Delay interval for lines sent, ports scanned
-k Keep inbound sockets open for multiple connects
-l Listen mode, for inbound connects
-n Suppress name/port resolutions
-p port Specify local port for remote connects
-r Randomize remote ports
-s addr Local source address
-T ToS Set IP Type of Service
-C Send CRLF as line-ending
-t Answer TELNET negotiation
-U Use UNIX domain socket
-u UDP mode
-v Verbose
-w secs Timeout for connects and final net reads
-X proto Proxy protocol: "4", "5" (SOCKS) or "connect"
-x addr[:port] Specify proxy address and port
-z Zero-I/O mode [used for scanning]
Port numbers can be individual or ranges: lo-hi [inclusive]
The basic syntax is:
# nc -z dest-ip dest-port
For example, to test connectivity on to port 80 you could run:
~ # nc -z 192.168.0.71 80 Connection to 192.168.0.71 80 port [tcp/http] succeeded!
You can also use netcat to test a range of ports on a remote host:
~ # nc -w 1 -z 192.168.0.71 80-85 Connection to 192.168.0.71 80 port [tcp/http] succeeded!
Netcat will report back with the ports it has found to be open within the specified range.
Troubleshooting network connectivity with ping and vmkping
You can test connectivity to remote ESXi host using the ping and vmkping utilities. Using vmkping to test connectivity via vMotion interfaces is a common practice. For example:
~ # vmkping 172.16.0.252 PING 172.16.0.252 (172.16.0.252): 56 data bytes 64 bytes from 172.16.0.252: icmp_seq=0 ttl=64 time=0.585 ms 64 bytes from 172.16.0.252: icmp_seq=1 ttl=64 time=0.341 ms 64 bytes from 172.16.0.252: icmp_seq=2 ttl=64 time=0.234 ms --- 172.16.0.252 ping statistics --- 3 packets transmitted, 3 packets received, 0% packet loss round-trip min/avg/max = 0.234/0.387/0.585 ms
If you use vmkping with the ‘-D’ switch, you can test the host’s IP stack as the command will automatically test configured IP addresses:
~ # vmkping -D PING 192.168.0.240 (192.168.0.240): 56 data bytes 64 bytes from 192.168.0.240: icmp_seq=0 ttl=64 time=0.028 ms 64 bytes from 192.168.0.240: icmp_seq=1 ttl=64 time=0.044 ms 64 bytes from 192.168.0.240: icmp_seq=2 ttl=64 time=0.040 ms --- 192.168.0.240 ping statistics --- 3 packets transmitted, 3 packets received, 0% packet loss round-trip min/avg/max = 0.028/0.037/0.044 ms PING 10.10.0.10 (10.10.0.10): 56 data bytes 64 bytes from 10.10.0.10: icmp_seq=0 ttl=64 time=0.152 ms 64 bytes from 10.10.0.10: icmp_seq=1 ttl=64 time=0.043 ms 64 bytes from 10.10.0.10: icmp_seq=2 ttl=64 time=0.036 ms --- 10.10.0.10 ping statistics --- 3 packets transmitted, 3 packets received, 0% packet loss round-trip min/avg/max = 0.036/0.077/0.152 ms
Troubleshooting SSL port connectivity with openssl
You can use the open ssl client present on an ESXi host to test connectivity to an ssl port – for example to vCenter or to another host. To do so:
~ # openssl s_client -connect 192.168.0.240:443 CONNECTED(00000003)
The output will also contain details about the certificate, which can be useful when troubleshooting certificate problems.
Capturing Traffic with tcpdump-uw
You can learn more about capturing traffic using tcpdump here. I’ll cover a couple of examples here however.
To display packets on interface vmk0 you can run:
~ # tcpdump-uw -i vmk0
To output the traffic capture to a file you can run:
# tcpdump-uw -i vmk0 -s 1514 -w /vmfs/volumes/ESXi1-Local/traffic.pcap tcpdump-uw: listening on vmk0, link-type EN10MB (Ethernet), capture size 1514 bytes tcpdump-uw: pcap_loop: recvfrom: Interrupted system call 118 packets captured 118 packets received by filter 0 packets dropped by kernel
You can then open the resulting .pcap file in a tool such as Wireshark, for analysis:
Viewing Physical NIC Configuration
You can list the physical NICs installed in the host by using:
~ # esxcli network nic list Name PCI Device Driver Link Speed Duplex MAC Address MTU Description ------ ------------- ------ ---- ----- ------ ----------------- ---- ------------------------------------------------------- vmnic0 0000:003:00.0 tg3 Up 1000 Full 74:46:a0:fe:43:74 1500 Broadcom Corporation NetXtreme BCM5717 Gigabit Ethernet vmnic1 0000:003:00.1 tg3 Up 1000 Full 74:46:a0:fe:43:75 1500 Broadcom Corporation NetXtreme BCM5717 Gigabit Ethernet
Using the ‘esxcli network nic’ namespace you can also bring interfaces up and down (which is useful for testing), and you can view interface statistics.
~ # esxcli network nic
Usage: esxcli network nic {cmd} [cmd options]
Available Namespaces:
vlan Get VLAN information for a NIC
stats Get packet statistics for a NIC
Available Commands:
down Bring down the specified network device.
get Get the generic configuration of a network device
list This command will list the Physical NICs currently installed and loaded on the system.
set Set the general options for the specified ethernet device.
up Bring up the specified network device.
An example of how to display interface statistics is shown here:
~ # esxcli network nic stats get -n vmnic0 NIC statistics for vmnic0 Packets received: 28718 Packets sent: 20288 Bytes received: 4159681 Bytes sent: 4810909 Receive packets dropped: 42 Transmit packets dropped: 0 Total receive errors: 0 Receive length errors: 0 Receive over errors: 0 Receive CRC errors: 0 Receive frame errors: 0 Receive FIFO errors: 0 Receive missed errors: 0 Total transmit errors: 0 Transmit aborted errors: 0 Transmit carrier errors: 0 Transmit FIFO errors: 0 Transmit heartbeat errors: 0 Transmit window errors: 0