Troubleshoot VMKernel Related Network Configuration Issues

VMkernel interfaces are used for a number of functions, such as management traffic, vMotion, Fault Tolerance logging, iSCSI and NFS. VMkernel interfaces are given the name ‘vmkx’, where X is the number (in order that it was created on the host). You can see VMkernel interfaces clearly when looking at a host’s networking configuration:

Looking at the properties of the VMkernel portgroup you can see what type of traffic the vmk interface is being used for:

 Testing the Management Network using the DCUI

You can test management network connectivity using the ‘Test Management Network’ option available in the DCUI.

This can be useful to test management connectivity following installing ESXi. Common issues that may cause these tests to fail include setting an incorrect VLAN for the management network (or not setting one at all), specifying incorrect DNS or default gateway IPs, and selecting the wrong physical adapter to be used for management traffic.

Troubleshooting VMkernel Issues using the CLI

There are a number of CLI commands available to help you in troubleshooting VMKernel issues. To list the VMKernel ports on a ESXi host you can run:

~ # esxcli network ip interface list
vmk0
   Name: vmk0
   MAC Address: 00:0c:29:3c:c3:4e
   Enabled: true
   Portset: vSwitch0
   Portgroup: Management Network
   VDS Name: N/A
   VDS UUID: N/A
   VDS Port: N/A
   VDS Connection: -1
   MTU: 1500
   TSO MSS: 65535
   Port ID: 33554438

vmk1
   Name: vmk1
   MAC Address: 00:50:56:6f:64:44
   Enabled: true
   Portset: vSwitch1
   Portgroup: ISCSI_1
   VDS Name: N/A
   VDS UUID: N/A
   VDS Port: N/A
   VDS Connection: -1
   MTU: 1500
   TSO MSS: 65535
   Port ID: 50331654

If you wanted to add a new VMKernel port you can do so using the following:

•# esxcli network ip interface add --interface-name=vmk6 –-portgroup-name=“FT”

You can then assign it an IP address using:

esxcli network ip interface ipv4 set --ipv4=10.20.10.1 --netmask=255.255.255.0 --type=static --interface-name=vmk6

To remove a VMK interface, you can use:

esxcli network ip interface remove --interface-name=vmk6

Testing VMKernel Interface Connectivity

I talking about using vmkping to test connectivity to vmkernel interfaces in this post. There is also the option to use ‘esxcli network diag ping’:

~ # esxcli network diag ping
Error: Missing required parameter -H|--host

Usage: esxcli network diag ping [cmd options]

Description:
  ping                  Send ICMP echo requests to network hosts.

Cmd options:
  -c|--count=     Specify the number of packets to send.
  -D|--debug            VMKPing debug mode.
  -d|--df               Set DF bit on IPv4 packets.
  -H|--host=       Specify the host to send packets to. (required)
  -I|--interface=  Specify the outgoing interface.
  -i|--interval=  Set the interval for sending packets in seconds.
  -4|--ipv4             Ping with ICMPv4 echo requests.
  -6|--ipv6             Ping with ICMPv6 echo requests.
  -N|--nexthop=    Override the system's default route selection, in dotted quad notation. (IPv4 only. Requires interface option)
  -s|--size=      Set the payload size of the packets to send.
  -t|--ttl=       Set IPv4 Time To Live or IPv6 Hop Limit
  -W|--wait=      Set the timeout to wait if no responses are received in seconds.

For example, to test connectivity to another hosts management IP I could run:

~ # esxcli network diag ping -H 192.168.0.235
   Summary:
         Duplicated: 0
         Host Addr: 192.168.0.235
         Packet Lost: 0
         Recieved: 3
         Roundtrip Avg MS: 354
         Roundtrip Max MS: 532
         Roundtrip Min MS: 247
         Transmitted: 3
   Trace:
         Detail:
         Dup: false
         Host: 192.168.0.235
         ICMPSeq: 0
         Received Bytes: 64
         Roundtrip Time MS: 532
         TTL: 64

         Detail:
         Dup: false
         Host: 192.168.0.235
         ICMPSeq: 1
         Received Bytes: 64
         Roundtrip Time MS: 248
         TTL: 64

         Detail:
         Dup: false
         Host: 192.168.0.235
         ICMPSeq: 2
         Received Bytes: 64
         Roundtrip Time MS: 285
         TTL: 64

You can use the ‘-I’ switch to specify the interface that the ping will be sent from. I’ve used this in the following example to confirm connectivity to my iSCSI presented storage:

~ # esxcli network diag ping -H 10.10.0.11 -I vmk3 Summary: Duplicated: 0 Host Addr: 10.10.0.11 Packet Lost: 0 Recieved: 3 Roundtrip Avg MS: 494 Roundtrip Max MS: 818 Roundtrip Min MS: 323 Transmitted: 3

Analyze troubleshooting data to determine if the root cause for a given network problem originates in the physical infrastructure or vSphere environment

So, I’ve added this section in here as it is on the VCAP-DCA blueprint, but I didn’t feel that it warrants it’s own article, mainly as I can’t think of a great deal to put about it. Earlier in this article there are examples of how to run tests to check connectivity to external IP addresses. The results of these tests should help determine whether there is an issue with the physical network or with the host.

I would say, run through the usual troubleshooting steps to determine where the problem originates. If you suspect a networking issue, check to see whether it is affected more than one host. If possible, check hosts connected to different physical switches. Work through the OSI layer, determining if there are any physical (cabling) issues, through to layer 2, then layer three issues.