This post will look at the ‘esxcli storage‘ namespace, with a focus on troubleshooting. I’ve written previously, in the course of covering the VCAP-DCA objectives, about using ESXCLI to w0rk with storage issues in this article, so there will be some overlap here.
Listing Device and Path Information
The first thing to do when troubleshooting storage using esxcli is to list the attached storage devices:
esxcli storage core device list
This will list all attached storage devices, example output shown below:
t10.F405E46494C4540013C625565687D2A6A75633D293877753 Display Name: OPNFILER iSCSI Disk (t10.F405E46494C4540013C625565687D2A6A75633D293877753) Has Settable Display Name: true Size: 5088 Device Type: Direct-Access Multipath Plugin: NMP Devfs Path: /vmfs/devices/disks/t10.F405E46494C4540013C625565687D2A6A75633D293877753 Vendor: OPNFILER Model: VIRTUAL-DISK Revision: 0 SCSI Level: 4 Is Pseudo: false Status: on Is RDM Capable: true Is Local: false Is Removable: false Is SSD: false Is Offline: false Is Perennially Reserved: false Thin Provisioning Status: unknown Attached Filters: VAAI Status: unknown Other UIDs: vml.0100000000316c526565782d6a7a65332d39787735564952545541
You can narrow down the output by specifying the storage device:
~ # esxcli storage core device list -d t10.F405E46494C4540013C625565687D2A6A75633D293877753 t10.F405E46494C4540013C625565687D2A6A75633D293877753 Display Name: OPNFILER iSCSI Disk (t10.F405E46494C4540013C625565687D2A6A75633D293877753) Has Settable Display Name: true Size: 5088 Device Type: Direct-Access Multipath Plugin: NMP Devfs Path: /vmfs/devices/disks/t10.F405E46494C4540013C625565687D2A6A75633D293877753 Vendor: OPNFILER Model: VIRTUAL-DISK Revision: 0 SCSI Level: 4 Is Pseudo: false Status: on Is RDM Capable: true Is Local: false Is Removable: false Is SSD: false Is Offline: false Is Perennially Reserved: false Thin Provisioning Status: unknown Attached Filters: VAAI Status: unknown Other UIDs: vml.0100000000316c526565782d6a7a65332d39787735564952545541
It’s clear from the output that this device is using VMware’s native multipathing plugin:
Multipath Plugin: NMP
To view a host’s paths to the storage device you can use the ‘esxcli storage core path list’ command. Use the ‘-d’ switch and specify the device:
~ # esxcli storage core path list -d t10.F405E46494C4540013C625565687D2A6A75633D293877753 iqn.1998-01.com.vmware:esxi1-47be1f31-00023d000002,iqn.2006-01.com.openfiler:tsn.a75226350df7,t,1-t10.F405E46494C4540013C625565687D2A6A75633D293877753 UID: iqn.1998-01.com.vmware:esxi1-47be1f31-00023d000002,iqn.2006-01.com.openfiler:tsn.a75226350df7,t,1-t10.F405E46494C4540013C625565687D2A6A75633D293877753 Runtime Name: vmhba33:C1:T0:L0 Device: t10.F405E46494C4540013C625565687D2A6A75633D293877753 Device Display Name: OPNFILER iSCSI Disk (t10.F405E46494C4540013C625565687D2A6A75633D293877753) Adapter: vmhba33 Channel: 1 Target: 0 LUN: 0 Plugin: NMP State: active Transport: iscsi Adapter Identifier: iqn.1998-01.com.vmware:esxi1-47be1f31 Target Identifier: 00023d000002,iqn.2006-01.com.openfiler:tsn.a75226350df7,t,1 Adapter Transport Details: iqn.1998-01.com.vmware:esxi1-47be1f31 Target Transport Details: IQN=iqn.2006-01.com.openfiler:tsn.a75226350df7 Alias= Session=00023d000002 PortalTag=1 iqn.1998-01.com.vmware:esxi1-47be1f31-00023d000001,iqn.2006-01.com.openfiler:tsn.a75226350df7,t,1-t10.F405E46494C4540013C625565687D2A6A75633D293877753 UID: iqn.1998-01.com.vmware:esxi1-47be1f31-00023d000001,iqn.2006-01.com.openfiler:tsn.a75226350df7,t,1-t10.F405E46494C4540013C625565687D2A6A75633D293877753 Runtime Name: vmhba33:C0:T0:L0 Device: t10.F405E46494C4540013C625565687D2A6A75633D293877753 Device Display Name: OPNFILER iSCSI Disk (t10.F405E46494C4540013C625565687D2A6A75633D293877753) Adapter: vmhba33 Channel: 0 Target: 0 LUN: 0 Plugin: NMP State: active Transport: iscsi Adapter Identifier: iqn.1998-01.com.vmware:esxi1-47be1f31 Target Identifier: 00023d000001,iqn.2006-01.com.openfiler:tsn.a75226350df7,t,1 Adapter Transport Details: iqn.1998-01.com.vmware:esxi1-47be1f31 Target Transport Details: IQN=iqn.2006-01.com.openfiler:tsn.a75226350df7 Alias= Session=00023d000001 PortalTag=1
The output shows that there are two paths to this storage device and that both are active. This matches what is seen in the vSphere client for this device:
If a path was down, you should see the ‘dead’ state (though I’ve shortened the output here):
~ # esxcli storage core path list -d t10.F405E46494C4540013C625565687D2A6A75633D293877753 iqn.1998-01.com.vmware:esxi1-47be1f31-00023d000002,iqn.2006-01.com.openfiler:tsn.a75226350df7,t,1-t10.F405E46494C4540013C625565687D2A6A75633D293877753 UID: iqn.1998-01.com.vmware:esxi1-47be1f31-00023d000002,iqn.2006-01.com.openfiler:tsn.a75226350df7,t,1-t10.F405E46494C4540013C625565687D2A6A75633D293877753 Runtime Name: vmhba33:C1:T0:L0 Device: t10.F405E46494C4540013C625565687D2A6A75633D293877753 Device Display Name: OPNFILER iSCSI Disk (t10.F405E46494C4540013C625565687D2A6A75633D293877753) Adapter: vmhba33 Channel: 1 Target: 0 LUN: 0 Plugin: NMP State: dead Transport: iscsi Adapter Identifier: iqn.1998-01.com.vmware:esxi1-47be1f31 Target Identifier: 00023d000002,iqn.2006-01.com.openfiler:tsn.a75226350df7,t,1 Adapter Transport Details: Unavailable or path is unclaimed Target Transport Details: Unavailable or path is unclaimed
To investigate this dead path, we can run a command to output more details on this path by referencing the runtime name with the ‘-p’ switch:
~ # esxcli storage core path list -p vmhba33:C1:T0:L0 iqn.1998-01.com.vmware:esxi1-47be1f31-00023d000002,iqn.2006-01.com.openfiler:tsn.a75226350df7,t,1-t10.F405E46494C4540013C625565687D2A6A75633D293877753 UID: iqn.1998-01.com.vmware:esxi1-47be1f31-00023d000002,iqn.2006-01.com.openfiler:tsn.a75226350df7,t,1-t10.F405E46494C4540013C625565687D2A6A75633D293877753 Runtime Name: vmhba33:C1:T0:L0 Device: t10.F405E46494C4540013C625565687D2A6A75633D293877753 Device Display Name: OPNFILER iSCSI Disk (t10.F405E46494C4540013C625565687D2A6A75633D293877753) Adapter: vmhba33 Channel: 1 Target: 0 LUN: 0 Plugin: NMP State: dead Transport: iscsi Adapter Identifier: iqn.1998-01.com.vmware:esxi1-47be1f31 Target Identifier: 00023d000002,iqn.2006-01.com.openfiler:tsn.a75226350df7,t,1 Adapter Transport Details: Unavailable or path is unclaimed Target Transport Details: Unavailable or path is unclaimed
Again, we can see that the path State is ‘dead’. Next steps would be to check other hosts to see if they can see the storage device in order to help determine root cause.
Looking at Storage Related Statistics using ESXCLI
Along with using tools such as esxtop and vscsistats, you can also use ESXCLI to view storage statistics. To do so, use the following command:
~ # esxcli storage core device stats get
This will produce output for each attached storage device such as that shown below
t10.F405E46494C4540013C625565687D2A6A75633D293877753 Device: t10.F405E46494C4540013C625565687D2A6A75633D293877753 Successful Commands: 1010 Blocks Read: 67936 Blocks Written: 58 Read Operations: 711 Write Operations: 58 Reserve Operations: 15 Reservation Conflicts: 0 Failed Commands: 50 Failed Blocks Read: 0 Failed Blocks Written: 0 Failed Read Operations: 0 Failed Write Operations: 0 Failed Reserve Operations: 0
Pay attention to any of the ‘Failed’ metrics. Any number other than zero could indicate a problem with the device, or connectivity to it.
Troubleshooting Storage Claimrules
Sometimes when troubleshooting storage issues it will be necessary to modify or add claimrules for a device. I have written about this in detail here.
ESXCLI Storage Command Examples
| Operation | Command | Information |
| List the SCSI stats for the SCSI Paths in the system for FC/iSCSI LUNs | esxcli storage core path stats get | Displays the quantity of read and write operations and failures |
| Lists all the SCSI paths for FC/iSCSI LUNs | esxcli storage core path list | Displays each path – Includes the CTL, Adapter, Device, transport mode, state, and plugin type/details |
| Only get path information for a single device | esxcli storage core path list -d deviceID | Displays path information for specified device – Includes the CTL, Adapter, Device, transport mode, state, and plugin type/details |
| List the adapters in the system | esxcli storage core adapter list | Displays the HBA, driver, state, UID, and Description of each installed adapter |
| Turn off a path | esxcli storage core path set –state off –path vmhba??:C?:T?:L? | |
| Activate a path | esxcli storage core path set –state active –pathvmhba??:C?:T?:L? | |
| List devices claimed by Native Multipath Plugin | esxcli storage nmp device list | Displays Device, Array Type/Config, and Path Selection Policy information |
| List the available Storage Array Type Plugins | esxcli storage nmp satp list | Displays Name, Default Path Selection Policy and Description for each SATP |
| Lists the available Path Selection Policies | esxcli storage nmp psp list | Displays the name and description of all path selection policies |
| Lists the paths claimed by the NMP and shows the SATP & PSP information associated with the path | esxcli storage nmp path list | Displays the NMP claimed path SATP and PSP information |