Using ESXCLI to Troubleshoot Multipathing and PSA-related Issues

by admin

This post will look at the ‘esxcli storage‘ namespace, with a focus on troubleshooting. I’ve written previously, in the course of covering the VCAP-DCA objectives, about using ESXCLI to w0rk with storage issues in this article, so there will be some overlap here.

Listing Device and Path Information

The first thing to do when troubleshooting storage using esxcli is to list the attached storage devices:

esxcli storage core device list

This will list all attached storage devices, example output shown below:

t10.F405E46494C4540013C625565687D2A6A75633D293877753
   Display Name: OPNFILER iSCSI Disk (t10.F405E46494C4540013C625565687D2A6A75633D293877753)
   Has Settable Display Name: true
   Size: 5088
   Device Type: Direct-Access
   Multipath Plugin: NMP
   Devfs Path: /vmfs/devices/disks/t10.F405E46494C4540013C625565687D2A6A75633D293877753
   Vendor: OPNFILER
   Model: VIRTUAL-DISK
   Revision: 0
   SCSI Level: 4
   Is Pseudo: false
   Status: on
   Is RDM Capable: true
   Is Local: false
   Is Removable: false
   Is SSD: false
   Is Offline: false
   Is Perennially Reserved: false
   Thin Provisioning Status: unknown
   Attached Filters:
   VAAI Status: unknown
   Other UIDs: vml.0100000000316c526565782d6a7a65332d39787735564952545541

You can narrow down the output by specifying the storage device:

~ # esxcli storage core device list -d t10.F405E46494C4540013C625565687D2A6A75633D293877753
t10.F405E46494C4540013C625565687D2A6A75633D293877753
   Display Name: OPNFILER iSCSI Disk (t10.F405E46494C4540013C625565687D2A6A75633D293877753)
   Has Settable Display Name: true
   Size: 5088
   Device Type: Direct-Access
   Multipath Plugin: NMP
   Devfs Path: /vmfs/devices/disks/t10.F405E46494C4540013C625565687D2A6A75633D293877753
   Vendor: OPNFILER
   Model: VIRTUAL-DISK
   Revision: 0
   SCSI Level: 4
   Is Pseudo: false
   Status: on
   Is RDM Capable: true
   Is Local: false
   Is Removable: false
   Is SSD: false
   Is Offline: false
   Is Perennially Reserved: false
   Thin Provisioning Status: unknown
   Attached Filters:
   VAAI Status: unknown
   Other UIDs: vml.0100000000316c526565782d6a7a65332d39787735564952545541

It’s clear from the output that this device is using VMware’s native multipathing plugin:

Multipath Plugin: NMP

To view a host’s paths to the storage device you can use the ‘esxcli storage core path list’ command. Use the ‘-d’ switch and specify the device:

~ # esxcli storage core path list -d t10.F405E46494C4540013C625565687D2A6A75633D293877753
iqn.1998-01.com.vmware:esxi1-47be1f31-00023d000002,iqn.2006-01.com.openfiler:tsn.a75226350df7,t,1-t10.F405E46494C4540013C625565687D2A6A75633D293877753
   UID: iqn.1998-01.com.vmware:esxi1-47be1f31-00023d000002,iqn.2006-01.com.openfiler:tsn.a75226350df7,t,1-t10.F405E46494C4540013C625565687D2A6A75633D293877753
   Runtime Name: vmhba33:C1:T0:L0
   Device: t10.F405E46494C4540013C625565687D2A6A75633D293877753
   Device Display Name: OPNFILER iSCSI Disk (t10.F405E46494C4540013C625565687D2A6A75633D293877753)
   Adapter: vmhba33
   Channel: 1
   Target: 0
   LUN: 0
   Plugin: NMP
   State: active
   Transport: iscsi
   Adapter Identifier: iqn.1998-01.com.vmware:esxi1-47be1f31
   Target Identifier: 00023d000002,iqn.2006-01.com.openfiler:tsn.a75226350df7,t,1
   Adapter Transport Details: iqn.1998-01.com.vmware:esxi1-47be1f31
   Target Transport Details: IQN=iqn.2006-01.com.openfiler:tsn.a75226350df7 Alias= Session=00023d000002 PortalTag=1

iqn.1998-01.com.vmware:esxi1-47be1f31-00023d000001,iqn.2006-01.com.openfiler:tsn.a75226350df7,t,1-t10.F405E46494C4540013C625565687D2A6A75633D293877753
   UID: iqn.1998-01.com.vmware:esxi1-47be1f31-00023d000001,iqn.2006-01.com.openfiler:tsn.a75226350df7,t,1-t10.F405E46494C4540013C625565687D2A6A75633D293877753
   Runtime Name: vmhba33:C0:T0:L0
   Device: t10.F405E46494C4540013C625565687D2A6A75633D293877753
   Device Display Name: OPNFILER iSCSI Disk (t10.F405E46494C4540013C625565687D2A6A75633D293877753)
   Adapter: vmhba33
   Channel: 0
   Target: 0
   LUN: 0
   Plugin: NMP
   State: active
   Transport: iscsi
   Adapter Identifier: iqn.1998-01.com.vmware:esxi1-47be1f31
   Target Identifier: 00023d000001,iqn.2006-01.com.openfiler:tsn.a75226350df7,t,1
   Adapter Transport Details: iqn.1998-01.com.vmware:esxi1-47be1f31
   Target Transport Details: IQN=iqn.2006-01.com.openfiler:tsn.a75226350df7 Alias= Session=00023d000001 PortalTag=1

The output shows that there are two paths to this storage device and that both are active. This matches what is seen in the vSphere client for this device:

storage-paths

If a path was down, you should see the ‘dead’ state (though I’ve shortened the output here):

~ # esxcli storage core path list -d t10.F405E46494C4540013C625565687D2A6A75633D293877753
iqn.1998-01.com.vmware:esxi1-47be1f31-00023d000002,iqn.2006-01.com.openfiler:tsn.a75226350df7,t,1-t10.F405E46494C4540013C625565687D2A6A75633D293877753
   UID: iqn.1998-01.com.vmware:esxi1-47be1f31-00023d000002,iqn.2006-01.com.openfiler:tsn.a75226350df7,t,1-t10.F405E46494C4540013C625565687D2A6A75633D293877753
   Runtime Name: vmhba33:C1:T0:L0
   Device: t10.F405E46494C4540013C625565687D2A6A75633D293877753
   Device Display Name: OPNFILER iSCSI Disk (t10.F405E46494C4540013C625565687D2A6A75633D293877753)
   Adapter: vmhba33
   Channel: 1
   Target: 0
   LUN: 0
   Plugin: NMP
   State: dead
   Transport: iscsi
   Adapter Identifier: iqn.1998-01.com.vmware:esxi1-47be1f31
   Target Identifier: 00023d000002,iqn.2006-01.com.openfiler:tsn.a75226350df7,t,1
   Adapter Transport Details: Unavailable or path is unclaimed
   Target Transport Details: Unavailable or path is unclaimed

To investigate this dead path, we can run a command to output more details on this path by referencing the runtime name with the ‘-p’ switch:

~ # esxcli storage core path list -p vmhba33:C1:T0:L0
iqn.1998-01.com.vmware:esxi1-47be1f31-00023d000002,iqn.2006-01.com.openfiler:tsn.a75226350df7,t,1-t10.F405E46494C4540013C625565687D2A6A75633D293877753
   UID: iqn.1998-01.com.vmware:esxi1-47be1f31-00023d000002,iqn.2006-01.com.openfiler:tsn.a75226350df7,t,1-t10.F405E46494C4540013C625565687D2A6A75633D293877753
   Runtime Name: vmhba33:C1:T0:L0
   Device: t10.F405E46494C4540013C625565687D2A6A75633D293877753
   Device Display Name: OPNFILER iSCSI Disk (t10.F405E46494C4540013C625565687D2A6A75633D293877753)
   Adapter: vmhba33
   Channel: 1
   Target: 0
   LUN: 0
   Plugin: NMP
   State: dead
   Transport: iscsi
   Adapter Identifier: iqn.1998-01.com.vmware:esxi1-47be1f31
   Target Identifier: 00023d000002,iqn.2006-01.com.openfiler:tsn.a75226350df7,t,1
   Adapter Transport Details: Unavailable or path is unclaimed
   Target Transport Details: Unavailable or path is unclaimed

Again, we can see that the path State is ‘dead’. Next steps would be to check other hosts to see if they can see the storage device in order to help determine root cause.

Looking at Storage Related Statistics using ESXCLI

Along with using tools such as esxtop and vscsistats, you can also use ESXCLI to view storage statistics. To do so, use the following command:

~ # esxcli storage core device stats get

This will produce output for each attached storage device such as that shown below

t10.F405E46494C4540013C625565687D2A6A75633D293877753
   Device: t10.F405E46494C4540013C625565687D2A6A75633D293877753
   Successful Commands: 1010
   Blocks Read: 67936
   Blocks Written: 58
   Read Operations: 711
   Write Operations: 58
   Reserve Operations: 15
   Reservation Conflicts: 0
   Failed Commands: 50
   Failed Blocks Read: 0
   Failed Blocks Written: 0
   Failed Read Operations: 0
   Failed Write Operations: 0
   Failed Reserve Operations: 0

Pay attention to any of the ‘Failed’ metrics. Any number other than zero could indicate a problem with the device, or connectivity to it.

Troubleshooting Storage Claimrules

Sometimes when troubleshooting storage issues it will be necessary to modify or add claimrules for a device. I have written about this in detail here.

ESXCLI Storage Command Examples

Operation Command Information
List the SCSI stats for the SCSI Paths in the system for FC/iSCSI LUNs esxcli storage core path stats get Displays the quantity of read and write operations and failures
Lists all the SCSI paths for FC/iSCSI LUNs esxcli storage core path list Displays each path – Includes the CTL, Adapter, Device, transport mode, state, and plugin type/details
Only get path information for a single device esxcli storage core path list -d deviceID Displays path information for specified device – Includes the CTL, Adapter, Device, transport mode, state, and plugin type/details
List the adapters in the system esxcli storage core adapter list Displays the HBA, driver, state, UID, and Description of each installed adapter
Turn off a path esxcli storage core path set –state off –path vmhba??:C?:T?:L?  
Activate a path esxcli storage core path set –state active –pathvmhba??:C?:T?:L?  
List devices claimed by Native Multipath Plugin esxcli storage nmp device list Displays Device, Array Type/Config, and Path Selection Policy information
List the available Storage Array Type Plugins esxcli storage nmp satp list Displays Name, Default Path Selection Policy and Description for each SATP
Lists the available Path Selection Policies esxcli storage nmp psp list Displays the name and description of all path selection policies
Lists the paths claimed by the NMP and shows the SATP & PSP information associated with the path esxcli storage nmp path list Displays the NMP claimed path SATP and PSP information

Keep up to date with new posts on Buildvirtual.net - Follow us on Twitter:
Be Sociable, Share!

Leave a Comment

*

{ 1 trackback }

Previous post:

Next post: