Skip to main content
SAN hosts and cloud clients

Use VMware vSphere 8.x with ONTAP

Contributors netapp-ranuk netapp-pcarriga

You can configure ONTAP SAN host settings for the VMware vSphere 8.x release with FC, FCoE, and iSCSI protocols.

Hypervisor SAN booting

What you'll need

If you decide to use SAN booting, it must be supported by your configuration. You can use the NetApp Interoperability Matrix Tool to verify that your OS, HBA, HBA firmware and the HBA boot BIOS, and ONTAP version are supported.

Steps

  1. Map the SAN boot LUN to the host.

  2. Verify that multiple paths are available.

    Note Multiple paths become available after the host OS is up and running on the paths.

  3. Enable SAN booting in the server BIOS for the ports to which the SAN boot LUN is mapped.

    For information on how to enable the HBA BIOS, see your vendor-specific documentation.

  4. Reboot the host to verify that the boot is successful.

Multipathing

ESXi provides an extensible multipathing module called Native Multipathing Plug-In (NMP) that manages the sub-plugins, Storage Array Type Plugins (SATPs) and Path Selection Plugins (PSPs). By default, these SATP rules are available in ESXi.

For NetApp ONTAP storage, VMW_SATP_ALUA plugin is used by default with VMW_PSP_RR as a path selection policy (PSP). You can confirm by using the following command:

`esxcli storage nmp satp rule list -s VMW_SATP_ALUA`

Example output:

Name           Device  Vendor    Model         Driver  Transport  Options
---------------------------------------------------------------------------------
VMW_SATP_ALUA          LSI       INF-01-00                        reset_on_attempted_reserve  system
VMW_SATP_ALUA          NETAPP                                     reset_on_attempted_reserve  system

Rule Group  Claim Options  Default PSP  PSP Options     Description
------------------------------------------------------------------
tpgs_on        VMW_PSP_MRU                  NetApp E-Series arrays with ALUA support
tpgs_on        VMW_PSP_RR                   NetApp arrays with ALUA support

Non-ASA configurations

For non-ASA configurations, there should be two groups of paths with different priorities. The paths with the higher priorities are Active/Optimized, meaning they are serviced by the controller where the aggregate is located. The paths with the lower priorities are active but are non-optimized because they are served from a different controller. The non-optimized paths are only used when no optimized paths are available.

Example

The following example displays the correct output for an ONTAP LUN with two Active/Optimized paths and two Active/Non-Optimized paths:

# esxcli storage nmp device list -d naa.600a0980383148693724545244395855

Example output:

naa.600a0980383148693724545244395855
   Device Display Name: NETAPP Fibre Channel Disk (naa.600a0980383148693724545244395855)
   Storage Array Type: VMW_SATP_ALUA
   Storage Array Type Device Config: {implicit_support=on; explicit_support=off; explicit_allow=on; alua_followover=on; action_OnRetryErrors=off; {TPG_id=1000,TPG_state=ANO}{TPG_id=1001,TPG_state=AO}}
   Path Selection Policy: VMW_PSP_RR
   Path Selection Policy Device Config: {policy=rr,iops=1000,bytes=10485760,useANO=0; lastPathIndex=1: NumIOsPending=0,numBytesPending=0}
   Path Selection Policy Device Custom Config:
   Working Paths: vmhba4:C0:T0:L11, vmhba3:C0:T0:L11
   Is USB: false
# esxcli storage nmp path list -d naa.600a0980383148693724545244395855

Example output:

fc.20000024ff7f4a51:21000024ff7f4a51-fc.2009d039ea3ab21f:2003d039ea3ab21f-naa.600a0980383148693724545244395855
   Runtime Name: vmhba4:C0:T0:L11
   Device: naa.600a0980383148693724545244395855
   Device Display Name: NETAPP Fibre Channel Disk (naa.600a0980383148693724545244395855)
   Group State: active
   Array Priority: 0
   Storage Array Type Path Config: {TPG_id=1001,TPG_state=AO,RTP_id=4,RTP_health=UP}
   Path Selection Policy Path Config: PSP VMW_PSP_RR does not support path configuration.

fc.20000024ff7f4a50:21000024ff7f4a50-fc.2009d039ea3ab21f:2002d039ea3ab21f-naa.600a0980383148693724545244395855
   Runtime Name: vmhba3:C0:T0:L11
   Device: naa.600a0980383148693724545244395855
   Device Display Name: NETAPP Fibre Channel Disk (naa.600a0980383148693724545244395855)
   Group State: active
   Array Priority: 0
   Storage Array Type Path Config: {TPG_id=1001,TPG_state=AO,RTP_id=3,RTP_health=UP}
   Path Selection Policy Path Config: PSP VMW_PSP_RR does not support path configuration.

fc.20000024ff7f4a51:21000024ff7f4a51-fc.2009d039ea3ab21f:2001d039ea3ab21f-naa.600a0980383148693724545244395855
   Runtime Name: vmhba4:C0:T3:L11
   Device: naa.600a0980383148693724545244395855
   Device Display Name: NETAPP Fibre Channel Disk (naa.600a0980383148693724545244395855)
   Group State: active unoptimized
   Array Priority: 0
   Storage Array Type Path Config: {TPG_id=1000,TPG_state=ANO,RTP_id=2,RTP_health=UP}
   Path Selection Policy Path Config: PSP VMW_PSP_RR does not support path configuration.

fc.20000024ff7f4a50:21000024ff7f4a50-fc.2009d039ea3ab21f:2000d039ea3ab21f-naa.600a0980383148693724545244395855
   Runtime Name: vmhba3:C0:T3:L11
   Device: naa.600a0980383148693724545244395855
   Device Display Name: NETAPP Fibre Channel Disk (naa.600a0980383148693724545244395855)
   Group State: active unoptimized
   Array Priority: 0
   Storage Array Type Path Config: {TPG_id=1000,TPG_state=ANO,RTP_id=1,RTP_health=UP}
   Path Selection Policy Path Config: PSP VMW_PSP_RR does not support path configuration.

All SAN Array configurations

In All SAN Array (ASA) configurations, all paths to a given LUN are active and optimized. This improves performance by serving I/O operations through all paths at the same time.

Example

The following example displays the correct output for an ONTAP LUN:

esxcli storage nmp device list -d naa.600a098038304759563f4e7837574453

Example output:

naa.600a098038314962485d543078486c7a
   Device Display Name: NETAPP Fibre Channel Disk (naa.600a098038314962485d543078486c7a)
   Storage Array Type: VMW_SATP_ALUA
   Storage Array Type Device Config: {implicit_support=on; explicit_support=off; explicit_allow=on; alua_followover=on; action_OnRetryErrors=off; {TPG_id=1001,TPG_state=AO}{TPG_id=1000,TPG_state=AO}}
   Path Selection Policy: VMW_PSP_RR
   Path Selection Policy Device Config: {policy=rr,iops=1000,bytes=10485760,useANO=0; lastPathIndex=3: NumIOsPending=0,numBytesPending=0}
   Path Selection Policy Device Custom Config:
   Working Paths: vmhba4:C0:T0:L14, vmhba4:C0:T1:L14, vmhba3:C0:T0:L14, vmhba3:C0:T1:L14
   Is USB: false
# esxcli storage nmp path list -d naa.600a098038314962485d543078486c7a

Example output:

fc.200034800d756a75:210034800d756a75-fc.2018d039ea936319:2015d039ea936319-naa.600a098038314962485d543078486c7a
   Runtime Name: vmhba4:C0:T0:L14
   Device: naa.600a098038314962485d543078486c7a
   Device Display Name: NETAPP Fibre Channel Disk (naa.600a098038314962485d543078486c7a)
   Group State: active
   Array Priority: 0
   Storage Array Type Path Config: {TPG_id=1000,TPG_state=AO,RTP_id=2,RTP_health=UP}
   Path Selection Policy Path Config: PSP VMW_PSP_RR does not support path configuration.

fc.200034800d756a75:210034800d756a75-fc.2018d039ea936319:2017d039ea936319-naa.600a098038314962485d543078486c7a
   Runtime Name: vmhba4:C0:T1:L14
   Device: naa.600a098038314962485d543078486c7a
   Device Display Name: NETAPP Fibre Channel Disk (naa.600a098038314962485d543078486c7a)
   Group State: active
   Array Priority: 0
   Storage Array Type Path Config: {TPG_id=1001,TPG_state=AO,RTP_id=4,RTP_health=UP}
   Path Selection Policy Path Config: PSP VMW_PSP_RR does not support path configuration.

fc.200034800d756a74:210034800d756a74-fc.2018d039ea936319:2014d039ea936319-naa.600a098038314962485d543078486c7a
   Runtime Name: vmhba3:C0:T0:L14
   Device: naa.600a098038314962485d543078486c7a
   Device Display Name: NETAPP Fibre Channel Disk (naa.600a098038314962485d543078486c7a)
   Group State: active
   Array Priority: 0
   Storage Array Type Path Config: {TPG_id=1000,TPG_state=AO,RTP_id=1,RTP_health=UP}
   Path Selection Policy Path Config: PSP VMW_PSP_RR does not support path configuration.

fc.200034800d756a74:210034800d756a74-fc.2018d039ea936319:2016d039ea936319-naa.600a098038314962485d543078486c7a
   Runtime Name: vmhba3:C0:T1:L14
   Device: naa.600a098038314962485d543078486c7a
   Device Display Name: NETAPP Fibre Channel Disk (naa.600a098038314962485d543078486c7a)
   Group State: active
   Array Priority: 0
   Storage Array Type Path Config: {TPG_id=1001,TPG_state=AO,RTP_id=3,RTP_health=UP}
   Path Selection Policy Path Config: PSP VMW_PSP_RR does not support path configuration.

vVol

Virtual Volumes (vVols) are a VMware object type that corresponds to a Virtual Machine (VM) disk, its snapshots, and fast clones.

ONTAP tools for VMware vSphere includes the VASA Provider for ONTAP, which provides the integration point for a VMware vCenter to leverage vVols based storage. When you deploy the ONTAP tools Open Virtualization Appliance (OVA), it is automatically registered with the vCenter server and enables the VASA Provider.

When you create a vVols datastore using the vCenter user interface, it guides you to create FlexVols as backup storage for the datastore. vVols within vVols datastores are accessed by ESXi hosts using a protocol endpoint (PE). In SAN environments, one 4MB LUN is created on each FlexVol in the datastore for use as a PE. A SAN PE is an administrative logical unit (ALU). vVols are subsidiary logical units (SLUs).

Standard requirements and best practices for SAN environments apply when using vVols, including (but not limited to) the following:

  • Create at least one SAN LIF on each node per SVM you intend to use. The best practice is to create at least two per node, but no more than necessary.

  • Eliminate any single point of failure. Use multiple VMkernel network interfaces on different network subnets that use NIC teaming when multiple virtual switches are used, or use multiple physical NICs connected to multiple physical switches to provide HA and increased throughput.

  • Configure zoning, VLANs, or both as required for host connectivity.

  • Verify that all required initiators are logged into the target LIFs on the desired SVM.

Note You must deploy ONTAP tools for VMware vSphere to enable the VASA Provider. The VASA Provider will manage all of your iGroup settings for you, therefore there is no need to create or manage iGroups in a vVols environment.

NetApp does not recommend changing any vVols settings from default at this time.

Refer to the NetApp Interoperability Matrix Tool for specific versions of ONTAP tools, or legacy VASA Provider for your specific versions of vSphere and ONTAP.

For detailed information on provisioning and managing vVols, refer to ONTAP tools for VMware vSphere documentation, TR-4597, and TR-4400.

ATS locking

ATS locking is mandatory for VAAI compatible storage and upgraded VMFS5 and is required for proper interoperability and optimal VMFS shared storage I/O performance with ONTAP LUNs. Refer to VMware documentation for details on enabling ATS locking.

Settings Default ONTAP Recommended Description

HardwareAcceleratedLocking

1

1

Helps enable the use of Atomic Test and Set (ATS) locking

Disk IOPs

1000

1

IOPS limit: The Round Robin PSP defaults to an IOPS limit of 1000. In this default case, a new path is used after 1000 I/O operations are issued.

Disk/QFullSampleSize

0

32

The count of QUEUE FULL or BUSY conditions it takes before ESXi starts throttling.
Note Enable Space-alloc setting for all the LUNs mapped to VMware vSphere for UNMAP to work. For more details, refer to ONTAP Documentation.

Guest OS timeouts

You can manually configure the virtual machines with the recommended guest OS tunings. After tuning updates, you must reboot the guest for the updates to take effect.

GOS timeout values:

Guest OS Type Timeouts

Linux variants

disk timeout = 60

Windows

disk timeout = 60

Solaris

disk timeout = 60
busy retry = 300
not ready retry = 300
reset retry = 30
max.throttle = 32
min.throttle = 8

Validate the vSphere tunable

You can use the following command to verify the HardwareAcceleratedLocking setting.

esxcli system settings advanced list --option /VMFS3/HardwareAcceleratedLocking

   Path: /VMFS3/HardwareAcceleratedLocking
   Type: integer
   Int Value: 1
   Default Int Value: 1
   Min Value: 0
   Max Value: 1
   String Value:
   Default String Value:
   Valid Characters:
   Description: Enable hardware accelerated VMFS locking (requires compliant hardware). Please see http://kb.vmware.com/kb/2094604 before disabling this option.

Validate the Disk IOPs setting

You can use the following command to verify the IOPs setting.

esxcli storage nmp device list -d naa.600a098038304731783f506670553355

naa.600a098038304731783f506670553355
   Device Display Name: NETAPP Fibre Channel Disk (naa.600a098038304731783f506670553355)
   Storage Array Type: VMW_SATP_ALUA
   Storage Array Type Device Config: {implicit_support=on; explicit_support=off; explicit_allow=on; alua_followover=on; action_OnRetryErrors=off; {TPG_id=1000,TPG_state=ANO}{TPG_id=1001,TPG_state=AO}}
   Path Selection Policy: VMW_PSP_RR
   Path Selection Policy Device Config: {policy=rr,iops=1,bytes=10485760,useANO=0; lastPathIndex=0: NumIOsPending=0,numBytesPending=0}
   Path Selection Policy Device Custom Config:
   Working Paths: vmhba4:C0:T0:L82, vmhba3:C0:T0:L82
   Is USB: false

Validate the QFullSampleSize

You can use the following command to verify the QFullSampleSize.

esxcli system settings advanced list --option /Disk/QFullSampleSize

   Path: /Disk/QFullSampleSize
   Type: integer
   Int Value: 32
   Default Int Value: 0
   Min Value: 0
   Max Value: 64
   String Value:
   Default String Value:
   Valid Characters:
   Description: Default I/O samples to monitor for detecting non-transient queue full condition. Should be nonzero to enable queue depth throttling. Device specific QFull options will take precedence over this value if set.

Known issues

The VMware vSphere 8.x with ONTAP release has the following known issues:

NetApp Bug ID Title Description

1543660

I/O error occurs when Linux VMs using vNVMe adapters encounter a long all paths down (APD) window

Linux VMs running vSphere 8.x and later and using virtual NVMe (vNVME) adapters encounter an I/O error because the vNVMe retry operation is disabled by default. To avoid a disruption on Linux VMs running older kernels during an all paths down (APD) or a heavy I/O load, VMware has introduced a tunable "VSCSIDisableNvmeRetry" to disable the vNVMe retry operation.
Related information