Using VMware vSphere 7.x with NetApp ONTAP

Contributors Download PDF of this page

Introduction

This document provides guidance on ONTAP SAN host configuration settings for the vSphere 7.x release with FC, FCoE and iSCSI protocols.

Hypervisor SAN Booting

Before you begin

If you decide to use SAN booting, it must be supported by your configuration. You can use the NetApp Interoperability Matrix Tool to verify that your OS, HBA, HBA firmware and the HBA boot BIOS, and ONTAP version are supported.

Steps
  1. Map the SAN boot LUN to the host.

  2. Verify multiple paths are available.

    Remember, multiple paths will only be available after the host OS is up and running on the paths.

  3. Enable SAN booting in the server BIOS for the ports to which the SAN boot LUN is mapped.

    For information on how to enable the HBA BIOS, see your vendor-specific documentation.

  4. Reboot the host to verify the boot is successful.

Multipathing

ESXi provides an extensible multipathing module called Native Multipathing Plug-In (NMP) that manages the sub-plugins Storage Array Type Plugins (SATPs), and Path Selection Plugins (PSPs). These SATP rules are available by default in ESXi.

For NetApp ONTAP storage, VMW_SATP_ALUA plugin is used by default with VMW_PSP_RR as a path selection policy (PSP). This can be confirmed by using the below command.

esxcli storage nmp satp rule list -s VMW_SATP_ALUA

Name           Device  Vendor    Model         Driver  Transport  Options
-------------  ------  --------  ------------  ------  ---------  --------------------------
VMW_SATP_ALUA          NETAPP                                     reset_on_attempted_reserve


Rule Group  Claim Options  Default PSP  PSP Options     Description
 ----------  -------------  -----------  --------------  -----------
system      tpgs_on        VMW_PSP_RR                   NetApp arrays with ALUA support

Non-ASA Configuration

For non-ASA configuration there should be two groups of paths with different priorities. The paths with the higher priorities are Active/Optimized, meaning they are serviced by the controller where the aggregate is located. The paths with the lower priorities are active but are non-optimized because they are served from a different controller. The non-optimized paths are only used when no optimized paths are available.

Example

The following example displays the correct output for an ONTAP LUN with two Active/Optimized paths and two Active/Non-Optimized paths:

esxcli storage nmp device list -d naa.600a098038313530772b4d673979372f

naa.600a098038313530772b4d673979372f
   Device Display Name: NETAPP Fibre Channel Disk (naa.600a098038313530772b4d673979372f)
   Storage Array Type: VMW_SATP_ALUA
   Storage Array Type Device Config: {implicit_support=on; explicit_support=off; explicit_allow=on; alua_followover=on; action_OnRetryErrors=off; {TPG_id=1000,TPG_state=AO}{TPG_id=1001,TPG_state=ANO}}
   Path Selection Policy: VMW_PSP_RR
   Path Selection Policy Device Config: {policy=rr,iops=1,bytes=10485760,useANO=0; lastPathIndex=1: NumIOsPending=0,numBytesPending=0}
   Path Selection Policy Device Custom Config:
   Working Paths: vmhba3:C0:T3:L21, vmhba4:C0:T2:L21
   Is USB: false

esxcli storage nmp path list -d naa.600a098038313530772b4d673979372f

fc.20000090fae0ec8e:10000090fae0ec8e-fc.201000a098dfe3d1:200b00a098dfe3d1-naa.600a098038313530772b4d673979372f
   Runtime Name: vmhba3:C0:T2:L21
   Device: naa.600a098038313530772b4d673979372f
   Device Display Name: NETAPP Fibre Channel Disk (naa.600a098038313530772b4d673979372f)
   Group State: active unoptimized
   Array Priority: 0
   Storage Array Type Path Config: {TPG_id=1001,TPG_state=ANO,RTP_id=29,RTP_health=UP}
   Path Selection Policy Path Config: PSP VMW_PSP_RR does not support path configuration.

fc.20000090fae0ec8e:10000090fae0ec8e-fc.201000a098dfe3d1:200700a098dfe3d1-naa.600a098038313530772b4d673979372f
   Runtime Name: vmhba3:C0:T3:L21
   Device: naa.600a098038313530772b4d673979372f
   Device Display Name: NETAPP Fibre Channel Disk (naa.600a098038313530772b4d673979372f)
   Group State: active
   Array Priority: 0
   Storage Array Type Path Config: {TPG_id=1000,TPG_state=AO,RTP_id=25,RTP_health=UP}
   Path Selection Policy Path Config: PSP VMW_PSP_RR does not support path configuration.

fc.20000090fae0ec8f:10000090fae0ec8f-fc.201000a098dfe3d1:200800a098dfe3d1-naa.600a098038313530772b4d673979372f
   Runtime Name: vmhba4:C0:T2:L21
   Device: naa.600a098038313530772b4d673979372f
   Device Display Name: NETAPP Fibre Channel Disk (naa.600a098038313530772b4d673979372f)
   Group State: active
   Array Priority: 0
   Storage Array Type Path Config: {TPG_id=1000,TPG_state=AO,RTP_id=26,RTP_health=UP}
   Path Selection Policy Path Config: PSP VMW_PSP_RR does not support path configuration.

fc.20000090fae0ec8f:10000090fae0ec8f-fc.201000a098dfe3d1:200c00a098dfe3d1-naa.600a098038313530772b4d673979372f
   Runtime Name: vmhba4:C0:T3:L21
   Device: naa.600a098038313530772b4d673979372f
   Device Display Name: NETAPP Fibre Channel Disk (naa.600a098038313530772b4d673979372f)
   Group State: active unoptimized
   Array Priority: 0
   Storage Array Type Path Config: {TPG_id=1001,TPG_state=ANO,RTP_id=30,RTP_health=UP}
   Path Selection Policy Path Config: PSP VMW_PSP_RR does not support path configuration.

All SAN Array Configuration

In All SAN Array (ASA) configurations, all paths to a given Logical Unit (LUN) are active and optimized. This means I/O can be served through all paths at the same time, thereby enabling better performance.

Example

The following example displays the correct output for an ONTAP LUN:

esxcli storage nmp device list -d naa.600a098038304759563f4e7837574453

naa.600a098038304759563f4e7837574453
   Device Display Name: NETAPP Fibre Channel Disk (naa.600a098038304759563f4e7837574453)
   Storage Array Type: VMW_SATP_ALUA
   Storage Array Type Device Config: {implicit_support=on; explicit_support=off; explicit_allow=on; alua_followover=on; action_OnRetryErrors=off; {TPG_id=1001,TPG_state=AO}{TPG_id=1000,TPG_state=AO}}
   Path Selection Policy: VMW_PSP_RR
   Path Selection Policy Device Config: {policy=rr,iops=1,bytes=10485760,useANO=0; lastPathIndex=2: NumIOsPending=0,numBytesPending=0}
   Path Selection Policy Device Custom Config:
   Working Paths: vmhba4:C0:T0:L9, vmhba3:C0:T1:L9, vmhba3:C0:T0:L9, vmhba4:C0:T1:L9
   Is USB: false

esxcli storage nmp device list -d naa.600a098038304759563f4e7837574453

fc.20000024ff171d37:21000024ff171d37-fc.202300a098ea5e27:204a00a098ea5e27-naa.600a098038304759563f4e7837574453
   Runtime Name: vmhba4:C0:T0:L9
   Device: naa.600a098038304759563f4e7837574453
   Device Display Name: NETAPP Fibre Channel Disk (naa.600a098038304759563f4e7837574453)
   Group State: active
   Array Priority: 0
   Storage Array Type Path Config: {TPG_id=1000,TPG_state=AO,RTP_id=6,RTP_health=UP}
   Path Selection Policy Path Config: PSP VMW_PSP_RR does not support path configuration.

fc.20000024ff171d36:21000024ff171d36-fc.202300a098ea5e27:201d00a098ea5e27-naa.600a098038304759563f4e7837574453
   Runtime Name: vmhba3:C0:T1:L9
   Device: naa.600a098038304759563f4e7837574453
   Device Display Name: NETAPP Fibre Channel Disk (naa.600a098038304759563f4e7837574453)
   Group State: active
   Array Priority: 0
   Storage Array Type Path Config: {TPG_id=1001,TPG_state=AO,RTP_id=3,RTP_health=UP}
   Path Selection Policy Path Config: PSP VMW_PSP_RR does not support path configuration.

fc.20000024ff171d36:21000024ff171d36-fc.202300a098ea5e27:201b00a098ea5e27-naa.600a098038304759563f4e7837574453
   Runtime Name: vmhba3:C0:T0:L9
   Device: naa.600a098038304759563f4e7837574453
   Device Display Name: NETAPP Fibre Channel Disk (naa.600a098038304759563f4e7837574453)
   Group State: active
   Array Priority: 0
   Storage Array Type Path Config: {TPG_id=1000,TPG_state=AO,RTP_id=1,RTP_health=UP}
   Path Selection Policy Path Config: PSP VMW_PSP_RR does not support path configuration.

fc.20000024ff171d37:21000024ff171d37-fc.202300a098ea5e27:201e00a098ea5e27-naa.600a098038304759563f4e7837574453
   Runtime Name: vmhba4:C0:T1:L9
   Device: naa.600a098038304759563f4e7837574453
   Device Display Name: NETAPP Fibre Channel Disk (naa.600a098038304759563f4e7837574453)
   Group State: active
   Array Priority: 0
   Storage Array Type Path Config: {TPG_id=1001,TPG_state=AO,RTP_id=4,RTP_health=UP}
   Path Selection Policy Path Config: PSP VMW_PSP_RR does not support path configuration.

vVol

Virtual Volumes (vVols) are a VMware object type that corresponds to a Virtual Machine (VM) disk, and its snapshots and fast-clones.

ONTAP tools for VMware vSphere includes the VASA Provider for ONTAP, which provides the integration point for a VMware vCenter to leverage vVols based storage. When you deploy the ONTAP tools OVA, it is automatically registered with the vCenter server and enables the VASA Provider.

When you create a vVols datastore using the vCenter user interface, it guides you to create FlexVols as backup storage for the datastore. vVols within a vVols datastores are access by ESXi hosts using a protocol endpoint (PE). In SAN environments, one 4MB LUN is created on each FlexVol in the datastore for use as a PE. A SAN PE is an administrative logical unit (ALU). vVols are subsidiary logical units (SLUs).

Standard requirements and best practices for SAN environments apply when using vVols, including (but not limited to) the following:

  1. Create at least one SAN LIF on each node per SVM you intend to use. The best practice is to create at least two per node, but no more than necessary.

  2. Eliminate any single point of failure. use multiple VMkernel network interfaces on different network subnets that use NIC teaming when multiple virtual switches are used. Or use multiple physical NICs connected to multiple physical switches to provide HA and increased throughput.

  3. Configure zoning and/or VLANs as required for host connectivity.

  4. Ensure all required initiators are logged into the target LIFs on the desired SVM.

Note You must deploy ONTAP tools for VMware vSphere to enabled the VASA Provider. The VASA Provider will manage all of your igroup settings for you, so there is no need to create or manage igroups in a vVols environment.

NetApp does not recommend changing any vVols settings from the default at this time.

Refer to the NetApp Interoperability Matrix Tool for specific versions of ONTAP tools, or legacy VASA Provider for your specific versions of vSphere and ONTAP.

For detailed information on provisioning and managing vVols, please refer to ONTAP tools for VMware vSphere documentation as well as TR-4597 and TR-4400.

ATS Locking

ATS locking is mandatory for VAAI compatible storage and upgraded VMFS5, and is required for proper interoperability and optimal VMFS shared storage I/O performance with ONTAP LUNs. Refer to VMware documentation for details on enabling ATS locking.

Settings Default ONTAP Recommended Description

HardwareAcceleratedLocking

1

1

Helps enable the use of Atomic Test and Set (ATS) locking

Disk IOPs

1000

1

IOPS limit: The Round Robin PSP defaults to an IOPS limit of 1000. In this default case, a new path is used after 1000 I/O operations are issued.

Disk/QFullSampleSize

0

32

The count of QUEUE FULL or BUSY conditions it takes before ESXi starts throttling.

Note Enable Space-alloc setting for all the LUN’s mapped to VMware vSphere for UNMAP to work. For More details, refer to ONTAP Documentation.

Guest OS timeouts

You can manually configure the virtual machines with the recommended guest OS tunings. After tuning updates, you must reboot the guest for the updates to take effect.

GOS timeout values:

Guest OS Type Timeouts

Linux variants

disk timeout = 60

Windows

disk timeout = 60

Solaris

disk timeout = 60
busy retry = 300
not ready retry = 300
reset retry = 30
max.throttle = 32
min.throttle = 8

Validating the vSphere tunable

Use the following command to verify the HardwareAcceleratedLocking setting.

esxcli system settings advanced list --option /VMFS3/HardwareAcceleratedLocking

   Path: /VMFS3/HardwareAcceleratedLocking
   Type: integer
   Int Value: 1
   Default Int Value: 1
   Min Value: 0
   Max Value: 1
   String Value:
   Default String Value:
   Valid Characters:
   Description: Enable hardware accelerated VMFS locking (requires compliant hardware). Please see http://kb.vmware.com/kb/2094604 before disabling this option.

Validating the Disk IOPs setting

Use the following command to verify the IOPs setting.

esxcli storage nmp device list -d naa.600a098038304731783f506670553355

naa.600a098038304731783f506670553355
   Device Display Name: NETAPP Fibre Channel Disk (naa.600a098038304731783f506670553355)
   Storage Array Type: VMW_SATP_ALUA
   Storage Array Type Device Config: {implicit_support=on; explicit_support=off; explicit_allow=on; alua_followover=on; action_OnRetryErrors=off; {TPG_id=1000,TPG_state=ANO}{TPG_id=1001,TPG_state=AO}}
   Path Selection Policy: VMW_PSP_RR
   Path Selection Policy Device Config: {policy=rr,iops=1,bytes=10485760,useANO=0; lastPathIndex=0: NumIOsPending=0,numBytesPending=0}
   Path Selection Policy Device Custom Config:
   Working Paths: vmhba4:C0:T0:L82, vmhba3:C0:T0:L82
   Is USB: false

Validating the QFullSampleSize

Use the following command to verify the QFullSampleSize

esxcli system settings advanced list --option /Disk/QFullSampleSize

   Path: /Disk/QFullSampleSize
   Type: integer
   Int Value: 32
   Default Int Value: 0
   Min Value: 0
   Max Value: 64
   String Value:
   Default String Value:
   Valid Characters:
   Description: Default I/O samples to monitor for detecting non-transient queue full condition. Should be nonzero to enable queue depth throttling. Device specific QFull options will take precedence over this value if set.

Known Issues

There are no known issues.