NVMe-oF Host Configuration for ESXi 7.0 with ONTAP

09/07/2021 Contributors Download PDF of this page

Supportability

NVME/FC is supported on ONTAP 9.7 or later for ESXi 7.0.

ESXi initiator host can run both NVMe/FC & FCP traffic through the same adapter ports. See the Hardware Universe for a list of supported FC adapters and controllers. For the most current list of supported configurations & versions, see the NetApp Interoperability Matrix.

Known limitations

The following are not supported:

RDM mapping
VVols

Enabling NVMe/FC with ANA

Disable the HppManageDegradedPaths parameter for improved interoperability with ONTAP:
```
# esxcfg-advcfg -s 0 /Misc/HppManageDegradedPaths
```
Reboot the host.

After reboot, verify that the HppManageDegradedPaths parameter is now disabled:

# esxcfg-advcfg -g /Misc/HppManageDegradedPaths
Value of HppManageDegradedPaths is 0

Check the ESXi host NQN string and verify that it matches with the host NQN string for the corresponding subsystem on the ONTAP array.

Example

# esxcli nvme info get
Host NQN: nqn.2014-08.com.vmware:nvme:chat-54-113

*> vserver nvme subsystem host show -vserver co_nv_fc_esx
Vserver Subsystem Host NQN
------- --------- ----------------------------------------------------------
co_nv_fc_esx
        subsys_chat_54_113_nvme
                  nqn.2014-08.com.vmware:nvme:chat-54-113

Configuring the Broadcom FC adapter for NVMe/FC

Install the recommended lpfc driver by copying it to a temporary folder and then executing the following command:

# esxcli software vib install -d /tmp/t/Emulex-FCoE-FC-lpfc-12.4.224.0-offline-bundle-13621872.zip --no-sig-check
Installation Result
   Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.
   Reboot Required: true
   VIBs Installed: EMU_bootbank_lpfc_12.4.224.0-1OEM.688.0.0.13621872
   VIBs Removed: EMU_bootbank_lpfc_12.4.211.6-1OEM.688.0.0.13621872
   VIBs Skipped:

If necessary, set the lpfc driver parameter lpfc_enable_fc4_type=3 for enabling NVMe/FC support in the lpfc driver:

This parameter is set by default for the LPe35000-series adapters. You must perform the following step to set it manually for LPe32000-series & LPe31000-series adapters.
```
# esxcli system module parameters set -m lpfc -p lpfc_enable_fc4_type=3
```

Use the elxmgmt utility to upgrade the Broadcom FC adapter firmware to the recommended version:

# esxcli software vib install -d /tmp/t/Emulex-elxmgmt-6.8.7-12.4.211.7.zip --no-sig-check
Installation Result
   Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.
   Reboot Required: true
   VIBs Installed: EMU_bootbank_emu-esx-elxmgmt_12.4.211.7-01
   VIBs Removed:
   VIBs Skipped:
…

Reboot the host.

After reboot, verify that the recommended lpfc driver and adapter firmware versions have applied and the initiator ports are online:

# esxcli storage san fc list
 Adapter: vmhba3
   Port ID: 010600
   Node Name: 20:00:00:90:fa:e0:ec:8e
   Port Name: 10:00:00:90:fa:e0:ec:8e
   Speed: 32 Gbps
   Port Type: NPort
   Port State: ONLINE
   Model Description: Emulex LightPulse LPe32002-M2 2-Port 32Gb Fibre Channel Adapter
   Hardware Version: 0000000c
   OptionROM Version: 12.4.217.2
   Firmware Version: 12.4.217.2
   Driver Name: lpfc
   DriverVersion: 12.4.224.0

   Adapter: vmhba4
   Port ID: 010F00
   Node Name: 20:00:00:90:fa:e0:ec:8f
   Port Name: 10:00:00:90:fa:e0:ec:8f
   Speed: 32 Gbps
   Port Type: NPort
   Port State: ONLINE
   Model Description: Emulex LightPulse LPe32002-M2 2-Port 32Gb Fibre Channel Adapter
   Hardware Version: 0000000c
   OptionROM Version: 12.4.217.2
   Firmware Version: 12.4.217.2
   Driver Name: lpfc
   DriverVersion: 12.4.224.0

Validating NVMe/FC

Verify that the ONTAP target NVMe/FC controllers are properly discovered on the ESXi host:

# esxcli nvme controller list

Name                                                                                                                             Controller Number  Adapter  Transport Type  Is Online
-------------------------------------------------------------------------------------------------------------------------------  -----------------  -------  --------------  ---------
nqn.1992-08.com.netapp:sn.e7f89c2c245d11e9975300a098dfce55:subsystem.interop_57_vm_01#vmhba32#204900a098dfe3d1:204a00a098dfe3d1                259  vmhba32  FC                  false
nqn.1992-08.com.netapp:sn.e7f89c2c245d11e9975300a098dfce55:subsystem.interop_57_vm_09#vmhba32#204900a098dfe3d1:204a00a098dfe3d1                263  vmhba32  FC                  false
nqn.1992-08.com.netapp:sn.e7f89c2c245d11e9975300a098dfce55:subsystem.interop_57_vm_11#vmhba32#204900a098dfe3d1:204a00a098dfe3d1                267  vmhba32  FC                  false
nqn.1992-08.com.netapp:sn.e7f89c2c245d11e9975300a098dfce55:subsystem.interop_57_vm_10#vmhba32#204900a098dfe3d1:204a00a098dfe3d1                265  vmhba32  FC                  false
nqn.1992-08.com.netapp:sn.e7f89c2c245d11e9975300a098dfce55:subsystem.interop_57_vm_02#vmhba32#204900a098dfe3d1:204a00a098dfe3d1                261  vmhba32  FC                  false

Verify that the NVMe/FC namespaces are properly created:

The UUIDs in the following example represent the NVMe/FC namespace devices.

#esxcfg-mpath -b
uuid.0d12b7cd97344be8a53b7913f8f72f04 : NVMe Fibre Channel Disk (uuid.0d12b7cd97344be8a53b7913f8f72f04)
   vmhba65:C0:T9:L30 LUN:30 state:active fc Adapter: WWNN: 20:00:00:90:fa:e0:ec:8f WWPN: 10:00:00:90:fa:e0:ec:8f  Target: WWNN: 20:49:00:a0:98:df:e3:d1 WWPN: 20:4d:00:a0:98:df:e3:d1
   vmhba64:C0:T9:L30 LUN:30 state:active fc Adapter: WWNN: 20:00:00:90:fa:e0:ec:8e WWPN: 10:00:00:90:fa:e0:ec:8e  Target: WWNN: 20:49:00:a0:98:df:e3:d1 WWPN: 20:4c:00:a0:98:df:e3:d1
   vmhba64:C0:T5:L30 LUN:30 state:standby fc Adapter: WWNN: 20:00:00:90:fa:e0:ec:8e WWPN: 10:00:00:90:fa:e0:ec:8e  Target: WWNN: 20:49:00:a0:98:df:e3:d1 WWPN: 20:4a:00:a0:98:df:e3:d1
   vmhba65:C0:T0:L30 LUN:30 state:standby fc Adapter: WWNN: 20:00:00:90:fa:e0:ec:8f WWPN: 10:00:00:90:fa:e0:ec:8f  Target: WWNN: 20:49:00:a0:98:df:e3:d1 WWPN: 20:4b:00:a0:98:df:e3:d1

uuid.49de7683950d47c9898f51443d893910 : NVMe Fibre Channel Disk (uuid.49de7683950d47c9898f51443d893910)
   vmhba65:C0:T12:L39 LUN:39 state:active fc Adapter: WWNN: 20:00:00:90:fa:e0:ec:8f WWPN: 10:00:00:90:fa:e0:ec:8f  Target: WWNN: 20:3a:00:a0:98:df:e3:d1 WWPN: 20:27:00:a0:98:df:e3:d1
   vmhba65:C0:T13:L39 LUN:39 state:standby fc Adapter: WWNN: 20:00:00:90:fa:e0:ec:8f WWPN: 10:00:00:90:fa:e0:ec:8f  Target: WWNN: 20:3a:00:a0:98:df:e3:d1 WWPN: 20:29:00:a0:98:df:e3:d1
   vmhba64:C0:T12:L39 LUN:39 state:active fc Adapter: WWNN: 20:00:00:90:fa:e0:ec:8e WWPN: 10:00:00:90:fa:e0:ec:8e  Target: WWNN: 20:3a:00:a0:98:df:e3:d1 WWPN: 20:3b:00:a0:98:df:e3:d1
   vmhba64:C0:T13:L39 LUN:39 state:standby fc Adapter: WWNN: 20:00:00:90:fa:e0:ec:8e WWPN: 10:00:00:90:fa:e0:ec:8e  Target: WWNN: 20:3a:00:a0:98:df:e3:d1 WWPN: 20:28:00:a0:98:df:e3:d1

In ONTAP 9.7, the default block size for a NVMe/FC namespace is 4K. This default size is not compatible with ESXi. Therefore, when creating namespaces for ESXi, you must set the namespace block size 512b. You can do this using the vserver nvme namespace create command.

Example

vserver nvme namespace create -vserver vs_1 -path /vol/nsvol/namespace1 -size 100g -ostype vmware -block-size 512B

Refer to the ONTAP 9 Command man pages for additional details.

Verify the status of the individual ANA paths of the respective NVMe/FC namespace devices:

# esxcli storage hpp path list
fc.20000090fae0ec8f:10000090fae0ec8f-fc.204900a098dfe3d1:204d00a098dfe3d1-uuid.1aa669c5376240a28ae47d8d549586ea
   Runtime Name: vmhba65:C0:T9:L33
   Device: uuid.1aa669c5376240a28ae47d8d549586ea
   Device Display Name: NVMe Fibre Channel Disk (uuid.1aa669c5376240a28ae47d8d549586ea)
   Path State: active

fc.20000090fae0ec8e:10000090fae0ec8e-fc.204900a098dfe3d1:204a00a098dfe3d1-uuid.1aa669c5376240a28ae47d8d549586ea
   Runtime Name: vmhba64:C0:T5:L33
   Device: uuid.1aa669c5376240a28ae47d8d549586ea
   Device Display Name: NVMe Fibre Channel Disk (uuid.1aa669c5376240a28ae47d8d549586ea)
   Path State: standby