NVMe/FC Host Configuration for Oracle Linux 8.3 with ONTAP
You can configure NVMe over Fibre Channel (NVMe/FC) on initiator hosts running Oracle Linux 8.3 and ONTAP as the target.
Supportability
NVMe over Fabrics or NVMe-oF (including NVMe/FC) is supported with Oracle Linux 8.3 with Asymmetric Namespace Access (ANA) required for surviving storage failovers (SFOs) on the ONTAP array. ANA is the ALUA equivalent in the NVMe-oF environment and is currently implemented with in-kernel NVMe Multipath. Using this procedure, you can enable NVMe-oF with in-kernel NVMe Multipath using ANA on OL 8.3 and ONTAP as the target.
You can use the configuration settings provided in this document to configure cloud clients connected to Cloud Volumes ONTAP and Amazon FSx for ONTAP. |
Features
-
Oracle Linux 8.3 has in-kernel NVMe multipath enabled by default for NVMe namespaces.
-
With Oracle Linux 8.3,
nvme-fc auto-connect
scripts are included in the native nvme-cli package. You can use these native auto-connect scripts instead of installing external vendor provided outbox auto-connect scripts. -
With Oracle Linux 8.3, a native
udev
rule is provided as part of thenvme-cli
package which enables round-robin load balancing for NVMe multipath. Therefore, you do not need to manually create this rule anymore. -
With Oracle Linux 8.3, both NVMe and SCSI traffic can be run on the same co-existent host. In fact, that is expected to be the commonly deployed host configuration. So for SCSI, you can configure dm-multipath as usual for SCSI LUNs resulting in mpath devices whereas NVMe multipath can be used to configure NVMe-oF multipath devices (for example,
/dev/nvmeXnY
) on the host. -
With Oracle Linux 8.3, the NetApp plugin in the native
nvme-cli
package is capable of displaying ONTAP details as well as ONTAP namespaces.
Known limitations
SAN booting using the NVMe-oF protocol is currently not supported.
Configuration requirements
Refer to the NetApp Interoperability Matrix Tool (IMT) for the current list of supported configurations.
Enable NVMe/FC with Oracle Linux 8.3
-
Install Oracle Linux 8.3 GA on the server. After the installation is complete, verify that you are running the specified Oracle Linux 8.3 GA kernel. See the NetApp Interoperability Matrix Tool for the most current list of supported versions.
# uname -r 5.4.17-2011.7.4.el8uek.x86_64
-
Install the
nvme-cli
package.# rpm -qa|grep nvme-cli nvme-cli-1.12-2.el8.x86_64_
-
On the Oracle Linux 8.3 host, check the hostnqn string at
/etc/nvme/hostnqn
and verify that it matches thehostnqn
string for the corresponding subsystem on the ONTAP array.#cat /etc/nvme/hostnqn nqn.2014-08.org.nvmexpress:uuid:446c21ab-f4c1-47ed-9a8f-1def96f3fed2 ::> vserver nvme subsystem host show -vserver vs_coexistance_2 Vserver Subsystem Host NQN ------- --------- ---------------------------------------------------------- vs_coexistance_2 nvme_1 nqn.2014-08.org.nvmexpress:uuid:446c21ab-f4c1-47ed-9a8f-1def96f3fed2
If the hostnqn strings do not match, you should use the vserver modify
command to update the hostnqn string on your corresponding ONTAP array subsystem to match to the hostnqn string from/etc/nvme/hostnqn
on the host. -
Reboot the host.
If you intend to run both NVMe and SCSI traffic on the same Oracle Linux 8.3 co-existent host, NetApp recommends using the in-kernel NVMe multipath for ONTAP namespaces and
dm-multipath
for ONTAP LUNs respectively. This also means the ONTAP namespaces should be blacklisted indm-multipath
to preventdm-multipath
from claiming these namespace devices. This can be done by adding the enable_foreign setting to the/etc/multipath.conf
file:#cat /etc/multipath.conf defaults { enable_foreign NONE }
Restart the multipathd daemon by running the systemctl restart multipathd command to let the new setting take effect.
Configure the Broadcom FC adapter for NVMe/FC
-
Verify that you are using the supported adapter. For the most current list of supported adapters, see the NetApp Interoperability Matrix Tool.
#cat /sys/class/scsi_host/host*/modelname LPe36002-M2 LPe36002-M2
#cat /sys/class/scsi_host/host*/modeldesc Emulex LPe36002-M64 2-Port 64Gb Fibre Channel Adapter Emulex LPe36002-M64 2-Port 64Gb Fibre Channel Adapter
-
Verify that you are using the recommended Broadcom lpfc firmware and inbox driver. For the most current list of supported adapter drivers and firmware versions, see the NetApp Interoperability Matrix Tool.
#cat /sys/class/scsi_host/host*/fwrev 12.8.351.49, sli-4:6:d 12.8.351.49, sli-4:6:d
#cat /sys/module/lpfc/version 0:12.6.0.3
-
Verify that the
lpfc_enable_fc4_type
parameter is set to 3.#cat /sys/module/lpfc/parameters/lpfc_enable_fc4_type 3
-
Verify that the initiator ports are up and running, and that you can see the target LIFs.
#cat /sys/class/fc_host/host*/port_name 0x100000109bf0447b 0x100000109bf0447c
#cat /sys/class/fc_host/host*/port_state Online Online
#cat /sys/class/scsi_host/host*/nvme_info NVME Initiator Enabled XRI Dist lpfc0 Total 6144 IO 5894 ELS 250 NVME LPORT lpfc0 WWPN x100000109bf0447b WWNN x200000109bf0447b DID x022400 ONLINE NVME RPORT WWPN x20e1d039ea243510 WWNN x20e0d039ea243510 DID x0a0314 TARGET DISCSRVC ONLINE NVME RPORT WWPN x20e4d039ea243510 WWNN x20e0d039ea243510 DID x0a0713 TARGET DISCSRVC ONLINE NVME Statistics LS: Xmt 00000003b6 Cmpl 00000003b6 Abort 00000000 LS XMIT: Err 00000000 CMPL: xb 00000000 Err 00000000 Total FCP Cmpl 00000000be1425e8 Issue 00000000be1425f2 OutIO 000000000000000a abort 00000251 noxri 00000000 nondlp 00000000 qdepth 00000000 wqerr 00000000 err 00000000 FCP CMPL: xb 00000c5b Err 0000d176 NVME Initiator Enabled XRI Dist lpfc1 Total 6144 IO 5894 ELS 250 NVME LPORT lpfc1 WWPN x100000109bf0447c WWNN x200000109bf0447c DID x021600 ONLINE NVME RPORT WWPN x20e2d039ea243510 WWNN x20e0d039ea243510 DID x0a0213 TARGET DISCSRVC ONLINE NVME RPORT WWPN x20e3d039ea243510 WWNN x20e0d039ea243510 DID x0a0614 TARGET DISCSRVC ONLINE NVME Statistics LS: Xmt 0000000419 Cmpl 0000000419 Abort 00000000 LS XMIT: Err 00000000 CMPL: xb 00000000 Err 00000000 Total FCP Cmpl 00000000be37ff65 Issue 00000000be37ff84 OutIO 000000000000001f abort 0000025a noxri 00000000 nondlp 00000000 qdepth 00000000 wqerr 00000000 err 00000000 FCP CMPL: xb 00000c89 Err 0000cd87
Enable 1MB I/O size
ONTAP reports an MDTS (Max Data Transfer Size) of 8 in the Identify Controller data. This means the maximum I/O request size can be up to 1MB. To issue I/O requests of size 1 MB for a Broadcom NVMe/FC host, you should increase the lpfc
value of the lpfc_sg_seg_cnt
parameter to 256 from the default value of 64.
The following steps don't apply to Qlogic NVMe/FC hosts. |
-
Set the
lpfc_sg_seg_cnt
parameter to 256:cat /etc/modprobe.d/lpfc.conf
options lpfc lpfc_sg_seg_cnt=256
-
Run the
dracut -f
command, and reboot the host: -
Verify that
lpfc_sg_seg_cnt
is 256:cat /sys/module/lpfc/parameters/lpfc_sg_seg_cnt
The expected value is 256.
Configure the Marvell/QLogic FC adapter for NVMe/FC
The native inbox qla2xxx driver included in the OL 8.3 GA kernel has the latest upstream fixes. These fixes are essential for ONTAP support.
-
Verify that you are running the supported adapter driver and firmware versions:
#cat /sys/class/fc_host/host*/symbolic_name QLE2742 FW:v9.10.11 DVR:v10.01.00.25-k QLE2742 FW:v9.10.11 DVR:v10.01.00.25-k
-
Verify that the
ql2xnvmeenable
parameter is set which enables the Marvell adapter to function as an NVMe/FC initiator.#cat /sys/module/qla2xxx/parameters/ql2xnvmeenable 1
Validate NVMe/FC
-
Verify the following NVMe/FC settings on the Oracle Linux 8.3 host.
#cat /sys/module/nvme_core/parameters/multipath Y #cat /sys/class/nvme-subsystem/nvme-subsys*/model NetApp ONTAP Controller NetApp ONTAP Controller #cat /sys/class/nvme-subsystem/nvme-subsys*/iopolicy round-robin round-robin
-
Verify that the namespaces are created and correctly discoverd on the host.
# nvme list Node SN Model Namespace Usage Format FW Rev ---------------- -------------------- ---------------------------------------- --------- ------------------ /dev/nvme0n1 81Ec-JRMlkL9AAAAAAAB NetApp ONTAP Controller 1 37.58 GB / 37.58 GB 4 KiB + 0 B FFFFFFFF /dev/nvme0n10 81Ec-JRMlkL9AAAAAAAB NetApp ONTAP Controller 10 37.58 GB / 37.58 GB 4 KiB + 0 B FFFFFFFF /dev/nvme0n11 81Ec-JRMlkL9AAAAAAAB NetApp ONTAP Controller 11 37.58 GB / 37.58 GB 4 KiB + 0 B FFFFFFFF /dev/nvme0n12 81Ec-JRMlkL9AAAAAAAB NetApp ONTAP Controller 12 37.58 GB / 37.58 GB 4 KiB + 0 B FFFFFFFF /dev/nvme0n13 81Ec-JRMlkL9AAAAAAAB NetApp ONTAP Controller 13 37.58 GB / 37.58 GB 4 KiB + 0 B FFFFFFFF /dev/nvme0n14 81Ec-JRMlkL9AAAAAAAB NetApp ONTAP Controller 14 37.58 GB / 37.58 GB 4 KiB + 0 B FFFFFFFF /dev/nvme0n15 81Ec-JRMlkL9AAAAAAAB NetApp ONTAP Controller 15 37.58 GB / 37.58 GB 4 KiB + 0 B FFFFFFFF /dev/nvme0n16 81Ec-JRMlkL9AAAAAAAB NetApp ONTAP Controller 16 37.58 GB / 37.58 GB 4 KiB + 0 B FFFFFFFF /dev/nvme0n17 81Ec-JRMlkL9AAAAAAAB NetApp ONTAP Controller 17 37.58 GB / 37.58 GB 4 KiB + 0 B FFFFFFFF /dev/nvme0n18 81Ec-JRMlkL9AAAAAAAB NetApp ONTAP Controller 18 37.58 GB / 37.58 GB 4 KiB + 0 B FFFFFFFF /dev/nvme0n19 81Ec-JRMlkL9AAAAAAAB NetApp ONTAP Controller 19 37.58 GB / 37.58 GB 4 KiB + 0 B FFFFFFFF /dev/nvme0n2 81Ec-JRMlkL9AAAAAAAB NetApp ONTAP Controller 2 37.58 GB / 37.58 GB 4 KiB + 0 B FFFFFFFF /dev/nvme0n20 81Ec-JRMlkL9AAAAAAAB NetApp ONTAP Controller 20 37.58 GB / 37.58 GB 4 KiB + 0 B FFFFFFFF /dev/nvme0n3 81Ec-JRMlkL9AAAAAAAB NetApp ONTAP Controller 3 37.58 GB / 37.58 GB 4 KiB + 0 B FFFFFFFF /dev/nvme0n4 81Ec-JRMlkL9AAAAAAAB NetApp ONTAP Controller 4 37.58 GB / 37.58 GB 4 KiB + 0 B FFFFFFFF /dev/nvme0n5 81Ec-JRMlkL9AAAAAAAB NetApp ONTAP Controller 5 37.58 GB / 37.58 GB 4 KiB + 0 B FFFFFFFF /dev/nvme0n6 81Ec-JRMlkL9AAAAAAAB NetApp ONTAP Controller 6 37.58 GB / 37.58 GB 4 KiB + 0 B FFFFFFFF /dev/nvme0n7 81Ec-JRMlkL9AAAAAAAB NetApp ONTAP Controller 7 37.58 GB / 37.58 GB 4 KiB + 0 B FFFFFFFF /dev/nvme0n8 81Ec-JRMlkL9AAAAAAAB NetApp ONTAP Controller 8 37.58 GB / 37.58 GB 4 KiB + 0 B FFFFFFFF /dev/nvme0n9 81Ec-JRMlkL9AAAAAAAB NetApp ONTAP Controller 9 37.58 GB / 37.58 GB 4 KiB + 0 B FFFFFFFF
-
Verify that the controller state of each path is live and has correct ANA status.
# nvme list-subsys /dev/nvme0n1 nvme-subsys0 - NQN=nqn.1992-08.com.netapp:sn.b79f5c6e4d0911edb3a0d039ea243511:subsystem.nvme_1 \ + +- nvme214 fc traddr=nn-0x20e0d039ea243510:pn-0x20e4d039ea243510 host_traddr=nn-0x200000109bf0447b:pn-0x100000109bf0447b live non-optimized +- nvme219 fc traddr=nn-0x20e0d039ea243510:pn-0x20e2d039ea243510 host_traddr=nn-0x200000109bf0447c:pn-0x100000109bf0447c live optimized +- nvme223 fc traddr=nn-0x20e0d039ea243510:pn-0x20e1d039ea243510 host_traddr=nn-0x200000109bf0447b:pn-0x100000109bf0447b live optimized +- nvme228 fc traddr=nn-0x20e0d039ea243510:pn-0x20e3d039ea243510 host_traddr=nn-0x200000109bf0447c:pn-0x100000109bf0447c live non-optimized
-
Verify the NetApp plug-in displays correct values for each ONTAP namespace devices.
#nvme netapp ontapdevices -o column Device Vserver Namespace Path NSID UUID Size ---------------- ------------------------- -------------------------------------------------- ---- --------- /dev/nvme0n1 LPE36002_ASA_BL /vol/fcnvme_1_0_0/fcnvme_ns 1 ae10e16d-1fa4-49c2-8594-02bf6f3b1af1 37.58GB /dev/nvme0n10 LPE36002_ASA_BL /vol/fcnvme_1_0_9/fcnvme_ns 10 2cf00782-e2bf-40fe-8495-63e4501727cd 37.58GB /dev/nvme0n11 LPE36002_ASA_BL /vol/fcnvme_1_1_9/fcnvme_ns 11 fbefbe6c-90fe-46a2-8a51-47bad9e2eb95 37.58GB /dev/nvme0n12 LPE36002_ASA_BL /vol/fcnvme_1_1_0/fcnvme_ns 12 0e9cc8fa-d821-4f1c-8944-3003dcded864 37.58GB /dev/nvme0n13 LPE36002_ASA_BL /vol/fcnvme_1_1_1/fcnvme_ns 13 31f03b13-aaf9-4a3f-826b-d126ef007991 37.58GB /dev/nvme0n14 LPE36002_ASA_BL /vol/fcnvme_1_1_8/fcnvme_ns 14 bcf4627c-5bf9-4a51-a920-5da174ec9876 37.58GB /dev/nvme0n15 LPE36002_ASA_BL /vol/fcnvme_1_1_7/fcnvme_ns 15 239fd09d-11db-46a3-8e94-b5ebe6eb2421 37.58GB /dev/nvme0n16 LPE36002_ASA_BL /vol/fcnvme_1_1_2/fcnvme_ns 16 1d8004df-f2e8-48c8-8ccb-ce45f18a15ae 37.58GB /dev/nvme0n17 LPE36002_ASA_BL /vol/fcnvme_1_1_3/fcnvme_ns 17 4f7afbcf-3ace-4e6c-9245-cbf5bd155ef4 37.58GB /dev/nvme0n18 LPE36002_ASA_BL /vol/fcnvme_1_1_4/fcnvme_ns 18 b022c944-6ebf-4986-a28c-8d9e8ec130c9 37.58GB /dev/nvme0n19 LPE36002_ASA_BL /vol/fcnvme_1_1_5/fcnvme_ns 19 c457d0c7-bfea-43aa-97ef-c749d8612a72 37.58GB /dev/nvme0n2 LPE36002_ASA_BL /vol/fcnvme_1_0_1/fcnvme_ns 2 d2413d8b-e82e-4412-89d3-c9a751ed7716 37.58GB /dev/nvme0n20 LPE36002_ASA_BL /vol/fcnvme_1_1_6/fcnvme_ns 20 650e0d93-967d-4415-874a-36bf9c93c952 37.58GB /dev/nvme0n3 LPE36002_ASA_BL /vol/fcnvme_1_0_2/fcnvme_ns 3 09d89d9a-7835-423f-93e7-f6f3ece1dcbc 37.58GB /dev/nvme0n4 LPE36002_ASA_BL /vol/fcnvme_1_0_3/fcnvme_ns 4 d8e99326-a67c-469f-b3e9-e0e4a38c8a76 37.58GB /dev/nvme0n5 LPE36002_ASA_BL /vol/fcnvme_1_0_4/fcnvme_ns 5 c91c71f9-3e04-4844-b376-30acab6311f1 37.58GB /dev/nvme0n6 LPE36002_ASA_BL /vol/fcnvme_1_0_5/fcnvme_ns 6 4e8b4345-e5b1-4aa4-ae1a-adf0de2879ea 37.58GB /dev/nvme0n7 LPE36002_ASA_BL /vol/fcnvme_1_0_6/fcnvme_ns 7 ef715a16-a946-4bb8-8735-74f214785874 37.58GB /dev/nvme0n8 LPE36002_ASA_BL /vol/fcnvme_1_0_7/fcnvme_ns 8 4b038502-966c-49fd-9631-a17f23478ae0 37.58GB /dev/nvme0n9 LPE36002_ASA_BL /vol/fcnvme_1_0_8/fcnvme_ns 9 f565724c-992f-41f6-83b5-da1fe741c09b 37.58GB
#nvme netapp ontapdevices -o json { "ONTAPdevices" : [ { "Device" : "/dev/nvme0n1", "Vserver" : "LPE36002_ASA_BL", "Namespace_Path" : "/vol/fcnvme_1_0_0/fcnvme_ns", "NSID" : 1, "UUID" : "ae10e16d-1fa4-49c2-8594-02bf6f3b1af1", "Size" : "37.58GB", "LBA_Data_Size" : 4096, "Namespace_Size" : 9175040 }, { "Device" : "/dev/nvme0n10", "Vserver" : "LPE36002_ASA_BL", "Namespace_Path" : "/vol/fcnvme_1_0_9/fcnvme_ns", "NSID" : 10, "UUID" : "2cf00782-e2bf-40fe-8495-63e4501727cd", "Size" : "37.58GB", "LBA_Data_Size" : 4096, "Namespace_Size" : 9175040 }, { "Device" : "/dev/nvme0n11", "Vserver" : "LPE36002_ASA_BL", "Namespace_Path" : "/vol/fcnvme_1_1_9/fcnvme_ns", "NSID" : 11, "UUID" : "fbefbe6c-90fe-46a2-8a51-47bad9e2eb95", "Size" : "37.58GB", "LBA_Data_Size" : 4096, "Namespace_Size" : 9175040 }, { "Device" : "/dev/nvme0n12", "Vserver" : "LPE36002_ASA_BL", "Namespace_Path" : "/vol/fcnvme_1_1_0/fcnvme_ns", "NSID" : 12, "UUID" : "0e9cc8fa-d821-4f1c-8944-3003dcded864", "Size" : "37.58GB", "LBA_Data_Size" : 4096, "Namespace_Size" : 9175040 }, { "Device" : "/dev/nvme0n13", "Vserver" : "LPE36002_ASA_BL", "Namespace_Path" : "/vol/fcnvme_1_1_1/fcnvme_ns", "NSID" : 13, "UUID" : "31f03b13-aaf9-4a3f-826b-d126ef007991", "Size" : "37.58GB", "LBA_Data_Size" : 4096, "Namespace_Size" : 9175040 },
Known issues
The NVMe-oF host configuration for OL 8.3 with ONTAP has the following known issues:
NetApp Bug ID | Title | Description |
---|---|---|
1517321 |
Oracle Linux 8.3 NVMe-oF Hosts create duplicate Persistent Discovery Controllers |
On Oracle Linux 8.3 NVMe over Fabrics (NVMe-oF) hosts, you can use the |