NVMe/FC host configuration for RHEL 8.3 with ONTAP

12/19/2023 Contributors

NVMe/FC is supported on ONTAP 9.6 or later for Red Hat Enterprise Linux (RHEL) 8.3. The RHEL 8.3 host runs both NVMe and SCSI traffic through the same FC initiator adapter ports. See the Hardware Universe for a list of supported FC adapters and controllers.

See the NetApp Interoperability Matrix Tool for the most current list of supported configurations.

Features

There are no new features in this release.

Known limitations

For RHEL 8.3, in-kernel NVMe multipath is disabled by default. You can enable it manually.
SAN booting using the NVMe-oF protocol is currently not supported.

Enable NVMe/FC on RHEL 8.3

You can use the following procedure to enable NVMe/FC.

Steps

Install Red Hat Enterprise Linux 8.3 GA on the server.
If you are upgrading from RHEL 8.2 to RHEL 8.3 using the yum update/upgrade command, your /etc/nvme/host* files might be lost. To avoid file loss, use the following procedure:
Steps
1. Backup your /etc/nvme/host* files.
2. If you have a manually edited udev rule, remove it:
  
  /lib/udev/rules.d/71-nvme-iopolicy-netapp-ONTAP.rules
3. Perform the upgrade.
4. After the upgrade is complete, run the following command:
  
  yum remove nvme-cli
5. Restore the host files at /etc/nvme/.
  
  yum install nvmecli
6. Copy the original /etc/nvme/host* contents from the backup to the actual host files at /etc/nvme/.
After the installation is complete, verify that you're running the specified RHEL kernel:
```
# uname -r
4.18.0-240.el8.x86_64
```
See the NetApp Interoperability Matrix Tool for the most current list of supported versions.

Install the nvme-cli package:

# rpm -qa|grep nvme-cli
nvme-cli-1.12-2.el8.x86_64

Enable in-kernel NVMe multipath.

# grubby --args=nvme_core.multipath=Y --update-kernel /boot/vmlinuz-4.18.0-240.el8.x86_64

On the RHEL 8.3 host, check the host NQN string at /etc/nvme/hostnqn verify that it matches the host NQN string for the corresponding subsystem on the ONTAP array:
```
# cat /etc/nvme/hostnqn
```
Example output:
```
# cat /etc/nvme/hostnqn
nqn.2014-08.org.nvmexpress:uuid:9ed5b327-b9fc-4cf5-97b3-1b5d986345d1
```

Verify that the hostnqn string matches the hostnqn string for the corresponding subsystem on the ONTAP array:

vserver nvme subsystem host show -vserver vs_fcnvme_141

Example output

::> vserver nvme subsystem host show -vserver vs_fcnvme_141
Vserver         Subsystem        Host           NQN
-----------     --------------- ----------- ---------------
vs_fcnvme_141    nvme_141_1                 nqn.2014-08.org.nvmexpress:uuid:9ed5b327-b9fc-4cf5-97b3-1b5d986345d1

If the host NQN strings do not match, use the vserver modify command to update the host NQN string on the corresponding ONTAP array subsystem to match to host NQN string from /etc/nvme/hostnqn on the host.

Reboot the host.

Optionally, update the enable_foreign setting.

If you intend to run both NVMe and SCSI traffic on the same RHEL 8.3 co-existent host, NetApp recommends that you use in-kernel NVMe multipath for ONTAP namespaces and dm-multipath for ONTAP LUNs, respectively. You should also blacklist the ONTAP namespaces in dm-multipath to prevent dm-multipath from claiming these namespace devices. You can do this by adding the enable_foreign setting to the /etc/multipath.conf, as shown below:

# cat /etc/multipath.conf
defaults {
   enable_foreign NONE
}

Restart the multipathd daemon by running a systemctl restart multipathd.

Validate NVMe/FC

You can use the following procedure to validate NVMe/FC.

Steps

Verify the following NVMe/FC settings:

# cat /sys/module/nvme_core/parameters/multipath
Y

# cat /sys/class/nvme-subsystem/nvme-subsys*/model
NetApp ONTAP Controller
NetApp ONTAP Controller

# cat /sys/class/nvme-subsystem/nvme-subsys*/iopolicy
round-robin
round-robin

Verify that the namespaces are created and properly discovered on the host.

/dev/nvme0n1     814vWBNRwf9HAAAAAAAB  NetApp ONTAP Controller                1                  85.90 GB / 85.90 GB     4 KiB + 0 B   FFFFFFFF
/dev/nvme0n2     814vWBNRwf9HAAAAAAAB  NetApp ONTAP Controller                2                  85.90 GB / 85.90 GB     4 KiB + 0 B   FFFFFFFF
/dev/nvme0n3     814vWBNRwf9HAAAAAAAB  NetApp ONTAP Controller                3                  85.90 GB / 85.90 GB     4 KiB + 0 B   FFFFFFFF

Verify the status of the ANA paths.

# nvme list-subsys /dev/nvme0n1
nvme-subsys0 - NQN=nqn.1992-08.com.netapp:sn.5f5f2c4aa73b11e9967e00a098df41bd:subsystem.nvme_141_1
\
+- nvme0 fc traddr=nn-0x203700a098dfdd91:pn-0x203800a098dfdd91 host_traddr=nn-0x200000109b1c1204:pn-0x100000109b1c1204 live inaccessible
+- nvme1 fc traddr=nn-0x203700a098dfdd91:pn-0x203900a098dfdd91 host_traddr=nn-0x200000109b1c1204:pn-0x100000109b1c1204 live inaccessible
+- nvme2 fc traddr=nn-0x203700a098dfdd91:pn-0x203a00a098dfdd91 host_traddr=nn-0x200000109b1c1205:pn-0x100000109b1c1205 live optimized
+- nvme3 fc traddr=nn-0x203700a098dfdd91:pn-0x203d00a098dfdd91 host_traddr=nn-0x200000109b1c1205:pn-0x100000109b1c1205 live optimized

Verify the NetApp plug-in for ONTAP devices:

Column

# nvme netapp ontapdevices -o column

Example output

Device               Vserver            Namespace Path                           NSID                      UUID                     Size
--------------- --------------- ---------------------------------------------  -------- --------------------------------------  ---------
/dev/nvme0n1      vs_fcnvme_141     /vol/fcnvme_141_vol_1_1_0/fcnvme_141_ns        1      72b887b1-5fb6-47b8-be0b-33326e2542e2    85.90GB
/dev/nvme0n2      vs_fcnvme_141     /vol/fcnvme_141_vol_1_0_0/fcnvme_141_ns        2      04bf9f6e-9031-40ea-99c7-a1a61b2d7d08    85.90GB
/dev/nvme0n3      vs_fcnvme_141     /vol/fcnvme_141_vol_1_1_1/fcnvme_141_ns        3      264823b1-8e03-4155-80dd-e904237014a4    85.90GB

JSON

# nvme netapp ontapdevices -o json

Example output

{
"ONTAPdevices" : [
    {
        "Device" : "/dev/nvme0n1",
        "Vserver" : "vs_fcnvme_141",
        "Namespace_Path" : "/vol/fcnvme_141_vol_1_1_0/fcnvme_141_ns",
        "NSID" : 1,
        "UUID" : "72b887b1-5fb6-47b8-be0b-33326e2542e2",
        "Size" : "85.90GB",
        "LBA_Data_Size" : 4096,
        "Namespace_Size" : 20971520
    },
    {
        "Device" : "/dev/nvme0n2",
        "Vserver" : "vs_fcnvme_141",
        "Namespace_Path" : "/vol/fcnvme_141_vol_1_0_0/fcnvme_141_ns",
        "NSID" : 2,
        "UUID" : "04bf9f6e-9031-40ea-99c7-a1a61b2d7d08",
        "Size" : "85.90GB",
        "LBA_Data_Size" : 4096,
        "Namespace_Size" : 20971520
      },
      {
         "Device" : "/dev/nvme0n3",
         "Vserver" : "vs_fcnvme_141",
         "Namespace_Path" : "/vol/fcnvme_141_vol_1_1_1/fcnvme_141_ns",
         "NSID" : 3,
         "UUID" : "264823b1-8e03-4155-80dd-e904237014a4",
         "Size" : "85.90GB",
         "LBA_Data_Size" : 4096,
         "Namespace_Size" : 20971520
       },
  ]

Configure the Broadcom FC adapter for NVMe/FC

You can use the following procedure to configure a Broadcom FC adapter.

For the most current list of supported adapters, see the NetApp Interoperability Matrix Tool.

Steps

Verify that you are using the supported adapter.

# cat /sys/class/scsi_host/host*/modelname
LPe32002-M2
LPe32002-M2

# cat /sys/class/scsi_host/host*/modeldesc
Emulex LightPulse LPe32002-M2 2-Port 32Gb Fibre Channel Adapter
Emulex LightPulse LPe32002-M2 2-Port 32Gb Fibre Channel Adapter

Verify that lpfc_enable_fc4_type is set to "3".

# cat /sys/module/lpfc/parameters/lpfc_enable_fc4_type
3

Verify that the initiator ports are up and running and can see the target LIFs.

# cat /sys/class/fc_host/host*/port_name
0x100000109b1c1204
0x100000109b1c1205

# cat /sys/class/fc_host/host*/port_state
Online
Online

# cat /sys/class/scsi_host/host*/nvme_info
NVME Initiator Enabled
XRI Dist lpfc0 Total 6144 IO 5894 ELS 250
NVME LPORT lpfc0 WWPN x100000109b1c1204 WWNN x200000109b1c1204 DID x011d00 ONLINE
NVME RPORT WWPN x203800a098dfdd91 WWNN x203700a098dfdd91 DID x010c07 TARGET DISCSRVC ONLINE
NVME RPORT WWPN x203900a098dfdd91 WWNN x203700a098dfdd91 DID x011507 TARGET DISCSRVC ONLINE
NVME Statistics
LS: Xmt 0000000f78 Cmpl 0000000f78 Abort 00000000
LS XMIT: Err 00000000 CMPL: xb 00000000 Err 00000000
Total FCP Cmpl 000000002fe29bba Issue 000000002fe29bc4 OutIO 000000000000000a
abort 00001bc7 noxri 00000000 nondlp 00000000 qdepth 00000000 wqerr 00000000 err 00000000
FCP CMPL: xb 00001e15 Err 0000d906
NVME Initiator Enabled
XRI Dist lpfc1 Total 6144 IO 5894 ELS 250
NVME LPORT lpfc1 WWPN x100000109b1c1205 WWNN x200000109b1c1205 DID x011900 ONLINE
NVME RPORT WWPN x203d00a098dfdd91 WWNN x203700a098dfdd91 DID x010007 TARGET DISCSRVC ONLINE
NVME RPORT WWPN x203a00a098dfdd91 WWNN x203700a098dfdd91 DID x012a07 TARGET DISCSRVC ONLINE
NVME Statistics
LS: Xmt 0000000fa8 Cmpl 0000000fa8 Abort 00000000
LS XMIT: Err 00000000 CMPL: xb 00000000 Err 00000000
Total FCP Cmpl 000000002e14f170 Issue 000000002e14f17a OutIO 000000000000000a
abort 000016bb noxri 00000000 nondlp 00000000 qdepth 00000000 wqerr 00000000 err 00000000
FCP CMPL: xb 00001f50 Err 0000d9f8

Enable 1 MB I/O size (optional).

The lpfc_sg_seg_cnt parameter needs to be set to 256 for the lpfc driver to issue I/O requests up to 1 MB in size.
```
# cat /etc/modprobe.d/lpfc.conf
options lpfc lpfc_sg_seg_cnt=256
```
Run a dracut -f command and then reboot the host.
After the host boots up, verify that lpfc_sg_seg_cnt is set to 256.
```
# cat /sys/module/lpfc/parameters/lpfc_sg_seg_cnt
256
```

Verify that you are using the recommended Broadcom lpfc firmware as well as the inbox driver:

# cat /sys/class/scsi_host/host*/fwrev
12.8.340.8, sli-4:2:c
12.8.340.8, sli-4:2:c

# cat /sys/module/lpfc/version
0:12.8.0.1

NVMe/FC host configuration for RHEL 8.3 with ONTAP

Creating your file...

Features

Known limitations

Enable NVMe/FC on RHEL 8.3

Validate NVMe/FC

Configure the Broadcom FC adapter for NVMe/FC