适用于采用ONTAP的Oracle Linux 9.4的NVMe-oF主机配置
NetApp SAN主机配置支持采用非对称命名空间访问(AANA)的基于网络结构的NVMe (NVMe-oF)协议。在NVMe-oF环境中、ANA相当于iSCSI和FCP环境中的非对称逻辑单元访问(AMUA)多路径功能。ANA是使用内核NVMe多路径功能实施的。
对于采用ONTAP存储的Oracle Linux 9.4、NVMe-oF主机配置提供以下支持和功能。在开始配置过程之前、您还应查看已知限制。
-
提供支持:
-
支持基于TCP的NVMe (NVMe/TCP)以及基于光纤通道的NVMe (NVMe/FC)。本机NVMe-CLI软件包中的NetApp插件可显示NVMe/FC和NVMe/TCP命名库的ONTAP详细信息。
-
在同一主机上同时运行NVMe和SCSI流量。例如、您可以为SCSI LUN的SCSI mpath设备配置dm-Multipath、并使用NVMe多路径在主机上配置NVMe-oF命名空间设备。
有关支持的配置的其他详细信息、请参见 "NetApp 互操作性表工具"。
-
-
可用功能:
-
从ONTAP 9.12.1开始、NVMe-oF引入了对安全带内身份验证的支持。您可以在Oracle Linux 9.4中对NVMe-oF使用安全带内身份验证
-
默认情况下、支持为NVMe命名空间启用内核NVMe多路径、因此无需显式设置。
-
-
已知限制:
-
目前不支持使用NVMe-oF协议启动SAN。
-
验证软件版本
您可以使用以下过程验证支持的最低Oracle Linux 9.4软件版本。
-
在服务器上安装Oracle Linux 9.4 GA。安装完成后、验证您是否正在运行指定的Oracle Linux 9.4 GA内核。
uname -r
5.15.0-205.149.5.1.el9uek.x86_64
-
安装
NVMe-CLI
软件包:rpm -qa|grep nvme-cli
nvme-cli-2.6-5.el9.x86_64
-
安装
libnvme
软件包:rpm -qa|grep libnvme
libnvme-1.6-1.el9.x86_64
-
在Oracle Linux 9.4主机上,检查
hostnqn
`/etc/nvme/hostnqn`以下位置的字符串:cat /etc/nvme/hostnqn
nqn.2014-08.org.nvmexpress:uuid:9c5d23fe-21c5-472f-9aa4-dc68de0882e9
-
验证是否已
hostnqn
字符串与匹配hostnqn
ONTAP 阵列上对应子系统的字符串:vserver nvme subsystem host show -vserver vs_coexistence_149
显示示例
Vserver Subsystem Priority Host NQN ------- --------- -------- ------------------------------------------------ vs_coexistence_149 nvme regular nqn.2014-08.org.nvmexpress:uuid:9c5d23fe-21c5-472f-9aa4-dc68de0882e9 nvme_1 regular nqn.2014-08.org.nvmexpress:uuid:9c5d23fe-21c5-472f-9aa4-dc68de0882e9 nvme_2 regular nqn.2014-08.org.nvmexpress:uuid:9c5d23fe-21c5-472f-9aa4-dc68de0882e9 nvme_3 regular nqn.2014-08.org.nvmexpress:uuid:9c5d23fe-21c5-472f-9aa4-dc68de0882e9 4 entries were displayed.
如果 hostnqn`字符串不匹配、您可以使用 `vserver modify`命令更新 `hostnqn`相应ONTAP阵列子系统上的字符串、使其与主机上的字符串 `/etc/nvme/hostnqn`匹配 `hostnqn
。
配置 NVMe/FC
您可以使用Broadcom/Emulex FC或Marvell/Qlogic FC适配器配置NVMe/FC。对于配置有Broadcom适配器的NVMe/FC、可以启用大小为1 MB的I/O请求。
为Broadcom/Emulex适配器配置NVMe/FC。
-
验证您使用的适配器型号是否受支持:
-
cat /sys/class/scsi_host/host*/modelname
LPe32002-M2 LPe32002-M2
-
cat /sys/class/scsi_host/host*/modeldesc
Emulex LightPulse LPe32002-M2 2-Port 32Gb Fibre Channel Adapter Emulex LightPulse LPe32002-M2 2-Port 32Gb Fibre Channel Adapter
-
-
确认您使用的是建议的Broadcom
lpfc
固件和内置驱动程序:-
cat /sys/class/scsi_host/host*/fwrev
14.4.317.7, sli-4:2:c 14.4.317.7, sli-4:2:c
-
cat /sys/module/lpfc/version
0:14.2.0.13
有关支持的适配器驱动程序和固件版本的最新列表、请参见 "NetApp 互操作性表工具"。
-
-
请验证
lpfc_enable_fc4_type
设置为3
:cat /sys/module/lpfc/parameters/lpfc_enable_fc4_type
3
-
验证是否可以查看启动程序端口:
cat /sys/class/fc_host/host*/port_name
0x100000109b3c081f 0x100000109b3c0820
-
验证启动程序端口是否联机:
cat /sys/class/fc_host/host*/port_state
Online Online
-
验证NVMe/FC启动程序端口是否已启用且目标端口是否可见:
cat /sys/class/scsi_host/host*/nvme_info
显示示例
NVME Initiator Enabled XRI Dist lpfc0 Total 6144 IO 5894 ELS 250 NVME LPORT lpfc0 WWPN x100000109b3c081f WWNN x200000109b3c081f DID x081600 ONLINE NVME RPORT WWPN x2020d039eab0dadc WWNN x201fd039eab0dadc DID x08010c TARGET DISCSRVC ONLINE NVME RPORT WWPN x2024d039eab0dadc WWNN x201fd039eab0dadc DID x08030c TARGET DISCSRVC ONLINE NVME Statistics LS: Xmt 00000027d8 Cmpl 00000027d8 Abort 00000000 LS XMIT: Err 00000000 CMPL: xb 00000000 Err 00000000 Total FCP Cmpl 00000000315454fa Issue 00000000314de6a4 OutIO fffffffffff991aa abort 00000be4 noxri 00000000 nondlp 00001903 qdepth 00000000 wqerr 00000000 err 00000000 FCP CMPL: xb 00000c92 Err 0000bda4 NVME Initiator Enabled XRI Dist lpfc1 Total 6144 IO 5894 ELS 250 NVME LPORT lpfc1 WWPN x100000109b3c0820 WWNN x200000109b3c0820 DID x081b00 ONLINE NVME RPORT WWPN x2027d039eab0dadc WWNN x201fd039eab0dadc DID x08020c TARGET DISCSRVC ONLINE NVME RPORT WWPN x2025d039eab0dadc WWNN x201fd039eab0dadc DID x08040c TARGET DISCSRVC ONLINE NVME Statistics LS: Xmt 00000026ac Cmpl 00000026ac Abort 00000000 LS XMIT: Err 00000000 CMPL: xb 00000000 Err 00000000 Total FCP Cmpl 00000000312a5478 Issue 00000000312465a2 OutIO fffffffffffa112a abort 00000b01 noxri 00000000 nondlp 00001ae4 qdepth 00000000 wqerr 00000000 err 00000000 FCP CMPL: xb 00000b53 Err 0000ba63
为Marvell/QLogic适配器配置NVMe/FC。
Oracle Linux 9.4 GA内核中包含的本机内置qla2xxx驱动程序已进行了最新修复。这些修复程序对于ONTAP支持至关重要。 |
-
验证您是否正在运行受支持的适配器驱动程序和固件版本:
cat /sys/class/fc_host/host*/symbolic_name
QLE2872 FW:v9.15.00 DVR:v10.02.09.100-k QLE2872 FW:v9.15.00 DVR:v10.02.09.100-k
-
请验证
ql2xnvmeenable
已设置。这样、Marvell适配器便可用作NVMe/FC启动程序:cat /sys/module/qla2xxx/parameters/ql2xnvmeenable
1
启用 1 MB I/O 大小(可选)
ONTAP会在"识别 控制器"数据中报告MDTS (MAX Data传输大小)为8。这意味着最大I/O请求大小最多可以为1 MB。要向Broadcom NVMe/FC主机发出大小为1 MB的I/O请求、应将参数的值 `lpfc_sg_seg_cnt`从默认值64增加 `lpfc`到256。
这些步骤不适用于逻辑NVMe/FC主机。 |
-
将 `lpfc_sg_seg_cnt`参数设置为256:
cat /etc/modprobe.d/lpfc.conf
options lpfc lpfc_sg_seg_cnt=256
-
运行 `dracut -f`命令并重新启动主机。
-
验证的预期值是否 `lpfc_sg_seg_cnt`为256:
cat /sys/module/lpfc/parameters/lpfc_sg_seg_cnt
配置 NVMe/TCP
NVMe/TCP协议不支持此 auto-connect`操作。相反、您可以通过手动执行NVMe/TCP或 `connect-all`操作来发现NVMe/TCP子系统和命名路径 `connect
。
-
验证启动程序端口是否可以通过受支持的NVMe/TCP LIF提取发现日志页面数据:
nvme discover -t tcp -w host-traddr -a traddr
显示示例
nvme discover -t tcp -w 192.168.166.4 -a 192.168.166.56 Discovery Log Number of Records 10, Generation counter 15 =====Discovery Log Entry 0====== trtype: tcp adrfam: ipv4 subtype: current discovery subsystem treq: not specified portid: 13 trsvcid: 8009 subnqn: nqn.1992-08.com.netapp:sn.cf84a53c81b111ef8446d039ea9ea481:discovery traddr: 192.168.165.56 eflags: explicit discovery connections, duplicate discovery information sectype: none =====Discovery Log Entry 1====== trtype: tcp adrfam: ipv4 subtype: current discovery subsystem treq: not specified portid: 9 trsvcid: 8009 subnqn: nqn.1992-08.com.netapp:sn.cf84a53c81b111ef8446d039ea9ea481:discovery traddr: 192.168.166.56 eflags: explicit discovery connections, duplicate discovery information sectype: none =====Discovery Log Entry 2====== trtype: tcp adrfam: ipv4 subtype: nvme subsystem treq: not specified portid: 13 trsvcid: 4420 subnqn: nqn.1992-08.com.netapp:sn.cf84a53c81b111ef8446d039ea9ea481:subsystem.nvme_tcp_2 traddr: 192.168.165.56 eflags: none sectype: none
-
验证其他NVMe/TCP启动程序-目标LIF组合是否可以成功提取发现日志页面数据:
nvme discover -t tcp -w host-traddr -a traddr
nvme discover -t tcp -w 192.168.166.4 -a 192.168.166.56 nvme discover -t tcp -w 192.168.165.3 -a 192.168.165.56
-
运行
nvme connect-all
在节点中所有受支持的NVMe/TCP启动程序-目标SIP上运行命令:nvme connect-all -t tcp -w host-traddr -a traddr
nvme connect-all -t tcp -w 192.168.166.4 -a 192.168.166.56 nvme connect-all -t tcp -w 192.168.165.3 -a 192.168.165.56
从Oracle Linux 9.4开始、NVMe/TCP的默认设置 ctrl_loss_tmo timeout`已关闭、并且对重试次数没有限制(无限期重试)。使用或 `nvme connect-all`命令(选项-l)时,您不必手动配置特定的 `ctrl_loss_tmo timeout`持续时间 `nvme connect
。通过此默认行为、NVMe/TCP控制器在发生路径故障时不会发生超时、并会无限期保持连接。
验证 NVMe-oF
要支持ONTAP LUN正确运行、请验证内核NVMe多路径状态、ANA状态和ONTAP命名空间是否适用于NVMe-oF配置。
-
在Oracle Linux 9.4主机上验证以下NVMe/FC设置:
-
cat /sys/module/nvme_core/parameters/multipath
Y
-
cat /sys/class/nvme-subsystem/nvme-subsys*/model
NetApp ONTAP Controller NetApp ONTAP Controller
-
cat /sys/class/nvme-subsystem/nvme-subsys*/iopolicy
round-robin round-robin
-
-
验证是否已在主机上创建并正确发现命名空间:
nvme list
显示示例
Node SN Model --------------------------------------------------------- /dev/nvme0n1 81K2iBXAYSG6AAAAAAAB NetApp ONTAP Controller /dev/nvme0n2 81K2iBXAYSG6AAAAAAAB NetApp ONTAP Controller /dev/nvme0n3 81K2iBXAYSG6AAAAAAAB NetApp ONTAP Controller Namespace Usage Format FW Rev ----------------------------------------------------------- 1 3.78GB/10.74GB 4 KiB + 0 B FFFFFFFF 2 3.78GB/10.74GB 4 KiB + 0 B FFFFFFFF 3 3.78GB/10.74GB 4 KiB + 0 B FFFFFFFF
-
验证每个路径的控制器状态是否为活动状态且是否具有正确的ANA状态:
NVMe/FCnvme list-subsys /dev/nvme0n1
显示示例
nvme-subsys0 - NQN=nqn.1992-08.com.netapp:sn.5f074d527b7011ef8446d039ea9ea481:subsystem.nvme hostnqn=nqn.2014-08.org.nvmexpress:uuid:060fd513-83be-4c3e-aba1-52e169056dcf iopolicy=round-robin \ +- nvme10 fc traddr=nn-0x201fd039eab0dadc:pn-0x2024d039eab0dadc,host_traddr=nn-0x200000109b3c081f:pn-0x100000109b3c081f live non-optimized +- nvme15 fc traddr=nn-0x201fd039eab0dadc:pn-0x2020d039eab0dadc,host_traddr=nn-0x200000109b3c081f:pn-0x100000109b3c081f live optimized +- nvme7 fc traddr=nn-0x201fd039eab0dadc:pn-0x2025d039eab0dadc,host_traddr=nn-0x200000109b3c0820:pn-0x100000109b3c0820 live non-optimized +- nvme9 fc traddr=nn-0x201fd039eab0dadc:pn-0x2027d039eab0dadc,host_traddr=nn-0x200000109b3c0820:pn-0x100000109b3c0820 live optimized
NVMe/TCPnvme list-subsys /dev/nvme1n22
显示示例
nvme-subsys0 - NQN=nqn.1992-08.com.netapp:sn.cf84a53c81b111ef8446d039ea9ea481:subsystem.nvme_tcp_1 hostnqn=nqn.2014-08.org.nvmexpress:uuid:9796c1ec-0d34-11eb-b6b2-3a68dd3bab57 iopolicy=round-robin \ +- nvme2 tcp traddr=192.168.166.56,trsvcid=4420,host_traddr=192.168.166.4,src_addr=192.168.166.4 live optimized +- nvme4 tcp traddr=192.168.165.56,trsvcid=4420,host_traddr=192.168.165.3,src_addr=192.168.165.3 live non-optimized
-
验证NetApp插件是否为每个ONTAP 命名空间设备显示正确的值:
列nvme netapp ontapdevices -o column
显示示例
Device Vserver Namespace Path ----------------------- ------------------------------ /dev/nvme0n1 vs_coexistence_147 /vol/fcnvme_1_1_0/fcnvme_ns /dev/nvme0n2 vs_coexistence_147 /vol/fcnvme_1_1_1/fcnvme_ns /dev/nvme0n3 vs_coexistence_147 /vol/fcnvme_1_1_2/fcnvme_ns NSID UUID Size ------------------------------------------------------------ 1 e605babf-1b54-417d-843b-bc14355b70c5 10.74GB 2 b8dbecc7-14c5-4d84-b948-73c7abf5af43 10.74GB 3 ba24d1a3-1911-4351-83a9-1c843d04633c 10.74GB
JSONnvme netapp ontapdevices -o json
显示示例
{ "ONTAPdevices":[ { "Device":"/dev/nvme0n1", "Vserver":"vs_coexistence_147", "Namespace_Path":"/vol/fcnvme_1_1_0/fcnvme_ns", "NSID":1, "UUID":"e605babf-1b54-417d-843b-bc14355b70c5", "Size":"10.74GB", "LBA_Data_Size":4096, "Namespace_Size":2621440 }, { "Device":"/dev/nvme0n2", "Vserver":"vs_coexistence_147", "Namespace_Path":"/vol/fcnvme_1_1_1/fcnvme_ns", "NSID":2, "UUID":"b8dbecc7-14c5-4d84-b948-73c7abf5af43", "Size":"10.74GB", "LBA_Data_Size":4096, "Namespace_Size":2621440 }, { "Device":"/dev/nvme0n3", "Vserver":"vs_coexistence_147", "Namespace_Path":"/vol/fcnvme_1_1_2/fcnvme_ns", "NSID":3, "UUID":"c236905d-a335-47c4-a4b1-89ae30de45ae", "Size":"10.74GB", "LBA_Data_Size":4096, "Namespace_Size":2621440 }, ] }
设置安全带内身份验证
从ONTAP 9.12.1开始、Oracle Linux 9.4主机与ONTAP控制器之间可通过NVMe/TCP和NVMe/FC进行安全带内身份验证。
要设置安全身份验证、每个主机或控制器都必须与关联 DH-HMAC-CHAP
密钥、它是NVMe主机或控制器的NQN与管理员配置的身份验证密钥的组合。要对其对等方进行身份验证、NVMe主机或控制器必须识别与对等方关联的密钥。
您可以使用命令行界面或Config JSON文件设置安全带内身份验证。如果需要为不同的子系统指定不同的dhchap密钥、则必须使用config JSON文件。
使用命令行界面设置安全带内身份验证。
-
获取主机NQN:
cat /etc/nvme/hostnqn
-
为OL 9.4主机生成dhchap密钥。
以下输出说明了 `gen-dhchap-key`命令参数:
nvme gen-dhchap-key -s optional_secret -l key_length {32|48|64} -m HMAC_function {0|1|2|3} -n host_nqn • -s secret key in hexadecimal characters to be used to initialize the host key • -l length of the resulting key in bytes • -m HMAC function to use for key transformation 0 = none, 1- SHA-256, 2 = SHA-384, 3=SHA-512 • -n host NQN to use for key transformation
在以下示例中、将生成一个随机dhchap密钥、其中HMAC设置为3 (SHA-512)。
# nvme gen-dhchap-key -m 3 -n nqn.2014-08.org.nvmexpress:uuid:9796c1ec-0d34-11eb-b6b2-3a68dd3bab57 DHHC-1:03:zSq3+upTmknih8+6Ro0yw6KBQNAXjHFrOxQJaE5i916YdM/xsUSTdLkHw2MMmdFuGEslj6+LhNdf5HF0qfroFPgoQpU=:
-
在ONTAP控制器上、添加主机并指定两个dhchap密钥:
vserver nvme subsystem host add -vserver <svm_name> -subsystem <subsystem> -host-nqn <host_nqn> -dhchap-host-secret <authentication_host_secret> -dhchap-controller-secret <authentication_controller_secret> -dhchap-hash-function {sha-256|sha-512} -dhchap-group {none|2048-bit|3072-bit|4096-bit|6144-bit|8192-bit}
-
主机支持两种类型的身份验证方法:单向和双向。在主机上、连接到ONTAP控制器并根据所选身份验证方法指定dhchap密钥:
nvme connect -t tcp -w <host-traddr> -a <tr-addr> -n <host_nqn> -S <authentication_host_secret> -C <authentication_controller_secret>
-
验证
nvme connect authentication
命令、验证主机和控制器dhchap密钥:-
验证主机dhchap密钥:
cat /sys/class/nvme-subsystem/<nvme-subsysX>/nvme*/dhchap_secret
显示单向配置的示例输出
cat /sys/class/nvme-subsystem/nvme-subsys0/nvme*/dhchap_secret DHHC-1:01:OKIc4l+fs+fmpAj0hMK7ay8tTIzjccUWSCak/G2XjgJpKZeK: DHHC-1:01:OKIc4l+fs+fmpAj0hMK7ay8tTIzjccUWSCak/G2XjgJpKZeK:
-
验证控制器dhchap密钥:
cat /sys/class/nvme-subsystem/<nvme-subsysX>/nvme*/dhchap_ctrl_secret
显示双向配置的示例输出
cat /sys/class/nvme-subsystem/nvme-subsys0/nvme*/dhchap_ctrl_secret DHHC-1:03:zSq3+upTmknih8+6Ro0yw6KBQNAXjHFrOxQJaE5i916YdM/xsUSTdLkHw2MMmdFuGEslj6+LhNdf5HF0qfroFPgoQpU=: DHHC-1:03:zSq3+upTmknih8+6Ro0yw6KBQNAXjHFrOxQJaE5i916YdM/xsUSTdLkHw2MMmdFuGEslj6+LhNdf5HF0qfroFPgoQpU=:
-
如果ONTAP控制器配置中有多个NVMe子系统、则可以将文件与命令结合 nvme connect-all`使用 `/etc/nvme/config.json
。
要生成JSON文件、可以使用 `-o`选项。有关更多语法选项、请参见NVMe Connect-all手册页。
-
配置 JSON 文件:
显示示例
cat /etc/nvme/config.json [ { "hostnqn":"nqn.2014-08.org.nvmexpress:uuid:9796c1ec-0d34-11eb-b6b2-3a68dd3bab57", "hostid":"9796c1ec-0d34-11eb-b6b2-3a68dd3bab57", "dhchap_key":"DHHC-1:01:OKIc4l+fs+fmpAj0hMK7ay8tTIzjccUWSCak\/G2XjgJpKZeK:", "subsystems":[ { "nqn":"nqn.1992-08.com.netapp:sn.cf84a53c81b111ef8446d039ea9ea481:subsystem.nvme_tcp_1", "ports":[ { "transport":"tcp", "traddr":"192.168.165.56", "host_traddr":"192.168.165.3", "trsvcid":"4420", "dhchap_key":"DHHC-1:01:OKIc4l+fs+fmpAj0hMK7ay8tTIzjccUWSCak\/G2XjgJpKZeK:", "dhchap_ctrl_key":"DHHC-1:03:zSq3+upTmknih8+6Ro0yw6KBQNAXjHFrOxQJaE5i916YdM\/xsUSTdLkHw2MMmdFuGEslj6+LhNdf5HF0qfroFPgoQpU=:" }, { "transport":"tcp", "traddr":"192.168.166.56", "host_traddr":"192.168.166.4", "trsvcid":"4420", "dhchap_key":"DHHC-1:01:OKIc4l+fs+fmpAj0hMK7ay8tTIzjccUWSCak\/G2XjgJpKZeK:", "dhchap_ctrl_key":"DHHC-1:03:zSq3+upTmknih8+6Ro0yw6KBQNAXjHFrOxQJaE5i916YdM\/xsUSTdLkHw2MMmdFuGEslj6+LhNdf5HF0qfroFPgoQpU=:" } ] } ] } ]
在上述示例中, dhchap_key`对应于, `dhchap_ctrl_key`对应 `dhchap_ctrl_secret`于 `dhchap_secret
。 -
使用config JSON文件连接到ONTAP控制器:
nvme connect-all -J /etc/nvme/config.json
显示示例
traddr=192.168.165.56 is already connected traddr=192.168.165.56 is already connected traddr=192.168.165.56 is already connected traddr=192.168.165.56 is already connected traddr=192.168.165.56 is already connected traddr=192.168.165.56 is already connected traddr=192.168.166.56 is already connected traddr=192.168.166.56 is already connected traddr=192.168.166.56 is already connected traddr=192.168.166.56 is already connected traddr=192.168.166.56 is already connected traddr=192.168.166.56 is already connected
-
验证是否已为每个子系统的相应控制器启用dhchap密码:
-
验证主机dhchap密钥:
cat /sys/class/nvme-subsystem/nvme-subsys0/nvme0/dhchap_secret
DHHC-1:01:OKIc4l+fs+fmpAj0hMK7ay8tTIzjccUWSCak/G2XjgJpKZeK:
-
验证控制器dhchap密钥:
cat /sys/class/nvme-subsystem/nvme-subsys0/nvme0/dhchap_ctrl_secret
DHHC-1:03:zSq3+upTmknih8+6Ro0yw6KBQNAXjHFrOxQJaE5i916YdM/xsUSTdLkHw2MMmdFuGEslj6+LhNdf5HF0qfroFPgoQpU=:
-
已知问题
具有ONTAP版本的Oracle Linux 9.4没有已知问题。