適用於 ONTAP 的 SUSE Linux Enterprise Server 15 SP5 的 NVMe 主機組態
含非對稱命名空間存取( ANA )的 SUSE Linux Enterprise Server ( SLES ) 15 SP5 支援 NVMe over Fabrics ( NVMe over Fabric 、 NVMe of )、包括 NVMe over Fibre Channel ( NVMe / FC )和其他傳輸。在 NVMe 環境中、 ANA 等同於 iSCSI 和 FCP 環境中的 ALUA 多重路徑、並以核心內建 NVMe 多重路徑來實作。
下列支援適用於 ONTAP 15 SP5 的 NVMe 主機組態:
-
NVMe和SCSI流量都可以在相同的共存主機上執行。因此、對於 SCSI LUN 、您可以為 SCSI mpath 裝置設定 dm-multipath 、而您可以使用 NVMe 多重路徑在主機上設定 NVMe 命名空間裝置。
-
支援 NVMe over TCP ( NVMe / TCP )、以及 NVMe / FC 。原生的 NetApp 外掛程式
nvme-cli
套件會同時顯示 NVMe / FC 和 NVMe / TCP 命名空間的 ONTAP 詳細資料。
如需支援組態的詳細資訊、請參閱 "NetApp 互通性對照表工具"。
功能
-
支援 NVMe 安全頻內驗證
-
使用獨特的探索 NQN 支援持續探索控制器( PDC )
已知限制
-
目前不支援使用 NVMe 型傳輸協定進行 SAN 開機。
-
沒有
sanlun
支援 NVMe 。因此、在 SLES 15 SP5 主機上、無法為 NVMe 提供主機公用程式支援。您可以仰賴原生 NVMe - CLI 套件中的 NetApp 外掛程式來執行所有 NVMe 傳輸。
設定NVMe/FC
您可以為 Broadcom / Emulex FC 或 Marvell/Qlogic FC 介面卡設定 NVMe / FC 。
-
確認您使用的是建議的介面卡機型:
cat /sys/class/scsi_host/host*/modelname
-
輸出範例 * :
LPe32002 M2 LPe32002-M2
-
-
驗證介面卡型號說明:
cat /sys/class/scsi_host/host*/modeldesc
-
輸出範例 * :
Emulex LightPulse LPe32002-M2 2-Port 32Gb Fibre Channel Adapter Emulex LightPulse LPe32002-M2 2-Port 32Gb Fibre Channel Adapter
-
-
確認您使用的是建議的 Emulex 主機匯流排介面卡( HBA )韌體版本:
cat /sys/class/scsi_host/host*/fwrev
-
輸出範例 * :
14.0.639.20, sli-4:2:c 14.0.639.20, sli-4:2:c
-
-
確認您使用的是建議的 lpfc 驅動程式版本:
cat /sys/module/lpfc/version
-
輸出範例 * :
0:14.2.0.13
-
-
確認您可以檢視啟動器連接埠:
cat /sys/class/fc_host/host*/port_name
-
輸出範例 * :
0x100000109b579d5e 0x100000109b579d5f
-
-
驗證啟動器連接埠是否在線上:
cat /sys/class/fc_host/host*/port_state
-
輸出範例 * :
Online Online
-
-
確認已啟用 NVMe / FC 啟動器連接埠、且目標連接埠可見:
cat /sys/class/scsi_host/host*/nvme_info
-
輸出範例 * :
在此範例中、會啟用一個啟動器連接埠、並與兩個目標生命體連線。
NVME Initiator Enabled XRI Dist lpfc0 Total 6144 IO 5894 ELS 250 NVME LPORT lpfc0 WWPN x100000109b579d5e WWNN x200000109b579d5e DID x011c00 ONLINE NVME RPORT WWPN x208400a098dfdd91 WWNN x208100a098dfdd91 DID x011503 TARGET DISCSRVC ONLINE NVME RPORT WWPN x208500a098dfdd91 WWNN x208100a098dfdd91 DID x010003 TARGET DISCSRVC *ONLINE NVME Statistics LS: Xmt 0000000e49 Cmpl 0000000e49 Abort 00000000 LS XMIT: Err 00000000 CMPL: xb 00000000 Err 00000000 Total FCP Cmpl 000000003ceb594f Issue 000000003ce65dbe OutIO fffffffffffb046f abort 00000bd2 noxri 00000000 nondlp 00000000 qdepth 00000000 wqerr 00000000 err 00000000 FCP CMPL: xb 000014f4 Err 00012abd NVME Initiator Enabled XRI Dist lpfc1 Total 6144 IO 5894 ELS 250 NVME LPORT lpfc1 WWPN x100000109b579d5f WWNN x200000109b579d5f DID x011b00 ONLINE NVME RPORT WWPN x208300a098dfdd91 WWNN x208100a098dfdd91 DID x010c03 TARGET DISCSRVC ONLINE NVME RPORT WWPN x208200a098dfdd91 WWNN x208100a098dfdd91 DID x012a03 TARGET DISCSRVC ONLINE NVME Statistics LS: Xmt 0000000e50 Cmpl 0000000e50 Abort 00000000 LS XMIT: Err 00000000 CMPL: xb 00000000 Err 00000000 Total FCP Cmpl 000000003c9859ca Issue 000000003c93515e OutIO fffffffffffaf794 abort 00000b73 noxri 00000000 nondlp 00000000 qdepth 00000000 wqerr 00000000 err 00000000 FCP CMPL: xb 0000159d Err 000135c3
-
-
重新啟動主機。
-
SLES 15 SP5 核心隨附的原生收件匣 qla2xxx 驅動程式具有 ONTAP 支援所需的最新修正程式。確認您執行的是支援的介面卡驅動程式和韌體版本:
cat /sys/class/fc_host/host*/symbolic_name
-
輸出範例 * :
QLE2742 FW:v9.12.01 DVR: v10.02.08.300-k QLE2742 FW:v9.12.01 DVR: v10.02.08.300-k
-
-
確認
ql2xnvmeenable
參數設為 1 :cat /sys/module/qla2xxx/parameters/ql2xnvmeenable 1
啟用1MB I/O大小(選用)
ONTAP 在識別控制器資料中報告的 MDTS ( MAX Data 傳輸大小)為 8 、表示最大 I/O 要求大小可達 1MB 。不過、若要針對 Broadcom NVMe / FC 主機發出大小為 1 MB 的 I/O 要求、您必須增加 lpfc
的價值 lpfc_sg_seg_cnt
從預設值 64 到 256 。
-
將「lfc_sg_seg_cnt"參數設為256。
# cat /etc/modprobe.d/lpfc.conf options lpfc lpfc_sg_seg_cnt=256
-
執行「dracut -f」命令、然後重新啟動主機。
-
驗證「lfc_sg_seg_cnt"是否為256。
# cat /sys/module/lpfc/parameters/lpfc_sg_seg_cnt 256
|
這不適用於 Qlogic NVMe / FC 主機。 |
啟用 NVMe 服務
中包含兩個 NVMe / FC 開機服務 nvme-cli
不過、套件( only ) nvmefc-boot-connections.service
可在系統開機期間啟動; nvmf-autoconnect.service
未啟用。因此、您需要手動啟用 nvmf-autoconnect.service
以在系統開機期間啟動。
-
啟用
nvmf-autoconnect.service
:# systemctl enable nvmf-autoconnect.service Created symlink /etc/systemd/system/default.target.wants/nvmf- autoconnect.service → /usr/lib/systemd/system/nvmf-autoconnect.service.
-
重新啟動主機。
-
請確認
nvmf-autoconnect.service
和nvmefc-boot-connections.service
系統開機後正在執行:-
輸出範例: *
# systemctl status nvmf-autoconnect.service nvmf-autoconnect.service - Connect NVMe-oF subsystems automatically during boot Loaded: loaded (/usr/lib/systemd/system/nvmf-autoconnect.service; enabled; vendor preset: disabled) Active: inactive (dead) since Thu 2023-05-25 14:55:00 IST; 11min ago Process: 2108 ExecStartPre=/sbin/modprobe nvme-fabrics (code=exited, status=0/SUCCESS) Process: 2114 ExecStart=/usr/sbin/nvme connect-all (code=exited, status=0/SUCCESS) Main PID: 2114 (code=exited, status=0/SUCCESS) systemd[1]: Starting Connect NVMe-oF subsystems automatically during boot... nvme[2114]: traddr=nn-0x201700a098fd4ca6:pn-0x201800a098fd4ca6 is already connected systemd[1]: nvmf-autoconnect.service: Deactivated successfully. systemd[1]: Finished Connect NVMe-oF subsystems automatically during boot. # systemctl status nvmefc-boot-connections.service nvmefc-boot-connections.service - Auto-connect to subsystems on FC-NVME devices found during boot Loaded: loaded (/usr/lib/systemd/system/nvmefc-boot- connections.service; enabled; vendor preset: enabled) Active: inactive (dead) since Thu 2023-05-25 14:55:00 IST; 11min ago Main PID: 1647 (code=exited, status=0/SUCCESS) systemd[1]: Starting Auto-connect to subsystems on FC-NVME devices found during boot... systemd[1]: nvmefc-boot-connections.service: Succeeded. systemd[1]: Finished Auto-connect to subsystems on FC-NVME devices found during boot.
-
設定NVMe/TCP
您可以使用下列程序來設定 NVMe / TCP 。
-
確認啟動器連接埠可在支援的NVMe/TCP LIF中擷取探索記錄頁面資料:
nvme discover -t tcp -w <host-traddr> -a <traddr>
-
輸出範例 * :
# nvme discover -t tcp -w 192.168.1.4 -a 192.168.1.31 Discovery Log Number of Records 8, Generation counter 18 =====Discovery Log Entry 0====== trtype: tcp adrfam: ipv4 subtype: current discovery subsystem treq: not specified portid: 0 trsvcid: 8009 subnqn: nqn.1992- 08.com.netapp:sn.48391d66c0a611ecaaa5d039ea165514:discovery traddr: 192.168.2.117 eflags: explicit discovery connections, duplicate discovery information sectype: none =====Discovery Log Entry 1====== trtype: tcp adrfam: ipv4 subtype: current discovery subsystem treq: not specified portid: 1 trsvcid: 8009 subnqn: nqn.1992- 08.com.netapp:sn.48391d66c0a611ecaaa5d039ea165514:discovery traddr: 192.168.1.117 eflags: explicit discovery connections, duplicate discovery information sectype: none =====Discovery Log Entry 2====== trtype: tcp adrfam: ipv4 subtype: current discovery subsystem treq: not specified portid: 2 trsvcid: 8009 subnqn: nqn.1992- 08.com.netapp:sn.48391d66c0a611ecaaa5d039ea165514:discovery traddr: 192.168.2.116 eflags: explicit discovery connections, duplicate discovery information sectype: none =====Discovery Log Entry 3====== trtype: tcp adrfam: ipv4 subtype: current discovery subsystem treq: not specified portid: 3 trsvcid: 8009 subnqn: nqn.1992- 08.com.netapp:sn.48391d66c0a611ecaaa5d039ea165514:discovery traddr: 192.168.1.116 eflags: explicit discovery connections, duplicate discovery information sectype: none =====Discovery Log Entry 4====== trtype: tcp adrfam: ipv4 subtype: nvme subsystem treq: not specified portid: 0 trsvcid: 4420 subnqn: nqn.1992- 08.com.netapp:sn.48391d66c0a611ecaaa5d039ea165514:subsystem.subsys_CLIEN T116 traddr: 192.168.2.117 eflags: not specified sectype: none =====Discovery Log Entry 5====== trtype: tcp adrfam: ipv4 subtype: nvme subsystem treq: not specified portid: 1 trsvcid: 4420 subnqn: nqn.1992- 08.com.netapp:sn.48391d66c0a611ecaaa5d039ea165514:subsystem.subsys_CLIEN T116 traddr: 192.168.1.117 eflags: not specified sectype: none =====Discovery Log Entry 6====== trtype: tcp adrfam: ipv4 subtype: nvme subsystem treq: not specified portid: 2 trsvcid: 4420 subnqn: nqn.1992- 08.com.netapp:sn.48391d66c0a611ecaaa5d039ea165514:subsystem.subsys_CLIEN T116 traddr: 192.168.2.116 eflags: not specified sectype: none =====Discovery Log Entry 7====== trtype: tcp adrfam: ipv4 subtype: nvme subsystem treq: not specified portid: 3 trsvcid: 4420 subnqn: nqn.1992- 08.com.netapp:sn.48391d66c0a611ecaaa5d039ea165514:subsystem.subsys_CLIEN T116 traddr: 192.168.1.116 eflags: not specified sectype: none
-
-
確認所有其他的 NVMe / TCP 啟動器目標 LIF 組合都能成功擷取探索記錄頁面資料:
nvme discover -t tcp -w <host-traddr> -a <traddr>
-
輸出範例: *
# nvme discover -t tcp -w 192.168.1.4 -a 192.168.1.32 # nvme discover -t tcp -w 192.168.2.5 -a 192.168.2.36 # nvme discover -t tcp -w 192.168.2.5 -a 192.168.2.37
-
-
執行
nvme connect-all
跨所有節點支援的 NVMe / TCP 啟動器目標生命體執行命令:nvme connect-all -t tcp -w host-traddr -a traddr -l <ctrl_loss_timeout_in_seconds>
-
輸出範例: *
# nvme connect-all -t tcp -w 192.168.1.4 -a 192.168.1.31 -l -1 # nvme connect-all -t tcp -w 192.168.1.4 -a 192.168.1.32 -l -1 # nvme connect-all -t tcp -w 192.168.2.5 -a 192.168.1.36 -l -1 # nvme connect-all -t tcp -w 192.168.2.5 -a 192.168.1.37 -l -1
+
NetApp 建議您設定 ctrl-loss-tmo
選項-1
如此一來、一旦路徑遺失、 NVMe / TCP 啟動器就會嘗試無限期重新連線。 -
驗證NVMe
您可以使用下列程序來驗證 NVMe 。
-
確認已啟用核心內建 NVMe 多重路徑:
cat /sys/module/nvme_core/parameters/multipath Y
-
確認主機具有適用於 ONTAP NVMe 命名空間的正確控制器機型:
cat /sys/class/nvme-subsystem/nvme-subsys*/model
-
輸出範例: *
NetApp ONTAP Controller NetApp ONTAP Controller
-
-
確認個別 ONTAP NVMe I/O 控制器的 NVMe I/O 原則:
cat /sys/class/nvme-subsystem/nvme-subsys*/iopolicy
-
輸出範例: *
round-robin round-robin
-
-
確認主機可以看到 ONTAP 命名空間:
nvme list -v
-
輸出範例: *
Subsystem Subsystem-NQN Controllers ---------------- ------------------------------------------------------------------------------------ ----------------------- nvme-subsys0 nqn.1992- 08.com.netapp:sn.0501daf15dda11eeab68d039eaa7a232:subsystem.unidir_dhcha p nvme0, nvme1, nvme2, nvme3 Device SN MN FR TxPort Asdress Subsystem Namespaces -------- -------------------- ---------------------------------------- -------- --------------------------------------------- nvme0 81LGgBUqsI3EAAAAAAAE NetApp ONTAP Controller FFFFFFFF tcp traddr=192.168.2.214,trsvcid=4420,host_traddr=192.168.2.14 nvme-subsys0 nvme0n1 nvme1 81LGgBUqsI3EAAAAAAAE NetApp ONTAP Controller FFFFFFFF tcp traddr=192.168.2.215,trsvcid=4420,host_traddr=192.168.2.14 nvme-subsys0 nvme0n1 nvme2 81LGgBUqsI3EAAAAAAAE NetApp ONTAP Controller FFFFFFFF tcp traddr=192.168.1.214,trsvcid=4420,host_traddr=192.168.1.14 nvme-subsys0 nvme0n1 nvme3 81LGgBUqsI3EAAAAAAAE NetApp ONTAP Controller FFFFFFFF tcp traddr=192.168.1.215,trsvcid=4420,host_traddr=192.168.1.14 nvme-subsys0 nvme0n1 Device Generic NSID Usage Format Controllers ------------ ------------ ---------- ------------------------------------------------------------- /dev/nvme0n1 /dev/ng0n1 0x1 1.07 GB / 1.07 GB 4 KiB + 0 B nvme0, nvme1, nvme2, nvme3
-
-
確認每個路徑的控制器狀態均為有效、且具有正確的ANA狀態:
nvme list-subsys /dev/<subsystem_name>
NVMe / FC-
輸出範例 *
# nvme list-subsys /dev/nvme1n1 nvme-subsys1 - NQN=nqn.1992-08.com.netapp:sn.04ba0732530911ea8e8300a098dfdd91:subsystem.nvme_145_1 \ +- nvme2 fc traddr=nn-0x208100a098dfdd91:pn- 0x208200a098dfdd91,host_traddr=nn-0x200000109b579d5f:pn-0x100000109b579d5f live optimized +- nvme3 fc traddr=nn-0x208100a098dfdd91:pn- 0x208500a098dfdd91,host_traddr=nn-0x200000109b579d5e:pn-0x100000109b579d5e live optimized +- nvme4 fc traddr=nn-0x208100a098dfdd91:pn- 0x208400a098dfdd91,host_traddr=nn-0x200000109b579d5e:pn-0x100000109b579d5e live non-optimized +- nvme6 fc traddr=nn-0x208100a098dfdd91:pn- 0x208300a098dfdd91,host_traddr=nn-0x200000109b579d5f:pn-0x100000109b579d5f live non-optimized
NVMe / TCP-
輸出範例 *
# nvme list-subsys nvme-subsys0 - NQN=nqn.1992-08.com.netapp:sn.0501daf15dda11eeab68d039eaa7a232:subsystem.unidir_dhchap hostnqn=nqn.2014-08.org.nvmexpress:uuid:e58eca24-faff-11ea-8fee-3a68dd3b5c5f iopolicy=round-robin +- nvme0 tcp traddr=192.168.2.214,trsvcid=4420,host_traddr=192.168.2.14 live +- nvme1 tcp traddr=192.168.2.215,trsvcid=4420,host_traddr=192.168.2.14 live +- nvme2 tcp traddr=192.168.1.214,trsvcid=4420,host_traddr=192.168.1.14 live +- nvme3 tcp traddr=192.168.1.215,trsvcid=4420,host_traddr=192.168.1.14 live
-
-
驗證NetApp外掛程式是否顯示每ONTAP 個版本名稱空間裝置的正確值:
欄位nvme netapp ontapdevices -o column
-
輸出範例 * :
Device Vserver Namespace Path NSID UUID Size ---------------- ------------------------- ----------------------------------------------------------------------------------------------- /dev/nvme0n1 vs_CLIENT114 /vol/CLIENT114_vol_0_10/CLIENT114_ns10 1 c6586535-da8a-40fa-8c20-759ea0d69d33 1.07GB
JSONnvme netapp ontapdevices -o json
-
輸出範例 * :
{ "ONTAPdevices":[ { "Device":"/dev/nvme0n1", "Vserver":"vs_CLIENT114", "Namespace_Path":"/vol/CLIENT114_vol_0_10/CLIENT114_ns10", "NSID":1, "UUID":"c6586535-da8a-40fa-8c20-759ea0d69d33", "Size":"1.07GB", "LBA_Data_Size":4096, "Namespace_Size":262144 } ] }
-
建立持續探索控制器
從 ONTAP 9.11.1 開始、您可以使用下列程序、為 SLES 15 SP5 主機建立持續探索控制器( PDC )。需要有 PDC 才能自動偵測 NVMe 子系統新增或移除案例、以及探索記錄頁面資料的變更。
-
確認探索記錄頁面資料可用、並可透過啟動器連接埠和目標 LIF 組合擷取:
nvme discover -t <trtype> -w <host-traddr> -a <traddr>
輸出範例:
Discovery Log Number of Records 16, Generation counter 14 =====Discovery Log Entry 0====== trtype: tcp adrfam: ipv4 subtype: current discovery subsystem treq: not specified portid: 0 trsvcid: 8009 subnqn: nqn.1992-08.com.netapp:sn.0501daf15dda11eeab68d039eaa7a232:discovery traddr: 192.168.1.214 eflags: explicit discovery connections, duplicate discovery information sectype: none =====Discovery Log Entry 1====== trtype: tcp adrfam: ipv4 subtype: current discovery subsystem treq: not specified portid: 0 trsvcid: 8009 subnqn: nqn.1992-08.com.netapp:sn.0501daf15dda11eeab68d039eaa7a232:discovery traddr: 192.168.1.215 eflags: explicit discovery connections, duplicate discovery information sectype: none =====Discovery Log Entry 2====== trtype: tcp adrfam: ipv4 subtype: current discovery subsystem treq: not specified portid: 0 trsvcid: 8009 subnqn: nqn.1992-08.com.netapp:sn.0501daf15dda11eeab68d039eaa7a232:discovery traddr: 192.168.2.215 eflags: explicit discovery connections, duplicate discovery information sectype: none =====Discovery Log Entry 3====== trtype: tcp adrfam: ipv4 subtype: current discovery subsystem treq: not specified portid: 0 trsvcid: 8009 subnqn: nqn.1992-08.com.netapp:sn.0501daf15dda11eeab68d039eaa7a232:discovery traddr: 192.168.2.214 eflags: explicit discovery connections, duplicate discovery information sectype: none =====Discovery Log Entry 4====== trtype: tcp adrfam: ipv4 subtype: nvme subsystem treq: not specified portid: 0 trsvcid: 4420 subnqn: nqn.1992-08.com.netapp:sn.0501daf15dda11eeab68d039eaa7a232:subsystem.unidir_none traddr: 192.168.1.214 eflags: none sectype: none =====Discovery Log Entry 5====== trtype: tcp adrfam: ipv4 subtype: nvme subsystem treq: not specified portid: 0 trsvcid: 4420 subnqn: nqn.1992-08.com.netapp:sn.0501daf15dda11eeab68d039eaa7a232:subsystem.unidir_none traddr: 192.168.1.215 eflags: none sectype: none =====Discovery Log Entry 6====== trtype: tcp adrfam: ipv4 subtype: nvme subsystem treq: not specified portid: 0 trsvcid: 4420 subnqn: nqn.1992-08.com.netapp:sn.0501daf15dda11eeab68d039eaa7a232:subsystem.unidir_none traddr: 192.168.2.215 eflags: none sectype: none =====Discovery Log Entry 7====== trtype: tcp adrfam: ipv4 subtype: nvme subsystem treq: not specified portid: 0 trsvcid: 4420 subnqn: nqn.1992-08.com.netapp:sn.0501daf15dda11eeab68d039eaa7a232:subsystem.unidir_none traddr: 192.168.2.214 eflags: none sectype: none =====Discovery Log Entry 8====== trtype: tcp adrfam: ipv4 subtype: nvme subsystem treq: not specified portid: 0 trsvcid: 4420 subnqn: nqn.1992-08.com.netapp:sn.0501daf15dda11eeab68d039eaa7a232:subsystem.subsys_CLIENT114 traddr: 192.168.1.214 eflags: none sectype: none =====Discovery Log Entry 9====== trtype: tcp adrfam: ipv4 subtype: nvme subsystem treq: not specified portid: 0 trsvcid: 4420 subnqn: nqn.1992-08.com.netapp:sn.0501daf15dda11eeab68d039eaa7a232:subsystem.subsys_CLIENT114 traddr: 192.168.1.215 eflags: none sectype: none =====Discovery Log Entry 10====== trtype: tcp adrfam: ipv4 subtype: nvme subsystem treq: not specified portid: 0 trsvcid: 4420 subnqn: nqn.1992-08.com.netapp:sn.0501daf15dda11eeab68d039eaa7a232:subsystem.subsys_CLIENT114 traddr: 192.168.2.215 eflags: none sectype: none =====Discovery Log Entry 11====== trtype: tcp adrfam: ipv4 subtype: nvme subsystem treq: not specified portid: 0 trsvcid: 4420 subnqn: nqn.1992-08.com.netapp:sn.0501daf15dda11eeab68d039eaa7a232:subsystem.subsys_CLIENT114 traddr: 192.168.2.214 eflags: none sectype: none =====Discovery Log Entry 12====== trtype: tcp adrfam: ipv4 subtype: nvme subsystem treq: not specified portid: 0 trsvcid: 4420 subnqn: nqn.1992-08.com.netapp:sn.0501daf15dda11eeab68d039eaa7a232:subsystem.unidir_dhchap traddr: 192.168.1.214 eflags: none sectype: none =====Discovery Log Entry 13====== trtype: tcp adrfam: ipv4 subtype: nvme subsystem treq: not specified portid: 0 trsvcid: 4420 subnqn: nqn.1992-08.com.netapp:sn.0501daf15dda11eeab68d039eaa7a232:subsystem.unidir_dhchap traddr: 192.168.1.215 eflags: none sectype: none =====Discovery Log Entry 14====== trtype: tcp adrfam: ipv4 subtype: nvme subsystem treq: not specified portid: 0 trsvcid: 4420 subnqn: nqn.1992-08.com.netapp:sn.0501daf15dda11eeab68d039eaa7a232:subsystem.unidir_dhchap traddr: 192.168.2.215 eflags: none sectype: none =====Discovery Log Entry 15====== trtype: tcp adrfam: ipv4 subtype: nvme subsystem treq: not specified portid: 0 trsvcid: 4420 subnqn: nqn.1992-08.com.netapp:sn.0501daf15dda11eeab68d039eaa7a232:subsystem.unidir_dhchap traddr: 192.168.2.214 eflags: none sectype: none
-
建立探索子系統的 PDC :
nvme discover -t <trtype> -w <host-traddr> -a <traddr> -p
-
輸出範例: *
nvme discover -t tcp -w 192.168.1.16 -a 192.168.1.116 -p
-
-
從 ONTAP 控制器、確認已建立 PDC :
vserver nvme show-discovery-controller -instance -vserver vserver_name
-
輸出範例: *
vserver nvme show-discovery-controller -instance -vserver vs_nvme175 Vserver Name: vs_CLIENT116 Controller ID: 00C0h Discovery Subsystem NQN: nqn.1992- 08.com.netapp:sn.48391d66c0a611ecaaa5d039ea165514:discovery Logical Interface UUID: d23cbb0a-c0a6-11ec-9731-d039ea165abc Logical Interface: CLIENT116_lif_4a_1 Node: A400-14-124 Host NQN: nqn.2014-08.org.nvmexpress:uuid:12372496-59c4-4d1b-be09- 74362c0c1afc Transport Protocol: nvme-tcp Initiator Transport Address: 192.168.1.16 Host Identifier: 59de25be738348f08a79df4bce9573f3 Admin Queue Depth: 32 Header Digest Enabled: false Data Digest Enabled: false Vserver UUID: 48391d66-c0a6-11ec-aaa5-d039ea165514
-
設定安全的頻內驗證
從 ONTAP 9.12.1 開始、在 SLES 15 SP5 主機和 ONTAP 控制器之間、透過 NVMe / TCP 和 NVMe / FC 支援安全頻內驗證。
若要設定安全驗證、每個主機或控制器都必須與相關聯 DH-HMAC-CHAP
金鑰、這是 NVMe 主機或控制器的 NQN 組合、以及管理員設定的驗證密碼。若要驗證其對等端點、 NVMe 主機或控制器必須識別與對等端點相關的金鑰。
您可以使用 CLI 或組態 JSON 檔案來設定安全的頻內驗證。如果您需要為不同的子系統指定不同的 dhchap 金鑰、則必須使用組態 JSON 檔案。
-
取得主機 NQN :
cat /etc/nvme/hostnqn
-
為 SLES15 SP5 主機產生 dhchap 金鑰:
nvme gen-dhchap-key -s optional_secret -l key_length {32|48|64} -m HMAC_function {0|1|2|3} -n host_nqn • -s secret key in hexadecimal characters to be used to initialize the host key • -l length of the resulting key in bytes • -m HMAC function to use for key transformation 0 = none, 1- SHA-256, 2 = SHA-384, 3=SHA-512 • -n host NQN to use for key transformation
在下列範例中、會產生一個隨機的 dhchap 金鑰、其中 HMAC 設為 3 ( SHA-512 )。
# nvme gen-dhchap-key -m 3 -n nqn.2014-08.org.nvmexpress:uuid:d3ca725a- ac8d-4d88-b46a-174ac235139b DHHC-1:03:J2UJQfj9f0pLnpF/ASDJRTyILKJRr5CougGpGdQSysPrLu6RW1fGl5VSjbeDF1n1DEh3nVBe19nQ/LxreSBeH/bx/pU=:
-
在 ONTAP 控制器上、新增主機並指定兩個 dhchap 金鑰:
vserver nvme subsystem host add -vserver <svm_name> -subsystem <subsystem> -host-nqn <host_nqn> -dhchap-host-secret <authentication_host_secret> -dhchap-controller-secret <authentication_controller_secret> -dhchap-hash-function {sha-256|sha-512} -dhchap-group {none|2048-bit|3072-bit|4096-bit|6144-bit|8192-bit}
-
主機支援兩種驗證方法:單向和雙向。在主機上、連線至 ONTAP 控制器、並根據所選的驗證方法指定 dhchap 金鑰:
nvme connect -t tcp -w <host-traddr> -a <tr-addr> -n <host_nqn> -S <authentication_host_secret> -C <authentication_controller_secret>
-
驗證
nvme connect authentication
命令驗證主機和控制器 dhchap 金鑰:-
驗證主機 dhchap 金鑰:
$cat /sys/class/nvme-subsystem/<nvme-subsysX>/nvme*/dhchap_secret
-
單向組態的輸出範例: *
# cat /sys/class/nvme-subsystem/nvme-subsys1/nvme*/dhchap_secret DHHC-1:03:je1nQCmjJLUKD62mpYbzlpuw0OIws86NB96uNO/t3jbvhp7fjyR9bIRjOHg8wQtye1JCFSMkBQH3pTKGdYR1OV9gx00=: DHHC-1:03:je1nQCmjJLUKD62mpYbzlpuw0OIws86NB96uNO/t3jbvhp7fjyR9bIRjOHg8wQtye1JCFSMkBQH3pTKGdYR1OV9gx00=: DHHC-1:03:je1nQCmjJLUKD62mpYbzlpuw0OIws86NB96uNO/t3jbvhp7fjyR9bIRjOHg8wQtye1JCFSMkBQH3pTKGdYR1OV9gx00=: DHHC-1:03:je1nQCmjJLUKD62mpYbzlpuw0OIws86NB96uNO/t3jbvhp7fjyR9bIRjOHg8wQtye1JCFSMkBQH3pTKGdYR1OV9gx00=:
-
-
驗證控制器 dhchap 按鍵:
$cat /sys/class/nvme-subsystem/<nvme-subsysX>/nvme*/dhchap_ctrl_secret
-
雙向組態輸出範例: *
-
# cat /sys/class/nvme-subsystem/nvme-subsys6/nvme*/dhchap_ctrl_secret DHHC-1:03:WorVEV83eYO53kV4Iel5OpphbX5LAphO3F8fgH3913tlrkSGDBJTt3crXeTUB8fCwGbPsEyz6CXxdQJi6kbn4IzmkFU=: DHHC-1:03:WorVEV83eYO53kV4Iel5OpphbX5LAphO3F8fgH3913tlrkSGDBJTt3crXeTUB8fCwGbPsEyz6CXxdQJi6kbn4IzmkFU=: DHHC-1:03:WorVEV83eYO53kV4Iel5OpphbX5LAphO3F8fgH3913tlrkSGDBJTt3crXeTUB8fCwGbPsEyz6CXxdQJi6kbn4IzmkFU=: DHHC-1:03:WorVEV83eYO53kV4Iel5OpphbX5LAphO3F8fgH3913tlrkSGDBJTt3crXeTUB8fCwGbPsEyz6CXxdQJi6kbn4IzmkFU=:
-
您可以使用 /etc/nvme/config.json
使用檔案 nvme connect-all
ONTAP 控制器組態上有多個 NVMe 子系統可用時的命令。
您可以使用產生 JSON 檔案 -o
選項。如需更多語法選項、請參閱 NVMe CONNECT ALL 手冊頁。
-
設定Json檔案:
# cat /etc/nvme/config.json [ { "hostnqn":"nqn.2014-08.org.nvmexpress:uuid:12372496-59c4-4d1b-be09-74362c0c1afc", "hostid":"3ae10b42-21af-48ce-a40b-cfb5bad81839", "dhchap_key":"DHHC-1:03:Cu3ZZfIz1WMlqZFnCMqpAgn/T6EVOcIFHez215U+Pow8jTgBF2UbNk3DK4wfk2EptWpna1rpwG5CndpOgxpRxh9m41w=:" }, { "hostnqn":"nqn.2014-08.org.nvmexpress:uuid:12372496-59c4-4d1b-be09-74362c0c1afc", "subsystems":[ { "nqn":"nqn.1992-08.com.netapp:sn.48391d66c0a611ecaaa5d039ea165514:subsystem.subsys_CLIENT116", "ports":[ { "transport":"tcp", "traddr":"192.168.1.117", "host_traddr":"192.168.1.16", "trsvcid":"4420", "dhchap_ctrl_key":"DHHC-1:01:0h58bcT/uu0rCpGsDYU6ZHZvRuVqsYKuBRS0Nu0VPx5HEwaZ:" }, { "transport":"tcp", "traddr":"192.168.1.116", "host_traddr":"192.168.1.16", "trsvcid":"4420", "dhchap_ctrl_key":"DHHC-1:01:0h58bcT/uu0rCpGsDYU6ZHZvRuVqsYKuBRS0Nu0VPx5HEwaZ:" }, { "transport":"tcp", "traddr":"192.168.2.117", "host_traddr":"192.168.2.16", "trsvcid":"4420", "dhchap_ctrl_key":"DHHC-1:01:0h58bcT/uu0rCpGsDYU6ZHZvRuVqsYKuBRS0Nu0VPx5HEwaZ:" }, { "transport":"tcp", "traddr":"192.168.2.116", "host_traddr":"192.168.2.16", "trsvcid":"4420", "dhchap_ctrl_key":"DHHC-1:01:0h58bcT/uu0rCpGsDYU6ZHZvRuVqsYKuBRS0Nu0VPx5HEwaZ:" } ] } ] } ] [NOTE] In the preceding example, `dhchap_key` corresponds to `dhchap_secret` and `dhchap_ctrl_key` corresponds to `dhchap_ctrl_secret`.
-
使用組態 JSON 檔案連線至 ONTAP 控制器:
nvme connect-all -J /etc/nvme/config.json
-
輸出範例 * :
traddr=192.168.2.116 is already connected traddr=192.168.1.116 is already connected traddr=192.168.2.117 is already connected traddr=192.168.1.117 is already connected traddr=192.168.2.117 is already connected traddr=192.168.1.117 is already connected traddr=192.168.2.116 is already connected traddr=192.168.1.116 is already connected traddr=192.168.2.116 is already connected traddr=192.168.1.116 is already connected traddr=192.168.2.117 is already connected traddr=192.168.1.117 is already connected
-
-
確認已為每個子系統的個別控制器啟用 dhchap 機密:
-
驗證主機 dhchap 金鑰:
# cat /sys/class/nvme-subsystem/nvme-subsys0/nvme0/dhchap_secret
-
輸出範例: *
DHHC-1:01:NunEWY7AZlXqxITGheByarwZdQvU4ebZg9HOjIr6nOHEkxJg:
-
-
驗證控制器 dhchap 按鍵:
# cat /sys/class/nvme-subsystem/nvme-subsys0/nvme0/dhchap_ctrl_secret
-
輸出範例: *
-
DHHC- 1:03:2YJinsxa2v3+m8qqCiTnmgBZoH6mIT6G/6f0aGO8viVZB4VLNLH4z8CvK7pV YxN6S5fOAtaU3DNi12rieRMfdbg3704=:
-
已知問題
在 ONTAP 版本中、 SLES 15 SP5 沒有已知問題。