Skip to main content
ONTAP SAN Host Utilities

Configure AIX with NVMe-oF for ONTAP storage

Contributors netapp-ranuk netapp-pcarriga netapp-sarajane
PDFs

The IBM AIX and Virtual I/O Server (VIOS)/PowerVM hosts supports the NVMe/FC protocol with Asymmetric Namespace Access (ANA). ANA is equivalent to asymmetric logical unit access (ALUA) multipathing in iSCSI and FCP environments.

For additional details on supported configurations, see the Interoperability Matrix Tool (IMT).

About this task

You can use the following support and features with the NVMe-oF host configuration for AIX hosts. You should also review the known limitations before starting the configuration process.

  • Support available:

    • Beginning with ONTAP 9.13.1, NVMe/FC support is added for IBM AIX 7.2 TL5 SP6, AIX 7.3 TL1 SP2, and VIOS 3.1.4.21 with SAN boot support for both physical and virtual stacks. See the IBM documentation for more information on setting up SAN boot support.

    • NVMe/FC is supported with Power9 and Power10 IBM servers.

    • A separate PCM (Path Control Module), such as Host Utilities for AIX SCSI Multipath I/O (MPIO) support, isn't required for NVMe devices.

    • Virtualization support with NetApp (VIOS/PowerVM) is introduced with VIOS 3.1.4.21. This is only supported through NPIV (N_PortID Virtualization) storage virtualization mode using the Power10 IBM server.

  • Known limitations:

    • Qlogic/Marvel 32G FC HBAs on an AIX host doesn't support NVMe/FC.

    • SAN boot isn't supported for NVMe/FC devices using Power9 IBM server.

Before you begin

  • Verify that you have 32GB FC Emulex adapters (EN1A, EN1B, EN1L, EN1M) or 64GB FC adapters (EN1N, EN1P) with adapter firmware 12.4.257.30 and later versions.

  • If you have a MetroCluster configuration, NetApp recommends changing the AIX NVMe/FC default APD (All Path Down) time for supporting MetroCluster unplanned switchover events to avoid the AIX operating system enforcing a shorter I/O timeout. For additional information and the recommended changes to default settings, refer to NetApp Bugs Online - 1553249.

  • Depending on your AIX version, the Asymmetric Namespace Access Transition Timeout (ANATT) for the AIX host OS is 30 seconds or 60 seconds by default. If the ANATT default for your host is 30 seconds, you need to install an IBM Interim Fix (ifix) from the IBM website that sets the ANATT to 60 seconds to ensure that all ONTAP workflows are non-disruptive.

    Note

    For NVMe/FC AIX support, you must install an ifix on the GA version of the AIX OS. The ifix isn't required for the VIOS/PowerVM OS.

    You need to install the ifixes on an AIX version with no previously installed ifixes related to devices.pciex.pciexclass.010802.rte on the system. Previously installed ifixes can conflict with the new installation.
    Set ANATT to 60 seconds

    The default ANATT for the AIX level 72-TL5-SP6-2320 and AIX level 73-TL1-SP2-2320 releases is 30 seconds. IBM provides an ifix that sets the ANATT to 60 seconds. The ifix is available through IBM case ID TS018079082 and you can install it for the following AIX releases:

    • For AIX level 72-TL5-SP6-2320, install the IJ46710s6a.230509.epkg.Z package.

    • For AIX level 73-TL1-SP2-2320, install the IJ46711s2a.230509.epkg.Z package.

    Default ANATT is 60 seconds

    The default ANATT is 60 seconds for the following AIX releases:

    • AIX level 73-TL2-SP3-2446

    • AIX level 73-TL2-SP2-2420

    • AIX level 72-TL5-SP8-2420

    Optionally, set ANATT to 120 seconds

    IBM provides an ifix that sets the ANATT to 120 seconds. When you set the ANATT to 120 seconds, it enhances performance during ONTAP storage failover events. The ifix is available through IBM case ID TS012877410 and you can install it for the following AIX releases:

    • For AIX level 73-TL3-SP0-2446, install the IJ53487s0a.250130.epkg.Z package.

    • For AIX level 72-TL5-SP9-2446, install the IJ53445s9a.250130.epkg.Z package.

    Note

    The minimum server firmware version for Power9 servers for NVMe/FC support is FW 950.

    The minimum server firmware version for Power10 servers for NVMe/FC support is FW 1010.

    For more information on managing ifixes, see Managing Interim Fixes on AIX.

Step 1: Confirm the multipath configuration for your host

When you install the AIX OS, IBM MPIO used for NVMe multipathing is enabled by default.

Steps

  1. Verify that NVMe multipathing is enabled:

    lsmpio -l hdisk1
    Show example 
    name     path_id  status   path_status  parent  connection
hdisk1  8         Enabled  Sel,Opt       nvme12  fcnvme0, 9
hdisk1  9         Enabled  Sel,Non       nvme65  fcnvme1, 9
hdisk1  10        Enabled  Sel,Opt       nvme37  fcnvme1, 9
hdisk1  11        Enabled  Sel,Non       nvme60  fcnvme0, 9

Step 2: Configure NVMe/FC

You need to configure NVMe/FC for Broadcom/Emulex adapters on VIOS because the NVMe/FC protocol support is disabled in the Virtual Fibre Channel (vFC) on VIOS. The NVMe/FC protocol support is enabled in the physical FC by default.

Steps

  1. Verify that you are using the supported adapter.

  2. Retrieve a list of virtual adapters:

    lsmap -all -npiv
    Show example 
    Name          Physloc                            ClntID ClntName       ClntOS
------------- ---------------------------------- ------ -------------- -------
vfchost0      U9105.22A.785DB61-V2-C2                 4 s1022-iop-mcc- AIX
Status:LOGGED_IN
FC name:fcs4                    FC loc code:U78DA.ND0.WZS01UY-P0-C7-T0
Ports logged in:3
Flags:0xea<LOGGED_IN,STRIP_MERGE,SCSI_CLIENT,NVME_CLIENT>
VFC client name:fcs0            VFC client DRC:U9105.22A.785DB61-V4-C2

  3. Enable support for the NVMe/FC protocol on an adapter by running the ioscli vfcctrl command on the VIOS:

    vfcctrl -enable -protocol nvme -vadapter vfchost0
    Example output
    The "nvme" protocol for "vfchost0" is enabled.

  4. Verify that the support has been enabled on the adapter:

    lsattr -El vfchost0
    Show example 
    alt_site_wwpn       WWPN to use - Only set after migration   False
current_wwpn  0     WWPN to use - Only set after migration   False
enable_nvme   yes   Enable or disable NVME protocol for NPIV True
label               User defined label                       True
limit_intr    false Limit NPIV Interrupt Sources             True
map_port      fcs4  Physical FC Port                         False
num_per_nvme  0     Number of NPIV NVME queues per range     True
num_per_range 0     Number of NPIV SCSI queues per range     True

  5. Enable the NVMe/Fc protocol for all adapters:

    1. Change the dflt_enabl_nvme attribute value of viosnpiv0 pseudo device to yes.

    2. Set the enable_nvme attribute value to yes for all the VFC host devices.

      chdev -l viosnpiv0 -a dflt_enabl_nvme=yes
      lsattr -El viosnpiv0
      Show example 
      bufs_per_cmd    10  NPIV Number of local bufs per cmd                    True
dflt_enabl_nvme yes Default NVME Protocol setting for a new NPIV adapter True
num_local_cmds  5   NPIV Number of local cmds per channel                True
num_per_nvme    8   NPIV Number of NVME queues per range                 True
num_per_range   8   NPIV Number of SCSI queues per range                 True
secure_va_info  no  NPIV Secure Virtual Adapter Information              True

  6. Enable the NVMe/Fc protocol for selected adapters by changing the enable_nvme value of the VFC host device attribute to yes.

  7. Verify that FC-NVMe Protocol Device has been created on the server:

    lsdev |grep fcnvme
    Exmaple output
    fcnvme0       Available 00-00-02    FC-NVMe Protocol Device
fcnvme1       Available 00-01-02    FC-NVMe Protocol Device

  8. Record the host NQN from the server:

    lsattr -El fcnvme0
    Show example 
    attach     switch                                                               How this adapter is connected  False
autoconfig available                                                            Configuration State            True
host_nqn   nqn.2014-08.org.nvmexpress:uuid:64e039bd-27d2-421c-858d-8a378dec31e8 Host NQN (NVMe Qualified Name) True
    lsattr -El fcnvme1
    Show example 
    attach     switch                                                               How this adapter is connected  False
autoconfig available                                                            Configuration State            True
host_nqn   nqn.2014-08.org.nvmexpress:uuid:64e039bd-27d2-421c-858d-8a378dec31e8 Host NQN (NVMe Qualified Name) True

  9. Check the host NQN and verify that it matches the host NQN string for the corresponding subsystem on the ONTAP array:

    vserver nvme subsystem host show -vserver vs_s922-55-lpar2
    Example output
    Vserver         Subsystem                Host NQN
------- --------- ----------------------------------------------------------
vs_s922-55-lpar2 subsystem_s922-55-lpar2 nqn.2014-08.org.nvmexpress:uuid:64e039bd-27d2-421c-858d-8a378dec31e8

  10. Verify that the initiator ports are up and running and you can see the target LIFs.

Step 3: Validate NVMe/FC

Verify that the ONTAP namespaces are correct for the NVMe/FC configuration.

Steps

  1. Verify that the ONTAP namespaces correctly reflect on the host:

    lsdev -Cc disk |grep NVMe
    Example output
    hdisk1  Available 00-00-02 NVMe 4K Disk

  2. Optionally, check the multipathing status:

    lsmpio -l hdisk1
    Show example 
    name     path_id  status   path_status  parent  connection
hdisk1  8        Enabled  Sel,Opt      nvme12  fcnvme0, 9
hdisk1  9        Enabled  Sel,Non      nvme65  fcnvme1, 9
hdisk1  10       Enabled  Sel,Opt      nvme37  fcnvme1, 9
hdisk1  11       Enabled  Sel,Non      nvme60  fcnvme0, 9

Step 4: Review the known issues

The NVMe/FC host configuration for AIX with ONTAP storage has the following known issues:

Burt ID Title Description

1553249

AIX NVMe/FC default APD time to be modified for supporting MCC Unplanned Switchover events

By default, AIX operating systems use an all path down (APD) timeout value of 20sec for NVMe/FC. However, ONTAP MetroCluster automatic unplanned switchover (AUSO) and TieBreaker initiated switchover workflows might take a little longer than the APD timeout window, causing I/O errors.

1546017

AIX NVMe/FC caps ANATT at 60s, instead of 120s as advertised by ONTAP

ONTAP advertises the ANA(asymmetric namespace access) transition timeout in controller identify at 120sec. Currently, with ifix, AIX reads the ANA transition timeout from controller identify, but effectively clamps it to 60sec if it is over that limit.

1541386

AIX NVMe/FC hits EIO after ANATT expiry

For any storage failover (SFO) events, if the ANA(asymmetric namespace access) transitioning exceeds the ANA transition timeout cap on a given path, the AIX NVMe/FC host fails with an I/O error despite having alternate healthy paths available to the namespace.

1541380

AIX NVMe/FC waits for half/full ANATT to expire before resuming I/O after ANA AEN

IBM AIX NVMe/FC does not support some Asynchronous notifications (AENs) that ONTAP publishes. This sub-optimal ANA handling will result in sub optimal performance during SFO operations.

Step 5: Troubleshoot

Before troubleshooting any NVMe/FC failures, verify that you are running a configuration that is compliant with the IMT specifications. If you continue to have issues, contact NetApp support.