Skip to main content

Configure NICs for NFS over RDMA

Contributors netapp-ahibbard netapp-aherbin netapp-thomi
PDFs

NFS over RDMA requires NIC configuration for both the client system and storage platform.

Storage platform configuration

For information about target RNIC support, refer to the NetApp Hardware Universe.

If you are using a high-availability (HA) configuration, both nodes must use the same RNIC to support RDMA failover. The NIC must be RoCE capable.

  • Beginning with ONTAP 9.10.1, you can view a list of RDMA offload protocols with the command:

    network port show -rdma-protocols roce

  • Beginning with ONTAP 9.16.1, it is recommended to use priority flow control (PFC). Configure PFC using the network port modify command:

    network port modify -node <nodename> -port <portname> -flowcontrol-admin pfc -pfc-queues-admin 3

  • Prior to ONTAP 9.16.1, it is recommended to use the default global flow control (GFC). If the flow-control setting has been changed, configure GFC using the network port modify command:

    network port modify -node <nodename> -port <portname> -flowcontrol-admin full

Client system configuration

For information on supported client operating systems for NFS over RDMA, refer to the NetApp Interoperability Matrix (IMT). For RoCE v2 supported RNICs, refer to the respective RNIC vendor documentation.

Although the client and server can be directly connected, the use of switches is recommended for improved failover performance.

The client, server, any switches, and all ports on switches must be configured using jumbo frames. The flow-control configuration on the clients and switches should match ONTAP's flow-control configuration. Beginning with ONTAP 9.16.1, it is best practice to enable and configure priority flow control on ONTAP, the switches, and the clients. Prior to ONTAP 9.16.1, it is recommended to use global flow control.

After this configuration is confirmed, you can mount the NFS export using RDMA.

System Manager

You must be using ONTAP 9.12.1 or later to configure network interfaces with NFS over RDMA using System Manager.

Steps

  1. Check if RDMA is supported. Navigate to Network > Ethernet Ports and select the appropriate node in the group view. When you expand the node, look at the RDMA protocols field for a given port: the value RoCE denotes RDMA is supported; a dash (-) indicates it's not supported.

  2. To add a VLAN, select + VLAN. Select the appropriate node. In the Port dropdown menu, the available ports display the text RoCE Enabled if they support RDMA. No text is displayed if they do not support RDMA.

  3. Follow the workflow in Enable NAS storage for Linux servers using NFS to configure a new NFS server.

    When adding network interfaces, you will have the option to select Use RoCE ports. Select this option for any network interfaces that you want to use NFS over RDMA.

CLI

  1. Check if RDMA access is enabled on the NFS server with the command:

    vserver nfs show-vserver <SVM_name>

    By default, -rdma should be enabled. If it is not, enable RDMA access on the NFS server:

    vserver nfs modify -vserver <SVM_name> -rdma enabled

  2. Mount the client via NFSv4.0 over RDMA:

    1. The input for the proto parameter depends on the server IP protocol version. If it is IPv4, use proto=rdma. If it is IPv6, use proto=rdma6.

    2. Specify the NFS target port as port=20049 instead of the standard port 2049:

      mount -o vers=4,minorversion=0,proto=rdma,port=20049 <Server_IP_address>:/<volume_path> <mount_point>

  3. OPTIONAL: If you need to unmount the client, run the command unmount <mount_path>

More information