Skip to main content

Configure NICs for NFS over RDMA

Contributors netapp-ahibbard netapp-aherbin netapp-thomi
PDFs

NFS over RDMA requires NIC configuration for both the client system and storage platform.

Storage platform configuration

An X1148 RDMA adapter needs to be installed on the server. If you are using an HA configuration, you must have a corresponding X1148 adapter on the failover partner so RDMA service can continue during failover. The NIC must be ROCE capable.

Beginning with ONTAP 9.10.1, you can view a list of RDMA offload protocols with the command: network port show -rdma-protocols roce

Client system configuration

Clients must be using Mellanox RDMA-capable NIC cards (e.g. X1148) and Mellanox OFED network software. Consult Mellanox documentation for supported models and versions. Although the client and server can be directly connected, the use of switches is recommended due to improved failover performance with a switch.

The client, server, any switches, and all ports on switches must be configured using Jumbo frames. Also ensure that priority flow-control is in effect on any switches.

Once this configuration is confirmed, you can mount the NFS.

System Manager

You must be using ONTAP 9.12.1 or later to configure network interfaces with NFS over RDMA using System Manager.

Steps

  1. Check if RDMA is supported. Navigate to Network > Ethernet Ports and select the appropriate node in the group view. When you expand the node, look at the RDMA protocols field for a given port: the value RoCE denotes RDMA is supported; a dash (-) indicates it's not supported.

  2. To add a VLAN, select + VLAN. Select the appropriate node. In the Port dropdown menu, the available ports display the text RoCE Enabled if they support RDMA. No text is displayed if they do not support RDMA.

  3. Follow the workflow in Enable NAS storage for Linux servers using NFS to configure a new NFS server.

    When adding network interfaces, you will have the option to select Use RoCE ports. Select this option for any network interfaces that you want to use NFS over RDMA.

CLI

  1. Check if RDMA access is enabled on the NFS server with the command:

    vserver nfs show-vserver SVM_name

    By default, -rdma should be enabled. If it is not, enable RDMA access on the NFS server:

    vserver nfs modify -vserver SVM_name -rdma enabled

  2. Mount the client via NFSv4.0 over RDMA:

    1. The input for the proto parameter depends on the server IP protocol version. If it is IPv4, use proto=rdma. If it is IPv6, use proto=rdma6.

    2. Specify the NFS target port as port=20049 instead of the standard port 2049:

      mount -o vers=4,minorversion=0,proto=rdma,port=20049 Server_IP_address:/volume_path mount_point

  3. OPTIONAL: If you need to unmount the client, run the command unmount mount_path

More information