Configure NICs for NFS over RDMA

Contributors

NFS over RDMA requires NIC configuration for both the client system and storage platform.

Storage platform configuration

An X1148 RDMA adapter needs to be installed on the server. If you are using an HA configuration, you must have a corresponding X1148 adapter on the failover partner so RDMA service can continue during failover. The NIC must be ROCE capable.

Beginning in ONTAP 9.10.1, you can view a list of RDMA offload protocols with the command: network port show -rdma-protocols roce

Client system configuration

Clients must be using Mellanox RDMA-capable NIC cards (e.g. X1148) and Mellanox OFED network software. Consult Mellanox documentation for supported models and versions. Although the client and server can be directly connected, the use of switches is recommended due to improved failover performance with a switch.

The client, server, and any switches, and all ports on switches must be configured using Jumbo frames. Also ensure that priority flow-control is in effect on any switches.

Once this configuration is confirmed, you can mount the NFS.

Steps
  1. Check if RDMA access is enabled on the NFS server with the command:

    vserver nfs show-vserver vserver_name

    By default, -rdma should be enabled. If it is not, enable RDMA access on the NFS server:

    vserver nfs modify -vserver vserver_name -rdma enabled

  2. Mount the client via NFSv4.0 over RDMA:

    1. The input for the proto parameter depends on the server IP protocol version. If it is v4, use proto=rdma. If it is v6, use proto=rdma6.

    2. Specify the NFS target port as port=20049 instead of the standard port of 2049:

      mount -o vers=4,minorversion=0,proto=rdma,port=20049 Server_IP_address:/volume_path mount_point

  3. OPTIONAL: If you need to unmount the client, run the command unmount mount_path