Skip to main content

NFS over RDMA overview

Contributors netapp-ahibbard netapp-aherbin netapp-mwallis netapp-dbagwell
PDFs

NFS over RDMA utilizes RDMA adapters, allowing data to be copied directly between storage system memory and host system memory, circumventing CPU interruptions and overhead.

NFS over RDMA configurations are designed for customers with latency sensitive or high-bandwidth workloads such as machine learning and analytics. NVIDIA has extended NFS over RDMA to enable GPU Direct Storage (GDS). GDS further accelerates GPU-enabled workloads by bypassing the CPU and main memory altogether, using RDMA to transfer data between the storage system and GPU memory directly.

Beginning with ONTAP 9.10.1, NFS over RDMA configurations are supported for the NFSv4.0 protocol when used with the Mellanox CX-5 or CX-6 adapter, which provides support for RDMA using version 2 of the RoCE protocol. GDS is only supported using NVIDIA Tesla- and Ampere-family GPUs with Mellanox NIC cards and MOFED software. Refer to the chart in requirements to understand NFS version support in subsequent ONTAP releases.

Note NFS mount sizes greater than 64k result in unstable performance with NFS over RDMA configurations.
Requirements

  • Storages systems must be running ONTAP 9.10.1 or later.

  • Ensure you are running the correct version of ONTAP for the NFS version you want to use.

    NFS version ONTAP support

    NFSv4.0

    ONTAP 9.10.1 and later

    NFSv4.1

    ONTAP 9.14.1 and later

    NFSv3

    ONTAP 9.15.1 and later

    • You can configure NFS over RDMA with System Manager beginning with ONTAP 9.12.1. In ONTAP 9.10.1 and 9.11.1, you need to use the CLI to configure NFS over RDMA.

  • Both nodes in the HA pair must be the same version.

  • Storage system controllers must have RDMA support

    Beginning in ONTAP…​ The following controllers support RDMA…​

    9.10.1 and later

    • AFF A400

    • AFF A700

    • AFF A800

    ONTAP 9.14.1 and later

    • AFF C-Series

    • AFF A900

    ONTAP 9.15.1 and later

    • AFF A1K

    • AFF A90

    • AFF A70

    ONTAP 9.16.1 and later

    • AFF A50

    • AFF A30

    • AFF A20

  • Data LIFs must be configured to support RDMA.

  • Clients must be using Mellanox RDMA-capable NIC cards and Mellanox OFED (MOFED) network software. For information about adapter support, refer to the NetApp Hardware Universe. NFS over RDMA supported adapters display "RoCE" in the description field.

Note Interface groups are not supported with NFS over RDMA.
Next steps
Related information