pNFS in ONTAP works with both FlexVol volumes and FlexGroup volumes, but for the best overall results, use FlexGroup volumes.

FlexGroup volumes provide:

Learn about FlexGroup volumes management

Not all NFS clients support pNFS, but most modern clients do. RHEL 6.4 and Fedora 17 were the first supported pNFS clients (roughly in 2014), so it is reasonable to assume that client versions released in the past few years fully support the feature. ONTAP's NFS support stance is one of "if the client supports the feature and is RFC compliant, and we support the feature, then the combination is supported." However, it is a best practice to ensure that pNFS is supported by the client OS vendor.

ONTAP provides the ability to nondisruptively move volumes across nodes or aggregates in the same cluster to provide capacity and performance balance flexibility. When a volume move takes place in ONTAP, the pNFS device mappings are automatically updated to inform clients to use the new volume-to-interface relationship if necessary.

Learn about moving a volume

ONTAP provides the ability to move network interfaces across nodes in the same cluster to provide performance balance and maintenance flexibility. Like volume moves, when a network interface migration takes place in ONTAP, the pNFS device mappings are automatically updated to inform clients to use the new volume-to-interface relationship if necessary.

However, because NFSv4.1 is a stateful protocol, a network interface migration can be disruptive to clients that are actively using the NFS mount. It is a best practice to conduct network interface migrations in a maintenance window and notify clients of potential network disruptions.

pNFS follows the same storage failover considerations as NFSv4.1. These are covered in detail in NetApp Technical Report 4067: NFS Best Practice and Implementation Guide. In general, any storage failovers/givebacks involving pNFS should be done in a maintenance window, with potential storage disruptions expected due to the statefulness of the protocol.

Metadata operations are small in size and can be large in numbers depending on the workload (Are you creating a large number of files? Are you running "find" commands?) and total file count. As a result, workloads that are high in metadata calls can be taxing on the CPU of the NFS server and can potentially bottleneck over a single connection. pNFS (and NFSv4.x in general) is not suited for performance-dependent high metadata workloads, as the statefulness, the locking mechanisms, and some of the security features of the protocol version can negatively impact CPU utilization and latency. These workload types (such as high GETATTR or SETATTR) generally fare better with NFSv3.

The metadata server in pNFS is established at the initial mount of an NFS export. When the mount point is established, it remains in place until it is remounted or the data interface is moved. Because of this, it is a best practice to ensure that multiple clients accessing the same volume mount to different nodes and data interfaces across the SVM. This approach provides load balancing of the metadata servers across nodes and CPU resources while maximizing the network interfaces in the cluster. One way to accomplish this is to establish a round robin DNS setup, which is covered in NetApp Technical Report 4523: DNS Load Balancing in ONTAP.

NFSv4.x provides security functionality in many ways (covered in detail in NetApp Technical Report 4067: NFS Best Practice and Implementation Guide). NFSv4.x ID domains is one of those ways, where a client and server must agree on the ID domains when attempting to authenticate users and groups in an NFS export. One of the side effects of an ID domain mismatch would be the user or group showing up as an anonymized user (essentially squashed) to prevent unwanted access. With NFSv4.x (and also pNFS), it is a best practice to ensure the NFSv4.x ID domains match on client and server.

As mentioned previously, nconnect in ONTAP can help improve performance in some workloads. With pNFS, it is important to understand that while nconnect can improve performance by greatly increasing the total number of TCP connections to the storage system, it can also create issues when many clients are leveraging the mount option by overwhelming the TCP connections on the storage. The NetApp Hardware Universe covers the TCP connection limits per node.

When a node's TCP connection limits are exceeded, no new TCP connections are allowed until existing connections are freed. This can create complications in environments that might experience mount storms.

The following table shows how pNFS with nconnect might overwhelm TCP connection limits:

¹ Exceeds most ONTAP single node TCP connection limits

Session trunking in ONTAP can be used to increase throughput and path resiliency to NFSv4.x mounts. When used with pNFS, each node in a cluster can establish a session trunk. However, session trunks require at least two interfaces per node, and pNFS requires at least one interface per node to work as intended. Additionally, all interfaces in the SVM must be routable to the NFS clients. Session trunking and pNFS do not work properly when also leveraging nconnect. Consider nconnect and session trunking as mutually exclusive features.

Learn about NFS trunking

pNFS requires a routable network interface on each node in a cluster to function properly. If other network interfaces that are not routable to NFS clients exist in the same SVM as the NFS server hosting pNFS, ONTAP will still advertise those interfaces in the device mapping to clients. When the NFS client attempts to access data via the interfaces in a different subnet, they will not be able to connect, and it will create an outage. It is a best practice to only allow network interfaces in an SVM that can be accessed by clients when using pNFS.

NFSv4.0 is an option that can be enabled in an ONTAP NFS server alongside NFSv4.1. However, pNFS does not work over NFSv4.0. If NFSv4.0 is enabled in the NFS server, clients can potentially unknowingly mount that protocol version and will not be able to leverage pNFS. As a result, it is a best practice to explicitly disable NFSv4.0 when using pNFS. NFSv4.1 must still be enabled and can work independently of NFSv4.0.

NFSv4.1 referrals will localize the mount path from a client to the network interface on the node that owns a volume. pNFS localizes the data path, and the mount path becomes a metadata server.

While the two features can feasibly be used together, using NFSv4.1 referrals with pNFS might result in the undesired effect of stacking multiple metadata servers on the same node and reducing the ability to spread metadata servers across multiple cluster nodes. If metadata servers are not spread evenly across a cluster when using pNFS, then a single node's CPU can get overwhelmed with metadata requests and create a performance bottleneck.

As such, it is a best practice to avoid using NFSv4.1 referrals when using pNFS. Instead, spread the mount points across multiple network interfaces and nodes in the cluster.

Learn about enabling or disabling NFSv4 referrals

With NFS Kerberos, it is possible to encrypt authentication with krb5 and to further encrypt data packets with krb5i and krb5p. This is enabled on a per-network interface basis in a SVM and is covered in full detail in NetApp Technical Report 4616: NFS Kerberos in ONTAP with Microsoft Active Directory.

Because pNFS can redirect data traffic across nodes and network interfaces in the SVM, NFS Kerberos must be enabled and functional on each network interface in the SVM. If any network interface in the SVM is not enabled for Kerberos, then pNFS will not be able to function properly when attempting to access data volumes on those interfaces.

For example, when running a read test using parallel dd on a pNFS-enabled SVM with two network interfaces (only one enabled for Kerberos), the files located on the Kerberos-enabled interface performed well, while the files on the node with the interface without Kerberos enabled never were able to complete their reads. When Kerberos was enabled on both interfaces, all files were able to perform as expected.

NFS Kerberos can be used with pNFS provided NFS Kerberos is enabled on all network interfaces in the SVM. Keep in mind that NFS Kerberos can incur a performance penalty due to encryption/decryption of the packets, so it is a best practice to test pNFS with NFS Kerberos thoroughly with your workloads to ensure that any performance hit is not overly impactful to the workload.

Below is an example of parallel read performance when using krb5 (authentication) and krb5p (end to end encryption) with pNFS on a RHEL 9.5 client. Krb5p saw a 70% performance degradation in this test.

Learn about Kerberos with NFS for strong security

NFSv4.2 was added to ONTAP 9.8 and is the latest NFSv4.x version available (RFC-7862). NFSv4.2 does not have an explicit option to enable/disable it. Instead, it is enabled/disabled alongside NFSv4.1 (-4.1 enabled). If a client supports NFSv4.2, it will negotiate the highest supported version of NFS during the mount command if not specified otherwise with the minorversion=2 mount option.

NFSv4.2 in ONTAP supports the following functionality:

pNFS was introduced with NFSv4.1, but is also supported with NFSv4.2, as well as its accompanying features.

Learn about ONTAP support for NFSv4.2