Performance benchmark tips
Suggest changes
PDFs
This page provides configuration guidance for reproducing the NetApp AFX performance benchmark results described in Benchmark results. These are general considerations and configuration guidelines rather than a complete test procedure.
For NFS protocol background and recommended practices, see TR-4067: NFS in ONTAP best practice and implementation guide and NFSv4.x performance enhancements.
NAS benchmarking scripts for volume creation, LIF management, and client tuning are available in the Benchmarking/ONTAP repository.
Performance testing considerations
Performance testing is generally intended to find the edges of what a system can achieve in specific workload scenarios. Most of the time, the ONTAP default settings do not need to be adjusted to see good performance for most workloads, but in some cases, you may want to adjust some configurations to optimize the system.
NFS export policy configuration
Export policies in ONTAP are how access to NFS exports are controlled. Every volume requires an export policy to be set, with rules configured in the policy. If no rules are configured in a policy, that will be considered as "no access." For more information, see NFS securing and the ONTAP export policies documentation.
For performance benchmarking, we are likely less concerned with permissions to the clients being used for testing and instead would like to avoid any permission-related errors. As a result, it is recommended to ensure that any export policy rule is set to wide-open access for the data volume, but only to the clients being used to access datasets. In addition, the vsroot policy must allow read access in the policy for path traversal.
vsroot policy (uses "default" policy)
export-policy rule create -policyname default -clientmatch 0/0 -rorule any -rwrule never
data volume policy
export-policy create -policyname [name] -vserver [SVM] export-policy rule create -policyname [name] -clientmatch [client1],[client2],..[clientN] -rorule any -rwrule any -superuser any -anon 0 -chown-mode unrestricted -ntfs-unix-security-ops ignore
NFS server configuration
The following NFS server options were used for internal AFX performance benchmark runs. Other NFS server options can be enabled as well (for example, NFSv4 ACL support), but they are outside the scope of these benchmark tips. For NFSv4.1, pNFS, and session trunking performance context on AFX, see NFSv4.x performance enhancements.
| Option | Value |
|---|---|
v4.1 |
Enabled |
v4-id-domain |
[domain name]* |
v4.1-pnfs |
Enabled |
v4.1-trunking |
Enabled |
tcp-max-transfer-size |
262144 |
v3-64bit-identifiers |
Enabled |
v4-64bit-identifiers |
Enabled |
v3 |
Enabled |
v4.1-read-delegation |
Disabled |
v4.1-write-delegation |
Disabled |
rdma |
Enabled (optional) |
v4.0 |
Disabled |
v3-hide-snapshot |
Enabled |
mount-rootonly |
Disabled |
nfs-rootonly |
Disabled |
*The NFSv4 ID domain can be any value, provided it matches what is set on the NFS clients.
To verify the NFSv4 ID domain on the client:
# nfsidmap -d user-domain.company.com # cat /etc/idmapd.conf | grep Domain Domain = user-domain.company.com
|
If those commands do not work, you likely have not installed the proper NFS packages to leverage NFSv4.x. For example, nfs-common for Debian/Ubuntu-based systems. |
The following CLI command applies the NFS server configuration used for internal performance benchmark runs. A script that connects to the cluster via SSH and applies this configuration is available at ONTAP/NAS/configure-nfs-server.sh.
|
|
RDMA reduces latency and increases throughput for most workloads — roughly 10-30%. For NFS over RDMA, see the ONTAP NFS over RDMA documentation. |
set advanced; nfs modify -vserver [SVM] -v3 enabled -v4.1 enabled -v4.0 disabled -v4-id-domain [yourdomain.com] -v4.1-pnfs enabled -v4.1-trunking enabled -v4-64bit-identifiers enabled -v3-64bit-identifiers enabled -chown-mode unrestricted -tcp-max-xfer-size 262144 -mount-rootonly disabled -nfs-rootonly disabled -v3-hide-snapshot enabled -rdma [enabled optional]
Volume configuration
The following section covers volume creation and configuration considerations and shows what options should be used during the initial creation. For how FlexGroup volumes work in AFX, see FlexGroup volume management improvements, Volume types, and TR-4571: NetApp ONTAP FlexGroup volumes best practices and implementation guide.
| Setting | Value |
|---|---|
Volume type |
FlexGroup |
Creation method |
CLI |
set diag; vol create -vserver [SVM] -volume [name] -size [size] -junction-path [/path] -unix-permissions 777 -files-set-maximum true -maxdir-size 4G -is-large-size-enabled true -snapshot-policy none -autosize-mode grow_shrink -policy [data policy] -snapdir-access false
Defaults changed:
-
Snapshot policy disabled
-
Maxfiles increased
-
Maxdirsize increased
-
Large size enabled
-
Autosize enabled
-
UNIX permissions 777
-
File System Analytics disabled
-
Snapdir access false
Special considerations — FlexGroup volumes
If the workload creates many large files (>1GB) in a single folder, modify the volume's workflow to data:
AFX::> set advanced; vol modify -vserver [SVM] -volume [FlexGroup name] -workflow data
If the workload is VMware datastores hosted on a FlexGroup volume, disable the -gdd option (Granular Data Distribution). GDD is not supported with NFS copy offload for virtualization on AFX; see ONTAP features that have been removed or are unsupported with AFX.
AFX::> set diag; vol modify -vserver [SVM] -volume [FlexGroup name] -granular-data disabled
Why snapshots are disabled
In general, snapshots in ONTAP are not detrimental to overall system performance. For how snapshots and replication work on AFX, see Snapshots and Data Protection. However, in benchmarking, we would disable snapshots for two reasons:
-
Granular control over the environment — We can create snapshots when we want to create them to better control the scenarios when snapshots are created and make it easier to measure their impact.
-
Data churn — Many benchmark tests will create a lot of data and then subsequently delete that data, which will rapidly grow the size of any existing snapshots on the volume. To avoid space allocation issues, we disable snapshots.
|
|
Snapdir access is also disabled to prevent unwanted filesystem crawls across snapshots. |
Mount options
Mount options, in general, are variable depending on use cases. This section attempts to show a general use case mount option list that was used in our internal performance testing. If the NFS server is configured properly as shown in NFS server configuration, then you won't need to specify NFS versions or wsize/rsize options.
Base options (applies to all clients/scenarios):
-o tcp,hard,intr
Session trunking (OS-specific)
Session trunking is also listed in the Benchmark results configuration (trunkdiscovery). For NFSv4.x multipath IO context, see NFSv4.x performance enhancements.
RHEL: trunkdiscovery
Ubuntu (depends on version): trunkdiscovery (newer releases)
max_connect=4 (older releases)
RDMA specific options
For NFS over RDMA, see the ONTAP NFS over RDMA documentation.
rdma,write=eager
Other mount options
Some workloads might benefit from special mount options during benchmark tests. The following table shows some of those options and what use cases might see performance improvements.
Alternate NFS mount options and their use cases
| NFS mount option | Description | Use case |
|---|---|---|
|
Disables "close-to-open" cache consistency. The client will not revalidate cached file attributes when a file is opened, relying instead on cached data even if it may be stale. |
Read-heavy, single-client workloads where files rarely change (e.g., static web content, software libraries, read-only media archives). Not suitable for multi-client writes. |
|
Enables close-to-open cache consistency (default). When a file is opened, the client checks with the server to ensure its cached copy is still valid. Guarantees that a file closed on one client is seen as up-to-date when opened on another. |
Multi-client shared access workloads where data consistency matters (e.g., shared home directories, collaborative development environments). |
|
Sets all attribute cache timeouts (acregmin, acregmax, acdirmin, acdirmax) to a single value in seconds. Controls how long the client trusts cached file/directory metadata before revalidating with the server. |
Simplifying tuning when you want a uniform caching policy. High values benefit read-heavy/static workloads; low values (e.g., actimeo=0) suit rapidly changing data needing near-real-time consistency. |
|
Sets the minimum time (seconds) the client caches attributes for a regular file before revalidating. Default is typically 3s. |
Workloads with frequently updated small files where you want to fine-tune freshness vs. performance (e.g., log ingestion, config file polling). |
|
Sets the maximum time (seconds) the client caches attributes for a regular file. Default is typically 60s. |
Mostly-static file workloads where extending the max reduces metadata chatter (e.g., large read-only datasets, package repositories). |
|
Sets the minimum time (seconds) the client caches attributes for a directory before revalidating. Default is typically 30s. |
Workloads with frequent directory listing or creation (e.g., build systems, mail spools with Maildir format) where directory freshness matters. |
|
Sets the maximum time (seconds) the client caches attributes for a directory. Default is typically 60s. |
Stable directory structures that rarely change — raising this reduces GETATTR calls (e.g., large static directory trees, archival storage). |
|
Specifies the client IP address to advertise to the NFS server for callback communication (NFSv4/v4.1+). The server uses this address to send delegation recalls and other callbacks. |
Multi-homed clients or environments with complex networking (e.g., clients with multiple NICs, VPN/overlay networks, or NAT) where the auto-detected address may be incorrect. |
|
Creates multiple TCP connections (up to the specified count) to the NFS server for a single mount, allowing parallel I/O over separate connections. |
High-throughput, I/O-intensive workloads that are bottlenecked by a single TCP stream (e.g., large file transfers, HPC/scientific computing, video editing, database backups, big data analytics). Particularly beneficial on high-bandwidth networks (10 GbE+). |
|
Specifies what Kerberos security flavor to use with the mount. Requires Kerberos configuration to work properly. See TR-4616: NFS Kerberos in ONTAP with Active Directory for details. |
Securing NFS mounts; has a negative impact on performance. |
Client configuration
Client configuration used for internal benchmarking, including NIC and OS tuning scripts, is covered in the Benchmarking/Clients repository. The following contains some additional configuration considerations. This list is not considered exhaustive and will be modified over time to accommodate additional use cases/scenarios.
NFS readahead
Client NFS read-ahead (below) is a Linux mount-point setting. It is separate from ONTAP volume-level aggressive readahead (Cross File Sequential Read) on AFX.
NFS read-ahead predictively requests blocks from a file in advance of I/O requests by the application. It's designed to improve client sequential read throughput. Until recently, all modern Linux distributions set the read-ahead value to be equivalent of 15 times the mounted filesystem's rsize.
RHEL 8.3 and Ubuntu 18.04 introduced changes that might negatively impact client sequential read performance. Unlike earlier releases, these distributions set read-ahead to a default of 128 KiB regardless of the rsize mount option used. Upgrading from releases with the larger read-ahead value to releases with the 128-KiB default experienced decreases in sequential read performance. However, read-ahead values may be tuned upward both dynamically and persistently. For example, testing with SAS GRID found the 15,360 KiB read value optimal compared to 3,840 KiB, 960 KiB, and 128 KiB. Not enough tests have been run beyond 15,360 KiB to determine positive or negative impact.
NFS read-ahead is defined at the mount point for an NFS filesystem. To display the current value, grep the mount path in /proc/self/mountinfo and use the device number to view /sys/class/bdi/<device>/read_ahead_kb:
# grep /mountpath/ /proc/self/mountinfo | awk '{ print $3 }'
# cat /sys/class/bdi/<device>/read_ahead_kb
A convenience script for viewing or dynamically setting read-ahead is provided in Linux NFS read-ahead best practices for Azure NetApp Files.
Network configuration
The following section covers some general networking guidance for client and storage networking. These are based on the internal benchmark tests we have done. For AFX networking differences, see Networking and Hardware: Switches.
Storage network considerations
-
At least one data IP address per node, per SVM (more per node is better)
-
Maximum of 16 IP addresses per node, per SVM
-
All data interfaces routable to all clients
-
Bonded/teamed ports per node using LACP
-
MTU size of 9000 (end to end)
-
Priority flow control (PFC priority 3) enabled when using RDMA
-
DNS round robin/load balancing configured to include all data interface IP addresses
Client network considerations
-
Dual NICs bonded for RoCE (LACP)
-
RX/TX ring buffers set to 8192
-
PFC and DSCP configured per NIC (when using RDMA)
-
RoCE traffic class set using cma_roce_tos (when using RDMA)
-
sunrpc.rdma_slot_table_entries set to 200
Additional ONTAP feature considerations
The following covers specific ONTAP feature configuration modifications that are not covered in NFS export policy configuration, Volume configuration, Mount options, Client configuration, or Network configuration.
Disable storage efficiencies
In general, storage efficiencies are one of the selling points for a production workload, as they can generate large space savings across a variety of workloads. However, in performance benchmarking, they offer less value (as many simulated workloads are either uncompressible or are zero-byte workloads and always deduplicating). As a result, we recommend disabling all storage efficiencies while performing benchmark tests.
For how storage efficiency works on AFX, including global deduplication, see Deduplication domains, ONTAP storage efficiency technical reports, and Dynamic Storage Efficiency in ONTAP 9.19.1.
vol efficiency off -volume [name] -vserver [SVM] aggr efficiency modify -aggregate data* -cross-volume-background-dedupe false -cross-volume-inline-dedupe false aggr efficiency wise-tsse modify -aggregate data* -enable-workload-informed-tsse false
Aggressive readahead
Aggressive readahead can help improve read performance for specific workloads (such as sequentially named files). Feature configuration, caveats, and statistics are covered in Aggressive read-ahead. See also Cross File Sequential Read in AFX SVM administration.