With the proven computing performance of NVIDIA DGX SuperPOD combined with NetApp’s industry-leading data security, data governance and multi-tenancy capabilities, customers can deploy the most efficient and agile infrastructure for next-generation workloads. This document describes the high-level architecture and key features that help customers deliver faster time-to-market and return on investment for AI/ML initiatives.

NVIDIA DGX SuperPOD offers a turnkey AI data center solution for
organizations, seamlessly delivering world-class computing, software tools, expertise, and continuous innovation. DGX SuperPOD delivers everything customers need to deploy AI/ML and HPC workloads with minimal setup time and maximum productivity. Figure 1 shows the high-level components of DGX SuperPOD.

Figure 1) NVIDIA DGX SuperPOD with NetApp AFF A90 storage systems.

DGX SuperPOD provides the following benefits:

As organizations embrace artificial intelligence (AI) and machine learning (ML) initiatives, the demand for robust, scalable, and efficient infrastructure solutions has never been greater. At the heart of these initiatives lies the challenge of managing and training increasingly complex AI models while ensuring data security, accessibility, and resource optimization. The evolution of agentic AI and sophisticated model training requirements has created unprecedented demands on computational and storage infrastructure. Organizations must
now handle massive datasets, support multiple concurrent training workloads, and maintain high-performance computing environments while ensuring data protection and regulatory compliance. Traditional infrastructure solutions often struggle to meet these demands, leading to operational inefficiencies and delayed time-to-value for AI projects. This solution offers the following key benefits:

DGX SuperPOD is based on the concept of a Scalable Unit (SU) that includes 32 DGX B200 systems and all of the other components necessary to deliver the required connectivity and eliminate any performance bottlenecks in the infrastructure. Customers can start with a single or multiple SUs and add additional SUs as needed to meet their requirements. This document describes the storage configuration for a single SU, and Table 1 shows the components needed for larger configurations.

The DGX SuperPOD reference architecture includes multiple networks, and the AFF A90 storage system is connected to several of them. For more information about DGX SuperPOD networking please refer to the
NVIDIA DGX SuperPOD Reference Architecture.

For this solution the high-performance storage fabric is an Ethernet network based on the NVIDIA Spectrum SN5600 switch with 64 800Gb ports in a Spine/Leaf configuration. The In-band network provides user access for other functions such as home directories and general file shares and is also based on SN5600 switches, and the out of band (OOB) network is for device-level system administrator access using SN2201 switches.

The storage fabric is a leaf-spine architecture where the DGX systems connect to one pair of leaf switches and the storage system connects to another pair of leaf switches. Multiple 800Gb ports are used to connect each leaf switch to a pair of spine switches, creating multiple high-bandwidth paths through the network for aggregate performance and redundancy. For connectivity to the AFF A90 storage system, each 800Gb
port is broken into four 200Gb ports using the appropriate copper or optical breakout cables. To support clients mounting the storage system with NFS over RDMA the storage fabric is configured for RDMA over Converged Ethernet (RoCE), which guarantees lossless packet delivery in the network. Figure 3 shows the storage network topology of this solution.

Figure 3) Storage fabric topology.

The NetApp AFF A90 storage system is a 4RU chassis containing 2
controllers that operate as high availability partners (HA Pair) for each other, with up to 48 2.5 inch form-factor solid state disks (SSD). Each controller is connected to both the SN5600 storage leaf switches using four 200Gb Ethernet connections, and there are 2 logical IP interfaces on each physical port. The storage cluster supports NFS v4.1
with Parallel NFS (pNFS) that enables clients to establish connections directly to every controller in the cluster. Additionally, session trunking combines the performance from multiple physical interfaces into a single session, enabling even single-threaded workloads to access more network bandwidth than is possible with traditional ethernet bonding.Combining all of these features with RDMA enables the AFF A90 storage system to deliver low latency and high throughput that scales
linearly for workloads leveraging NVIDIA GPUDirect Storage™.

For connectivity to the in-band network the AFF A90 controllers have additional 200Gb Ethernet interfaces configured in a LACP interface group providing general NFS v3 and v4 services as well as S3 access to shared filesystems if desired. All of the controllers and storage cluster switches are connected to the OOB network for remote administrative access.

To enable high performance and scalability, the storage controllers form a storage cluster that enables the entire performance and capacity of the cluster nodes to be combined into a single namespace called a FlexGroup with data distributed across the disks of every node in the
cluster. With the new Granular Data Distribution feature released in ONTAP 9.16.1, individual files are separated and distributed across the FlexGroup to enable the highest levels of performance for single-file workloads. Figure 4 below shows how pNFS and NFS session trunking work together with FlexGroups and GDD to enable parallel access to large files leveraging every network interface and disk in the storage system.

Figure 4) pNFS, session trunking, FlexGroups and GDD.

This solution leverages multiple Storage Virtual Machines (SVM) to host volumes for both high-performance storage access as well as user home directories and other cluster artifacts on a management SVM. Each SVM is configured with network interfaces and FlexGroup volumes and QoS policy is implemented to ensure performance for the Data SVM. For more information on FlexGroups, Storage Virtual Machines and ONTAP QoS capabilities please refer to the ONTAP documentation.

This storage solution was validated in multiple stages by NetApp and NVIDIA to ensure that performance and scalability meet the requirements for NVIDIA DGX SuperPOD. The configuration was validated using a combination of synthetic workloads and real-world ML/DL workloads to verify both maximum performance and application interoperability. Table 3 below provides examples of typical workloads and their data requirements that are commonly seen in DGX SuperPOD deployments.

Table 3) SuperPOD workload examples.

Table 4 shows performance guidelines for the example workloads above. These values represent the storage throughput that can be generated by these workloads under ideal conditions.

Table 4) DGX SuperPOD performance guidelines.

The NVIDIA DGX SuperPOD with NetApp AFF A90 storage systems represents a significant advancement in AI infrastructure solutions. By addressing key challenges around security, data management, resource utilization, and scalability, it enables organizations to accelerate their AI initiatives while maintaining operational efficiency, data protection, and collaboration. The solution's integrated approach eliminates common bottlenecks in AI development pipelines, enabling data scientists and engineers to focus on innovation rather than infrastructure management.

To learn more about the information that is described in this document, review the following documents and/or websites: