AI Data Engine requirements
Suggest changes
PDFs
Before you deploy AI Data Engine (AIDE), review the networking, Kubernetes, sizing, and operating system requirements for your environment. For a single-node deployment, you provide a Linux VM and AI Data Engine provisions everything else automatically. For a multi-node deployment, you bring your own Kubernetes cluster.
Networking requirements
The following network connections must be open:
| Connection | Port | Protocol | Direction | Purpose |
|---|---|---|---|---|
Console agent to NetApp Console |
443 |
TCP |
Outbound |
HTTPS: connectivity to NetApp Console |
Console agent to ONTAP |
443 |
TCP |
Outbound |
HTTPS: ONTAP cluster discovery |
Console agent to AI Data Engine |
80, 443, 8080, 9000 |
TCP |
Bidirectional |
Communication between Console agent and AI Data Engine |
AI client to MCP server |
443 |
TCP |
Inbound |
HTTPS: AI client access to the AIDE cluster host name from your corporate network or VPN. AI clients connect to the host name on port 443. They don't connect to the internal service port. |
AI Data Engine to ONTAP (NFS) |
111, 2049 |
TCP/UDP |
To storage |
NFS data source access |
AI Data Engine to ONTAP (SMB) |
139, 445 |
TCP/UDP |
To storage |
SMB data source access |
Cluster nodes to NFS server (multi-node configuration volume) |
2049 |
TCP |
Outbound |
NFS access to the export that holds the AI Data Engine configuration. Applies to multi-node deployments only. If you use Kerberos, the admin host also mounts this export. |
AI Data Engine to Active Directory |
389, 636, 3268, 3269 |
TCP (UDP also for 389) |
Outbound |
LDAP (389), LDAPS (636), and Global Catalog (3268, 3269) for user authentication, SMB/CIFS scanning, and mapping NFS user and group IDs to Active Directory names. Port 389 uses both TCP and UDP; all other ports use TCP only. |
AI Data Engine to Kerberos KDC (Active Directory domain controller) |
88 |
TCP/UDP |
Outbound |
Kerberos authentication for SMB/CIFS data sources. Required only if you use Kerberos. Refer to Kerberos and Active Directory requirements. |
AI Data Engine to DNS server |
53 |
TCP/UDP |
Outbound |
Name resolution for file servers and the KDC when you use Kerberos |
AI Data Engine to NetApp services, container registry, and CDN |
443 |
TCP |
Outbound |
HTTPS: installer script downloads from a CDN, container image and Helm chart pulls from ECR, and component catalog downloads from S3 |
Outbound internet access
For an online installation, the following endpoints must be reachable. Where a row states a condition, the endpoint is needed only in that case.
| Endpoint | Purpose | Initiated by | Deployment type |
|---|---|---|---|
|
NetApp container registry (AIDE container images and Helm charts) |
Container runtime and installer |
Both |
|
NetApp container registry (dualstack/virtual private cloud (VPC) endpoint access) |
Container runtime and installer |
Both |
|
AWS ECR API (authentication and image manifest retrieval) |
Container runtime and installer |
Both |
|
AWS ECR registry credential refresh |
AI Data Engine |
Both |
|
AWS S3 (ECR container image layer storage) |
Container runtime |
Both |
|
AWS S3 (ECR container image layer storage) |
Container runtime |
Both |
|
AWS S3 (component version catalog) |
Installer and AI Data Engine |
Both |
|
AWS S3 (component version catalog) |
Installer and AI Data Engine |
Both |
|
AWS STS (temporary credential exchange for registry and S3 access) |
AI Data Engine |
Both |
|
AI Data Engine operator: version policy |
AI Data Engine |
Both |
|
NetApp Console registration |
Installer |
Both |
|
NetApp Console authentication |
AI Data Engine |
Both |
|
FSx and NetApp workloads discovery |
AI Data Engine |
Both |
|
Google Cloud NetApp Volumes discovery |
AI Data Engine |
Both |
|
The health monitor uses it to show the public IP address on the health dashboard. AI Data Engine works without it. |
AI Data Engine |
Both |
|
Fallback for obtaining the public IP address shown on the health dashboard. AI Data Engine works without it. |
AI Data Engine |
Both |
|
Fallback for obtaining the public IP address shown on the health dashboard. AI Data Engine works without it. |
AI Data Engine |
Both |
|
Fallback for obtaining the public IP address shown on the health dashboard. AI Data Engine works without it. |
AI Data Engine |
Both |
|
Proxy connectivity test, used only when you use an outbound proxy. A failed test doesn't block installation or runtime. This endpoint is separate from the Docker Hub endpoints that serve k3s system images. |
AI Data Engine |
Both |
|
Proxy connectivity test, used only when you use an outbound proxy. A failed test doesn't block installation or runtime. |
AI Data Engine |
Both |
|
AWS Cognito access, used for sending product usage metrics from AI Data Engine instances to NetApp. Also used for a proxy connectivity test. |
AI Data Engine |
Both |
|
Logs and metrics streaming |
AI Data Engine |
Both |
|
Required only if you use governed AI access (MCP). Microsoft Entra ID tenant discovery and signing-key retrieval. AIDE supports only Microsoft Entra ID in the Azure public (commercial) cloud. |
AI Data Engine |
Both |
|
CloudFront CDN for installer scripts |
Installer |
Both |
|
Helm download |
Installer |
Both |
|
kubectl download |
Installer |
Both |
|
jq download, and the k3s binary download on single-node deployments |
Installer |
Both |
|
GitHub download redirects |
Installer |
Both |
|
k3s runtime download |
Installer |
Single node only |
|
k3s SELinux policy package ( |
Installer |
Single node only (RHEL family) |
|
Docker Hub authentication for k3s system images |
Container runtime |
Single node only |
|
Docker Hub registry for k3s system images |
Container runtime |
Single node only |
|
Docker Hub image layer storage for k3s system images |
Container runtime |
Single node only |
The endpoints in this table are the same whether you connect directly to the internet or through a proxy.
If you use a proxy, allow each endpoint for HTTPS (CONNECT) on port 443 using the exact host names listed, without wildcards or a leading dot.
Allow d13jb5dapseb36.cloudfront.net, which hosts the installer download, by host name. Its IP addresses are shared with other sites and change often, so a rule based on IP addresses stops working.
For multi-node deployments, the cluster nodes make the container runtime connections. If the multi-node install command from NetApp Console specifies a different --registry value, allow that registry host name instead of the registry host names in the table.
See Learn about proxies for AI Data Engine. To deploy with a proxy, see Deploy AI Data Engine as a single-node deployment with a proxy or Deploy AI Data Engine as a multi-node deployment with a proxy.
|
Verify that your package managers have the appropriate proxy configuration if required in your environment. |
AI Data Engine also relies on NetApp Console for centralized authentication and console services. Endpoints such as https://api.bluexp.netapp.com and https://netapp-cloud-account.auth0.com are listed in the network access requirements for NetApp Console.
Verify that DNS resolution for these endpoints is working from the AI Data Engine host before you begin deployment.
Supported Kubernetes versions
AI Data Engine supports only the distributions and versions listed in this table.
| Deployment | Kubernetes distribution | Supported version | Provided by |
|---|---|---|---|
AIDE single-node |
k3s |
v1.35 |
The installer provisions k3s automatically. No action is required. |
AIDE multi-node |
RKE2 |
v1.34 to v1.36 (v1.36.x recommended) |
You provide the cluster, which can be shared or dedicated. |
AIDE multi-node |
OpenShift |
OpenShift 4.22.16 (Kubernetes v1.35.8) |
You provide the cluster, which can be shared or dedicated. |
AIDE multi-node |
Azure Kubernetes Service (AKS) |
v1.34 to v1.36 |
You provide the cluster, which can be shared or dedicated. |
AIDE single-node deployment requirements
You provide a Linux VM, and AI Data Engine provisions everything else automatically, including k3s. You don't need to set up or manage Kubernetes. For host preparation, disk space, and firewall requirements, see AIDE single-node deployment prerequisites.
VM sizing
These sizes are recommended baseline configurations optimized for a three to four day initial scan window, not hard limits. Refer to Flexible sizing guidelines for what drives these numbers and how to scale beyond them.
| Size | vCPU | RAM | Disk | Storage IOPS | Storage throughput | Network | Approximate files |
|---|---|---|---|---|---|---|---|
Small |
16 |
64 GB |
565 GB 1 |
8,000 |
1,000 MB/s |
1 GbE |
200 million |
Medium |
32 |
128 GB |
2,065 GB 1 |
12,000 |
1,500 MB/s |
1 GbE |
1 billion |
Large |
96 |
192 GB |
6,065 GB 1 |
16,000 |
2,000 MB/s |
10 GbE |
3 billion |
1 The disk value is the minimum free space required when the OS, k3s runtime, and AIDE data share a single disk. It includes 65 GB for the OS, k3s, kubelet, and logs, plus the AIDE data volume (500 GB for Small, 2,000 GB for Medium, and 6,000 GB for Large). If these components are on separate disks, see disk space in the single-node prerequisites.
|
|
NVMe SSD or other solid-state storage is recommended for all deployment sizes. Use block storage, not NFS, for the AIDE data volume on a single-node deployment. NFS-backed storage can cause underperformance. |
Flexible sizing guidelines
Small, Medium, and Large configurations are recommended baselines for a three to four day initial scan window, not hard limits enforced by the software.
- What drives the sizing recommendations
-
-
Total file and directory count: The total number of objects to catalog is the dominant driver of compute and storage requirements. Higher object counts require proportionally more memory, CPU, and storage.
-
Directory structure: Resource use varies with directory nesting depth and how files are distributed across shares and volumes. Deeply nested or highly fragmented structures can require more resources than the same file count in a flatter hierarchy.
-
Required scan throughput: Initial catalog completion speed is tied directly to CPU and memory allocation. Provisioning below the recommended size reduces throughput and extends the initial scan window.
-
Storage performance: High-performance storage is strongly recommended. Metadata indexing and event processing require sustained IOPS and throughput proportional to the deployment size.
-
- What happens if you undersize or outgrow the deployed size
-
Each configuration is tuned for a specific file count and scan rate. If your environment exceeds what the configuration supports, you can expect the following:
-
Extended initial catalog duration: Reduced compute lowers objects scanned per day. For example, a deployment targeting 3 billion objects but provisioned at a smaller size can take significantly longer than the three to four day target.
-
Reduced ongoing ingest throughput: New file events and incremental scan updates process more slowly under sustained load.
-
Stalled scanning: AI Data Engine components are configured for the size that you deploy. When the environment significantly exceeds that size, scanning can stall instead of only slowing down. To check for unhealthy workloads and resource constraints, refer to Monitor deployment health and investigate issues.
-
Storage capacity risk: Undersized storage relative to total object count can push the metadata index toward capacity thresholds. Contact NetApp Support to plan a storage expansion.
-
- Scale beyond the recommended size
-
AI Data Engine components are configured for the size that you deploy, so adding vCPU, memory, or storage to the host VM doesn't automatically increase capacity. If the number of files or the performance that you need exceeds the size that you deployed, contact NetApp Support to readjust the sizing.
A single-node deployment doesn't support high availability or horizontal scale-out. If your environment consistently exceeds 3 billion files, or you require high availability and fault tolerance, deploy an AIDE multi-node deployment instead.
Operating system
| OS | Supported versions |
|---|---|
Ubuntu |
22.04 LTS, 24.04 LTS (24.04 LTS recommended) |
Red Hat Enterprise Linux (RHEL) |
8.x, 9.x |
AIDE multi-node deployment requirements
You bring your own Kubernetes cluster, which can be shared with other workloads or dedicated to AI Data Engine. The installer deploys AI Data Engine onto your existing Kubernetes cluster and doesn't provision or manage it. For the cluster, storage, and installer host requirements, see AIDE multi-node deployment prerequisites.
|
|
NVMe or high-performance SSD storage is recommended for all multi-node deployments. |
Linux kernel and NFS nconnect support
Linux kernel 5.3 or later is recommended for the nconnect NFS mount option, which provides the throughput required by the shared AIDE configuration volume. The following distribution versions support nconnect and meet the kernel recommendation; RHEL and CentOS 8.3 use a Red Hat kernel backport.
| Distribution | Kernel version | nconnect support |
|---|---|---|
RHEL / CentOS 8.3+ |
4.18 (backport) |
Yes |
RHEL / CentOS 9.x |
5.14 |
Yes |
Ubuntu 20.04 |
5.4 |
Yes |
Ubuntu 22.04 |
5.15 |
Yes |
Node sizing
These sizes are recommended baseline cluster configurations optimized for a three to four day initial scan window, not hard capacity limits. Environments beyond 6 billion files require custom sizing with NetApp. Refer to Flexible sizing guidelines for details.
| Size | Nodes | vCPU per node | RAM per node | Disk per node | Storage IOPS per node | Storage throughput per node | Network per node | Approximate files |
|---|---|---|---|---|---|---|---|---|
Small |
3 |
32 |
128 GB |
~1.3 TB |
8,000 |
1,000 MB/s |
1 GbE |
1 billion |
Medium |
6 |
48 |
192 GB |
~2 TB |
12,000 |
1,500 MB/s |
10 GbE |
3 billion |
Large |
9 |
64 |
256 GB |
~2.7 TB |
16,000 |
2,000 MB/s |
10 GbE |
6 billion |
Flexible sizing guidelines
Small, Medium, and Large configurations are recommended baseline cluster configurations for a three to four day initial scan window, not hard capacity ceilings enforced by the software. Environments exceeding 6 billion files require custom sizing with NetApp.
- What drives the sizing recommendations
-
-
Total file and directory count: Total object count is the dominant driver of storage and compute requirements. Larger environments require additional cluster nodes to distribute processing load and maintain performance.
-
High availability replication: Multi-node deployments enable data replication across nodes by default for fault tolerance. Replication increases total storage and memory requirements compared to an equivalent single-node deployment.
-
Node count and workload distribution: Processing throughput and total catalog capacity scale with the number of nodes in the cluster. Additional nodes distribute both the indexing workload and storage capacity across the cluster.
-
Required scan throughput: Cluster-wide CPU and memory allocation governs the daily scan rate and how quickly the initial catalog completes.
-
Per-node storage performance: High-performance storage is strongly recommended on each node, and total cluster capacity scales proportionally with node count.
-
- What happens if you undersize or outgrow the deployed size
-
Each configuration is tuned for a specific file count and scan rate. If your environment exceeds what the configuration supports, you can expect the following:
-
Extended initial catalog duration: Fewer nodes or reduced per-node resources lower throughput, extending the initial catalog window proportionally.
-
Increased processing latency under peak load: Insufficient memory across the cluster can cause elevated processing latency during high-volume ingestion periods.
-
Stalled scanning: AI Data Engine components are configured for the size that you deploy. When the environment significantly exceeds that size, scanning can stall instead of only slowing down. To check for unhealthy workloads and resource constraints, refer to Monitor deployment health and investigate issues.
-
Storage capacity risk: Insufficient cluster storage for the target object count can push the metadata index toward capacity thresholds. Contact NetApp Support to plan a storage expansion.
-
- Scale beyond the recommended size
-
Horizontal scaling of an existing multi-node cluster isn't supported. Adding worker nodes or dedicated storage nodes doesn't automatically increase scan processing, event ingestion, or catalog capacity. If your environment exceeds the size that you deployed, or you need more scan processing or event ingestion capacity, contact NetApp Support or your NetApp representative to plan the change.
To scale beyond six billion files, contact your NetApp representative for node sizing guidance and custom scale.
Kerberos and Active Directory requirements
Review these requirements before you scan SMB/CIFS shares with Kerberos authentication or map NFS user and group IDs to Active Directory names. The time synchronization, DNS, and host package requirements apply to Kerberos only. AIDE is a Kerberos client: it authenticates to your existing Kerberos key distribution center (KDC) and doesn't act as a Kerberos server.
Time synchronization
Kerberos authentication fails when the clocks on the AIDE nodes and the KDC differ by more than the allowed clock skew. Active Directory allows a maximum difference of five minutes by default.
Verify the following:
-
Network Time Protocol (NTP) is active on all AIDE nodes.
-
The AIDE nodes use a time source that is synchronized with your domain controllers.
DNS
Meet the following DNS requirements:
-
Every AIDE node can resolve the file server and the KDC by fully qualified domain name (FQDN).
-
If you configure the nodes to use your Active Directory DNS server, that server must also resolve external names. Otherwise, AIDE can't reach the endpoints listed in Outbound internet access.
-
If you change the DNS settings on a multi-node deployment, configure your Active Directory DNS server as the only name server on every node. Name resolution in the cluster picks one of the listed name servers at random, so a name server that can't resolve Active Directory names causes intermittent failures.
To change the DNS settings, refer to "Fix name resolution" in Configure Kerberos for SMB/CIFS shares on a single-node deployment or Configure Kerberos for SMB/CIFS shares on a multi-node deployment.
Host packages
| Deployment | Required packages |
|---|---|
Single node |
Kerberos client tools: |
Multi-node |
|
cifs-utils is required on every AIDE node only for Kerberos-based SMB/CIFS scanning. The single-node installer installs it. On a multi-node deployment, you install it on every node. Kerberos scanning of an SMB/CIFS share fails if cifs-utils is missing on a node.
Active Directory attributes for NFS
To resolve NFS user and group IDs to Active Directory names, the uidNumber and gidNumber attributes must be populated, and replicated to the Global Catalog if AIDE queries it. Refer to Configure Active Directory identity mapping for NFS.
Active version and support contact
To find the active version of AI Data Engine and the NetApp Support contact information, go to Configuration > About AIDE.