Skip to main content
AI Data Engine

Learn about AI Data Engine deployment

Contributors netapp-dbagwell

Deploy AI Data Engine (AIDE) as an AIDE single-node deployment, where the installer sets up everything on a single Linux VM that you provide. If your environment requires Kubernetes-based high availability at large scale, you can deploy an AIDE multi-node deployment on a Kubernetes cluster that you bring.

AIDE single-node deployment

An AIDE single-node deployment installs on a single Linux VM that you provide. The installer provisions everything else automatically, including k3s, the Kubernetes runtime, and the supporting tools and host configuration that AI Data Engine requires. You don't need to install or manage Kubernetes, and you don't need Kubernetes expertise.

An AIDE single-node deployment is available in three sizes — Small, Medium, and Large — for environments up to approximately 200 million, 1 billion, and 3 billion files respectively. For full VM sizing and OS requirements, refer to AIDE requirements. These sizes are recommended baselines, not hard limits; see flexible sizing guidelines for what drives sizing and what to do if you outgrow a size.

AIDE multi-node deployment

Deploy an AIDE multi-node deployment if your environment requires high availability, fault tolerance, rolling upgrades, or capacity beyond the largest single-node deployment size. An AIDE multi-node deployment deploys onto an existing Kubernetes cluster. You must bring your own Kubernetes cluster (RKE2, OpenShift, or AKS). You provision and manage the cluster yourself, and it can be shared with other workloads or dedicated to AI Data Engine. The installer doesn't create or manage the cluster. The machine that you run the installer from is a lightweight management host and doesn't run AI Data Engine workloads.

An AIDE multi-node deployment is available in three sizes — Small, Medium, and Large — for environments up to approximately 1 billion, 3 billion, and 6 billion files respectively. For full node sizing and cluster requirements, refer to AIDE requirements. These sizes are recommended baselines, not hard limits: environments exceeding 6 billion files require custom sizing with NetApp. See flexible sizing guidelines for details.

Compare deployment types

AIDE single-node deployment AIDE multi-node deployment

Deployment type

Single Linux VM that you provide. The installer provisions k3s automatically.

Multi-node Kubernetes cluster. You bring your own RKE2, OpenShift, or AKS cluster.

Nodes

One Linux VM

Three to nine nodes (Small to Large)

Scan rate

50 million to 1 billion files per day, varies by size

250 million to 2 billion files per day, varies by size

Approximate files (Small to Large)

200 million to 3 billion

1 billion to 6 billion

Compute

16 to 96 vCPU, 64 to 192 GB RAM

32 to 64 vCPU, 128 to 256 GB RAM per node

Storage

565 GB to 6,065 GB SSD block storage, 8,000 to 16,000 IOPS, 1,000 to 2,000 MB/s throughput

4 to 24 TB total cluster capacity, 8,000 to 16,000 IOPS per node, 1,000 to 2,000 MB/s throughput per node

Network

1 GbE (10 GbE recommended for Large)

1 to 10 GbE

High availability

Not supported

Supported across nodes

When to choose it

You have a single site, want a fast start, and don't require high availability

Production scale or HA is a requirement

Deployment process

You can deploy AI Data Engine directly from NetApp Console. Go to AI data hub > AI Data Engine and deploy AI Data Engine. Use the wizard to choose a deployment type, and then run the installation command in an environment that meets the AIDE requirements.

NetApp Console doesn't provision the infrastructure for AI Data Engine. For a single-node deployment, you prepare a Linux VM, and the installer sets up the rest. For a multi-node deployment, you prepare a Kubernetes cluster. In both cases, the infrastructure must meet the requirements. Then you copy the pre-generated installation command from NetApp Console and run it on the target host. The command bootstraps the runtime and deploys the AI Data Engine stack automatically.

Single-node deployments typically complete within 45 minutes. Multi-node deployments typically complete within 30 minutes. The installer includes built-in timeouts and retries to handle transient issues while components start.

Deployment with a proxy

If your host or cluster reaches the internet only through an outbound proxy, deploy AI Data Engine with a proxy. The proxy steps differ for each deployment type, so each type has its own complete topic that replaces the standard deployment topic.

For the complete steps, see Learn about proxies for AI Data Engine for the supported proxy types, and Outbound internet access for the endpoints to allow on your proxy.