Skip to main content
AI Data Engine

AI Data Engine architecture

Contributors netapp-dbagwell
PDFs

AI Data Engine (AIDE) runs on infrastructure that you provide. The Metadata Engine scans your storage and builds a searchable catalog. NetApp Console manages your deployment through a Console agent. A Model Context Protocol (MCP) server option lets AI clients search the catalog within boundaries that you set.

Components

The AI Data Engine dashboard, investigation, and policies in NetApp Console all work from the metadata catalog. The following components build the catalog, connect it to NetApp Console, and provide governed access for AI clients.

Metadata Engine

The Metadata Engine is the core capability of AI Data Engine. It scans the data sources that you connect, extracts metadata, and stores it in a searchable catalog. It doesn't read or classify file contents. The Metadata Engine updates the catalog at each scan cycle, detecting additions, changes, and deletions across your storage systems.

NetApp Console agent

AI Data Engine requires a NetApp Console agent deployed in your environment. The Console agent is a general-purpose NetApp platform component that maintains connectivity between your environment and NetApp Console. It handles authentication and communication between NetApp Console and your AI Data Engine deployment. The scanning and catalog services run in your AI Data Engine deployment, not on the Console agent.

Note The Console agent isn't specific to AI Data Engine. If you already have a Console agent deployed for other NetApp services, you can use the same agent. For Console agent deployment instructions, refer to the NetApp Console agent documentation.

Governed Access Gateway

The Governed Access Gateway is the MCP server for AI Data Engine. It lets AI clients search the metadata catalog. The gateway runs in your AI Data Engine deployment, and AI clients connect to it over HTTPS from your corporate network or through VPN. The Console agent doesn't relay MCP traffic.

Microsoft Entra ID authenticates users, and the gateway limits each AI client to the collections that the user is authorized to search. The gateway supports metadata search and tagging only, and only power users can tag files. It doesn't retrieve file contents.

Learn more about governed AI access.

Deployment types

AI Data Engine supports two deployment types. Both run AI Data Engine services on Kubernetes.

AIDE single-node deployment

Runs on a single Linux VM that you provide. The installer provisions k3s and the supporting components automatically. A single-node deployment doesn't provide high availability or horizontal scaling.

AIDE multi-node deployment

Runs on a Kubernetes cluster that you provide and is intended for larger environments that require high availability.

For supported platforms, sizing, and infrastructure requirements, refer to AIDE requirements.

Each deployment type also has a separate procedure for environments that reach the internet only through an outbound proxy. Refer to Learn about proxies for AI Data Engine.

Learn more about deployment options.

Data flow

AI Data Engine uses two separate paths: one that builds the catalog and serves it to NetApp Console, and one that serves AI clients.

Metadata scanning and NetApp Console access

  1. You deploy AI Data Engine on infrastructure that you manage and connect it to a NetApp Console agent.

  2. You connect the data sources that you want AI Data Engine to catalog in Data sources. For more information, refer to Connect a data source.

  3. The Metadata Engine scans the data sources to extract metadata and indexes it in the catalog.

  4. In NetApp Console, the dashboard surfaces storage insights from the catalog, and Investigation lets you query the catalog directly with metadata filters.

  5. You can save search criteria as policies and re-run them to identify matching files, and you can act on files from Investigation.

Note The catalog reflects metadata collected during scanning. Changes to source files appear in the catalog after the affected files and directories are rescanned and the results are indexed.

Governed AI access

  1. You configure Microsoft Entra ID, enable collections for AI access, and choose which users can search each collection.

  2. A user connects an AI client to the HTTPS endpoint of the Governed Access Gateway and signs in with Microsoft Entra ID.

  3. The gateway verifies the user's access and passes authorized requests to the Metadata Engine.

  4. The AI client searches the metadata in the collections that the user can access and take actions on.