Skip to main content
AI Data Engine

What's new in AI Data Engine

Contributors netapp-dbagwell RSS
PDFs

AI Data Engine (AIDE) 9.18.1 is the initial release of NetApp's platform for AI data management. This release introduces a metadata engine and management workflows that enable organizations to catalog and organize unstructured data for AI workloads, providing the foundation for advanced governance and vectorization capabilities. Advanced governance (guardrails) and vectorization are available for customers who have the appropriate AI Data Engine licenses.

What's new in the AIDE 9.18.1 initial release

AIDE 9.18.1 introduces the following foundational capabilities:

Metadata Engine for AI data cataloging

The initial release includes a metadata engine that catalogs files and objects across ONTAP clusters.

Key features include:

  • Automated extraction of metadata (core and extended attributes, object tags) from local and remote ONTAP volumes on peered clusters.

  • Centralized querying and filtering REST APIs for applications requiring a global view of enterprise data.

  • Scalable metadata storage.

  • Automatic metadata extraction triggered during workspace creation.

Workspace management

Workspaces provide logical grouping of data sources (volumes) for AI projects.

The initial release supports:

  • Creation of workspaces spanning local and remote ONTAP volumes (using cluster peering).

  • Assignment of access controls to workspaces, supporting multi-user and multi-tenant environments.

  • Automatic metadata extraction and catalog population upon workspace creation.

Data Sync for automated data currency

Data Sync keeps metadata catalogs and data collections current as source data changes, without manual intervention.

Key features include:

  • Automated synchronization of data from remote or local ONTAP clusters using policy-driven SnapMirror replication.

  • Incremental updates that propagate only modified data, reducing overhead.

  • Configurable refresh intervals per workspace.

  • Workspace-level monitoring of sync status and activity.

Cluster setup and management

The initial release includes the following workflows:

  • Discovery and addition of data compute nodes (DCNs) during cluster setup.

  • Creation of dedicated metadata storage VMs for the metadata engine.

  • Configuration of Data Engine service interfaces for cluster-wide metadata access.

  • Peering with other ONTAP clusters to extend metadata cataloging across the data estate.

OpenID Connect (OIDC) authentication

  • OIDC/OAuth-based authentication for secure access to ONTAP System Manager and Data Engine Console with Microsoft Entra ID and Active Directory Federation Services (ADFS).

  • Role-based access controls for workspace and metadata management.

Advanced data curation and governance capabilities

The following capabilities are available for customers who have the appropriate AI Data Engine licenses:

  • Vectorization and RAG: Creation of data collections, embeddings, and retrieval endpoints in the AI Data Engine Console, using metadata from AIDE workspaces.

  • Guardrail-based governance: Definition of guardrail policies in the AI Data Engine Console and association of those policies with workspaces in ONTAP System Manager.

Supported hardware and platforms

AI Data Engine 9.18.1 runs on ONTAP AI data platform clusters that combine:

  • AFX 1K storage nodes

  • NetApp data compute nodes

Related information