Skip to main content
AI Data Engine

Connect a data source to AI Data Engine

Contributors netapp-dbagwell
PDFs

Connect the data sources that you want AI Data Engine (AIDE) to analyze, and choose which of their data resources to scan. AI Data Engine scans connected data sources to collect metadata, and the connection procedure depends on the source type. You complete these tasks in NetApp Console.

Before you begin

  • You must have the AI Data Engine administrator role in NetApp Console to connect and configure data sources.

  • For systems connected through NetApp Console discovery, use a Console agent to discover the storage systems in NetApp Console before you add them to AI Data Engine. This applies to all supported storage types, including on-premises ONTAP clusters, Cloud Volumes ONTAP, Amazon FSx for NetApp ONTAP, Google Cloud NetApp Volumes, and Azure NetApp Files. For on-premises ONTAP clusters, refer to Discover on-premises ONTAP clusters in the NetApp Console documentation. Direct discovery doesn't enable data services such as AI Data Engine.

  • Verify that the network connections required for AI Data Engine to reach your data sources are open. Refer to AI Data Engine requirements.

  • To scan SMB/CIFS shares with Kerberos authentication, complete the procedure for your deployment type in Kerberos authentication for SMB/CIFS shares in AI Data Engine first.

Add a data source

Bring your data into AI Data Engine so that you can analyze it. Use Add system on the Data sources page to connect a file shares group, a database server, or a StorageGRID system. After you add the data source, configure scanning to specify which of its resources AI Data Engine scans to collect metadata. Refer to Configure scanning.

Add a file shares group

Bring NFS or SMB shares into AI Data Engine as one named system so that you can manage them together. Create the group, add the shares, and then configure which shares to scan.

Before you begin

  • Identify a name for the group and the host and path of each share.

  • Verify that the AI Data Engine deployment can reach the share servers.

  • For NFS, configure the export access that AI Data Engine needs.

  • For SMB, prepare the authentication method and the access needed to read the shares.

Steps

  1. From the AI Data Engine UI, select Data sources.

  2. Select Add system, and then select Add File Shares Group.

  3. Enter a name for the group and continue.

  4. Select the protocol of the shares to add.

  5. Enter the shares to add and submit them.

    Note For SMB/CIFS shares that use Kerberos authentication, enter the share path as the FQDN followed by the share name, for example filer.corp.example.com/sharename, and leave the credentials empty. Refer to Kerberos authentication for SMB/CIFS shares in AI Data Engine.

  6. Review the result for each share. Resolve any failed shares, and then add them again.

  7. Configure scanning for the group. Refer to Configure scanning.

Add a database server

Bring the data in a database server into AI Data Engine so that you can scan it. Connect the database server, and then choose the schemas to scan.

Before you begin

  • Know the database type, server host name or IP address, port, and scan account credentials.

  • Verify that the account can read the schemas and tables that you want to scan.

  • Verify that AI Data Engine can reach the database server.

Steps

  1. From the AI Data Engine UI, select Data sources.

  2. Select Add system, and then select Add Database Server.

  3. Select the database type and enter the connection details.

  4. Submit the connection details.

  5. Configure scanning for the database server. Refer to Configure scanning.

Add StorageGRID

Bring your StorageGRID object data into AI Data Engine through its S3 endpoint so that you can scan it. Connect StorageGRID, and then choose the buckets to scan.

Before you begin

  • Prepare a system name, the StorageGRID S3 endpoint URL, and an S3 access key and secret key.

  • Verify that AI Data Engine can reach the endpoint.

Steps

  1. From the AI Data Engine UI, select Data sources.

  2. Select Add system, and then select Add StorageGRID.

  3. Enter the system name, S3 endpoint URL, access key, and secret key.

  4. Submit the connection details.

  5. Configure scanning for the StorageGRID system. Refer to Configure scanning.

Configure scanning

After you connect a system, choose which of its data resources AI Data Engine scans so that metadata collection covers the data that you need to analyze. Choose one of the following approaches:

  • Enable auto-scan for the entire system on the Data sources page. AI Data Engine scans all existing and future resources in the system.

  • Enable scanning for individual volumes, shares, buckets, or schemas on the system's View details page.

Steps

  • Enable auto-scan for a system:

    1. From the AI Data Engine UI, select Data sources.

    2. Locate the system and toggle the auto-scan switch to enable auto-scan.

      Note You must turn off auto-scan to manage scanning for individual resources.

  • Enable scanning for individual resources:

    1. From the AI Data Engine UI, select Data sources.

    2. Locate the system and select View details.

    3. In the data resources list, enable scanning for the volumes, shares, buckets, or schemas that you want to include.

    4. Return to Data sources, locate the system, and select Activate scans.

For more resource-level procedures, refer to Manage data sources.

Result

The system status on the Data sources page updates to show that scanning is running. The page shows the scan status of each resource.

If scans stop or pause unexpectedly, refer to Monitor deployment health and investigate issues.