Skip to main content
AI Data Engine

Take action on your data in AI Data Engine

Contributors netapp-dbagwell
PDFs

In AI Data Engine (AIDE), you can enforce your organization's data lifecycle policies by archiving, copying, tagging, or deleting the files that you identified with your filters. For example, a storage administrator who filters to find terabytes of VM snapshots not accessed in over a year can archive them in bulk to a cold storage share or delete them, depending on retention requirements and business policies.

About this task

To take action, select one or more items in the Investigation results table using the checkboxes. The action toolbar activates with four options: Delete, Copy, Archive, and Tag. To remove duplicate copies of a file, use Clean up duplicates. Refer to Investigate and clean up duplicate files.

The actions available to you depend on your assigned role:

Action AI Data Engine administrator AI Data Engine viewer

Delete

Yes

-

Copy

Yes

-

Archive

Yes

-

Tag

Yes

Yes

Clean up duplicates

Yes

-

For a full description of roles, refer to Manage users and roles.

Decide what to do

Choose an action based on how certain you are about the data:

Action When to use it

Delete

The data is no longer needed and has no retention requirements. Deletion is permanent, so use it only when you are sure.

Copy

The data should exist in a second location, for example for backup or migration purposes. Copying preserves the original in place. You can copy to an NFS or SMB folder, create a filtered FlexClone volume, or transfer items with NetApp Copy and Sync.

Archive

The data might still have value, but it doesn't belong in primary storage. Archiving moves it to a lower-cost destination while keeping it recoverable.

Tag

You need more time, or need to coordinate with another team, before committing to archive or delete. Tagging tracks the files without changing their location.

Clean up duplicates

Identical copies of a file exist in more than one location and you want to keep only one. You can tag, archive, or delete the redundant copies while retaining one copy. Deleting duplicates is permanent. Refer to Investigate and clean up duplicate files.

When data might still have value or require further review, choose tagging, archiving, or copying over deleting. Only delete when you are certain the data is safe to remove permanently.

Delete files

Permanently remove files that you no longer need and that have no retention requirements.

Caution Deletion is permanent and can't be undone. This action deletes source files even if they contain active data.
Before you begin

  • You must have the AI Data Engine administrator role.

Steps

  1. In the Investigation view, select the files you want to delete.

  2. Select Delete.

  3. In the Delete dialog, type permanently delete in the confirmation field.

  4. Select Delete.

Result

The selected files are permanently deleted from storage.

Investigate and clean up duplicate files

Find duplicate copies of a file and remove the redundant ones. Review the copy metadata and locations, and tag duplicates for review, archive them, or permanently delete duplicate copies while retaining one copy.

Before you begin

  • You must have the AI Data Engine administrator role.

Steps

  1. Select Investigation > Unstructured.

    The table displays file names, sizes, owners, systems, and tags.

  2. Optional: Select Add filter to narrow the list of files.

  3. Select a file name to open its details.

    Review the file's system, storage repository, path, size, timestamps, access, and owner. The Duplicates field displays the number of duplicate copies.

  4. Next to Duplicates, select View details.

    The Investigation view applies a Duplicates filter that includes files with duplicates and a File hash filter that matches the selected file.

  5. Review the matching files and their locations.

  6. Select the files you want to clean up.

    Review the Selected items count.

  7. Select Clean up duplicates.

  8. Review the estimated number of files matching the current filters that have duplicate copies.

    Note The count is an estimate. Exact duplicates are recalculated when the action runs, and only files confirmed identical are affected.

  9. Decide how to handle the duplicates:

    • Tag the duplicates for review:

      1. Under Action, select Tag.

      2. Select Add, and then choose the tag to apply. Repeat to add more tags.

      3. Select Submit.

    • Archive the duplicates to a destination volume:

      1. Under Action, select Archive.

      2. Under At the original location, select Remove cleanly or Leave a breadcrumb.

      3. Select the destination volume protocol. For NFS, enter the destination folder path in the format Hostname:/SHAREPATH. For SMB, enter the destination folder path, and then select Kerberos or NTLM authentication. For NTLM, enter your Active Directory user name and password.

      4. Select Submit.

    • Delete the duplicates and keep one copy:

      Caution Deletion is permanent and can't be undone.

      1. Under Action, select Delete.

      2. Under Keep copy with the, select Last accessed, Last modified, or Created.

      3. Under At the original location, select Remove cleanly or Leave a breadcrumb.

      4. Select Submit.

      5. In the Delete the duplicate copies dialog, review the number of files to be deleted and the information about retaining one copy.

      6. Enter permanently delete, and then select Delete.

To exit without submitting an action, select Cancel.

Copy selected items

Make selected items available in another location, for example, for backup or migration. You can copy items to an NFS or SMB folder, create a filtered copy of an ONTAP volume, or transfer items using Copy and Sync.

Choose the method that matches your goal:

  • Regular copy: Copy items to an NFS or SMB destination folder without changing the original files.

  • FlexClone: Create a point-in-time copy of an ONTAP volume and remove items that were not included in your query from the cloned volume. The original volume is not affected.

  • Sync: Transfer items between file or object storage, on premises or in the cloud, using Copy and Sync.

Before you begin

  • You must have the AI Data Engine administrator role.

Steps

  1. In AI Data Engine, select Investigation from the left navigation.

  2. Use filters to find the items you want to copy, and then select the items in the results table.

  3. Select Copy.

  4. Complete the procedure for your chosen copy method.

Copy items to an NFS or SMB folder

Use regular copy to place copies of selected items in a destination folder while leaving the original files unchanged.

Caution Copying or archiving files can change destination permissions, ownership, group information, and timestamps. Cross-protocol transfers can substantially broaden access. Review the destination security settings before transferring sensitive data, and verify permissions afterward.
Before you begin

  • Verify that the destination share allows access from AI Data Engine. NFS and SMB shares are supported regardless of where they are hosted.

  • For an SMB destination, have Kerberos or NTLM credentials available.

Steps

  1. In the Copy selected items dialog, select Regular copy.

  2. Choose the destination protocol and provide the destination details:

    • For NFS, enter the destination folder path in the format hostname:/path/to/destination.

    • For SMB, enter the destination folder path in the format \\hostname\share_name. Select Kerberos or NTLM as the SMB authentication type. For Kerberos, use the FQDN of the host and leave the credentials empty. Refer to Kerberos authentication for SMB/CIFS shares in AI Data Engine. For other methods, provide Active Directory credentials: enter the username in domain\user or user@domain format, and enter the password.

  3. Review how copying might affect file metadata:

    • Permissions, ownership, and group information might be preserved when possible. If source identities cannot be mapped, destination defaults might apply.

    • Source timestamps, including creation time, might not be preserved and might reflect the copy operation or destination behavior.

    • New directories use the destination's default permissions.

    • Cross-protocol transfers can broaden access. Review the destination's security settings before copying and verify permissions afterward.

  4. Select Submit.

Create a filtered copy of an ONTAP volume

Create a cleaned-up copy of an ONTAP volume that contains only the items in your query, ready to migrate. FlexClone creates a point-in-time copy of the volume and deletes the items outside your query from the clone only.

Before you begin

Confirm that all selected items belong to the same ONTAP volume. The FlexClone volume is created in the same aggregate as its parent volume.

Steps

  1. In the Copy selected items dialog, select FlexClone.

  2. Enter a Name for the cloned volume.

  3. Confirm that your query includes all the items you want to retain in the cloned volume.

    Items not included in the query are deleted from the clone. The original volume is not affected.

  4. Select Submit.

Transfer items using Copy and Sync

Transfer selected items between file or object storage locations, on premises or in the cloud.

Before you begin

Confirm that all selected items belong to the same storage repository. If your selection spans repositories, cancel the operation and select items from a single repository.

Steps

  1. In the Copy selected items dialog, select Sync.

  2. Review the number of items to be synced and confirm that it matches the items you intend to transfer.

  3. Select Sync to proceed with Copy and Sync.

Archive files

Move files that don't belong in primary storage to a cold storage destination while keeping them recoverable. You can optionally leave a breadcrumb file at the original location so that users can tell why a file was moved.

Caution Copying or archiving files can change destination permissions, ownership, group information, and timestamps. Cross-protocol transfers can substantially broaden access. Review the destination security settings before transferring sensitive data, and verify permissions afterward.
Note The credentials you provide for an SMB destination are separate from the Active Directory integration. To display user names and permissions in AI Data Engine, refer to Configure data classification thresholds and access scanning.
Before you begin

  • You must have the AI Data Engine administrator role.

Steps

  1. In the Investigation view, select the files you want to archive.

  2. Select Archive.

  3. In the Archive dialog, select the destination volume protocol:

    • For NFS, enter the destination folder path in the format hostname:/path/to/destination.

    • For SMB, enter the SMB destination folder path in the format \\Hostname\share_name. Select the authentication method. For Kerberos, use the FQDN of the host and leave the credentials empty. Refer to Kerberos authentication for SMB/CIFS shares in AI Data Engine. For other methods, provide Active Directory credentials: enter the username in domain\user or user@domain format, and enter the password.

  4. To create a breadcrumb file at the original location, enable Leave breadcrumb.

    A breadcrumb file helps users understand why a file was moved.

  5. Select Archive file.

Tag files

Track files that you want to revisit or coordinate on with another team, without changing their location. For example, apply an "Archive candidate" tag to identify files that require review before you archive or delete them.

Steps

  1. In the Investigation view, select the files you want to tag.

  2. Select Tag.

  3. Enter or select the tags to apply (for example, "Archive candidate").

  4. Confirm to apply the tags.