Data mover solution for AI

Contributors

The data mover solution for AI is based on customers' needs to process Hadoop data from AI operations. NetApp moves data from HDFS to NFS by using the NIPAM. In one use case, the customer needed to move data to NFS on the premises and another customer needed to move data from the Windows Azure Storage Blob to Cloud Volumes Service in order to process the data from the GPU cloud instances in the cloud.

The following diagram illustrates the data mover solution details.

Error: Missing Graphic Image

The following steps are required to build the data mover solution:

  1. ONTAP SAN provides HDFS, and NAS provides the NFS volume through NIPAM to the production data lake cluster.

  2. The customer’s data is in HDFS and NFS. The NFS data can be production data from other applications that is used for big data analytics and AI operations.

  3. NetApp FlexClone technology creates a clone of the production NFS volume and provisions it to the AI cluster on premises.

  4. Data from an HDFS SAN LUN is copied into an NFS volume with NIPAM and the hadoop distcp command. NIPAM uses the bandwidth of multiple network interfaces to transfer data. This process reduces the data copy time so that more data can be transferred.

  5. Both NFS volumes are provisioned to the AI cluster for AI operations.

  6. To process on-the-premises NFS data with GPUs in the cloud, the NFS volumes are mirrored to NetApp Private Storage (NPS) with NetApp SnapMirror technology and mounted to cloud service providers for GPUs.

  7. The customer wants to process data in EC2/EMR, HDInsight, or DataProc services in GPUs from cloud service providers. The Hadoop data mover moves the data from Hadoop services to the Cloud Volumes Services with NIPAM and the hadoop distcp command.

  8. The Cloud Volumes Service data is provisioned to AI through the NFS protocol.Data that is processed through AI can be sent on an on-premises location for big data analytics in addition to the NVIDIA cluster through NIPAM, SnapMirror, and NPS.

In this scenario, the customer has large file-count data in the NAS system at a remote location that is required for AI processing on the NetApp storage controller on premises. In this scenario, it’s better to use the XCP Migration Tool to migrate the data at a faster speed.

The hybrid-use-case customer can use Cloud Sync to migrate on-premises data from NFS, CIFS, and S3 data to the cloud and vice versa for AI processing by using GPUs such as those in an NVIDIA cluster. Both Cloud Sync and the XCP Migration Tool are used for the NFS data migration to NetApp ONTAP NFS.