Skip to main content
NetApp Solutions

Data mover solution for AI

Contributors kevin-hoke
PDFs

The data mover solution for AI is based on customers' needs to process Hadoop data from AI operations. NetApp moves data from HDFS to NFS by using the NIPAM. In one use case, the customer needed to move data to NFS on the premises and another customer needed to move data from the Windows Azure Storage Blob to Google Cloud NetApp Volumes in order to process the data from the GPU cloud instances in the cloud.

The following diagram illustrates the data mover solution details.

Figure showing input/output dialog or representing written content

The following steps are required to build the data mover solution:

  1. ONTAP SAN provides HDFS, and NAS provides the NFS volume through NIPAM to the production data lake cluster.

  2. The customer’s data is in HDFS and NFS. The NFS data can be production data from other applications that is used for big data analytics and AI operations.

  3. NetApp FlexClone technology creates a clone of the production NFS volume and provisions it to the AI cluster on premises.

  4. Data from an HDFS SAN LUN is copied into an NFS volume with NIPAM and the hadoop distcp command. NIPAM uses the bandwidth of multiple network interfaces to transfer data. This process reduces the data copy time so that more data can be transferred.

  5. Both NFS volumes are provisioned to the AI cluster for AI operations.

  6. To process on-the-premises NFS data with GPUs in the cloud, the NFS volumes are mirrored to NetApp Private Storage (NPS) with NetApp SnapMirror technology and mounted to cloud service providers for GPUs.

  7. The customer wants to process data in EC2/EMR, HDInsight, or DataProc services in GPUs from cloud service providers. The Hadoop data mover moves the data from Hadoop services to the Google Cloud NetApp Volumes with NIPAM and the hadoop distcp command.

  8. The Google Cloud NetApp Volumes data is provisioned to AI through the NFS protocol.Data that is processed through AI can be sent on an on-premises location for big data analytics in addition to the NVIDIA cluster through NIPAM, SnapMirror, and NPS.

In this scenario, the customer has large file-count data in the NAS system at a remote location that is required for AI processing on the NetApp storage controller on premises. In this scenario, it’s better to use the XCP Migration Tool to migrate the data at a faster speed.

The hybrid-use-case customer can use BlueXP Copy and Sync to migrate on-premises data from NFS, CIFS, and S3 data to the cloud and vice versa for AI processing by using GPUs such as those in an NVIDIA cluster. Both BlueXP Copy and Sync and the XCP Migration Tool are used for the NFS data migration to NetApp ONTAP NFS.