HDFS and MapR-FS to ONTAP NFS
Suggest changes
PDFs
For this solution, NetApp validated the migration of data from data lake (HDFS) and MapR cluster data to ONTAP NFS. The data resided in MapR-FS and HDFS. NetApp XCP introduced a new feature that directly migrates the data from a distributed file system such as HDFS and MapR-FS to ONTAP NFS. XCP uses async threads and HDFS C API calls to communicate and transfer data from MapR- FS as well as HDFS.
The below figure shows the data migration from data lake (HDFS) and MapR-FS to ONTAP NFS. With this new feature, you don’t have to export the source as an NFS share.
Why are customers moving from HDFS and MapR-FS to NFS?
Most of the Hadoop distributions such as Cloudera and Hortonworks use HDFS and MapR distributions uses their own filesystem called Mapr-FS to store data. HDFS and MapR-FS data provides the valuable insights to data scientists that can be leveraged in machine learning (ML) and deep learning (DL). The data in HDFS and MapR-FS is not shared, which means it cannot be used by other applications. Customers are looking for shared data, specifically in the banking sector where customers’ sensitive data is used by multiple applications. The latest version of Hadoop (3.x or later) supports NFS data source, which can be accessed without additional third-party software. With the new NetApp XCP feature, data can be moved directly from HDFS and MapR-FS to NetApp NFS in order to provide access to multiple applications
Testing was done in Amazon Web Services (AWS) to transfer the data from MapR-FS to NFS for the initial performance test with 12 MAPR nodes and 4 NFS servers.
| Quantity | Size | vCPU | Memory | Storage | Network | |
|---|---|---|---|---|---|---|
NFS server |
4 |
i3en.24xlarge |
96 |
488GiB |
8x 7500 NVMe SSD |
100 |
MapR nodes |
12 |
I3en.12xlarge |
48 |
384GiB |
4x 7500 NVMe SSD |
50 |
Based on initial testing, we obtained 20GBps throughput and were able to transfer 2PB per day of data.
For more information about HDFS data migration without exporting HDFS to NFS, see the “Deployment steps - NAS” section in TR-4863: TR-4863: Best-Practice Guidelines for NetApp XCP - Data Mover, File Migration, and Analytics.