Skip to main content
XCP

Configure the S3 connector

Contributors netapp-cgoff netapp-aoife
PDFs

Beginning with XCP 1.9.2, the Simple Storage Service (S3) connector enhances the scope of XCP data migration by enabling data migration from Hadoop Distributed File System (HDFS) file systems to S3 object storage.

Supported migration use cases

The following migration use cases are supported for the S3 connectors:

  • Migration from HDFS to NetApp StorageGRID

  • Migration from HDFS to Amazon S3

  • Migration from HDFS to NetApp ONTAP S3

Note Currently MapR is only qualified and supported for HDFS.
Supported features

Support for the scan, copy, verify, resume and delete commands is available for the S3 connectors.

Unsupported Features

Support for the sync command is not available for the S3 connectors.

Path Syntax

The path syntax for the S3 connector is s3://<bucket in S3>.

  • You can provide a specific S3 profile for the XCP commands using the -s3.profile option.

  • You can use the s3.endpoint option to modify the endpoint value to communicate with S3

Note Endpoint usage is mandatory for StorageGRID and ONTAP S3.

Set up an S3 connector

Steps

  1. To run the XCP command with the S3 connector, create a bucket in S3 by following the online documentation for the respective platforms:

  2. Install the Amazon Web Services (AWS) CLI packages and run the AWS CLI commands to configure the keys and Secure Sockets Layer (SSL) certificates for S3 accounts:

  3. Use the aws configure command to configure your credentials file. By default, the file's location is /root/.aws/credentials. The credentials file should specify the access key and secret access key.

  4. Use the aws configure set command to specify a CA certificate bundle, which is a file with the .pem extension that is used when verifying SSL certificates. By default, the file's location is /root/.aws/config.

    Example:

    [root@client1 ~]# aws configure
AWS Access Key ID [None]: <access_key>
AWS Secret Access Key [None]: <secret_key>
Default region name [None]:
Default output format [None]:
[root@client1 ~]# cat /root/.aws/credentials
[default]
aws_access_key_id = <access_key>
aws_secret_access_key = <secret_key>
[root@client1 ~]#
[root@client1 ~]# aws configure set default.ca_bundle /u/xxxx/s3/ca/aws_cacert.pem
[root@client1 ~]# cat /root/.aws/config
[default]
ca_bundle = /u/xxxx/s3/ca/aws_cacert.pem

  5. After the required setup configuration is completed, confirm that the AWS CLI commands can access the S3 buckets from the Linux client before running the XCP commands:
    aws s3 ls --endpoint-url <endpoint_url> s3://bucket-name/

    aws s3 ls --profile <profile> --endpoint-url <endpoint_url> s3://bucket-name

    Example:

    [root@client1 linux]# aws s3 ls --profile <profile> --endpoint <endpoint_url>  s3://<bucket-name>
                           PRE 1G/
                           PRE aws_files/
                           PRE copied_folders/
                           PRE d1/
                           PRE d2/
                           PRE giant_size_dirs/
                           PRE medium_size_dirs/
                           PRE small_size_dirs/
[root@client1 l