Getting started with the Cloud Sync service Edit on GitHub Request doc changes

Contributors netapp-bcammett

Getting started with the Cloud Sync service includes a few steps.

number 1 Verify requirements for the source and target

The most important requirement is verifying connectivity between the data broker and the source and target locations.

number 2 Choose where to run the NetApp data broker

The NetApp data broker software syncs data from a source to a target (this is called a sync relationship). You can run the data broker in AWS, Azure, Google Cloud Platform, or on your premises. Cloud Sync guides you through the installation process when you create a sync relationship, at which point you can deploy the data broker in the cloud or download an install script for your own Linux host.

number3 Create your first sync relationship

Start your free trial from Cloud Central, drag and drop your selections for the source and target, and follow the prompts to complete the setup.

number4 Pay for your sync relationships after your free trial ends

You can pay-as-you-go or purchase licenses directly from NetApp. Just go to the License Settings page in Cloud Sync.

Source and target requirements

The source and target in a supported sync relationship must meet the following requirements:

Networking requirements
  • The source and target must have a network connection to the data broker.

    For example, if an NFS server is in your data center and the data broker is in AWS, then you need a network connection (VPN or Direct Connect) from your network to the VPC.

  • NetApp recommends configuring the source, target, and data broker to use a Network Time Protocol (NTP) service. The time difference between the three components should not exceed 5 minutes.

Requirements for SMB/CIFS and NFS servers
  • The SMB/CIFS or NFS server can be a NetApp system or a non-NetApp system.

  • The file server must allow the data broker host to access the exports.

  • NFS versions 3, 4.0, 4.1, and 4.2 are supported.

    The desired version must be enabled on the server.

  • SMB versions 1.0, 2.0, 2.1, and 3.0 are supported.

  • If you want Cloud Sync to copy ACLs between SMB/CIFS shares, then you must run a deployment script on a Windows host and activate support when you create a sync relationship.

  • If you want to sync NFS data from an ONTAP system, you must ensure that access to the NFS export list for an SVM is enabled (vserver nfs modify -vserver svm_name -showmount enabled).

    The default setting for showmount is enabled starting with ONTAP 9.2.
Requirements for Azure Blob storage

When creating a sync relationship between an Azure Blob container and an NFS or CIFS server, you need to provide Cloud Sync with the storage account connection string:

Shows a connection string

If you want to sync data between two Azure Blob containers, then the connection string must include a shared access signature (SAS). You also have the option to use a SAS when syncing between a Blob container and an NFS or CIFS server.

The SAS must allow access to the Blob service and all resource types (Service, Container, and Object). The SAS must also include the following permissions:

  • For the source Blob container: Read and List

  • For the target Blob container: Read, Write, List, Add, and Create

    Shows a shared access signature

Requirements for Google Cloud Storage

Sync relationships that include GCP storage require a GCP data broker or an on-prem data broker that has GCP access. Cloud Sync guides you through the data broker installation process when you create a sync relationship.

Data broker requirements

The data broker can reside in AWS, in Azure, in Google Cloud Platform, or in your network. You should be aware of the following requirements:

Supported regions
  • AWS: All regions are supported except for the China and GovCloud (US) regions.

  • Azure: All regions are supported except for the China, US Gov, and US DoD regions.

  • Google Cloud Platform: All regions are supported.

Networking requirements
  • The data broker needs an outbound internet connection so it can poll the Cloud Sync service for tasks.

    Cloud Sync configures outbound internet access by default when you install a data broker. Note that you can configure the data broker to use a proxy server during the installation process.

    If you need to limit outbound connectivity, see the list of endpoints that the data broker contacts.

  • Port 443 must be open on the data broker host for outbound traffic.

  • NetApp recommends configuring the source, target, and data broker to use a Network Time Protocol (NTP) service. The time difference between the three components should not exceed 5 minutes.

Permissions required to deploy the data broker in AWS

The AWS user account that you use to deploy the data broker must have the permissions included in the NetApp-provided policy.

Requirements to use your own IAM role with the AWS data broker

You can deploy a data broker in AWS using your own IAM role, rather than the IAM role that Cloud Sync creates for you. You might use this option if your organization has strict security policies.

To use your own IAM role, follow the prompts in Cloud Sync to create a data broker in AWS. Enter the name of the IAM role when you specify details for the AWS CloudFormation stack:

A screenshot of the AWS CloudFormation Console that shows the IAM role name field.

The IAM role must meet the following requirements:

Permissions required to deploy the data broker in Google Cloud Platform

Ensure that the following GCP permissions are in place for the service account that you use to create the data broker:

  • Compute Admin

  • Deployment Manager Editor

  • Service Account User

  • Storage Admin

Permissions required to prepare an existing Linux host for AWS access

When you install the data broker on an existing Linux host and AWS S3 is the source or target in the relationship, the instructions in the Cloud Sync interface prompt you to create an AWS user that has programmatic access and the permissions included in the NetApp-provided policy. Watch the installation video for more details.

Permissions required to prepare an existing Linux host for GCP access

When you install the data broker on an existing Linux host and GCP storage is the source or target in the relationship, Cloud Sync prompts you to provide GCP access:

  • Create a GCP service account that has Storage Admin permissions, if you don’t already have one.

  • Create a service account key saved in JSON format. View GCP instructions.

    The file should contain at least the following properties: "project_id", "private_key", "client_email"

    When you create a key, the file gets generated and downloaded to your machine.
  • Save the JSON file to the Linux host.

Linux host requirements

If you install the data broker on your own host, then you must verify support for your configuration. Cloud Sync provides the installation script when you create a relationship and choose an on-prem data broker. Watch the installation video for details.

  • Operating system: Red Hat Enterprise Linux 7 or CentOS 7 or Fedora

    The command yum update all must be run on the host before you install the data broker.

    A Red Hat Enterprise Linux system must be registered with Red Hat Subscription Management. If it is not registered, the system cannot access repositories to update required 3rd party software during installation.

  • RAM: 16 GB

  • CPU: 4 cores

  • Free disk space: 10 GB

Creating your first sync relationship

When you create a sync relationship, the Cloud Sync service copies files from the source to the target. After the initial copy, the service syncs any changed data every 24 hours.

Steps
  1. Go to NetApp Cloud Central.

  2. Sign up or log in and then start a free trial of Cloud Sync.

  3. After you log in, review details about using the service after the free trial ends, and then click OK.

  4. On the Select Source & Target page, choose a source and target.

    The following steps provide an example of how to create a sync relationship from an NFS server to an S3 bucket.

  5. Review the details about how the service works and then click Continue.

  6. On the NFS Server page, enter the IP address or fully qualified domain name of the NFS server that you want to sync to AWS.

  7. On the Data Broker page, follow the prompts to create a data broker virtual machine in AWS, Azure, or Google Cloud Platform, or to install the data broker software an existing Linux host.

    If you choose AWS, the AWS CloudFormation Console loads in a separate browser tab. The Cloud Sync service uses a CloudFormation template to quickly provision the data broker instance. The Cloud Sync service does not have access to your AWS credentials because you are logging in directly to AWS.

    AWS example

    The following video shows how to launch the data broker instance in AWS:

    The instance takes approximately 5 to 10 minutes to deploy. You can monitor the progress from the AWS CloudFormation console, or you can return to the Cloud Sync service website, which automatically refreshes when the instance is available.

    Existing host example

    The following video shows how to install the data broker software on an on-premises Linux host, or on an existing Linux host in the cloud:

    If you install the software using AWS keys, the keys are stored on the data broker. NetApp does not use the keys outside of the data broker.
  8. After you install the data broker, click Continue.

    The following image shows a successfully deployed instance in AWS:

    This screenshot shows a successfully deployed instance. The interface shows details about the instance including AWS networking.

  9. On the Directories page, select a top-level directory or subdirectory.

    If you want to sync more than one directory on the NFS server, then you must create additional sync relationships after you are done.
  10. On the S3 Bucket page, choose the target location:

  11. On the Settings page, define how source files and folders are synced and maintained in the target location:

    Recently modified files

    Choose to exclude files that were recently modified prior to the scheduled sync.

    Delete files on target

    Choose to delete files from the target location, if they were deleted from the source. The default is to never deletes files from the target location.

    Retries

    Define the number of times that Cloud Sync should retry to sync a file before skipping it.

    File types

    Define the file types to include in each sync.

  12. On the Relationship Tags page, enter up to 9 relationship tags and then click Continue.

    The Cloud Sync service assigns the tags to each object that it syncs to the S3 bucket.

  13. Review the details of the sync relationship and then click Create Relationship.

  14. After the Cloud Sync service successfully creates the relationship, click View in Dashboard to view details about the data sync relationship.

Result

Congratulations, you have created your first sync relationship.

Paying for sync relationships after your free trial ends

There are two ways to pay for sync relationships after your 14-day free trial ends. The first option is to subscribe from AWS or Azure to pay-as-you-go or to pay annually. The second option is to purchase licenses directly from NetApp.

AWS or Azure subscriptions

Subscribing to the Cloud Sync service from AWS or Azure enables you to pay at an hourly rate, or to pay annually. You can subscribe through either AWS or Azure, depending on where you want to be billed.

With a pay-as-you-go subscription, the Cloud Sync service charges hourly based on the number of sync relationships that you create. For pricing details, go to the Cloud Sync service page.

An annual subscription provides a license for 20 sync relationships that you pay for up front.

Azure enables you to use the pay-as-you-go and annual subscriptions together. If you go above 20 sync relationships, you would pay for the additional relationships by the hour.

Licenses from NetApp

Another way to pay for sync relationships up front is by purchasing licenses directly from NetApp. Each license enables you to create up to 20 sync relationships. After you purchase a license, you must add it to Cloud Sync. For further details about licenses, see How Cloud Sync licenses work.

You can use these licenses with an AWS or Azure subscription. For example, if you have 25 sync relationships, you can pay for the first 20 sync relationships using a license and then pay-as-you-go from AWS or Azure with the remaining 5 sync relationships.

What if I don’t immediately pay after my free trial ends?

You won’t be able to create any additional relationships. Existing relationships are not deleted, but you cannot make any changes to them until you subscribe or enter a license.

Subscribing from AWS

AWS enables you to pay-as-you-go or to pay annually.

Steps to pay-as-you-go
  1. Go to the License Settings page.

  2. Select AWS Marketplace.

  3. Click Subscribe and then click Continue.

  4. Subscribe from the AWS Marketplace, and then log back in to the Cloud Sync service to complete the registration.

    The following video shows the process:

Steps to pay annually
  1. Go to the AWS Marketplace page.

  2. Click Continue to Subscribe.

  3. Select your contract options and click Create contract.

Subscribing from Azure

Azure enables you to pay-as-you-go or to pay annually.

What you’ll need

An Azure user account that has Contributor or Owner permissions in the relevant subscription.

Steps
  1. Go to the License Settings page.

  2. Select Azure Marketplace.

  3. Click Subscribe and then click Continue.

  4. In the Azure portal, click Create, select your options, and click Subscribe.

    Select Monthly to pay by the hour, or Yearly to pay for a year up front.

  5. When deployment is complete, click the name of the SaaS resource in the notification pop-up.

  6. Click Configure Account to return to Cloud Sync.

    The following video shows the process:

Purchasing licenses from NetApp and adding them to Cloud Sync

To pay for your sync relationships up front, you must purchase one or more licenses and add them to the Cloud Sync service.

Steps
  1. Purchase a license by contacting NetApp.

  2. Go to the License Settings page and add the license.