Backing up and restoring data using the NetApp Cloud Backup Service in AWS Edit on GitHub Request doc changes

Contributors netapp-bcammett

The NetApp Cloud Backup Service for Cloud Volumes ONTAP delivers fully-managed backup and restore capabilities for protection and long-term archive of your cloud data. You can integrate the Cloud Backup Service with Cloud Volumes ONTAP for AWS. Backups created by the service are stored in AWS S3 object storage.

The Cloud Backup Service is supported with Cloud Volumes ONTAP 9.4 and later.

To get started, install and configure the backup agent and then run commands to start backup and restore operations. If you need help, we encourage you to contact us by using the chat icon in Cloud Manager.

Contacting NetApp to use the Cloud Backup Service

Before you get started, send us your AWS account ID so that we can enable your account to use the Cloud Backup Service.

Step
  1. Send an email to ng-cloud-volume-ontap-preview@netapp.com that includes your AWS account ID.

Verifying your AWS permissions

To complete the steps on this page, your AWS account must have the following permissions:

ec2:DescribeVpcEndpoints
ec2:CreateVpcEndpoint
ec2:ModifyVpcEndpoint

Installing and configuring the Cloud Backup Service agent

We recommend that you install the backup agent on the Cloud Manager instance. An internet connection is required to download the agent package and a connection to Cloud Volumes ONTAP systems is required for backup and restore operations. The Cloud Manager instance should meet both of those requirements.

Downloading the agent package

Download the agent package from an S3 bucket and uncompress the files to a directory on the instance.

Steps
  1. SSH to the Cloud Manager EC2 instance.

  2. Download the agent for the Cloud Backup Service:

  3. Untar the compressed files into another existing directory:

    tar -xvf folder.tar -C /target/directory

Creating VPC interface endpoints to the Cloud Backup Service

Two VPC interface endpoints are required to communicate with the Cloud Backup Service.

Steps
  1. At the AWS CLI, run aws configure and provide your secret key and access key.

  2. Run the following command to create a VPC interface endpoint to the backup service:

    aws --region <region> cloudformation create-stack --stack-name <stack-name> --template-body file://vpc_endpoint.json --parameters ParameterKey=VPCId,ParameterValue=<vpc-id> ParameterKey=ServiceCbs,ParameterValue=<service-endpoint>

    • The region and VPC ID should correspond to the network location where your Cloud Volumes ONTAP systems are deployed.

    • The stack name can be any string that identifies the purpose of the CFT.

    • The template body specifies the location of the Cloud Formation Template that was included in the agent package that you downloaded and uncompressed to a directory.

    • The service endpoint must point to the backup service endpoint in the eu-west-1 region or the us-east-1 region:

      • eu-west-1: com.amazonaws.vpce.eu-west-1.vpce-svc-0cd9f6bf5706d5ec9

      • us-east-1: com.amazonaws.vpce.us-east-1.vpce-svc-046e3104ac8604f7e

  3. Now run the following command to create a VPC interface endpoint to the restore service:

    aws --region <region> cloudformation create-stack --stack-name <stack-name> --template-body file://vpc_endpoint.json --parameters ParameterKey=VPCId,ParameterValue=<vpc-id> ParameterKey=ServiceCbs,ParameterValue=<service-endpoint>

    The only difference between this command and the previous command is the value of the service endpoint. The service endpoint must point to the restore service endpoint in the eu-west-1 region or the us-east-1 region:

    • eu-west-1: com.amazonaws.vpce.eu-west-1.vpce-svc-04dd86976846bc84b

    • us-east-1: com.amazonaws.vpce.us-east-1.vpce-svc-0343dd25489869199

Configuring the subnets for the VPC interface endpoints

The VPC interface endpoints that you just created must be added to the subnets where the Cloud Volumes ONTAP systems reside.

Steps
  1. Go to the AWS Console, select the VPC service, and click Endpoints.

  2. Select the VPC interface endpoint for the backup service.

  3. Click the Subnets tab and click Manage Subnets.

    A screenshot of the AWS VPC Console that shows the Manage Subnets button in the Subnets tab after you select and endpoint.

  4. Choose the subnets where the Cloud Volumes ONTAP systems reside and click Modify Subnets.

  5. From the Subnets tab, copy the private IP address of the subnet where the Cloud Volumes ONTAP system resides.

    You’ll need this IP address later when you configure the Cloud Backup Service agent by modifying the config.json file.

  6. Repeat these steps for the restore service endpoint.

Generating a user token for authentication

Each backup and restore operation requires a user token for authentication.

Steps
  1. Go to the API Documentation for NetApp Cloud Central.

  2. Click Learn how to authenticate.

  3. Follow the instructions to generate an access token using regular access or federated access, while making a few changes to the body so it works for the Cloud Backup Service.

    1. For regular access, use the following audience and client_id:

      "audience": "https://cloudmanager.cloud.netapp.com",
      "client_id": "_CloudManagerClientID_"

      Obtain the client ID for Cloud Manager by using the following API:

      GET /occm/system/support-services
    2. For federated access, use the following audience:

      "audience": "https://cloudmanager.cloud.netapp.com",
  4. After you receive the token, copy the value into the user_token.json file by overwriting the existing value.

    You can find this file in the same directory where you uncompressed the agent package.

    NetApp Cloud Central uses the token to authenticate the user for all API calls between Cloud Volumes ONTAP and the Cloud Backup Service. If the token is invalid or expired, the API calls will fail and backup and restore operations will not start.

Configuring the Cloud Backup Service agent

Modify the agent’s configuration file by specifying the IP addresses of the network interfaces for the VPC endpoints. This enables the agent to contact the Cloud Backup Service.

About this task

If you update this configuration file after you start the agent, you’ll need to kill the service and then restart it. See Updating the config.json file.

Steps
  1. Go to the directory where you uncompressed the agent package.

  2. Edit the config.json file by specifying the IP addresses.

    {
              "LRSE_BACKUP_IP": "<ENI private IP for backup>",
              "LRSE_RESTORE_IP": "<ENI private IP for restore>",
              "CBS_ENDPOINT_IP" : "<ENI private IP for backup>",
              "CBS_ENDPOINT_PORT" : "8088",
              "SNAPMIRROR_POLICY_TRIES": 8,
              "SNAPMIRROR_RETRY_COUNT": 10,
              "SNAPMIRROR_POLL_INTERVAL":30
    }
    • LRSE_BACKUP_IP is the private IP address of the VPC interface endpoint that’s connected to the backup service. You can find the IP address in the AWS console. Go to the VPC service and select the VPC Endpoint. Click Subnets and find the IP address of the subnet where the Cloud Volumes ONTAP system resides.

      A screenshot of the AWS VPC Console that shows the network interfaces for a VPC Endpoint.

    • LRSE_RESTORE_IP is the private IP address of the VPC interface endpoint that’s connected to the restore service. Follow the same instructions provided for LRSE_BACKUP_IP.

    • CBS_ENDPOINT_IP should be the same as the LRSE_BACKUP_IP since we use the same VPC interface endpoint for making API calls.

What if I’m backing multiple Cloud Volumes ONTAP systems?

It’s okay to use the same IP addresses for multiple Cloud Volumes ONTAP systems, as long as the subnets are in the same Availability Zone. If you need to back up multiple systems that are spread across Availability Zones, contact us using the in-product chat and we’ll help you with your setup.

Starting the Cloud Backup Service agent

Now that you’ve installed and configured the agent, you need to start it.

Steps
  1. Run the following commands:

    chmod +x cvo-cbs-service
    chmod +x cvo-cbs-client
    ./cvo-cbs-service &

Preparing to back up volumes

When you run a backup operation, you need to specify a JSON file that includes information about the volume. You can use a JSON template to prepare a JSON file for each volume.

Preparing ad hoc backups

An ad hoc backup is an immediate, one-time backup. Prepare a separate JSON file for each volume that you want to backup.

Steps
  1. Create a copy of adhoc_backup.json and edit it by providing details about the volume.

    {
           "ownerId": "e7855e3e-006d-49f0-bd1e-2c0df8fec505",
           "ontapIP": "10.193.78.9",
           "username": "admin",
           "password": "netapp1!",
           "vserverName": "vs_seeni",
           "volumeName": "backup",
           "fileSystemId": "cf765c5f-84e6-4080-84a7-599ab8a31968",
           "sourceSnapshot": "snap10",
           "tag": ""
    }
    • ownerId: A unique identifier for all of the backup and restore operations associated with this Cloud Manager system. Run the "uuidgen" UNIX utility to generate an ID and use it in all JSON files for backup and restore operations.

    • ontapIP: The cluster management IP of the Cloud Volumes ONTAP system where the volume is located. Get this value from Cloud Manager by selecting the system from the Working Environments page.

    • username and password: The credentials for the Cloud Volumes ONTAP system.

    • vserverName: The name of the storage virtual machine (SVM) that contains data volumes. Get this value from Cloud Manager by opening the working environment and selecting Information.

    • volumeName: The name of the volume name that you want to backup.

    • fileSystemId: The file system UUID for the volume’s backup copy. This value must be unique for every volume because it’s used by the Cloud Backup Service to identify a volume. Generate an ID by running the "uuidgen" UNIX utility.

    • sourceSnapshot (optional): Specify the name of a Snapshot copy that you want to backup. If you omit this parameter, the Cloud Backup Service backs up the volume based on its existing state.

    • tag (optional): Specify a tag for the backup so you can search for it more easily.

Preparing scheduled backups

A scheduled backup triggers incremental backups at a defined interval. Prepare a separate JSON file for each volume that you want to backup.

Steps
  1. Create a copy of scheduled_backup.json and edit it by providing details about the volume.

    {
          "ownerId": "e7855e3e-006d-49f0-bd1e-2c0df8fec505",
          "ontapIP": "10.193.78.9",
          "username": "admin",
          "password": "netapp1!",
          "vserverName": "vs_seeni",
          "volumeName": "backup",
          "fileSystemId": "e2334e3e-226d-39f0-bd1e-1c0df6fec215",
          "snapmirrorPolicy": {
          "enabled": true,
          "daily-schedule": {
          "snapmirrorLabel": "sm_daily",
          "snapshotsToKeep": 24
          },
          "weekly-schedule": {
          "snapmirrorLabel": "sm_weekly",
          "snapshotsToKeep": 4
          },
          "monthly-schedule": {
          "snapmirrorLabel": "sm_monthly",
          "snapshotsToKeep": 40
         }
    }
    }
    • ownerId: A unique identifier for all of the backup and restore operations associated with this Cloud Manager system. Run the "uuidgen" UNIX utility to generate an ID and use it in all JSON files for backup and restore operations.

    • ontapIP: The cluster management IP of the Cloud Volumes ONTAP system where the volume is located. Get this value from Cloud Manager by selecting the system from the Working Environments page.

    • username and password: The credentials for the Cloud Volumes ONTAP system.

    • vserverName: The name of the storage virtual machine (SVM) that contains data volumes. Get this value from Cloud Manager by opening the working environment and selecting Information.

    • volumeName: The name of the volume name that you want to backup.

    • fileSystemId: The file system UUID for the volume’s backup copy. This value must be unique for every volume because it’s used by Cloud Backup Service to identify a volume. Generate an ID by running the "uuidgen" UNIX utility.

    • snapmirrorPolicy: Defines the SnapMirror policy for the scheduled backup.

    • enabled: Enables the policy.

    • daily-schedule: Defines daily scheduling information for the policy.

    • weekly-schedule: Defines weekly scheduling information for the policy.

    • monthly-schedule: Defines monthly scheduling information for the policy.

    • snapmirrorLabel: A SnapMirror label for the rule.

    • snapshotsToKeep: The number of Snapshot copies to keep.

  2. Create a Snapshot policy on the Cloud Volumes ONTAP system and modify the volume to use the Snapshot policy.

    For scheduled backups to work, a corresponding Snapshot policy must be configured on the Cloud Volumes ONTAP system and attached to the volume. The label for the Snapshot policy must match the value of the snapmirrorLabel that you specified in the JSON file.

    Example

    cluster1::> volume snapshot policy create -vserver vs0 -policy mysnappolicy -schedule1 hourly-count1 5 -prefix1 every_hour -snapmirror-label1 hrLabel

    cluster1::> volume modify -vserver vs0 -volume backup -snapshot-policy mysnappolicy

Preparing to restore volumes

When you restore a volume, the Cloud Backup Service restores the contents of the volume to a data protection volume that you must create beforehand. To prepare for a restore, create the new data protection volume and set up a JSON file that specifies details about the volume restore. You’ll specify the JSON file when you run the restore operation.

Steps
  1. Create the data protection volume to which you’ll restore the contents of the volume.

    Example

    cluster1::> vol create -volume restoreVol -aggregate aggr1 -size 100GB -state online -policy default -type DP

  2. Create a copy of restore.json and edit it by providing details about the volume.

    {
           "ownerId": "e7855e3e-006d-49f0-bd1e-2c0df8fec505",
           "ontapIP": "10.193.78.9",
           "username": "admin",
           "password": "netapp1!",
           "vserverName": "vs_seeni",
           "fileSystemId": "cf765c5f-84e6-4080-84a7-599ab8a31967",
           "destinationVolumeName": "retoreauth",
           "restoreSnapshot": ""
    }
    • ownerId: A unique identifier for all of the backup and restore operations associated with this Cloud Manager system.

    • ontapIP: The cluster management IP of the Cloud Volumes ONTAP system where the volume is located. Get this value from Cloud Manager by selecting the system from the Working Environments page.

    • username/password: The credentials for the Cloud Volumes ONTAP system.

    • vserverName: The name of the storage virtual machine (SVM) that contains data volumes. The value should match what you entered in the JSON file for the backup operation.

    • fileSystemId: The file system UUID for the volume’s backup copy. The value should match what you entered in the JSON file for the backup operation.

    • destinationVolumeName: Specify the name of the destination volume that you created in step 1. The volume must be a data protection (DP) volume.

    • restoreSnapshot: Specify the name of a Snapshot copy that you want to restore. If you don’t want to specify a specific Snapshot copy, enter an empty value as shown above.

Backing up and restoring volumes

Once you’re ready, start backing up and restoring your volumes.

Steps
  1. Run the following command from the Cloud Manager instance:

    ./cvo-cbs-client

  2. Select an action from the prompt:

    1) Backup a volume

    Run a one-time backup. When prompted, specify the absolute path for the JSON file that corresponds to the volume that you want to backup.

    2) Scheduled backup

    Use a scheduled backup to periodically trigger incremental backups. When prompted, specify the absolute path for the JSON file that corresponds to the volume that you want to backup.

    3) Restore to a DP volume

    Restore a volume that you previously backed up. When prompted, specify the absolute path for the JSON file that corresponds to the volume that you want to restore.

    4) Job Status

    Display the job status for backup and restore operations. Backup and restore operations are async operations, so you’ll get a job ID when you run an operation. Use that ID as input when this option prompts for the jobId.

    5) List Volume Backups

    List all backups corresponding to a volume. The fileSystemId that you specified in the backup JSON template must be provided as input to this option.

    6) Exit

    Exit the prompt.

Example 1

Choose a operation to be performed:1
Enter backup volume details json file path:/home/ubuntu/cvo-cbs-agent/adhoc_backup.json
Processing Backup request...
Adhoc backup initiated successfully.
Get Backup status using JobID 45

Example 2

Choose a operation to be performed:5
Enter filesystem id to list backups: cf761c4f-84e6-4080-84a7-599ab8b31965
Processing List backups for:  cf761c4f-84e6-4080-84a7-599ab8b31965
Total Backups are: 1
--------------------Backup 0 details--------------------
backup Id = db682289-b896-d248-ac29-a13e4e8e1bbb
backup Name = adhoc_2019-04-06_150037
backup type = adhoc
completion time = 2019-04-06T15:02:23.000Z
creation time = 2019-04-06T15:00:51.000Z
size =  1.2582912e+09
status = Backup Complete

Example 3

Choose a operation to be performed:3
Enter restore volume details json file path: /home/ubuntu/cvo-cbs-agent/restore.json
Processing volume restore request ...
Restore initiated successfully.
Track Backup status using JobID 47

Mounting a restored volume

After you restore the volume, you’ll need to mount it to view the files that it contains.

Steps
  1. Connect to the Cloud Volumes ONTAP CLI.

  2. Add a junction path:

    volume mount -vserver <vserver_name> -volume <volume_name> -junction-path /<junction-path-name>

  3. Verify that the volume is in the desired mount state:

    volume show -vserver <vserver_name> -volume <volume_name> -junction

  4. Mount the volume to the client.

Administering

Updating the config.json file

If you need to update the config.json file after you start the agent, you’ll need to kill the service and then restart it.

Steps
  1. Stop the Cloud Backup Service agent:

    ps -ef | grep -i cvo-cbs-service
    kill -9 <service PID>

  2. Update the configuration file.

  3. Start the agent.

Troubleshooting unauthorized access

If there is a problem with the access token, you might receive the following error message:

Post to CVO api server returned status code =401 and error Unauthorized

If this happens, generate a new token and update the user_token.json file. For details, see Generating a user token for authentication.

Troubleshooting the fail to open database error

If you receive the following database error, kill the previously running cvo-cbs-service.

Error setting up pending jobs db: timeout
  1. Stop the Cloud Backup Service agent:

    ps -ef | grep -i cvo-cbs-service
    kill -9 <service PID>

  2. Start the agent.