Backing up on-premises ONTAP data to Amazon S3

Complete a few steps to get started backing up volume data from your on-premises ONTAP systems to Amazon S3 storage.

Note that "on-premises ONTAP systems" includes FAS, AFF, and ONTAP Select systems.

Quick start

Get started quickly by following these steps. Details for each step are provided in the following sections in this topic.

One Identify the configuration method you’ll use

Choose whether you’ll connect your on-premises ONTAP cluster directly to AWS S3 over the public internet, or whether you’ll use a VPN or AWS Direct Connect and route traffic through a private VPC Endpoint interface to AWS S3.

Two Prepare your BlueXP Connector

If you already have a Connector deployed in your AWS VPC or on your premises, then you’re all set. If not, then you’ll need to create a Connector to back up ONTAP data to AWS S3 storage. You’ll also need to customize network settings for the Connector so that it can connect to AWS S3.

Three Prepare your on-premises ONTAP cluster

Discover your ONTAP cluster in BlueXP, verify that the cluster meets minimum requirements, and customize network settings so the cluster can connect to AWS S3.

Four Prepare Amazon S3 as your backup target

Set up permissions for the Connector to create and manage the S3 bucket. You’ll also need to set up permissions for the on-premises ONTAP cluster so it can read and write data to the S3 bucket.

Optionally, you can set up your own custom-managed keys for data encryption instead of using the default Amazon S3 encryption keys. See how to get your AWS S3 environment ready to receive ONTAP backups.

Five Enable BlueXP backup and recovery on the system

Select the working environment and click Enable > Backup Volumes next to the Backup and recovery service in the right-panel. Then follow the setup wizard to define the default backup policy and number of backups to retain, and select the volumes you want to back up.

Network diagrams for connection options

There are two connection methods you can use when configuring backups from on-premises ONTAP systems to AWS S3.

  • Public connection - Directly connect the ONTAP system to AWS S3 using a public S3 endpoint.

  • Private connection - Use a VPN or AWS Direct Connect and route traffic through a VPC Endpoint interface that uses a private IP address.

The following diagram shows the public connection method and the connections that you need to prepare between the components. You can use a Connector that you’ve installed on your premises, or a Connector that you’ve deployed in the AWS VPC.

A diagram showing how BlueXP backup and recovery communicates over a public connection with the volumes on the cluster and the AWS S3 storage where the backup files are located.

The following diagram shows the private connection method and the connections that you need to prepare between the components. You can use a Connector that you’ve installed on your premises, or a Connector that you’ve deployed in the AWS VPC.

A diagram showing how BlueXP backup and recovery communicates over a private connection with the volumes on the cluster and the AWS S3 storage where the backup files are located.

Prepare your Connector

The BlueXP Connector is the main software for BlueXP functionality. A Connector is required to back up and restore your ONTAP data.

Creating or switching Connectors

If you already have a Connector deployed in your AWS VPC or on your premises, then you’re all set. If not, then you’ll need to create a Connector in either of those locations to back up ONTAP data to AWS S3 storage. You can’t use a Connector that’s deployed in another cloud provider.

Connector networking requirements

  • Ensure that the network where the Connector is installed enables the following connections:

    • An HTTPS connection over port 443 to the BlueXP backup and recovery service and to your S3 object storage (see the list of endpoints)

    • An HTTPS connection over port 443 to your ONTAP cluster management LIF

    • Additional inbound and outbound security group rules are required for AWS and AWS GovCloud deployments. See Rules for the Connector in AWS for details.

  • Ensure that the Connector has permissions to manage the S3 bucket.

  • If you have a Direct Connect or VPN connection from your ONTAP cluster to the VPC, and you want communication between the Connector and S3 to stay in your AWS internal network (a private connection), you’ll need to enable a VPC Endpoint interface to S3. See how to set up a VPC endpoint interface.

Prepare your ONTAP cluster

Discover your ONTAP cluster in BlueXP

You need to discover your on-premises ONTAP cluster in BlueXP before you can start backing up volume data. You’ll need to know the cluster management IP address and the password for the admin user account to add the cluster.

ONTAP requirements

  • Minimum of ONTAP 9.7P5; ONTAP 9.8P13 and later is recommended.

  • A SnapMirror license (included as part of the Premium Bundle or Data Protection Bundle).

    Note: The "Hybrid Cloud Bundle" is not required when using BlueXP backup and recovery.

  • Time and time zone are set correctly.

Cluster networking requirements

  • The cluster requires an inbound HTTPS connection from the Connector to the cluster management LIF.

  • An intercluster LIF is required on each ONTAP node that hosts the volumes you want to back up. These intercluster LIFs must be able to access the object store.

    The cluster initiates an outbound HTTPS connection over port 443 from the intercluster LIFs to Amazon S3 storage for backup and restore operations. ONTAP reads and writes data to and from object storage — the object storage never initiates, it just responds.

  • The intercluster LIFs must be associated with the IPspace that ONTAP should use to connect to object storage. Learn more about IPspaces.

    When you set up BlueXP backup and recovery, you are prompted for the IPspace to use. You should choose the IPspace that these LIFs are associated with. That might be the "Default" IPspace or a custom IPspace that you created.

    If you use are using a different IPspace than "Default", then you might need to create a static route to get access to the object storage.

    All intercluster LIFs within the IPspace must have access to the object store. If you can’t configure this for the current IPspace, then you’ll need to create a dedicated IPspace where all intercluster LIFs have access to the object store.

  • DNS servers must have been configured for the storage VM where the volumes are located. See how to configure DNS services for the SVM.

  • Update firewall rules, if necessary, to allow BlueXP backup and recovery connections from ONTAP to object storage through port 443 and name resolution traffic from the storage VM to the DNS server over port 53 (TCP/UDP).

  • If you are using a Private VPC Interface Endpoint in AWS for the S3 connection, then in order for HTTPS/443 to be used, you’ll need to load the S3 endpoint certificate into the ONTAP cluster. See how to set up a VPC endpoint interface and load the S3 certificate.

  • Ensure that your ONTAP cluster has permissions to access the S3 bucket.

Verify license requirements

  • Before you can activate BlueXP backup and recovery for your cluster, you’ll need to either subscribe to a pay-as-you-go (PAYGO) BlueXP Marketplace offering from AWS, or purchase and activate a BlueXP backup and recovery BYOL license from NetApp. These licenses are for your account and can be used across multiple systems.

  • You need to have an AWS subscription for the object storage space where your backups will be located.

    You can create backups from on-premises systems to Amazon S3 in all regions where Cloud Volumes ONTAP is supported; including AWS GovCloud regions. You specify the region where backups will be stored when you set up the service.

Prepare your AWS environment

Set up S3 permissions

You’ll need to configure two sets of permissions:

  • Permissions for the Connector to create and manage the S3 bucket.

  • Permissions for the on-premises ONTAP cluster so it can read and write data to the S3 bucket.

Steps
  1. Confirm that the following S3 permissions (from the latest BlueXP policy) are part of the IAM role that provides the Connector with permissions. If they are not, see the AWS Documentation: Editing IAM policies.

    {
              "Sid": "backupPolicy",
              "Effect": "Allow",
              "Action": [
                  "s3:DeleteBucket",
                  "s3:GetLifecycleConfiguration",
                  "s3:PutLifecycleConfiguration",
                  "s3:PutBucketTagging",
                  "s3:ListBucketVersions",
                  "s3:GetObject",
                  "s3:DeleteObject",
                  "s3:PutObject",
                  "s3:ListBucket",
                  "s3:ListAllMyBuckets",
                  "s3:GetBucketTagging",
                  "s3:GetBucketLocation",
                  "s3:GetBucketPolicyStatus",
                  "s3:GetBucketPublicAccessBlock",
                  "s3:GetBucketAcl",
                  "s3:GetBucketPolicy",
                  "s3:PutBucketPolicy",
                  "s3:PutBucketOwnershipControls",
                  "s3:PutBucketPublicAccessBlock",
                  "s3:PutEncryptionConfiguration",
                  "s3:GetObjectVersionTagging",
                  "s3:GetBucketObjectLockConfiguration",
                  "s3:GetObjectVersionAcl",
                  "s3:PutObjectTagging",
                  "s3:DeleteObjectTagging",
                  "s3:GetObjectRetention",
                  "s3:DeleteObjectVersionTagging",
                  "s3:PutBucketObjectLockConfiguration",
                  "s3:ListBucketByTags",
                  "s3:DeleteObjectVersion",
                  "s3:GetObjectTagging",
                  "s3:PutBucketVersioning",
                  "s3:PutObjectVersionTagging",
                  "s3:GetBucketVersioning",
                  "s3:BypassGovernanceRetention",
                  "s3:PutObjectRetention",
                  "s3:GetObjectVersion",
                  "athena:StartQueryExecution",
                  "athena:GetQueryResults",
                  "athena:GetQueryExecution",
                  "glue:GetDatabase",
                  "glue:GetTable",
                  "glue:CreateTable",
                  "glue:CreateDatabase",
                  "glue:GetPartitions",
                  "glue:BatchCreatePartition",
                  "glue:BatchDeletePartition"
              ],
              "Resource": [
                  "arn:aws:s3:::netapp-backup-*"
              ]
          },
    Note When creating backups in AWS China regions, you need to change the AWS Resource Name "arn" under all Resource sections in the IAM policies from "aws" to "aws-cn"; for example arn:aws-cn:s3:::netapp-backup-*.
  2. When activating the service, the Backup wizard will prompt you to enter an access key and secret key. These credentials are passed to the ONTAP cluster so that ONTAP can back up and restore data to the S3 bucket. For that, you’ll need to create an IAM user with the following permissions:

    {
        "Version": "2012-10-17",
         "Statement": [
            {
               "Action": [
                    "s3:GetObject",
                    "s3:PutObject",
                    "s3:DeleteObject",
                    "s3:ListBucket",
                    "s3:ListAllMyBuckets",
                    "s3:GetBucketLocation",
                    "s3:PutEncryptionConfiguration"
                ],
                "Resource": "arn:aws:s3:::netapp-backup-*",
                "Effect": "Allow",
                "Sid": "backupPolicy"
            }
        ]
    }
    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Action": [
                    "s3:ListBucket",
                    "s3:GetBucketLocation"
                ],
                "Resource": "arn:aws:s3:::netapp-backup*",
                "Effect": "Allow"
            },
            {
                "Action": [
                    "s3:GetObject",
                    "s3:PutObject",
                    "s3:DeleteObject",
                    "s3:ListAllMyBuckets",
                    "s3:PutObjectTagging",
                    "s3:GetObjectTagging",
                    "s3:RestoreObject",
                    "s3:GetBucketObjectLockConfiguration",
                    "s3:GetObjectRetention",
                    "s3:PutBucketObjectLockConfiguration",
                    "s3:PutObjectRetention"
                ],
                "Resource": "arn:aws:s3:::netapp-backup*/*",
                "Effect": "Allow"
            }
        ]
    }

Set up customer-managed AWS keys for data encryption

If you want to use the default Amazon S3 encryption keys to encrypt the data passed between your on-prem cluster and the S3 bucket, then you are all set because the default installation uses that type of encryption.

If you want to use your own customer-managed keys for data encryption instead of using the default keys, then you’ll need to have the encryption managed keys already set up before you start the BlueXP backup and recovery wizard. See how to use your own keys.

Configure your system for a private connection using a VPC endpoint interface

If you want to use a standard public internet connection, then all the permissions are set by the Connector and there is nothing else you need to do. This type of connection is shown in the first diagram.

If you want to have a more secure connection over the internet from your on-prem data center to the VPC, there’s an option to select an AWS PrivateLink connection in the Backup activation wizard. It’s required if you plan to use a VPN or AWS Direct Connect to connect your on-premises system through a VPC Endpoint interface that uses a private IP address. This type of connection is shown in the second diagram.

  1. Create an Interface endpoint configuration using the Amazon VPC console or the command line. See details about using AWS PrivateLink for Amazon S3.

  2. Modify the security group configuration that’s associated with the BlueXP Connector. You must change the policy to "Custom" (from "Full Access"), and you must add the S3 permissions from the backup policy as shown earlier.

    A screenshot of the AWS security group associated with the Connector.

    If you’re using port 80 (HTTP) for communication to the private endpoint, you’re all set. You can enable BlueXP backup and recovery on the cluster now.

    If you’re using port 443 (HTTPS) for communication to the private endpoint, you must copy the certificate from the VPC S3 endpoint and add it to your ONTAP cluster, as shown in the next 4 steps.

  3. Obtain the DNS name of the endpoint from the AWS Console.

    A screenshot of the DNS name of the VPC endpoint from the AWS Console.

  4. Obtain the certificate from the VPC S3 endpoint. You do this by logging into the VM that hosts the BlueXP Connector and running the following command. When entering the DNS name of the endpoint, add “bucket” to the beginning, replacing the “*”:

    [ec2-user@ip-10-160-4-68 ~]$ openssl s_client -connect bucket.vpce-0ff5c15df7e00fbab-yxs7lt8v.s3.us-west-2.vpce.amazonaws.com:443 -showcerts
  5. From the output of this command, copy the data for the S3 certificate (all data between, and including, the BEGIN / END CERTIFICATE tags):

    Certificate chain
    0 s:/CN=s3.us-west-2.amazonaws.com`
       i:/C=US/O=Amazon/OU=Server CA 1B/CN=Amazon
    -----BEGIN CERTIFICATE-----
    MIIM6zCCC9OgAwIBAgIQA7MGJ4FaDBR8uL0KR3oltTANBgkqhkiG9w0BAQsFADBG
    …
    …
    GqvbOz/oO2NWLLFCqI+xmkLcMiPrZy+/6Af+HH2mLCM4EsI2b+IpBmPkriWnnxo=
    -----END CERTIFICATE-----
  6. Log into the ONTAP cluster CLI and apply the certificate you copied using the following command (substitute your own storage VM name):

    cluster1::> security certificate install -vserver cluster1 -type server-ca
    Please enter Certificate: Press <Enter> when done

Enable BlueXP backup and recovery

Enable BlueXP backup and recovery at any time directly from the on-premises working environment.

Steps
  1. From the Canvas, select the working environment and click Enable > Backup Volumes next to the Backup and recovery service in the right-panel.

    If the Amazon S3 destination for your backups exists as a working environment on the Canvas, you can drag the cluster onto the Amazon S3 working environment to initiate the setup wizard.

    A screenshot that shows the Backup and recovery Enable button which is available after you select a working environment.

  2. Select Amazon Web Services as your provider and click Next.

  3. Enter the provider details and click Next.

    1. The AWS Account, the AWS Access Key, and the Secret Key used to store the backups.

      The access key and secret key are for the IAM user you created to give the ONTAP cluster access to the S3 bucket.

    2. The AWS region where the backups will be stored.

    3. Whether you’ll use the default Amazon S3 encryption keys, or choose your own customer-managed keys from your AWS account, to manage encryption of your data. (See how to use your own keys).

      A screenshot that shows the cloud provider details when backing up volumes from an ONTAP system to AWS S3.

  4. If you don’t have an existing BlueXP backup and recovery license for your account, you’ll be prompted at this point to select the type of charging method that you want to use. You can subscribe to a pay-as-you-go (PAYGO) BlueXP Marketplace offering from AWS (or if you have multiple subscriptions you’ll need to select one), or purchase and activate a BlueXP backup and recovery BYOL license from NetApp. Learn how to set up BlueXP backup and recovery licensing.

  5. Enter the networking details and click Next.

    1. The IPspace in the ONTAP cluster where the volumes you want to back up reside. The intercluster LIFs for this IPspace must have outbound internet access.

    2. Optionally, choose whether you’ll use an AWS PrivateLink that you have previously configured. See details about using AWS PrivateLink for Amazon S3.

      A screenshot that shows the networking details when backing up volumes from an ONTAP system to AWS S3.

  6. Enter the backup policy details that will be used for your default policy and click Next. You can select an existing policy, or you can create a new policy by entering your selections in each section:

    1. Enter the name for the default policy. You don’t need to change the name.

    2. Define the backup schedule and choose the number of backups to retain. See the list of existing policies you can choose.

    3. Optionally, when using ONTAP 9.11.1 and greater, you can choose to protect your backups from deletion and ransomware attacks by configuring one of the DataLock and Ransomware Protection settings. DataLock protects your backup files from being modified or deleted, and Ransomware protection scans your backup files to look for evidence of a ransomware attack in your backup files. Learn more about the available DataLock settings.

    4. Optionally, when using ONTAP 9.10.1 and greater, you can choose to tier backups to either S3 Glacier or S3 Glacier Deep Archive storage after a certain number of days for further cost optimization. Learn more about using archival tiers.

      A screenshot that shows the BlueXP backup and recovery settings where you can choose your schedule and backup retention.

      Important: If you plan to use DataLock, you must enable it in your first policy when activating BlueXP backup and recovery.

  7. Select the volumes that you want to back up using the defined backup policy in the Select Volumes page. If you want to assign different backup policies to certain volumes, you can create additional policies and apply them to those volumes later.

    • To back up all existing volumes and any volumes added in the future, check the box "Back up all existing and future volumes…​". We recommend this option so that all your volumes will be backed up and you’ll never have to remember to enable backups for new volumes.

    • To back up only existing volumes, check the box in the title row (button backup all volumes).

    • To back up individual volumes, check the box for each volume (button backup 1 volume).

      A screenshot of selecting the volumes that will be backed up.

    • If there are any local Snapshot copies for read/write volumes in this working environment that match the backup schedule label you just selected for this working environment (for example, daily, weekly, etc.), an additional prompt is displayed "Export existing Snapshot copies to object storage as backup copies". Check this box if you want all historic Snapshots to be copied to object storage as backup files to ensure the most complete protection for your volumes.

  8. Click Activate Backup and BlueXP backup and recovery starts taking the initial backups of your volumes.

Result

An S3 bucket is created automatically in the service account indicated by the S3 access key and secret key you entered, and the backup files are stored there. The Volume Backup Dashboard is displayed so you can monitor the state of the backups. You can also monitor the status of backup and restore jobs using the Job Monitoring panel.

What’s next?