Back up on-premises ONTAP data to Amazon S3

11/22/2024 Contributors

Complete a few steps to get started backing up volume data from your on-premises ONTAP systems to a secondary storage system and to Amazon S3 cloud storage.

"On-premises ONTAP systems" include FAS, AFF, and ONTAP Select systems.

Quick start

Get started quickly by following these steps. Details for each step are provided in the following sections in this topic.

Identify the connection method you'll use

Choose whether you'll connect your on-premises ONTAP cluster directly to AWS S3 over the public internet, or whether you'll use a VPN or AWS Direct Connect and route traffic through a private VPC Endpoint interface to AWS S3.

Identify the connection method.

Prepare your BlueXP Connector

If you already have a Connector deployed in your AWS VPC or on your premises, then you're all set. If not, then you'll need to create a BlueXP Connector to back up ONTAP data to AWS S3 storage. You'll also need to customize network settings for the Connector so that it can connect to AWS S3.

Learn how to create a Connector and how to define required network settings.

Verify license requirements

You'll need to check license requirements for both AWS and BlueXP.

Refer to Verify license requirements.

Prepare your ONTAP clusters

Discover your ONTAP clusters in BlueXP, verify that the clusters meet minimum requirements, and customize network settings so the clusters can connect to AWS S3.

Learn how to get your ONTAP clusters ready.

Prepare Amazon S3 as your backup target

Set up permissions for the Connector to create and manage the S3 bucket. You'll also need to set up permissions for the on-premises ONTAP cluster so it can read and write data to the S3 bucket.

Optionally, you can set up your own custom-managed keys for data encryption instead of using the default Amazon S3 encryption keys. Learn how to get your AWS S3 environment ready to receive ONTAP backups.

Activate backups on your ONTAP volumes

Select the working environment and click Enable > Backup Volumes next to the Backup and recovery service in the right-panel. Then follow the setup wizard to select the replication and backup policies that you'll use and the volumes you want to back up.

Activate backups on your ONTAP volumes.

Identify the connection method

Choose which of the two connection methods you will use when configuring backups from on-premises ONTAP systems to AWS S3.

Public connection - Directly connect the ONTAP system to AWS S3 using a public S3 endpoint.
Private connection - Use a VPN or AWS Direct Connect and route traffic through a VPC Endpoint interface that uses a private IP address.

Optionally, you can connect to a secondary ONTAP system for replicated volumes using the public or private connection as well.

The following diagram shows the public connection method and the connections that you need to prepare between the components. You can use a Connector that you've installed on your premises, or a Connector that you've deployed in the AWS VPC.

A diagram showing how BlueXP backup and recovery communicates over a public connection with the volumes on the cluster and the AWS S3 storage where the backup files are located.

The following diagram shows the private connection method and the connections that you need to prepare between the components. You can use a Connector that you've installed on your premises, or a Connector that you've deployed in the AWS VPC.

A diagram showing how BlueXP backup and recovery communicates over a private connection with the volumes on the cluster and the AWS S3 storage where the backup files are located.

Prepare your BlueXP Connector

The BlueXP Connector is the main software for BlueXP functionality. A Connector is required to back up and restore your ONTAP data.

Create or switch Connectors

If you already have a Connector deployed in your AWS VPC or on your premises, then you're all set.

If not, then you'll need to create a Connector in one of those locations to back up ONTAP data to AWS S3 storage. You can't use a Connector that's deployed in another cloud provider.

Learn about Connectors
Install a Connector in AWS
Install a Connector in your premises
Install a Connector in an AWS GovCloud region

BlueXP backup and recovery is supported in GovCloud regions when the Connector is deployed in the cloud - not when it's installed in your premises. Additionally, you must deploy the Connector from the AWS Marketplace. You can't deploy the Connector in a Government region from the BlueXP SaaS website.

Prepare Connector networking requirements

Ensure that the following networking requirements are met:

Ensure that the network where the Connector is installed enables the following connections:
- An HTTPS connection over port 443 to the BlueXP backup and recovery service and to your S3 object storage (see the list of endpoints)
- An HTTPS connection over port 443 to your ONTAP cluster management LIF
- Additional inbound and outbound security group rules are required for AWS and AWS GovCloud deployments. See Rules for the Connector in AWS for details.
Ensure that the Connector has permissions to manage the S3 bucket.
If you have a Direct Connect or VPN connection from your ONTAP cluster to the VPC, and you want communication between the Connector and S3 to stay in your AWS internal network (a private connection), you'll need to enable a VPC Endpoint interface to S3. See how to set up a VPC endpoint interface.

Verify license requirements

You'll need to verify license requirements for both AWS and BlueXP:

Before you can activate BlueXP backup and recovery for your cluster, you'll need to either subscribe to a pay-as-you-go (PAYGO) BlueXP Marketplace offering from AWS, or purchase and activate a BlueXP backup and recovery BYOL license from NetApp. These licenses are for your account and can be used across multiple systems.
- For BlueXP backup and recovery PAYGO licensing, you'll need a subscription to the NetApp BlueXP offering from the AWS Marketplace. Billing for BlueXP backup and recovery is done through this subscription.
- For BlueXP backup and recovery BYOL licensing, you'll need the serial number from NetApp that enables you to use the service for the duration and capacity of the license. Learn how to manage your BYOL licenses.
You need to have an AWS subscription for the object storage space where your backups will be located.

Supported regions

You can create backups from on-premises systems to Amazon S3 in all regions where Cloud Volumes ONTAP is supported; including AWS GovCloud regions. You specify the region where backups will be stored when you set up the service.

Prepare your ONTAP clusters

You'll need to prepare your source on-premises ONTAP system and any secondary on-premises ONTAP or Cloud Volumes ONTAP systems.

Preparing your ONTAP clusters involves the following steps:

Discover your ONTAP systems in BlueXP
Verify ONTAP system requirements
Verify ONTAP networking requirements for backing up data to object storage
Verify ONTAP networking requirements for replicating volumes

Discover your ONTAP systems in BlueXP

Both your source on-premises ONTAP system and any secondary on-premises ONTAP or Cloud Volumes ONTAP systems must be available on the BlueXP Canvas.

You'll need to know the cluster management IP address and the password for the admin user account to add the cluster.
Learn how to discover a cluster.

Verify ONTAP system requirements

Ensure that the following ONTAP requirements are met:

Minimum of ONTAP 9.8; ONTAP 9.8P13 and later is recommended.
A SnapMirror license (included as part of the Premium Bundle or Data Protection Bundle).

Note: The "Hybrid Cloud Bundle" is not required when using BlueXP backup and recovery.

Learn how to manage your cluster licenses.
Time and time zone are set correctly. Learn how to configure your cluster time.
If you are going to replicate data, you should verify that the source and destination systems are running compatible ONTAP versions before replicating data.

View compatible ONTAP versions for SnapMirror relationships.

Verify ONTAP networking requirements for backing up data to object storage

You must configure the following requirements on the system that connects to object storage.

For a fan-out backup architecture, configure the following settings on the primary system.
For a cascaded backup architecture, configure the following settings on the secondary system.

The following ONTAP cluster networking requirements are needed:

The cluster requires an inbound HTTPS connection from the Connector to the cluster management LIF.
An intercluster LIF is required on each ONTAP node that hosts the volumes you want to back up. These intercluster LIFs must be able to access the object store.

The cluster initiates an outbound HTTPS connection over port 443 from the intercluster LIFs to Amazon S3 storage for backup and restore operations. ONTAP reads and writes data to and from object storage — the object storage never initiates, it just responds.
The intercluster LIFs must be associated with the IPspace that ONTAP should use to connect to object storage. Learn more about IPspaces.

When you set up BlueXP backup and recovery, you are prompted for the IPspace to use. You should choose the IPspace that these LIFs are associated with. That might be the "Default" IPspace or a custom IPspace that you created.

If you use are using a different IPspace than "Default", then you might need to create a static route to get access to the object storage.

All intercluster LIFs within the IPspace must have access to the object store. If you can't configure this for the current IPspace, then you'll need to create a dedicated IPspace where all intercluster LIFs have access to the object store.
DNS servers must have been configured for the storage VM where the volumes are located. See how to configure DNS services for the SVM.
Update firewall rules, if necessary, to allow BlueXP backup and recovery connections from ONTAP to object storage through port 443 and name resolution traffic from the storage VM to the DNS server over port 53 (TCP/UDP).
If you are using a Private VPC Interface Endpoint in AWS for the S3 connection, then in order for HTTPS/443 to be used, you'll need to load the S3 endpoint certificate into the ONTAP cluster. See how to set up a VPC endpoint interface and load the S3 certificate.
Ensure that your ONTAP cluster has permissions to access the S3 bucket.

Verify ONTAP networking requirements for replicating volumes

If you plan to create replicated volumes on a secondary ONTAP system using BlueXP backup and recovery, ensure that the source and destination systems meet following networking requirements.

On-premises ONTAP networking requirements

If the cluster is in your premises, you should have a connection from your corporate network to your virtual network in the cloud provider. This is typically a VPN connection.
ONTAP clusters must meet additional subnet, port, firewall, and cluster requirements.

Because you can replicate to Cloud Volumes ONTAP or an on-premises systems, review peering requirements for on-premises ONTAP systems. View prerequisites for cluster peering in the ONTAP documentation.

Cloud Volumes ONTAP networking requirements

The instance's security group must include the required inbound and outbound rules: specifically, rules for ICMP and ports 11104 and 11105. These rules are included in the predefined security group.

Prepare Amazon S3 as your backup target

Preparing Amazon S3 as your backup target involves the following steps:

Set up S3 permissions.
(Optional) Create your own S3 buckets. (The service will create buckets for you if you want.)
(Optional) Set up customer-managed AWS keys for data encryption.
(Optional) Configure your system for a private connection using a VPC endpoint interface.

Set up S3 permissions

You'll need to configure two sets of permissions:

Permissions for the Connector to create and manage the S3 bucket.
Permissions for the on-premises ONTAP cluster so it can read and write data to the S3 bucket.

Steps

Ensure that the Connector has the required permissions. For details, see BlueXP policy permissions.

When creating backups in AWS China regions, you need to change the AWS Resource Name "arn" under all Resource sections in the IAM policies from "aws" to "aws-cn"; for example arn:aws-cn:s3:::netapp-backup-*.

When you activate the service, the Backup wizard will prompt you to enter an access key and secret key. These credentials are passed to the ONTAP cluster so that ONTAP can back up and restore data to the S3 bucket. For that, you'll need to create an IAM user with the following permissions.

Refer to the AWS Documentation: Creating a Role to Delegate Permissions to an IAM User.

Details

{
    "Version": "2012-10-17",
     "Statement": [
        {
           "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject",
                "s3:ListBucket",
                "s3:ListAllMyBuckets",
                "s3:GetBucketLocation",
                "s3:PutEncryptionConfiguration"
            ],
            "Resource": "arn:aws:s3:::netapp-backup-*",
            "Effect": "Allow",
            "Sid": "backupPolicy"
        },
        {
            "Action": [
                "s3:ListBucket",
                "s3:GetBucketLocation"
            ],
            "Resource": "arn:aws:s3:::netapp-backup*",
            "Effect": "Allow"
        },
        {
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject",
                "s3:ListAllMyBuckets",
                "s3:PutObjectTagging",
                "s3:GetObjectTagging",
                "s3:RestoreObject",
                "s3:GetBucketObjectLockConfiguration",
                "s3:GetObjectRetention",
                "s3:PutBucketObjectLockConfiguration",
                "s3:PutObjectRetention"
            ],
            "Resource": "arn:aws:s3:::netapp-backup*/*",
            "Effect": "Allow"
        }
    ]
}

Create your own buckets

By default, the service creates buckets for you. Or, if you want to use your own buckets, you can create them before you start the backup activation wizard and then select those buckets in the wizard.

Learn more about creating your own buckets.

If you create your own buckets, you should use a bucket name of “netapp-backup”. If you need to use a custom name, edit the ontapcloud-instance-policy-netapp-backup IAMRole for the existing CVOs and add the following list to the S3 permissions. You need to include “Resource”: “arn:aws:s3:::*” and assign all the necessary permissions that need to be associated with the bucket.

Details

"Action": [
"S3:ListBucket"
"S3:GetBucketLocation"
]
“Resource”: “arn:aws:s3:::*”,
"Effect": "Allow"
},
{
"Action": [
"S3:GetObject",
"S3:PutObject",
"S3:DeleteObject",
"S3:ListAllMyBuckets",
"S3:PutObjectTagging",
"S3:GetObjectTagging",
"S3:RestoreObject",
"S3:GetBucketObjectLockConfiguration",
"S3:GetObjectRetention",
"S3:PutBucketObjectLockConfiguration",
"S3:PutObjectRetention"
]
“Resource”: “arn:aws:s3:::*”,

Set up customer-managed AWS keys for data encryption

If you want to use the default Amazon S3 encryption keys to encrypt the data passed between your on-prem cluster and the S3 bucket, then you are all set because the default installation uses that type of encryption.

If instead you want to use your own customer-managed keys for data encryption rather than using the default keys, then you'll need to have the encryption managed keys already set up before you start the BlueXP backup and recovery wizard. Refer to how to use your own keys.

Configure your system for a private connection using a VPC endpoint interface

If you want to use a standard public internet connection, then all the permissions are set by the Connector and there is nothing else you need to do. This type of connection is shown in the first diagram.

If you want to have a more secure connection over the internet from your on-prem data center to the VPC, there's an option to select an AWS PrivateLink connection in the Backup activation wizard. It's required if you plan to use a VPN or AWS Direct Connect to connect your on-premises system through a VPC Endpoint interface that uses a private IP address. This type of connection is shown in the second diagram.

Steps

Create an Interface endpoint configuration using the Amazon VPC console or the command line. Refer to details about using AWS PrivateLink for Amazon S3.
Modify the security group configuration that's associated with the BlueXP Connector. You must change the policy to "Custom" (from "Full Access"), and you must add the S3 permissions from the backup policy as shown earlier.

If you're using port 80 (HTTP) for communication to the private endpoint, you're all set. You can enable BlueXP backup and recovery on the cluster now.

If you're using port 443 (HTTPS) for communication to the private endpoint, you must copy the certificate from the VPC S3 endpoint and add it to your ONTAP cluster, as shown in the next 4 steps.
Obtain the DNS name of the endpoint from the AWS Console.
Obtain the certificate from the VPC S3 endpoint. You do this by logging into the VM that hosts the BlueXP Connector and running the following command. When entering the DNS name of the endpoint, add “bucket” to the beginning, replacing the “*”:
```
[ec2-user@ip-10-160-4-68 ~]$ openssl s_client -connect bucket.vpce-0ff5c15df7e00fbab-yxs7lt8v.s3.us-west-2.vpce.amazonaws.com:443 -showcerts
```

From the output of this command, copy the data for the S3 certificate (all data between, and including, the BEGIN / END CERTIFICATE tags):

Certificate chain
0 s:/CN=s3.us-west-2.amazonaws.com`
   i:/C=US/O=Amazon/OU=Server CA 1B/CN=Amazon
-----BEGIN CERTIFICATE-----
MIIM6zCCC9OgAwIBAgIQA7MGJ4FaDBR8uL0KR3oltTANBgkqhkiG9w0BAQsFADBG
…
…
GqvbOz/oO2NWLLFCqI+xmkLcMiPrZy+/6Af+HH2mLCM4EsI2b+IpBmPkriWnnxo=
-----END CERTIFICATE-----

Log into the ONTAP cluster CLI and apply the certificate you copied using the following command (substitute your own storage VM name):
```
cluster1::> security certificate install -vserver cluster1 -type server-ca
Please enter Certificate: Press <Enter> when done
```

Activate backups on your ONTAP volumes

Activate backups at any time directly from your on-premises working environment.

A wizard takes you through the following major steps:

Select the volumes that you want to back up
Define the backup strategy
Review your selections

You can also Show the API commands at the review step, so you can copy the code to automate backup activation for future working environments.

Start the wizard

Steps

Access the Activate backup and recovery wizard using one of the following ways:
- From the BlueXP canvas, select the working environment and select Enable > Backup Volumes next to the Backup and recovery service in the right-panel.
  
  If the Amazon S3 destination for your backups exists as a working environment on the Canvas, you can drag the ONTAP cluster onto the Amazon S3 object storage.
- Select Volumes in the Backup and recovery bar. From the Volumes tab, select the Actions icon and select Activate Backup for a single volume (that does not already have replication or backup to object storage already enabled).
The Introduction page of the wizard shows the protection options including local Snapshots, replication, and backups. If you did the second option in this step, the Define Backup Strategy page appears with one volume selected.
Continue with the following options:
- If you already have a BlueXP Connector, you're all set. Just select Next.
- If you don't already have a BlueXP Connector, the Add a Connector option appears. Refer to Prepare your BlueXP Connector.

Select the volumes that you want to back up

Choose the volumes you want to protect. A protected volume is one that has one or more of the following: Snapshot policy, replication policy, backup to object policy.

You can choose to protect FlexVol or FlexGroup volumes; however, you cannot select a mix of these volumes when activating backup for a working environment. See how to activate backup for additional volumes in the working environment (FlexVol or FlexGroup) after you have configured backup for the initial volumes.

You can activate a backup only on a single FlexGroup volume at a time.
The volumes you select must have the same SnapLock setting. All volumes must have SnapLock Enterprise enabled or have SnapLock disabled.

Steps

Note that if the volumes you choose already have Snapshot or replication policies applied, then the policies you select later will overwrite these existing policies.

In the Select Volumes page, select the volume or volumes you want to protect.
- Optionally, filter the rows to show only volumes with certain volume types, styles, and more to make the selection easier.
- After you select the first volume, then you can select all FlexVol volumes (FlexGroup volumes can be selected one at a time only). To back up all existing FlexVol volumes, check one volume first and then check the box in the title row. ().
- To back up individual volumes, check the box for each volume ().
Select Next.

Define the backup strategy

Defining the backup strategy involves setting the following options:

Whether you want one or all of the backup options: local Snapshots, replication, and backup to object storage
Architecture
Local Snapshot policy
Replication target and policy

If the volumes you choose have different Snapshot and replication policies than the policies you select in this step, the existing policies will be overwritten.
Backup to object storage information (provider, encryption, networking, backup policy, and export options).

Steps

In the Define backup strategy page, choose one or all of the following. All three are selected by default:
- Local Snapshots: If you are performing replication or back up to object storage, local Snapshots must be created.
- Replication: Creates replicated volumes on another ONTAP storage system.
- Backup: Backs up volumes to object storage.
Architecture: If you chose replication and backup, choose one of the following flows of information:
- Cascading: Information flows from the primary to the secondary to object storage and from the secondary to object storage.
- Fan out: Information flows from the primary to the secondary and from the primary to object storage.
  
  For details about these architectures, refer to Plan your protection journey.
Local Snapshot: Choose an existing Snapshot policy or create a policy.

To create a custom policy before activating the Snapshot, refer to Create a policy.
To create a policy, select Create new policy and do the following:
- Enter the name of the policy.
- Select up to 5 schedules, typically of different frequencies.
  - For backup-to-object policies, set the DataLock and Ransomware Protection settings. For details on DataLock and Ransomware Protection, refer to Backup-to-object policy settings.
- Select Create.
Replication: Set the following options:
- Replication target: Select the destination working environment and SVM. Optionally, select the destination aggregate or aggregates and prefix or suffix that will be added to the replicated volume name.
- Replication policy: Choose an existing replication policy or create a policy.
  
  To create a custom policy before activating the replication, refer to Create a policy.
  
  To create a policy, select Create new policy and do the following:
  - Enter the name of the policy.
  - Select up to 5 schedules, typically of different frequencies.
  - Select Create.
Back up to Object: If you selected Backup, set the following options:
- Provider: Select Amazon Web Services.
- Provider settings: Enter the provider details and AWS region where the backups will be stored.
  
  The access key and secret key are for the IAM user you created to give the ONTAP cluster access to the S3 bucket.
- Bucket: Either choose an existing S3 bucket or create a new one. Refer to Add S3 buckets.
- Encryption key: If you created a new S3 bucket, enter encryption key information given to you from the provider. Choose whether you'll use the default Amazon S3 encryption keys, or choose your own customer-managed keys from your AWS account, to manage encryption of your data.
If you chose an existing bucket, encryption information is already available, so you don't need to enter it now.
- Networking: Choose the IPspace, and whether you'll be using a Private Endpoint. Private Endpoint is disabled by default.
  1. The IPspace in the ONTAP cluster where the volumes you want to back up reside. The intercluster LIFs for this IPspace must have outbound internet access.
  2. Optionally, choose whether you'll use an AWS PrivateLink that you have previously configured. See details about using AWS PrivateLink for Amazon S3.
- Backup policy: Select an existing backup policy or create a policy.
  
  To create a custom policy before activating the backup, refer to Create a policy.
  
  To create a policy, select Create new policy and do the following:
  - Enter the name of the policy.
  - Select up to 5 schedules, typically of different frequencies.
  - Select Create.
- Export existing Snapshot copies to object storage as backup copies: If there are any local Snapshot copies for volumes in this working environment that match the backup schedule label you just selected for this working environment (for example, daily, weekly, etc.), this additional prompt is displayed. Check this box to have all historic Snapshots copied to object storage as backup files to ensure the most complete protection for your volumes.
Select Next.

Review your selections

This is the chance to review your selections and make adjustments, if necessary.

Steps

In the Review page, review your selections.
Optionally check the box to Automatically synchronize the Snapshot policy labels with the replication and backup policy labels. This creates Snapshots with a label that matches the labels in the replication and backup policies.
Select Activate Backup.

Result

BlueXP backup and recovery starts taking the initial backups of your volumes. The baseline transfer of the replicated volume and the backup file includes a full copy of the primary storage system data. Subsequent transfers contain differential copies of the primary data contained in Snapshot copies.

A replicated volume is created in the destination cluster that will be synchronized with the primary storage volume.

The S3 bucket is created in the service account indicated by the S3 access key and secret key you entered, and the backup files are stored there. The Volume Backup Dashboard is displayed so you can monitor the state of the backups.

You can also monitor the status of backup and restore jobs using the Job Monitoring panel.

Show the API commands

You might want to display and optionally copy the API commands used in the Activate backup and recovery wizard. You might want to do this to automate backup activation in future working environments.

Steps

From the Activate backup and recovery wizard, select View API request.
To copy the commands to the clipboard, select the Copy icon.

What's next?

You can manage your backup files and backup policies. This includes starting and stopping backups, deleting backups, adding and changing the backup schedule, and more.
You can manage cluster-level backup settings. This includes changing the storage keys ONTAP uses to access cloud storage, changing the network bandwidth available to upload backups to object storage, changing the automatic backup setting for future volumes, and more.
You can also restore volumes, folders, or individual files from a backup file to a Cloud Volumes ONTAP system in AWS, or to an on-premises ONTAP system.

Back up on-premises ONTAP data to Amazon S3

Creating your file...

Quick start

Identify the connection method

Prepare your BlueXP Connector

Create or switch Connectors

Prepare Connector networking requirements

Verify license requirements

Prepare your ONTAP clusters

Discover your ONTAP systems in BlueXP

Verify ONTAP system requirements

Verify ONTAP networking requirements for backing up data to object storage

Verify ONTAP networking requirements for replicating volumes

On-premises ONTAP networking requirements

Cloud Volumes ONTAP networking requirements

Prepare Amazon S3 as your backup target

Set up S3 permissions

Create your own buckets

Set up customer-managed AWS keys for data encryption

Configure your system for a private connection using a VPC endpoint interface

Activate backups on your ONTAP volumes

Start the wizard

Select the volumes that you want to back up

Define the backup strategy

Review your selections

Show the API commands

What's next?