Skip to main content

Tiering data from on-premises ONTAP clusters to S3 object storage

Contributors netapp-tonacki netapp-bcammett
PDFs

Free space on your on-prem ONTAP clusters by tiering inactive data to any object storage service which uses the Simple Storage Service (S3) protocol.

At this time, MinIO object storage has been qualified.

Caution

Customers who want to use object stores that are not officially supported as a cloud tier can do so using these instructions. Customers must test and confirm that the object store meets their requirements.

NetApp does not support, nor is liable, for any issues arising from any third-party Object Store Service, specifically where it does not have agreed support arrangements with the third party with whom the product originated. It is acknowledged and agreed that NetApp shall not be liable for any associated damage or otherwise be required to provide support on that third-party product.

Quick start

Get started quickly by following these steps, or scroll down to the remaining sections for full details.

One Prepare to tier data to S3-compatible object storage

You need the following:

  • A source on-prem ONTAP cluster that's running ONTAP 9.8 or later, and a connection over a user-specified port to the destination S3-compatible object storage. Learn how to discover a cluster.

  • The FQDN, Access Key, and Secret Key for the object storage server so that the ONTAP cluster can access the bucket.

  • A Connector installed on your premises.

  • Networking for the Connector that enables an outbound HTTPS connection to the source ONTAP cluster, to the S3-compatible object storage, and to the BlueXP tiering service.

Two Set up tiering

In BlueXP, select an on-prem working environment, click Enable for the Tiering service, and follow the prompts to tier data to S3-compatible object storage.

Three Set up licensing

Pay for BlueXP tiering through a pay-as-you-go subscription from your cloud provider, a NetApp BlueXP tiering bring-your-own-license, or a combination of both:

Requirements

Verify support for your ONTAP cluster, set up your networking, and prepare your object storage.

The following image shows each component and the connections that you need to prepare between them:

An architecture image that shows the BlueXP tiering service with a connection to the Connector on your premises, the Connector with a connection to your ONTAP cluster, and a connection between the ONTAP cluster and object storage. Active data resides on the ONTAP cluster, while inactive data resides in object storage.

Note Communication between the Connector and the S3-compatible object storage server is for object storage setup only.

Preparing your ONTAP clusters

Your source ONTAP clusters must meet the following requirements when tiering data to S3-compatible object storage.

Supported ONTAP platforms

You can tier data from AFF systems, or FAS systems with all-SSD aggregates or all-HDD aggregates.

Supported ONTAP version

ONTAP 9.8 or later

Cluster networking requirements

  • The ONTAP cluster initiates an HTTPS connection over a user-specified port to S3-compatible object storage (the port is configurable during tiering setup).

    The source ONTAP system reads and writes data to and from object storage. The object storage never initiates, it just responds.

  • An inbound connection is required from the Connector, which must reside on your premises.

    A connection between the cluster and the BlueXP tiering service is not required.

  • An intercluster LIF is required on each ONTAP node that hosts the volumes you want to tier. The LIF must be associated with the IPspace that ONTAP should use to connect to object storage.

    When you set up data tiering, BlueXP tiering prompts you for the IPspace to use. You should choose the IPspace that each LIF is associated with. That might be the "Default" IPspace or a custom IPspace that you created. Learn more about LIFs and IPspaces.

Supported volumes and aggregates

The total number of volumes that BlueXP tiering can tier might be less than the number of volumes on your ONTAP system. That's because volumes can't be tiered from some aggregates. Refer to the ONTAP documentation for functionality or features not supported by FabricPool.

Tip BlueXP tiering supports both FlexVol and FlexGroup volumes.

Discovering an ONTAP cluster

You need to create an on-prem ONTAP working environment in the BlueXP Canvas before you can start tiering cold data.

Learn how to discover a cluster.

Preparing S3-compatible object storage

S3-compatible object storage must meet the following requirements.

S3 credentials

When you set up tiering to S3-compatible object storage, you're prompted to create an S3 bucket or to select an existing S3 bucket. You need to provide BlueXP tiering with an S3 access key and secret key. BlueXP tiering uses the keys to access your bucket.

These access keys must be associated with a user who has the following permissions:

"s3:ListAllMyBuckets",
"s3:ListBucket",
"s3:GetObject",
"s3:PutObject",
"s3:DeleteObject",
"s3:CreateBucket"

Creating or switching Connectors

A Connector is required to tier data to the cloud. When tiering data to S3-compatible object storage, a Connector must be available on your premises. You'll either need to install a new Connector or make sure that the currently selected Connector resides on-prem.

Preparing networking for the Connector

Ensure that the Connector has the required networking connections.

Steps

  1. Ensure that the network where the Connector is installed enables the following connections:

    • An HTTPS connection over port 443 to the BlueXP tiering service (see the list of endpoints)

    • An HTTPS connection over port 443 to S3-compatible object storage

    • An HTTPS connection over port 443 to your ONTAP cluster management LIF

Tiering inactive data from your first cluster to S3-compatible object storage

After you prepare your environment, start tiering inactive data from your first cluster.

What you'll need

  • An on-premises working environment.

  • The FQDN of the S3-compatible object storage server and the port that will be used for HTTPS communications.

  • An access key and secret key that has the required S3 permissions.

Steps

  1. Select the on-prem ONTAP working environment.

  2. Click Enable for the Tiering service from the right panel.

    A screenshot that shows the Tiering option that appears on the right side of the screen after you select an on-prem ONTAP working environment.

  3. Define Object Storage Name: Enter a name for this object storage. It must be unique from any other object storage you may be using with aggregates on this cluster.

  4. Select Provider: Select S3 Compatible and click Continue.

  5. Complete the steps on the Create Object Storage pages:

    1. Server: Enter the FQDN of the S3-compatible object storage server, the port that ONTAP should use for HTTPS communication with the server, and the access key and secret key for an account that has the required S3 permissions.

    2. Bucket: Add a new bucket or select an existing bucket and click Continue.

    3. Cluster Network: Select the IPspace that ONTAP should use to connect to object storage and click Continue.

      Selecting the correct IPspace ensures that BlueXP tiering can set up a connection from ONTAP to your S3-compatible object storage.

      You can also set the network bandwidth available to upload inactive data to object storage by defining the "Maximum transfer rate". Select the Limited radio button and enter the maximum bandwidth that can be used, or select Unlimited to indicate that there is no limit.

  6. On the Success page click Continue to set up your volumes now.

  7. On the Tier Volumes page, select the volumes that you want to configure tiering for and click Continue:

    • To select all volumes, check the box in the title row () and click Configure volumes.

    • To select multiple volumes, check the box for each volume () and click Configure volumes.

    • To select a single volume, click the row (or edit pencil icon icon) for the volume.

      A screenshot that shows how to select a single volume, multiple volumes, or all volumes, and the modify selected volumes button.

  8. In the Tiering Policy dialog, select a tiering policy, optionally adjust the cooling days for the selected volumes, and click Apply.

    Learn more about volume tiering policies and cooling days.

    A screenshot that shows the configurable tiering policy settings.

Result

You've successfully set up data tiering from volumes on the cluster to S3-compatible object storage.

What's next?

Be sure to subscribe to the BlueXP tiering service.

You can review information about the active and inactive data on the cluster. Learn more about managing your tiering settings.

You can also create additional object storage in cases where you may want to tier data from certain aggregates on a cluster to different object stores. Or if you plan to use FabricPool Mirroring where your tiered data is replicated to an additional object store. Learn more about managing object stores.