Considerations for Cloud Storage Pools

If you plan to use a Cloud Storage Pool to move objects out of the StorageGRID system, you must review the considerations for configuring and using Cloud Storage Pools.

Information required to create a Cloud Storage Pool

Before you can create a Cloud Storage Pool, you must create the external S3 bucket that you will use for the Cloud Storage Pool. When you create the Cloud Storage Pool in StorageGRID, you must enter the following information about the bucket:

The Uniform Resource Identifier (URI) used to access the S3 bucket
The exact name of the S3 bucket
If an access key is required to access the S3 bucket, the access key ID and the secret access key
Optionally, a custom CA certificate to verify TLS connections to the S3 bucket

S3 permissions required for the Cloud Storage Pool bucket

The bucket policy for the external S3 bucket used for a Cloud Storage Pool must grant StorageGRID permission to move an object to the bucket, get an object's status, restore an object from Glacier storage when required, and more. Ideally, StorageGRID should have full-control access to the bucket (s3:*); however, if this is not possible, the bucket policy must grant the following S3 permissions to StorageGRID:

s3:AbortMultipartUpload
s3:DeleteObject
s3:GetObject
s3:ListBucket
s3:ListBucketMultipartUploads
s3:ListMultipartUploadParts
s3:PutObject
s3:RestoreObject

Considerations for the bucket lifecycle configuration

The movement of objects from StorageGRID to the external S3 bucket specified in the Cloud Storage Pool is controlled by ILM rules and the active ILM policy in StorageGRID. In contrast, the transition of objects from the Cloud Storage Pool to AWS Glacier (or to a storage solution that implements the Glacier storage class) is controlled by the external bucket's lifecycle configuration.

If you want to transition objects from the Cloud Storage Pool, you must create the appropriate lifecycle configuration for the external S3 bucket, and you must use a storage solution that implements the Glacier storage class and supports the S3 POST Object restore API.

For example, suppose you want all objects that are moved from StorageGRID to the Cloud Storage Pool to be transitioned to Glacier storage immediately. You would create a lifecycle configuration rule for the external S3 bucket that specifies a single action (Transition) as follows:

<LifecycleConfiguration>
  <Rule>
    <ID>Transition Rule</ID>
    <Filter>
       <Prefix></Prefix>
    </Filter>
    <Status>Enabled</Status>
    <Transition>
      <Days>0</Days>
      <StorageClass>GLACIER</StorageClass>
    </Transition>
  </Rule>
</LifecycleConfiguration>

This rule would transition all bucket objects to Glacier on the day they were created (that is, on the day they were moved from StorageGRID to the Cloud Storage Pool).

Attention: When configuring bucket lifecycle rules for the external bucket, never use Expiration actions to define when objects expire. Expiration actions cause the external storage system to delete expired objects. If you later attempt to access an expired object from StorageGRID, the deleted object will not be found.

Considerations for the ports used for Cloud Storage Pools

To ensure that the ILM rules can move objects to and from the specified Cloud Storage Pool, you must configure the network or networks that contain your system's Storage Nodes. You must ensure that the following ports can communicate with the Cloud Storage Pool.

By default, Cloud Storage Pools use the following ports:

80: For endpoint URIs that begin with http
443: For endpoint URIs that begin with https

You can specify a different port when you create or edit a Cloud Storage Pool.

If you use a non-transparent proxy server, you must also configure proxy settings to allow messages to be sent to external endpoints, such as an endpoint on the internet.

Considerations for object segmentation

The Segmentation storage option must be enabled in the Grid Manager. This option is enabled by default.

Considerations for AWS S3 costs

When StorageGRID connects to the external Cloud Storage Pool bucket, it issues various S3 requests to monitor connectivity and to ensure it can perform the required operations. These requests might include requests to put, post, copy, and list objects. While some additional AWS costs will be associated with these StorageGRID requests, the overall cost of using a Cloud Storage Pool will be only a small fraction of what you will pay to store the objects in S3.