StorageGRID Webscale platform services can help you implement a hybrid cloud strategy.
If the use of platform services is allowed for your tenant account, you can configure the following services for any S3 bucket:
- CloudMirror replication: The StorageGRID Webscale CloudMirror replication service is used to mirror specific objects from a StorageGRID Webscale bucket to a specified external destination.
For example, you could configure CloudMirror replication to mirror specific customer records placed in one bucket to a sister organization that owns its own instance of StorageGRID Webscale, essentially creating a cross-organizational hybrid grid.
- Notifications: Per-bucket event notifications are used to send notifications about specific actions performed on objects to a specified external Simple Notification Service (SNS).
For example, you could configure alerts to be sent to administrators about each object added to a bucket, where the objects represent log files associated with a critical system event.
- Search integration service: The search integration service is used to send S3 object metadata to a specified Elasticsearch index where the metadata can be searched or analyzed using the external service.
For example, you could configure your buckets to send S3 object metadata to a remote Elasticsearch service. You could then use Elasticsearch to perform searches across buckets, and perform sophisticated analyses of patterns present in your object metadata.
Because the target location for platform services is typically external to your StorageGRID Webscale deployment, platform services give you the power and flexibility that comes from using external storage resources, notification services, and search or analysis services for your data.
Any combination of platform services can be configured for a single S3 bucket. For example, you could configure both the CloudMirror service and notifications on a StorageGRID Webscale S3 bucket so that you can mirror specific objects to the AWS Simple Storage Service™, while sending a notification about each such object to a third party monitoring application to help you track your AWS expenses.
How platform services are configured
Platform services communicate with external endpoints that you configure using the Tenant Management Interface or Tenant Management API. Each endpoint represents an external destination, such as a StorageGRID Webscale S3 bucket, an Amazon Web Services bucket, a Simple Notification Service topic, or an Elasticsearch cluster hosted locally or on AWS.
After you create an endpoint, you can enable a platform service for a bucket by adding XML configuration to the bucket. The XML configuration identifies the objects that the bucket should act on, the action that the bucket should take, and the endpoint that the bucket should use for the service.
You must add separate XML configurations for each platform service that you want to configure. For example, if you want all objects whose keys start with "/images" to be replicated to an AWS S3 bucket, you must add a replication configuration to the source bucket. If you also want to send notifications when these objects are stored to the bucket, you must add a notifications configuration. Finally, if you want to index their object metadata, you would also add the metadata notification configuration that is used to implement search integration.
The format for the configuration XML is governed by the S3 REST APIs used to implement each StorageGRID Webscale platform service:
| Platform service | S3 REST API |
|---|---|
| CloudMirror replication | S3 bucket replication |
| Notifications | S3 bucket notification |
| Search integration | S3 bucket metadata notification (custom to StorageGRID Webscale) |
See the S3 Implementation Guide for more information on how StorageGRID Webscale implements these APIs.
How platform service messages are delivered
When you perform an operation on a bucket that is configured to trigger a platform services message, the message is generated and sent when the operation succeeds. For example, if a bucket is configured for CloudMirror replication, when you successfully store an object in the source bucket, a copy of that object is created and sent to the destination bucket. Delivery is asynchronous. Platform services messages are queued for delivery.
If queues grow too long or if an internal StorageGRID Webscale service responsible for platform services messages fails, operations on the source bucket fail. This failure prevents further undeliverable messages from being generated. If the destination encounters an issue that prevents it from accepting platform services messages — for example, if credentials are updated on the destination such that StorageGRID Webscale can no longer authenticate to the destination service — the operation on the bucket succeeds, but the platform services message is not delivered. In this circumstance, the grid administrator will see a Total Events (SMTT) alarm with an error message that explains the nature of the issue.
All platform services operations are performed on a per-site basis. That is, if you perform an S3 API Create operation on an object by connecting to an API Gateway Node at Data Center Site 1, the notification about that action will be triggered and sent from Data Center Site 1.
Because CloudMirror replications, notifications, and search integration data are sent directly from the site where an operation is performed, a grid administrator must configure networking for each site such that messages can be delivered to the destination services.