How StorageGRID Webscale implements the S3 REST API

Conflicting client requests

Conflicting client requests, such as a two clients writing to the same key, are resolved on a "latest-wins" basis. The timing for the "latest-wins" evaluation is based on when the StorageGRID Webscale system completes a given request, and not on when S3 clients begin an operation.

Consistency guarantees and controls

By default, StorageGRID Webscale guarantees read-after-write consistency for newly created objects. Any GET following a successfully completed PUT will be able to read the newly written data. Overwrites of existing objects, metadata updates, and deletes are eventually consistent. Overwrites generally take seconds or minutes to propagate, but can take up to 15 days.

StorageGRID Webscale allows you to control the consistency control used for each bucket or API request. You can change the consistency control to make a trade-off between the availability of the objects and the consistency of those objects across different Storage Nodes and sites, as required by your application.

For information on configuring consistency controls for buckets, see GET Bucket consistency request and PUT Bucket consistency request.

To set the consistency control for an individual API operation, consistency controls must be supported for the operation, and you must specify the consistency control in the request header. This example sets the consistency control to strong-site for a GET Object operation.

GET /bucket/object HTTP/1.1
Date: Sat, 29 Nov 2015 01:02:17 GMT
Authorization: AWS 9MOYPG9ACWPAJA1S72R5:jUGbYkLdBApjCWBgK4TxvOjfock=
Host: test.com
Consistency-Control: strong-site

You can set the Consistency-Control header to one of the following values:

• all: Provides the highest guarantee of read-after-write consistency. All nodes receive the data immediately, or the request will fail.
• strong-global: Guarantees read-after-write consistency for all client requests across all sites.
• strong-site: Guarantees read-after-write consistency for all client requests within a site.
• default (read-after-write for new): Provides read-after-write consistency for new objects and eventual consistency for object updates. Offers high availability, and data protection guarantees. Matches AWS S3 consistency guarantees.
  Note: If your application attempts HEAD operations on keys that do not exist, set the consistency level to available, unless you require AWS S3 consistency guarantees. Otherwise, a high number of 500 Internal Server errors can result if one or more Storage Nodes are unavailable.
• available (eventual consistency for HEAD operations): Behaves the same as the default consistency level, but only provides eventual consistency for HEAD operations. Offers higher availability for HEAD operations than default if Storage Nodes are unavailable. Differs from AWS S3 consistency guarantees for HEAD operations only.
• weak: Provides eventual consistency and high availability, with minimal data protection guarantees, especially if a Storage Node fails or is unavailable. Suitable only for write-heavy workloads that require high availability, do not require read-after-write consistency, and can tolerate the potential loss of data if a node fails.

The following behavior is applied to operations on the bucket:

• Consistency controls only affect objects within a bucket. Setting the consistency control for a bucket does not impact the bucket itself; instead, it sets the consistency control for the S3 operations performed on the objects in the bucket.
  Note: In general, you should use the default consistency control value. If requests are not working correctly, change the application client behavior if possible. Or, configure the client to specify the consistency control for each API request. Set the consistency control at the bucket level only as a last resort.
• The default behavior of the S3 HEAD operation is to start with a low consistency. If the result of the first HEAD attempt is Not Found, the grid retries the operation with a higher consistency on each retry, until consistency control value all is used. If you use the HEAD operation on a key that does not exist, retries will eventually "bubble up" to the all consistency control, which requires all nodes to be online. In this case, a high number of 500 Internal Server errors can result if one or more Storage Nodes are unavailable. To prevent these errors, change the consistency control to available. This will stop the "bubbling up" of consistency behavior for HEAD operations, although read-after-write consistency on newly created object metadata will no longer be guaranteed.
• The "bubbling up" of consistency behavior also applies to GET operations. However, PUT and DELETE operations do not follow this pattern; they either succeed or fail at the specified consistency control.