Recommendations for implementing the S3 REST API

You should follow these recommendations when implementing the S3 REST API for use with StorageGRID Webscale.

Recommendations for HEADs to non-existent objects

If your application routinely checks to see if an object exists at a path where you do not expect the object to actually exist, you should use the "Available" consistency control. For example, you should use the "Available" consistency control if your application HEADs a location before PUT-ing to it.

Otherwise, if the HEAD operation does not find the object, you might receive a high number of 500 Internal Server errors if one or more Storage Nodes are unavailable.

You can set the "Available" consistency control for each bucket using the PUT Bucket consistency request, or you can specify the consistency control in the request header for an individual API operation.

Recommendations for object keys

You should not use random values as the first four characters of object keys. This is in contrast to AWS recommendations for key prefixes. Instead, you should use non-random, non-unique prefixes, such as image.

If you do follow the AWS recommendation to use random and unique characters in key prefixes, you should prefix the object keys with a directory name. That is, use this format:

mybucket/mydir/f8e3-image3132.jpg

Instead of this format:

mybucket/f8e3-image3132.jpg

Recommendations for "range reads"

If the Stored Object Compression grid option is enabled for StorageGRID Webscale, S3 client applications should avoid performing GET Object operations that specify a range of bytes be returned. These "range read" operations are inefficient because StorageGRID Webscale must effectively uncompress the objects to access the requested bytes. GET Object operations that request a small range of bytes from a very large object are especially inefficient; for example, it is very inefficient to read a 10 MB range from a 50 GB compressed object.

If ranges are read from compressed objects, client requests can time out.

Note: If you need to compress objects and your client application must use range reads, increase the read timeout for the application.