Operations for multipart uploads

Note: You should not exceed 1,000 concurrent multipart uploads to a single bucket because the results of List Multipart Uploads queries for that bucket might return incomplete results.
Note: Each multipart part except the last must be between 5 MB and 5 GB. The last part can be less than 5 MB. This client must follow these limits as it is not enforced by StorageGRID Webscale.

All of the multipart upload operations support StorageGRID Webscale consistency controls. For information on using the Consistency-Control header, see How StorageGRID Webscale implements the S3 REST API.

Operation Implementation
List Multipart Uploads

The following request headers are supported:

  • encoding-type
  • max-uploads
  • key-marker
  • prefix
  • upload-id-marker

The delimiter request parameter is not supported.

Versioning

Multipart upload consists of separate operations for initiating the upload, listing uploads, uploading parts, assembling the uploaded parts, and completing the upload. When the Complete Multipart Upload operation is performed, that is the point when objects are created (and versioned if applicable).

Initiate Multipart Upload

The x-amz-storage-class request header is supported with the following enumerated values:

  • STANDARD

    Default. Specifies a dual-commit ingest operation.

  • REDUCED_REDUNDANCY

    Specifies a single-commit ingest operation.

Note: The REDUCED_REDUNDANCY storage class is an option used to limit redundant storage for data that is better replicated elsewhere, such as using ILM policies. Therefore, specifying the REDUCED_REDUNDANCY value does not affect the specified ILM policy, and does not result in data being stored at lower levels of redundancy in the StorageGRID Webscale system.
CAUTION:
Be careful when ingesting objects using REDUCED_REDUNDANCY to create only a single initial copy of the object data. If the single copy is created on a Storage Node that fails, and ILM is not yet satisfied, the result is unrecoverable loss of data.

The following request headers are supported:

  • Content-Type
  • x-amz-meta- name-value pairs for user-defined metadata

To record the object creation time, so that you can use the User Defined Creation Time option for the reference time in an ILM rule, you need to store the value in a user-defined header named x-amz-meta-creation-time. For example: x-amz-meta-creation-time=1443399726. This field is evaluated as seconds since Jan 1, 1970. For more information, see "Reference time" in the Administrator Guide.

The x-amz-server-side-encryption header is not supported directly for Initiate Multipart Upload requests. If you require server-side encryption for a multipart upload, you need to specify the x-amz-server-side-encryption header for each of the upload parts, but you cannot specify it as part of the Initiate Multipart Upload header or the request fails.

The following request headers are not supported and return XNotImplemented:

  • x-amz-website-redirect-location
  • x-amz-server-side-encryption-customer-key

Versioning

Multipart upload consists of separate operations for initiating the upload, listing uploads, uploading parts, assembling the uploaded parts, and completing the upload. When the Complete Multipart Upload operation is performed, that is the point when objects are created (and versioned if applicable).

Upload Part

The following request headers are supported:

  • Content-Length
  • x-amz-server-side-encryption

    If you need to specify server-side encryption for a multipart upload, you need to specify the x-amz-server-side-encryption header for each of the individual upload parts.

The following request headers are not supported and return XNotImplemented:

  • x-amz-server-side-encryption-customer-algorithm
  • x-amz-server-side-encryption-customer-key
  • x-amz-server-side-encryption-customer-key-MD5

Versioning

Multipart upload consists of separate operations for initiating the upload, listing uploads, uploading parts, assembling the uploaded parts, and completing the upload. When the Complete Multipart Upload operation is performed, that is the point when objects are created (and versioned if applicable).

Upload Part - Copy

Implemented with all Amazon S3 REST API behavior.

This request reads and writes the object data specified in x-amz-copy-source-range within the StorageGRID Webscale system.

The following request headers are supported:

  • x-amz-copy-source-if-match
  • x-amz-copy-source-if-none-match
  • x-amz-copy-source-if-unmodified-since
  • x-amz-copy-source-if-modified-since

Versioning

Multipart upload consists of separate operations for initiating the upload, listing uploads, uploading parts, assembling the uploaded parts, and completing the upload. When the Complete Multipart Upload operation is performed, that is the point when objects are created (and versioned if applicable).

Complete Multipart Upload

Resolving conflicts

Conflicting client requests, such as two clients writing to the same key, are resolved on a "latest-wins" basis. The timing for the "latest-wins"evaluation is based on when the StorageGRID Webscale system completes a given request, and not on when S3 clients begin an operation.

Request headers

The x-amz-storage-class request header is supported with the following enumerated values:

  • STANDARD

    Default. Specifies a dual-commit ingest operation.

  • REDUCED_REDUNDANCY

    Specifies a single-commit ingest operation.

Note: The REDUCED_REDUNDANCY storage class is an option used to limit redundant storage for data that is better replicated elsewhere, such as with ILM policies. Therefore, specifying the REDUCED_REDUNDANCY value does not affect the specified ILM policy, and it does not result in data being stored at lower levels of redundancy in the StorageGRID Webscale system.
CAUTION:
Be very careful when configuring ILM policies for data configured with REDUCED_REDUNDANCY to replicate only a single copy of object data before the ILM policy is satisfied. If the ILM policy is not yet satisfied, and the single copy is created on a Storage Node that fails, the result is unrecoverable loss of data.
Attention: If a multipart upload is not completed within 15 days, the operation is marked as inactive and all associated data is deleted from the system.
Note: The ETag value returned is not an MD5 sum of the data, but follows the Amazon S3 API implementation of the ETag value for multipart objects.
Complete Multipart Upload (continued)

Versioning

This operation completes a multipart upload. If versioning is enabled for a bucket, the object version is created upon completion of the multipart upload.

If versioning is enabled for a bucket, a unique versionId is automatically generated for the version of the object being stored. This versionId is also returned in the response using the x-amz-version-id response header.

If versioning is suspended, the object version is stored with a null versionId and if a null version already exists it will be overwritten.

Note: When versioning is enabled for a bucket, completing a multipart upload always creates a new version, even if there are concurrent multipart uploads completed on the same object key. When versioning is not enabled for a bucket, it is possible to initiate a multipart upload and then have another multipart upload initiate and complete first on the same object key. On non-versioned buckets, the multipart upload that completes last takes precedence.

Failed replication, notification, or metadata notification

If the bucket where the multipart upload occurs is configured for a platform service, multipart upload succeeds even if the associated replication or notification action fails.

If this occurs, an alarm is raised in the Grid Management Interface on Total Events (SMTT). The Last Event message at Grid > site > Storage Node > SSM > Events displays " Failed to publish notifications for bucket-name object key " for the last object whose notification failed. Event messages are also listed in /var/local/log/bycasterr.log

A tenant can trigger the failed replication or notification by updating the object's metadata or tags. They can resubmit the existing values to avoid making unwanted changes.

Abort Multipart Upload Implemented with all Amazon S3 REST API behavior.
List Parts Implemented with all Amazon S3 REST API behavior.