StorageGRID 11.3 introduces a new Load Balancer service, support for high availability node groups, new alerts functionality, improvements to Cloud Storage Pools, enhancements to object ingest and delete processing, new StorageGRID appliances, and more.
New Load Balancer service
A new Load Balancer service is included on Gateway Nodes and on all Admin Nodes. This service provides Layer 7 load balancing of S3 and Swift traffic from clients to Storage Nodes. The legacy Connection Load Balancer (CLB) service on Gateway Nodes is still supported; however, configuring endpoints for the new Load Balancer service is recommended ().
Administering StorageGRID
High availability groups
You can now create high availability (HA) groups of Admin Nodes and Gateway Nodes (). HA groups use virtual IP addresses to provide active-backup access to Gateway Node or Admin Node services. For example, you can create an HA group of Gateway Nodes and Admin Nodes to provide highly available data connections for S3 and Swift clients. Or, you can create an HA group of Admin Nodes to provide highly available connections to the Grid Manager and the Tenant Manager.
If required, you can achieve an active-active configuration by using round-robin DNS or a third-party load balancer and multiple HA groups.
Administering StorageGRID
Untrusted Client Network feature
You can use the Untrusted Client Network feature to secure the StorageGRID nodes on the Client Network from hostile attacks. The new feature allows you to specify that a given node only accept inbound connections on ports explicitly configured as load balancer endpoints ().
For example, you might want a Gateway Node to refuse all inbound traffic on the Client Network except for HTTPS S3 requests. Or, you might want to enable outbound S3 platform service traffic from a Storage Node, while preventing any inbound connections to that Storage Node on the Client Network.
Administering StorageGRID
New alerts functionality
A new alerts system is available to preview in StorageGRID 11.3. The alerts system is designed to be easier to use and more powerful than the legacy alarms system.
Attention: For StorageGRID 11.3, consider the alerts system to be a supplement to the alarms system, not a replacement for it. You must continue to use the alarms system as your primary tool for detecting and resolving any issues with your system.
Some of the benefits of the new alerts system include the following:
- Multiple alerts of the same type are reported in one email notification to reduce the number of emails received.
- The Alerts page provides a user friendly interface for viewing current problems across your StorageGRID system. You can expand and collapse groups of alerts and sort the listing by severity, location, or time triggered.
- Alerts use intuitive names and descriptions to help you understand quickly what the problem is, and they provide the recommended actions for resolving the alert.
- If you need to temporarily suppress the notifications at one or more severity levels, you can easily silence a specific alert rule for the entire grid, a single site, or a single node.
- You can create custom alert rules to target the specific conditions that are relevant to your situation and provide your own recommended actions. To define the conditions for different alert severities, you create expressions using the Prometheus metrics listed in the Metrics section of the Grid Management API.
Note: As part of this enhancement, the existing alarms and monitoring information was moved from the instructions for administering StorageGRID to the new instructions for monitoring and troubleshooting StorageGRID.
Monitoring and troubleshooting StorageGRID
Enhancements to Cloud Storage Pools
In addition to using a Cloud Storage Pool to tier object data from StorageGRID to an external location, you can now use Cloud Storage Pools for backup. You can also configure more than one Cloud Storage Pool endpoint. Specifically:
- You can now back up an object to a Cloud Storage Pool while one or more copies of that object also exist in StorageGRID.
- You can now create Cloud Storage Pools that connect to Microsoft Azure Blob storage endpoints. You can also create Cloud Storage Pools within the AWS Secret Region.
- You can now create up to 10 Cloud Storage Pools, each with a unique cloud endpoint. However, you cannot store the same object in more than one Cloud Storage Pool at a time.
- You can now use ILM rules to restore objects from a Cloud Storage Pool back to StorageGRID. The ILM process automatically issues the required transition requests for objects that are in a non-retrievable state.
- The lifecycle policy on the external S3 bucket used for a Cloud Storage Pool can now include transitions to S3 Glacier Deep Archive.
Administering StorageGRID
Enhancements to object ingest
When creating an ILM rule, you can now indicate whether you want the rule's placement instructions to be satisfied when the objects are ingested. Previously, StorageGRID used dual commit—it made two interim copies during ingest and evaluated ILM later.
On upgrade, existing ILM rules continue to use dual commit. After upgrade is complete, you can configure ILM rules to use the new ingest behavior by selecting one of these options: Balanced (which attempts to make all required copies during ingest and performs dual commit if that is not possible) or Strict (which fails ingest if StorageGRID cannot immediately make all required copies).
The Balanced and Strict options cannot be used for some types of object placements. In addition, these options are not recommended for use with erasure-coded objects when objects are larger than 4 MB or if the erasure-coding scheme creates more than seven fragments. (That is, only the 2+1, 4+1, 4+2, and 6+1 erasure coding schemes are recommended.)
Administering StorageGRID
Enhancements to object deletion
StorageGRID 11.3 improves delete performance and introduces synchronous deletion, which enables content to be removed from the grid more quickly in response to client requests.
In previous releases, StorageGRID always provided an immediate response to client delete requests and queued object copies for deletion later. With synchronous deletion, StorageGRID attempts to remove all object copies before providing a client response. This change means that clients might sometimes receive a slower response, even though objects are generally being removed more quickly than they were in the past.
In addition, when an S3 versioned object is deleted, StorageGRID now creates a delete marker as the current version of the object. This behavior matches AWS S3 behavior.
Administering StorageGRID
Enhancements to object capacity
StorageGRID 11.3 optimizes database operations and metadata space allocations to increase the grid’s object capacity. These changes significantly increase the number of objects per node that a StorageGRID deployment can support in many circumstances. The exact number depends on factors such as how many times ILM rules change object placements and how much user metadata and tags are stored per object.
As part of these changes, more space is now reserved for metadata on volume 0 of Storage Nodes that have 128 GB or more of RAM. When you upgrade, the size of the metadata reservation is automatically increased to 4 TB for these larger Storage Nodes, unless the Metadata Reserved Space (CAWM) setting has been changed from its default value of 3 TB ().
Administering StorageGRID
Changes to metadata usage reporting
StorageGRID 11.2 and earlier under-reported the amount of metadata used by approximately 10%. After upgrade to StorageGRID 11.3, the reported metadata usage will increase and reflect the actual value. To see the value for used metadata, select , and hover over the Storage Used – Object Metadata graph. A pop-up displays Used (%), Used, and Total (allowed) values.
Monitoring and troubleshooting StorageGRID
Changes to ILM processing for Last Access Time
Changing the Last Access Time for an object no longer adds the object to an ILM queue for immediate processing. Instead, the object's placements are re-evaluated during background ILM processing. If you use Last Access Time as a reference time for an ILM rule, you should check and update the time periods you have specified for object placements. Placements should typically last for more than one month.
Administering StorageGRID
Enhancements to the Tenant Manager
- When configuring identity federation for the Tenant Manager, you can now use LDAP over SSL (LDAPS) to secure communications between StorageGRID and the LDAP server. STARTTLS is still the recommended method for securing identity federation.
- You can now prevent tenants from being able to access the Grid Manager by opening port 9443 on the external firewall and closing port 443. All Grid Manager and internal traffic is rejected on port 9443.
- Several enhancements were made to S3 platform services, including the following:
- Basic HTTP authentication is now supported for connections to Elasticsearch clusters. You specify authentication credentials when creating an endpoint for the search integration platform service.
- The search integration service now includes the bucket's region in the metadata notifications that it sends to an endpoint.
- If you have internet access, you can now access the StorageGRID 11.3 Documentation Center directly from the Tenant Manager ().
Using tenant accounts
Enhancements to S3 REST API support
- Support for server-side encryption with customer-provided keys (SSE-C) has been added to object operations in the StorageGRID S3 REST API.
- S3 clients can now configure a bucket lifecycle to control how long their objects are retained. If an S3 bucket lifecycle exists, the Expiration action in the lifecycle will always override ILM rule settings. As part of this change, support has been added for the S3 DELETE, GET, and PUT Bucket lifecycle operations (Expiration and NoncurrentVersionExpiration actions only) and for the x-amz-expiration response header, which returns expiry-date and rule-id.
- Improvements were made to the validation and processing of the Content-Encoding field.
- Clients can now use the TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 cipher when establishing a Transport Layer Security (TLS) session with StorageGRID.
Implementing S3 client applications
Administering StorageGRID
Enhancements to Swift REST API support
- Clients can now use the TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 cipher when establishing a Transport Layer Security (TLS) session with StorageGRID.
Implementing Swift client applications
Audit message changes
- New fields were added to the SDEL, SGET, SPOS, and SPUT audit messages to provide additional auditing information for operations on S3 object and bucket subresources:
- S3SR: Identifies the S3 subresource being operated on.
- SRCF: Shows the new configuration. Included only in requests that set a non-sensitive subresource configuration:
- PUT Bucket lifecycle
- PUT Bucket policy
- PUT Bucket compliance
- PUT Bucket consistency
- PUT Bucket last access time
- PUT Bucket versioning
- POST Object restore
- CNCH: For requests with a Consistency-Control header, shows the value of the Consistency-Control header.
- Several fields were changed in the following audit messages:
- PUT Bucket compliance: ATYPE was changed from SUPD to SPUT. CMPS was replaced by SRCF, which is the whole request XML body.
- PUT Bucket versioning: VSST was replaced by SRCF. (VSST is still used in audit message where the bucket is versioned.)
- POST Object restore: RRDA (Restore Days) and RRTI (Restore Tier) were replaced by SRCF.
- PUT Object tagging: ATYP was changed from SUPD to SPUT.
Understanding audit messages
Changes to the internal firewall
The firewall service inside of StorageGRID has changed from UFW to nftables, and it has moved to inside the Docker container. This change allows for some firewall ports to be opened only when configured, such as the ports used by the new Load Balancer service. During the upgrade to StorageGRID 11.3, the open ports are reset to the default set.
Note: During the upgrade precheck process, any custom firewall ports that you might have opened are flagged. You must contact technical support before proceeding with the upgrade.
New StorageGRID appliances
- The SG1000 services appliance can operate as a Gateway Node or an Admin Node to provide high availability grid administration and load balancing services.
- The all-flash SGF6024 storage appliance includes 24 flash drives, two EF570 storage controllers, and the same compute controller used for the SG6060 appliance.
- The SG6060 storage appliance can now optionally support one or two 60-drive expansion shelves, which must be installed during initial installation.
SG1000 appliance installation and maintenance
SG6000 appliance installation and maintenance
NAS Bridge enhancements
- You can now define Secondary file systems. If you lose access to the data on the Primary file system, having a Secondary file system on standby allows you to recover the data by performing an
emergency takeover.
- The NAS Bridge API includes new attributes for SMB file systems:
- Mandatory attribute: ntfs_file_attrs. Set this attribute to true for access-control list (ACL) support and to track Windows NT file system (NTFS) style file attributes. Note that these tracking operations decrease performance.
- Optional attributes: allow_write and allow_read. Use these attributes to specify user names or group names to restrict read or write access to the share.
Administering NAS Bridge
Using the NAS Bridge Management API