Monitoring object metadata capacity for each Storage Node

You must monitor the metadata usage for each Storage Node to ensure that adequate space remains available for essential database operations. You must add new Storage Nodes at each site before object metadata exceeds 100% of the allowed metadata space.

Before you begin

About this task

StorageGRID maintains object metadata for each object or object version that it stores. The metadata for an object or object version can include the following types of information:
  • System metadata, including the object ID (UUID), the object name, the name of the container, the tenant account name or ID, the logical size of the object, the date and time the object was first created, and the date and time the object was last modified.
  • Any custom user metadata key-value pairs associated with the object.
  • For S3 objects, any object tag key-value pairs associated with the object.
  • For replicated object copies, the current storage location of each copy.
  • For erasure-coded object copies, the current storage location of each fragment.
  • For object copies in a Cloud Storage Pool, the location of the object, including the name of the external bucket and the object's unique identifier.
  • For segmented objects and multipart objects, segment identifiers and data sizes.

Three copies of object metadata are maintained at each site to provide redundancy and to protect object metadata from loss. The three copies are evenly distributed across all Storage Nodes at each site using space reserved on storage volume 0 of each Storage Node.

The total space reserved for metadata on volume 0 of each Storage Node is a system-wide setting known as the Metadata Reserved Space. (To see this value, go to Configuration > Storage Options > Overview.)

The Metadata Reserved Space is subdivided into the space available for object metadata (the Metadata Allowed Space) and the space required for essential database operations, such as compaction and repair.
Metadata Allowed Space Volume 0
StorageGRID uses the following Prometheus expression to measure how full the Metadata Allowed Space is:
storagegrid_storage_utilization_metadata_bytes/storagegrid_storage_utilization_metadata_allowed_bytes

When this Prometheus expression reaches certain thresholds, the Low metadata storage alert is triggered.

Note: This Prometheus expression is equivalent to the legacy CDLP (Metadata Used Space (Percent)) attribute, which triggers the legacy CDLP alarm at the same thresholds.

Some of the factors that can increase metadata usage include the size and quantity of user metadata and tags, the total number of parts in an multipart upload, and the frequency of changes to ILM storage locations.

To ensure that adequate space remains for object metadata, follow these steps to monitor the metadata capacity of a Storage Node.

Procedure

  1. Select Nodes > Storage Node > Storage.
  2. Hover your cursor over the Storage Used - Object Metadata graph to see the percentage of allowed space consumed by object metadata.
    The value for Used % is the current value of the Prometheus expression shown above.

    Storage Used - Object Metadata Graph
  3. If the Used % value is 70% or higher, expand the StorageGRID system by adding Storage Nodes.
    When you add the new nodes, the system automatically rebalances object metadata across all Storage Nodes within the site.