Troubleshooting the Low metadata storage alert

If the Low metadata storage alert is triggered, you must add new Storage Nodes.

Before you begin

About this task

StorageGRID reserves a certain amount of space on volume 0 of each Storage Node for object metadata. This space is known as the actual reserved space, and it is subdivided into the space allowed for object metadata (the allowed metadata space) and the space required for essential database operations, such as compaction and repair. The allowed metadata space governs overall object capacity.
Metadata Allowed Space Volume 0

If object metadata consumes more than 100% of the space allowed for metadata, database operations cannot run efficiently and errors will occur.

StorageGRID uses the following Prometheus metric to measure how full the allowed metadata space is:
storagegrid_storage_utilization_metadata_bytes/storagegrid_storage_utilization_metadata_allowed_bytes

When this Prometheus expression reaches certain thresholds, the Low metadata storage alert is triggered.

In the following example, object metadata is using more than 100% of the allowed metadata space. This is a critical situation, which will result in inefficient database operation and errors.


Metadata Dashboard Alarm
Attention: If the size of volume 0 is smaller than the Metadata Reserved Space storage option (for example, in a non-production environment), the calculation for the Low metadata storage alert might be inaccurate.

Procedure

  1. Select Alerts > Current.
  2. From the table of alerts, expand the Low metadata storage alert group, if required, and select the specific alert you want to view.
  3. Review the details in the alert dialog box.
  4. If a major or critical Low metadata storage alert has been triggered, perform an expansion to add Storage Nodes immediately.
    Note: Because StorageGRID keeps complete copies of all object metadata at each site, the metadata capacity of the entire grid is limited by the metadata capacity of the smallest site. If you need to add metadata capacity to one site, you should also expand any other sites by the same number of Storage Nodes.
    After you perform the expansion, StorageGRID redistributes the existing object metadata to the new nodes, which increases the overall metadata capacity of the grid. No user action is required. The Low metadata storage alert is cleared.