Troubleshooting the Low metadata storage alert

If the Low metadata storage alert or the legacy CDLP alarm is triggered, you must add new Storage Nodes.

About this task

StorageGRID reserves a certain amount of space on each Storage Node for metadata. This space is known as the Metadata Reserved Space (CAWM). The Metadata Reserved Space is subdivided into the space available for object metadata (the Metadata Allowed Space) and the space required for essential database operations, such as compaction and repair.

If object metadata consumes more than 100% of the Metadata Allowed Space, database operations cannot run efficiently and errors will occur.

StorageGRID uses the following Prometheus metric to measure how full the Metadata Allowed Space is:

storagegrid_storage_utilization_metadata_bytes/storagegrid_storage_utilization_metadata_allowed_bytes

When this Prometheus expression reaches certain thresholds, the Low metadata storage alert is triggered.

Note: This Prometheus expression is equivalent to the legacy CDLP (Metadata Used Space (Percent)) attribute, which triggers the legacy CDLP alarm at the same thresholds.

• Minor: Object metadata is using 70% or more of the Metadata Allowed Space. You should add new Storage Nodes as soon as possible.
• Major: Object metadata is using 90% or more of the Metadata Allowed Space. You must add new Storage Nodes immediately.
  Attention: When the legacy CDLP alarm is triggered at the major level, a warning appears on the Dashboard. If this warning appears, you must add new Storage Nodes immediately. You must never allow object metadata to use more than 100% of the allowed space.
• Critical: Object metadata is using 100% or more of the Metadata Allowed Space and is starting to consume the space required for essential database operations. You must stop the ingest of new objects, and you must add new Storage Nodes immediately.

When you add the new nodes, StorageGRID evenly distributes object metadata across all Storage Nodes at each site to increase the overall metadata capacity of the grid. No other user action is required. The Low metadata storage alert (and the legacy CDLP alarm) are cleared.

Note: Because StorageGRID keeps all object metadata at every site, the metadata capacity of the entire grid is limited by the metadata capacity of the smallest site. If you need to add metadata capacity to one site, you should also expand any other sites by adding the same number of Storage Nodes.

In the following example, object metadata is using more than 100% of the Metadata Allowed Space. This is a critical situation, which will result in inefficient database operation and errors.

Attention: If the first storage volume is smaller than the Metadata Reserved Space (for example, in a non-production environment), the calculation for the Low metadata storage alert (and the legacy CDLP alarm) might be inaccurate.

1. To view details about the Low metadata storage alert:
   1. Select Alerts > Current.
   2. From the table of alerts, expand the Low metadata storage alert group, if required, and select the specific alert you want to view.
   3. Review the details in the alert dialog box.
2. To view details about the legacy CDLP alarm:
   1. Select Support. Then, in the Alarms (legacy) section of the menu, select Current Alarms.
   2. Review the details for the alarm in the table.

      CDLP (Metadata Used Space (Percent)) alarms of all severities (minor, major, and critical) are displayed on this page.

   
   
   
   Note: You can also see the CDLP alarm for a Storage Node by selecting Support > Grid Topology > Storage Node > DDS > Data Store > Alarms.
3. If a major or critical Low metadata storage alert (or legacy CDLP alarm) has been triggered, perform an expansion to add Storage Nodes immediately.
   When you add the new nodes, the system automatically rebalances object metadata across all Storage Nodes at each site, and the alerts and alarms clear.