Monitoring object metadata capacity for each Storage Node

You must monitor the metadata usage for each Storage Node to ensure that adequate space remains available for essential database operations. If the CDLP alarm and the Low metadata storage alert are triggered, you must add new Storage Nodes.

Before you begin

You must be signed in to the Grid Manager using a supported browser.

About this task

StorageGRID maintains three copies of object metadata at each site to provide redundancy and to protect object metadata from loss. The three copies are load balanced across all Storage Nodes at each site using space reserved on storage volume 0 of each Storage Node.

The total space reserved on each Storage Node for metadata is known as the Metadata Reserved Space (CAWM). The Metadata Reserved Space is subdivided into the space available for object metadata (the Metadata Allowed Space, or CEMS) and the space required for essential database operations, such as compaction and repair.

If the space used by object metadata is more than 100% of the Metadata Allowed Space, database operations cannot run efficiently and errors will occur.

The Metadata Used Space (Percent) attribute, or CDLP, measures how full the Metadata Allowed Space is. When the Metadata Used Space (Percent) reaches the following thresholds, the CDLP alarm and the Low metadata storage alert are triggered:

Minor: Object metadata is using 70% or more of the Metadata Allowed Space. You should add new Storage Nodes as soon as possible.
Major: Object metadata is using 90% or more of the Metadata Allowed Space. You must add new Storage Nodes immediately.
Attention: When the CDLP alarm is triggered at the major level, a warning appears on the Dashboard. If this warning appears, you must add new Storage Nodes immediately. You must never allow object metadata to use more than 100% of the allowed space.
Critical: Object metadata is using 100% or more of the Metadata Allowed Space and is starting to consume the space required for essential database operations. You must stop the ingest of new objects, and you must add new Storage Nodes immediately.

When you add the new nodes, the system automatically rebalances object metadata across all Storage Nodes, and the CDLP alarm and the Low metadata storage alert are cleared.

In the following example, object metadata is using more than 100% of the Metadata Allowed Space. This is a critical situation, which will result in inefficient database operation and errors.

Attention: If the first storage volume is smaller than the Metadata Reserved Space (for example, in a non-production environment), the calculation for the CDLP alarm and the Low metadata storage alert might be inaccurate.

Steps

Select Nodes > Storage Node > Storage.
Hover over the Storage Used - Object Metadata graph to see the percentage of allowed space consumed by object metadata.
The value for Used % is the Metadata Used Space (Percent) (CDLP) attribute.
If the Used % value is 70% or higher, expand the StorageGRID system by adding Storage Nodes.
To view details about the CDLP alarm:
1. Select Alarms. Then, in the Alarms section of the menu, select Current Alarms.
2. Review the details for the alarm in the table.
  
  CDLP (Metadata Used Space (Percent)) alarms of all severities (minor, major, and critical) are displayed on this page.
Example

Note: You can also see the CDLP alarm for a Storage Node by selecting Support > Grid Topology > Storage Node > DDS > Data Store > Alarms.
To view details about the Low metadata storage alert:
1. Select Alarms. Then, in the Alerts (Preview) section of the menu, select Alerts.
2. From the table of alerts, expand the Low metadata storage alert group, if required, and select the specific alert you want to view.
3. Review the details in the alert dialog box.
If a major or critical CDLP alarm or Low metadata storage alert has been triggered, perform an expansion to add Storage Nodes immediately.
When you add the new nodes, the system automatically rebalances object metadata across all Storage Nodes, and the alarms clear.