Monitoring object metadata capacity for each Storage Node

You must monitor the metadata usage for each Storage Node to ensure that adequate space remains available for essential database operations. If the CDLP alarm and the Low metadata storage alert are triggered, you must add new Storage Nodes.

Before you begin

About this task

StorageGRID maintains three copies of object metadata at each site to provide redundancy and to protect object metadata from loss. The three copies are load balanced across all Storage Nodes at each site using space reserved on storage volume 0 of each Storage Node.

The total space reserved on each Storage Node for metadata is known as the Metadata Reserved Space (CAWM). The Metadata Reserved Space is subdivided into the space available for object metadata (the Metadata Allowed Space, or CEMS) and the space required for essential database operations, such as compaction and repair.
Metadata Reserved Space

If the space used by object metadata is more than 100% of the Metadata Allowed Space, database operations cannot run efficiently and errors will occur.

The Metadata Used Space (Percent) attribute, or CDLP, measures how full the Metadata Allowed Space is. When the Metadata Used Space (Percent) reaches the following thresholds, the CDLP alarm and the Low metadata storage alert are triggered:

When you add the new nodes, the system automatically rebalances object metadata across all Storage Nodes, and the CDLP alarm and the Low metadata storage alert are cleared.

In the following example, object metadata is using more than 100% of the Metadata Allowed Space. This is a critical situation, which will result in inefficient database operation and errors.


Metadata Dashboard Alarm
Attention: If the first storage volume is smaller than the Metadata Reserved Space (for example, in a non-production environment), the calculation for the CDLP alarm and the Low metadata storage alert might be inaccurate.

Steps

  1. Select Nodes > Storage Node > Storage.
  2. Hover over the Storage Used - Object Metadata graph to see the percentage of allowed space consumed by object metadata.
    The value for Used % is the Metadata Used Space (Percent) (CDLP) attribute.


    Storage Used - Object Metadata Graph

  3. If the Used % value is 70% or higher, expand the StorageGRID system by adding Storage Nodes.
  4. To view details about the CDLP alarm:
    1. Select Alarms. Then, in the Alarms section of the menu, select Current Alarms.
    2. Review the details for the alarm in the table.

      CDLP (Metadata Used Space (Percent)) alarms of all severities (minor, major, and critical) are displayed on this page.

    Example


    Metadata Used Space Alarm (CDLP)

    Note: You can also see the CDLP alarm for a Storage Node by selecting Support > Grid Topology > Storage Node > DDS > Data Store > Alarms.
  5. To view details about the Low metadata storage alert:
    1. Select Alarms. Then, in the Alerts (Preview) section of the menu, select Alerts.
    2. From the table of alerts, expand the Low metadata storage alert group, if required, and select the specific alert you want to view.
    3. Review the details in the alert dialog box.
  6. If a major or critical CDLP alarm or Low metadata storage alert has been triggered, perform an expansion to add Storage Nodes immediately.
    When you add the new nodes, the system automatically rebalances object metadata across all Storage Nodes, and the alarms clear.