Monitoring object metadata capacity per Storage Node

StorageGRID reserves space on storage volume 0 of each Storage Node for the Cassandra database. This database stores three copies of all object metadata as well as certain configuration data. To ensure that adequate space remains available for essential Cassandra operations, you must monitor the metadata attributes for each Storage Node and add new Storage Nodes as required.

Before you begin

You must be signed in to the Grid Manager using a supported browser.

About this task

The total space reserved on each Storage Node for metadata is known as Metadata Reserved Space, or CAWM. CAWM is subdivided into the space available for object metadata (the Metadata Allowed Space, or CEMS) and the space required for essential Cassandra operations, such as compaction and repair.

If object metadata uses more than 100% of the Metadata Allowed Space, Cassandra operations cannot run efficiently and errors will occur. For this reason, you must closely monitor how much object metadata space has been used.

The Metadata Used Space (Percent) attribute, or CDLP, measures how full the Metadata Allowed Space is. When the Metadata Used Space (Percent) reaches certain thresholds, the CDLP alarm is triggered, as follows:

CDLP = 70% (minor alarm): You should add new Storage Nodes as soon as possible.
CDLP = 90% (major alarm): You should add new Storage Nodes immediately.
CDLP = 100% (critical alarm): You must add new Storage Nodes immediately and you must stop the ingest of new objects.

Note that when the Metadata Used Space (Percent) attribute reaches 70% (that is, when the Metadata Allowed Space becomes 70% full), the CDLP alarm is triggered as a minor alarm. You should add new Storage Nodes in an expansion procedure as soon as possible. When you add the new nodes, the system automatically rebalances object metadata across all Storage Nodes, and the alarms clear.

Attention: When the Metadata Used Space (Percent) attribute reaches 90%, the CDLP alarm is triggered as a major alarm, and a warning appears on the Dashboard. If this warning appears, you must add new Storage Nodes immediately. You must never allow object metadata to use more than 100% of the allowed space.

Metadata Dashboard Alarm

Attention: If the first storage volume is smaller than the Metadata Reserved Space, the CDLP calculation might be inaccurate.

Steps

Select Nodes > Storage Node > Storage.
View the Storage Used - Object Metadata graph.
If the Used % (CDLP Metadata Used Space (Percent)) value is 70% or higher, expand the StorageGRID system by adding Storage Nodes.
To view alarm details, select the Alarms tab from the Data Store Overview page, or select Alarms from the top menu bar.
Example

CDLP (Metadata Used Space (Percent)) alarms of all severities (minor, major, and critical) are displayed on this page. In the example, all the alarms listed are critical.
If a major or critical CDLP alarm appears on the Current Alarms page, add Storage Nodes immediately.
When you add the new nodes, the system automatically rebalances object metadata across all Storage Nodes, and the alarms clear.