As the ingest rate increased, the number of ingests pending and the number of ILM evaluations pending increased because the LDR and CMS services could not keep up with the increased ingest rate.
Trends suggest that something must have changed in the StorageGRID Webscale deployment’s environment to account for the increased number of objects being ingested into the system. Did new client applications start ingesting objects? Is legacy data being migrated to the StorageGRID Webscale system? Data migration can cause queues to build up to a point where operations are disrupted. For guidelines on how to manage data migration, see the Administrator Guide.
Queues should be resolved early by a prompt response to alarms with a severity of Notice. The following figure shows where ingest and replication queues are likely to build up and where alarms are likely to be triggered.
- LDR storage available
If an Insufficient Free Space alarm is triggered, the amount of available space available has dropped below the Swapout No Create Watermark. The Insufficient Free Space alarm can be found on the Grid Management Interface screen.
- Object replication queue
Following ingest, copies of object data are made according to the StorageGRID Webscale system’s ILM policy. During periods of high ingest, queues might form as it can become difficult for the CMS and LDR services to keep up with the required ILM activities.
For LDR services, content replication queues are likely to form on both the source and destination LDR services. The number of Inbound Replications – Queued (RIRQ) on the destination LDR service and of Output Replications – Queued (RORQ) on the source LDR service are likely to increase and failures can trigger alarms. The Inbound Replications – Failed (RIRF) and Outbound Replications – Failed (RORF) alarms should clear after system activity goes back down. The replication attributes can be found for each of the Storage Nodes on the Grid Management Interface screen.
