Running foreground verification

Foreground verification enables you to verify the existence of data on a Storage Node. Missing object data might indicate that an issue exists with the underlying storage device.

Before you begin

You have ensured that the following grid tasks are not running:
- Grid Expansion: Add Server (GEXP), when adding a Storage Node
- Storage Node Decommissioning (LDCM) on the same Storage Node
If these grid tasks are running, wait for them to complete or release their lock.
You have ensured that the storage is online. (Select Support. Then, in the Tools section of the menu, select Grid Topology. Then, select Storage Node > LDR > Storage > Overview > Main. Ensure that Storage State - Current is Online.)
You have ensured that the following recovery procedures are not running on the same Storage Node:
- Recovery of a failed storage volume
- Recovery of a Storage Node with a failed system drive
Foreground verification does not provide useful information while recovery procedures are in progress.

About this task

Foreground verification checks for both missing replicated object data and missing erasure-coded object data:

If foreground verification finds large amounts of missing object data, there is likely an issue with the Storage Node's storage that needs to be investigated and addressed.
If foreground verification finds a serious storage error associated with erasure-coded data, it will notify you. You must perform storage volume recovery to repair the error.

You can configure foreground verification to check all of a Storage Node's object stores or only specific object stores.

If foreground verification finds missing object data, the StorageGRID system attempts to replace it. If a replacement copy cannot be made, the LOST (Lost Objects) alarm might be triggered.

Foreground verification generates an LDR Foreground Verification grid task that, depending on the number of objects stored on a Storage Node, can take days or weeks to complete. It is possible to select multiple Storage Nodes at the same time; however, these grid tasks are not run simultaneously. Instead, they are queued and run one after the other until completion. When foreground verification is in progress on a Storage Node, you cannot start another foreground verification task on that same Storage Node even though the option to verify additional volumes might appear to be available for the Storage Node.

If a Storage Node other than the one where foreground verification is being run goes offline, the grid task continues to run until the % Complete attribute reaches 99.99 percent. The % Complete attribute then falls back to 50 percent and waits for the Storage Node to return to online status. When the Storage Node's state returns to online, the LDR Foreground Verification grid task continues until it completes.

Procedure

Select Storage Node > LDR > Verification.
Click Configuration > Main .
Under Foreground Verification, select the check box for each storage volume ID you want to verify.
Click Apply Changes.
Wait until the page auto-refreshes and reloads before you leave the page. Once refreshed, object stores become unavailable for selection on that Storage Node.
An LDR Foreground Verification grid task is generated and runs until it completes, pauses, or is aborted.
Monitor missing objects or missing fragments:
1. Select Storage Node > LDR > Verification.
2. On the Overview tab under Verification Results, note the value of Missing Objects Detected.
  
  Note: The same value is reported as Lost Objects on the Nodes page. Go to Nodes > Storage Node, and select the Objects tab.
  
  If the number of Missing Objects Detected is large (if there are a hundreds of missing objects), there is likely an issue with the Storage Node's storage. Contact technical support.
3. Select Storage Node > LDR > Erasure Coding.
4. On the Overview tab under Verification Results, note the value of Missing Fragments Detected.
  If the number of Missing Fragments Detected is large (if there are a hundreds of missing fragments), there is likely an issue with the Storage Node's storage. Contact technical support.
If foreground verification does not detect a significant number of missing replicated object copies or a significant number of missing fragments, then the storage is operating normally.
Monitor the completion of the foreground verification grid task:
1. Select Support. Then, in the Tools section of the menu, select Grid Topology. Then select site > Admin Node > CMN > Grid Task > Overview > Main.
2. Verify that the foreground verification grid task is progressing without errors.
  Note: A notice-level alarm is triggered on grid task status (SCAS) if the foreground verification grid task pauses.
3. If the grid task pauses with a critical storage error, recover the affected volume and then run foreground verification on the remaining volumes to check for additional errors.
  Attention: If the foreground verification grid task pauses with the message Encountered a critical storage error in volume volID, you must perform the procedure for recovering a failed storage volume. See the recovery and maintenance instructions.

After you finish

If you still have concerns about data integrity, go to LDR > Verification > Configuration > Main and increase the background Verification Rate. Background verification checks the correctness of all stored object data and repairs any issues that it finds. Finding and repairing potential issues as quickly as possible reduces the risk of data loss.