Running foreground verification

Foreground verification enables you to verify the existence of data on a Storage Node. Missing object data might indicate that an issue exists with the underlying storage device.

Before you begin

About this task

Foreground verification checks for both missing replicated object data and missing erasure coded object data:
  • If foreground verification finds large amounts of missing object data, there is likely an issue with the Storage Node's storage that needs to be investigated and addressed.
  • If foreground verification finds a serious storage error associated with erasure coded data, it will notify you. You must perform storage volume recovery to repair the error.

You can configure foreground verification to check all of a Storage Node's object stores or only specific object stores.

If foreground verification finds missing object data, the StorageGRID Webscale system attempts to replace it. If a replacement copy cannot be made, the LOST (Lost Objects) alarm might be triggered.

Foreground verification generates an LDR Foreground Verification grid task that, depending on the number of objects stored on a Storage Node, can take days or weeks to complete. It is possible to select multiple Storage Nodes at the same time; however, these grid tasks are not run simultaneously. Instead, they are queued and run one after the other until completion. When foreground verification is in progress on a Storage Node, you cannot start another foreground verification task on that same Storage Node even though the option to verify additional volumes might appear to be available for the Storage Node.

If a Storage Node other than the one where foreground verification is being run goes offline, the grid task continues to run until the % Complete attribute reaches 99.99 percent. The % Complete attribute then falls back to 50 percent and waits for the Storage Node to return to online status. When the Storage Node's state returns to online, the LDR Foreground Verification grid task continues until it completes.

Steps

  1. Select Storage Node > LDR > Verification.
  2. Click Configuration > Main .
  3. Under Foreground Verification, select the check box for each storage volume ID you want to verify.

    Foreground Verification Configuration page
  4. Click Apply Changes.

    Wait until the page auto-refreshes and reloads before you leave the page. Once refreshed, object stores become unavailable for selection on that Storage Node.

    An LDR Foreground Verification grid task is generated and runs until it completes, pauses, or is aborted.
  5. Monitor missing objects or missing fragments:
    1. Select Storage Node > LDR > Verification.
    2. On the Overview tab under Verification Results, note the value of Missing Objects Detected.
      If the count for the attribute Missing Objects Detected is large (if there are a hundreds of missing objects), there is likely an issue with the Storage Node's storage. In this case, cancel foreground verification by aborting the Foreground Verification grid task, resolve the storage issue, and then rerun foreground verification for the Storage Node.
    3. Select Storage Node > LDR > Erasure Coding.
    4. On the Overview tab under Verification Results, note the value of Missing Fragments Detected.
      If the count for the attribute Missing Fragments Detected is large (if there are a hundreds of missing fragments), there is likely an issue with the Storage Node's storage. In this case, cancel foreground verification by aborting the Foreground Verification grid task, resolve the storage issue, and then rerun foreground verification for the Storage Node.

    If foreground verification does not detect a significant number of missing replicated object copies or a significant number of missing fragments, then the storage is operating normally.

  6. Monitor the completion of the foreground verification grid task:
    1. Select Support > Grid Topology. Then select site > Admin Node > CMN > Grid Task > Overview > Main.
    2. Verify that the foreground verification grid task is progressing without errors.
      Note: A notice-level alarm is triggered on grid task status (SCAS) if the foreground verification grid task pauses.
    3. If the grid task pauses with a critical storage error, recover the affected volume and then run foreground verification on the remaining volumes to check for additional errors.
      Attention: If the foreground verification grid task pauses with the message Encountered a critical storage error in volume volID you must perform the procedure for recovering a failed storage volume. See the recovery and maintenance instructions.

After you finish

If you still have concerns about data integrity, go to LDR > Verification > Configuration > Main and increase the background Verification Rate. Background verification checks the correctness of all stored object data and repairs any issues that it finds. Finding and repairing potential issues as quickly as possible reduces the risk of data loss.