Skip to main content

Investigate lost objects

Contributors netapp-pcarriga
PDFs

When the Objects lost alert is triggered, you must investigate immediately. Collect information about the affected objects and contact technical support.

Before you begin
About this task

The Objects lost alert indicates that StorageGRID believes that there are no copies of an object in the grid. Data might have been permanently lost.

Investigate lost object alerts immediately. You might need to take action to prevent further data loss. In some cases, you might be able to restore a lost object if you take prompt action.

Steps

  1. Select NODES.

  2. Select Storage Node > Objects.

  3. Review the number of Lost objects shown in the Object counts table.

    This number indicates the total number of objects this grid node detects as missing from the entire StorageGRID system. The value is the sum of the Lost objects counters of the Data store component within the LDR and DDS services.

    Nodes Storage Nodes Object Page lost object

  4. From an Admin Node, access the audit log to determine the unique identifier (UUID) of the object that triggered the Objects lost alert:

    1. Log in to the grid node:

      1. Enter the following command: ssh admin@grid_node_IP

      2. Enter the password listed in the Passwords.txt file.

      3. Enter the following command to switch to root: su -

      4. Enter the password listed in the Passwords.txt file. When you are logged in as root, the prompt changes from $ to #.

    2. Change to the directory where the audit logs are located. Enter: cd /var/local/log/

      Note

      Learn about the destinations for audit information.

    3. Use grep to extract the Object Lost (OLST) audit messages. Enter: grep OLST audit_file_name

    4. Note the UUID value included in the message.

      >Admin: # grep OLST audit.log
2020-02-12T19:18:54.780426 [AUDT:[CBID(UI64):0x38186FE53E3C49A5][UUID(CSTR):926026C4-00A4-449B-AC72-BCCA72DD1311]
[PATH(CSTR):"source/cats"][NOID(UI32):12288733][VOLI(UI64):3222345986][RSLT(FC32):NONE][AVER(UI32):10]
[ATIM(UI64):1581535134780426][ATYP(FC32):OLST][ANID(UI32):12448208][AMID(FC32):ILMX][ATID(UI64):7729403978647354233]]

  5. Look up the metadata for the lost object by using the UUID:

    1. Select ILM > Object metadata lookup.

    2. Enter the UUID, and select Look Up.

    3. Review the locations in the metadata, and take the appropriate action:

      Metadata Conclusion

      Object <object_identifier> not found

      If the object is not found, the message "ERROR":"" is returned.

      If the object is not found, you can reset the count of Objects lost to clear the alert. The lack of an object indicates that the object was intentionally deleted.

      Locations > 0

      If there are locations listed in the output, the Objects lost alert might be a false positive.

      Confirm that the objects exist. Use the Node ID and filepath listed in the output to confirm that the object file is in the listed location.

      (The procedure for searching for potentially lost objects explains how to use the Node ID to find the correct Storage Node.)

      If the objects exist, you can reset the count of Objects lost to clear the alert.

      Locations = 0

      If there are no locations listed in the output, the object is potentially missing. You can try to search for and restore the object yourself, or you can contact technical support.

      Technical support might ask you to determine if there is a storage recovery procedure in progress. See the information about restoring object data using Grid Manager and restoring object data to a storage volume.