Skip to main content

Investigate potentially lost objects

Contributors netapp-lhalbert netapp-pcarriga
PDFs

When the Objects potentially lost alert is triggered, you must investigate immediately. Collect information about the affected objects and contact technical support.

Before you begin
About this task

The Objects potentially lost alert indicates that, according to the available information in StorageGRID, there are no copies of an object in the grid. Data might have been permanently lost.

Investigate lost object alerts immediately. You might need to take action to prevent further data loss. In some cases, you might be able to restore a lost object if you take prompt action.

Caution If more than 10 objects are reported lost, contact technical support. Don't follow this procedure yourself.
Steps

  1. Select Nodes.

  2. Select Storage Node > Objects.

  3. Review the number of Lost objects shown in the Object counts table.

    This number indicates the total number of objects this grid node detects as missing from the entire StorageGRID system. The value is the sum of the Lost objects counters of the Data store component within the LDR and DDS services.

  4. From an Admin Node, access the audit log to determine the unique identifier (UUID) of the object that triggered the Objects potentially lost alert:

    1. Log in to the grid node:

      1. Enter the following command: ssh admin@grid_node_IP

      2. Enter the password listed in the Passwords.txt file.

      3. Enter the following command to switch to root: su -

      4. Enter the password listed in the Passwords.txt file. When you are logged in as root, the prompt changes from $ to #.

    2. Change to the directory where the audit logs are located.

      The audit log directory and applicable nodes depend on your audit destination settings.

      Option Destination

      Local nodes (default)

      /var/local/log/localaudit.log

      Admin Nodes/local nodes

      • Admin Nodes (primary and non-primary): /var/local/audit/export/audit.log

      • All nodes: The /var/local/log/localaudit.log file is typically empty or missing in this mode.

      External syslog server

      /var/local/log/localaudit.log

      Depending on your audit destination settings, enter: cd /var/local/log or /var/local/audit/export/

      To learn more, refer to Select log location.

    3. Use grep to extract the Object Lost (OLST) audit messages. Enter: grep OLST audit_file_name

    4. Note the UUID value included in the message.

      Admin: # grep OLST audit.log
2020-02-12T19:18:54.780426 [AUDT:[CBID(UI64):0x38186FE53E3C49A5][UUID(CSTR):"926026C4-00A4-449B-AC72-BCCA72DD1311"]
[PATH(CSTR):"source/cats"][NOID(UI32):12288733][VOLI(UI64):3222345986][RSLT(FC32):NONE][AVER(UI32):10]
[ATIM(UI64):1581535134780426][ATYP(FC32):OLST][ANID(UI32):12448208][AMID(FC32):ILMX][ATID(UI64):7729403978647354233]]

  5. Look up the metadata for the lost object by using the UUID:

    1. Select ILM > Object metadata lookup.

    2. Enter the UUID, and select Look Up.

    3. Review the locations in the metadata, and take the appropriate action:

      Metadata Conclusion

      Object <object_identifier> not found

      If the object isn't found, the message "ERROR":"" is returned.

      If the object isn't found, reset the Objects potentially lost counter to clear the alert. The lack of an object indicates that the object was intentionally deleted.

      Locations > 0

      If there are locations listed in the output, the Objects potentially lost alert might be a false positive.

      Confirm that the objects exist. Use the Node ID and filepath listed in the output to confirm that the object file is in the listed location.

      If the objects exist, reset the Objects potentially lost counter to clear the alert.

      Locations = 0

      If there are no locations listed in the output, the object is potentially missing. Contact technical support.

      Technical support might ask you to determine if there is a storage recovery procedure in progress. See the information about restoring object data using Grid Manager and restoring object data to a storage volume.

  6. After you address lost object issues, reset the Objects potentially lost counter to ensure alerts aren't false positives:

    1. Select Nodes.

    2. Select Storage Node > Tasks.

    3. In the Reset Objects potentially lost counter section, select Reset.