Searching for and restoring potentially lost objects

It might be possible to find and restore objects that have triggered a Lost Objects (LOST) alarm, that you have identified as potentially lost.

1. Search the audit logs for possible object locations:
   1. Log in to the Admin Node as admin and su to root using the password listed in the Passwords.txt file.
   2. Change to the directory where the audit logs are located: cd /var/local/audit/export/
   3. Use grep to extract the audit messages associated with the potentially lost object and send them to an output file. Enter: grep hexadecimal-cbid-value audit_file_name > output_file_name
      For example:

      Admin: # grep 0x2E2C7E93FD5E4ED4 audit.log > messages_about_lost_object.txt

   4. Use grep to extract the Object Rules Met (ORLM) audit messages for the lost object. Enter: grep ORLM output_file_name
      For example:

      Admin: # grep ORLM messages_about_lost_objects.txt

      An ORLM audit message looks like this sample message.

      2017-08-01.txt.gz:2017-08-01T23:13:29.648789 [AUDT:[CBID(UI64):0x2E2C7E93FD5E4ED4][RULE(CSTR):"Make 2 Copies"][STAT(FC32):DONE][CSIZ(UI64):10][SPAR(UI64):0][UUID(CSTR):"2A8F30F0-11D2-49E4-B649-7FA875AC018B"][LOCS(CSTR):"CLDI 12099341, CLDI 12018810"][RSLT(FC32):SUCS][AVER(UI32):10][ATIM(UI64):1501629209648789][ATYP(FC32):ORLM][ANID(UI32):12099341][AMID(FC32):OBDI][ATID(UI64):15559490536956943145]]

   5. Find the LOCS field in the ORLM message.
      If present, the value of CLDI in LOCS is the node ID of the LDR and the volume ID where a copy of the object might be found.
      If you find an object location, you might be able to restore the object.
   6. Find the Storage Node for this LDR node ID.
      There are two ways to use the node ID to find the Storage Node:

      • In the Grid Manager, select Support > Grid Topology. Then select Data Center > Storage Node > LDR. The LDR node ID is in the Node Information table. Review the information for each Storage Node until you find the one that hosts this LDR.
      • Download and unzip the Recovery Package for the grid. There is a \docs directory in the SAID package. If you open the index.html file, the Servers Summary shows all node IDs for all grid nodes.
2. Look for the object on the Storage Node where the audit message indicates it might be found:
   1. Log in to the Storage Node as admin and su to root using the password listed in the Passwords.txt file.
   2. Change directories: cd /var/local/rangedb
   3. Look for the object copy. Enter:find . -name “hexadecimal-cbid-value”
      For example, enter:

      DC-SN1: # find . -name “0x2E2C7E93FD5E4ED4” 

      Note: This find command may take a long time (days) to complete.
      If the object exists on one of the node's storage volumes, the find command returns the file path to the object from your current location. You can use this truncated file path to restore the object.
3. Restore the lost object:
   1. Telnet to localhost 1402 to access the LDR console.
   2. Enter: cd /proc/CMSI
   3. Enter: Object_Found file_path_of_object
      The full file path of the object includes /var/local/rangedb, but use the truncated file path you found in the previous step. For example, enter:

      Object_Found 1/p/12/0E/0x2E2C7E93FD5E4ED4p

      Issuing the Object_Found command notifies the grid of the object's location. It also triggers the active ILM policy, which will make additional copies as specified in the policy.
   Note: If the Storage Node where you found the object is offline, you can copy the object to any Storage Node that is online. Place the object in the lowest sub-directory of any /var/local/rangedb of the online Storage Node. Then issue the Object_Found command using the truncated file path to the object.
4. Reset count of lost objects in the Grid Manager.