Restoring object data to a storage volume

After recovering a storage volume on a Storage Node where the system drive also failed and was recovered, you can restore object data to the recovered storage volume from other Storage Nodes and Archive Nodes.

Before you begin

You must run the repair-data commands while logged in on the primary Admin Node.
You must have acquired the Volume ID of the restored storage volume. Select Grid > Storage Node > LDR > Storage > Overview > Main.
You must have confirmed that the recovered Storage Node displays in green in the Grid Topology tree and shows all services as Online.

About this task

To correctly recover failed storage volumes, you need to know both the device names of the failed storage volumes and their volume IDs.

At installation, each storage device is assigned a file system universal unique identifier (UUID) and is mounted to a rangedb directory on the Storage Node using that assigned file system UUID. The file system UUID and the rangedb directory are listed in the /etc/fstab file. The device name, rangedb directory, and the size of the mounted volume are displayed in the StorageGRID Webscale system at Grid > site > failed Storage Node > SSM > Resources > Overview > Main.

In the following example, device /dev/sdb has a volume size of 4 TB, is mounted to /var/local/rangedb/0, using the device name /dev/disk/by-uuid/822b0547-3b2b-472e-ad5e-e1cf1809faba in the /etc/fstab file:
Volume size sample

Using the repair-data script

To restore object data, you run the repair-data script. This script begins the process of restoring object data and works with ILM scanning to ensure that ILM rules are met. You use different options with the repair-data script to restore object data that is protected using object replication and data that is procected using erasure coding.

Note: You can enter repair-data --help at the Primary Admin Node command line for more information on using the repair-data script.

Replicated data

The repair-data start-replicated-node-repair and repair-data start-replicated-volume-repair commands are used for restoring replicated data. Although Cassandra inconsistencies might be present and failed repairs are not tracked, you can monitor replicated repair status to a degree. Use a combination of the following attributes to monitor repairs and to determine as well as possible if replicated repairs are complete:

Use the Repairs Attempted (XRPA) attribute (Summary Attribute page > ILM Activity) to track the progress of replicated repairs. This attribute increases each time an LDR tries to repair a high-risk object. When this attribute does not increase for a period longer than the current scan period (provided by the Scan Period – Estimated attribute), it means that ILM scanning found no high-risk objects that need to be repaired on any nodes.
Note: High-risk objects are objects that are at risk of being completely lost. This does not include objects that do not satisfy their ILM configuration.
Use the Scan Period – Estimated (XSCM) attribute to estimate when a policy change will be applied to previously ingested objects. If the Repairs Attempted attribute does not increase for a period longer than the current scan period, it is probable that replicated repairs are done. Note that the scan period can change. The Scan Period – Estimated (XSCM) attribute is at the Summary level and is the maximum of all node scan periods. You can query the Scan Period – Estimated attribute history at the Summary level to determine an appropriate timeframe for your grid.

Erasure coded (EC) data

The repair-data start-ec-node-repair and repair-data start-ec-volume-repair commands are used for restoring erasure coded data. Repairs of erasure coded data can begin while some Storage Nodes are offline. Repair will complete after all nodes are available. You can track repairs of erasure coded data by using the repair-data show-ec-repair-status command.

Notes on data recovery

If the only remaining copy of object data is located on an Archive Node, object data is retrieved from the Archive Node. Due to the latency associated with retrievals from external archival storage systems, restoring object data to a Storage Node from an Archive Node takes longer than restoring copies from other Storage Nodes.

Note: If the ILM policy is configured to replicate a single copy only, and those objects are lost, they cannot be recovered. You must still perform the "Restoring object data to a storage volume" procedure to purge lost object information from the database. For more information about ILM policies, see the Administrator Guide.

Steps

From the service laptop, log in to the primary Admin Node:
1. Enter the following command: ssh admin@primary_Admin_Node_IP
2. Enter the password listed in the Passwords.txt file.
3. Enter the following command to switch to root: su -
4. Enter the password listed in the Passwords.txt file.
  When you are logged in as root, the prompt changes from $ to #.
Use the /etc/hosts file to find the host name of the Storage Node for the restored storage volumes. To see a list of all nodes in the grid, enter the following: cat /etc/hosts
If all storage volumes have failed, use the repair-data node-repair command to repair the entire node.
Run both of the following commands if your grid has both replicated and erasure coded data.
- Replicated data: Use the repair-data start-replicated-node-repair command with the --nodes option to repair the entire Storage Node.
  The following command uses the sample node name SG-DC-SN3:
  repair-data start-replicated-node-repair --nodes SG-DC-SN3
- Erasure coded data: Use the repair-data start-ec-node-repair command with the --nodes option to repair the entire Storage Node.
  The following command uses the sample node name SG-DC-SN3:
  repair-data start-ec-node-repair --nodes SG-DC-SN3
  The operation returns a unique repair ID that identifies this repair_data operation. Use this repair ID to track the progress and result of the repair_data operation. No other feedback is returned as the recovery process completes.
  
  Note: Repairs of erasure coded data can begin while some Storage Nodes are offline. Repair will complete after all nodes are available.
Note: You cannot run repair-data operations for more than one node at the same time. To recover multiple nodes, contact technical support.

As object data is restored, if the StorageGRID Webscale system cannot locate replicated object data, the LOST (Lost Objects) alarm triggers. Alarms may be triggered on Storage Nodes throughout the system. Action should be taken to determine the cause of the loss and if recovery is possible. For more information, see the Troubleshooting Guide.
If only some of the volumes have failed, use the repair-datavolume-repair command to repair one volume or a range of volumes.
Enter the volume IDs in hexadecimal, where 0000 is the first volume and 000F is the sixteenth volume. You can specify one volume, or a range of volumes.

Run both of the following commands if your grid has both replicated and erasure coded data.
- Replicated data: Use the start-replicated-volume-repair command with the --nodes and --volume-range options.
  The following command uses the sample node name SG-DC-SN3 and restores object data to all volumes in the range 0003 to 000B:
  repair-data start-replicated-volume-repair --nodes SG-DC-SN3 --volume-range 0003,000B
  For replicated data, you can run more than one repair-data operation at the same time for the same node. You might want to do this if you need to restore two volumes that are not in a range, such as 0000 and 000A.
- Erasure coded data: Use the start-ec-volume-repair command with the --nodes and --volume-range options.
  The following command uses the sample node name SG-DC-SN3 and restores data to a single volume 000A:
  repair-data start-ec-volume-repair --nodes SG-DC-SN3 --volume-range 000A
  For erasure coded data, you must wait until one repair-data start ec-volume-repair operation completes before starting a second repair-data operation for the same node.
  
  The repair-data operation returns a unique repair ID that identifies this repair_data operation. Use this repair ID to track the progress and result of the repair_data operation. No other feedback is returned as the recovery process completes.
  Note: Repairs of erasure coded data can begin while some Storage Nodes are offline. Repair will complete after all nodes are available.
Note: You cannot run repair-data operations for more than one node at the same time. To recover multiple nodes, contact technical support.

As object data is restored, if the StorageGRID Webscale system cannot locate replicated object data, the LOST (Lost Objects) alarm triggers. Alarms may be triggered on Storage Nodes throughout the system. Action should be taken to determine the cause of the loss and if recovery is possible. For more information, see the Troubleshooting Guide.

Track the status of the repair of erasure coded data and make sure it completes successfully. Choose one of the following:

Use this command to determine the current status or result of the repair-data operation: repair-data show-ec-repair-status --repair-id repair ID
Status displays for the specified repair ID.

Use this command to list all repairs: repair-data show-ec-repair-status

The output lists information, including repair ID, for all previously and currently running repairs.

root@DC1-ADM1:~ # repair-data show-ec-repair-status                      

 Repair ID   Scope                   Start Time  End Time  State  Est Bytes Affected Bytes Repaired  Retry Repair
==========================================================================================================
 949283   DC1-S-99-10(Volumes: 1,2) 2016-11-30T15:27:06.9  Success   17359            17359           No
 949292   DC1-S-99-10(Volumes: 1,2) 2016-11-30T15:37:06.9  Failure   17359            0               Yes
 949294   DC1-S-99-10(Volumes: 1,2) 2016-11-30T15:47:06.9  Failure   17359            0               Yes
 949299   DC1-S-99-10(Volumes: 1,2) 2016-11-30T15:57:06.9  Failure   17359            0               Yes

If the output shows that the repair operation failed, use the --repair-id option to retry a failed repair.
The following command for retrying a failed node repair for erasure coded data uses a sample repair with ID 83930030303133434: repair-data start-ec-node-repair --repair-id 83930030303133434

The following command for retrying a failed erasure coded data volume repair uses a sample repair with ID 83930030303133434: repair-data start-ec-volume-repair --repair-id 83930030303133434