Restore object data to StorageGRID storage volume where system drive is intact
Suggest changes
PDFs
After recovering a storage volume on a Storage Node where the system drive is intact, you can restore the replicated or erasure-coded object data that was lost when the storage volume failed.
-
If the automatic procedure failed to remount preserved storage volumes and reformat failed storage volumes, you've performed one of these procedures:
-
You've confirmed that the Storage Node where you are restoring objects has a Connection State of Connected
on the Nodes > Overview tab in the Grid Manager. -
You've confirmed the following:
-
A grid expansion to add a Storage Node isn't in process.
-
A Storage Node decommission isn't in process or failed.
-
A recovery of a failed storage volume isn't in process.
-
A recovery of a Storage Node with a failed system drive isn't in process.
-
An EC rebalance job isn't in process.
-
Appliance node cloning isn't in process.
-
After you've replaced the drives and the volumes have been formatted, Grid Manager displays the volumes as candidates for restoration on the Maintenance > Volume restoration > Nodes to restore tab.
Whenever possible, restore object data using the Volume restoration page in the Grid Manager. You can either enable automatic restore mode to automatically start volume restoration when the volumes are ready to be restored or manually perform volume restoration. Follow these guidelines:
-
If the volumes are listed at Maintenance > Volume restoration > Nodes to restore, restore object data as described in the steps below. The volumes will be listed if:
-
Some, but not all, storage volumes in a node have failed
-
All storage volumes in a node have failed and are being replaced with the same number of volumes or more volumes
The Volume restoration page in the Grid Manager also allows you to monitor the volume restoration process and view restoration history.
If the recovered Storage Node contains fewer volumes than the node it's replacing, contact technical support for assistance.
-
You can restore two types of object data:
-
Replicated data objects are restored from other locations, assuming that the grid's ILM rules were configured to make object copies available.
-
If an ILM rule was configured to store only one replicated copy and that copy existed on a storage volume that failed, you won't be able to recover the object.
-
If the only remaining copy of an object is in a Cloud Storage Pool, StorageGRID must issue multiple requests to the Cloud Storage Pool endpoint to restore object data.
-
-
Erasure-coded (EC) data objects are restored by reassembling the stored fragments. Corrupt or lost fragments are recreated by the erasure-coding algorithm from the remaining data and parity fragments.
Repairs of erasure-coded data can begin while some Storage Nodes are offline. However, if all erasure-coded data cannot be accounted for, the repair can't be completed. Repair will complete after all nodes are available.
|
Volume restoration is dependent on the availability of resources where object copies are stored. Progress of volume restoration is nonlinear and might take days or weeks to complete. |
Enable automatic restore mode
When you enable Automatic restore mode, volume restoration automatically starts when the volumes are ready to be restored.
-
In Grid Manager go to Maintenance > Volume restoration.
-
Select the Nodes to restore tab, then slide the toggle for Automatic restore mode to the enabled position.
-
When the confirmation dialog box appears, review the details.
-
You won't be able to start volume restoration jobs manually on any nodes.
-
Volume restorations will begin automatically only when no other maintenance procedures are in progress.
-
You can monitor the status of the job from the progress monitoring page.
-
StorageGRID automatically retries volume restorations that fail to start.
-
-
When you understand the results of enabling Automatic restore mode, select Yes in the confirmation dialog box.
You can disable Automatic restore mode at any time.
Manually restore failed volume or node
Follow these steps to restore a failed volume or node.
-
In Grid Manager go to Maintenance > Volume restoration.
-
Select the Nodes to restore tab, then slide the toggle for Automatic restore mode to the disabled position.
The number on the tab indicates the number of nodes with volumes requiring restoration.
-
Expand each node to see the volumes in it that need restoration and their status.
-
Correct any issues preventing restoration of each volume. Issues will be indicated when you select Waiting for manual steps, if it displays as the volume status.
-
Select a node to restore where all the volumes indicate a Ready to restore status.
You can only restore the volumes for one node at a time.
Each volume in the node must indicate that it's ready to restore.
-
Select Start restore.
-
Address any warnings that might appear or select Start anyway to ignore the warnings and start the restoration.
Nodes are moved from the Nodes to restore tab to the Restoration progress tab when the restoration starts.
If a volume restoration can't be started, the node returns to the Nodes to restore tab.
View restoration progress
The Restoration progress tab shows the status of the volume restoration process and information about the volumes for a node being restored.
Data repair rates for replicated and erasure-coded objects in all volumes are averages summarizing all restorations in process. The percentage of objects in those volumes that are intact and don't require restoration is also indicated.
|
|
Replicated data restoration is dependent on the availability of resources where the replicated copies are stored. Progress of replicated data restoration is nonlinear and might take days or weeks to complete. |
The Restoration jobs section displays information about volume restorations started from Grid Manager.
-
The number in the Restoration jobs section heading indicates the number of volumes that are either being restored or queued for restoration.
-
The table displays information about each volume in a node being restored and its progress.
-
The progress for each node displays the percentage for each job.
-
Expand the Details column to display the restoration start time and job ID.
-
-
If a volume restoration fails:
-
The Status column indicates
failed (attempting retry), and will be retried automatically. -
If multiple restoration jobs have failed, the most recent job will be retried automatically first.
-
The EC repair failure alert is triggered if the retries continue to fail. Follow the steps in the alert to resolve the issue.
-