Recovering from Storage Node failures

The procedure for recovering a failed Storage Node failure depends upon the type of failure, and the type of Storage Node that has failed.

Use this table to help decide how to respond to a Storage Node failure. Links to the complete procedures follow the table.

Issue Action Notes
More than one Storage Node has failed? Contact technical support. Under some circumstances, recovering more than one Storage Node might affect the integrity of the Cassandra database. Technical support can determine when it is safe to begin recovery of a second Storage Node.
A second Storage Node fails less than 15 days after a Storage Node failure or recovery?
  • Includes the case where a Storage Node fails while recovery of another Storage Node is still in progress.
Contact technical support.
Storage Node has been offline for more than 15 days? Recover a Storage Node down > 15 days This procedure is required to ensure Cassandra database integrity.
Appliance Storage Node has failed?

Recover an appliance Storage Node

Major steps:

  1. Prepare appliance for recovery.
  2. Select Start Recovery to configure the replacement appliance.
  3. Remount and reformat storage volumes.
  4. Restore object data.
  5. Check storage state.
The recovery procedure for appliance Storage Nodes is the same for all failures.
Storage volumes have failed?

(system drive is intact)

Recover from storage volume failure

Major steps:

  1. Identify and unmount failed volumes.
  2. Recover failed volumes and rebuild Cassandra if necessary.
  3. Restore object data.
  4. Check storage state.
This procedure is used for virtual Storage Nodes.
System drive and possibly storage volumes have failed?

Recover from system drive failure

Major steps:

  1. Replace node.
  2. Select Start Recovery to configure the replacement Storage Node.
  3. Remount and reformat storage volumes.
  4. Restore object data.
  5. Check storage state.
The node replacement procedure varies depending on the deployment platform.
Linux: For Linux deployments, Storage Node recovery steps after node replacement might be different than shown here. See Linux node replacement for details.