Decommissioning grid nodes

You can decommission and permanently remove one or more Storage Nodes, API Gateway Nodes, or non-primary Admin Nodes from the StorageGRID Webscale system. You cannot decommission primary Admin Nodes or Archive Nodes.

Before you begin

You must be signed in to the Grid Management Interface using a supported browser.
You must have Maintenance or Root Access permissions.
You must have the provisioning passphrase.
You must not be running repair-data.
You must confirm that no other non-decommissioning maintenance procedures are running while the decommissioning procedure is running, unless the decommissioning procedure is paused.
Note: Confirm that Storage Node recovery is not in progress anywhere in the grid. If it is, you must wait until any Cassandra rebuild performed as part of the recovery is complete. You can then proceed with decommissioning.

Attention: The grid node must be available and online to perform decommissioning. Do not remove grid node's virtual machine or other resources until instructed to do so in this procedure.

About this task

The StorageGRID Webscale system prevents you from decommissioning a grid node if it will leave the StorageGRID Webscale in an invalid state. The following is enforced:

Storage Nodes cannot be decommissioned if their removal affects ADC quorum. ADC quorum requires at least three Storage Nodes at each data center site. If there are more than three Storage Nodes at a given data center site, a simple majority of those Storage Nodes must have ADC ((0.5 x Storage Nodes with ADC) + 1) .
The system must at all times include enough Storage Nodes to satisfy the active ILM policy.

If a Storage Node fails while you are decommissioning another Storage Node, wait for the decommissioning process to complete before recovering the failed Storage Node.

Decommissioning procedures may take days or weeks to complete. If needed, you can pause the decommissioning procedure during specific stages.

The decommissioning procedure performs the following tasks:

Revokes certificates for the grid node.
This notifies the rest of the StorageGRID Webscale system that the grid node has been decommissioned and is no longer available for system operations.
Disables StorageGRID services and gracefully shuts down the grid node.
Removes the decommissioned grid node from the Grid Topology tree in the StorageGRID Webscale system.
For Storage Nodes:
- Changes the state of the Storage Node being decommissioned to Offline.
- Checks the version of the software on the Storage Node being decommissioned in the StorageGRID Webscale system to ensure that the software version is consistent with the instructions in the grid task.
- Changes the state of the Storage Node being decommissioned to Read-only.
- Copies Cassandra data from the Storage Node being decommissioned to other Cassandra nodes.
- Migrates object data off the decommissioned Storage Node, truncating file lengths to zero to mark progress.
  
  If a Storage Node is offline while decommissioning is in progress, the Storage Node detects the pending decommissioning task when it rejoins the StorageGRID Webscale system and proceeds with data migration as described above.

Note: The grid task does not securely remove content from the storage volumes of the decommissioned Storage Node. To ensure data security, you must wipe the decommissioned grid node’s drives after decommissioning is complete.

Steps

For systems with erasure coded objects, confirm that repair-data is not running:
1. From the service laptop, log in to the primary Admin Node:
  1. Enter the following command: ssh admin@grid_node_IP
  2. Enter the password listed in the Passwords.txt file.
  3. Enter the following command to switch to root: su -
  4. Enter the password listed in the Passwords.txt file.
  When you are logged in as root, the prompt changes from $ to #.
2. Check for running repairs: repair-data show-ec-repair-status
  State must be Success in order to continue with the decommissioning process.
Select Maintenance > Decommission.
The Decommission page appears.
Select the checkbox beside each grid node you want to decommission and enter the provisioning passphrase.
Click Start Decommission.
The confirmation dialog appears.
Click OK in the confirmation dialog box.
The decommissioning procedure starts and the progress page appears. While active, the progress of the decommissioning procedure is displayed on the progress page. All of the decommissioned nodes are listed in the table. The table shows the Progress bar showing percent complete and the current stage of the procedure for each node. Additionally, a new recovery package is generated as a result of the grid configuration change.

Note: For Storage Nodes, you must not change the LDR > Storage > Storage State-Desired after the decommissioning procedure starts. Changing the state might result in some content not being copied to other locations.
Download the new Recovery Package by accessing the new package on the Maintenance > Recovery package page.
Note: Download the Recovery Package as soon as possible after provisioning to ensure recoverability during the decommission procedure.

When the decommissioning process is complete, the progress page closes and the Decommission selection page appears.
After the grid node automatically shuts down, you can remove the grid node's virtual machine or other resources.

After you finish

If you have not done so already, be sure to download the new Recovery Package.

After the decommissioning procedure, ensure that the drives of the decommissioned grid node are wiped clean. Use a commercially available data wiping tool or service to permanently and securely remove data from the drives.