Restoring Prometheus metrics when recovering a non-primary Admin Node
-
PDF of this doc site
- Install and upgrade software
-
Install and maintain hardware
- SG5700 storage appliances
- SG5600 storage appliances
-
Configure and manage
- Administer StorageGRID
- Use StorageGRID
-
Monitor and troubleshoot
- Monitor a StorageGRID system
Collection of separate PDF docs
Creating your file...
Optionally, you can retain the historical metrics maintained by Prometheus on a non-primary Admin Node that has failed.
-
The recovered Admin Node must be installed and running.
-
The StorageGRID system must include at least two Admin Nodes.
-
You must have the
Passwords.txt
file. -
You must have the provisioning passphrase.
If an Admin Node fails, the metrics maintained in the Prometheus database on the Admin Node are lost. When you recover the Admin Node, the software installation process creates a new Prometheus database. After the recovered Admin Node is started, it records metrics as if you had performed a new installation of the StorageGRID system.
If you restored a non-primary Admin Node, you can restore the historical metrics by copying the Prometheus database from the primary Admin Node (the source Admin Node) to the recovered Admin Node.
Copying the Prometheus database might take an hour or more. Some Grid Manager features will be unavailable while services are stopped on the source Admin Node. |
-
Log in to the source Admin Node:
-
Enter the following command:
ssh admin@grid_node_IP
-
Enter the password listed in the
Passwords.txt
file. -
Enter the following command to switch to root:
su -
-
Enter the password listed in the
Passwords.txt
file.
-
-
From the source Admin Node, stop the Prometheus service:
service prometheus stop
-
Complete the following steps on the recovered Admin Node:
-
Log in to the recovered Admin Node:
-
Enter the following command:
ssh admin@grid_node_IP
-
Enter the password listed in the
Passwords.txt
file. -
Enter the following command to switch to root:
su -
-
Enter the password listed in the
Passwords.txt
file.
-
-
Stop the Prometheus service:
service prometheus stop
-
Add the SSH private key to the SSH agent. Enter:
ssh-add
-
Enter the SSH Access Password listed in the
Passwords.txt
file. -
Copy the Prometheus database from the source Admin Node to the recovered Admin Node:
/usr/local/prometheus/bin/prometheus-clone-db.sh Source_Admin_Node_IP
-
When prompted, press Enter to confirm that you want to destroy the new Prometheus database on the recovered Admin Node.
The original Prometheus database and its historical data are copied to the recovered Admin Node. When the copy operation is done, the script starts the recovered Admin Node. The following status appears:
Database cloned, starting services
-
When you no longer require passwordless access to other servers, remove the private key from the SSH agent. Enter:
ssh-add -D
-
-
Restart the Prometheus service on the source Admin Node.
service prometheus start