Skip to main content

Copy Prometheus metrics

Contributors
PDFs

After adding a new Admin Node, you can optionally copy the historical metrics maintained by Prometheus from the primary Admin Node to the new Admin Node. Copying the metrics ensures that historical metrics are consistent between Admin Nodes.

Before you begin

  • The new Admin Node is installed and running.

  • You have the Passwords.txt file.

  • You have the provisioning passphrase.

About this task

When you add an Admin Node, the software installation process creates a new Prometheus database. You can keep the historical metrics consistent between nodes by copying the Prometheus database from the primary Admin Node (the source Admin Node) to the new Admin Node.

Note Copying the Prometheus database might take an hour or more. Some Grid Manager features will be unavailable while services are stopped on the source Admin Node.
Steps

  1. Log in to the source Admin Node:

    1. Enter the following command: ssh admin@grid_node_IP

    2. Enter the password listed in the Passwords.txt file.

    3. Enter the following command to switch to root: su -

    4. Enter the password listed in the Passwords.txt file.

  2. From the source Admin Node, stop the Prometheus service: service prometheus stop

  3. Complete the following steps on the new Admin Node:

    1. Log in to the new Admin Node:

      1. Enter the following command: ssh admin@grid_node_IP

      2. Enter the password listed in the Passwords.txt file.

      3. Enter the following command to switch to root: su -

      4. Enter the password listed in the Passwords.txt file.

    2. Stop the Prometheus service: service prometheus stop

    3. Add the SSH private key to the SSH agent. Enter:ssh-add

    4. Enter the SSH Access Password listed in the Passwords.txt file.

    5. Copy the Prometheus database from the source Admin Node to the new Admin Node: /usr/local/prometheus/bin/prometheus-clone-db.sh Source_Admin_Node_IP

    6. When prompted, press Enter to confirm that you want to destroy the new Prometheus database on the new Admin Node.

      The original Prometheus database and its historical data are copied to the new Admin Node. When the copy operation is done, the script starts the new Admin Node. The following status appears:

      Database cloned, starting services

    7. When you no longer require passwordless access to other servers, remove the private key from the SSH agent. Enter:

      ssh-add -D

  4. Restart the Prometheus service on the source Admin Node.

    service prometheus start