Skip to main content
Install and maintain

Automated boot media recovery from the partner node - AFF C80

Contributors netapp-lisa netapp-jsnyder
PDFs

After installing the new boot media device in your AFF C80 storage system, you can start the automated boot media recovery process to restore the configuration from the partner node. During the recovery process, the system checks whether encryption is enabled and determines the type of key encryption in use. If key encryption is enabled, the system guides you through the appropriate steps to restore it.

The automated boot media recovery process is supported only in ONTAP 9.17.1 and later. If your storage system is running an earlier version of ONTAP, use the manual boot recovery procedure.

Before you begin

  • For OKM, you need the cluster-wide passphrase and also the backup data.

  • For EKM, you need copies of the following files from the partner node:

    • /cfcard/kmip/servers.cfg file.

    • /cfcard/kmip/certs/client.crt file.

    • /cfcard/kmip/certs/client.key file.

    • /cfcard/kmip/certs/CA.pem file.

Steps

  1. From the LOADER prompt, enter the command:

    boot_recovery -partner

    The screen displays the following message:

    Starting boot media recovery (BMR) process. Press Ctrl-C to abort…

  2. Monitor the boot media install recovery process.

    The process completes and displays the Installation complete message.

  3. The system checks for encryption and encryption type and displays one of two messages. Depending on what message is displayed, take one of the following actions:

    Important Occasionally, the process may not be able to identify if key manager is configured on the system. It will display an error message, ask if key manager is configured for the system, and then ask what type of key manager is configured. The process will resume after you resolve the issue.
    Show example of configuration error finding prompts 
    Error when fetching key manager config from partner ${partner_ip}: ${status}

Has key manager been configured on this system

Is the key manager onboard
    If you see this message…​ Do this…​

    key manager is not configured. Exiting.

    Encryption is not installed on the system. Complete the following steps:

    1. Log into the node when the login prompt is displayed and give back the storage:

      storage failover giveback -ofnode impaired_node_name

    2. Go to step 5 to enable automatic giveback if it was disabled.

    key manager is configured.

    Go to step 4 to restore the appropriate key manager.

    The node accesses the boot menu and runs:

    • Option 10 for systems with Onboard Key Manager (OKM).

    • Option 11 for systems with External Key Manager (EKM).

  4. Select the appropriate key manager restoration process.

    Onboard Key Manager (OKM)

    If OKM is detected, the system displays the following message and begins running BootMenu Option 10.

    key manager is configured.
Entering Bootmenu Option 10...

This option must be used only in disaster recovery procedures. Are you sure? (y or n):

    1. Enter Y at the prompt to confirm you want to start the OKM recovery process.

    2. Enter the following when prompted:

      1. The passphrase

      2. The passphrase again when prompted to confirm

      3. Backup data for onboard key manager

        Show example of passphrase and backup data prompts 
        Enter the passphrase for onboard key management:
-----BEGIN PASSPHRASE-----
<passphrase_value>
-----END PASSPHRASE-----
Enter the passphrase again to confirm:
-----BEGIN PASSPHRASE-----
<passphrase_value>
-----END PASSPHRASE-----
Enter the backup data:
-----BEGIN BACKUP-----
<passphrase_value>
-----END BACKUP-----

    3. Continue to monitor the recovery process as it restores the appropriate files from the partner node.

      When the recovery process is complete, the node will reboot. The following messages indicate a successful recovery:

      Trying to recover keymanager secrets....
Setting recovery material for the onboard key manager
Recovery secrets set successfully
Trying to delete any existing km_onboard.keydb file.

Successfully recovered keymanager secrets.

    4. When the node reboots, verify the boot media recovery was successful by confirming that the system is back online and operational.

    5. Return the impaired controller to normal operation by giving back its storage:

      storage failover giveback -ofnode impaired_node_name

    6. After the partner node is fully up and serving data, synchronize the OKM keys across the cluster.

      security key-manager onboard sync

    External Key Manager (EKM)

    If EKM is detected, the system displays the following message and begins running BootMenu Option 11.

    key manager is configured.
Entering Bootmenu Option 11...

    1. The next step depends on which version of ONTAP your system is running:

      If your system is running…​ Do this…​

      ONTAP 9.16.0

      1. Press Ctlr-C to exit BootMenu Option 11.

      2. Press Ctlr-C to exit the EKM configuration process and return to the boot menu.

      3. Select BootMenu Option 8.

      4. Reboot the node.

        If AUTOBOOT is set, the node reboots and uses the configuration files from the partner node.

        If AUTOBOOT is not set, enter the appropriate boot command. The node reboots and uses the configuration files from the partner node.

      5. Reboot the node so that EKM protects the boot media partition.

      6. Proceed to step c.

      ONTAP 9.16.1 and later

      Proceed to the next step.

    2. Enter the following EKM configuration setting when prompted:

      Action Example

      Enter the client certificate contents from the /cfcard/kmip/certs/client.crt file.
      Show example of client certificate contents 
      -----BEGIN CERTIFICATE-----
<certificate_value>
-----END CERTIFICATE-----

      Enter the client key file contents from the /cfcard/kmip/certs/client.key file.
      Show example of client key file contents 
      -----BEGIN RSA PRIVATE KEY-----
<key_value>
-----END RSA PRIVATE KEY-----

      Enter the KMIP server CA(s) file contents from the /cfcard/kmip/certs/CA.pem file.
      Show example of KMIP server file contents 
      -----BEGIN CERTIFICATE-----
<KMIP_certificate_CA_value>
-----END CERTIFICATE-----

      Enter the server configuration file contents from the /cfcard/kmip/servers.cfg file.
      Show example of server configuration file contents 
      xxx.xxx.xxx.xxx:5696.host=xxx.xxx.xxx.xxx
xxx.xxx.xxx.xxx:5696.port=5696
xxx.xxx.xxx.xxx:5696.trusted_file=/cfcard/kmip/certs/CA.pem
xxx.xxx.xxx.xxx:5696.protocol=KMIP1_4
1xxx.xxx.xxx.xxx:5696.timeout=25
xxx.xxx.xxx.xxx:5696.nbio=1
xxx.xxx.xxx.xxx:5696.cert_file=/cfcard/kmip/certs/client.crt
xxx.xxx.xxx.xxx:5696.key_file=/cfcard/kmip/certs/client.key
xxx.xxx.xxx.xxx:5696.ciphers="TLSv1.2:kRSA:!CAMELLIA:!IDEA:!RC2:!RC4:!SEED:!eNULL:!aNULL"
xxx.xxx.xxx.xxx:5696.verify=true
xxx.xxx.xxx.xxx:5696.netapp_keystore_uuid=<id_value>

      If prompted, enter the ONTAP Cluster UUID from the partner.

      You can check the cluster UUID from the partner node using the cluster identify show command.
      Show example of ONTAP Cluster UUID 
      Notice: bootarg.mgwd.cluster_uuid is not set or is empty.
Do you know the ONTAP Cluster UUID? {y/n} y
Enter the ONTAP Cluster UUID: <cluster_uuid_value>


System is ready to utilize external key manager(s).

      If prompted, enter the temporary network interface and settings for the node.

      You need to enter:

      1. The IP address for the port

      2. The netmask for the port

      3. The IP address of the default gateway
      Show example of a temporary network setting 
      In order to recover key information, a temporary network interface needs to be
configured.

Select the network port you want to use (for example, 'e0a')
e0M

Enter the IP address for port : xxx.xxx.xxx.xxx
Enter the netmask for port : xxx.xxx.xxx.xxx
Enter IP address of default gateway: xxx.xxx.xxx.xxx
Trying to recover keys from key servers....
[discover_versions]
[status=SUCCESS reason= message=]

    3. Depending on whether the key is successfully restored, take one of the following actions:

      • If you see kmip2_client: Successfully imported the keys from external key server: xxx.xxx.xxx.xxx:5696 in the output, the EKM configuration has been successfully restored.

        The process attempts to restore the appropriate files from the partner node and reboots the node. Go to step d.

      • If the key is not successfully restored, the system will halt and indicate that it could not restore the key. The error and warning messages are displayed. You must rerun the recovery process:

        boot_recovery -partner

        Show example of key recovery error and warning messages 
        ERROR: kmip_init: halting this system with encrypted mroot...
WARNING: kmip_init: authentication keys might not be available.
********************************************************
*                 A T T E N T I O N                    *
*                                                      *
*       System cannot connect to key managers.         *
*                                                      *
********************************************************
ERROR: kmip_init: halting this system with encrypted mroot...
.
Terminated

Uptime: 11m32s
System halting...

LOADER-B>

    4. When the node reboots, verify that the boot media recovery was successful by confirming that the system is back online and operational.

    5. Return the controller to normal operation by giving back its storage:

      storage failover giveback -ofnode impaired_node_name

  1. If automatic giveback was disabled, reenable it:

    storage failover modify -node local -auto-giveback true

  2. If AutoSupport is enabled, restore automatic case creation:

    system node autosupport invoke -node * -type all -message MAINT=END

What's next

After you've restored the ONTAP image and the node is up and serving data, you return the failed part to NetApp.