Restore encryption keys after manual boot recovery - AFF A70 and AFF A90
Restore encryption on the replacement boot media in your AFF A70 or AFF A90 system to ensure continued data protection. The replacement process involves verifying key availability, reapplying encryption settings, and confirming secure access to your data.
If your system is running in ONTAP 9.17.1 and later, use the automatic boot recovery procedure.
Complete the appropriate steps to restore encryption on your system based on your key manager type. If you are unsure which key manager your system uses, check the settings you captured at the beginning of the boot media replacement procedure.
Restore the Onboard Key Manager (OKM) configuration from the ONTAP boot menu.
Ensure you have the following information available:
-
Cluster-wide passphrase entered while enabling onboard key management
-
Verification that you have the correct passphrase and backup data using the How to verify onboard key management backup and cluster-wide passphrase procedure
On the impaired controller:
-
Connect the console cable to the impaired controller.
-
From the ONTAP boot menu, select the appropriate option:
ONTAP version Select this option ONTAP 9.8 or later
Select option 10.
Show example boot menu
Please choose one of the following: (1) Normal Boot. (2) Boot without /etc/rc. (3) Change password. (4) Clean configuration and initialize all disks. (5) Maintenance mode boot. (6) Update flash from backup config. (7) Install new software first. (8) Reboot node. (9) Configure Advanced Drive Partitioning. (10) Set Onboard Key Manager recovery secrets. (11) Configure node for external key management. Selection (1-11)? 10
ONTAP 9.7 and earlier
Select the hidden option
recover_onboard_keymanagerShow example boot menu
Please choose one of the following: (1) Normal Boot. (2) Boot without /etc/rc. (3) Change password. (4) Clean configuration and initialize all disks. (5) Maintenance mode boot. (6) Update flash from backup config. (7) Install new software first. (8) Reboot node. (9) Configure Advanced Drive Partitioning. Selection (1-19)? recover_onboard_keymanager
-
Confirm that you want to continue the recovery process when prompted:
Show example prompt
This option must be used only in disaster recovery procedures. Are you sure? (y or n): -
Enter the cluster-wide passphrase twice.
While entering the passphrase, the console does not show any input.
Show example prompt
Enter the passphrase for onboard key management:Enter the passphrase again to confirm: -
Enter the backup information:
-
Paste the entire content from the BEGIN BACKUP line through the END BACKUP line, including the dashes.
Show example prompt
Enter the backup data: --------------------------BEGIN BACKUP-------------------------- 0123456789012345678901234567890123456789012345678901234567890123 1234567890123456789012345678901234567890123456789012345678901234 2345678901234567890123456789012345678901234567890123456789012345 3456789012345678901234567890123456789012345678901234567890123456 4567890123456789012345678901234567890123456789012345678901234567 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA 0123456789012345678901234567890123456789012345678901234567890123 1234567890123456789012345678901234567890123456789012345678901234 2345678901234567890123456789012345678901234567890123456789012345 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA ---------------------------END BACKUP---------------------------
-
Press Enter twice at the end of the input.
The recovery process completes and displays the following message:
Successfully recovered keymanager secrets.Show example prompt
Trying to recover keymanager secrets.... Setting recovery material for the onboard key manager Recovery secrets set successfully Trying to delete any existing km_onboard.wkeydb file. Successfully recovered keymanager secrets. *********************************************************************************** * Select option "(1) Normal Boot." to complete recovery process. * * Run the "security key-manager onboard sync" command to synchronize the key database after the node reboots. ***********************************************************************************
Do not proceed if the displayed output is anything other than Successfully recovered keymanager secrets. Perform troubleshooting to correct the error.
-
-
Select option
1from the boot menu to continue booting into ONTAP.Show example prompt
*********************************************************************************** * Select option "(1) Normal Boot." to complete the recovery process. * *********************************************************************************** (1) Normal Boot. (2) Boot without /etc/rc. (3) Change password. (4) Clean configuration and initialize all disks. (5) Maintenance mode boot. (6) Update flash from backup config. (7) Install new software first. (8) Reboot node. (9) Configure Advanced Drive Partitioning. (10) Set Onboard Key Manager recovery secrets. (11) Configure node for external key management. Selection (1-11)? 1
-
Confirm that the controller's console displays the following message:
Waiting for giveback…(Press Ctrl-C to abort wait)On the partner controller:
-
Giveback the impaired controller:
storage failover giveback -fromnode local -only-cfo-aggregates trueOn the impaired controller:
-
After booting with only the CFO aggregate, synchronize the key manager:
security key-manager onboard sync -
Enter the cluster-wide passphrase for the Onboard Key Manager when prompted.
Show example prompt
Enter the cluster-wide passphrase for the Onboard Key Manager: All offline encrypted volumes will be brought online and the corresponding volume encryption keys (VEKs) will be restored automatically within 10 minutes. If any offline encrypted volumes are not brought online automatically, they can be brought online manually using the "volume online -vserver <vserver> -volume <volume_name>" command.
If the sync is successful, the cluster prompt is returned with no additional messages. If the sync fails, an error message appears before returning to the cluster prompt. Do not continue until the error is corrected and the sync runs successfully. -
Verify that all keys are synced:
security key-manager key query -restored falseThe command should return no results. If any results appear, repeat the sync command until no results are returned.
On the partner controller:
-
Giveback the impaired controller:
storage failover giveback -fromnode local -
Restore automatic giveback if you disabled it:
storage failover modify -node local -auto-giveback true -
If AutoSupport is enabled, restore automatic case creation:
system node autosupport invoke -node * -type all -message MAINT=END
Restore the External Key Manager configuration from the ONTAP boot menu.
Gather the following files from another cluster node or from your backup:
-
/cfcard/kmip/servers.cfgfile or the KMIP server address and port -
/cfcard/kmip/certs/client.crtfile (client certificate) -
/cfcard/kmip/certs/client.keyfile (client key) -
/cfcard/kmip/certs/CA.pemfile (KMIP server CA certificates)
On the impaired controller:
-
Connect the console cable to the impaired controller.
-
Select option
11from the ONTAP boot menu.Show example boot menu
(1) Normal Boot. (2) Boot without /etc/rc. (3) Change password. (4) Clean configuration and initialize all disks. (5) Maintenance mode boot. (6) Update flash from backup config. (7) Install new software first. (8) Reboot node. (9) Configure Advanced Drive Partitioning. (10) Set Onboard Key Manager recovery secrets. (11) Configure node for external key management. Selection (1-11)? 11
-
Confirm you have gathered the required information when prompted:
Show example prompt
Do you have a copy of the /cfcard/kmip/certs/client.crt file? {y/n} Do you have a copy of the /cfcard/kmip/certs/client.key file? {y/n} Do you have a copy of the /cfcard/kmip/certs/CA.pem file? {y/n} Do you have a copy of the /cfcard/kmip/servers.cfg file? {y/n} -
Enter the client and server information when prompted:
-
Enter the client certificate (client.crt) file contents, including the BEGIN and END lines.
-
Enter the client key (client.key) file contents, including the BEGIN and END lines.
-
Enter the KMIP server CA(s) (CA.pem) file contents, including the BEGIN and END lines.
-
Enter the KMIP server IP address.
-
Enter the KMIP server port (press Enter to use the default port 5696).
Show example
Enter the client certificate (client.crt) file contents: -----BEGIN CERTIFICATE----- <certificate_value> -----END CERTIFICATE----- Enter the client key (client.key) file contents: -----BEGIN RSA PRIVATE KEY----- <key_value> -----END RSA PRIVATE KEY----- Enter the KMIP server CA(s) (CA.pem) file contents: -----BEGIN CERTIFICATE----- <certificate_value> -----END CERTIFICATE----- Enter the IP address for the KMIP server: 10.10.10.10 Enter the port for the KMIP server [5696]: System is ready to utilize external key manager(s). Trying to recover keys from key servers.... kmip_init: configuring ports Running command '/sbin/ifconfig e0M' .. .. kmip_init: cmd: ReleaseExtraBSDPort e0M
The recovery process completes and displays the following message:
Successfully recovered keymanager secrets.Show example
System is ready to utilize external key manager(s). Trying to recover keys from key servers.... Performing initialization of OpenSSL Successfully recovered keymanager secrets.
-
-
Select option
1from the boot menu to continue booting into ONTAP.Show example prompt
*************************************************************************** * Select option "(1) Normal Boot." to complete the recovery process. * *************************************************************************** (1) Normal Boot. (2) Boot without /etc/rc. (3) Change password. (4) Clean configuration and initialize all disks. (5) Maintenance mode boot. (6) Update flash from backup config. (7) Install new software first. (8) Reboot node. (9) Configure Advanced Drive Partitioning. (10) Set Onboard Key Manager recovery secrets. (11) Configure node for external key management. Selection (1-11)? 1
-
Restore automatic giveback if you disabled it:
storage failover modify -node local -auto-giveback true -
If AutoSupport is enabled, restore automatic case creation:
system node autosupport invoke -node * -type all -message MAINT=END
After restoring encryption on the boot media, you need to return the failed part to NetApp.