Skip to main content
Install and maintain

Replace the NVRAM module and/or NVRAM DIMMs - AFF A900

Contributors dougthomp netapp-martyh thrisun netapp-adityaw
PDFs

The NVRAM module consists of the NVRAM11 and DIMMs. You can replace a failed NVRAM module or the DIMMs inside the NVRAM module. To replace a failed NVRAM module, you must remove it from the chassis, move the DIMMs to the replacement module, and install the replacement NVRAM module into the chassis.

To replace and NVRAM DIMM, you must remove the NVRAM module from the chassis, replace the failed DIMM in the module, and then reinstall the NVRAM module.

About this task

Because the system ID is derived from the NVRAM module, if replacing the module, disks belonging to the system are reassigned to a new system ID.

Before you begin

  • All disk shelves must be working properly.

  • If your system is in an HA pair, the partner controller must be able to take over the controller associated with the NVRAM module that is being replaced.

  • This procedure uses the following terminology:

    • The impaired controller is the controller on which you are performing maintenance.

    • The healthy controller is the HA partner of the impaired controller.

  • This procedure includes steps for automatically reassigning disks to the controller module associated with the new NVRAM module. You must reassign the disks when directed to in the procedure. Completing the disk reassignment before giveback can cause issues.

  • You must replace the failed component with a replacement FRU component you received from your provider.

  • You cannot change any disks or disk shelves as part of this procedure.

Step 1: Shut down the impaired controller

Shut down or take over the impaired controller using one of the following options.

Option 1: Most systems

To shut down the impaired controller, you must determine the status of the controller and, if necessary, take over the controller so that the healthy controller continues to serve data from the impaired controller storage.

About this task

  • If you have a SAN system, you must have checked event messages (cluster kernel-service show) for impaired controller SCSI blade. The cluster kernel-service show command displays the node name, quorum status of that node, availability status of that node, and operational status of that node.

    Each SCSI-blade process should be in quorum with the other nodes in the cluster. Any issues must be resolved before you proceed with the replacement.

  • If you have a cluster with more than two nodes, it must be in quorum. If the cluster is not in quorum or a healthy controller shows false for eligibility and health, you must correct the issue before shutting down the impaired controller; see Synchronize a node with the cluster.

Steps

  1. If AutoSupport is enabled, suppress automatic case creation by invoking an AutoSupport message: system node autosupport invoke -node * -type all -message MAINT=number_of_hours_downh

    The following AutoSupport message suppresses automatic case creation for two hours: cluster1:> system node autosupport invoke -node * -type all -message MAINT=2h

  2. Disable automatic giveback from the console of the healthy controller: storage failover modify –node local -auto-giveback false

    Note When you see Do you want to disable auto-giveback?, enter y.

  3. Take the impaired controller to the LOADER prompt:

    If the impaired controller is displaying…​ Then…​

    The LOADER prompt

    Go to the next step.

    Waiting for giveback…​

    Press Ctrl-C, and then respond y when prompted.

    System prompt or password prompt

    Take over or halt the impaired controller from the healthy controller: storage failover takeover -ofnode impaired_node_name

    When the impaired controller shows Waiting for giveback…​, press Ctrl-C, and then respond y.
Option 2: Controller is in a MetroCluster
Note Do not use this procedure if your system is in a two-node MetroCluster configuration.

To shut down the impaired controller, you must determine the status of the controller and, if necessary, take over the controller so that the healthy controller continues to serve data from the impaired controller storage.

  • If you have a cluster with more than two nodes, it must be in quorum. If the cluster is not in quorum or a healthy controller shows false for eligibility and health, you must correct the issue before shutting down the impaired controller; see Synchronize a node with the cluster.

  • If you have a MetroCluster configuration, you must have confirmed that the MetroCluster Configuration State is configured and that the nodes are in an enabled and normal state (metrocluster node show).

Steps

  1. If AutoSupport is enabled, suppress automatic case creation by invoking an AutoSupport message: system node autosupport invoke -node * -type all -message MAINT=number_of_hours_downh

    The following AutoSupport message suppresses automatic case creation for two hours: cluster1:*> system node autosupport invoke -node * -type all -message MAINT=2h

  2. Disable automatic giveback from the console of the healthy controller: storage failover modify –node local -auto-giveback false

  3. Take the impaired controller to the LOADER prompt:

    If the impaired controller is displaying…​ Then…​

    The LOADER prompt

    Go to the next step.

    Waiting for giveback…​

    Press Ctrl-C, and then respond y when prompted.

    System prompt or password prompt (enter system password)

    Take over or halt the impaired controller from the healthy controller: storage failover takeover -ofnode impaired_node_name

    When the impaired controller shows Waiting for giveback…​, press Ctrl-C, and then respond y.

Step 2: Replace the NVRAM module

To replace the NVRAM module, locate it in slot 6 in the chassis and follow the specific sequence of steps.

  1. If you are not already grounded, properly ground yourself.

  2. Remove the target NVRAM module from the chassis:

    1. Depress the lettered and numbered cam button.

      The cam button moves away from the chassis.

    2. Rotate the cam latch down until it is in a horizontal position.

      The NVRAM module disengages from the chassis and moves out a few inches.

    3. Remove the NVRAM module from the chassis by pulling on the pull tabs on the sides of the module face.

      Animation - Replace the NVRAM module
      drw a900 move remove NVRAM module

    Callout number 1

    Lettered and numbered cam latch

    Callout number 2

    Cam latch completely unlocked

  3. Set the NVRAM module on a stable surface and remove the cover from the NVRAM module by pushing down on the blue locking button on the cover, and then, while holding down the blue button, slide the lid off the NVRAM module.

    drw a900 remove NVRAM module contents

    Callout number 1

    Cover locking button

    Callout number 2

    DIMM and DIMM ejector tabs

  4. Remove the DIMMs, one at a time, from the old NVRAM module and install them in the replacement NVRAM module.

  5. Close the cover on the module.

  6. Install the replacement NVRAM module into the chassis:

    1. Align the module with the edges of the chassis opening in slot 6.

    2. Gently slide the module into the slot until the lettered and numbered cam latch begins to engage with the I/O cam pin, and then push the cam latch all the way up to lock the module in place.

Step 3: Replace a NVRAM DIMM

To replace NVRAM DIMMs in the NVRAM module, you must remove the NVRAM module, open the module, and then replace the target DIMM.

  1. If you are not already grounded, properly ground yourself.

  2. Remove the target NVRAM module from the chassis:

    1. Depress the lettered and numbered cam button.

      The cam button moves away from the chassis.

    2. Rotate the cam latch down until it is in a horizontal position.

      The NVRAM module disengages from the chassis and moves out a few inches.

    3. Remove the NVRAM module from the chassis by pulling on the pull tabs on the sides of the module face.

      Animation - Replace NVRAM DIMM
      drw a900 move remove NVRAM module

    Callout number 1

    Lettered and numbered cam latch

    Callout number 2

    cam latch completely unlocked

  3. Set the NVRAM module on a stable surface and remove the cover from the NVRAM module by pushing down on the blue locking button on the cover, and then, while holding down the blue button, slide the lid off the NVRAM module.

    drw a900 remove NVRAM module contents

    Callout number 1

    Cover locking button

    Callout number 2

    DIMM and DIMM ejector tabs

  4. Locate the DIMM to be replaced inside the NVRAM module, and then remove it by pressing down on the DIMM locking tabs and lifting the DIMM out of the socket.

  5. Install the replacement DIMM by aligning the DIMM with the socket and gently pushing the DIMM into the socket until the locking tabs lock in place.

  6. Close the cover on the module.

  7. Install the NVRAM module into the chassis:

    1. Align the module with the edges of the chassis opening in slot 6.

    2. Gently slide the module into the slot until the lettered and numbered cam latch begins to engage with the I/O cam pin, and then push the cam latch all the way up to lock the module in place.

Step 4: Reboot the controller

After you replace the FRU, you must reboot the controller module.

  1. To boot ONTAP from the LOADER prompt, enter bye.

Step 5: Reassign disks

You must confirm the system ID change when you boot the replacement controller and then verify that the change was implemented.

Caution Disk reassignment is only needed when replacing the NVRAM module and does not apply to NVRAM DIMM replacement.
Steps

  1. If the replacement controller is in Maintenance mode (showing the *> prompt), exit Maintenance mode and go to the LOADER prompt: halt

  2. From the LOADER prompt on the replacement controller, boot the controller and entering y if you are prompted to override the system ID due to a system ID mismatch.

  3. Wait until the Waiting for giveback…​ message is displayed on the console of the controller with the replacement module and then, from the healthy controller, verify that the new partner system ID has been automatically assigned: storage failover show

    In the command output, you should see a message that the system ID has changed on the impaired controller, showing the correct old and new IDs. In the following example, node2 has undergone replacement and has a new system ID of 151759706.

    node1:> storage failover show
                                    Takeover
Node              Partner           Possible     State Description
------------      ------------      --------     -------------------------------------
node1             node2             false        System ID changed on partner (Old:
                                                  151759755, New: 151759706), In takeover
node2             node1             -            Waiting for giveback (HA mailboxes)

  4. Give back the controller:

    1. From the healthy controller, give back the replaced controller's storage: storage failover giveback -ofnode replacement_node_name

      The replacement controller takes back its storage and completes booting.

      If you are prompted to override the system ID due to a system ID mismatch, you should enter y.

      Note If the giveback is vetoed, you can consider overriding the vetoes.

      For more information, see the Manual giveback commands topic to override the veto.

    2. After the giveback has been completed, confirm that the HA pair is healthy and that takeover is possible: storage failover show

      The output from the storage failover show command should not include the System ID changed on partner message.

  5. Verify that the disks were assigned correctly: storage disk show -ownership

    The disks belonging to the replacement controller should show the new system ID. In the following example, the disks owned by node1 now show the new system ID, 151759706:

    node1:> storage disk show -ownership

Disk  Aggregate Home  Owner  DR Home  Home ID    Owner ID  DR Home ID Reserver  Pool
----- ------    ----- ------ -------- -------    -------    -------  ---------  ---
1.0.0  aggr0_1  node1 node1  -        151759706  151759706  -       151759706 Pool0
1.0.1  aggr0_1  node1 node1           151759706  151759706  -       151759706 Pool0
.
.
.

  6. If the system is in a MetroCluster configuration, monitor the status of the controller: metrocluster node show

    The MetroCluster configuration takes a few minutes after the replacement to return to a normal state, at which time each controller will show a configured state, with DR Mirroring enabled and a mode of normal. The metrocluster node show -fields node-systemid command output displays the old system ID until the MetroCluster configuration returns to a normal state.

  7. If the controller is in a MetroCluster configuration, depending on the MetroCluster state, verify that the DR home ID field shows the original owner of the disk if the original owner is a controller on the disaster site.

    This is required if both of the following are true:

  8. If your system is in a MetroCluster configuration, verify that each controller is configured: metrocluster node show - fields configuration-state

    node1_siteA::> metrocluster node show -fields configuration-state

dr-group-id            cluster node           configuration-state
-----------            ---------------------- -------------- -------------------
1 node1_siteA          node1mcc-001           configured
1 node1_siteA          node1mcc-002           configured
1 node1_siteB          node1mcc-003           configured
1 node1_siteB          node1mcc-004           configured

4 entries were displayed.

  9. Verify that the expected volumes are present for each controller: vol show -node node-name

  10. If storage encryption is enabled, you must restore functionality.

  11. If you disabled automatic takeover on reboot, enable it from the healthy controller: storage failover modify -node replacement-node-name -onreboot true

Step 6: Restore Storage and Volume Encryption functionality

If you have storage encryption enabled, use the appropriate procedure.

Important This step does not apply to NVRAM DIMM replacement.
Option 1: Using Onboard Key Manager
Steps

  1. Boot the node to the boot menu.

  2. Select option 10, Set onboard key management recovery secrets.

  3. Enter the passphrase for the onboard key manager you obtained from the customer.

  4. At the prompt, paste the backup key data from the output of security key-manager backup show OR security key-manager onboard show-backup command.

    Example of backup data:

    --------------------------BEGIN BACKUP--------------------------

    TmV0QXBwIEtleSBCbG9iAAEAAAAEAAAAcAEAAAAAAADuD+byAAAAACEAAAAAAAAA QAAAAAAAAABvOlH0AAAAAMh7qDLRyH1DBz12piVdy9ATSFMT0C0TlYFss4PDjTaV dzRYkLd1PhQLxAWJwOIyqSr8qY1SEBgm1IWgE5DLRqkiAAAAAAAAACgAAAAAAAAA 3WTh7gAAAAAAAAAAAAAAAAIAAAAAAAgAZJEIWvdeHr5RCAvHGclo+wAAAAAAAAAA IgAAAAAAAAAoAAAAAAAAAEOTcR0AAAAAAAAAAAAAAAACAAAAAAAJAGr3tJA/ LRzUQRHwv+1aWvAAAAAAAAAAACQAAAAAAAAAgAAAAAAAAACdhTcvAAAAAJ1PXeBf ml4NBsSyV1B4jc4A7cvWEFY6lLG6hc6tbKLAHZuvfQ4rIbYAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA . . . . H4nPQM0nrDRYRa9SCv8AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAA

    ---------------------------END BACKUP---------------------------

    Note The controller returns to the boot menu.

  5. Select option 1, Normal Boot

  6. Give back only the CFO aggregates with the storage failover giveback -fromnode local -only-cfo-aggregates true command.

    • If the command fails because of a failed disk, physically disengage the failed disk, but leave the disk in the slot until a replacement is received.

    • If the command fails because of an open CIFS session, check with the customer how to close out CIFS sessions.

      Note Terminating CIFS can cause loss of data.

    • If the command fails because the partner "not ready", wait 5 minutes for the NVRAMs to synchronize.

    • If the command fails because of an NDMP, SnapMirror, or SnapVault process, disable the process. See the appropriate content for more information.

  7. Once the giveback completes, check the failover and giveback status with the storage failover show and storage failover show-giveback commands.

    Only the CFO aggregates (root aggregate and CFO style data aggregates) will be shown.

  8. Run the security key-manager onboard sync:

    1. Run the security key-manager onboard sync command and then enter the passphrase when prompted.

    2. Enter the security key-manager key-query command to see a detailed view of all keys stored in the onboard key manager and verify that the Restored column = yes/true for all authentication keys.

      Note If the Restored column = anything other than yes/true, contact Customer Support.

    3. Wait 10 minutes for the key to synchronize across the cluster.

  9. Move the console cable to the partner controller.

  10. Give back the target controller using the storage failover giveback -fromnode local command.

  11. Check the giveback status, three minutes after it reports complete, using the storage failover show command.

    If giveback is not complete after 20 minutes, contact Customer Support.

  12. At the clustershell prompt, enter the net int show -is-home false command to list the logical interfaces that are not on their home controller and port.

    If any interfaces are listed as false, revert those interfaces back to their home port using the net int revert command.

  13. Move the console cable to the target controller and run the version -v command to check the ONTAP versions.

  14. Restore automatic giveback if you disabled it by using the storage failover modify -node local -auto-giveback true command.

  15. Reset the MSID if it was previously set and was captured at the beginning of this procedure:

    1. Assign a data authentication key to a FIPS drive or SED using the storage encryption disk modify -disk disk_ID -data-key-id key_ID command.

      Note You can use the security key-manager key query -key-type NSE-AK command to view key IDs.

    2. Verify that the authentication keys have been assigned using the storage encryption disk show command.

Option 2: Using External Manager

  1. Boot the controller to the boot menu.

  2. Select option 11, Configure node for external key management.

  3. Enter the management certificate information at the prompts.

    Note The controller returns to the boot menu after the management certificate information is completed.

  4. Select option 1, Normal Boot

  5. Move the console cable to the partner controller and give back the target controller storage using the storage failover giveback -fromnode local -only-cfo-aggregates true local command.

    • If the command fails because of a failed disk, physically disengage the failed disk, but leave the disk in the slot until a replacement is received.

    • If the command fails because of an open CIFS sessions, check with customer how to close out CIFS sessions.

      Note Terminating CIFS can cause loss of data.

    • If the command fails because the partner is "not ready", wait 5 minutes for the NVMEMs to synchronize.

    • If the command fails because of an NDMP, SnapMirror, or SnapVault process, disable the process. See the appropriate content for more information.

  6. Wait 3 minutes and check the failover status with the storage failover show command.

  7. At the clustershell prompt, enter the net int show -is-home false command to list the logical interfaces that are not on their home controller and port.

    If any interfaces are listed as false, revert those interfaces back to their home port using the net int revert command.

  8. Move the console cable to the target controller and run the version -v command to check the ONTAP versions.

  9. Restore automatic giveback if you disabled it by using the storage failover modify -node local -auto-giveback true command.

  10. Use the storage encryption disk show at the clustershell prompt, to review the output.

  11. Use the security key-manager key-query command to display the encryption and authentication keys that are stored on the key management servers.

    • If the Restored column = yes/true, you are done and can proceed to complete the replacement process.

    • If the Key Manager type = external and the Restored column = anything other than yes/true, use the security key-manager external restore command to restore the key IDs of the authentication keys.

      Note If the command fails, contact Customer Support.

    • If the Key Manager type = onboard and the Restored column = anything other than yes/true, use the security key-manager onboard sync command to re-sync the Key Manager type.

      Use the security key-manager key-query command to verify that the Restored column = yes/true for all authentication keys.

  12. Connect the console cable to the partner controller.

  13. Give back the controller using the storage failover giveback -fromnode local command.

  14. Restore automatic giveback if you disabled it by using the storage failover modify -node local -auto-giveback true command.

  15. Reset the MSID if it was previously set and was captured at the beginning of this procedure:

    1. Assign a data authentication key to a FIPS drive or SED using the storage encryption disk modify -disk disk_ID -data-key-id key_ID command.

      Note You can use the security key-manager key query -key-type NSE-AK command to view key IDs.

    2. Verify that the authentication keys have been assigned using the storage encryption disk show command.

Step 7: Return the failed part to NetApp

Return the failed part to NetApp, as described in the RMA instructions shipped with the kit. See the Part Return & Replacements page for further information.