Upgrade storage firmware

Contributors netapp-dbagwell netapp-amitha netapp-mwallis amgrissino Download PDF of this page

Starting with Element 12.0 and management services version 2.14, you can perform firmware-only upgrades on your H-series storage nodes using the NetApp Hybrid Cloud Control UI and REST API. This procedure does not upgrade Element software and enables you to upgrade storage firmware outside of a major Element release.

What you’ll need
  • Admin privileges: You have storage cluster administrator permissions to perform the upgrade.

  • System time sync: You have ensured that the system time on all nodes is synced and that NTP is correctly configured for the storage cluster and nodes. Each node must be configured with a DNS nameserver in the per-node web UI (https://[IP address]:442) with no unresolved cluster faults related to time skew.

  • System ports: If you are using NetApp Hybrid Cloud Control for upgrades, you have ensured that the necessary ports are open. See Network ports for more information.

  • Management node: For NetApp Hybrid Cloud Control UI and API, the management node in your environment is running version 11.3.

  • Management services: You have updated your management services bundle to the latest version.

For H610S storage nodes running Element software version 12.0, you should apply D-patch SUST-909 before you upgrade to storage firmware bundle 2.27. Contact NetApp Support to obtain the D-patch before you upgrade. See Storage Firmware Bundle 2.27 Release Notes.
You must upgrade to the latest management services bundle before upgrading the firmware on your storage nodes. If you are updating your Element software to version 12.2, you need management services 2.14.60 or later to proceed.
Upgrade options

Choose one of the following storage firmware upgrade options:

Use NetApp Hybrid Cloud Control UI to upgrade storage firmware

You can use the NetApp Hybrid Cloud Control UI to upgrade the firmware of the storage nodes in your cluster.

What you’ll need
For potential issues while upgrading storage clusters using NetApp Hybrid Cloud Control and their workarounds, see the KB article.
The upgrade process takes approximately 30 minutes per node.
Steps
  1. Open a web browser and browse to the IP address of the management node:

    https://<ManagementNodeIP>
  2. Log in to NetApp Hybrid Cloud Control by providing the storage cluster administrator credentials.

  3. Click Upgrade near the top right of the interface.

  4. On the Upgrades page, select Storage.

    The Storage tab lists the storage clusters that are part of your installation. If a cluster is inaccessible by NetApp Hybrid Cloud Control, it will not be displayed on the Upgrades page. If you have clusters running Element 12.0 or later, you will see the current firmware bundle version listed for these clusters. If the nodes in a single cluster have different firmware versions on them or as the upgrade progresses, you will see Multiple in the Current Firmware Bundle Version column. You can click Multiple to navigate to the Nodes page to compare firmware versions. If all your clusters are running Element versions earlier than 12.0, you will not see any information about firmware bundle version numbers. This information is also available on the Nodes page. See View your inventory.
    If the cluster is up to date and/or no upgrade packages are available, the Element and Firmware Only tabs are not displayed. These tabs are also not displayed when an upgrade is in progress. If the Element tab is displayed, but not the Firmware Only tab, no firmware packages are available.
  5. Choose from the following options and perform the set of steps that are applicable to your cluster:

    Option Steps

    Your management node has external connectivity.

    1. Click the drop-down arrow next to the cluster you are upgrading.

    2. Click Firmware Only, and select from the upgrade versions available.

    3. Click Begin Upgrade.

    The Upgrade Status changes during the upgrade to reflect the status of the process. It also changes in response to actions you take, such as pausing the upgrade, or if the upgrade returns an error. See Upgrade status changes.
    While the upgrade is in progress, you can leave the page and come back to it later to continue monitoring the progress. The page does not dynamically update status and current version if the cluster row is collapsed. The cluster row must be expanded to update the table or you can refresh the page.

    You can download logs after the upgrade is complete.

    Your management node is within a dark site without external connectivity.

    1. Click the drop-down arrow next to the cluster you are upgrading.

    2. Click Browse to upload the upgrade package that you downloaded.

    3. Wait for the upload to complete. A progress bar shows the status of the upload.

    The file upload will be lost if you navigate away from the browser window.

    An on-screen message is displayed after the file is successfully uploaded and validated. Validation might take several minutes. If you navigate away from the browser window at this stage, the file upload is preserved.
    You can download logs after the upgrade is complete. For information about the various upgrade status changes, see Upgrade status changes.

Upgrade status changes

Here are the different states that the Upgrade Status column in the UI shows before, during, and after the upgrade process:

Upgrade state Description

Up to Date

The cluster was upgraded to the latest Element version available or the firmware was upgraded to the latest version.

Unable to Detect

NetApp Hybrid Cloud Control shows this status instead of Versions Available when it does not have external connectivity to reach the online software repository. This status is also displayed when the storage service API returns an upgrade status that is not in the enumerated list of possible upgrade statuses.

Versions Available

Newer versions of Element and/or storage firmware are available for upgrade.

In Progress

The upgrade is in progress. A progress bar shows the upgrade status. On-screen messages also show node-level faults and display the node ID of each node in the cluster as the upgrade progresses. You can monitor the status of each node using the Element UI or the NetApp Element plug-in for vCenter Server UI.

Upgrade Pausing

You can choose to pause the upgrade. Depending on the state of the upgrade process, the pause operation can succeed or fail. You will see a UI prompt asking you to confirm the pause operation. To ensure that the cluster is in a safe spot before pausing an upgrade, it can take up to two hours for the upgrade operation to be completely paused. To resume the upgrade, click Resume.

Paused

You paused the upgrade. Click Resume to resume the process.

Error

An error has occurred during the upgrade. You can download the error log and send it to NetApp Support. After you resolve the error, you can return to the page, and click Resume. When you resume the upgrade, the progress bar goes backwards for a few minutes while the system runs the health check and checks the current state of the upgrade.

What happens if an upgrade fails using NetApp Hybrid Cloud Control

If a drive or node fails during an upgrade, the Element UI will show cluster faults. The upgrade process does not proceed to the next node, and waits for the cluster faults to resolve. The progress bar in the UI shows that the upgrade is waiting for the cluster faults to resolve. At this stage, clicking Pause in the UI will not work, because the upgrade waits for the cluster to be healthy. You will need to engage NetApp Support to assist with the failure investigation.

NetApp Hybrid Cloud Control has a pre-set three-hour waiting period, during which one of the following scenarios can happen:

  • The cluster faults get resolved within the three-hour window, and upgrade resumes. You do not need to take any action in this scenario.

  • The problem persists after three hours, and the upgrade status shows Error with a red banner. You can resume the upgrade by clicking Resume after the problem is resolved.

  • NetApp Support has determined that the upgrade needs to be temporarily aborted to take corrective action before the three-hour window. Support will use the API to abort the upgrade.

Aborting the cluster upgrade while a node is being updated might result in the drives being ungracefully removed from the node. If the drives are ungracefully removed, adding the drives back during an upgrade will require manual intervention by NetApp Support. The node might be taking longer to do firmware updates or post update syncing activities. If the upgrade progress seems stalled, contact NetApp Support for assistance.

Use NetApp Hybrid Cloud Control API to upgrade storage firmware

You can use APIs to upgrade storage nodes in a cluster to the latest Element software version. You can use an automation tool of your choice to run the APIs. The API workflow documented here uses the REST API UI available on the management node as an example.

Steps
  1. Do one of the following depending on your connection:

    Option Steps

    Your management node has external connectivity.

    1. Verify the repository connection:

      1. Open the management node REST API UI on the management node:

        https://[management node IP]/package-repository/1/
      2. Click Authorize and complete the following:

        1. Enter the cluster user name and password.

        2. Enter the client ID as mnode-client.

        3. Click Authorize to begin a session.

        4. Close the authorization window.

      3. From the REST API UI, click GET ​/packages​/remote-repository​/connection.

      4. Click Try it out.

      5. Click Execute.

      6. If code 200 is returned, go to the next step. If there is no connection to the remote repository, establish the connection or use the dark site option.

    2. Find the upgrade package ID:

      1. From the REST API UI, click GET /packages.

      2. Click Try it out.

      3. Click Execute.

      4. From the response, copy and save the firmware package ID for use in a later step.

    Your management node is within a dark site without external connectivity.

    1. Download the storage firmware upgrade package to a device that is accessible to the management node:

      • For NetApp HCI systems, go to the NetApp HCI software download page and download the latest storage firmware image.

      • For SolidFire storage systems, go to the Element software download page and download the latest storage firmware image.

    2. Upload the storage firmware upgrade package to the management node:

      1. Open the management node REST API UI on the management node:

        https://[management node IP]/package-repository/1/
      2. Click Authorize and complete the following:

        1. Enter the cluster user name and password.

        2. Enter the client ID as mnode-client.

        3. Click Authorize to begin a session.

        4. Close the authorization window.

      3. From the REST API UI, click POST /packages.

      4. Click Try it out.

      5. Click Browse and select the upgrade package.

      6. Click Execute to initiate the upload.

      7. From the response, copy and save the package ID ("id") for use in a later step.

    3. Verify the status of the upload.

      1. From the REST API UI, click GET​ /packages​/{id}​/status.

      2. Click Try it out.

      3. Enter the firmware package ID you copied in the previous step in id.

      4. Click Execute to initiate the status request.

        The response indicates state as SUCCESS when complete.

  2. Locate the installation asset ID:

    1. Open the management node REST API UI on the management node:

      https://[management node IP]/inventory/1/
    2. Click Authorize and complete the following:

      1. Enter the cluster user name and password.

      2. Enter the client ID as mnode-client.

      3. Click Authorize to begin a session.

      4. Close the authorization window.

    3. From the REST API UI, click GET /installations.

    4. Click Try it out.

    5. Click Execute.

    6. From the response, copy the installation asset ID (id).

      "id": "abcd01e2-xx00-4ccf-11ee-11f111xx9a0b",
      "management": {
        "errors": [],
        "inventory": {
          "authoritativeClusterMvip": "10.111.111.111",
          "bundleVersion": "2.14.19",
          "managementIp": "10.111.111.111",
          "version": "1.4.12"
    7. From the REST API UI, click GET /installations/{id}.

    8. Click Try it out.

    9. Paste the installation asset ID into the id field.

    10. Click Execute.

    11. From the response, copy and save the storage cluster ID ("id") of the cluster you intend to upgrade for use in a later step.

      "storage": {
        "errors": [],
        "inventory": {
          "clusters": [
            {
              "clusterUuid": "a1bd1111-4f1e-46zz-ab6f-0a1111b1111x",
              "id": "a1bd1111-4f1e-46zz-ab6f-a1a1a111b012",
  3. Run the storage firmware upgrade:

    1. Open the storage REST API UI on the management node:

      https://[management node IP]/storage/1/
    2. Click Authorize and complete the following:

      1. Enter the cluster user name and password.

      2. Enter the client ID as mnode-client.

      3. Click Authorize to begin a session.

      4. Close the window.

    3. Click POST /upgrades.

    4. Click Try it out.

    5. Enter the upgrade package ID in the parameter field.

    6. Enter the storage cluster ID in the parameter field.

    7. Click Execute to initiate the upgrade.

      The response should indicate state as initializing:

      {
        "_links": {
          "collection": "https://localhost:442/storage/upgrades",
          "self": "https://localhost:442/storage/upgrades/3fa85f64-1111-4562-b3fc-2c963f66abc1",
          "log": https://localhost:442/storage/upgrades/3fa85f64-1111-4562-b3fc-2c963f66abc1/log
        },
        "storageId": "114f14a4-1a1a-11e9-9088-6c0b84e200b4",
        "upgradeId": "334f14a4-1a1a-11e9-1055-6c0b84e2001b4",
        "packageId": "774f14a4-1a1a-11e9-8888-6c0b84e200b4",
        "config": {},
        "state": "initializing",
        "status": {
          "availableActions": [
            "string"
          ],
          "message": "string",
          "nodeDetails": [
            {
              "message": "string",
              "step": "NodePreStart",
              "nodeID": 0,
              "numAttempt": 0
            }
          ],
          "percent": 0,
          "step": "ClusterPreStart",
          "timestamp": "2020-04-21T22:10:57.057Z",
          "failedHealthChecks": [
            {
              "checkID": 0,
              "name": "string",
              "displayName": "string",
              "passed": true,
              "kb": "string",
              "description": "string",
              "remedy": "string",
              "severity": "string",
              "data": {},
              "nodeID": 0
            }
          ]
        },
        "taskId": "123f14a4-1a1a-11e9-7777-6c0b84e123b2",
        "dateCompleted": "2020-04-21T22:10:57.057Z",
        "dateCreated": "2020-04-21T22:10:57.057Z"
      }
    8. Copy the upgrade ID ("upgradeId") that is part of the response.

  4. Verify the upgrade progress and results:

    1. Click GET ​/upgrades/{upgradeId}.

    2. Click Try it out.

    3. Enter the upgrade ID from the previous step in upgradeId.

    4. Click Execute.

    5. Do one of the following if there are problems or special requirements during the upgrade:

      Option Steps

      You need to correct cluster health issues due to failedHealthChecks message in the response body.

      1. Go to the specific KB article listed for each issue or perform the specified remedy.

      2. If a KB is specified, complete the process described in the relevant KB article.

      3. After you have resolved cluster issues, reauthenticate if needed and click PUT ​/upgrades/{upgradeId}.

      4. Click Try it out.

      5. Enter the upgrade ID from the previous step in upgradeId.

      6. Enter "action":"resume" in the request body.

        {
          "action": "resume"
        }
      7. Click Execute.

      You need to pause the upgrade because the maintenance window is closing or for another reason.

      1. Reauthenticate if needed and click PUT ​/upgrades/{upgradeId}.

      2. Click Try it out.

      3. Enter the upgrade ID from the previous step in upgradeId.

      4. Enter "action":"pause" in the request body.

        {
          "action": "pause"
        }
      5. Click Execute.

    6. Run the GET ​/upgrades/{upgradeId} API multiple times, as needed, until the process is complete.

      During the upgrade, the status indicates running if no errors are encountered. As each node is upgraded, the step value changes to NodeFinished.

      The upgrade has finished successfully when the percent value is 100 and the state indicates finished.

Find more information