Skip to main content
Element Software

Upgrade storage firmware

Contributors netapp-pcarriga netapp-yvonneo
PDFs

Starting with Element 12.0 and management services version 2.14, you can perform firmware-only upgrades on your storage nodes using the NetApp Hybrid Cloud Control UI and REST API. This procedure does not upgrade Element software and enables you to upgrade storage firmware outside of a major Element release.

What you'll need

  • Admin privileges: You have storage cluster administrator permissions to perform the upgrade.

  • System time sync: You have ensured that the system time on all nodes is synced and that NTP is correctly configured for the storage cluster and nodes. Each node must be configured with a DNS nameserver in the per-node web UI (https://[IP address]:442) with no unresolved cluster faults related to time skew.

  • System ports: If you are using NetApp Hybrid Cloud Control for upgrades, you have ensured that the necessary ports are open. See Network ports for more information.

  • Management node: For NetApp Hybrid Cloud Control UI and API, the management node in your environment is running version 11.3.

  • Management services: You have updated your management services bundle to the latest version.

Important For H610S storage nodes running Element software version 12.0, you should apply D-patch SUST-909 before you upgrade to storage firmware bundle 2.27. Contact NetApp Support to obtain the D-patch before you upgrade. See Storage Firmware Bundle 2.27 Release Notes.
Important You must upgrade to the latest management services bundle before upgrading the firmware on your storage nodes. If you are updating your Element software to version 12.2 or later, you need management services 2.14.60 or later to proceed.
Note For a complete matrix of firmware and driver firmware for your hardware, see supported storage firmware versions for SolidFire storage nodes.

  • Upgrade process time: You have scheduled sufficient time to perform your upgrade. When you upgrade to Element software 12.5 or later, the upgrade process time varies depending on your current Element software version and firmware updates.

    Storage Node Current Element software version Approximate software and firmware install time per node1 Approximate data synchronization time per node2 Approximate total upgrade time per node

    All SolidFire and NetApp H-series nodes with up-to-date firmware3

    12.x

    15 minutes

    10 to 15 minutes

    20 to 30 minutes

    H610S and H410S

    12.x and 11.8

    60 minutes

    30 to 60 minutes

    90 to 120 minutes

    H610S

    11.7 and earlier

    90 minutes

    40 to 70 minutes

    130 to 160 minutes
    You must also perform a complete node shutdown and power disconnect for each H610S node.

    1For a complete matrix of firmware and driver firmware for your hardware, see supported storage firmware versions for SolidFire storage nodes.

    2If you combine a cluster with a heavy write IOPS load with a longer firmware update time, the data synchronization time will increase.

    3Beginning with Element 12.7, the SF2405 and SF9608 storage nodes and FC0025 and SF-FCN-01 FC nodes are not supported. If you attempt to upgrade one of these nodes to Element 12.7, you will see an error stating that this node is not supported by Element 12.7.

  • End User License Agreement (EULA): Beginning with management services 2.20.69, you must accept and save the EULA before using the NetApp Hybrid Cloud Control UI or API to upgrade storage firmware:

    1. Open the IP address of the management node in a web browser:

      https://<ManagementNodeIP>

    2. Log in to NetApp Hybrid Cloud Control by providing the storage cluster administrator credentials.

    3. Select Upgrade near the top right of the interface.

    4. The EULA pops up. Scroll down, select I accept for current and all future updates, and select Save.

Upgrade options

Choose one of the following storage firmware upgrade options:

Use NetApp Hybrid Cloud Control UI to upgrade storage firmware

You can use the NetApp Hybrid Cloud Control UI to upgrade the firmware of the storage nodes in your cluster.

What you'll need
Caution For potential issues while upgrading storage clusters using NetApp Hybrid Cloud Control and their workarounds, see the KB article.
Tip The upgrade process takes approximately 30 minutes per storage node. If you are upgrading an Element storage cluster to storage firmware newer than version 2.76, individual storage nodes will only reboot during the upgrade if new firmware was written to the node.
Steps

  1. Open the IP address of the management node in a web browser:

    https://<ManagementNodeIP>

  2. Log in to NetApp Hybrid Cloud Control by providing the storage cluster administrator credentials.

  3. Select Upgrade near the top right of the interface.

  4. On the Upgrades page, select Storage.

    Note

    The Storage tab lists the storage clusters that are part of your installation. If a cluster is inaccessible by NetApp Hybrid Cloud Control, it will not be displayed on the Upgrades page. If you have clusters running Element 12.0 or later, you will see the current firmware bundle version listed for these clusters. If the nodes in a single cluster have different firmware versions on them or as the upgrade progresses, you will see Multiple in the Current Firmware Bundle Version column. You can select Multiple to navigate to the Nodes page to compare firmware versions. If all your clusters are running Element versions earlier than 12.0, you will not see any information about firmware bundle version numbers.

    If the cluster is up to date and/or no upgrade packages are available, the Element and Firmware Only tabs are not displayed. These tabs are also not displayed when an upgrade is in progress. If the Element tab is displayed, but not the Firmware Only tab, no firmware packages are available.

  5. Select the drop-down arrow next to the cluster you are upgrading.

  6. Select Browse to upload the upgrade package that you downloaded.

  7. Wait for the upload to complete. A progress bar shows the status of the upload.

    Caution The file upload will be lost if you navigate away from the browser window.

    An on-screen message is displayed after the file is successfully uploaded and validated. Validation might take several minutes. If you navigate away from the browser window at this stage, the file upload is preserved.

  8. Select Firmware Only, and select from the upgrade versions available.

  9. Select Begin Upgrade.

    Tip The Upgrade Status changes during the upgrade to reflect the status of the process. It also changes in response to actions you take, such as pausing the upgrade, or if the upgrade returns an error. See Upgrade status changes.
    Note While the upgrade is in progress, you can leave the page and come back to it later to continue monitoring the progress. The page does not dynamically update status and current version if the cluster row is collapsed. The cluster row must be expanded to update the table or you can refresh the page.

You can download logs after the upgrade is complete.

Upgrade status changes

Here are the different states that the Upgrade Status column in the UI shows before, during, and after the upgrade process:

Upgrade state Description

Up to Date

The cluster was upgraded to the latest Element version available or the firmware was upgraded to the latest version.

Unable to Detect

This status is displayed when the storage service API returns an upgrade status that is not in the enumerated list of possible upgrade statuses.

Versions Available

Newer versions of Element and/or storage firmware are available for upgrade.

In Progress

The upgrade is in progress. A progress bar shows the upgrade status. On-screen messages also show node-level faults and display the node ID of each node in the cluster as the upgrade progresses. You can monitor the status of each node using the Element UI or the NetApp Element plug-in for vCenter Server UI.

Upgrade Pausing

You can choose to pause the upgrade. Depending on the state of the upgrade process, the pause operation can succeed or fail. You will see a UI prompt asking you to confirm the pause operation. To ensure that the cluster is in a safe spot before pausing an upgrade, it can take up to two hours for the upgrade operation to be completely paused. To resume the upgrade, select Resume.

Paused

You paused the upgrade. Select Resume to resume the process.

Error

An error has occurred during the upgrade. You can download the error log and send it to NetApp Support. After you resolve the error, you can return to the page, and select Resume. When you resume the upgrade, the progress bar goes backwards for a few minutes while the system runs the health check and checks the current state of the upgrade.

What happens if an upgrade fails using NetApp Hybrid Cloud Control

If a drive or node fails during an upgrade, the Element UI will show cluster faults. The upgrade process does not proceed to the next node, and waits for the cluster faults to resolve. The progress bar in the UI shows that the upgrade is waiting for the cluster faults to resolve. At this stage, selecting Pause in the UI will not work, because the upgrade waits for the cluster to be healthy. You will need to engage NetApp Support to assist with the failure investigation.

NetApp Hybrid Cloud Control has a pre-set three-hour waiting period, during which one of the following scenarios can happen:

  • The cluster faults get resolved within the three-hour window, and upgrade resumes. You do not need to take any action in this scenario.

  • The problem persists after three hours, and the upgrade status shows Error with a red banner. You can resume the upgrade by selecting Resume after the problem is resolved.

  • NetApp Support has determined that the upgrade needs to be temporarily aborted to take corrective action before the three-hour window. Support will use the API to abort the upgrade.

Caution Aborting the cluster upgrade while a node is being updated might result in the drives being ungracefully removed from the node. If the drives are ungracefully removed, adding the drives back during an upgrade will require manual intervention by NetApp Support. The node might be taking longer to do firmware updates or post update syncing activities. If the upgrade progress seems stalled, contact NetApp Support for assistance.

Use NetApp Hybrid Cloud Control API to upgrade storage firmware

You can use APIs to upgrade storage nodes in a cluster to the latest Element software version. You can use an automation tool of your choice to run the APIs. The API workflow documented here uses the REST API UI available on the management node as an example.

Steps

  1. Download the storage firmware upgrade package to a device that is accessible to the management node; go to the Element software downloads page and download the latest storage firmware image.

  2. Upload the storage firmware upgrade package to the management node:

    1. Open the management node REST API UI on the management node:

      https://<ManagementNodeIP>/package-repository/1/

    2. Select Authorize and complete the following:

      1. Enter the cluster user name and password.

      2. Enter the client ID as mnode-client.

      3. Select Authorize to begin a session.

      4. Close the authorization window.

    3. From the REST API UI, select POST /packages.

    4. Select Try it out.

    5. Select Browse and select the upgrade package.

    6. Select Execute to initiate the upload.

    7. From the response, copy and save the package ID ("id") for use in a later step.

  3. Verify the status of the upload.

    1. From the REST API UI, select GET​ /packages​/{id}​/status.

    2. Select Try it out.

    3. Enter the firmware package ID you copied in the previous step in id.

    4. Select Execute to initiate the status request.

      The response indicates state as SUCCESS when complete.

  4. Locate the installation asset ID:

    1. Open the management node REST API UI on the management node:

      https://<ManagementNodeIP>/inventory/1/

    2. Select Authorize and complete the following:

      1. Enter the cluster user name and password.

      2. Enter the client ID as mnode-client.

      3. Select Authorize to begin a session.

      4. Close the authorization window.

    3. From the REST API UI, select GET /installations.

    4. Select Try it out.

    5. Select Execute.

    6. From the response, copy the installation asset ID (id).

      "id": "abcd01e2-xx00-4ccf-11ee-11f111xx9a0b",
"management": {
  "errors": [],
  "inventory": {
    "authoritativeClusterMvip": "10.111.111.111",
    "bundleVersion": "2.14.19",
    "managementIp": "10.111.111.111",
    "version": "1.4.12"

    7. From the REST API UI, select GET /installations/{id}.

    8. Select Try it out.

    9. Paste the installation asset ID into the id field.

    10. Select Execute.

    11. From the response, copy and save the storage cluster ID ("id") of the cluster you intend to upgrade for use in a later step.

      "storage": {
  "errors": [],
  "inventory": {
    "clusters": [
      {
        "clusterUuid": "a1bd1111-4f1e-46zz-ab6f-0a1111b1111x",
        "id": "a1bd1111-4f1e-46zz-ab6f-a1a1a111b012",

  5. Run the storage firmware upgrade:

    1. Open the storage REST API UI on the management node:

      https://<ManagementNodeIP>/storage/1/

    2. Select Authorize and complete the following:

      1. Enter the cluster user name and password.

      2. Enter the client ID as mnode-client.

      3. Select Authorize to begin a session.

      4. Close the window.

    3. Select POST /upgrades.

    4. Select Try it out.

    5. Enter the upgrade package ID in the parameter field.

    6. Enter the storage cluster ID in the parameter field.

    7. Select Execute to initiate the upgrade.

      The response should indicate state as initializing:

      {
  "_links": {
    "collection": "https://localhost:442/storage/upgrades",
    "self": "https://localhost:442/storage/upgrades/3fa85f64-1111-4562-b3fc-2c963f66abc1",
    "log": https://localhost:442/storage/upgrades/3fa85f64-1111-4562-b3fc-2c963f66abc1/log
  },
  "storageId": "114f14a4-1a1a-11e9-9088-6c0b84e200b4",
  "upgradeId": "334f14a4-1a1a-11e9-1055-6c0b84e2001b4",
  "packageId": "774f14a4-1a1a-11e9-8888-6c0b84e200b4",
  "config": {},
  "state": "initializing",
  "status": {
    "availableActions": [
      "string"
    ],
    "message": "string",
    "nodeDetails": [
      {
        "message": "string",
        "step": "NodePreStart",
        "nodeID": 0,
        "numAttempt": 0
      }
    ],
    "percent": 0,
    "step": "ClusterPreStart",
    "timestamp": "2020-04-21T22:10:57.057Z",
    "failedHealthChecks": [
      {
        "checkID": 0,
        "name": "string",
        "displayName": "string",
        "passed": true,
        "kb": "string",
        "description": "string",
        "remedy": "string",
        "severity": "string",
        "data": {},
        "nodeID": 0
      }
    ]
  },
  "taskId": "123f14a4-1a1a-11e9-7777-6c0b84e123b2",
  "dateCompleted": "2020-04-21T22:10:57.057Z",
  "dateCreated": "2020-04-21T22:10:57.057Z"
}

    8. Copy the upgrade ID ("upgradeId") that is part of the response.

  6. Verify the upgrade progress and results:

    1. Select GET ​/upgrades/{upgradeId}.

    2. Select Try it out.

    3. Enter the upgrade ID from the previous step in upgradeId.

    4. Select Execute.

    5. Do one of the following if there are problems or special requirements during the upgrade:

      Option Steps

      You need to correct cluster health issues due to failedHealthChecks message in the response body.

      1. Go to the specific KB article listed for each issue or perform the specified remedy.

      2. If a KB is specified, complete the process described in the relevant KB article.

      3. After you have resolved cluster issues, reauthenticate if needed and select PUT ​/upgrades/{upgradeId}.

      4. Select Try it out.

      5. Enter the upgrade ID from the previous step in upgradeId.

      6. Enter "action":"resume" in the request body.

        {
  "action": "resume"
}

      7. Select Execute.

      You need to pause the upgrade because the maintenance window is closing or for another reason.

      1. Reauthenticate if needed and select PUT ​/upgrades/{upgradeId}.

      2. Select Try it out.

      3. Enter the upgrade ID from the previous step in upgradeId.

      4. Enter "action":"pause" in the request body.

        {
  "action": "pause"
}

      5. Select Execute.

    6. Run the GET ​/upgrades/{upgradeId} API multiple times, as needed, until the process is complete.

      During the upgrade, the status indicates running if no errors are encountered. As each node is upgraded, the step value changes to NodeFinished.

      The upgrade has finished successfully when the percent value is 100 and the state indicates finished.

Find more information