Skip to main content
HCI
1.10

Upgrade Element software

Contributors netapp-pcarriga netapp-yvonneo netapp-mwallis netapp-dbagwell
PDFs

To upgrade NetApp Element software, you can use the NetApp Hybrid Cloud Control UI or REST API. Certain operations are suppressed during an Element software upgrade, such as adding and removing nodes, adding and removing drives, and commands associated with initiators, volume access groups, and virtual networks, among others.

Note Beginning with Element 12.5, NetApp HealthTools is no longer supported for Element software upgrades. If you are running Element 11.0 or 11.1, you must first upgrade to Element 12.3 using HealthTools and then upgrade to Element 12.5 or later using NetApp Hybrid Cloud Control.
What you'll need

  • Admin privileges: You have storage cluster administrator permissions to perform the upgrade.

  • Valid upgrade path: You have checked upgrade path information for the Element version you are upgrading to and verified that the upgrade path is valid. See NetApp KB: Upgrade matrix for storage clusters running NetApp Element Software

  • System time sync: You have ensured that the system time on all nodes is synced and that NTP is correctly configured for the storage cluster and nodes. Each node must be configured with a DNS nameserver in the per-node web UI (https://[IP address]:442) with no unresolved cluster faults related to time skew.

  • System ports: If you are using NetApp Hybrid Cloud Control for upgrades, you have ensured that the necessary ports are open. See Network ports for more information.

  • Management node: For NetApp Hybrid Cloud Control UI and API, the management node in your environment is running version 11.3.

  • Management services: You have updated your management services bundle to the latest version.

Important You must upgrade to the latest management services bundle before upgrading your Element software to version 12.5 or later. If you are updating your Element software to version 12.5 or later, you need management services 2.21.61 or later to proceed.

  • Cluster health: You have verified that the cluster is ready to be upgraded. See Run Element storage health checks prior to upgrading storage.

  • Updated baseboard management controller (BMC) for H610S nodes: You have upgraded the BMC version for your H610S nodes. See the release notes and upgrade instructions.

  • Upgrade process time: You have scheduled sufficient time to perform your upgrade. When you upgrade to Element software 12.5 or later, the upgrade process time varies depending on your current Element software version and firmware updates.

    Storage Node Current Element software version Approximate software and firmware install time per node1 Approximate data synchronization time per node2 Approximate total upgrade time per node

    All SolidFire and NetApp H-series nodes with up-to-date firmware

    12.x

    15 minutes

    10 to 15 minutes

    20 to 30 minutes

    H610S and H410S

    12.x and 11.8

    60 minutes

    30 to 60 minutes

    90 to 120 minutes

    H610S

    11.7 and earlier

    90 minutes

    40 to 70 minutes

    130 to 160 minutes
    You must also perform a complete node shutdown and power disconnect for each H610S node.

    1For a complete matrix of firmware and driver firmware for your hardware, see supported firmware and ESXi driver versions for NetApp HCI and firmware versions for NetApp HCI storage nodes.

    2If you combine a cluster with a heavy write IOPS load with a longer firmware update time, the data synchronization time will increase.

  • End User License Agreement (EULA): Beginning with management services 2.20.69, you must accept and save the EULA before using the NetApp Hybrid Cloud Control UI or API to upgrade Element software:

    1. Open the IP address of the management node in a web browser:

      https://<ManagementNodeIP>

    2. Log in to NetApp Hybrid Cloud Control by providing the storage cluster administrator credentials.

    3. Select Upgrade near the top right of the interface.

    4. The EULA pops up. Scroll down, select I accept for current and all future updates, and select Save.

Upgrade options

Choose one of the following Element software upgrade options:

Important If you are upgrading an H610S series node to Element 12.5 or later and the node is running a version of Element software earlier than 11.8, you will need to perform the additional upgrade steps in this KB article for each storage node. If you are running Element 11.8 or later, the additional upgrade steps are not required.

Use NetApp Hybrid Cloud Control UI to upgrade Element storage

Using the NetApp Hybrid Cloud Control UI, you can upgrade a storage cluster.

Caution For potential issues while upgrading storage clusters using NetApp Hybrid Cloud Control and their workarounds, see this KB article.
Steps

  1. Open the IP address of the management node in a web browser:

    https://<ManagementNodeIP>

  2. Log in to NetApp Hybrid Cloud Control by providing the storage cluster administrator credentials.

  3. Select Upgrade near the top right of the interface.

  4. On the Upgrades page, select Storage.

    The Storage tab lists the storage clusters that are part of your installation. If a cluster is inaccessible by NetApp Hybrid Cloud Control, it will not be displayed on the Upgrades page.

  5. Choose from the following options and perform the set of steps that are applicable to your cluster:

    Option Steps

    All clusters running Element 11.8 and later

    1. Select Browse to upload the upgrade package that you downloaded.

    2. Wait for the upload to complete. A progress bar shows the status of the upload.

      Caution The file upload will be lost if you navigate away from the browser window.

      An on-screen message is displayed after the file is successfully uploaded and validated. Validation might take several minutes. If you navigate away from the browser window at this stage, the file upload is preserved.

    3. Select Begin Upgrade.

      Tip The Upgrade Status changes during the upgrade to reflect the status of the process. It also changes in response to actions you take, such as pausing the upgrade, or if the upgrade returns an error. See Upgrade status changes.
      Note While the upgrade is in progress, you can leave the page and come back to it later to continue monitoring the progress. The page does not dynamically update status and current version if the cluster row is collapsed. The cluster row must be expanded to update the table or you can refresh the page.

      You can download logs after the upgrade is complete.

    You are upgrading an H610S cluster running Element version earlier than 11.8.

    1. Select the drop-down arrow next to the cluster you are upgrading, and select from the upgrade versions available.

    2. Select Begin Upgrade.
      After the upgrade is complete, the UI prompts you to perform additional upgrade steps.

    3. Complete the additional steps required in the KB article, and acknowledge in the UI that you have completed them.

    You can download logs after the upgrade is complete. For information about the various upgrade status changes, see Upgrade status changes.

Upgrade status changes

Here are the different states that the Upgrade Status column in the UI shows before, during, and after the upgrade process:

Upgrade state Description

Up to Date

The cluster was upgraded to the latest Element version available.

Versions Available

Newer versions of Element and/or storage firmware are available for upgrade.

In Progress

The upgrade is in progress. A progress bar shows the upgrade status. On-screen messages also show node-level faults and display the node ID of each node in the cluster as the upgrade progresses. You can monitor the status of each node using the Element UI or the NetApp Element plug-in for vCenter Server UI.

Upgrade Pausing

You can choose to pause the upgrade. Depending on the state of the upgrade process, the pause operation can succeed or fail. You will see a UI prompt asking you to confirm the pause operation. To ensure that the cluster is in a safe spot before pausing an upgrade, it can take up to two hours for the upgrade operation to be completely paused. To resume the upgrade, select Resume.

Paused

You paused the upgrade. Select Resume to resume the process.

Error

An error has occurred during the upgrade. You can download the error log and send it to NetApp Support. After you resolve the error, you can return to the page, and select Resume. When you resume the upgrade, the progress bar goes backwards for a few minutes while the system runs the health check and checks the current state of the upgrade.

Complete with Follow-up

Only for H610S nodes upgrading from Element version earlier than 11.8. After phase 1 of the upgrade process is complete, this state prompts you to perform additional upgrade steps (see the KB article). After you complete these additional steps and acknowledge that you have completed it, the status changes to Up to Date.

Use NetApp Hybrid Cloud Control API to upgrade Element storage

You can use APIs to upgrade storage nodes in a cluster to the latest Element software version. You can use an automation tool of your choice to run the APIs. The API workflow documented here uses the REST API UI available on the management node as an example.

Steps

  1. Download the storage upgrade package to a device that is accessible to the management node; go to the NetApp HCI software downloads page and download the latest storage node image.

  2. Upload the storage upgrade package to the management node:

    1. Open the management node REST API UI on the management node:

      https://<ManagementNodeIP>/package-repository/1/

    2. Select Authorize and complete the following:

      1. Enter the cluster user name and password.

      2. Enter the client ID as mnode-client.

      3. Select Authorize to begin a session.

      4. Close the authorization window.

    3. From the REST API UI, select POST /packages.

    4. Select Try it out.

    5. Select Browse and select the upgrade package.

    6. Select Execute to initiate the upload.

    7. From the response, copy and save the package ID ("id") for use in a later step.

  3. Verify the status of the upload.

    1. From the REST API UI, select GET​ /packages​/{id}​/status.

    2. Select Try it out.

    3. Enter the package ID you copied in the previous step in id.

    4. Select Execute to initiate the status request.

      The response indicates state as SUCCESS when complete.

  4. Locate the storage cluster ID:

    1. Open the management node REST API UI on the management node:

      https://<ManagementNodeIP>/inventory/1/

    2. Select Authorize and complete the following:

      1. Enter the cluster user name and password.

      2. Enter the client ID as mnode-client.

      3. Select Authorize to begin a session.

      4. Close the authorization window.

    3. From the REST API UI, select GET /installations.

    4. Select Try it out.

    5. Select Execute.

    6. From the response, copy the installation asset ID ("id").

    7. From the REST API UI, select GET /installations/{id}.

    8. Select Try it out.

    9. Paste the installation asset ID into the id field.

    10. Select Execute.

    11. From the response, copy and save the storage cluster ID ("id") of the cluster you intend to upgrade for use in a later step.

  5. Run the storage upgrade:

    1. Open the storage REST API UI on the management node:

      https://<ManagementNodeIP>/storage/1/

    2. Select Authorize and complete the following:

      1. Enter the cluster user name and password.

      2. Enter the client ID as mnode-client.

      3. Select Authorize to begin a session.

      4. Close the authorization window.

    3. Select POST /upgrades.

    4. Select Try it out.

    5. Enter the upgrade package ID in the parameter field.

    6. Enter the storage cluster ID in the parameter field.

      The payload should look similar to the following example:

      {
  "config": {},
  "packageId": "884f14a4-5a2a-11e9-9088-6c0b84e211c4",
  "storageId": "884f14a4-5a2a-11e9-9088-6c0b84e211c4"
}

    7. Select Execute to initiate the upgrade.

      The response should indicate the state as initializing:

      {
  "_links": {
    "collection": "https://localhost:442/storage/upgrades",
    "self": "https://localhost:442/storage/upgrades/3fa85f64-1111-4562-b3fc-2c963f66abc1",
    "log": https://localhost:442/storage/upgrades/3fa85f64-1111-4562-b3fc-2c963f66abc1/log
  },
  "storageId": "114f14a4-1a1a-11e9-9088-6c0b84e200b4",
  "upgradeId": "334f14a4-1a1a-11e9-1055`-6c0b84e2001b4",
  "packageId": "774f14a4-1a1a-11e9-8888-6c0b84e200b4",
  "config": {},
  "state": "initializing",
  "status": {
    "availableActions": [
      "string"
    ],
    "message": "string",
    "nodeDetails": [
      {
        "message": "string",
        "step": "NodePreStart",
        "nodeID": 0,
        "numAttempt": 0
      }
    ],
    "percent": 0,
    "step": "ClusterPreStart",
    "timestamp": "2020-04-21T22:10:57.057Z",
    "failedHealthChecks": [
      {
        "checkID": 0,
        "name": "string",
        "displayName": "string",
        "passed": true,
        "kb": "string",
        "description": "string",
        "remedy": "string",
        "severity": "string",
        "data": {},
        "nodeID": 0
      }
    ]
  },
  "taskId": "123f14a4-1a1a-11e9-7777-6c0b84e123b2",
  "dateCompleted": "2020-04-21T22:10:57.057Z",
  "dateCreated": "2020-04-21T22:10:57.057Z"
}

    8. Copy the upgrade ID ("upgradeId") that is part of the response.

  6. Verify the upgrade progress and results:

    1. Select GET ​/upgrades/{upgradeId}.

    2. Select Try it out.

    3. Enter the upgrade ID from the previous step in upgradeId.

    4. Select Execute.

    5. Do one of the following if there are problems or special requirements during the upgrade:

      Option Steps

      You need to correct cluster health issues due to failedHealthChecks message in the response body.

      1. Go to the specific KB article listed for each issue or perform the specified remedy.

      2. If a KB is specified, complete the process described in the relevant KB article.

      3. After you have resolved cluster issues, reauthenticate if needed and select PUT ​/upgrades/{upgradeId}.

      4. Select Try it out.

      5. Enter the upgrade ID from the previous step in upgradeId.

      6. Enter "action":"resume" in the request body.

        {
  "action": "resume"
}

      7. Select Execute.

      You need to pause the upgrade because the maintenance window is closing or for another reason.

      1. Reauthenticate if needed and select PUT ​/upgrades/{upgradeId}.

      2. Select Try it out.

      3. Enter the upgrade ID from the previous step in upgradeId.

      4. Enter "action":"pause" in the request body.

        {
  "action": "pause"
}

      5. Select Execute.

      If you are upgrading an H610S cluster running an Element version earlier than 11.8, you see the state finishedNeedsAck in the response body. You must perform additional upgrade steps for each H610S storage node.

      1. Complete the additional upgrade steps in this KB article for each node.

      2. Reauthenticate if needed and select PUT ​/upgrades/{upgradeId}.

      3. Select Try it out.

      4. Enter the upgrade ID from the previous step in upgradeId.

      5. Enter "action":"acknowledge" in the request body.

        {
  "action": "acknowledge"
}

      6. Select Execute.

    6. Run the GET ​/upgrades/{upgradeId} API multiple times, as needed, until the process is complete.

      During the upgrade, the status indicates running if no errors are encountered. As each node is upgraded, the step value changes to NodeFinished.

      The upgrade has finished successfully when the percent value is 100 and the state indicates finished.

What happens if an upgrade fails using NetApp Hybrid Cloud Control

If a drive or node fails during an upgrade, the Element UI will show cluster faults. The upgrade process does not proceed to the next node, and waits for the cluster faults to resolve. The progress bar in the UI shows that the upgrade is waiting for the cluster faults to resolve. At this stage, selecting Pause in the UI will not work, because the upgrade waits for the cluster to be healthy. You will need to engage NetApp Support to assist with the failure investigation.

NetApp Hybrid Cloud Control has a pre-set three-hour waiting period, during which one of the following scenarios can happen:

  • The cluster faults get resolved within the three-hour window, and upgrade resumes. You do not need to take any action in this scenario.

  • The problem persists after three hours, and the upgrade status shows Error with a red banner. You can resume the upgrade by selecting Resume after the problem is resolved.

  • NetApp Support has determined that the upgrade needs to be temporarily aborted to take corrective action before the three-hour window. Support will use the API to abort the upgrade.

Caution Aborting the cluster upgrade while a node is being updated might result in the drives being ungracefully removed from the node. If the drives are ungracefully removed, adding the drives back during an upgrade will require manual intervention by NetApp Support. The node might be taking longer to do firmware updates or post update syncing activities. If the upgrade progress seems stalled, contact NetApp Support for assistance.

Find more information