Skip to main content
A newer release of this product is available.

Upgrade Element software

Contributors netapp-pcarriga netapp-yvonneo
PDFs

To upgrade NetApp Element software, you can use the NetApp Hybrid Cloud Control UI, REST API, or the HealthTools suite of tools. Certain operations are suppressed during an Element software upgrade, such as adding and removing nodes, adding and removing drives, and commands associated with initiators, volume access groups, and virtual networks, among others.

What you'll need

  • Admin privileges: You have storage cluster administrator permissions to perform the upgrade.

  • Valid upgrade path: You have checked upgrade path information for the Element version you are upgrading to and verified that the upgrade path is valid.
    NetApp KB: Upgrade matrix for storage clusters running NetApp Element Software (login required)

  • System time sync: You have ensured that the system time on all nodes is synced and that NTP is correctly configured for the storage cluster and nodes. Each node must be configured with a DNS nameserver in the per-node web UI (https://[IP address]:442) with no unresolved cluster faults related to time skew.

  • System ports: If you are using NetApp Hybrid Cloud Control for upgrades, you have ensured that the necessary ports are open. See Network ports for more information.

  • Management node: For NetApp Hybrid Cloud Control UI and API, the management node in your environment is running version 11.3.

  • Management services: You have updated your management services bundle to the latest version.

Important You must upgrade to the latest management services bundle before upgrading your Element software to version 12. If you are updating your Element software to version 12.2, you need management services 2.14.60 or later to proceed.

  • Cluster health: You have verified that the cluster is ready to be upgraded. See Run Element storage health checks prior to upgrading storage.

  • Updated BMC for H610S nodes: You have upgraded the BMC version for your H610S nodes. See the release notes and upgrade instructions.

  • End User License Agreement (EULA): Beginning with management services 2.20.69, you must accept and save the EULA before using the NetApp Hybrid Cloud Control UI or API to upgrade Element software:

    1. Open the IP address of the management node in a web browser:

      https://<ManagementNodeIP>

    2. Log in to NetApp Hybrid Cloud Control by providing the storage cluster administrator credentials.

    3. Select Upgrade near the top right of the interface.

    4. The EULA pops up. Scroll down, select I accept for current and all future updates, and select Save.

Upgrade options

Choose one of the following Element software upgrade options:

Important If you are upgrading an H610S series node to Element 12.2 and the node is running a version of Element earlier than 11.8, you will need to perform additional upgrade steps (phase 2) for each storage node. If you are running Element 11.8 or later, the additional upgrade steps (phase 2) are not required.

Use NetApp Hybrid Cloud Control UI to upgrade Element storage

Using the NetApp Hybrid Cloud Control UI, you can upgrade a storage cluster.

Caution For potential issues while upgrading storage clusters using NetApp Hybrid Cloud Control and their workarounds, see the KB article.
Tip The upgrade process takes approximately 30 minutes per node for non-H610S platforms.
Steps

  1. Open the IP address of the management node in a web browser:

    https://<ManagementNodeIP>

  2. Log in to NetApp Hybrid Cloud Control by providing the storage cluster administrator credentials.

  3. Select Upgrade near the top right of the interface.

  4. On the Upgrades page, select Storage.

    The Storage tab lists the storage clusters that are part of your installation. If a cluster is inaccessible by NetApp Hybrid Cloud Control, it will not be displayed on the Upgrades page.

  5. Choose from the following options and perform the set of steps that are applicable to your cluster:

    Option Steps

    All clusters running Element 11.8 and later

    1. Select Browse to upload the upgrade package that you downloaded.

    2. Wait for the upload to complete. A progress bar shows the status of the upload.

      Caution The file upload will be lost if you navigate away from the browser window.

      An on-screen message is displayed after the file is successfully uploaded and validated. Validation might take several minutes. If you navigate away from the browser window at this stage, the file upload is preserved.

    3. Select Begin Upgrade.

      Tip The Upgrade Status changes during the upgrade to reflect the status of the process. It also changes in response to actions you take, such as pausing the upgrade, or if the upgrade returns an error. See Upgrade status changes.
      Note While the upgrade is in progress, you can leave the page and come back to it later to continue monitoring the progress. The page does not dynamically update status and current version if the cluster row is collapsed. The cluster row must be expanded to update the table or you can refresh the page.

      You can download logs after the upgrade is complete.

    You are upgrading an H610S cluster running Element version earlier than 11.8.

    1. Select the drop-down arrow next to the cluster you are upgrading, and select from the upgrade versions available.

    2. Select Begin Upgrade.
      After the upgrade is complete, the UI prompts you to perform phase 2 of the process.

    3. Complete the additional steps required (phase 2) in the KB article, and acknowledge in the UI that you have completed phase 2.

    You can download logs after the upgrade is complete. For information about the various upgrade status changes, see Upgrade status changes.

Upgrade status changes

Here are the different states that the Upgrade Status column in the UI shows before, during, and after the upgrade process:

Upgrade state Description

Up to Date

The cluster was upgraded to the latest Element version available.

Versions Available

Newer versions of Element and/or storage firmware are available for upgrade.

In Progress

The upgrade is in progress. A progress bar shows the upgrade status. On-screen messages also show node-level faults and display the node ID of each node in the cluster as the upgrade progresses. You can monitor the status of each node using the Element UI or the NetApp Element plug-in for vCenter Server UI.

Upgrade Pausing

You can choose to pause the upgrade. Depending on the state of the upgrade process, the pause operation can succeed or fail. You will see a UI prompt asking you to confirm the pause operation. To ensure that the cluster is in a safe spot before pausing an upgrade, it can take up to two hours for the upgrade operation to be completely paused. To resume the upgrade, select Resume.

Paused

You paused the upgrade. Select Resume to resume the process.

Error

An error has occurred during the upgrade. You can download the error log and send it to NetApp Support. After you resolve the error, you can return to the page, and select Resume. When you resume the upgrade, the progress bar goes backwards for a few minutes while the system runs the health check and checks the current state of the upgrade.

Complete with Follow-up

Only for H610S nodes upgrading from Element version earlier than 11.8. After phase 1 of the upgrade process is complete, this state prompts you to perform phase 2 of the upgrade (see the KB article). After you complete phase 2 and acknowledge that you have completed it, the status changes to Up to Date.

Use NetApp Hybrid Cloud Control API to upgrade Element storage

You can use APIs to upgrade storage nodes in a cluster to the latest Element software version. You can use an automation tool of your choice to run the APIs. The API workflow documented here uses the REST API UI available on the management node as an example.

Steps

  1. Download the storage upgrade package to a device that is accessible to the management node; go to the NetApp HCI software downloads page and download the latest storage node image.

  2. Upload the storage upgrade package to the management node:

    1. Open the management node REST API UI on the management node:

      https://<ManagementNodeIP>/package-repository/1/

    2. Select Authorize and complete the following:

      1. Enter the cluster user name and password.

      2. Enter the client ID as mnode-client.

      3. Select Authorize to begin a session.

      4. Close the authorization window.

    3. From the REST API UI, select POST /packages.

    4. Select Try it out.

    5. Select Browse and select the upgrade package.

    6. Select Execute to initiate the upload.

    7. From the response, copy and save the package ID ("id") for use in a later step.

  3. Verify the status of the upload.

    1. From the REST API UI, select GET​ /packages​/{id}​/status.

    2. Select Try it out.

    3. Enter the package ID you copied in the previous step in id.

    4. Select Execute to initiate the status request.

      The response indicates state as SUCCESS when complete.

  4. Locate the storage cluster ID:

    1. Open the management node REST API UI on the management node:

      https://<ManagementNodeIP>/inventory/1/

    2. Select Authorize and complete the following:

      1. Enter the cluster user name and password.

      2. Enter the client ID as mnode-client.

      3. Select Authorize to begin a session.

      4. Close the authorization window.

    3. From the REST API UI, select GET /installations.

    4. Select Try it out.

    5. Select Execute.

    6. From the response, copy the installation asset ID ("id").

    7. From the REST API UI, select GET /installations/{id}.

    8. Select Try it out.

    9. Paste the installation asset ID into the id field.

    10. Select Execute.

    11. From the response, copy and save the storage cluster ID ("id") of the cluster you intend to upgrade for use in a later step.

  5. Run the storage upgrade:

    1. Open the storage REST API UI on the management node:

      https://<ManagementNodeIP>/storage/1/

    2. Select Authorize and complete the following:

      1. Enter the cluster user name and password.

      2. Enter the client ID as mnode-client.

      3. Select Authorize to begin a session.

      4. Close the authorization window.

    3. Select POST /upgrades.

    4. Select Try it out.

    5. Enter the upgrade package ID in the parameter field.

    6. Enter the storage cluster ID in the parameter field.

      The payload should look similar to the following example:

      {
  "config": {},
  "packageId": "884f14a4-5a2a-11e9-9088-6c0b84e211c4",
  "storageId": "884f14a4-5a2a-11e9-9088-6c0b84e211c4"
}

    7. Select Execute to initiate the upgrade.

      The response should indicate the state as initializing:

      {
  "_links": {
    "collection": "https://localhost:442/storage/upgrades",
    "self": "https://localhost:442/storage/upgrades/3fa85f64-1111-4562-b3fc-2c963f66abc1",
    "log": https://localhost:442/storage/upgrades/3fa85f64-1111-4562-b3fc-2c963f66abc1/log
  },
  "storageId": "114f14a4-1a1a-11e9-9088-6c0b84e200b4",
  "upgradeId": "334f14a4-1a1a-11e9-1055`-6c0b84e2001b4",
  "packageId": "774f14a4-1a1a-11e9-8888-6c0b84e200b4",
  "config": {},
  "state": "initializing",
  "status": {
    "availableActions": [
      "string"
    ],
    "message": "string",
    "nodeDetails": [
      {
        "message": "string",
        "step": "NodePreStart",
        "nodeID": 0,
        "numAttempt": 0
      }
    ],
    "percent": 0,
    "step": "ClusterPreStart",
    "timestamp": "2020-04-21T22:10:57.057Z",
    "failedHealthChecks": [
      {
        "checkID": 0,
        "name": "string",
        "displayName": "string",
        "passed": true,
        "kb": "string",
        "description": "string",
        "remedy": "string",
        "severity": "string",
        "data": {},
        "nodeID": 0
      }
    ]
  },
  "taskId": "123f14a4-1a1a-11e9-7777-6c0b84e123b2",
  "dateCompleted": "2020-04-21T22:10:57.057Z",
  "dateCreated": "2020-04-21T22:10:57.057Z"
}

    8. Copy the upgrade ID ("upgradeId") that is part of the response.

  6. Verify the upgrade progress and results:

    1. Select GET ​/upgrades/{upgradeId}.

    2. Select Try it out.

    3. Enter the upgrade ID from the previous step in upgradeId.

    4. Select Execute.

    5. Do one of the following if there are problems or special requirements during the upgrade:

      Option Steps

      You need to correct cluster health issues due to failedHealthChecks message in the response body.

      1. Go to the specific KB article listed for each issue or perform the specified remedy.

      2. If a KB is specified, complete the process described in the relevant KB article.

      3. After you have resolved cluster issues, reauthenticate if needed and select PUT ​/upgrades/{upgradeId}.

      4. Select Try it out.

      5. Enter the upgrade ID from the previous step in upgradeId.

      6. Enter "action":"resume" in the request body.

        {
  "action": "resume"
}

      7. Select Execute.

      You need to pause the upgrade because the maintenance window is closing or for another reason.

      1. Reauthenticate if needed and select PUT ​/upgrades/{upgradeId}.

      2. Select Try it out.

      3. Enter the upgrade ID from the previous step in upgradeId.

      4. Enter "action":"pause" in the request body.

        {
  "action": "pause"
}

      5. Select Execute.

      If you are upgrading an H610S cluster running an Element version earlier than 11.8, you see the state finishedNeedsAck in the response body. You need to perform additional upgrade steps (phase 2) for each H610S storage node.

      1. See [Upgrading H610S storage nodes to Element 12.2 or later (phase 2)] and complete the process for each node.

      2. Reauthenticate if needed and select PUT ​/upgrades/{upgradeId}.

      3. Select Try it out.

      4. Enter the upgrade ID from the previous step in upgradeId.

      5. Enter "action":"acknowledge" in the request body.

        {
  "action": "acknowledge"
}

      6. Select Execute.

    6. Run the GET ​/upgrades/{upgradeId} API multiple times, as needed, until the process is complete.

      During the upgrade, the status indicates running if no errors are encountered. As each node is upgraded, the step value changes to NodeFinished.

      The upgrade has finished successfully when the percent value is 100 and the state indicates finished.

What happens if an upgrade fails using NetApp Hybrid Cloud Control

If a drive or node fails during an upgrade, the Element UI will show cluster faults. The upgrade process does not proceed to the next node, and waits for the cluster faults to resolve. The progress bar in the UI shows that the upgrade is waiting for the cluster faults to resolve. At this stage, selecting Pause in the UI will not work, because the upgrade waits for the cluster to be healthy. You will need to engage NetApp Support to assist with the failure investigation.

NetApp Hybrid Cloud Control has a pre-set three-hour waiting period, during which one of the following scenarios can happen:

  • The cluster faults get resolved within the three-hour window, and upgrade resumes. You do not need to take any action in this scenario.

  • The problem persists after three hours, and the upgrade status shows Error with a red banner. You can resume the upgrade by selecting Resume after the problem is resolved.

  • NetApp Support has determined that the upgrade needs to be temporarily aborted to take corrective action before the three-hour window. Support will use the API to abort the upgrade.

Caution Aborting the cluster upgrade while a node is being updated might result in the drives being ungracefully removed from the node. If the drives are ungracefully removed, adding the drives back during an upgrade will require manual intervention by NetApp Support. The node might be taking longer to do firmware updates or post update syncing activities. If the upgrade progress seems stalled, contact NetApp Support for assistance.

Upgrade Element software at connected sites using HealthTools

Steps

  1. Download the storage upgrade package; go to the NetApp HCI software downloads page and download the latest storage node image to a device that is not the management node.

    Note You need the latest version of HealthTools to upgrade Element storage software.

  2. Copy the ISO file to the management node in an accessible location like /tmp.

    When you upload the ISO file, make sure that the name of the file does not change, otherwise later steps will fail.

  3. Optional: Download the ISO from the management node to the cluster nodes before the upgrade.

    This step reduces the upgrade time by pre-staging the ISO on the storage nodes and running additional internal checks to ensure that the cluster is in a good state to be upgraded. Performing this operation will not put the cluster into "upgrade" mode or restrict any of the cluster operations.

    sfinstall <MVIP> -u <cluster_username> <path-toinstall-file-ISO> --stage
    Note Omit the password from the command line to allow sfinstall to prompt for the information. For passwords that contain special characters, add a backslash (\) before each special character. For example, mypass!@1 should be entered as mypass\!\@.

    Example
    See the following sample input:

    sfinstall 10.117.0.244 -u admin /tmp/solidfire-rtfisodium-11.0.0.345.iso --stage

    The output for the sample shows that sfinstall attempts to verify if a newer version of sfinstall is available:

    sfinstall 10.117.0.244 -u admin
/tmp/solidfire-rtfisodium-11.0.0.345.iso 2018-10-01 16:52:15:
Newer version of sfinstall available.
This version: 2018.09.01.130, latest version: 2018.06.05.901.
The latest version of the HealthTools can be downloaded from:
https:// mysupport.netapp.com/NOW/cgi-bin/software/
or rerun with --skip-version-check

    See the following sample excerpt from a successful pre-stage operation:

    Note When staging completes, the message will display Storage Node Upgrade Staging Successful after the upgrade event. 
    flabv0004 ~ # sfinstall -u admin
10.117.0.87 solidfire-rtfi-sodium-patch3-11.3.0.14171.iso --stage
2019-04-03 13:19:58: sfinstall Release Version: 2019.01.01.49 Management Node Platform:
Ember Revision: 26b042c3e15a Build date: 2019-03-12 18:45
2019-04-03 13:19:58: Checking connectivity to MVIP 10.117.0.87
2019-04-03 13:19:58: Checking connectivity to node 10.117.0.86
2019-04-03 13:19:58: Checking connectivity to node 10.117.0.87
...
2019-04-03 13:19:58: Successfully connected to cluster and all nodes
...
2019-04-03 13:20:00: Do you want to continue? ['Yes', 'No']: Yes
...
2019-04-03 13:20:55: Staging install pack on cluster nodes
2019-04-03 13:20:55: newVersion: 11.3.0.14171
2019-04-03 13:21:01: nodeToStage: nlabp2814, nlabp2815, nlabp2816, nlabp2813
2019-04-03 13:21:02: Staging Node nlabp2815 mip=[10.117.0.87] nodeID=[2] (1 of 4 nodes)
2019-04-03 13:21:02: Node Upgrade serving image at
http://10.117.0.204/rtfi/solidfire-rtfisodium-
patch3-11.3.0.14171/filesystem.squashfs
...
2019-04-03 13:25:40: Staging finished. Repeat the upgrade command without the --stage option to start the upgrade.

    The staged ISOs will be automatically deleted after the upgrade completes. However, if the upgrade has not started and needs to be rescheduled, ISOs can be manually de-staged using the command:

    sfinstall <MVIP> -u <cluster_username> --destage

    After the upgrade has started, the de-stage option is no longer available.

  4. Start the upgrade with the sfinstall command and the path to the ISO file:

    sfinstall <MVIP> -u <cluster_username> <path-toinstall-file-ISO>

    Example

    See the following sample input command:

    sfinstall 10.117.0.244 -u admin /tmp/solidfire-rtfi-sodium-11.0.0.345.iso

    The output for the sample shows that sfinstall attempts to verify if a newer version of sfinstall is available:

    sfinstall 10.117.0.244 -u admin /tmp/solidfire-rtfi-sodium-11.0.0.345.iso
2018-10-01 16:52:15: Newer version of sfinstall available.
This version: 2018.09.01.130, latest version: 2018.06.05.901.
The latest version of the HealthTools can be downloaded from:
https://mysupport.netapp.com/NOW/cgi-bin/software/ or rerun with --skip-version-check

    See the following sample excerpt from a successful upgrade. Upgrade events can be used to monitor the progress of the upgrade.

    # sfinstall 10.117.0.161 -u admin solidfire-rtfi-sodium-11.0.0.761.iso
2018-10-11 18:28
Checking connectivity to MVIP 10.117.0.161
Checking connectivity to node 10.117.0.23
Checking connectivity to node 10.117.0.24
...
Successfully connected to cluster and all nodes
###################################################################
You are about to start a new upgrade
10.117.0.161
10.3.0.161
solidfire-rtfi-sodium-11.0.0.761.iso
Nodes:
10.117.0.23 nlabp1023 SF3010 10.3.0.161
10.117.0.24 nlabp1025 SF3010 10.3.0.161
10.117.0.26 nlabp1027 SF3010 10.3.0.161
10.117.0.28 nlabp1028 SF3010 10.3.0.161
###################################################################
Do you want to continue? ['Yes', 'No']: yes
...
Watching for new network faults. Existing fault IDs are set([]).
Checking for legacy network interface names that need renaming
Upgrading from 10.3.0.161 to 11.0.0.761 upgrade method=rtfi
Waiting 300 seconds for cluster faults to clear
Waiting for caches to fall below threshold
...
Installing mip=[10.117.0.23] nodeID=[1] (1 of 4 nodes)
Starting to move primaries.
Loading volume list
Moving primary slice=[7] away from mip[10.117.0.23] nodeID[1] ssid[11] to new ssid[15]
Moving primary slice=[12] away from mip[10.117.0.23] nodeID[1] ssid[11] to new ssid[15]
...
Installing mip=[10.117.114.24] nodeID=[2] (2 of 4 nodes)
Starting to move primaries.
Loading volume list
Moving primary slice=[5] away from mip[10.117.114.24] nodeID[2] ssid[7] to new ssid[11]
...
Install of solidfire-rtfi-sodium-11.0.0.761 complete.
Removing old software
No staged builds present on nodeID=[1]
No staged builds present on nodeID=[2]
...
Starting light cluster block service check
Important If you are upgrading an H610S series node to Element 12.2 and the node is running a version of Element earlier than 11.8, you will need to perform additional upgrade steps (phase 2) for each storage node. If you are running Element 11.8 or later, the additional upgrade steps (phase 2) are not required.

Upgrade Element software at dark sites using HealthTools

You can use the HealthTools suite of tools to update NetApp Element software at a dark site.

What you'll need

  1. Go to the NetApp HCI software downloads page.

  2. Select the correct software release and download the latest storage node image to a computer that is not the management node.

    Note You need the latest version of HealthTools to upgrade Element storage software.

  3. Download this JSON file from the NetApp Support Site on a computer that is not the management node and rename it to metadata.json.

  4. Copy the ISO file to the management node in an accessible location like /tmp.

    Tip You can do this by using, for example, SCP. When you upload the ISO file, make sure that the name of the file does not change, otherwise later steps will fail.
Steps

  1. Run the sfupdate-healthtools command:

    sfupdate-healthtools <path-to-healthtools-package>

  2. Check the installed version:

    sfupdate-healthtools -v

  3. Check the latest version against the metadata JSON file:

    sfupdate-healthtools -l --metadata=<path-to-metadata-json>

  4. Ensure that the cluster is ready:

    sudo sfupgradecheck -u <cluster_username> -p <cluster_password> MVIP --metadata=<path-to-metadata-json>

  5. Run the sfinstall command with the path to the ISO file and the metadata JSON file:

    sfinstall -u <cluster_username> <MVIP> <path-toinstall-file-ISO> --metadata=<path-to-metadata-json-file>

    See the following sample input command:

    sfinstall -u admin 10.117.78.244 /tmp/solidfire-rtfi-11.3.0.345.iso --metadata=/tmp/metadata.json

    Optional You can add the --stage flag to the sfinstall command to pre-stage the upgrade in advance.

Important If you are upgrading an H610S series node to Element 12.2 and the node is running a version of Element earlier than 11.8, you will need to perform additional upgrade steps (phase 2) for each storage node. If you are running Element 11.8 or later, the additional upgrade steps (phase 2) are not required.

What happens if an upgrade fails using HealthTools

If the software upgrade fails, you can pause the upgrade.

Tip You should pause an upgrade only with Ctrl-C. This enables the system to clean itself up.

When sfinstall waits for cluster faults to clear and if any failure causes the faults to remain, sfinstall will not proceed to the next node.

Steps

  1. You should stop sfinstall with Ctrl+C.

  2. Contact NetApp Support to assist with the failure investigation.

  3. Resume the upgrade with the same sfinstall command.

  4. When an upgrade is paused by using Ctrl+C, if the upgrade is currently upgrading a node, choose one of these options:

    • Wait: Allow the currently upgrading node to finish before resetting the cluster constants.

    • Continue: Continue the upgrade, which cancels the pause.

    • Abort: Reset the cluster constants and abort the upgrade immediately.

      Note Aborting the cluster upgrade while a node is being updated might result in the drives being ungracefully removed from the node. If the drives are ungracefully removed, adding the drives back during an upgrade will require manual intervention by NetApp Support. The node might be taking longer to do firmware updates or post update syncing activities. If the upgrade progress seems stalled, contact NetApp Support for assistance.

Upgrading H610S storage nodes to Element 12.2 (phase 2)

If you are upgrading an H610S series node to Element 12.2 and the node is running a version of Element earlier than 11.8, the upgrade process involves two phases.

Phase 1, which is performed first, follows the same steps as the standard upgrade to Element 12.2 process. It installs Element Software and all 5 firmware updates in a rolling fashion across the cluster one node at a time. Due to the firmware payload, the process is estimated to take approximately 1.5 to 2 hours per H610S node, including a single cold-boot cycle at the end of the upgrade for each node.

Phase 2 involves completing steps to perform a complete node shutdown and power disconnect for each H610S node that are described in a required KB. This phase is estimated to take approximately one hour per H610S node.

Important After you complete phase 1, four of the five firmware updates are activated during the cold boot on each H610S node; however, the Complex Programmable Logic Device (CPLD) firmware requires a complete power disconnect and reconnect to fully install. The CPLD firmware update protects against NVDIMM failures and metadata drive eviction during future reboots or power cycles. This power reset is estimated to take approximately one hour per H610S node. It requires shutting down the node, removing power cables or disconnecting power via a smart PDU, waiting approximately 3 minutes, and reconnecting power.
What you'll need

  • You have completed phase 1 of the H610S upgrade process and have upgraded your storage nodes using one the standard Element storage upgrade procedures.

Note Phase 2 requires on-site personnel.
Steps

  1. (Phase 2) Complete the power reset process required for each H610S node in the cluster:

Note If the cluster also has non-H610S nodes, these non-H610S nodes are exempt from phase 2 and do not need to be shut down or have their power disconnected.

  1. Contact NetApp Support for assistance and to schedule this upgrade.

  2. Follow the phase 2 upgrade procedure in this KB that is required to complete an upgrade for each H610S node.

Find more information