Hot-add a shelf - shelves with IOM12/IOM12B modules

09/30/2024 Contributors

You can hot-add one or more disk shelves with IOM12/IOM12B modules to an existing stack of disk shelves with IOM12/IOM12B modules or hot-add a stack of one or more disk shelves with IOM12/IOM12B modules directly to a SAS HBA or an onboard SAS port on the controller.

About this task

You cannot use this procedure to mix a stack: hot-add a shelf with IOM12/IOM12B modules to a stack of shelves that has IOM6 modules. If you need to mix a stack, use Hot-add IOM12 shelves to a stack of IOM6 shelves.

Requirements for hot-adding disk shelves with IOM12/IOM12B modules

Your system must meet certain requirements before hot-adding disk shelves with IOM12/IOM12B modules.

State of your system

Your system and version of ONTAP must support the disk shelves you are hot-adding, including the IOMs, disk drives, and SAS cables.

NetApp Hardware Universe
Your system must have less than the maximum number of disk drives supported, by at least the number of disk shelves you plan to hot-add.

You cannot have exceeded the maximum number of disk drives supported for your system after hot-adding disk shelves.

NetApp Hardware Universe

If you are hot-adding a stack of one or more disk shelves (directly to the platform controllers), your system must have enough available PCI SAS HBA or onboard SAS ports or a combination of both.

If you need to install an additional PCI SAS HBA, the best practice is to use 12Gb SAS HBAs to keep controller-to-stack connectivity at 12Gbs for maximum performance.

Using 6Gb SAS HBAs or a combination of 6Gb SAS HBAs and 12Gb SAS HBAs is supported; however, IOM12 module connections to 6Gb SAS HBAs are negotiated down to 6Gbs, resulting in lower performance.

Your system cannot have any SAS cabling error messages.

Download and run Config Advisor to verify that your SAS connections are cabled correctly.

You must correct any cabling errors using the corrective actions provided by the error messages.

NetApp Downloads: Config Advisor

Using mini-SAS HD SAS optical cables

If you are using mini-SAS HD SAS optical cables or a mix of mini-SAS HD SAS optical cables and SAS copper cables in the stack of disk shelves, you must have met the rules in Mini-SAS HD SAS optical cable rules.
If you are hot-adding a disk shelf with mini-SAS HD SAS optical cables to a stack of disk shelves that is connected with SAS copper cables, you can temporarily have both cable types in the stack.

After hot-adding the disk shelf, you must replace the SAS copper cables for the rest of the shelf-to-shelf connections in the stack and the controller-to-stack connections so that the stack meets the rules in Mini-SAS HD SAS optical cable rules. This means that you must have ordered the appropriate number of mini-SAS HD SAS optical cables.

Considerations for hot-adding disk shelves with IOM12/IOM12B modules

You should familiarize yourself with aspects and best practices about this procedure before hot-adding disk shelves.

General considerations

If you are hot-adding a disk shelf with IOM12/IOM12B modules to an existing stack (of disk shelves with IOM12/IOM12B modules), you can hot-add the disk shelf to either end—the logical first or last disk shelf—of the stack.

For single-path HA and single-path configurations, as applicable to AFF A200, AFF A220, FAS2600 series, and FAS2700 systems, you hot-add disk shelves to the end of the stack that does not have controller connections.
Disk shelves with IOM12/IOM12B modules must be in their own unique stack; they cannot be added to a stack that has shelves with IOM6 modules or IOM3 modules.
This procedure assumes your configuration is using in-band ACP.

For configurations that have in-band ACP enabled, in-band ACP is automatically enabled on hot-added disk shelves. For configurations in which in-band ACP is not enabled, hot-added disk shelves operate without any ACP functionality.
Nondisruptive stack consolidation is not supported.

You cannot use this procedure to hot-add disk shelves that were hot-removed from another stack in the same system when the system is powered on and serving data (I/O is in progress).

Best practice considerations

The best practice is to have the current version of the Disk Qualification Package (DQP) installed before hot-adding a disk shelf.

Having the current version of the DQP installed allows your system to recognize and utilize newly qualified disk drives; therefore, avoiding system event messages about having non-current disk drive information. You also avoid the possible prevention of disk partitioning because disk drives are not recognized. The DQP also notifies you of non-current disk drive firmware.

NetApp Downloads: Disk Qualification Package
The best practice is to run Config Advisor before and after hot-adding a disk shelf.

Running Config Advisor before hot-adding a disk shelf provides a snapshot of the SAS connectivity, verifies disk shelf (IOM) firmware versions, and allows you to verify shelf IDs already in use on your system. Running Config Advisor after hot-adding a disk shelf allows you to verify SAS connections are cabled correctly and that shelf IDs are unique within the HA pair or single-controller configuration.

If any SAS cabling or duplicate shelf ID errors are generated, follow the corrective actions provided.

You need network access to download Config Advisor.

NetApp Downloads: Config Advisor
The best practice is to have the current versions of disk shelf (IOM) firmware and disk drive firmware on your system before adding new disk shelves, shelf FRU components, or SAS cables.

Current versions of firmware can be found on the NetApp Support Site.

NetApp Downloads: Disk Shelf Firmware

NetApp Downloads: Disk Drive Firmware

SAS cable handling considerations

Visually inspect the SAS port to verify the proper orientation of the connector before plugging it in.

The SAS cable connectors are keyed. When oriented correctly into a SAS port, the connector clicks into place and if the disk shelf power is on at the time, the disk shelf SAS port LNK LED illuminates green. For disk shelves, you insert a SAS cable connector with the pull tab oriented down (on the underside of the connector).

For controllers, the orientation of SAS ports can vary depending on the platform model; therefore, the correct orientation of the SAS cable connector varies.
To prevent degraded performance, do not twist, fold, pinch, or step on the cables.

Cables have a minimum bend radius. Cable manufacturer specifications define the minimum bend radius; however, a general guideline for minimum bend radius is 10 times the cable diameter.
Using Velcro wraps instead of tie-wraps to bundle and secure system cables allows for easier cable adjustments.

DS460C drive handling considerations

The drives are packaged separately from the shelf chassis.

You should take inventory of the drives.

After you unpack the drives, you should save the packaging materials for future use.

Possible loss of data access: If in the future, you move the shelf to a different part of the data center or transport the shelf to a different location, you need to remove the drives from the drive drawers to avoid possible damage to the drive drawers and drives.

Keep disk drives in their ESD bag until you are ready to install them.

When handling the drives, always wear an ESD wrist strap grounded to an unpainted surface on your storage enclosure chassis to prevent static discharges.

If a wrist strap is unavailable, touch an unpainted surface on your storage enclosure chassis before handling the disk drive.

Install disk shelves with IOM12/IOM12B modules for a hot-add

For each disk shelf you are hot-adding, you install the disk shelf into a rack, connect the power cords, power on the disk shelf, and set the disk shelf ID before cabling the SAS connections.

Steps

Install the rack mount kit (for two-post or four-post rack installations) that came with your disk shelf using the installation flyer that came with the kit.

If you are installing multiple disk shelves, you should install them from the bottom to the top of the rack for the best stability.

Do not flange-mount the disk shelf into a telco-type rack; the disk shelf's weight can cause it to collapse in the rack under its own weight.
Install and secure the disk shelf onto the support brackets and rack using the installation flyer that came with the kit.

To make a disk shelf lighter and easier to maneuver, remove the power supplies and I/O modules (IOMs).

For DS460C disk shelves, although the drives are packaged separately, which makes the shelf lighter, an empty DS460C shelf still weighs approximately 132 lb (60kg); therefore, exercise the following caution when moving a shelf.

It is recommended that you use a mechanized lift or four people using the lift handles to safely move an empty DS460C shelf.

Your DS460C shipment was packaged with four detachable lift handles (two for each side). To use the lift handles, you install them by inserting the tabs of the handles into the slots in the side of the shelf and pushing up until they click into place. Then, as you slide the disk shelf onto the rails, you detach one set of handles at a time using the thumb latch. The following illustration shows how to attach a lift handle.
Reinstall any power supplies and IOMs you removed prior to installing your disk shelf into the rack.

If you are installing a DS460C disk shelf, install the drives into the drive drawers; otherwise, go to the next step.

Always wear an ESD wrist strap grounded to an unpainted surface on your storage enclosure chassis to prevent static discharges.

If a wrist strap is unavailable, touch an unpainted surface on your storage enclosure chassis before handling the disk drive.

If you purchased a partially populated shelf, meaning that the shelf has less than the 60 drives it supports, for each drawer, install the drives as follows:

Install the first four drives into the front slots (0, 3, 6, and 9).

Risk of equipment malfunction: To allow for proper air flow and prevent overheating, always install the first four drives into the front slots (0, 3, 6, and 9).
For the remaining drives, evenly distribute them across each drawer.

The following illustration shows how the drives are numbered from 0 to 11 in each drive drawer within the shelf.
1. Open the top drawer of the shelf.
2. Remove a drive from its ESD bag.
3. Raise the cam handle on the drive to vertical.
4. Align the two raised buttons on each side of the drive carrier with the matching gap in the drive channel on the drive drawer.
  
  Raised button on the right side of the drive carrier
5. Lower the drive straight down, and then rotate the cam handle down until the drive snaps into place under the orange release latch.
6. Repeat the previous substeps for each drive in the drawer.
  
  You must be sure that slots 0, 3, 6, and 9 in each drawer contain drives.
7. Carefully push the drive drawer back into the enclosure.
  
  Possible loss of data access: Never slam the drawer shut. Push the drawer in slowly to avoid jarring the drawer and causing damage to the storage array.
8. Close the drive drawer by pushing both levers towards the center.
9. Repeat these steps for each drawer in the disk shelf.
10. Attach the front bezel.

If you are adding multiple disk shelves, repeat the previous steps for each disk shelf you are installing.
Connect the power supplies for each disk shelf:
1. Connect the power cords first to the disk shelves, securing them in place with the power cord retainer, and then connect the power cords to different power sources for resiliency.
2. Turn on the power supplies for each disk shelf and wait for the disk drives to spin up.
Set the shelf ID for each disk shelf you are hot-adding to an ID that is unique within the HA pair or single-controller configuration.

If you have a platform model with an internal disk shelf, shelf IDs must be unique across the internal disk shelf and externally attached disk shelves.

You can use the following substeps to change shelf IDs, or for more detailed instructions, use Change a shelf ID.
1. If needed, verify shelf IDs already in use by running Config Advisor.
  
  You can also run the storage shelf show -fields shelf-id command to see a list of shelf IDs already in use (and duplicates if present) in your system.
2. Access the shelf ID button behind the left end cap.
3. Change the shelf ID to a valid ID (00 through 99).
4. Power-cycle the disk shelf to make the shelf ID take effect.
  
  Wait at least 10 seconds before turning the power back on to complete the power cycle.
  
  The shelf ID blinks and the operator display panel amber LED blinks until you power cycle the disk shelf.
5. Repeat substeps a through d for each disk shelf you are hot-adding.

Cable disk shelves with IOM12/IOM12B modules for a hot-add

You cable the SAS connections (shelf-to-shelf and controller-to-stack) as applicable for hot-added disk shelves so they have connectivity to the system.

Before you begin

You must have met the requirements in Requirements for hot-adding disk shelves with IOM12 modules and installed, powered on, and set shelf IDs for each disk shelf as instructed in Install disk shelves with IOM12 modules for a hot-add.

About this task

For an explanation and examples of shelf-to-shelf “standard” cabling and shelf-to-shelf “double-wide” cabling, see Shelf-to-shelf SAS connection rules.
For instructions about how to read a worksheet to cable controller-to-stack connections, see How to read a worksheet to cable controller-to-stack connections for multipathed connectivity or How to read a worksheet to cable controller-to-stack connections for quad-pathed connectivity.
After you have cabled the hot-added disk shelves, ONTAP recognizes them: disk ownership is assigned if disk ownership automatic assignment is enabled; disk shelf (IOM) firmware and disk drive firmware should automatically update if needed; and if in-band ACP is enabled on your configuration, it is automatically enabled on the hot-added disk shelves.

Firmware updates can take up to 30 minutes.

Steps

If you want to manually assign disk ownership for the disk shelves you are hot-adding, you need to disable disk ownership automatic assignment if it is enabled; otherwise, go to the next step.

You need to manually assign disk ownership if disks in the stack are owned by both controllers in an HA pair.

You disable disk ownership automatic assignment before cabling the hot-added disk shelves and then later, in step 7, you reenable it after cabling the hot-added disk shelves.
1. Verify if disk ownership automatic assignment is enabled:storage disk option show
  
  If you have an HA pair, you can enter the command at the console of either controller.
  
  If disk ownership automatic assignment is enabled, the output shows “on” (for each controller) in the “Auto Assign” column.
2. If disk ownership automatic assignment is enabled, you need to disable it:storage disk option modify -node _node_nam_e -autoassign off
  
  You need to disable disk ownership automatic assignment on both controllers in an HA pair.

If you are hot-adding a stack of disk shelves directly to a controller, complete the following substeps; otherwise, go to step 3.

If the stack you are hot-adding has more than one disk shelf, cable the shelf-to-shelf connections; otherwise, go to substep b.

If…	Then…
You are cabling a stack with multipath HA, tri-path HA, multipath, single-path HA, or single-path connectivity to the controllers	Cable the shelf-to-shelf connections as “standard” connectivity (using IOM ports 3 and 1): Beginning with the logical first shelf in the stack, connect IOM A port 3 to the next shelf's IOM A port 1 until each IOM A in the stack is connected. Repeat substep i for IOM B.
You are cabling a stack with quad-path HA or quad-path connectivity to the controllers	Cable the shelf-to-shelf connections as “double-wide” connectivity: You cable the standard connectivity using IOM ports 3 and 1 and then the double-wide connectivity using IOM ports 4 and 2. Beginning with the logical first shelf in the stack, connect IOM A port 3 to the next shelf's IOM A port 1 until each IOM A in the stack is connected. Beginning with the logical first shelf in the stack, connect IOM A port 4 to the next shelf's IOM A port 2 until each IOM A in the stack is connected. Repeat substeps i and ii for IOM B.

If…

Then…

You are cabling a stack with multipath HA, tri-path HA, multipath, single-path HA, or single-path connectivity to the controllers

Cable the shelf-to-shelf connections as “standard” connectivity (using IOM ports 3 and 1):

Beginning with the logical first shelf in the stack, connect IOM A port 3 to the next shelf's IOM A port 1 until each IOM A in the stack is connected.
Repeat substep i for IOM B.

You are cabling a stack with quad-path HA or quad-path connectivity to the controllers

Cable the shelf-to-shelf connections as “double-wide” connectivity: You cable the standard connectivity using IOM ports 3 and 1 and then the double-wide connectivity using IOM ports 4 and 2.

Beginning with the logical first shelf in the stack, connect IOM A port 3 to the next shelf's IOM A port 1 until each IOM A in the stack is connected.
Beginning with the logical first shelf in the stack, connect IOM A port 4 to the next shelf's IOM A port 2 until each IOM A in the stack is connected.
Repeat substeps i and ii for IOM B.

Check the controller-to-stack cabling worksheets and cabling examples to see whether a completed worksheet exists for your configuration.

Controller-to-stack cabling worksheets and cabling examples for platforms with internal storage

Controller-to-stack cabling worksheets and cabling examples for multipath HA configurations

Controller-to-stack cabling worksheet and cabling example for a quad-path HA configuration with two quad-port SAS HBAs
If there is a completed worksheet for your configuration, cable the controller-to-stack connections using the completed worksheet; otherwise, go to the next substep.
If there is no completed worksheet for your configuration, fill out the appropriate worksheet template, and then cable the controller-to-stack connections using the completed worksheet.

Controller-to-stack cabling worksheet template for multipathed connectivity

Controller-to-stack cabling worksheet template for quad-pathed connectivity
Verify that all cables are securely fastened.

If you are hot-adding one or more disk shelves to an end (the logical first or last disk shelf) of an existing stack, complete the applicable substeps for your configuration; otherwise, go to the next step.

Make sure that you wait at least 70 seconds between disconnecting a cable and reconnecting it, and if you are replacing a cable for a longer one.

If you are…	Then…
Hot-adding a disk shelf to an end of a stack that has multipath HA, tri-path HA, multipath, quad-path HA, or quad-path connectivity to the controllers	Disconnect any cables from IOM A of the disk shelf at the end of the stack that are connected to any controllers; otherwise, go to substep e. Leave the other end of these cables connected to the controllers, or replace cables with longer cables if needed. Cable the shelf-to-shelf connection(s) between IOM A of the disk shelf at the end of the stack and IOM A of the disk shelf you are hot-adding. Reconnect any cables that you removed in substep a to the same port(s) on IOM A of the disk shelf you are hot-adding; otherwise, go to the next substep. Verify that all cables are securely fastened. Repeat substeps a through d for IOM B; otherwise, go to Step 4.
Hot-adding a disk shelf to an end of the stack in a single-path HA or single-path configuration, as applicable to AFF A200, AFF A220, FAS2600 series and FAS2700 systems. These instructions are for hot-adding to the end of the stack that does not have controller-to-stack connections.	Cable the shelf-to-shelf connection between IOM A of the disk shelf in the stack and IOM A of the disk shelf you are hot-adding. Verify that the cable is securely fastened. Repeat applicable substeps for IOM B.

If you are…

Then…

Hot-adding a disk shelf to an end of a stack that has multipath HA, tri-path HA, multipath, quad-path HA, or quad-path connectivity to the controllers

Disconnect any cables from IOM A of the disk shelf at the end of the stack that are connected to any controllers; otherwise, go to substep e.

Leave the other end of these cables connected to the controllers, or replace cables with longer cables if needed.
Cable the shelf-to-shelf connection(s) between IOM A of the disk shelf at the end of the stack and IOM A of the disk shelf you are hot-adding.
Reconnect any cables that you removed in substep a to the same port(s) on IOM A of the disk shelf you are hot-adding; otherwise, go to the next substep.
Verify that all cables are securely fastened.
Repeat substeps a through d for IOM B; otherwise, go to Step 4.

Hot-adding a disk shelf to an end of the stack in a single-path HA or single-path configuration, as applicable to AFF A200, AFF A220, FAS2600 series and FAS2700 systems.

These instructions are for hot-adding to the end of the stack that does not have controller-to-stack connections.

Cable the shelf-to-shelf connection between IOM A of the disk shelf in the stack and IOM A of the disk shelf you are hot-adding.
Verify that the cable is securely fastened.
Repeat applicable substeps for IOM B.

If you hot-added a disk shelf with mini-SAS HD SAS optical cables to a stack of disk shelves connected with SAS copper cables, replace the SAS copper cables; otherwise, go to the next step.

The stack must meet the requirements stated in the Requirements for hot-adding disk shelves with IOM12/IOM12B modules section of this procedure.

Replace cables one at a time and make sure that you wait at least 70 seconds between disconnecting a cable and connecting a new one.
Download and run Config Advisor to verify that your SAS connections are cabled correctly.

NetApp Downloads: Config Advisor

If any SAS cabling errors are generated, follow the corrective actions provided.

Verify SAS connectivity for each hot-added disk shelf: storage shelf show -shelf shelf_name -connectivity

You must run this command for each disk shelf you hot-added.

For example, the following output shows hot-added disk shelf 2.5 is connected to initiator ports 1a and 0d (port pair 1a/0d) on each controller (in a FAS8080 multipath HA configuration with one quad-port SAS HBA):

cluster1::> storage shelf show -shelf 2.5 -connectivity

           Shelf Name: 2.5
             Stack ID: 2
             Shelf ID: 5
            Shelf UID: 40:0a:09:70:02:2a:2b
        Serial Number: 101033373
          Module Type: IOM12
                Model: DS224C
         Shelf Vendor: NETAPP
           Disk Count: 24
      Connection Type: SAS
          Shelf State: Online
               Status: Normal

Paths:

Controller     Initiator   Initiator Side Switch Port   Target Side Switch Port   Target Port   TPGN
------------   ---------   --------------------------   -----------------------   -----------   ------
stor-8080-1    1a           -                           -                          -             -
stor-8080-1    0d           -                           -                          -             -
stor-8080-2    1a           -                           -                          -             -
stor-8080-2    0d           -                           -                          -             -

Errors:
------
-

If you disabled disk ownership automatic assignment in Step 1, manually assign disk ownership, and then reenable disk ownership automatic assignment if needed:
1. Display all unowned disks:storage disk show -container-type unassigned
2. Assign each disk:storage disk assign -disk disk_name -owner owner_name
  
  You can use the wildcard character to assign more than one disk at once.
3. Reenable disk ownership automatic assignment if needed:storage disk option modify -node node_name -autoassign on
  
  You need to reenable disk ownership automatic assignment on both controllers in an HA pair.
If your configuration is running in-band ACP, verify that in-band ACP was automatically enabled on hot-added disk shelves: storage shelf acp show

In the output, “in-band” is listed as “active” for each node.

Move or transport DS460C shelves

If in the future, you move DS460C shelves to a different part of the data center or transport the shelves to a different location, you need to remove the drives from the drive drawers to avoid possible damage to the drive drawers and drives.

If when you installed DS460C shelves as part of your shelf hot-add, you saved the drive packaging materials, use these to repackage the drives before moving them.

If you did not save the packaging materials, you should place drives on cushioned surfaces or use alternate cushioned packaging. Never stack drives on top of each other.
Before handling drives, wear an ESD wrist strap grounded to an unpainted surface on your storage enclosure chassis.

If a wrist strap is unavailable, touch an unpainted surface on your storage enclosure chassis before handling a drive.
You should take steps to handle drives carefully:
- Always use two hands when removing, installing, or carrying a drive to support its weight.
  
  Do not place hands on the drive boards exposed on the underside of the drive carrier.
- Be careful not to bump drives against other surfaces.
- Drives should be kept away from magnetic devices.
  
  Magnetic fields can destroy all data on a drive and cause irreparable damage to the drive circuitry.