Hot-adding a stack of SAS disk shelves and bridges to a MetroCluster system

04/11/2024 Contributors

PDFs

You can hot-add (nondisruptively add) an entire stack, including the bridges, to the MetroCluster system. There must be available ports on the FC switches and you must update switch zoning to reflect the changes.

About this task

This procedure can be used to add a stack using FibreBridge 7600N or 7500N bridges.
This procedure is written with the assumption that you are using the recommended bridge management interfaces: the ATTO ExpressNAV GUI and the ATTO QuickNAV utility.
- You use the ATTO ExpressNAV GUI to configure and manage a bridge, and to update the bridge firmware. You use the ATTO QuickNAV utility to configure the bridge Ethernet management 1 port.
- You can use other management interfaces, if needed. These options include using a serial port or Telnet to configure and manage a bridge, and to configure the Ethernet management 1 port, and using FTP to update the bridge firmware. If you choose any of these management interfaces, your system must meet the applicable requirements in Other bridge management interfaces

Preparing to hot-add a stack of SAS disk shelves and bridges

Preparing to hot-add a stack of SAS disk shelves and a pair of bridges involves downloading documents as well as the disk drive and disk shelf firmware.

Before you begin

Your system must be a supported configuration and must be running a supported version of ONTAP.

NetApp Interoperability Matrix Tool
All disk drives and disk shelves in the system must be running the latest firmware version.

You might want to update the disk and shelf firmware throughout the MetroCluster configuration prior to adding shelves.

Upgrade, revert, or downgrade
Each FC switch must have one FC port available for one bridge to connect to it.

You might need to upgrade the FC switch depending on the FC switch compatibility.
The computer you are using to set up the bridges must be running an ATTO supported web browser to use the ATTO ExpressNAV GUI: Internet Explorer 8 or 9, or Mozilla Firefox 3.

The ATTO Product Release Notes have an up-to-date list of supported web browsers. You can access this document using the information in the steps.

Steps

Download or view the following documents from the NetApp Support Site:
- NetApp Interoperability Matrix Tool
- The Installation and Service Guide for your disk shelf model.
Download content from the ATTO website and from the NetApp website:
1. Go to the ATTO FibreBridge Description page.
2. Using the link on the ATTO FibreBridge Description page, access the ATTO web site and download the following:
  - ATTO FibreBridge Installation and Operation Manual for your bridge model.
  - ATTO QuickNAV utility (to the computer you are using for setup).
3. Go to the ATTO FibreBridge Firmware Download page by clicking Continue at the end of the ATTO FibreBridge Description page, and then do the following:
  - Download the bridge firmware file as directed on the download page.
    
    In this step, you are only completing the download portion of the instructions provided in the links. You update the firmware on each bridge later, when instructed to do so in the Hot-adding the stack of shelves section.
  - Make a copy of the ATTO FibreBridge Firmware Download page and release notes for reference later.

Download the latest disk and disk shelf firmware, and make a copy of the installation portion of the instructions for reference later.

All disk shelves in the MetroCluster configuration (both the new shelves and existing shelves) must be running the same firmware version.

In this step, you are only completing the download portion of the instructions provided in the links and making a copy of the installation instructions. You update the firmware on each disk and disk shelf later, when instructed to do so in the Hot-adding the stack of shelves section.

Download the disk firmware and make a copy of the disk firmware instructions for reference later.

NetApp Downloads: Disk Drive Firmware
Download the disk shelf firmware and make a copy of the disk shelf firmware instructions for reference later.

NetApp Downloads: Disk Shelf Firmware

Gather the hardware and information needed to use the recommended bridge management interfaces—the ATTO ExpressNAV GUI and ATTO QuickNAV utility:
1. Acquire a standard Ethernet cable to connect from the bridge Ethernet management 1 port to your network.
2. Determine a non-default user name and password for accessing the bridges.
  
  It is recommended that you change the default user name and password.
3. Obtain an IP address, subnet mask, and gateway information for the Ethernet management 1 port on each bridge.
4. Disable VPN clients on the computer you are using for setup.
  
  Active VPN clients cause the QuickNAV scan for bridges to fail.
Acquire four screws for each bridge to flush-mount the bridge “L” brackets securely to the front of the rack.

The openings in the bridge “L” brackets are compliant with rack standard ETA-310-X for 19-inch (482.6 mm) racks.
If necessary, update the FC switch zoning to accommodate the new bridges that are being added to the configuration.

If you are using the Reference Configuration Files provided by NetApp, the zones have been created for all ports, so you do not need to make any zoning updates. There must be a storage zone for each switch port that connects to the FC ports of the bridge.

Hot-adding a stack of SAS disk shelves and bridges

You can hot-add a stack of SAS disk shelves and bridges to increase the capacity of the bridges.

The system must meet all of the requirements to hot-add a stack of SAS disk shelves and bridges.

Preparing to hot-add a stack of SAS disk shelves and bridges

Hot-adding a stack of SAS disk shelves and bridges is a nondisruptive procedure if all of the interoperability requirements are met.

NetApp Interoperability Matrix Tool

Using the Interoperability Matrix Tool to find MetroCluster information
Multipath HA is the only supported configuration for MetroCluster systems that are using bridges.

Both controller modules must have access through the bridges to the disk shelves in each stack.
You should hot-add an equal number of disk shelves at each site.
If you will be using in-band management of the bridge rather than IP management, the steps for configuring the Ethernet port and IP settings can be skipped, as noted in the relevant steps.

Beginning with ONTAP 9.8, the storage bridge command is replaced with system bridge. The following steps show the storage bridge command, but if you are running ONTAP 9.8 or later, the system bridge command is preferred.

If you insert a SAS cable into the wrong port, when you remove the cable from a SAS port, you must wait at least 120 seconds before plugging the cable into a different SAS port. If you fail to do so, the system will not recognize that the cable has been moved to another port.

Steps

Properly ground yourself.

From the console of either controller module, check whether your system has disk autoassignment enabled:

storage disk option show

The Auto Assign column indicates whether disk autoassignment is enabled.

Node        BKg. FW. Upd.  Auto Copy   Auto Assign  Auto Assign Policy
----------  -------------  ----------  -----------  ------------------
node_A_1             on           on           on           default
node_A_2             on           on           on           default
2 entries were displayed.

Disable the switch ports for the new stack.
If configuring for in-band management, connect a cable from FibreBridge RS-232 serial port to the serial (COM) port on a personal computer.

The serial connection will be used for initial configuration, and then in-band management via ONTAP and the FC ports can be used to monitor and manage the bridge.
If configuring for IP management, configure the Ethernet management 1 port for each bridge by following the procedure in section 2.0 of the ATTO FibreBridge Installation and Operation Manual for your bridge model.

In systems running ONTAP 9.5 or later, in-band management can be used to access the bridge via the FC ports rather than the Ethernet port. Beginning with ONTAP 9.8, only in-band management is supported and SNMP management is deprecated.

When running QuickNAV to configure an Ethernet management port, only the Ethernet management port that is connected by the Ethernet cable is configured. For example, if you also wanted to configure the Ethernet management 2 port, you would need to connect the Ethernet cable to port 2 and run QuickNAV.

Configure the bridge.

If you retrieved the configuration information from the old bridge, use the information to configure the new bridge.

Be sure to make note of the user name and password that you designate.

The ATTO FibreBridge Installation and Operation Manual for your bridge model has the most current information on available commands and how to use them.

Do not configure time synchronization on ATTO FibreBridge 7600N or 7500N. The time synchronization for ATTO FibreBridge 7600N or 7500N is set to the cluster time after the bridge is discovered by ONTAP. It is also synchronized periodically once a day. The time zone used is GMT and is not changeable.

If configuring for IP management, configure the IP settings of the bridge.

To set the IP address without the QuickNAV utility, you need to have a serial connection to the FibreBridge.

If using the CLI, you must run the following commands:

set ipaddress mp1 ip-address

set ipsubnetmask mp1 subnet-mask

set ipgateway mp1 x.x.x.x

set ipdhcp mp1 disabled

set ethernetspeed mp1 1000
Configure the bridge name.

The bridges should each have a unique name within the MetroCluster configuration.

Example bridge names for one stack group on each site:
- bridge_A_1a
- bridge_A_1b
- bridge_B_1a
- bridge_B_1b If using the CLI, you must run the following command:
  
  set bridgename bridgename
If running ONTAP 9.4 or earlier, enable SNMP on the bridge:
set SNMP enabled

In systems running ONTAP 9.5 or later, in-band management can be used to access the bridge via the FC ports rather than the Ethernet port. Beginning with ONTAP 9.8, only in-band management is supported and SNMP management is deprecated.

Configure the bridge FC ports.

Configure the data rate/speed of the bridge FC ports.

The supported FC data rate depends on your model bridge.

The FibreBridge 7600N bridge supports up to 32, 16, or 8 Gbps.

The FibreBridge 7500N bridge supports up to 16, 8, or 4 Gbps.

The FCDataRate speed you select is limited to the maximum speed supported by both the bridge and the switch to which the bridge port connects. Cabling distances must not exceed the limitations of the SFPs and other hardware.

If using the CLI, you must run the following command:

set FCDataRate port-number port-speed

If you are configuring a FibreBridge 7500N bridge, configure the connection mode that the port uses to "ptp".

The FCConnMode setting is not required when configuring a FibreBridge 7600N bridge.

If using the CLI, you must run the following command:

set FCConnMode port-number ptp
If you are configuring a FibreBridge 7600N or 7500N bridge, you must configure or disable the FC2 port.
- If you are using the second port, you must repeat the previous substeps for the FC2 port.
- If you are not using the second port, then you must disable the port:
  FCPortDisable port-number
If you are configuring a FibreBridge 7600N or 7500N bridge, disable the unused SAS ports:
SASPortDisable sas-port

SAS ports A through D are enabled by default. You must disable the SAS ports that are not being used. If only SAS port A is used, then SAS ports B, C, and D must be disabled.

Secure access to the bridge and save the bridge's configuration.
1. From the controller prompt check the status of the bridges:
  
  storage bridge show
  
  The output shows which bridge is not secured.
2. Check the status of the unsecured bridge's ports:
  info
  
  The output shows the status of Ethernet ports MP1 and MP2.
3. If Ethernet port MP1 is enabled, run the following command:
  set EthernetPort mp1 disabled
  
  If Ethernet port MP2 is also enabled, repeat the previous substep for port MP2.
4. Save the bridge's configuration.
  
  You must run the following commands:
  
  SaveConfiguration
  
  FirmwareRestart
  
  You are prompted to restart the bridge.
Update the FibreBridge firmware on each bridge.

If the new bridge is the same type as the partner bridge upgrade to the same firmware as the partner bridge. If the new bridge is a different type to the partner bridge, upgrade to the latest firmware supported by the bridge and version of ONTAP. See the section "Updating firmware on a FibreBridge bridge" in MetroCluster Maintenance.
Cable the disk shelves to the bridges:
1. Daisy-chain the disk shelves in each stack.
  
  The Installation Guide for your disk shelf model provides detailed information about daisy-chaining disk shelves.
2. For each stack of disk shelves, cable IOM A of the first shelf to SAS port A on FibreBridge A, and then cable IOM B of the last shelf to SAS port A on FibreBridge B.
  
  Fabric-attached MetroCluster installation and configuration
  
  Stretch MetroCluster installation and configuration
  
  Each bridge has one path to its stack of disk shelves; bridge A connects to the A-side of the stack through the first shelf, and bridge B connects to the B-side of the stack through the last shelf.
  
  The bridge SAS port B is disabled.

Verify that each bridge can detect all of the disk drives and disk shelves to which the bridge is connected.

If you are using the…	Then…
ATTO ExpressNAV GUI	In a supported web browser, enter the IP address of a bridge in the browser box. You are brought to the ATTO FibreBridge home page, which has a link. Click the link, and then enter your user name and the password that you designated when you configured the bridge. The ATTO FibreBridge status page appears with a menu to the left. Click Advanced in the menu. View the connected devices: `sastargets` Click Submit.
Serial port connection	View the connected devices: `sastargets`

If you are using the…

Then…

ATTO ExpressNAV GUI

In a supported web browser, enter the IP address of a bridge in the browser box.

You are brought to the ATTO FibreBridge home page, which has a link.
Click the link, and then enter your user name and the password that you designated when you configured the bridge.

The ATTO FibreBridge status page appears with a menu to the left.
Click Advanced in the menu.
View the connected devices:
sastargets
Click Submit.

Serial port connection

View the connected devices:

sastargets

The output shows the devices (disks and disk shelves) to which the bridge is connected. The output lines are sequentially numbered so that you can quickly count the devices.

If the text response truncated appears at the beginning of the output, you can use Telnet to connect to the bridge, and then view all of the output by using the sastargets command.

The following output shows that 10 disks are connected:

Tgt VendorID ProductID        Type        SerialNumber
  0 NETAPP   X410_S15K6288A15 DISK        3QP1CLE300009940UHJV
  1 NETAPP   X410_S15K6288A15 DISK        3QP1ELF600009940V1BV
  2 NETAPP   X410_S15K6288A15 DISK        3QP1G3EW00009940U2M0
  3 NETAPP   X410_S15K6288A15 DISK        3QP1EWMP00009940U1X5
  4 NETAPP   X410_S15K6288A15 DISK        3QP1FZLE00009940G8YU
  5 NETAPP   X410_S15K6288A15 DISK        3QP1FZLF00009940TZKZ
  6 NETAPP   X410_S15K6288A15 DISK        3QP1CEB400009939MGXL
  7 NETAPP   X410_S15K6288A15 DISK        3QP1G7A900009939FNTT
  8 NETAPP   X410_S15K6288A15 DISK        3QP1FY0T00009940G8PA
  9 NETAPP   X410_S15K6288A15 DISK        3QP1FXW600009940VERQ

Verify that the command output shows that the bridge is connected to all of the appropriate disks and disk shelves in the stack.

If the output is…	Then…
Correct	Repeat Step 11 for each remaining bridge.
Not correct	Check for loose SAS cables or correct the SAS cabling by repeating Step 10. Repeat Step 11.

If the output is…

Then…

Correct

Repeat Step 11 for each remaining bridge.

Not correct

Check for loose SAS cables or correct the SAS cabling by repeating Step 10.
Repeat Step 11.

If you are configuring a fabric-attached MetroCluster configuration, cable each bridge to the local FC switches, using the cabling shown in the table for your configuration, switch model, and FC-to-SAS bridge model:

Brocade and Cisco switches use different port numbering, as shown in the following tables.

On Brocade switches, the first port is numbered “0”.

On Cisco switches, the first port is numbered “1”.

Configurations using FibreBridge 7500N or 7600N using both FC ports (FC1 and FC2)

DR GROUP 1

Brocade 6505

Brocade 6510, Brocade DCX 8510-8

Brocade 6520

Brocade G620, Brocade G620-1, Brocade G630, Brocade G630-1

Brocade G720

Component

Port

Switch 1

Switch 2

Switch 1

Switch 2

Switch 1

Switch 2

Switch 1

Switch 2

Switch 1

Switch 2

Stack 1

bridge_x_1a

FC1

FC2

bridge_x_1B

FC1

FC2

Stack 2

bridge_x_2a

FC1

FC2

bridge_x_2B

FC1

FC2

Stack 3

bridge_x_3a

FC1

FC2

bridge_x_3B

FC1

FC2

Stack y

bridge_x_ya

FC1

FC2

bridge_x_yb

FC1

FC2

Additional bridges can be cabled to ports 16, 17, 20 and 21 in G620, G630, G620-1, and G630-1 switches.

Configurations using FibreBridge 7500N or 7600N using both FC ports (FC1 and FC2)

DR GROUP 2

Brocade G620, Brocade G620-1, Brocade G630, Brocade G630-1

Brocade 6510, Brocade DCX 8510-8

Brocade 6520

Brocade G720

Component

Port

Switch 1

Switch 2

Switch 1

Switch 2

Switch 1

Switch 2

Switch 1

switch 2

Stack 1

bridge_x_51a

FC1

FC2

bridge_x_51b

FC1

FC2

Stack 2

bridge_x_52a

FC1

FC2

bridge_x_52b

FC1

FC2

Stack 3

bridge_x_53a

FC1

FC2

bridge_x_53b

FC1

FC2

Stack y

bridge_x_5ya

FC1

FC2

bridge_x_5yb

FC1

FC2

Additional bridges can be cabled to ports 36 - 39 in G620, G630, G620-1, and G-630-1 switches.

Configurations using FibreBridge 7500N or 7600N using one FC port (FC1 or FC2) only

DR GROUP 1

Brocade 6505

Brocade 6510, Brocade DCX 8510-8

Brocade 6520

Brocade G620, brocade G620-1, Brocade G630, Brocade G630-1

Brocade G720

Component

Port

Switch 1

Switch 2

Switch 1

Switch 2

Switch 1

Switch 2

Switch 1

Switch 2

Switch 1

Switch 2

Stack 1

bridge_x_1a

bridge_x_1b

Stack 2

bridge_x_2a

bridge_x_2b

Stack 3

bridge_x_3a

bridge_x_4b

Stack y

bridge_x_ya

bridge_x_yb

Additional bridges can be cabled to ports 12 - 17, 20 and 21 in G620, G630, G620-1, and G630-1 switches. Additional bridges can be cabled to ports 16 - 17, 20 and 21 G720 switches.

Configurations using FibreBridge 7500N or 7600N using one FC port (FC1 or FC2) only

DR GROUP 2

Brocade G720

Brocade G620, Brocade G620-1, Brocade G630, Brocade G630-1

Brocade 6510, Brocade DCX 8510-8

Brocade 6520

Stack 1

bridge_x_51a

bridge_x_51b

Stack 2

bridge_x_52a

bridge_x_52b

Stack 3

bridge_x_53a

bridge_x_54b

Stack y

bridge_x_ya

bridge_x_yb

Additional bridges can be cabled to ports 32 through 39 in G620, G630, G620-1, and G630-1 switches. Additional bridges can be cabled to ports 36 through 39 in G720 switches.

If you are configuring a bridge-attached MetroCluster system, cable each bridge to the controller modules:
1. Cable FC port 1 of the bridge to a 16 Gb or 8 Gb FC port on the controller module in cluster_A.
2. Cable FC port 2 of the bridge to the same speed FC port of the controller module in cluster_A.
3. Repeat these substeps on other subsequent bridges until all of the bridges have been cabled.
Update the disk drive firmware to the most current version from the system console:

disk_fw_update

You must run this command on both controller modules.

NetApp Downloads: Disk Drive Firmware
Update the disk shelf firmware to the most current version by using the instructions for the downloaded firmware.

You can run the commands in the procedure from the system console of either controller module.

NetApp Downloads: Disk Shelf Firmware

If your system does not have disk autoassignment enabled, assign disk drive ownership.

Disk and aggregate management

If you are splitting the ownership of a single stack of disk shelves among multiple controller modules, you must disable disk autoassignment on both nodes in the cluster (storage disk option modify -autoassign off *) before assigning disk ownership; otherwise, when you assign any single disk drive, the remaining disk drives might be automatically assigned to the same controller module and pool.

You must not add disk drives to aggregates or volumes until after the disk drive firmware and disk shelf firmware have been updated and the verification steps in this task have been completed.

Enable the switch ports for the new stack.
Verify the operation of the MetroCluster configuration in ONTAP:
1. Check whether the system is multipathed:
  node run -node node-name sysconfig -a
2. Check for any health alerts on both clusters:
  system health alert show
3. Confirm the MetroCluster configuration and that the operational mode is normal:
  metrocluster show
4. Perform a MetroCluster check:
  metrocluster check run
5. Display the results of the MetroCluster check:
  metrocluster check show
6. Check for any health alerts on the switches (if present):
  storage switch show
7. Run Config Advisor.
  
  NetApp Downloads: Config Advisor
8. After running Config Advisor, review the tool's output and follow the recommendations in the output to address any issues discovered.
If applicable, repeat this procedure for the partner site.

Related information

In-band management of the FC-to-SAS bridges

Hot-adding a stack of SAS disk shelves and bridges to a MetroCluster system

Creating your file...

Preparing to hot-add a stack of SAS disk shelves and bridges

Hot-adding a stack of SAS disk shelves and bridges