Skip to main content
ONTAP MetroCluster

Replacing a single FC-to-SAS bridge

Contributors netapp-aoife netapp-pcarriga netapp-aherbin netapp-martyh netapp-thomi netapp-ranuk
PDFs

You can nondisruptively replace a bridge with a same model bridge or with a new model bridge.

Before you begin

You need the admin password and access to an FTP or SCP server.

About this task

This procedure is nondisruptive and takes approximately 60 minutes to complete.

This procedure uses the bridge CLI to configure and manage a bridge, and to update the bridge firmware and the ATTO QuickNAV utility to configure the bridge Ethernet management 1 port. You can use other interfaces if they meet the requirements.

Requirements for using other interfaces to configure and manage FibreBridge bridges

Related information

Replacing a pair of FibreBridge 6500N bridges with 7600N or 7500N bridges

Verifying storage connectivity

Before replacing bridges, you should verify bridge and storage connectivity. Familiarizing yourself with the command output enables you to subsequently confirm connectivity after making configuration changes.

About this task

You can issue these commands from the admin prompt of any of the controller modules in the MetroCluster configuration at the site undergoing maintenance.

Steps

  1. Confirm connectivity to the disks by entering the following command on any one of the MetroCluster nodes:

    run local sysconfig -v

    The output shows the disks attached to the initiator ports on the controller, and identifies the shelves connected to the FC-to-SAS bridges:

    node_A_1> run local sysconfig -v
NetApp Release 9.3.2X18: Sun Dec 13 01:23:24 PST 2017
System ID: 4068741258 (node_A_1); partner ID: 4068741260 (node_B_1)
System Serial Number: 940001025471 (node_A_1)
System Rev: 70
System Storage Configuration: Multi-Path HA**<=== Configuration should be multi-path HA**
.
.
.
slot 0: FC Host Adapter 0g (QLogic 8324 rev. 2, N-port, <UP>)**<=== Initiator port**
		Firmware rev:      7.5.0
		Flash rev:         0.0.0
		Host Port Id:      0x60130
		FC Node Name:      5:00a:098201:bae312
		FC Port Name:      5:00a:098201:bae312
		SFP Vendor:        UTILITIES CORP.
		SFP Part Number:   FTLF8529P3BCVAN1
		SFP Serial Number: URQ0Q9R
		SFP Capabilities:  4, 8 or 16 Gbit
		Link Data Rate:    16 Gbit
		Switch Port:       brcd6505-fcs40:1
  **<List of disks visible to port\>**
		 ID     Vendor   Model            FW    Size
		brcd6505-fcs29:12.126L1527     : NETAPP   X302_HJUPI01TSSM NA04 847.5GB (1953525168 512B/sect)
		brcd6505-fcs29:12.126L1528     : NETAPP   X302_HJUPI01TSSA NA02 847.5GB (1953525168 512B/sect)
		.
		.
		.
		**<List of FC-to-SAS bridges visible to port\>**
		FC-to-SAS Bridge:
		brcd6505-fcs40:12.126L0        : ATTO     FibreBridge6500N 1.61  FB6500N102980
		brcd6505-fcs42:13.126L0        : ATTO     FibreBridge6500N 1.61  FB6500N102980
		brcd6505-fcs42:6.126L0         : ATTO     FibreBridge6500N 1.61  FB6500N101167
		brcd6505-fcs42:7.126L0         : ATTO     FibreBridge6500N 1.61  FB6500N102974
		.
		.
		.
  **<List of storage shelves visible to port\>**
		brcd6505-fcs40:12.shelf6: DS4243  Firmware rev. IOM3 A: 0200  IOM3 B: 0200
		brcd6505-fcs40:12.shelf8: DS4243  Firmware rev. IOM3 A: 0200  IOM3 B: 0200
		.
		.
		.

Hot-swapping a bridge with a replacement bridge of the same model

You can hot-swap a failed bridge with another bridge of the same model.

About this task

If you will be using in-band management of the bridge rather than IP management, the steps for configuring the Ethernet port and IP settings can be skipped, as noted in the relevant steps.

Note Beginning with ONTAP 9.8, the storage bridge command is replaced with system bridge. The following steps show the storage bridge command, but if you are running ONTAP 9.8 or later, the system bridge command is preferred.
Steps

  1. If the old bridge is accessible, you can retrieve the configuration information.

    If…​

    Then…​

    You are using IP management

    Connect to the old bridge with a Telnet connection and copy the output of the bridge configuration.

    You are using in-band management

    Use the ONTAP CLI to retrieve the configuration information with the following commands:

    storage bridge run-cli -name bridge-name -command "info"

    storage bridge run-cli -name bridge-name -command "sasportlist"

    1. Enter the command:

      storage bridge run-cli -name bridge_A1 -command "info"

      info

Device Status         = Good
Unsaved Changes       = None
Device                = "FibreBridge 7500N"
Serial Number         = FB7500N100000
Device Version        = 3.10
Board Revision        = 7
Build Number          = 007A
Build Type            = Release
Build Date            = "Aug 20 2019" 11:01:24
Flash Revision        = 0.02
Firmware Version      = 3.10
BCE Version (FPGA 1)  = 15
BAU Version (FPGA 2)  = 33
User-defined name     = "bridgeA1"
World Wide Name       = 20 00 00 10 86 A1 C7 00
MB of RAM Installed   = 512
FC1 Node Name         = 20 00 00 10 86 A1 C7 00
FC1 Port Name         = 21 00 00 10 86 A1 C7 00
FC1 Data Rate         = 16Gb
FC1 Connection Mode   = ptp
FC1 FW Revision       = 11.4.337.0
FC2 Node Name         = 20 00 00 10 86 A1 C7 00
FC2 Port Name         = 22 00 00 10 86 A1 C7 00
FC2 Data Rate         = 16Gb
FC2 Connection Mode   = ptp
FC2 FW Revision       = 11.4.337.0
SAS FW Revision       = 3.09.52
MP1 IP Address        = 10.10.10.10
MP1 IP Subnet Mask    = 255.255.255.0
MP1 IP Gateway        = 10.10.10.1
MP1 IP DHCP           = disabled
MP1 MAC Address       = 00-10-86-A1-C7-00
MP2 IP Address        = 0.0.0.0 (disabled)
MP2 IP Subnet Mask    = 0.0.0.0
MP2 IP Gateway        = 0.0.0.0
MP2 IP DHCP           = enabled
MP2 MAC Address       = 00-10-86-A1-C7-01
SNMP                  = enabled
SNMP Community String = public
PS A Status           = Up
PS B Status           = Up
Active Configuration  = NetApp

Ready.

    2. Enter the command:

      storage bridge run-cli -name bridge_A1 -command "sasportlist"

      SASPortList

;Connector      PHY     Link            Speed   SAS Address
;=============================================================
Device  A       1       Up              6Gb     5001086000a1c700
Device  A       2       Up              6Gb     5001086000a1c700
Device  A       3       Up              6Gb     5001086000a1c700
Device  A       4       Up              6Gb     5001086000a1c700
Device  B       1       Disabled        12Gb    5001086000a1c704
Device  B       2       Disabled        12Gb    5001086000a1c704
Device  B       3       Disabled        12Gb    5001086000a1c704
Device  B       4       Disabled        12Gb    5001086000a1c704
Device  C       1       Disabled        12Gb    5001086000a1c708
Device  C       2       Disabled        12Gb    5001086000a1c708
Device  C       3       Disabled        12Gb    5001086000a1c708
Device  C       4       Disabled        12Gb    5001086000a1c708
Device  D       1       Disabled        12Gb    5001086000a1c70c
Device  D       2       Disabled        12Gb    5001086000a1c70c
Device  D       3       Disabled        12Gb    5001086000a1c70c
Device  D       4       Disabled        12Gb    5001086000a1c70c

  2. If the bridge is in a fabric-attached MetroCluster configuration, disable all of the switch ports that connect to the bridge FC port or ports.

  3. From the ONTAP cluster prompt, remove the bridge undergoing maintenance from health monitoring:

    1. Remove the bridge:
      storage bridge remove -name bridge-name

    2. View the list of monitored bridges and confirm that the removed bridge is not present:
      storage bridge show

  4. Properly ground yourself.

  5. Power down the ATTO bridge and remove the power cables connected to the bridge.

  6. Disconnect the cables that are connected to the old bridge.

    You should make note of the port to which each cable was connected.

  7. Remove the old bridge from the rack.

  8. Install the new bridge into the rack.

  9. Reconnect the power cord and, if configuring for IP access to the bridge, a shielded Ethernet cable.

    Important You must not reconnect the SAS or FC cables at this time.

  10. Connect the bridge to a power source, and then turn it on.

    The bridge Ready LED might take up to 30 seconds to illuminate, indicating that the bridge has completed its power-on self test sequence.

  11. If configuring for in-band management, connect a cable from FibreBridge RS-232 serial port to the serial (COM) port on a personal computer.

    The serial connection will be used for initial configuration, and then in-band management via ONTAP and the FC ports can be used to monitor and manage the bridge.

  12. If configuring for IP management, configure the Ethernet management 1 port for each bridge by following the procedure in section 2.0 of the ATTO FibreBridge Installation and Operation Manual for your bridge model.

    In systems running ONTAP 9.5 or later, in-band management can be used to access the bridge via the FC ports rather than the Ethernet port. Beginning with ONTAP 9.8, only in-band management is supported and SNMP management is deprecated.

    When running QuickNAV to configure an Ethernet management port, only the Ethernet management port that is connected by the Ethernet cable is configured. For example, if you also wanted to configure the Ethernet management 2 port, you would need to connect the Ethernet cable to port 2 and run QuickNAV.

  13. Configure the bridge.

    If you retrieved the configuration information from the old bridge, use the information to configure the new bridge.

    Be sure to make note of the user name and password that you designate.

    The ATTO FibreBridge Installation and Operation Manual for your bridge model has the most current information on available commands and how to use them.

    Note Do not configure time synchronization on ATTO FibreBridge 7600N or 7500N. The time synchronization for ATTO FibreBridge 7600N or 7500N is set to the cluster time after the bridge is discovered by ONTAP. It is also synchronized periodically once a day. The time zone used is GMT and is not changeable.

    1. If configuring for IP management, configure the IP settings of the bridge.

      To set the IP address without the QuickNAV utility, you need to have a serial connection to the FibreBridge.

      If using the CLI, you must run the following commands:

      set ipaddress mp1 _ip-address

      set ipsubnetmask mp1 subnet-mask

      set ipgateway mp1 x.x.x.x

      set ipdhcp mp1 disabled

      set ethernetspeed mp1 1000

    2. Configure the bridge name.

      The bridges should each have a unique name within the MetroCluster configuration.

      Example bridge names for one stack group on each site:

      • bridge_A_1a

      • bridge_A_1b

      • bridge_B_1a

      • bridge_B_1b

        If using the CLI, you must run the following command:

        set bridgename bridgename

    3. If running ONTAP 9.4 or earlier, enable SNMP on the bridge:

      set SNMP enabled

      In systems running ONTAP 9.5 or later, in-band management can be used to access the bridge via the FC ports rather than the Ethernet port. Beginning with ONTAP 9.8, only in-band management is supported and SNMP management is deprecated.

  14. Configure the bridge FC ports.

    1. Configure the data rate/speed of the bridge FC ports.

      The supported FC data rate depends on your model bridge.

      • The FibreBridge 7600N bridge supports up to 32, 16, or 8 Gbps.

      • The FibreBridge 7500N bridge supports up to 16, 8, or 4 Gbps.

        Note The FCDataRate speed you select is limited to the maximum speed supported by both the bridge and the switch to which the bridge port connects. Cabling distances must not exceed the limitations of the SFPs and other hardware.

        If using the CLI, you must run the following command:

        set FCDataRate port-number port-speed

    2. If you are configuring a FibreBridge 7500N, configure the connection mode that the port uses to "ptp".

      Note The FCConnMode setting is not required when configuring a FibreBridge 7600N bridge.

      If using the CLI, you must run the following command:

      set FCConnMode port-number ptp

    3. If you are configuring a FibreBridge 7600N or 7500N bridge, you must configure or disable the FC2 port.

      • If you are using the second port, you must repeat the previous substeps for the FC2 port.

      • If you are not using the second port, then you must disable the port:

        FCPortDisable port-number

    4. If you are configuring a FibreBridge 7600N or 7500N bridge, disable the unused SAS ports:

      SASPortDisable sas-port

      Note SAS ports A through D are enabled by default. You must disable the SAS ports that are not being used. If only SAS port A is used, then SAS ports B, C, and D must be disabled.

  15. Secure access to the bridge and save the bridge's configuration.

    1. From the controller prompt check the status of the bridges: storage bridge show

      The output shows which bridge is not secured.

    2. Check the status of the unsecured bridge's ports:

      info

      The output shows the status of Ethernet ports MP1 and MP2.

    3. If Ethernet port MP1 is enabled, run the following command:

      set EthernetPort mp1 disabled

      Note If Ethernet port MP2 is also enabled, repeat the previous substep for port MP2.

    4. Save the bridge's configuration.

      You must run the following commands:

      SaveConfiguration

      FirmwareRestart

      You are prompted to restart the bridge.

  16. Connect the FC cables to the same ports on the new bridge.

  17. Update the FibreBridge firmware on each bridge.

    If the new bridge is the same type as the partner bridge, upgrade to the same firmware as the partner bridge. If the new bridge is a different type to the partner bridge, upgrade to the latest firmware supported by the bridge and version of ONTAP. See Updating firmware on a FibreBridge bridge

  18. Reconnect the SAS cables to the same ports on the new bridge.

    You must replace the cables connecting the bridge to the top or bottom of the shelf stack. The FibreBridge 7600N and 7500N bridges require mini-SAS cables for these connections.

    Note Wait at least 10 seconds before connecting the port. The SAS cable connectors are keyed; when oriented correctly into a SAS port, the connector clicks into place and the disk shelf SAS port LNK LED illuminates green. For disk shelves, you insert a SAS cable connector with the pull tab oriented down (on the underside of the connector). For controllers, the orientation of SAS ports can vary depending on the platform model; therefore, the correct orientation of the SAS cable connector varies.

  19. Verify that each bridge can see all of the disk drives and disk shelves to which the bridge is connected.

    If you are using the…​

    Then…​

    ATTO ExpressNAV GUI

    1. In a supported web browser, enter the IP address of the bridge in the browser box.

      You are brought to the ATTO FibreBridge homepage, which has a link.

    2. Click the link, and then enter your user name and the password that you designated when you configured the bridge.

      The ATTO FibreBridge status page appears with a menu to the left.

    3. Click Advanced in the menu.

    4. View the connected devices:

      sastargets

    5. Click Submit.

    Serial port connection

    View the connected devices:

    sastargets

    The output shows the devices (disks and disk shelves) to which the bridge is connected. The output lines are sequentially numbered so that you can quickly count the devices.

    Note If the text response truncated appears at the beginning of the output, you can use Telnet to connect to the bridge, and then view all of the output by using the sastargets command.

    The following output shows that 10 disks are connected:

    Tgt VendorID ProductID        Type SerialNumber
  0 NETAPP   X410_S15K6288A15 DISK 3QP1CLE300009940UHJV
  1 NETAPP   X410_S15K6288A15 DISK 3QP1ELF600009940V1BV
  2 NETAPP   X410_S15K6288A15 DISK 3QP1G3EW00009940U2M0
  3 NETAPP   X410_S15K6288A15 DISK 3QP1EWMP00009940U1X5
  4 NETAPP   X410_S15K6288A15 DISK 3QP1FZLE00009940G8YU
  5 NETAPP   X410_S15K6288A15 DISK 3QP1FZLF00009940TZKZ
  6 NETAPP   X410_S15K6288A15 DISK 3QP1CEB400009939MGXL
  7 NETAPP   X410_S15K6288A15 DISK 3QP1G7A900009939FNTT
  8 NETAPP   X410_S15K6288A15 DISK 3QP1FY0T00009940G8PA
  9 NETAPP   X410_S15K6288A15 DISK 3QP1FXW600009940VERQ

  20. Verify that the command output shows that the bridge is connected to all of the appropriate disks and disk shelves in the stack.

    If the output is…​

    Then…​

    Correct

    Repeat Step 19 for each remaining bridge.

    Not correct

    1. Check for loose SAS cables or correct the SAS cabling by repeating Step 18.

    2. Repeat Step 19.

  21. If the bridge is in a fabric-attached MetroCluster configuration, re-enable the FC switch port that you disabled at the beginning of this procedure.

    This should be the port that connects to the bridge.

  22. From the system console of both controller modules, verify that all of the controller modules have access through the new bridge to the disk shelves (that is, that the system is cabled for Multipath HA):

    run local sysconfig

    Note It might take up to a minute for the system to complete discovery.

    If the output does not indicate Multipath HA, you must correct the SAS and FC cabling because not all of the disk drives are accessible through the new bridge.

    The following output states that the system is cabled for Multipath HA:

    NetApp Release 8.3.2: Tue Jan 26 01:41:49 PDT 2016
System ID: 1231231231 (node_A_1); partner ID: 4564564564 (node_A_2)
System Serial Number: 700000123123 (node_A_1); partner Serial Number: 700000456456 (node_A_2)
System Rev: B0
System Storage Configuration: Multi-Path HA
System ACP Connectivity: NA
    Important When the system is not cabled as Multipath HA, restarting a bridge might cause loss of access to the disk drives and result in a multi-disk panic.

  23. If running ONTAP 9.4 or earlier, verify that the bridge is configured for SNMP.

    If you are using the bridge CLI, run the following command:

    get snmp

  24. From the ONTAP cluster prompt, add the bridge to health monitoring:

    1. Add the bridge, using the command for your version of ONTAP:

      ONTAP version

      Command

      9.5 and later

      storage bridge add -address 0.0.0.0 -managed-by in-band -name bridge-name

      9.4 and earlier

      storage bridge add -address bridge-ip-address -name bridge-name

    2. Verify that the bridge has been added and is properly configured:

      storage bridge show

      It might take as long as 15 minutes to reflect all data because of the polling interval. The ONTAP health monitor can contact and monitor the bridge if the value in the “Status” column is “ok”, and other information, such as the worldwide name (WWN), is displayed.

      The following example shows that the FC-to-SAS bridges are configured:

      controller_A_1::> storage bridge show

Bridge              Symbolic Name Is Monitored  Monitor Status  Vendor Model                Bridge WWN
------------------  ------------- ------------  --------------  ------ -----------------    ----------
ATTO_10.10.20.10  atto01        true          ok              Atto   FibreBridge 7500N   	20000010867038c0
ATTO_10.10.20.11  atto02        true          ok              Atto   FibreBridge 7500N   	20000010867033c0
ATTO_10.10.20.12  atto03        true          ok              Atto   FibreBridge 7500N   	20000010867030c0
ATTO_10.10.20.13  atto04        true          ok              Atto   FibreBridge 7500N   	2000001086703b80

4 entries were displayed

 controller_A_1::>

  25. Verify the operation of the MetroCluster configuration in ONTAP:

    1. Check whether the system is multipathed:
      node run -node node-name sysconfig -a

    2. Check for any health alerts on both clusters:
      system health alert show

    3. Confirm the MetroCluster configuration and that the operational mode is normal:
      metrocluster show

    4. Perform a MetroCluster check:
      metrocluster check run

    5. Display the results of the MetroCluster check:
      metrocluster check show

    6. Check for any health alerts on the switches (if present):
      storage switch show

    7. Run Config Advisor.

      NetApp Downloads: Config Advisor

    8. After running Config Advisor, review the tool's output and follow the recommendations in the output to address any issues discovered.

Related information

In-band management of the FC-to-SAS bridges

Hot-swapping a FibreBridge 7500N with a 7600N bridge

You can hot-swap a FibreBridge 7500N bridge with a 7600N bridge.

About this task

If you will be using in-band management of the bridge rather than IP management, the steps for configuring the Ethernet port and IP settings can be skipped, as noted in the relevant steps.

Note Beginning with ONTAP 9.8, the storage bridge command is replaced with system bridge. The following steps show the storage bridge command, but if you are running ONTAP 9.8 or later, the system bridge command is preferred.
Steps

  1. If the bridge is in a fabric-attached MetroCluster configuration, disable all of the switch ports that connect to the bridge FC port or ports.

  2. From the ONTAP cluster prompt, remove the bridge undergoing maintenance from health monitoring:

    1. Remove the bridge:
      storage bridge remove -name bridge-name

    2. View the list of monitored bridges and confirm that the removed bridge is not present:
      storage bridge show

  3. Properly ground yourself.

  4. Remove the power cables connected to the bridge to power down the bridge.

  5. Disconnect the cables that are connected to the old bridge.

    You should make note of the port to which each cable was connected.

  6. Remove the old bridge from the rack.

  7. Install the new bridge into the rack.

  8. Reconnect the power cord and shielded Ethernet cable.

    Important You must not reconnect the SAS or FC cables at this time.

  9. Connect the bridge to a power source, and then turn it on.

    The bridge Ready LED might take up to 30 seconds to illuminate, indicating that the bridge has completed its power-on self test sequence.

  10. If configuring for in-band management, connect a cable from FibreBridge RS-232 serial port to the serial (COM) port on a personal computer.

    The serial connection will be used for initial configuration, and then in-band management via ONTAP and the FC ports can be used to monitor and manage the bridge.

  11. If configuring for in-band management, connect a cable from FibreBridge RS-232 serial port to the serial (COM) port on a personal computer.

    The serial connection will be used for initial configuration, and then in-band management via ONTAP and the FC ports can be used to monitor and manage the bridge.

  12. If configuring for IP management, configure the Ethernet management 1 port for each bridge by following the procedure in section 2.0 of the ATTO FibreBridge Installation and Operation Manual for your bridge model.

    In systems running ONTAP 9.5 or later, in-band management can be used to access the bridge via the FC ports rather than the Ethernet port. Beginning with ONTAP 9.8, only in-band management is supported and SNMP management is deprecated.

    When running QuickNAV to configure an Ethernet management port, only the Ethernet management port that is connected by the Ethernet cable is configured. For example, if you also wanted to configure the Ethernet management 2 port, you would need to connect the Ethernet cable to port 2 and run QuickNAV.

  13. Configure the bridges.

    Be sure to make note of the user name and password that you designate.

    The ATTO FibreBridge Installation and Operation Manual for your bridge model has the most current information on available commands and how to use them.

    Note Do not configure time synchronization on FibreBridge 7600N. The time synchronization for FibreBridge 7600N is set to the cluster time after the bridge is discovered by ONTAP. It is also synchronized periodically once a day. The time zone used is GMT and is not changeable.

    1. If configuring for IP management, configure the IP settings of the bridge.

      To set the IP address without the QuickNAV utility, you need to have a serial connection to the FibreBridge.

      If using the CLI, you must run the following commands:

      set ipaddress mp1 ip-address

      set ipsubnetmask mp1 subnet-mask

      set ipgateway mp1 x.x.x.x

      set ipdhcp mp1 disabled

      set ethernetspeed mp1 1000

    2. Configure the bridge name.

      The bridges should each have a unique name within the MetroCluster configuration.

      Example bridge names for one stack group on each site:

      • bridge_A_1a

      • bridge_A_1b

      • bridge_B_1a

      • bridge_B_1b

        If using the CLI, you must run the following command:

        set bridgename bridgename

    3. If running ONTAP 9.4 or earlier, enable SNMP on the bridge:
      set SNMP enabled

      In systems running ONTAP 9.5 or later, in-band management can be used to access the bridge via the FC ports rather than the Ethernet port. Beginning with ONTAP 9.8, only in-band management is supported and SNMP management is deprecated.

  14. Configure the bridge FC ports.

    1. Configure the data rate/speed of the bridge FC ports.

      The supported FC data rate depends on your model bridge.

      • The FibreBridge 7600N bridge supports up to 32, 16, or 8 Gbps.

      • The FibreBridge 7500N bridge supports up to 16, 8, or 4 Gbps.

        Note The FCDataRate speed you select is limited to the maximum speed supported by both the bridge and the FC port of the controller module or switch to which the bridge port connects. Cabling distances must not exceed the limitations of the SFPs and other hardware.

        If using the CLI, you must run the following command:

        set FCDataRate port-number port-speed

    2. You must configure or disable the FC2 port.

      • If you are using the second port, you must repeat the previous substeps for the FC2 port.

      • If you are not using the second port, then you must disable the unused port:

        FCPortDisable port-number

        The following example shows the disabling of FC port 2:

        FCPortDisable 2

Fibre Channel Port 2 has been disabled.

    3. Disable the unused SAS ports:

      SASPortDisable sas-port

      Note SAS ports A through D are enabled by default. You must disable the SAS ports that are not being used.

      If only SAS port A is used, then SAS ports B, C, and D must be disabled. The following example shows disabling of SAS port B. You must similarly disable SAS ports C and D:

      SASPortDisable b

SAS Port B has been disabled.

  15. Secure access to the bridge and save the bridge's configuration.

    1. From the controller prompt check the status of the bridges:

      storage bridge show

      The output shows which bridge is not secured.

    2. Check the status of the unsecured bridge's ports:

      info

      The output shows the status of Ethernet ports MP1 and MP2.

    3. If Ethernet port MP1 is enabled, run the following command:

      set EthernetPort mp1 disabled

      Note If Ethernet port MP2 is also enabled, repeat the previous substep for port MP2.

    4. Save the bridge's configuration.

      You must run the following commands:

      SaveConfiguration

      FirmwareRestart

      You are prompted to restart the bridge.

  16. Connect the FC cables to the same ports on the new bridge.

  17. Update the FibreBridge firmware on each bridge.

    Update firmware on a FibreBridge bridge

  18. Reconnect the SAS cables to the same ports on the new bridge.

    Note Wait at least 10 seconds before connecting the port. The SAS cable connectors are keyed; when oriented correctly into a SAS port, the connector clicks into place and the disk shelf SAS port LNK LED illuminates green. For disk shelves, you insert a SAS cable connector with the pull tab oriented down (on the underside of the connector). For controllers, the orientation of SAS ports can vary depending on the platform model; therefore, the correct orientation of the SAS cable connector varies.

  19. Verify that each bridge can see all of the disk drives and disk shelves to which the bridge is connected:

    sastargets

    The output shows the devices (disks and disk shelves) to which the bridge is connected. The output lines are sequentially numbered so that you can quickly count the devices.

    The following output shows that 10 disks are connected:

    Tgt VendorID ProductID        Type        SerialNumber
  0 NETAPP   X410_S15K6288A15 DISK        3QP1CLE300009940UHJV
  1 NETAPP   X410_S15K6288A15 DISK        3QP1ELF600009940V1BV
  2 NETAPP   X410_S15K6288A15 DISK        3QP1G3EW00009940U2M0
  3 NETAPP   X410_S15K6288A15 DISK        3QP1EWMP00009940U1X5
  4 NETAPP   X410_S15K6288A15 DISK        3QP1FZLE00009940G8YU
  5 NETAPP   X410_S15K6288A15 DISK        3QP1FZLF00009940TZKZ
  6 NETAPP   X410_S15K6288A15 DISK        3QP1CEB400009939MGXL
  7 NETAPP   X410_S15K6288A15 DISK        3QP1G7A900009939FNTT
  8 NETAPP   X410_S15K6288A15 DISK        3QP1FY0T00009940G8PA
  9 NETAPP   X410_S15K6288A15 DISK        3QP1FXW600009940VERQ

  20. Verify that the command output shows that the bridge is connected to all of the appropriate disks and disk shelves in the stack.

    If the output is…​

    Then…​

    Correct

    Repeat the previous step for each remaining bridge.

    Not correct

    1. Check for loose SAS cables or correct the SAS cabling by repeating Step 18.

    2. Repeat the previous step.

  21. If the bridge is in a fabric-attached MetroCluster configuration, reenable the FC switch port that you disabled at the beginning of this procedure.

    This should be the port that connects to the bridge.

  22. From the system console of both controller modules, verify that all of the controller modules have access through the new bridge to the disk shelves (that is, that the system is cabled for Multipath HA):

    run local sysconfig

    Note It might take up to a minute for the system to complete discovery.

    If the output does not indicate Multipath HA, you must correct the SAS and FC cabling because not all of the disk drives are accessible through the new bridge.

    The following output states that the system is cabled for Multipath HA:

    NetApp Release 8.3.2: Tue Jan 26 01:41:49 PDT 2016
System ID: 1231231231 (node_A_1); partner ID: 4564564564 (node_A_2)
System Serial Number: 700000123123 (node_A_1); partner Serial Number: 700000456456 (node_A_2)
System Rev: B0
System Storage Configuration: Multi-Path HA
System ACP Connectivity: NA
    Important When the system is not cabled as Multipath HA, restarting a bridge might cause loss of access to the disk drives and result in a multi-disk panic.

  23. If running ONTAP 9.4 or earlier, verify that the bridge is configured for SNMP.

    If you are using the bridge CLI, run the following command:

    get snmp

  24. From the ONTAP cluster prompt, add the bridge to health monitoring:

    1. Add the bridge, using the command for your version of ONTAP:

      ONTAP version

      Command

      9.5 and later

      storage bridge add -address 0.0.0.0 -managed-by in-band -name bridge-name

      9.4 and earlier

      storage bridge add -address bridge-ip-address -name bridge-name

    2. Verify that the bridge has been added and is properly configured:

      storage bridge show

      It might take as long as 15 minutes to reflect all data because of the polling interval. The ONTAP health monitor can contact and monitor the bridge if the value in the “Status” column is “ok”, and other information, such as the worldwide name (WWN), is displayed.

      The following example shows that the FC-to-SAS bridges are configured:

      controller_A_1::> storage bridge show

Bridge              Symbolic Name Is Monitored  Monitor Status  Vendor Model                Bridge WWN
------------------  ------------- ------------  --------------  ------ -----------------    ----------
ATTO_10.10.20.10  atto01        true          ok              Atto   FibreBridge 7500N   	20000010867038c0
ATTO_10.10.20.11  atto02        true          ok              Atto   FibreBridge 7500N   	20000010867033c0
ATTO_10.10.20.12  atto03        true          ok              Atto   FibreBridge 7500N   	20000010867030c0
ATTO_10.10.20.13  atto04        true          ok              Atto   FibreBridge 7500N   	2000001086703b80

4 entries were displayed

 controller_A_1::>

  25. Verify the operation of the MetroCluster configuration in ONTAP:

    1. Check whether the system is multipathed:
      node run -node node-name sysconfig -a

    2. Check for any health alerts on both clusters:
      system health alert show

    3. Confirm the MetroCluster configuration and that the operational mode is normal:
      metrocluster show

    4. Perform a MetroCluster check:
      metrocluster check run

    5. Display the results of the MetroCluster check:

      metrocluster check show

    6. Check for any health alerts on the switches (if present):
      storage switch show

    7. Run Config Advisor.

      NetApp Downloads: Config Advisor

    8. After running Config Advisor, review the tool's output and follow the recommendations in the output to address any issues discovered.

Related information

In-band management of the FC-to-SAS bridges

Hot-swapping a FibreBridge 6500N bridge with a FibreBridge 7600N or 7500N bridge

You can hot-swap a FibreBridge 6500N bridge with a FibreBridge 7600N or 7500N bridge to replace a failed bridge or upgrade your bridge in a fabric-attached or a bridge-attached MetroCluster configuration.

About this task

  • This procedure is for hot-swapping a single FibreBridge 6500N bridge with single FibreBridge 7600N or 7500N bridge.

  • When you hot-swap a FibreBridge 6500N bridge with a FibreBridge 7600N or 7500N bridge, you must use only one FC port and one SAS port on the FibreBridge 7600N or 7500N bridge.

  • If you will be using in-band management of the bridge rather than IP management, the steps for configuring the Ethernet port and IP settings can be skipped, as noted in the relevant steps.

Important If you are hot-swapping both FibreBridge 6500N bridges in a pair, you must use the Consolidate Multiple Storage Stacks procedure for zoning instructions. By replacing both FibreBridge 6500N bridges on the bridge, you can take advantage of the additional ports on the FibreBridge 7600N or 7500N bridge.
Note Beginning with ONTAP 9.8, the storage bridge command is replaced with system bridge. The following steps show the storage bridge command, but if you are running ONTAP 9.8 or later, the system bridge command is preferred.
Steps

  1. Do one of the following:

    • If the failed bridge is in a fabric-attached MetroCluster configuration, disable the switch port that connects to the bridge FC port.

    • If the failed bridge is in a stretch MetroCluster configuration, use either one of the available FC ports.

  2. From the ONTAP cluster prompt, remove the bridge undergoing maintenance from health monitoring:

    1. Remove the bridge:

      storage bridge remove -name bridge-name

    2. View the list of monitored bridges and confirm that the removed bridge is not present:

      storage bridge show

  3. Properly ground yourself.

  4. Turn off the power switch of the bridge.

  5. Disconnect the cables connected from the shelf to the FibreBridge 6500N bridge ports and power cables.

    You should make note of the ports that each cable was connected to.

  6. Remove the FibreBridge 6500N bridge that you need to replace from the rack.

  7. Install the new FibreBridge 7600N or 7500N bridge into the rack.

  8. Reconnect the power cord and, if necessary, the shielded Ethernet cable.

    Important Do not reconnect the SAS or FC cables at this time.

  9. If configuring for in-band management, connect a cable from FibreBridge RS-232 serial port to the serial (COM) port on a personal computer.

    The serial connection will be used for initial configuration, and then in-band management via ONTAP and the FC ports can be used to monitor and manage the bridge.

  10. If configuring for IP management, connect the Ethernet management 1 port on each bridge to your network by using an Ethernet cable.

    In systems running ONTAP 9.5 or later, in-band management can be used to access the bridge via the FC ports rather than the Ethernet port. Beginning with ONTAP 9.8, only in-band management is supported and SNMP management is deprecated.

    The Ethernet management 1 port enables you to quickly download the bridge firmware (using ATTO ExpressNAV or FTP management interfaces) and to retrieve core files and extract logs.

  11. If configuring for IP management, configure the Ethernet management 1 port for each bridge by following the procedure in section 2.0 of the ATTO FibreBridge Installation and Operation Manual for your bridge model.

    In systems running ONTAP 9.5 or later, in-band management can be used to access the bridge via the FC ports rather than the Ethernet port. Beginning with ONTAP 9.8, only in-band management is supported and SNMP management is deprecated.

    When running QuickNAV to configure an Ethernet management port, only the Ethernet management port that is connected by the Ethernet cable is configured. For example, if you also wanted to configure the Ethernet management 2 port, you would need to connect the Ethernet cable to port 2 and run QuickNAV.

  12. Configure the bridge.

    If you retrieved the configuration information from the old bridge, use the information to configure the new bridge.

    Be sure to make note of the user name and password that you designate.

    The ATTO FibreBridge Installation and Operation Manual for your bridge model has the most current information on available commands and how to use them.

    Note Do not configure time synchronization on ATTO FibreBridge 7600N or 7500N. The time synchronization for ATTO FibreBridge 7600N or 7500N is set to the cluster time after the bridge is discovered by ONTAP. It is also synchronized periodically once a day. The time zone used is GMT and is not changeable.

    1. If configuring for IP management, configure the IP settings of the bridge.

      To set the IP address without the QuickNAV utility, you need to have a serial connection to the FibreBridge.

      If using the CLI, you must run the following commands:

      set ipaddress mp1 ip-address

      set ipsubnetmask mp1 subnet-mask

      set ipgateway mp1 x.x.x.x

      set ipdhcp mp1 disabled

      set ethernetspeed mp1 1000

    2. Configure the bridge name.

      The bridges should each have a unique name within the MetroCluster configuration.

      Example bridge names for one stack group on each site:

      • bridge_A_1a

      • bridge_A_1b

      • bridge_B_1a

      • bridge_B_1b

        If using the CLI, you must run the following command:

        set bridgename bridgename

    3. If running ONTAP 9.4 or earlier, enable SNMP on the bridge:
      set SNMP enabled

      In systems running ONTAP 9.5 or later, in-band management can be used to access the bridge via the FC ports rather than the Ethernet port. Beginning with ONTAP 9.8, only in-band management is supported and SNMP management is deprecated.

  13. Configure the bridge FC ports.

    1. Configure the data rate/speed of the bridge FC ports.

      The supported FC data rate depends on your model bridge.

      • The FibreBridge 7600N bridge supports up to 32, 16, or 8 Gbps.

      • The FibreBridge 7500N bridge supports up to 16, 8, or 4 Gbps.

      • The FibreBridge 6500N bridge supports up to 8, 4, or 2 Gbps.

        Note The FCDataRate speed you select is limited to the maximum speed supported by both the bridge and the switch to which the bridge port connects. Cabling distances must not exceed the limitations of the SFPs and other hardware.

        If using the CLI, you must run the following command:

        set FCDataRate port-number port-speed

    2. If you are configuring a FibreBridge 7500N or 6500N bridge, configure the connection mode that the port uses to ptp.

      Note The FCConnMode setting is not required when configuring a FibreBridge 7600N bridge.

      If using the CLI, you must run the following command:

      set FCConnMode port-number ptp

    3. If you are configuring a FibreBridge 7600N or 7500N bridge, you must configure or disable the FC2 port.

      • If you are using the second port, you must repeat the previous substeps for the FC2 port.

      • If you are not using the second port, then you must disable the port:

        FCPortDisable port-number

    4. If you are configuring a FibreBridge 7600N or 7500N bridge, disable the unused SAS ports:

      SASPortDisable sas-port

      Note SAS ports A through D are enabled by default. You must disable the SAS ports that are not being used. If only SAS port A is used, then SAS ports B, C, and D must be disabled.

  14. Secure access to the bridge and save the bridge's configuration.

    1. From the controller prompt check the status of the bridges:

      storage bridge show

      The output shows which bridge is not secured.

    2. Check the status of the unsecured bridge's ports:

      info

      The output shows the status of Ethernet ports MP1 and MP2.

    3. If Ethernet port MP1 is enabled, run the following command:

      set EthernetPort mp1 disabled

      Note If Ethernet port MP2 is also enabled, repeat the previous substep for port MP2.

    4. Save the bridge's configuration.

      You must run the following commands:

      SaveConfiguration

      FirmwareRestart

      You are prompted to restart the bridge.

  15. Turn on Health Monitoring for the FibreBridge 7600N or 7500N bridge.

  16. Connect the FC cables to the Fibre Channel 1 ports on the new bridge.

    You must cable the FC port to the same switch or controller port that the FibreBridge 6500N bridge had been connected to.

  17. Update the FibreBridge firmware on each bridge.

    If the new bridge is the same type as the partner bridge, upgrade to the same firmware as the partner bridge. If the new bridge is a different type to the partner bridge, upgrade to the latest firmware and version of ONTAP supported by the bridge.

    Update firmware on a FibreBridge bridge

  18. Reconnect the SAS cables to the SAS A ports on the new bridge.

    The SAS port must be cabled to the same shelf port that the FibreBridge 6500N bridge had been connected to.

    Note Do not force a connector into a port. The mini-SAS cables are keyed; when oriented correctly into a SAS port, the SAS cable clicks into place and the disk shelf SAS port LNK LED illuminates green. For disk shelves, you insert a SAS cable connector with the pull tab oriented down (on the underside of the connector).For controllers, the orientation of SAS ports can vary depending on the platform model; therefore, the correct orientation of the SAS cable connector varies.

  19. Verify that the bridge can detect all of the disk drives and disk shelves it is connected to.

    If you are using the…​

    Then…​

    ATTO ExpressNAV GUI

    1. In a supported web browser, enter the IP address of the bridge in the browser box.

      You are brought to the ATTO FibreBridge homepage, which has a link.

    2. Click the link, and then enter your user name and the password that you designated when you configured the bridge.

      The ATTO FibreBridge status page appears with a menu to the left.

    3. Click Advanced in the menu.

    4. Enter the following command and then click Submit to see the list of disks visible to the bridge:

      sastargets

    Serial port connection

    Display the list of disks visible to the bridge:

    sastargets

    The output shows the devices (disks and disk shelves) that the bridge is connected to. Output lines are sequentially numbered so that you can quickly count the devices. For example, the following output shows that 10 disks are connected:

    Tgt VendorID ProductID        Type        SerialNumber
  0 NETAPP   X410_S15K6288A15 DISK        3QP1CLE300009940UHJV
  1 NETAPP   X410_S15K6288A15 DISK        3QP1ELF600009940V1BV
  2 NETAPP   X410_S15K6288A15 DISK        3QP1G3EW00009940U2M0
  3 NETAPP   X410_S15K6288A15 DISK        3QP1EWMP00009940U1X5
  4 NETAPP   X410_S15K6288A15 DISK        3QP1FZLE00009940G8YU
  5 NETAPP   X410_S15K6288A15 DISK        3QP1FZLF00009940TZKZ
  6 NETAPP   X410_S15K6288A15 DISK        3QP1CEB400009939MGXL
  7 NETAPP   X410_S15K6288A15 DISK        3QP1G7A900009939FNTT
  8 NETAPP   X410_S15K6288A15 DISK        3QP1FY0T00009940G8PA
  9 NETAPP   X410_S15K6288A15 DISK        3QP1FXW600009940VERQ
    Note If the text “response truncated” appears at the beginning of the output, you can use Telnet to access the bridge and enter the same command to see all of the output.

  20. Verify that the command output shows that the bridge is connected to all of the necessary disks and disk shelves in the stack.

    If the output is…​

    Then…​

    Correct

    Repeat the previous step for each remaining bridge.

    Not correct

    1. Check for loose SAS cables or correct the SAS cabling by repeating Step 18.

    2. Repeat the previous step for each remaining bridge.

  21. Reenable the FC switch port that connects to the bridge.

  22. Verify that all controllers have access through the new bridge to the disk shelves (that the system is cabled for Multipath HA), at the system console of both controllers:

    run local sysconfig

    Note It might take up to a minute for the system to complete discovery.

    For example, the following output shows that the system is cabled for Multipath HA:

    NetApp Release 8.3.2: Tue Jan 26 01:23:24 PST 2016
System ID: 1231231231 (node_A_1); partner ID: 4564564564 (node_A_2)
System Serial Number: 700000123123 (node_A_1); partner Serial Number: 700000456456 (node_A_2)
System Rev: B0
System Storage Configuration: Multi-Path HA
System ACP Connectivity: NA

    If the command output indicates that the configuration is mixed-path or single-path HA, you must correct the SAS and FC cabling because not all disk drives are accessible through the new bridge.

    Important When the system is not cabled as Multipath HA, restarting a bridge might cause loss of access to the disk drives and result in a multi-disk panic.

  23. From the ONTAP cluster prompt, add the bridge to health monitoring:

    1. Add the bridge, using the command for your version of ONTAP:

      ONTAP version

      Command

      9.5 and later

      storage bridge add -address 0.0.0.0 -managed-by in-band -name bridge-name

      9.4 and earlier

      storage bridge add -address bridge-ip-address -name bridge-name

    2. Verify that the bridge has been added and is properly configured:
      storage bridge show

      It might take as long as 15 minutes to reflect all data because of the polling interval. The ONTAP health monitor can contact and monitor the bridge if the value in the “Status” column is “ok”, and other information, such as the worldwide name (WWN), is displayed.

      The following example shows that the FC-to-SAS bridges are configured:

      controller_A_1::> storage bridge show

Bridge              Symbolic Name Is Monitored  Monitor Status  Vendor Model                Bridge WWN
------------------  ------------- ------------  --------------  ------ -----------------    ----------
ATTO_10.10.20.10  atto01        true          ok              Atto   FibreBridge 7500N   	20000010867038c0
ATTO_10.10.20.11  atto02        true          ok              Atto   FibreBridge 7500N   	20000010867033c0
ATTO_10.10.20.12  atto03        true          ok              Atto   FibreBridge 7500N   	20000010867030c0
ATTO_10.10.20.13  atto04        true          ok              Atto   FibreBridge 7500N   	2000001086703b80

4 entries were displayed

 controller_A_1::>

  24. Verify the operation of the MetroCluster configuration in ONTAP:

    1. Check whether the system is multipathed:

      node run -node node-name sysconfig -a

    2. Check for any health alerts on both clusters:
      system health alert show

    3. Confirm the MetroCluster configuration and that the operational mode is normal:

      metrocluster show

    4. Perform a MetroCluster check:

      metrocluster check run

    5. Display the results of the MetroCluster check:

      metrocluster check show

    6. Check for any health alerts on the switches (if present):

      storage switch show

    7. Run Config Advisor.

      NetApp Downloads: Config Advisor

    8. After running Config Advisor, review the tool's output and follow the recommendations in the output to address any issues discovered.

  25. After replacing the part, return the failed part to NetApp, as described in the RMA instructions shipped with the kit. See the Part Return & Replacements page for further information.

Related information

In-band management of the FC-to-SAS bridges