Skip to main content
Cluster and storage switches

Replace a Broadcom-supported BES-53248 cluster switch

Contributors netapp-yvonneo netapp-jolieg
PDFs

Follow these steps to replace a defective Broadcom-supported BES-53248 cluster switch in a cluster network. This is a nondisruptive procedure (NDU).

About the examples

The examples in this procedure use the following switch and node nomenclature:

  • The names of the existing BES-53248 switches are cs1 and cs2.

  • The name of the new BES-53248 switch is newcs2.

  • The node names are node1 and node2.

  • The cluster ports on each node are named e0a and e0b.

  • The cluster LIF names are node1_clus1 and node1_clus2 for node1, and node2_clus1 and node2_clus2 for node2.

  • The prompt for changes to all cluster nodes is cluster1::>

About the topology

This procedure is based on the following cluster network topology:

Show example topology 
cluster1::> network port show -ipspace Cluster

Node: node1
                                                                       Ignore
                                                  Speed(Mbps) Health   Health
Port      IPspace      Broadcast Domain Link MTU  Admin/Oper  Status   Status
--------- ------------ ---------------- ---- ---- ----------- -------- ------
e0a       Cluster      Cluster          up   9000  auto/10000 healthy  false
e0b       Cluster      Cluster          up   9000  auto/10000 healthy  false


Node: node2
                                                                       Ignore
                                                  Speed(Mbps) Health   Health
Port      IPspace      Broadcast Domain Link MTU  Admin/Oper  Status   Status
--------- ------------ ---------------- ---- ---- ----------- -------- ------
e0a       Cluster      Cluster          up   9000  auto/10000 healthy  false
e0b       Cluster      Cluster          up   9000  auto/10000 healthy  false


cluster1::> network interface show -vserver Cluster
            Logical    Status     Network            Current       Current Is
Vserver     Interface  Admin/Oper Address/Mask       Node          Port    Home
----------- ---------- ---------- ------------------ ------------- ------- ----
Cluster
            node1_clus1  up/up    169.254.209.69/16  node1         e0a     true
            node1_clus2  up/up    169.254.49.125/16  node1         e0b     true
            node2_clus1  up/up    169.254.47.194/16  node2         e0a     true
            node2_clus2  up/up    169.254.19.183/16  node2         e0b     true


cluster1::> network device-discovery show -protocol cdp
Node/       Local  Discovered
Protocol    Port   Device (LLDP: ChassisID)  Interface         Platform
----------- ------ ------------------------- ----------------  ----------------
node2      /cdp
            e0a    cs1                       0/2               BES-53248
            e0b    cs2                       0/2               BES-53248
node1      /cdp
            e0a    cs1                       0/1               BES-53248
            e0b    cs2                       0/1               BES-53248
(cs1)# show isdp neighbors

Capability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge,
                  S - Switch, H - Host, I - IGMP, r - Repeater

Device ID                Intf      Holdtime  Capability Platform         Port ID
------------------------ --------- --------- ---------- ---------------- ---------
node1                    0/1       175       H          FAS2750          e0a
node2                    0/2       152       H          FAS2750          e0a
cs2                      0/55      179       R          BES-53248        0/55
cs2                      0/56      179       R          BES-53248        0/56


(cs2)# show isdp neighbors

Capability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge,
                  S - Switch, H - Host, I - IGMP, r - Repeater

Device ID                Intf      Holdtime  Capability Platform         Port ID
------------------------ --------- --------- ---------- ---------------- ---------
node1                    0/1       129       H          FAS2750          e0b
node2                    0/2       165       H          FAS2750          e0b
cs1                      0/55      179       R          BES-53248        0/55
cs1                      0/56      179       R          BES-53248        0/56
Steps

  1. Review the Replacement requirements.

  2. If AutoSupport is enabled on this cluster, suppress automatic case creation by invoking an AutoSupport message:

    system node autosupport invoke -node * -type all -message MAINT=xh

    where x is the duration of the maintenance window in hours.

    Note The AutoSupport message notifies technical support of this maintenance task so that automatic case creation is suppressed during the maintenance window.

  3. Install the appropriate Reference Configuration File (RCF) and image on the switch, newcs2, and make any necessary site preparations.

    If necessary, verify, download, and install the appropriate versions of the RCF and EFOS software for the new switch. If you have verified that the new switch is correctly set up and does not need updates to the RCF and EFOS software, continue to step 2.

    1. You can download the applicable Broadcom EFOS software for your cluster switches from the Broadcom Ethernet Switch Support site. Follow the steps on the Download page to download the EFOS file for the version of ONTAP software you are installing.

    2. The appropriate RCF is available from the Broadcom Cluster Switches page. Follow the steps on the Download page to download the correct RCF for the version of ONTAP software you are installing.

  4. On the new switch, log in as admin and shut down all of the ports that will be connected to the node cluster interfaces (ports 1 to 16).

    Note If you purchased additional licenses for additional ports, shut down these ports too.

    If the switch that you are replacing is not functional and is powered down, the LIFs on the cluster nodes should have already failed over to the other cluster port for each node.

    Note No password is required to enter enable mode.
    Show example 
    User: admin
Password:
(newcs2)> enable
(newcs2)# config
(newcs2)(config)# interface 0/1-0/16
(newcs2)(interface 0/1-0/16)# shutdown
(newcs2)(interface 0/1-0/16)# exit
(newcs2)(config)# exit
(newcs2)#

  5. Verify that all cluster LIFs have auto-revert enabled:

    network interface show -vserver Cluster -fields auto-revert

    Show example topology 
    cluster1::> network interface show -vserver Cluster -fields auto-revert

Logical
Vserver   Interface    Auto-revert
--------- ------------ ------------
Cluster   node1_clus1  true
Cluster   node1_clus2  true
Cluster   node2_clus1  true
Cluster   node2_clus2  true

  6. Shut down the ISL ports 0/55 and 0/56 on the BES-53248 switch cs1:

    Show example topology 
    (cs1)# config
(cs1)(config)# interface 0/55-0/56
(cs1)(interface 0/55-0/56)# shutdown

  7. Remove all cables from the BES-53248 cs2 switch, and then connect them to the same ports on the BES-53248 newcs2 switch.

  8. Bring up the ISLs ports 0/55 and 0/56 between the cs1 and newcs2 switches, and then verify the port channel operation status.

    The Link State for port-channel 1/1 should be up and all member ports should be True under the Port Active heading.

    Show example

    This example enables ISL ports 0/55 and 0/56 and displays the Link State for port-channel 1/1 on switch cs1:

    (cs1)# config
(cs1)(config)# interface 0/55-0/56
(cs1)(interface 0/55-0/56)# no shutdown
(cs1)(interface 0/55-0/56)# exit
(cs1)# show port-channel 1/1

Local Interface................................ 1/1
Channel Name................................... Cluster-ISL
Link State..................................... Up
Admin Mode..................................... Enabled
Type........................................... Dynamic
Port-channel Min-links......................... 1
Load Balance Option............................ 7
(Enhanced hashing mode)

Mbr    Device/       Port       Port
Ports  Timeout       Speed      Active
------ ------------- ---------- -------
0/55   actor/long    100G Full  True
       partner/long
0/56   actor/long    100G Full  True
       partner/long

  9. On the new switch newcs2, re-enable all of the ports that are connected to the node cluster interfaces (ports 1 to 16).

    Note If you purchased additional licenses for additional ports, shut down these ports too.
    Show example 
    User:admin
Password:
(newcs2)> enable
(newcs2)# config
(newcs2)(config)# interface 0/1-0/16
(newcs2)(interface 0/1-0/16)# no shutdown
(newcs2)(interface 0/1-0/16)# exit
(newcs2)(config)# exit

  10. Verify that port e0b is up:

    network port show -ipspace Cluster

    Show example

    The output should be similar to the following:

    cluster1::> network port show -ipspace Cluster

Node: node1
                                                                        Ignore
                                                   Speed(Mbps) Health   Health
Port      IPspace      Broadcast Domain Link MTU   Admin/Oper  Status   Status
--------- ------------ ---------------- ---- ----- ----------- -------- -------
e0a       Cluster      Cluster          up   9000  auto/10000  healthy  false
e0b       Cluster      Cluster          up   9000  auto/10000  healthy  false

Node: node2
                                                                        Ignore
                                                   Speed(Mbps) Health   Health
Port      IPspace      Broadcast Domain Link MTU   Admin/Oper  Status   Status
--------- ------------ ---------------- ---- ----- ----------- -------- -------
e0a       Cluster      Cluster          up   9000  auto/10000  healthy  false
e0b       Cluster      Cluster          up   9000  auto/auto   -        false

  11. On the same node as you used in the previous step, wait for the cluster LIF node1_clus2 on node1 to auto-revert.

    Show example

    In this example, LIF node1_clus2 on node1 is successfully reverted if Is Home is true and the port is e0b.

    The following command displays information about the LIFs on both nodes. Bringing up the first node is successful if Is Home is true for both cluster interfaces and they show the correct port assignments, in this example e0a and e0b on node1.

    cluster::> network interface show -vserver Cluster

            Logical      Status     Network            Current    Current Is
Vserver     Interface    Admin/Oper Address/Mask       Node       Port    Home
----------- ------------ ---------- ------------------ ---------- ------- -----
Cluster
            node1_clus1  up/up      169.254.209.69/16  node1      e0a     true
            node1_clus2  up/up      169.254.49.125/16  node1      e0b     true
            node2_clus1  up/up      169.254.47.194/16  node2      e0a     true
            node2_clus2  up/up      169.254.19.183/16  node2      e0a     false

  12. Display information about the nodes in a cluster:

    cluster show

    Show example

    This example shows that the node health for node1 and node2 in this cluster is true:

    cluster1::> cluster show
Node   Health   Eligibility   Epsilon
------ -------- ------------  --------
node1  true     true          true
node2  true     true          true

  13. Confirm the following cluster network configuration:

    network port show

    network interface show

    Show example 
    cluster1::> network port show -ipspace Cluster
Node: node1
                                                                       Ignore
                                       Speed(Mbps)            Health   Health
Port      IPspace     Broadcast Domain Link MTU   Admin/Oper  Status   Status
--------- ----------- ---------------- ---- ----- ----------- -------- ------
e0a       Cluster     Cluster          up   9000  auto/10000  healthy  false
e0b       Cluster     Cluster          up   9000  auto/10000  healthy  false

Node: node2
                                                                       Ignore
                                        Speed(Mbps)           Health   Health
Port      IPspace      Broadcast Domain Link MTU  Admin/Oper  Status   Status
--------- ------------ ---------------- ---- ---- ----------- -------- ------
e0a       Cluster      Cluster          up   9000 auto/10000  healthy  false
e0b       Cluster      Cluster          up   9000 auto/10000  healthy  false


cluster1::> network interface show -vserver Cluster

            Logical    Status     Network            Current       Current Is
Vserver     Interface  Admin/Oper Address/Mask       Node          Port    Home
----------- ---------- ---------- ------------------ ------------- ------- ----
Cluster
            node1_clus1  up/up    169.254.209.69/16  node1         e0a     true
            node1_clus2  up/up    169.254.49.125/16  node1         e0b     true
            node2_clus1  up/up    169.254.47.194/16  node2         e0a     true
            node2_clus2  up/up    169.254.19.183/16  node2         e0b     true
4 entries were displayed.

  14. Verify that the cluster network is healthy:

    show isdp neighbors

    Show example 
    (cs1)# show isdp neighbors
Capability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge,
S - Switch, H - Host, I - IGMP, r - Repeater
Device ID    Intf    Holdtime    Capability    Platform    Port ID
---------    ----    --------    ----------    --------    --------
node1        0/1     175         H             FAS2750     e0a
node2        0/2     152         H             FAS2750     e0a
newcs2       0/55    179         R             BES-53248   0/55
newcs2       0/56    179         R             BES-53248   0/56

(newcs2)# show isdp neighbors
Capability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge,
S - Switch, H - Host, I - IGMP, r - Repeater

Device ID    Intf    Holdtime    Capability    Platform    Port ID
---------    ----    --------    ----------    --------    --------
node1        0/1     129         H             FAS2750     e0b
node2        0/2     165         H             FAS2750     e0b
cs1          0/55    179         R             BES-53248   0/55
cs1          0/56    179         R             BES-53248   0/56

  15. If you suppressed automatic case creation, re-enable it by invoking an AutoSupport message:

    system node autosupport invoke -node * -type all -message MAINT=END

What's next?

Configure switch health monitoring.