Skip to main content
Cluster and storage switches

Replace a Cisco Nexus 9336C-FX2 shared switch

Contributors netapp-jolieg netapp-yvonneo

You can replace a defective Nexus 9336C-FX2 shared switch. This is a nondisruptive procedure (NDU).

What you'll need

Before performing the switch replacement, make sure that:

  • In the existing cluster and network infrastructure:

    • The existing cluster is verified as completely functional, with at least one fully connected cluster switch.

    • All cluster ports are up.

    • All cluster logical interfaces (LIFs) are up and on their home ports.

    • The ONTAP cluster ping-cluster -node node1 command must indicate that basic connectivity and larger than PMTU communication are successful on all paths.

  • For the Nexus 9336C-FX2 replacement switch:

    • Management network connectivity on the replacement switch is functional.

    • Console access to the replacement switch is in place.

    • The node connections are ports 1/1 through 1/34:

    • All Inter-Switch Link (ISL) ports are disabled on ports 1/35 and 1/36.

    • The desired reference configuration file (RCF) and NX-OS operating system image switch is loaded onto the switch.

    • Any previous site customizations, such as STP, SNMP, and SSH, should be copied to the new switch.

About the examples

You must execute the command for migrating a cluster LIF from the node where the cluster LIF is hosted.

The examples in this procedure use the following switch and node nomenclature:

  • The names of the existing Nexus 9336C-FX2 switches are sh1 and sh2.

  • The name of the new Nexus 9336C-FX2 switches are newsh1 and newsh2.

  • The node names are node1 and node2.

  • The cluster ports on each node are named e3a and e3b.

  • The cluster LIF names are node1_clus1 and node1_clus2 for node1, and node2_clus1 and node2_clus2 for node2.

  • The prompt for changes to all cluster nodes is cluster1::*>.

    Note The following procedure is based on the following network topology:
Show example topology 
cluster1::*> network port show -ipspace Cluster

Node: node1
                                                                        Ignore
                                                  Speed(Mbps)  Health   Health
Port      IPspace      Broadcast Domain Link MTU  Admin/Oper   Status   Status
--------- ------------ ---------------- ---- ---- ------------ -------- ------
e3a       Cluster      Cluster          up   9000  auto/100000 healthy  false
e3b       Cluster      Cluster          up   9000  auto/100000 healthy  false

Node: node2
                                                                        Ignore
                                                  Speed(Mbps)  Health   Health
Port      IPspace      Broadcast Domain Link MTU  Admin/Oper   Status   Status
--------- ------------ ---------------- ---- ---- ------------ -------- ------
e3a       Cluster      Cluster          up   9000  auto/100000 healthy  false
e3b       Cluster      Cluster          up   9000  auto/100000 healthy  false
4 entries were displayed.


cluster1::*> network interface show -vserver Cluster
            Logical    Status     Network            Current       Current Is
Vserver     Interface  Admin/Oper Address/Mask       Node          Port    Home
----------- ---------- ---------- ------------------ ------------- ------- ----
Cluster
            node1_clus1  up/up    169.254.209.69/16  node1         e3a     true
            node1_clus2  up/up    169.254.49.125/16  node1         e3b     true
            node2_clus1  up/up    169.254.47.194/16  node2         e3a     true
            node2_clus2  up/up    169.254.19.183/16  node2         e3b     true
4 entries were displayed.

cluster1::*> network device-discovery show -protocol cdp
Node/       Local  Discovered
Protocol    Port   Device (LLDP: ChassisID)  Interface         Platform
----------- ------ ------------------------- ----------------  ----------------
node2      /cdp
            e3a    sh1                       Eth1/2            N9K-C9336C
            e3b    sh2                       Eth1/2            N9K-C9336C

node1      /cdp
            e3a    sh1                       Eth1/1            N9K-C9336C
            e3b    sh2                       Eth1/1            N9K-C9336C
4 entries were displayed.

sh1# show cdp neighbors
Capability Codes: R - Router, T - Trans-Bridge, B - Source-Route-Bridge
                  S - Switch, H - Host, I - IGMP, r - Repeater,
                  V - VoIP-Phone, D - Remotely-Managed-Device,
                  s - Supports-STP-Dispute
Device-ID          Local Intrfce  Hldtme Capability  Platform      Port ID
node1              Eth1/1         144    H           FAS2980       e3a
node2              Eth1/2         145    H           FAS2980       e3a
sh2                Eth1/35        176    R S I s     N9K-C9336C    Eth1/35
sh2 (FDO220329V5)   Eth1/36       176    R S I s     N9K-C9336C    Eth1/36
Total entries displayed: 4

sh2# show cdp neighbors
Capability Codes: R - Router, T - Trans-Bridge, B - Source-Route-Bridge
                  S - Switch, H - Host, I - IGMP, r - Repeater,
                  V - VoIP-Phone, D - Remotely-Managed-Device,
                  s - Supports-STP-Dispute
Device-ID          Local Intrfce  Hldtme Capability  Platform      Port ID
node1              Eth1/1         139    H           FAS2980       eb
node2              Eth1/2         124    H           FAS2980       eb
sh1                Eth1/35        178    R S I s     N9K-C9336C    Eth1/35
sh1                Eth1/36        178    R S I s     N9K-C9336C    Eth1/36
Total entries displayed: 4
Steps

  1. If AutoSupport is enabled on this cluster, suppress automatic case creation by invoking an AutoSupport message:

    system node autosupport invoke -node * -type all -message MAINT=xh

    Where x is the duration of the maintenance window in hours.

  2. Optional: Install the appropriate RCF and image on the switch, newsh2, and make any necessary site preparations.

    1. If necessary, verify, download, and install the appropriate versions of the RCF and NX-OS software for the new switch. If you have verified that the new switch is correctly set up and does not need updates to the RCF and NX-OS software, continue to Step 3.

    2. Go to the NetApp Cluster and Management Network Switches Reference Configuration File Description Page on the NetApp Support Site.

    3. Click the link for the Cluster Network and Management Network Compatibility Matrix, and then note the required switch software version.

    4. Click your browser's back arrow to return to the Description page, click CONTINUE, accept the license agreement, and then go to the Download page.

    5. Follow the steps on the Download page to download the correct RCF and NX-OS files for the version of ONTAP software you are installing.

  3. On the new switch, log in as admin and shut down all the ports that will be connected to the node cluster interfaces (ports 1/1 to 1/34).
    If the switch that you are replacing is not functional and is powered down, go to Step 4. The LIFs on the cluster nodes should have already failed over to the other cluster port for each node.

    Show example 
    newsh2# config
Enter configuration commands, one per line. End with CNTL/Z.
newsh2(config)# interface e1/1-34
newsh2(config-if-range)# shutdown

  1. Verify that all cluster LIFs have auto-revert enabled.

    network interface show - vserver Cluster -fields auto-revert

    Show example 
    cluster1::> network interface show -vserver Cluster -fields auto-revert
             Logical
Vserver      Interface     Auto-revert
------------ ------------- -------------
Cluster      node1_clus1   true
Cluster      node1_clus2   true
Cluster      node2_clus1   true
Cluster      node2_clus2   true
4 entries were displayed.

  1. Verify that all the cluster LIFs can communicate:

    cluster ping-cluster <node name>

    Show example 
    cluster1::*> cluster ping-cluster node2
Host is node2
Getting addresses from network interface table...
Cluster node1_clus1 169.254.209.69 node1 e3a
Cluster node1_clus2 169.254.49.125 node1 e3b
Cluster node2_clus1 169.254.47.194 node2 e3a
Cluster node2_clus2 169.254.19.183 node2 e3b
Local = 169.254.47.194 169.254.19.183
Remote = 169.254.209.69 169.254.49.125
Cluster Vserver Id = 4294967293
Ping status:
....
Basic connectivity succeeds on 4 path(s)
Basic connectivity fails on 0 path(s)
................
Detected 9000 byte MTU on 4 path(s):
Local 169.254.47.194 to Remote 169.254.209.69
Local 169.254.47.194 to Remote 169.254.49.125
Local 169.254.19.183 to Remote 169.254.209.69
Local 169.254.19.183 to Remote 169.254.49.125
Larger than PMTU communication succeeds on 4 path(s)
RPC status:
2 paths up, 0 paths down (tcp check)
2 paths up, 0 paths down (udp check)

  1. Shut down the ISL ports 1/35 and 1/36 on the Nexus 9336C-FX2 switch sh1.

    Show example 
    sh1# configure
Enter configuration commands, one per line. End with CNTL/Z.
sh1(config)# interface e1/35-36
sh1(config-if-range)# shutdown

  1. Remove all the cables from the Nexus 9336C-FX2 sh2 switch, and then connect them to the same ports on the Nexus C9336C-FX2 newsh2 switch.

  2. Bring up the ISLs ports 1/35 and 1/36 between the sh1 and newsh2 switches, and then verify the port channel operation status.

    Port-Channel should indicate Po1(SU) and Member Ports should indicate Eth1/35(P) and Eth1/36(P).

    Show example

    This example enables ISL ports 1/35 and 1/36 and displays the port channel summary on switch sh1.

    sh1# configure
Enter configuration commands, one per line. End with CNTL/Z.
sh1 (config)# int e1/35-36
sh1 (config-if-range)# no shutdown
sh1 (config-if-range)# show port-channel summary
Flags:  D - Down        P - Up in port-channel (members)
        I - Individual  H - Hot-standby (LACP only)
        s - Suspended   r - Module-removed
        b - BFD Session Wait
        S - Switched    R - Routed
        U - Up (port-channel)
        p - Up in delay-lacp mode (member)
        M - Not in use. Min-links not met
--------------------------------------------------------------------------------
Group Port-       Type     Protocol  Member       Ports
      Channel
--------------------------------------------------------------------------------
1     Po1(SU)     Eth      LACP      Eth1/35(P)   Eth1/36(P)

sh1 (config-if-range)#

  1. Verify that port e3b is up on all nodes:

    network port show ipspace Cluster

    Show example

    The output should be like the following:

    cluster1::*> network port show -ipspace Cluster

Node: node1
                                                                         Ignore
                                                   Speed(Mbps)  Health   Health
Port      IPspace      Broadcast Domain Link MTU   Admin/Oper   Status   Status
--------- ------------ ---------------- ---- ----- ---------- - - -------- ----
e3a       Cluster      Cluster          up   9000  auto/100000  healthy  false
e3b       Cluster      Cluster          up   9000  auto/100000  healthy  false

Node: node2
                                                                         Ignore
                                                   Speed(Mbps)  Health   Health
Port      IPspace      Broadcast Domain Link MTU   Admin/Oper   Status   Status
--------- ------------ ---------------- ---- ----- ----------- -  -------- ----
e3a       Cluster      Cluster          up   9000  auto/100000  healthy  false
e3b       Cluster      Cluster          up   9000  auto/auto    -        false
4 entries were displayed.

  1. On the same node you used in the previous step, revert the cluster LIF associated with the port in the previous step by using the network interface revert command.

    In this example, LIF node1_clus2 on node1 is successfully reverted if the Home value is true and the port is e3b.

    The following commands return LIF node1_clus2 on node1 to home port e3a and displays information about the LIFs on both nodes. Bringing up the first node is successful if the Is Home column is true for both cluster interfaces and they show the correct port assignments, in this example e3a and e3b on node1.

    Show example 
    cluster1::*> network interface show -vserver Cluster

            Logical      Status     Network            Current    Current Is
Vserver     Interface    Admin/Oper Address/Mask       Node       Port    Home
----------- ------------ ---------- ------------------ ---------- ------- -----
Cluster
            node1_clus1  up/up      169.254.209.69/16  node1      e3a     true
            node1_clus2  up/up      169.254.49.125/16  node1      e3b     true
            node2_clus1  up/up      169.254.47.194/16  node2      e3a     true
            node2_clus2  up/up      169.254.19.183/16  node2      e3a     false
4 entries were displayed.

  1. Display information about the nodes in a cluster:

    cluster show

    Show example

    This example shows that the node health for node1 and node2 in this cluster is true:

    cluster1::*> cluster show
Node          Health  Eligibility
------------- ------- ------------
node1         false   true
node2         true    true

  1. Verify that all physical cluster ports are up:

    network port show ipspace Cluster

    Show example 
    cluster1::*> network port show -ipspace Cluster

Node node1                                                                Ignore
                                                    Speed(Mbps)  Health   Health
Port      IPspace     Broadcast Domain  Link  MTU   Admin/Oper   Status   Status
--------- ----------- ----------------- ----- ----- ------------ -------- ------
e3a       Cluster     Cluster           up    9000  auto/100000  healthy  false
e3b       Cluster     Cluster           up    9000  auto/100000  healthy  false

Node: node2
                                                                          Ignore
                                                    Speed(Mbps)  Health   Health
Port      IPspace      Broadcast Domain Link  MTU   Admin/Oper   Status   Status
--------- ------------ ---------------- ----- ----- ------------ -------- ------
e3a       Cluster      Cluster          up    9000  auto/100000  healthy  false
e3b       Cluster      Cluster          up    9000  auto/100000  healthy  false
4 entries were displayed.

  1. Verify that all the cluster LIFs can communicate:

    cluster ping-cluster

    Show example 
    cluster1::*> cluster ping-cluster -node node2
Host is node2
Getting addresses from network interface table...
Cluster node1_clus1 169.254.209.69 node1 e3a
Cluster node1_clus2 169.254.49.125 node1 e3b
Cluster node2_clus1 169.254.47.194 node2 e3a
Cluster node2_clus2 169.254.19.183 node2 e3b
Local = 169.254.47.194 169.254.19.183
Remote = 169.254.209.69 169.254.49.125
Cluster Vserver Id = 4294967293
Ping status:
....
Basic connectivity succeeds on 4 path(s)
Basic connectivity fails on 0 path(s)
................
Detected 9000 byte MTU on 4 path(s):
Local 169.254.47.194 to Remote 169.254.209.69
Local 169.254.47.194 to Remote 169.254.49.125
Local 169.254.19.183 to Remote 169.254.209.69
Local 169.254.19.183 to Remote 169.254.49.125
Larger than PMTU communication succeeds on 4 path(s)
RPC status:
2 paths up, 0 paths down (tcp check)
2 paths up, 0 paths down (udp check)

  1. Confirm the following cluster network configuration:

    network port show

    Show example 
    cluster1::*> network port show -ipspace Cluster

Node: node1
                                                                        Ignore
                                       Speed(Mbps)             Health   Health
Port      IPspace     Broadcast Domain Link MTU   Admin/Oper   Status   Status
--------- ----------- ---------------- ---- ----- ------------ -------- ------
e3a       Cluster     Cluster          up   9000  auto/100000  healthy  false
e3b       Cluster     Cluster          up   9000  auto/100000  healthy  false

Node: node2
                                                                        Ignore
                                        Speed(Mbps)            Health   Health
Port      IPspace      Broadcast Domain Link MTU  Admin/Oper   Status   Status
--------- ------------ ---------------- ---- ---- ------------ -------- ------
e3a       Cluster      Cluster          up   9000 auto/100000  healthy  false
e3b       Cluster      Cluster          up   9000 auto/100000  healthy  false
4 entries were displayed.

cluster1::*> network interface show -vserver Cluster
            Logical    Status     Network            Current       Current Is
Vserver     Interface  Admin/Oper Address/Mask       Node          Port    Home
----------- ---------- ---------- ------------------ ------------- ------- ----
Cluster
            node1_clus1  up/up    169.254.209.69/16  node1         e3a     true
            node1_clus2  up/up    169.254.49.125/16  node1         e3b     true
            node2_clus1  up/up    169.254.47.194/16  node2         e3a     true
            node2_clus2  up/up    169.254.19.183/16  node2         e3b     true
4 entries were displayed.

cluster1::> network device-discovery show -protocol cdp
Node/       Local  Discovered
Protocol    Port   Device (LLDP: ChassisID)  Interface         Platform
----------- ------ ------------------------- ----------------  ----------------
node2      /cdp
            e3a    sh1    0/2               N9K-C9336C
            e3b    newsh2                    0/2               N9K-C9336C
node1      /cdp
            e3a    sh1                       0/1               N9K-C9336C
            e3b    newsh2                    0/1               N9K-C9336C
4 entries were displayed.

sh1# show cdp neighbors
Capability Codes: R - Router, T - Trans-Bridge, B - Source-Route-Bridge
                  S - Switch, H - Host, I - IGMP, r - Repeater,
                  V - VoIP-Phone, D - Remotely-Managed-Device,
                  s - Supports-STP-Dispute
Device-ID            Local Intrfce  Hldtme Capability  Platform      Port ID
node1                Eth1/1         144    H           FAS2980       e3a
node2                Eth1/2         145    H           FAS2980       e3a
newsh2               Eth1/35        176    R S I s     N9K-C9336C    Eth1/35
newsh2               Eth1/36        176    R S I s     N9K-C9336C    Eth1/36
Total entries displayed: 4

sh2# show cdp neighbors
Capability Codes: R - Router, T - Trans-Bridge, B - Source-Route-Bridge
                  S - Switch, H - Host, I - IGMP, r - Repeater,
                  V - VoIP-Phone, D - Remotely-Managed-Device,
                  s - Supports-STP-Dispute
Device-ID          Local Intrfce  Hldtme Capability  Platform      Port ID
node1              Eth1/1         139    H           FAS2980       e3b
node2              Eth1/2         124    H           FAS2980       eb
sh1                Eth1/35        178    R S I s     N9K-C9336C    Eth1/35
sh1                Eth1/36        178    R S I s     N9K-C9336C    Eth1/36
Total entries displayed: 4

  1. Enable the Ethernet switch health monitor log collection feature for collecting switch-related log files, using the following commands:

    • system switch ethernet log setup password

    • system switch ethernet log enable-collection

      Show example 
      cluster1::*> system switch ethernet log setup-password
Enter the switch name: <return>
The switch name entered is not recognized.
Choose from the following list:
sh1
sh2
cluster1::*> system switch ethernet log setup-password
Enter the switch name: sh1
RSA key fingerprint is e5:8b:c6:dc:e2:18:18:09:36:63:d9:63:dd:03:d9:cc
Do you want to continue? {y|n}::[n] y
Enter the password: <enter switch password>
Enter the password again: <enter switch password>
cluster1::*> system switch ethernet log setup-password
Enter the switch name: sh2
RSA key fingerprint is 57:49:86:a1:b9:80:6a:61:9a:86:8e:3c:e3:b7:1f:b1
Do you want to continue? {y|n}:: [n] y
Enter the password: <enter switch password>
Enter the password again: <enter switch password>
cluster1::*> system  switch ethernet log enable-collection
Do you want to enable cluster log collection for all nodes in the cluster? y|n}: [n] y
Enabling cluster switch log collection.
cluster1::*>
Note If any of these commands return an error, contact NetApp support.

  1. Move the storage ports from the old switch sh2 to the new switch newsh2.

  2. Verify the storage attached to HA pair 1, shared switch newsh2 is healthy.

  3. Verify the storage attached to HA pair 2, shared switch newsh2 is healthy:

    storage port show -port-type ENET

    Show example 
    storage::*> storage port show -port-type ENET
                                   Speed                            VLAN
Node    Port    Type    Mode       (Gb/s)      State     Status       ID
------- ------- ------- ---------- ----------- --------- --------- -----
node1
        e3a     ENET    storage          100   enabled   online       30
        e3b     ENET    storage            0   enabled   offline      30
        e7a     ENET    storage            0   enabled   offline      30
        e7b     ENET    storage          100   enabled   online       30

node2
        e3a     ENET    storage          100   enabled   online       30
        e3b     ENET    storage            0   enabled   offline      30
        e7a     ENET    storage            0   enabled   offline      30
        e7b     ENET    storage          100   enabled   online       30

  1. Verify that the shelves are correctly cabled:

    storage shelf port show -fields remote- device,remote-port

    Show example 
    cluster1::*> storage shelf port show -fields remote-device,remote-port
shelf id remote-port  remote-device
----- -- ------------ ----------------------------
3.20  0  Ethernet1/13 sh1
3.20  1  Ethernet1/13 newsh2
3.20  2  Ethernet1/14 sh1
3.20  3  Ethernet1/14 newsh2
3.30  0  Ethernet1/15 sh1
3.30  1  Ethernet1/15 newsh2
3.30  2  Ethernet1/16 sh1
3.30  3  Ethernet1/16 newsh2
8 entries were displayed.

  1. Remove the old switch sh2.

  2. Repeat these steps for the switch sh1 and new switch newsh1.

  3. If you suppressed automatic case creation, reenable it by invoking an AutoSupport message:

    system node autosupport invoke -node * -type all -message MAINT=END