Replace a NVIDIA SN2100 storage switch

07/30/2024 Contributors

PDFs

You can replace a defective NVIDIA SN2100 storage switch. This is a nondisruptive procedure.

What you'll need

Before installing the Cumulus software and RCFs on a NVIDIA SN2100 storage switch, ensure that:

Your system can support NVIDIA SN2100 storage switches.
You have downloaded the applicable RCFs.

The Hardware Universe provides full details of supported ports and their configurations.

The existing network configuration must have the following characteristics:

Make sure that all troubleshooting steps have been completed to confirm that your switch needs replacing.
Management connectivity must exist on both switches.

Make sure that all troubleshooting steps have been completed to confirm that your switch needs replacing.

The replacement NVIDIA SN2100 switch must have the following characteristics:

Management network connectivity is functional.
Console access to the replacement switch is in place.
The appropriate RCF and Cumulus operating system image is loaded onto the switch.
Initial customization of the switch is complete.

Procedure summary

This procedure replaces the second NVIDIA SN2100 storage switch sw2 with the new NVIDIA SN2100 switch nsw2. The two nodes are node1 and node2.

Steps to complete:

Confirm the switch to be replaced is sw2.
Disconnect the cables from switch sw2.
Reconnect the cables to switch nsw2.
Verify all device configurations on switch nsw2.

Steps

If AutoSupport is enabled on this cluster, suppress automatic case creation by invoking an AutoSupport message:

system node autosupport invoke -node * -type all - message MAINT=xh

x is the duration of the maintenance window in hours.
Change the privilege level to advanced, entering y when prompted to continue:

set -privilege advanced

Check on the health status of the storage node ports to make sure that there is connection to storage switch S1:

storage port show -port-type ENET

Show example

cluster1::*> storage port show -port-type ENET
                                  Speed                     VLAN
Node           Port Type  Mode    (Gb/s) State    Status      ID
-------------- ---- ----- ------- ------ -------- --------- ----
node1
               e3a  ENET  storage 100    enabled  online      30
               e3b  ENET  storage   0    enabled  offline     30
               e7a  ENET  storage   0    enabled  offline     30
               e7b  ENET  storage 100    enabled  online      30
node2
               e3a  ENET  storage 100    enabled  online      30
               e3b  ENET  storage   0    enabled  offline     30
               e7a  ENET  storage   0    enabled  offline     30
               e7b  ENET  storage 100    enabled  online      30
cluster1::*>

Verify that storage switch sw1 is available:

network device-discovery show -protocol lldp

Show example

cluster1::*> network device-discovery show -protocol lldp
Node/       Local  Discovered
Protocol    Port   Device (LLDP: ChassisID)  Interface         Platform
----------- ------ ------------------------- ----------------  ----------------
node1/lldp
            e0M    sw1 (00:ea:bd:68:6a:e8)   Eth1/46           -
            e0b    sw2 (6c:b2:ae:5f:a5:b2)   Ethernet1/16      -
            e0c    SHFFG1827000286 (d0:39:ea:1c:16:92)
                                             e0a               -
            e0e    sw3 (6c:b2:ae:5f:a5:ba)   Ethernet1/18      -
            e0f    SHFFG1827000286 (00:a0:98:fd:e4:a9)
                                             e0b               -
            e0g    sw4 (28:ac:9e:d5:4a:9c)   Ethernet1/11      -
            e0h    sw5 (6c:b2:ae:5f:a5:ca)   Ethernet1/22      -
            e1a    sw6 (00:f6:63:10:be:7c)   Ethernet1/33      -
            e1b    sw7 (00:f6:63:10:be:7d)   Ethernet1/34      -
            e2a    sw8 (b8:ce:f6:91:3d:88)   Ethernet1/35      -
Press <space> to page down, <return> for next line, or 'q' to quit...
10 entries were displayed.

Run the net show interface command on the working switch to confirm that you can see both nodes and all shelves:

net show interface

Show example

cumulus@sw1:~$ net show interface

State  Name    Spd   MTU    Mode        LLDP                  Summary
-----  ------  ----  -----  ----------  --------------------  --------------------
...
...
UP     swp1    100G  9216   Trunk/L2   node1 (e3a)             Master: bridge(UP)
UP     swp2    100G  9216   Trunk/L2   node2 (e3a)             Master: bridge(UP)
UP     swp3    100G  9216   Trunk/L2   SHFFG1826000112 (e0b)   Master: bridge(UP)
UP     swp4    100G  9216   Trunk/L2   SHFFG1826000112 (e0b)   Master: bridge(UP)
UP     swp5    100G  9216   Trunk/L2   SHFFG1826000102 (e0b)   Master: bridge(UP)
UP     swp6    100G  9216   Trunk/L2   SHFFG1826000102 (e0b)   Master: bridge(UP))
...
...

Verify the shelf ports in the storage system:

storage shelf port show -fields remote-device, remote-port

Show example

cluster1::*> storage shelf port show -fields remote-device, remote-port
shelf   id  remote-port   remote-device
-----   --  -----------   -------------
3.20    0   swp3          sw1
3.20    1   -             -
3.20    2   swp4          sw1
3.20    3   -             -
3.30    0   swp5          sw1
3.20    1   -             -
3.30    2   swp6          sw1
3.20    3   -             -
cluster1::*>

Remove all cables attached to storage switch sw2.
Reconnect all cables to the replacement switch nsw2.

Recheck the health status of the storage node ports:

storage port show -port-type ENET

Show example

cluster1::*> storage port show -port-type ENET
                                    Speed                     VLAN
Node             Port Type  Mode    (Gb/s) State    Status      ID
---------------- ---- ----- ------- ------ -------- --------- ----
node1
                 e3a  ENET  storage 100    enabled  online      30
                 e3b  ENET  storage   0    enabled  offline     30
                 e7a  ENET  storage   0    enabled  offline     30
                 e7b  ENET  storage 100    enabled  online      30
node2
                 e3a  ENET  storage 100    enabled  online      30
                 e3b  ENET  storage   0    enabled  offline     30
                 e7a  ENET  storage   0    enabled  offline     30
                 e7b  ENET  storage 100    enabled  online      30
cluster1::*>

Verify that both switches are available:

net device-discovery show -protocol lldp

Show example

cluster1::*> network device-discovery show -protocol lldp
Node/       Local  Discovered
Protocol    Port   Device (LLDP: ChassisID)  Interface         Platform
----------- ------ ------------------------- ----------------  ----------------
node1/lldp
            e0M    sw1 (00:ea:bd:68:6a:e8)   Eth1/46           -
            e0b    sw2 (6c:b2:ae:5f:a5:b2)   Ethernet1/16      -
            e0c    SHFFG1827000286 (d0:39:ea:1c:16:92)
                                             e0a               -
            e0e    sw3 (6c:b2:ae:5f:a5:ba)   Ethernet1/18      -
            e0f    SHFFG1827000286 (00:a0:98:fd:e4:a9)
                                             e0b               -
            e0g    sw4 (28:ac:9e:d5:4a:9c)   Ethernet1/11      -
            e0h    sw5 (6c:b2:ae:5f:a5:ca)   Ethernet1/22      -
            e1a    sw6 (00:f6:63:10:be:7c)   Ethernet1/33      -
            e1b    sw7 (00:f6:63:10:be:7d)   Ethernet1/34      -
            e2a    sw8 (b8:ce:f6:91:3d:88)   Ethernet1/35      -
Press <space> to page down, <return> for next line, or 'q' to quit...
10 entries were displayed.

Verify the shelf ports in the storage system:

storage shelf port show -fields remote-device, remote-port

Show example

cluster1::*> storage shelf port show -fields remote-device, remote-port
shelf   id    remote-port     remote-device
-----   --    -----------     -------------
3.20    0     swp3            sw1
3.20    1     swp3            nsw2
3.20    2     swp4            sw1
3.20    3     swp4            nsw2
3.30    0     swp5            sw1
3.20    1     swp5            nsw2
3.30    2     swp6            sw1
3.20    3     swp6            nsw2
cluster1::*>

Change the privilege level back to admin:

set -privilege admin
If you suppressed automatic case creation, re-enable it by invoking an AutoSupport message:

system node autosupport invoke -node * -type all -message MAINT=END

What's next?

Configure switch health monitoring.

Replace a NVIDIA SN2100 storage switch

Creating your file...