Review cabling and configuration considerations
Suggest changes
PDFs
Before configuring your NVIDIA SN2100 switch, review the following considerations.
NVIDIA port details
Switch ports |
Ports usage |
|---|---|
swp1s0-3 |
4x10GbE breakout cluster port nodes |
swp2s0-3 |
4x25GbE breakout cluster port nodes |
swp3-14 |
40/100GbE cluster port nodes |
swp15-16 |
100GbE Inter-Switch Link (ISL) ports |
See the Hardware Universe for more information on switch ports.
Link-up delays with optical connections
If you are experiencing link-up delays of more than five seconds, Cumulus Linux 5.4 and later includes support for fast link-up. You can configure the links by using the nv set command as follows:
nv set interface <interface-id> link fast-linkup on nv config apply reload the switchd
Show example
cumulus@cumulus-cs13:mgmt:~$ nv set interface swp5 link fast-linkup on cumulus@cumulus-cs13:mgmt:~$ nv config apply switchd need to reload on this config change Are you sure? [y/N] y applied [rev_id: 22] Only switchd reload required
Support for copper connections
The following configuration changes are required to fix this issue.
-
Identify the name for each interface using 40GbE/100GbE copper cables:
cumulus@cumulus:mgmt:~$ net show interface pluggables Interface Identifier Vendor Name Vendor PN Vendor SN Vendor Rev --------- ------------- -------------- ----------- ------------- ---------- swp3 0x11 (QSFP28) Molex 112-00576 93A2229911111 B0 swp4 0x11 (QSFP28) Molex 112-00576 93A2229922222 B0
-
Add the following two lines to the
/etc/cumulus/switchd.conffile for every port (swp<n>) that is using 40GbE/100GbE copper cables:-
interface.swp<n>.enable_media_depended_linkup_flow=TRUE -
interface.swp<n>.enable_short_tuning=TRUEFor example:
cumulus@cumulus:mgmt:~$ sudo nano /etc/cumulus/switchd.conf . . interface.swp3.enable_media_depended_linkup_flow=TRUE interface.swp3.enable_short_tuning=TRUE interface.swp4.enable_media_depended_linkup_flow=TRUE interface.swp4.enable_short_tuning=TRUE
-
-
Restart the
switchdservice:cumulus@cumulus:mgmt:~$ sudo systemctl restart switchd.service
-
Confirm that the ports are up:
cumulus@cumulus:mgmt:~$ net show interface all State Name Spd MTU Mode LLDP Summary ----- --------- ---- ----- ---------- ----------------- -------- UP swp3 100G 9216 Trunk/L2 Master: bridge(UP) UP swp4 100G 9216 Trunk/L2 Master: bridge(UP)
-
Identify the name for each interface using 40GbE/100GbE copper cables:
cumulus@cumulus:mgmt:~$ nv show interface pluggables Interface Identifier Vendor Name Vendor PN Vendor SN Vendor Rev --------- ------------- -------------- ----------- ------------- ---------- swp3 0x11 (QSFP28) Molex 112-00576 93A2229911111 B0 swp4 0x11 (QSFP28) Molex 112-00576 93A2229922222 B0
-
Configure the links using the
nv setcommand as follows:-
nv set interface <interface-id> link fast-linkup on -
nv config apply -
Reload the
switchdserviceFor example:
cumulus@cumulus:mgmt:~$ nv set interface swp5 link fast-linkup on cumulus@cumulus:mgmt:~$ nv config apply switchd need to reload on this config change Are you sure? [y/N] y applied [rev_id: 22] Only switchd reload required
-
-
Confirm that the ports are up:
cumulus@cumulus:mgmt:~$ net show interface all State Name Spd MTU Mode LLDP Summary ----- --------- ---- ----- ---------- ----------------- -------- UP swp3 100G 9216 Trunk/L2 Master: bridge(UP) UP swp4 100G 9216 Trunk/L2 Master: bridge(UP)
See the Knowledge Base article SN2100 switch fails to connect using 40/100GbE copper cables for further details.
On Cumulus Linux 4.4.2, copper connections are not supported on SN2100 switches with X1151A NIC, X1146A NIC, or onboard 100GbE ports. For example:
-
AFF A800 on ports e0a and e0b
-
AFF A320 on ports e0g and e0h
QSA adapter
When a QSA adapter is used to connect to the 10GbE/25GbE cluster ports on a platform, the link might not come up.
To resolve this issue, do the following:
-
For 10GbE, manually set the swp1s0-3 link speed to 10000 and set auto-negotiation to off.
-
For 25GbE, manually set the swp2s0-3 link speed to 25000 and set auto-negotiation to off.
|
When using 10GbE/25GbE QSA adapters, insert them in non-breakout 40GbE/100GbE ports (swp3-swp14). Do not insert the QSA adapter in a port that is configured for breakout. |
Setting interface speed on breakout ports
Depending on the transceiver in the switch port, you might need to set the speed on the switch interface to a fixed speed. If using 10GbE and 25GbE breakout ports, verify that auto-negotiation is off and set the interface speed on the switch.
For example:
cumulus@cumulus:mgmt:~$ net add int swp1s3 link autoneg off && net com
--- /etc/network/interfaces 2019-11-17 00:17:13.470687027 +0000
+++ /run/nclu/ifupdown2/interfaces.tmp 2019-11-24 00:09:19.435226258 +0000
@@ -37,21 +37,21 @@
alias 10G Intra-Cluster Node
link-autoneg off
link-speed 10000 <---- port speed set
mstpctl-bpduguard yes
mstpctl-portadminedge yes
mtu 9216
auto swp1s3
iface swp1s3
alias 10G Intra-Cluster Node
- link-autoneg off
+ link-autoneg on
link-speed 10000 <---- port speed set
mstpctl-bpduguard yes
mstpctl-portadminedge yes
mtu 9216
auto swp2s0
iface swp2s0
alias 25G Intra-Cluster Node
link-autoneg off
link-speed 25000 <---- port speed set
Check the interface and port status to verify that the settings are applied:
cumulus@cumulus:mgmt:~$ net show interface State Name Spd MTU Mode LLDP Summary ----- -------- ----- ----- ---------- --------------- -------------------------------------- . . UP swp1s0 10G 9216 Trunk/L2 cs07 (e4c) Master: br_default(UP) UP swp1s1 10G 9216 Trunk/L2 cs07 (e4d) Master: br_default(UP) UP swp1s2 10G 9216 Trunk/L2 cs08 (e4c) Master: br_default(UP) UP swp1s3 10G 9216 Trunk/L2 cs08 (e4d) Master: br_default(UP) . . UP swp3 40G 9216 Trunk/L2 cs03 (e4e) Master: br_default(UP) UP swp4 40G 9216 Trunk/L2 cs04 (e4e) Master: br_default(UP) DN swp5 N/A 9216 Trunk/L2 Master: br_default(UP) DN swp6 N/A 9216 Trunk/L2 Master: br_default(UP) DN swp7 N/A 9216 Trunk/L2 Master: br_default(UP) . . UP swp15 100G 9216 BondMember cs01 (swp15) Master: cluster_isl(UP) UP swp16 100G 9216 BondMember cs01 (swp16) Master: cluster_isl(UP) . .
For example:
cumulus@cumulus:mgmt:~$ nv set interface swp1s3 link auto-negotiate off cumulus@cumulus:mgmt:~$ nv set interface swp1s3 link speed 10G cumulus@cumulus:mgmt:~$ nv show interface swp1s3 link auto-negotiate off off off duplex full full full speed 10G 10G 10G fec auto auto auto mtu 9216 9216 9216 [breakout] state up up up
Check the interface and port status to verify that the settings are applied:
cumulus@cumulus:mgmt:~$ nv show interface State Name Spd MTU Mode LLDP Summary ----- -------- ----- ----- ---------- --------------- -------------------------------------- . . UP swp1s0 10G 9216 Trunk/L2 cs07 (e4c) Master: br_default(UP) UP swp1s1 10G 9216 Trunk/L2 cs07 (e4d) Master: br_default(UP) UP swp1s2 10G 9216 Trunk/L2 cs08 (e4c) Master: br_default(UP) UP swp1s3 10G 9216 Trunk/L2 cs08 (e4d) Master: br_default(UP) . . UP swp3 40G 9216 Trunk/L2 cs03 (e4e) Master: br_default(UP) UP swp4 40G 9216 Trunk/L2 cs04 (e4e) Master: br_default(UP) DN swp5 N/A 9216 Trunk/L2 Master: br_default(UP) DN swp6 N/A 9216 Trunk/L2 Master: br_default(UP) DN swp7 N/A 9216 Trunk/L2 Master: br_default(UP) . . UP swp15 100G 9216 BondMember cs01 (swp15) Master: cluster_isl(UP) UP swp16 100G 9216 BondMember cs01 (swp16) Master: cluster_isl(UP) . .