Skip to main content
Cluster and storage switches

Review cabling and configuration considerations

Contributors netapp-yvonneo netapp-jsnyder netapp-jolieg

Before configuring your NVIDIA SN2100 switch, review the following considerations.

NVIDIA port details

Switch ports

Ports usage

swp1s0-3

4x10GbE breakout cluster port nodes

swp2s0-3

4x25GbE breakout cluster port nodes

swp3-14

40/100GbE cluster port nodes

swp15-16

100GbE Inter-Switch Link (ISL) ports

See the Hardware Universe for more information on switch ports.

If you are experiencing link-up delays of more than five seconds, Cumulus Linux 5.4 and later includes support for fast link-up. You can configure the links by using the nv set command as follows:

nv set interface <interface-id> link fast-linkup on
nv config apply
reload the switchd
Show example
cumulus@cumulus-cs13:mgmt:~$ nv set interface swp5 link fast-linkup on
cumulus@cumulus-cs13:mgmt:~$ nv config apply
switchd need to reload on this config change

Are you sure? [y/N] y
applied [rev_id: 22]

Only switchd reload required

Support for copper connections

The following configuration changes are required to fix this issue.

Cumulus Linux 4.4.3
  1. Identify the name for each interface using 40GbE/100GbE copper cables:

    cumulus@cumulus:mgmt:~$ net show interface pluggables
    
    Interface  Identifier     Vendor Name     Vendor PN    Vendor SN      Vendor Rev
    ---------  -------------  --------------  -----------  -------------  ----------
    swp3       0x11 (QSFP28)  Molex           112-00576    93A2229911111  B0
    swp4       0x11 (QSFP28)  Molex           112-00576    93A2229922222  B0
  2. Add the following two lines to the /etc/cumulus/switchd.conf file for every port (swp<n>) that is using 40GbE/100GbE copper cables:

    • interface.swp<n>.enable_media_depended_linkup_flow=TRUE

    • interface.swp<n>.enable_short_tuning=TRUE

      For example:

      cumulus@cumulus:mgmt:~$ sudo nano /etc/cumulus/switchd.conf
      .
      .
      interface.swp3.enable_media_depended_linkup_flow=TRUE
      interface.swp3.enable_short_tuning=TRUE
      interface.swp4.enable_media_depended_linkup_flow=TRUE
      interface.swp4.enable_short_tuning=TRUE
  3. Restart the switchd service:

    cumulus@cumulus:mgmt:~$ sudo systemctl restart switchd.service
  4. Confirm that the ports are up:

    cumulus@cumulus:mgmt:~$ net show interface all
    
    State  Name      Spd   MTU    Mode       LLDP              Summary
    -----  --------- ----  -----  ---------- ----------------- --------
    UP     swp3      100G  9216   Trunk/L2                     Master: bridge(UP)
    UP     swp4      100G  9216   Trunk/L2                     Master: bridge(UP)
Cumulus Linux 5.x
  1. Identify the name for each interface using 40GbE/100GbE copper cables:

    cumulus@cumulus:mgmt:~$ nv show interface --view=pluggables
    
    Interface  Identifier     Vendor Name     Vendor PN    Vendor SN      Vendor Rev
    ---------  -------------  --------------  -----------  -------------  ----------
    swp3       0x11 (QSFP28)  Molex           112-00576    93A2229911111  B0
    swp4       0x11 (QSFP28)  Molex           112-00576    93A2229922222  B0
  2. Configure the links using the nv set command as follows:

    • nv set interface <interface-id> link fast-linkup on

    • nv config apply

    • Reload the switchd service

      For example:

      cumulus@cumulus:mgmt:~$ nv set interface swp5 link fast-linkup on
      cumulus@cumulus:mgmt:~$ nv config apply
      switchd need to reload on this config change
      
      Are you sure? [y/N] y
      applied [rev_id: 22]
      
      Only switchd reload required
  3. Confirm that the ports are up:

    cumulus@cumulus:mgmt:~$ net show interface all
    
    State  Name      Spd   MTU    Mode       LLDP              Summary
    -----  --------- ----  -----  ---------- ----------------- --------
    UP     swp3      100G  9216   Trunk/L2                     Master: bridge(UP)
    UP     swp4      100G  9216   Trunk/L2                     Master: bridge(UP)

See the Knowledge Base article SN2100 switch fails to connect using 40/100GbE copper cables for further details.

On Cumulus Linux 4.4.2, copper connections are not supported on SN2100 switches with X1151A NIC, X1146A NIC, or onboard 100GbE ports. For example:

  • AFF A800 on ports e0a and e0b

  • AFF A320 on ports e0g and e0h

QSA adapter

When a QSA adapter is used to connect to the 10GbE/25GbE cluster ports on a platform, the link might not come up.

To resolve this issue, do the following:

  • For 10GbE, manually set the swp1s0-3 link speed to 10000 and set auto-negotiation to off.

  • For 25GbE, manually set the swp2s0-3 link speed to 25000 and set auto-negotiation to off.

Note When using 10GbE/25GbE QSA adapters, insert them in non-breakout 40GbE/100GbE ports (swp3-swp14). Do not insert the QSA adapter in a port that is configured for breakout.

Setting interface speed on breakout ports

Depending on the transceiver in the switch port, you might need to set the speed on the switch interface to a fixed speed. If using 10GbE and 25GbE breakout ports, verify that auto-negotiation is off and set the interface speed on the switch.

Cumulus Linux 4.4.3

For example:

cumulus@cumulus:mgmt:~$ net add int swp1s3 link autoneg off && net com
--- /etc/network/interfaces     2019-11-17 00:17:13.470687027 +0000
+++ /run/nclu/ifupdown2/interfaces.tmp  2019-11-24 00:09:19.435226258 +0000
@@ -37,21 +37,21 @@
     alias 10G Intra-Cluster Node
     link-autoneg off
     link-speed 10000  <---- port speed set
     mstpctl-bpduguard yes
     mstpctl-portadminedge yes
     mtu 9216

auto swp1s3
iface swp1s3
     alias 10G Intra-Cluster Node
-    link-autoneg off
+    link-autoneg on
     link-speed 10000 <---- port speed set
     mstpctl-bpduguard yes
     mstpctl-portadminedge yes
     mtu 9216

auto swp2s0
iface swp2s0
     alias 25G Intra-Cluster Node
     link-autoneg off
     link-speed 25000 <---- port speed set

Check the interface and port status to verify that the settings are applied:

cumulus@cumulus:mgmt:~$ net show interface

State  Name      Spd    MTU    Mode        LLDP             Summary
-----  --------  -----  -----  ----------  ---------------  --------------------------------------
.
.
UP     swp1s0     10G   9216   Trunk/L2    cs07 (e4c)       Master: br_default(UP)
UP     swp1s1     10G   9216   Trunk/L2    cs07 (e4d)       Master: br_default(UP)
UP     swp1s2     10G   9216   Trunk/L2    cs08 (e4c)       Master: br_default(UP)
UP     swp1s3     10G   9216   Trunk/L2    cs08 (e4d)       Master: br_default(UP)
.
.
UP     swp3       40G   9216   Trunk/L2    cs03 (e4e)       Master: br_default(UP)
UP     swp4       40G   9216   Trunk/L2    cs04 (e4e)       Master: br_default(UP)
DN     swp5       N/A   9216   Trunk/L2                     Master: br_default(UP)
DN     swp6       N/A   9216   Trunk/L2                     Master: br_default(UP)
DN     swp7       N/A   9216   Trunk/L2                     Master: br_default(UP)
.
.
UP     swp15      100G  9216   BondMember  cs01 (swp15)     Master: cluster_isl(UP)
UP     swp16      100G  9216   BondMember  cs01 (swp16)     Master: cluster_isl(UP)
.
.
Cumulus Linux 5.x

For example:

cumulus@cumulus:mgmt:~$ nv set interface swp1s3 link auto-negotiate off
cumulus@cumulus:mgmt:~$ nv set interface swp1s3 link speed 10G
cumulus@cumulus:mgmt:~$ nv show interface swp1s3

link                                                                                            
  auto-negotiate        off                     off                     off                   
  duplex                full                    full                    full                  
  speed                 10G                     10G                     10G                   
  fec                   auto                    auto                    auto                  
  mtu                   9216                    9216                    9216                  
[breakout]                                                                                    
  state                 up                      up                      up

Check the interface and port status to verify that the settings are applied:

cumulus@cumulus:mgmt:~$ nv show interface

State  Name      Spd    MTU    Mode        LLDP             Summary
-----  --------  -----  -----  ----------  ---------------  --------------------------------------
.
.
UP     swp1s0     10G   9216   Trunk/L2    cs07 (e4c)       Master: br_default(UP)
UP     swp1s1     10G   9216   Trunk/L2    cs07 (e4d)       Master: br_default(UP)
UP     swp1s2     10G   9216   Trunk/L2    cs08 (e4c)       Master: br_default(UP)
UP     swp1s3     10G   9216   Trunk/L2    cs08 (e4d)       Master: br_default(UP)
.
.
UP     swp3       40G   9216   Trunk/L2    cs03 (e4e)       Master: br_default(UP)
UP     swp4       40G   9216   Trunk/L2    cs04 (e4e)       Master: br_default(UP)
DN     swp5       N/A   9216   Trunk/L2                     Master: br_default(UP)
DN     swp6       N/A   9216   Trunk/L2                     Master: br_default(UP)
DN     swp7       N/A   9216   Trunk/L2                     Master: br_default(UP)
.
.
UP     swp15      100G  9216   BondMember  cs01 (swp15)     Master: cluster_isl(UP)
UP     swp16      100G  9216   BondMember  cs01 (swp16)     Master: cluster_isl(UP)
.
.