Install the RCF script

Contributors netapp-yvonneo

Before installing the RCF script, ensure that the following are available on the switch:

  • Cumulus Linux 4.4.2 is installed.

  • IP address, subnet mask, and default gateway defined via DHCP or manually configured.

Important See Cabling and configuration considerations for caveats and further details.
Current RCF script versions

There are two RCF scripts available for Clustering and Storage applications. The procedure for each is the same.

  • Clustering: MSN2100-RCF-v1.8-Cluster

  • Storage: MSN2100-RCF-v1.8-Storage

Note The following example procedure shows how to download and apply the RCF script for Cluster switches.
Note Example command output uses switch management IP address 10.233.204.71, netmask 255.255.254.0 and default gateway 10.233.204.1.
Steps
  1. Display the available interfaces on the SN2100 switch:

    cumulus@cumulus:mgmt:~$ net show interface all
    
    State  Name   Spd  MTU    Mode         LLDP                Summary
    -----  -----  ---  -----  -----------  ------------------  --------------
    ...
    ...
    ADMDN  swp1   N/A  9216   NotConfigured
    ADMDN  swp2   N/A  9216   NotConfigured
    ADMDN  swp3   N/A  9216   NotConfigured
    ADMDN  swp4   N/A  9216   NotConfigured
    ADMDN  swp5   N/A  9216   NotConfigured
    ADMDN  swp6   N/A  9216   NotConfigured
    ADMDN  swp7   N/A  9216   NotConfigure
    ADMDN  swp8   N/A  9216   NotConfigured
    ADMDN  swp9   N/A  9216   NotConfigured
    ADMDN  swp10  N/A  9216   NotConfigured
    ADMDN  swp11  N/A  9216   NotConfigured
    ADMDN  swp12  N/A  9216   NotConfigured
    ADMDN  swp13  N/A  9216   NotConfigured
    ADMDN  swp14  N/A  9216   NotConfigured
    ADMDN  swp15  N/A  9216   NotConfigured
    ADMDN  swp16  N/A  9216   NotConfigured
  2. Copy the RCF python script to the switch:

    cumulus@cumulus:mgmt:~$ pwd
    /home/cumulus
    cumulus@cumulus:mgmt: /tmp$ scp <user>@<host:/<path>/MSN2100-RCF-v1.8-Cluster
    ssologin@10.233.204.71's password:
    MSN2100-RCF-v1.8-Cluster                 100% 8607   111.2KB/s         00:00
  3. Apply the RCF python script MSN2100-RCF-v1.8-Cluster:

    cumulus@cumulus:mgmt:/tmp$ sudo python3 MSN2100-RCF-v1.8-Cluster
    [sudo] password for cumulus:
    ...
    Step 1: Creating the banner file
    Step 2: Registering banner message
    Step 3: Updating the MOTD file
    Step 4: Ensuring passwordless use of cl-support command by admin
    Step 5: Disabling apt-get
    Step 6: Creating the interfaces
    Step 7: Adding the interface config
    Step 8: Disabling cdp
    Step 9: Adding the lldp config
    Step 10: Adding the RoCE base config
    Step 11: Modifying RoCE Config
    Step 12: Configure SNMP
    Step 13: Reboot the switch

    The RCF script completes the following steps:

    1. Updates the banner MOTD

    2. Disables the apt-get for OS updates

    3. Defines breakout and non-breakout interfaces

    4. Configures interfaces and SNMP

    5. Disables CDP

    6. Changes the LLDP configuration

    7. Adds a RoCE configuration

    8. Modifies the RoCE configuration for HA and Cluster RDMA

    9. Reboots the switch

      Note For any RCF python script issues that cannot be corrected, contact NetApp Support for assistance.
  4. Verify the configuration after the reboot:

    cumulus@cumulus:mgmt:~$ net show interface all
    
    State  Name      Spd   MTU    Mode       LLDP              Summary
    -----  --------- ----  -----  ---------- ----------------- --------
    ...
    ...
    DN     swp1s0    N/A   9216   Trunk/L2                     Master: bridge(UP)
    DN     swp1s1    N/A   9216   Trunk/L2                     Master: bridge(UP)
    DN     swp1s2    N/A   9216   Trunk/L2                     Master: bridge(UP)
    DN     swp1s3    N/A   9216   Trunk/L2                     Master: bridge(UP)
    DN     swp2s0    N/A   9216   Trunk/L2                     Master: bridge(UP)
    DN     swp2s1    N/A   9216   Trunk/L2                     Master: bridge(UP)
    DN     swp2s2    N/A   9216   Trunk/L2                     Master: bridge(UP)
    DN     swp2s3    N/A   9216   Trunk/L2                     Master: bridge(UP)
    UP     swp3      100G  9216   Trunk/L2                     Master: bridge(UP)
    UP     swp4      100G  9216   Trunk/L2                     Master: bridge(UP)
    DN     swp5      N/A   9216   Trunk/L2                     Master: bridge(UP)
    DN     swp6      N/A   9216   Trunk/L2                     Master: bridge(UP)
    DN     swp7      N/A   9216   Trunk/L2                     Master: bridge(UP)
    DN     swp8      N/A   9216   Trunk/L2                     Master: bridge(UP)
    DN     swp9      N/A   9216   Trunk/L2                     Master: bridge(UP)
    DN     swp10     N/A   9216   Trunk/L2                     Master: bridge(UP)
    DN     swp11     N/A   9216   Trunk/L2                     Master: bridge(UP)
    DN     swp12     N/A   9216   Trunk/L2                     Master: bridge(UP)
    DN     swp13     N/A   9216   Trunk/L2                     Master: bridge(UP)
    DN     swp14     N/A   9216   Trunk/L2                     Master: bridge(UP)
    UP     swp15     N/A   9216   BondMember                   Master: bond_15_16(UP)
    UP     swp16     N/A   9216   BondMember                   Master: bond_15_16(UP)
    ...
    ...
    
    cumulus@cumulus:mgmt:~$ net show roce config
    RoCE mode.......... lossless
    Congestion Control:
      Enabled SPs.... 0 2 5
      Mode........... ECN
      Min Threshold.. 150 KB
      Max Threshold.. 1500 KB
    PFC:
      Status......... enabled
      Enabled SPs.... 2 5
      Interfaces......... swp10-16,swp1s0-3,swp2s0-3,swp3-9
    
    DSCP                     802.1p  switch-priority
    -----------------------  ------  ---------------
    0 1 2 3 4 5 6 7               0                0
    8 9 10 11 12 13 14 15         1                1
    16 17 18 19 20 21 22 23       2                2
    24 25 26 27 28 29 30 31       3                3
    32 33 34 35 36 37 38 39       4                4
    40 41 42 43 44 45 46 47       5                5
    48 49 50 51 52 53 54 55       6                6
    56 57 58 59 60 61 62 63       7                7
    
    switch-priority  TC  ETS
    ---------------  --  --------
    0 1 3 4 6 7       0  DWRR 28%
    2                 2  DWRR 28%
    5                 5  DWRR 43%
  5. Verify information for the transceiver in the interface: net show interface pluggables

    cumulus@cumulus:mgmt:~$ net show interface pluggables
    Interface  Identifier     Vendor Name  Vendor PN        Vendor SN       Vendor Rev
    ---------  -------------  -----------  ---------------  --------------  ----------
    swp3       0x11 (QSFP28)  Amphenol     112-00574        APF20379253516  B0
    swp4       0x11 (QSFP28)  AVAGO        332-00440        AF1815GU05Z     A0
    swp15      0x11 (QSFP28)  Amphenol     112-00573        APF21109348001  B0
    swp16      0x11 (QSFP28)  Amphenol     112-00573        APF21109347895  B0
  6. Verify that the nodes each have a connection to each switch: net show lldp

    cumulus@cumulus:mgmt:~$ net show lldp
    
    LocalPort  Speed  Mode        RemoteHost              RemotePort
    ---------  -----  ----------  ----------------------  -----------
    swp3       100G   Trunk/L2    sw1                     e3a
    swp4       100G   Trunk/L2    sw2                     e3b
    swp15      100G   BondMember  sw13                    swp15
    swp16      100G   BondMember  sw14                    swp16
  7. Verify the health of cluster ports on the cluster.

    1. Verify that e0d ports are up and healthy across all nodes in the cluster: network port show -role cluster

      cluster1::*> network port show -role cluster
      
      Node: node1
                                                                             Ignore
                                                        Speed(Mbps) Health   Health
      Port      IPspace      Broadcast Domain Link MTU  Admin/Oper  Status   Status
      --------- ------------ ---------------- ---- ---- ----------- -------- ------
      e3a       Cluster      Cluster          up   9000  auto/10000 healthy  false
      e3b       Cluster      Cluster          up   9000  auto/10000 healthy  false
      
      Node: node2
                                                                             Ignore
                                                        Speed(Mbps) Health   Health
      Port      IPspace      Broadcast Domain Link MTU  Admin/Oper  Status   Status
      --------- ------------ ---------------- ---- ---- ----------- -------- ------
      e3a       Cluster      Cluster          up   9000  auto/10000 healthy  false
      e3b       Cluster      Cluster          up   9000  auto/10000 healthy  false
    2. Verify the switch health from the cluster (this might not show switch sw2, since LIFs are not homed on e0d).

      cluster1::*> network device-discovery show -protocol lldp
      Node/       Local  Discovered
      Protocol    Port   Device (LLDP: ChassisID)  Interface Platform
      ----------- ------ ------------------------- --------- ----------
      node1/lldp
                  e3a    sw1 (b8:ce:f6:19:1a:7e)   swp3      -
                  e3b    sw2 (b8:ce:f6:19:1b:96)   swp3      -
      
      node2/lldp
                  e3a    sw1 (b8:ce:f6:19:1a:7e)   swp4      -
                  e3b    sw2 (b8:ce:f6:19:1b:96)   swp4      -
      
      
      cluster1::*> system switch ethernet show -is-monitoring-enabled-operational true
      Switch                      Type               Address          Model
      --------------------------- ------------------ ---------------- -----
      sw1                         cluster-network    10.233.205.90    MSN2100-CB2RC
           Serial Number: MNXXXXXXGD
            Is Monitored: true
                  Reason: None
        Software Version: Cumulus Linux version 4.4.2 running on Mellanox
                          Technologies Ltd. MSN2100
          Version Source: LLDP
      
      sw2                         cluster-network    10.233.205.91    MSN2100-CB2RC
           Serial Number: MNCXXXXXXGS
            Is Monitored: true
                  Reason: None
        Software Version: Cumulus Linux version 4.4.2 running on Mellanox
                          Technologies Ltd. MSN2100
          Version Source: LLDP