Skip to main content
Install and maintain

Install the RCF script on NVIDIA SN2100 storage switches

Contributors netapp-camdenc netapp-yvonneo netapp-jolieg

Install the reference configuration file (RCF) script on NVIDIA SN2100 storage switches to configure port assignments and network settings.

Before you begin

Before installing the RCF script, make sure that the following are available on the switch:

  • Supported version of Cumulus Linux is installed.

  • Management port IP address, subnet mask, and default gateway defined via DHCP or manually configured.

RCF script version

Select the appropriate storage RCF script version based on your current ONTAP and Cumulus Linux version. Download RCFs from NVIDIA SN2100 software download page.

About the examples

The following example procedure shows how to download and apply the RCF script for Storage switches.

Step 1: Install the RCF on the switch

  1. Display the available interfaces on the SN2100 switch:

    nv show interface
    Show example
    cumulus@cumulus:mgmt:~$ nv show interface
    
    State  Name   Spd  MTU    Mode         LLDP                Summary
    -----  -----  ---  -----  -----------  ------------------  --------------
    ...
    ...
    ADMDN  swp1   N/A  9216   NotConfigured
    ADMDN  swp2   N/A  9216   NotConfigured
    ADMDN  swp3   N/A  9216   NotConfigured
    ADMDN  swp4   N/A  9216   NotConfigured
    ADMDN  swp5   N/A  9216   NotConfigured
    ADMDN  swp6   N/A  9216   NotConfigured
    ADMDN  swp7   N/A  9216   NotConfigured
    ADMDN  swp8   N/A  9216   NotConfigured
    ADMDN  swp9   N/A  9216   NotConfigured
    ADMDN  swp10  N/A  9216   NotConfigured
    ADMDN  swp11  N/A  9216   NotConfigured
    ADMDN  swp12  N/A  9216   NotConfigured
    ADMDN  swp13  N/A  9216   NotConfigured
    ADMDN  swp14  N/A  9216   NotConfigured
    ADMDN  swp15  N/A  9216   NotConfigured
    ADMDN  swp16  N/A  9216   NotConfigured
  2. Copy the RCF python script to the switch:

    admin@sw1:mgmt:~$ pwd
    /home/cumulus
    cumulus@cumulus:mgmt:~$ cd /tmp
    cumulus@cumulus:mgmt:/tmp$ scp <user>@<host:/<path>/RCF-v1.14-MSN2100-ST1
    ssologin@10.233.204.71's password:
    RCF-v1.14-MSN2100-ST1                 100% 8607   111.2KB/s         00:00
  3. Apply the RCF python script RCF-v1.14-MSN2100-ST1:

    cumulus@cumulus:mgmt:/tmp$ RCF-v1.14-MSN2100-ST1
    [sudo] password for cumulus:
    ...
    Step 1: Creating the banner file
    Step 2: Registering banner message
    Step 3: Updating the MOTD file
    Step 4: Ensuring passwordless use of cl-support command by admin
    Step 5: Disabling apt-get
    Step 6: Creating the interfaces
    Step 7: Adding the interface config
    Step 8: Disabling cdp
    Step 9: Adding the lldp config
    Step 10: Adding the RoCE base config
    Step 11: Modifying RoCE Config
    Step 12: Configure SNMP
    Step 13: Reboot the switch

    The RCF script completes the steps listed above.

    Note For any RCF python script issues that cannot be corrected, contact NetApp Support for assistance.
  4. Reapply any previous customizations to the switch configuration. Refer to Review cabling and configuration considerations for details of any further changes required.

  5. Connect the controller storage ports and NS224 shelf ports to the switches.

Step 2: Verify the switch and storage connectivity

  1. Verify the configuration after the reboot:

    nv show interface
    Show example
    cumulus@cumulus:mgmt:~$ nv show interface
    Interface   Admin Status  Oper Status  Speed  MTU    Type      Remote Host                         Remote Port  Summary
    ----------  ------------  -----------  -----  -----  --------  ----------------------------------  -----------  --------------------------------------------------------
    bond998     up            down                9216   bond
    bond999     up            down                9216   bond
    br_default  up            up                  9216   bridge                                                     IPv6 Address:               fe80::1234:5678:9abc:def0/64
    eth0        up            up           100M   1500   eth       mgmt-switch                         Eth105/1/17  IPv4 Address:                             192.0.2.209/24
                                                                                                                    Address type:                                    primary
                                                                                                                    IPv6 Address:                        2001:db8:1::495e/64
                                                                                                                    IPv6 Address:               fe80::1234:5678:9abc:def0/64
    lo          up            unknown             65536  loopback                                                   IPv4 Address:                                127.0.0.1/8
                                                                                                                    Address type:                                    primary
                                                                                                                    IPv6 Address:                                    ::1/128
    mgmt        up            up                  65575  vrf                                                        IPv4 Address:                                127.0.0.1/8
                                                                                                                    IPv4 Address:                                127.0.1.1/8
                                                                                                                    Address type:                                    primary
                                                                                                                    Address type:                                  secondary
                                                                                                                    IPv6 Address:                                    ::1/128
    swp1        up            up           100G   9216   swp       node1                               e3b
    swp2        up            up           100G   9216   swp       node2                               e3b
    swp3        up            down                9216   swp
    swp4        up            down                9216   swp
    swp5        up            down                9216   swp
    swp6        up            up           100G   9216   swp       952347XXXXXX                        e1b
    swp7        up            down                9216   swp
    swp8        up            down                9216   swp
    swp9        up            down                9216   swp
    swp10       up            down                9216   swp
    swp11       up            down                9216   swp
    swp12       up            down                9216   swp
    swp13       up            down                9216   swp
    swp14       up            down                9216   swp
    swp15       up            down                9216   swp
    swp16       up            down                9216   swp
  2. Verify information for the transceiver in the interface:

    Cumulus Linux 5.4
    nv show interface --view=pluggables
    Show example
    cumulus@cumulus:mgmt:~$ nv show interface --view=pluggables
    Interface  Identifier     Vendor Name  Vendor PN        Vendor SN       Vendor Rev
    ---------  -------------  -----------  ---------------  --------------  ----------
    swp3       0x11 (QSFP28)  Amphenol     112-00574        APF20379253516  B0
    swp4       0x11 (QSFP28)  AVAGO        332-00440        AF1815GU05Z     A0
    swp15      0x11 (QSFP28)  Amphenol     112-00573        APF21109348001  B0
    swp16      0x11 (QSFP28)  Amphenol     112-00573        APF21109347895  B0
    Cumulus Linux 5.11 and later
    nv show platform transceiver
    Show example
    cumulus@cumulus:mgmt:~$ nv show platform transceiver
    Transceiver  Identifier  Vendor name   Vendor PN  Vendor SN       Vendor revision
    -----------  ----------  ------------  ---------  --------------  ---------------
    swp1         QSFP28      FINISAR CORP  332-00440  XA7C325         A1
    swp2         QSFP28      FINISAR CORP  332-00440  XA7C33W         A1
    swp5         QSFP28      FINISAR CORP  332-00440  XA7C1RV         A1
    swp6         QSFP28      FINISAR CORP  332-00440  X4QANNS         A1
  3. Verify that the nodes each have a connection to each switch:

    Cumulus Linux 5.4
    nv show interface --view=lldp
    Show example
    cumulus@cumulus:mgmt:~$ net show lldp
    
    LocalPort  Speed  Mode        RemoteHost              RemotePort
    ---------  -----  ----------  ----------------------  -----------
    swp3       100G   Trunk/L2    sw1                     e3a
    swp4       100G   Trunk/L2    sw2                     e3b
    swp15      100G   Trunk/L2    sw13                    swp15
    swp16      100G   Trunk/L2    sw14                    swp16
    Cumulus Linux 5.11 and later
    nv show interface lldp
    Show example
    cumulus@cumulus:mgmt:~$ nv show interface lldp
    Interface  Speed  Type  Remote Host                         Remote Port
    ---------  -----  ----  ----------------------------------  -----------
    eth0       100M   eth   mgmt-switch                          Eth105/1/15
    swp1       100G   swp   cluster1-01                          e3a
    swp2       100G   swp   cluster1-02                          e3a
    swp5       100G   swp   952347XXXXXX                         e1a
    swp6       100G   swp   952347XXXXXX                         e1a
  4. Verify the health of storage ports and switch connectivity.

    1. Verify the status of node storage ports:

      storage port show -port-type ENET
      Show example
      cluster1::*> storage port show -port-type ENET
      
      
                                            Speed
      Node               Port Type  Mode    (Gb/s) State    Status
      ------------------ ---- ----- ------- ------ -------- ---------------
      node1
                         e3a  ENET  storage    100 enabled  online
                         e3b  ENET  storage    100 enabled  online
      node2
                         e3a  ENET  storage    100 enabled  online
                         e3b  ENET  storage    100 enabled  online
      4 entries were displayed.
  5. Verify the shelf ports in the storage system:

    storage shelf port show -fields remote-device, remote-port
    Show example
    cluster1::*> storage shelf port show -fields remote-device, remote-port
    shelf   id    remote-port     remote-device
    -----   --    -----------     -------------
    3.20    0     swp3            sw1
    3.20    1     swp3            sw2
    3.20    2     swp4            sw1
    3.20    3     swp4            sw2
    3.30    0     swp5            sw1
    3.30    1     swp5            sw2
    3.30    2     swp6            sw1
    3.30    3     swp6            sw2
    cluster1::*>
  6. Verify that the switches are available:

    network device-discovery show -protocol lldp
    Show example
    cluster1::*> network device-discovery show -protocol lldp
    Node/       Local  Discovered
    Protocol    Port   Device (LLDP: ChassisID)  Interface         Platform
    ----------- ------ ------------------------- ----------------  ----------------
    node1
                e0M    mgmt-switch
                                                 Eth123/1/41       -
                e1a    data-switch
                                                 Ethernet1/25      -
                e1b    data-switch
                                                 Ethernet1/26      -
    
                sw1    storage-switch
                                                 swp1              -
                sw2    storage-switch
                                                 swp1              -
                cs1    cluster1-01
                                                 swp3              -
                cs2    cluster2-02
                                                 swp3              -
    node2
                e0M    mgmt-switch
                                                 Eth123/1/42       -
                e1a    data-switch
                                                 Ethernet1/27      -
                e1b    data-switch
                                                 Ethernet1/28      -
                sw1    storage-switch
                                                 swp2              -
                sw2    storage-switch
                                                 swp2              -
                cs1    cluster1-01
                                                 swp4              -
                cs2    cluster2-02
                                                 swp4              -
    18 entries were displayed.
  7. Verify that the switches are being monitored.

    system switch ethernet show -is-monitoring-enabled-operational true
    Show example
    cluster1::*> system switch ethernet show -is-monitoring-enabled-operational true
    Switch                      Type               Address          Model
    --------------------------- ------------------ ---------------- -----
    sw1                         storage-network    192.0.2.71       MSN2100-CB2RC
         Serial Number: MNXXXXXXGD
          Is Monitored: true
                Reason: None
      Software Version: Cumulus Linux version 5.11.1 running on Mellanox
                        Technologies Ltd. MSN2100
        Version Source: LLDP
    
    sw2                         storage-network    192.0.2.72       MSN2100-CB2RC
         Serial Number: MNCXXXXXXGS
          Is Monitored: true
                Reason: None
      Software Version: Cumulus Linux version 5.11.1 running on Mellanox
                        Technologies Ltd. MSN2100
        Version Source: LLDP
  8. Repeat these verification steps on the partner (or other) switch.

What's next?

After you've installed or upgraded the RCF, you can install the CSHM file.