Skip to main content
Install and maintain

Upgrade your Reference Configuration File (RCF)

Contributors netapp-yvonneo
PDFs

You upgrade your RCF version when you have an existing version of the RCF file installed on your operational switches.

Before you begin

Make sure you have the following:

  • A current backup of the switch configuration.

  • A fully functioning cluster (no errors in the logs or similar issues).

  • The current RCF.

  • If you are updating your RCF version, you need a boot configuration in the RCF that reflects the desired boot images.

    If you need to change the boot configuration to reflect the current boot images, you must do so before reapplying the RCF so that the correct version is instantiated on future reboots.

Note No operational inter-switch link (ISL) is needed during this procedure. This is by design because RCF version changes can affect ISL connectivity temporarily. To ensure non-disruptive cluster operations, the following procedure migrates all of the cluster LIFs to the operational partner switch while performing the steps on the target switch.
Caution Before installing a new switch software version and RCFs, you must erase the switch settings and perform basic configuration. You must be connected to the switch using the serial console or have preserved basic configuration information prior to erasing the switch settings.

Step 1: Prepare for the upgrade

  1. Display the cluster ports on each node that are connected to the cluster switches:

    network device-discovery show

    Show example 
    cluster1::*> network device-discovery show
Node/       Local  Discovered
Protocol    Port   Device (LLDP: ChassisID)  Interface         Platform
----------- ------ ------------------------- ----------------  --------
cluster1-01/cdp
            e0a    cs1                       Ethernet1/7       N3K-C3232C
            e0d    cs2                       Ethernet1/7       N3K-C3232C
cluster1-02/cdp
            e0a    cs1                       Ethernet1/8       N3K-C3232C
            e0d    cs2                       Ethernet1/8       N3K-C3232C
cluster1-03/cdp
            e0a    cs1                       Ethernet1/1/1     N3K-C3232C
            e0b    cs2                       Ethernet1/1/1     N3K-C3232C
cluster1-04/cdp
            e0a    cs1                       Ethernet1/1/2     N3K-C3232C
            e0b    cs2                       Ethernet1/1/2     N3K-C3232C
cluster1::*>

  2. Check the administrative and operational status of each cluster port.

    1. Verify that all the cluster ports are up with a healthy status:

      network port show –role cluster

      Show example 
      cluster1::*> network port show -role cluster
Node: cluster1-01
                                                                       Ignore
                                                  Speed(Mbps) Health   Health
Port      IPspace      Broadcast Domain Link MTU  Admin/Oper  Status   Status
--------- ------------ ---------------- ---- ---- ----------- -------- ------
e0a       Cluster      Cluster          up   9000  auto/100000 healthy false
e0d       Cluster      Cluster          up   9000  auto/100000 healthy false
Node: cluster1-02
                                                                       Ignore
                                                  Speed(Mbps) Health   Health
Port      IPspace      Broadcast Domain Link MTU  Admin/Oper  Status   Status
--------- ------------ ---------------- ---- ---- ----------- -------- ------
e0a       Cluster      Cluster          up   9000  auto/100000 healthy false
e0d       Cluster      Cluster          up   9000  auto/100000 healthy false
8 entries were displayed.
Node: cluster1-03
   Ignore
                                                  Speed(Mbps) Health   Health
Port      IPspace      Broadcast Domain Link MTU  Admin/Oper  Status   Status
--------- ------------ ---------------- ---- ---- ----------- -------- ------
e0a       Cluster      Cluster          up   9000  auto/10000 healthy  false
e0b       Cluster      Cluster          up   9000  auto/10000 healthy  false
Node: cluster1-04
                                                                       Ignore
                                                  Speed(Mbps) Health   Health
Port      IPspace      Broadcast Domain Link MTU  Admin/Oper  Status   Status
--------- ------------ ---------------- ---- ---- ----------- -------- ------
e0a       Cluster      Cluster          up   9000  auto/10000 healthy  false
e0b       Cluster      Cluster          up   9000  auto/10000 healthy  false
cluster1::*>

    2. Verify that all the cluster interfaces (LIFs) are on the home port:

      network interface show -role cluster

      Show example 
      cluster1::*> network interface show -role cluster
            Logical            Status     Network           Current      Current Is
Vserver     Interface          Admin/Oper Address/Mask      Node         Port    Home
----------- ------------------ ---------- ----------------- ------------ ------- ----
Cluster
            cluster1-01_clus1  up/up     169.254.3.4/23     cluster1-01  e0a     true
            cluster1-01_clus2  up/up     169.254.3.5/23     cluster1-01  e0d     true
            cluster1-02_clus1  up/up     169.254.3.8/23     cluster1-02  e0a     true
            cluster1-02_clus2  up/up     169.254.3.9/23     cluster1-02  e0d     true
            cluster1-03_clus1  up/up     169.254.1.3/23     cluster1-03  e0a     true
            cluster1-03_clus2  up/up     169.254.1.1/23     cluster1-03  e0b     true
            cluster1-04_clus1  up/up     169.254.1.6/23     cluster1-04  e0a     true
            cluster1-04_clus2  up/up     169.254.1.7/23     cluster1-04  e0b     true
8 entries were displayed.
cluster1::*>

    3. Verify that the cluster displays information for both cluster switches:

      system cluster-switch show -is-monitoring-enabled-operational true

      Show example 
      cluster1::*> system cluster-switch show -is-monitoring-enabled-operational true
Switch                      Type               Address          Model
--------------------------- ------------------ ---------------- ---------------
cs1                         cluster-network    10.233.205.92    NX3232C
     Serial Number: FOXXXXXXXGS
      Is Monitored: true
            Reason: None
  Software Version: Cisco Nexus Operating System (NX-OS) Software, Version
                    9.3(4)
    Version Source: CDP
cs2                         cluster-network    10.233.205.93    NX3232C
     Serial Number: FOXXXXXXXGD
      Is Monitored: true
            Reason: None
  Software Version: Cisco Nexus Operating System (NX-OS) Software, Version
                    9.3(4)
    Version Source: CDP
2 entries were displayed.

  3. Disable auto-revert on the cluster LIFs.

    cluster1::*> network interface modify -vserver Cluster -lif * -auto-revert false

Step 2: Configure ports

  1. On cluster switch cs2, shut down the ports connected to the cluster ports of the nodes.

    cs2> enable
cs2# configure
cs2(config)# interface eth1/1/1-2,eth1/7-8
cs2(config-if-range)# shutdown
cs2(config-if-range)# exit
cs2# exit
    Caution Make sure to shutdown all connected cluster ports to avoid any network connection issues. See the Knowledge Base article Node out of quorum when migrating cluster LIF during switch OS upgrade for further details.

  2. Verify that the cluster ports have failed over to the ports hosted on cluster switch cs1. This might take a few seconds.

    network interface show -role cluster

    Show example 
    cluster1::*> network interface show -role cluster
            Logical           Status     Network            Current       Current Is
Vserver     Interface         Admin/Oper Address/Mask       Node          Port    Home
----------- ----------------- ---------- ------------------ ------------- ------- ----
Cluster
            cluster1-01_clus1 up/up      169.254.3.4/23     cluster1-01   e0a     true
            cluster1-01_clus2 up/up      169.254.3.5/23     cluster1-01   e0a     false
            cluster1-02_clus1 up/up      169.254.3.8/23     cluster1-02   e0a     true
            cluster1-02_clus2 up/up      169.254.3.9/23     cluster1-02   e0a     false
            cluster1-03_clus1 up/up      169.254.1.3/23     cluster1-03   e0a     true
            cluster1-03_clus2 up/up      169.254.1.1/23     cluster1-03   e0a     false
            cluster1-04_clus1 up/up      169.254.1.6/23     cluster1-04   e0a     true
            cluster1-04_clus2 up/up      169.254.1.7/23     cluster1-04   e0a     false
8 entries were displayed.
cluster1::*>

  3. Verify that the cluster is healthy:

    cluster show

    Show example 
    cluster1::*> cluster show
Node                 Health  Eligibility   Epsilon
-------------------- ------- ------------  -------
cluster1-01          true    true          false
cluster1-02          true    true          false
cluster1-03          true    true          true
cluster1-04          true    true          false
4 entries were displayed.
cluster1::*>

  4. If you have not already done so, save a copy of the current switch configuration by copying the output of the following command to a text file:

    show running-config

  5. Record any custom additions between the current running-config and the RCF file in use (such as an SNMP configuration for your organization).

  6. Save basic configuration details to the write_erase.cfg file on the bootflash.

    Note

    Make sure to configure the following: * Username and password * Management IP address * Default gateway * Switch name

    cs2# show run | section "switchname" > bootflash:write_erase.cfg

    cs2# show run | section "hostname" >> bootflash:write_erase.cfg

    cs2# show run | i "username admin password" >> bootflash:write_erase.cfg

    cs2# show run | section "vrf context management" >> bootflash:write_erase.cfg

    cs2# show run | section "interface mgmt0" >> bootflash:write_erase.cfg

  7. When upgrading to RCF version 1.12 and later, run the following commands:

    cs2# echo "hardware access-list tcam region racl-lite 512" >> bootflash:write_erase.cfg

    cs2# echo "hardware access-list tcam region qos 256" >> bootflash:write_erase.cfg

    See the Knowledge Base article How to clear configuration on a Cisco interconnect switch while retaining remote connectivity for further details.

  8. Verify that the write_erase.cfg file is populated as expected:

    show file bootflash:write_erase.cfg

  9. Issue the write erase command to erase the current saved configuration:

    cs2# write erase

    Warning: This command will erase the startup-configuration.

    Do you wish to proceed anyway? (y/n) [n] y

  10. Copy the previously saved basic configuration into the startup configuration.

    cs2# copy bootflash:write_erase.cfg startup-config

  11. Reboot the switch cs2:

    cs2# reload

    This command will reboot the system. (y/n)? [n] y

  12. After the management IP address is reachable again, log in to the switch through SSH.

    You may need to update host file entries related to the SSH keys.

  13. Copy the RCF to the bootflash of switch cs2 using one of the following transfer protocols: FTP, TFTP, SFTP, or SCP. For more information on Cisco commands, see the appropriate guide in the Cisco Nexus 3000 Series NX-OS Command Reference guides.

    Show example

    This example shows TFTP being used to copy an RCF to the bootflash on switch cs2:

    cs2# copy tftp: bootflash: vrf management
Enter source filename: Nexus_3232C_RCF_v1.6-Cluster-HA-Breakout.txt
Enter hostname for the tftp server: 172.22.201.50
Trying to connect to tftp server......Connection to Server Established.
TFTP get operation was successful
Copy complete, now saving to disk (please wait)...

  14. Apply the RCF previously downloaded to the bootflash.

    For more information on Cisco commands, see the appropriate guide in the Cisco Nexus 3000 Series NX-OS Command Reference guides.

    Show example

    This example shows the RCF file Nexus_3232C_RCF_v1.6-Cluster-HA-Breakout.txt being installed on switch cs2:

    cs2# copy Nexus_3232C_RCF_v1.6-Cluster-HA-Breakout.txt running-config echo-commands
    Caution

    Make sure to read thoroughly the Installation notes, Important Notes, and banner sections of your RCF. You must read and follow these instructions to ensure the proper configuration and operation of the switch.

  15. Verify that the RCF file is the correct newer version:

    show running-config

    When you check the output to verify you have the correct RCF, make sure that the following information is correct:

    • The RCF banner

    • The node and port settings

    • Customizations

      The output varies according to your site configuration. Check the port settings and refer to the release notes for any changes specific to the RCF that you have installed.

  16. Reapply any previous customizations to the switch configuration. Refer to Review cabling and configuration considerations for details of any further changes required.

  17. After you verify the RCF versions and switch settings are correct, copy the running-config file to the startup-config file.

    For more information on Cisco commands, see the appropriate guide in the Cisco Nexus 3000 Series NX-OS Command Reference guides.

    cs2# copy running-config startup-config [########################################] 100% Copy complete

  18. Reboot switch cs2. You can ignore the "cluster ports down" events reported on the nodes while the switch reboots.

    cs2# reload
This command will reboot the system. (y/n)?  [n] y

  19. Verify the health of cluster ports on the cluster.

    1. Verify that e0d ports are up and healthy across all nodes in the cluster:

      network port show -role cluster

      Show example 
      cluster1::*> network port show -role cluster
Node: cluster1-01
                                                                       Ignore
                                                  Speed(Mbps) Health   Health
Port      IPspace      Broadcast Domain Link MTU  Admin/Oper  Status   Status
--------- ------------ ---------------- ---- ---- ----------- -------- ------
e0a       Cluster      Cluster          up   9000  auto/10000 healthy  false
e0b       Cluster      Cluster          up   9000  auto/10000 healthy  false
Node: cluster1-02
                                                                       Ignore
                                                  Speed(Mbps) Health   Health
Port      IPspace      Broadcast Domain Link MTU  Admin/Oper  Status   Status
--------- ------------ ---------------- ---- ---- ----------- -------- ------
e0a       Cluster      Cluster          up   9000  auto/10000 healthy  false
e0b       Cluster      Cluster          up   9000  auto/10000 healthy  false
Node: cluster1-03
                                                                       Ignore
                                                  Speed(Mbps) Health   Health
Port      IPspace      Broadcast Domain Link MTU  Admin/Oper  Status   Status
--------- ------------ ---------------- ---- ---- ----------- -------- ------
e0a       Cluster      Cluster          up   9000  auto/100000 healthy false
e0d       Cluster      Cluster          up   9000  auto/100000 healthy false
Node: cluster1-04
                                                                       Ignore
                                                  Speed(Mbps) Health   Health
Port      IPspace      Broadcast Domain Link MTU  Admin/Oper  Status   Status
--------- ------------ ---------------- ---- ---- ----------- -------- ------
e0a       Cluster      Cluster          up   9000  auto/100000 healthy false
e0d       Cluster      Cluster          up   9000  auto/100000 healthy false
8 entries were displayed.

    2. Verify the switch health from the cluster (this might not show switch cs2, since LIFs are not homed on e0d).

      Show example 
      cluster1::*> network device-discovery show -protocol cdp
Node/       Local  Discovered
Protocol    Port   Device (LLDP: ChassisID)  Interface         Platform
----------- ------ ------------------------- ----------------- --------
cluster1-01/cdp
            e0a    cs1                       Ethernet1/7       N3K-C3232C
            e0d    cs2                       Ethernet1/7       N3K-C3232C
cluster01-2/cdp
            e0a    cs1                       Ethernet1/8       N3K-C3232C
            e0d    cs2                       Ethernet1/8       N3K-C3232C
cluster01-3/cdp
            e0a    cs1                       Ethernet1/1/1     N3K-C3232C
            e0b    cs2                       Ethernet1/1/1     N3K-C3232C
cluster1-04/cdp
            e0a    cs1                       Ethernet1/1/2     N3K-C3232C
            e0b    cs2                       Ethernet1/1/2     N3K-C3232C
cluster1::*> system cluster-switch show -is-monitoring-enabled-operational true
Switch                      Type               Address          Model
--------------------------- ------------------ ---------------- -----
cs1                         cluster-network    10.233.205.90    N3K-C3232C
     Serial Number: FOXXXXXXXGD
      Is Monitored: true
            Reason: None
  Software Version: Cisco Nexus Operating System (NX-OS) Software, Version
                    9.3(4)
    Version Source: CDP
cs2                         cluster-network    10.233.205.91    N3K-C3232C
     Serial Number: FOXXXXXXXGS
      Is Monitored: true
            Reason: None
  Software Version: Cisco Nexus Operating System (NX-OS) Software, Version
                    9.3(4)
    Version Source: CDP
2 entries were displayed.
      Note

      You might observe the following output on the cs1 switch console depending on the RCF version previously loaded on the switch 2020 Nov 17 16:07:18 cs1 %$ VDC-1 %$ %STP-2-UNBLOCK_CONSIST_PORT: Unblocking port port-channel1 on VLAN0092. Port consistency restored. 2020 Nov 17 16:07:23 cs1 %$ VDC-1 %$ %STP-2-BLOCK_PVID_PEER: Blocking port-channel1 on VLAN0001. Inconsistent peer vlan. 2020 Nov 17 16:07:23 cs1 %$ VDC-1 %$ %STP-2-BLOCK_PVID_LOCAL: Blocking port-channel1 on VLAN0092. Inconsistent local vlan.
    Note It can take up to 5 minutes for the cluster nodes to report as healthy.

  20. On cluster switch cs1, shut down the ports connected to the cluster ports of the nodes.

    Show example

    The following example uses the interface example output from step 1:

    cs1(config)# interface eth1/1/1-2,eth1/7-8
cs1(config-if-range)# shutdown

  21. Verify that the cluster LIFs have migrated to the ports hosted on switch cs2. This might take a few seconds.

    network interface show -role cluster

    Show example 
    cluster1::*> network interface show -role cluster
            Logical            Status     Network            Current             Current Is
Vserver     Interface          Admin/Oper Address/Mask       Node                Port    Home
----------- ------------------ ---------- ------------------ ------------------- ------- ----
Cluster
            cluster1-01_clus1  up/up      169.254.3.4/23     cluster1-01         e0d     false
            cluster1-01_clus2  up/up      169.254.3.5/23     cluster1-01         e0d     true
            cluster1-02_clus1  up/up      169.254.3.8/23     cluster1-02         e0d     false
            cluster1-02_clus2  up/up      169.254.3.9/23     cluster1-02         e0d     true
            cluster1-03_clus1  up/up      169.254.1.3/23     cluster1-03         e0b     false
            cluster1-03_clus2  up/up      169.254.1.1/23     cluster1-03         e0b     true
            cluster1-04_clus1  up/up      169.254.1.6/23     cluster1-04         e0b     false
            cluster1-04_clus2  up/up      169.254.1.7/23     cluster1-04         e0b     true
8 entries were displayed.
cluster1::*>

  22. Verify that the cluster is healthy:

    cluster show

    Show example 
    cluster1::*> cluster show
Node                 Health   Eligibility   Epsilon
-------------------- -------- ------------- -------
cluster1-01          true     true          false
cluster1-02          true     true          false
cluster1-03          true     true          true
cluster1-04          true     true          false
4 entries were displayed.
cluster1::*>

  23. Repeat Steps 4 to 19 on switch cs1.

  24. Enable auto-revert on the cluster LIFs.

    cluster1::*> network interface modify -vserver Cluster -lif * -auto-revert true

Step 3: Verify the cluster network configuration and cluster health

  1. Verify that the switch ports connected to the cluster ports are up.

    show interface brief | grep up

    Show example 
    cs1# show interface brief | grep up
.
.
Eth1/1/1      1       eth  access up      none                    10G(D) --
Eth1/1/2      1       eth  access up      none                    10G(D) --
Eth1/7        1       eth  trunk  up      none                   100G(D) --
Eth1/8        1       eth  trunk  up      none                   100G(D) --
.
.

  2. Verify that the ISL between cs1 and cs2 is functional:

    show port-channel summary

    Show example 
    cs1# show port-channel summary
Flags:  D - Down        P - Up in port-channel (members)
        I - Individual  H - Hot-standby (LACP only)
        s - Suspended   r - Module-removed
        b - BFD Session Wait
        S - Switched    R - Routed
        U - Up (port-channel)
        p - Up in delay-lacp mode (member)
        M - Not in use. Min-links not met
--------------------------------------------------------------------------------
Group Port-       Type     Protocol  Member Ports
      Channel
--------------------------------------------------------------------------------
1     Po1(SU)     Eth      LACP      Eth1/31(P)   Eth1/32(P)
cs1#

  3. Verify that the cluster LIFs have reverted to their home port:

    network interface show -role cluster

    Show example 
    cluster1::*> network interface show -role cluster
            Logical            Status     Network            Current             Current Is
Vserver     Interface          Admin/Oper Address/Mask       Node                Port    Home
----------- ------------------ ---------- ------------------ ------------------- ------- ----
Cluster
            cluster1-01_clus1  up/up      169.254.3.4/23     cluster1-01         e0d     true
            cluster1-01_clus2  up/up      169.254.3.5/23     cluster1-01         e0d     true
            cluster1-02_clus1  up/up      169.254.3.8/23     cluster1-02         e0d     true
            cluster1-02_clus2  up/up      169.254.3.9/23     cluster1-02         e0d     true
            cluster1-03_clus1  up/up      169.254.1.3/23     cluster1-03         e0b     true
            cluster1-03_clus2  up/up      169.254.1.1/23     cluster1-03         e0b     true
            cluster1-04_clus1  up/up      169.254.1.6/23     cluster1-04         e0b     true
            cluster1-04_clus2  up/up      169.254.1.7/23     cluster1-04         e0b     true
8 entries were displayed.
cluster1::*>

    If any cluster LIFS have not returned to their home ports, revert them manually: network interface revert -vserver vserver_name -lif lif_name

  4. Verify that the cluster is healthy:

    cluster show

    Show example 
    cluster1::*> cluster show
Node                 Health  Eligibility   Epsilon
-------------------- ------- ------------- -------
cluster1-01          true    true          false
cluster1-02          true    true          false
cluster1-03          true    true          true
cluster1-04          true    true          false
4 entries were displayed.
cluster1::*>

  5. Verify the connectivity of the remote cluster interfaces:

ONTAP 9.9.1 and later

You can use the network interface check cluster-connectivity command to start an accessibility check for cluster connectivity and then display the details: network interface check cluster-connectivity start and network interface check cluster-connectivity show

cluster1::*> network interface check cluster-connectivity start

NOTE: Wait for a number of seconds before running the show command to display the details.

cluster1::*> network interface check cluster-connectivity show
                                  Source              Destination         Packet
Node   Date                       LIF                 LIF                 Loss
------ -------------------------- ------------------- ------------------- -----------
cluster1-01
       3/5/2022 19:21:18 -06:00   cluster1-01_clus2   cluster1-02_clus1   none
       3/5/2022 19:21:20 -06:00   cluster1-01_clus2   cluster1-02_clus2   none
.
.
cluster1-02
       3/5/2022 19:21:18 -06:00   cluster1-02_clus2   cluster1-01_clus1   none
       3/5/2022 19:21:20 -06:00   cluster1-02_clus2   cluster1-01_clus2   none
.
.
cluster1-03
.
.
.
.
cluster1-04
.
.
.
.
All ONTAP releases

For all ONTAP releases, you can also use the cluster ping-cluster -node <name> command to check the connectivity: cluster ping-cluster -node <name>

cluster1::*> cluster ping-cluster -node local
Host is cluster1-03
Getting addresses from network interface table...
Cluster cluster1-03_clus1 169.254.1.3 cluster1-03 e0a
Cluster cluster1-03_clus2 169.254.1.1 cluster1-03 e0b
Cluster cluster1-04_clus1 169.254.1.6 cluster1-04 e0a
Cluster cluster1-04_clus2 169.254.1.7 cluster1-04 e0b
Cluster cluster1-01_clus1 169.254.3.4 cluster1-01 e0a
Cluster cluster1-01_clus2 169.254.3.5 cluster1-01 e0d
Cluster cluster1-02_clus1 169.254.3.8 cluster1-02 e0a
Cluster cluster1-02_clus2 169.254.3.9 cluster1-02 e0d
Local = 169.254.1.3 169.254.1.1
Remote = 169.254.1.6 169.254.1.7 169.254.3.4 169.254.3.5 169.254.3.8 169.254.3.9
Cluster Vserver Id = 4294967293
Ping status:
............
Basic connectivity succeeds on 12 path(s)
Basic connectivity fails on 0 path(s)
................................................
Detected 9000 byte MTU on 12 path(s):
    Local 169.254.1.3 to Remote 169.254.1.6
    Local 169.254.1.3 to Remote 169.254.1.7
    Local 169.254.1.3 to Remote 169.254.3.4
    Local 169.254.1.3 to Remote 169.254.3.5
    Local 169.254.1.3 to Remote 169.254.3.8
    Local 169.254.1.3 to Remote 169.254.3.9
    Local 169.254.1.1 to Remote 169.254.1.6
    Local 169.254.1.1 to Remote 169.254.1.7
    Local 169.254.1.1 to Remote 169.254.3.4
    Local 169.254.1.1 to Remote 169.254.3.5
    Local 169.254.1.1 to Remote 169.254.3.8
    Local 169.254.1.1 to Remote 169.254.3.9
Larger than PMTU communication succeeds on 12 path(s)
RPC status:
6 paths up, 0 paths down (tcp check)
6 paths up, 0 paths down (udp check)
What's next?

After you've upgraded your RCF, you verify the SSH configuration.