Skip to main content
Cluster and storage switches

Install NX-OS software and Reference Configuration Files (RCFs)

Contributors netapp-martyh netapp-karunesh

Install NX-OS software and RCF on a Cisco Nexus 9336C- FX2 cluster switch

The Cisco NX-OS software and reference configuration file (RCF) must be installed on Cisco Nexus 9336C-FX2 cluster switches.

Before you begin

The following conditions must exist before you install the NX-OS software and RCF on the cluster switch:

  • The cluster must be fully functioning (there should be no errors in the logs or similar issues).

  • You must have checked or set your desired boot configuration in the RCF to reflect the desired boot images if you are installing only NX-OS and keeping your current RCF version.

  • If you need to change the boot configuration to reflect the current boot images, you must do so before reapplying the RCF so that the correct version is instantiated on future reboots.

  • You must have a console connection to the switch, required when installing the RCF.

  • You must have consulted the switch compatibility table on the Cisco Ethernet switch page for the supported ONTAP, NX-OS, and RCF versions.
    See Cisco Ethernet Switches for more information.

  • There can be command dependencies between the command syntax in the RCF and that found in versions of NX-OS.

  • You must have referred to the appropriate software and upgrade guides available on the Cisco web site for complete documentation on the Cisco switch upgrade and downgrade procedures on Cisco Nexus 9000 Series Switches.
    See Cisco Nexus 9000 Series Switches for more information.

  • You must have the current RCF.

Initial setup

The examples in this procedure use two nodes. These nodes use two 100GbE cluster interconnect ports e3a and e3b, as per the A400 controller.
See the Hardware Universe to verify the correct cluster ports on your platforms.

Note The command outputs might vary depending on different releases of ONTAP.

The examples in this procedure use the following switch and node nomenclature:

  • The names of the two Cisco switches are cs1 and cs2.

  • The node names are cluster1-01 and cluster1-02.

  • The cluster LIF names are cluster1-01_clus1 and cluster1-01_clus2 for cluster1-01 and cluster1-02_clus1 and cluster1-02_clus2 for cluster1-02.

  • The cluster1::*> prompt indicates the name of the cluster.

Note The procedure requires the use of both ONTAP commands and Cisco Nexus 9000 Series Switches commands; ONTAP commands are used unless otherwise indicated.
Steps

  1. If AutoSupport is enabled on this cluster, suppress automatic case creation by invoking an AutoSupport message: system node autosupport invoke -node * -type all -message MAINT=x h

    Where x is the duration of the maintenance window in hours.

  1. Change the privilege level to advanced, entering y when prompted to continue:
    set -privilege advanced

    The advanced prompt (*>) appears.

  2. Display how many cluster interconnect interfaces are configured in each node for each cluster interconnect switch:
    network device-discovery show -protocol cdp

cluster1:: network device-discovery show -protocol cdp
Node/       Local  Discovered
Protocol    Port   Device (LLDP: ChassisID)  Interface         Platform
----------- ------ ------------------------- ----------------- --------
cluster1-02/cdp
            e3a    cs1                       Eth1/2            N9K-C9336C
            e3b    cs2                       Eth1/2            N9K-C9336C
cluster1-01/cdp
            e3a    cs1                       Eth1/1            N9K-C9336C
            e3b    cs2                       Eth1/1            N9K-C9336C
4 entries were displayed.

  1. Check the administrative or operational status of each cluster interface:

    1. Display the network port attributes:
      network port show –ipspace Cluster

cluster1::*> network port show -ipspace Cluster
Node: cluster1-02
                                                  Speed(Mbps)  Health
Port      IPspace      Broadcast Domain Link MTU  Admin/Oper   Status
--------- ------------ ---------------- ---- ---- ------------ ------
e3a       Cluster      Cluster          up   9000  auto/100000 healthy
e3b       Cluster      Cluster          up   9000  auto/100000 healthy

Node: cluster1-01
                                                  Speed(Mbps)  Health
Port      IPspace      Broadcast Domain Link MTU  Admin/Oper   Status
--------- ------------ ---------------- ---- ---- ------------ ------
e3a       Cluster      Cluster          up   9000  auto/100000 healthy
e3b       Cluster      Cluster          up   9000  auto/100000 healthy
4 entries were displayed.

  1. Display information about the LIFs:
    network interface show - vserver Cluster

cluster1::*> network interface show -vserver Cluster
        Logical            Status     Network            Current    Current Is
Vserver Interface          Admin/Oper Address/Mask       Node        Port   Home
------- ------------------ ---------- ------------------ ----------- ------ ----
Cluster
        cluster1-01_clus1  up/up      169.254.209.69/16  cluster1-01  e3a   true
        cluster1-01_clus2  up/up      169.254.49.125/16  cluster1-01  e3b   true
        cluster1-02_clus1  up/up      169.254.47.194/16  cluster1-02  e3a   true
        cluster1-02_clus2  up/up      169.254.19.183/16  cluster1-02  e3b   true
4 entries were displayed.

  1. Ping the remote cluster LIFs:
    cluster ping-cluster -node node-name

cluster1::*> cluster ping-cluster -node cluster1-02
Host is cluster1-02
Getting addresses from network interface table...
Cluster cluster1-01_clus1 169.254.209.69 cluster1-01     e3a
Cluster cluster1-01_clus2 169.254.49.125 cluster1-01     e3b
Cluster cluster1-02_clus1 169.254.47.194 cluster1-02     e3a
Cluster cluster1-02_clus2 169.254.19.183 cluster1-02     e3b
Local = 169.254.47.194 169.254.19.183
Remote = 169.254.209.69 169.254.49.125
Cluster Vserver Id = 4294967293
Ping status:

Basic connectivity succeeds on 4 path(s)
Basic connectivity fails on 0 path(s)

Detected 9000 byte MTU on 4 path(s):
    Local 169.254.19.183 to Remote 169.254.209.69
    Local 169.254.19.183 to Remote 169.254.49.125
    Local 169.254.47.194 to Remote 169.254.209.69
    Local 169.254.47.194 to Remote 169.254.49.125
Larger than PMTU communication succeeds on 4 path(s)
RPC status:
2 paths up, 0 paths down (tcp check)
2 paths up, 0 paths down (udp check)

  1. Verify that the auto-revert command is enabled on all cluster LIFs:
    network interface show - vserver Cluster -fields auto-revert

cluster1::*> network interface show -vserver Cluster -fields auto-revert
          Logical
Vserver   Interface            Auto-revert
--------- ––––––-------------- ------------
Cluster
          cluster1-01_clus1    true
          cluster1-01_clus2    true
          cluster1-02_clus1    true
          cluster1-02_clus2    true
4 entries were displayed.

  1. Enable the Ethernet switch health monitor log collection feature for collecting switch-related log files, using the following commands:

    • system switch ethernet log setup-password

    • system switch ethernet log enable-collection

cluster1::*> system switch ethernet log setup password
Enter the switch name: <return>
The switch name entered is not recognized.
Choose from the following list:
cs1
cs2
cluster1::*> system switch ethernet log setup-password
Enter the switch name: cs1
RSA key fingerprint is e5:8b:c6:dc:e2:18:18:09:36:63:d9:63:dd:03:d9:cc
Do you want to continue? {y|n}::[n] y
Enter the password: <enter switch password>
Enter the password again: <enter switch password>
cluster1::*> system switch ethernet log setup-password
Enter the switch name: cs2
RSA key fingerprint is 57:49:86:a1:b9:80:6a:61:9a:86:8e:3c:e3:b7:1f:b1
Do you want to continue? {y|n}:: [n] y
Enter the password: <enter switch password>
Enter the password again: <enter switch password>
cluster1::*> system  switch ethernet log enable-collection
Do you want to enable cluster log collection for all nodes in the cluster? {y|n}: [n] y
Enabling cluster switch log collection.
cluster1::*>
Note If any of these commands return an error, contact NetApp support.

Install the NX-OS software on a Cisco Nexus 9336C- FX2 cluster switch

You can use this procedure to install the NX-OS software on the Cisco Nexus 9336C-FX2 cluster switch.

Steps

  1. Connect the cluster switch to the management network.

  2. Use the ping command to verify connectivity to the server hosting the NX-OS software and the RCF.

    This example verifies that the switch can reach the server at IP address 172.19.2.1:

cs2# ping 172.19.2.1
Pinging 172.19.2.1 with 0 bytes of data:
Reply From 172.19.2.1: icmp_seq = 0. time= 5910 usec.

  1. Copy the NX-OS software and EPLD images to the Nexus 9336C-FX2 switch.

cs2# copy sftp: bootflash: vrf management
Enter source filename: /code/nxos.9.3.5.bin
Enter hostname for the sftp server: 172.19.2.1
Enter username: user1
Outbound-ReKey for 172.19.2.1:22
Inbound-ReKey for 172.19.2.1:22
user1@172.19.2.1's password:
sftp> progress
Progress meter enabled
sftp> get   /code/nxos.9.3.5.bin  /bootflash/nxos.9.3.5.bin
/code/nxos.9.3.5.bin  100% 1261MB   9.3MB/s   02:15
sftp> exit
Copy complete, now saving to disk (please wait)...
Copy complete.
cs2# copy sftp: bootflash: vrf management
Enter source filename: /code/n9000-epld.9.3.5.img
Enter hostname for the sftp server: 172.19.2.1
Enter username: user1
Outbound-ReKey for 172.19.2.1:22
Inbound-ReKey for 172.19.2.1:22
user1@172.19.2.1's password:
sftp> progress
Progress meter enabled
sftp> get   /code/n9000-epld.9.3.5.img  /bootflash/n9000-epld.9.3.5.img
/code/n9000-epld.9.3.5.img  100%  161MB   9.5MB/s   00:16
sftp> exit
Copy complete, now saving to disk (please wait)...
Copy complete.

  1. Verify the running version of the NX-OS software:
    show version

cs2# show version
Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Copyright (C) 2002-2020, Cisco and/or its affiliates.
All rights reserved.
The copyrights to certain works contained in this software are
owned by other third parties and used and distributed under their own
licenses, such as open source.  This software is provided "as is," and unless
otherwise stated, there is no warranty, express or implied, including but not
limited to warranties of merchantability and fitness for a particular purpose.
Certain components of this software are licensed under
the GNU General Public License (GPL) version 2.0 or
GNU General Public License (GPL) version 3.0  or the GNU
Lesser General Public License (LGPL) Version 2.1 or
Lesser General Public License (LGPL) Version 2.0.
A copy of each such license is available at
http://www.opensource.org/licenses/gpl-2.0.php and
http://opensource.org/licenses/gpl-3.0.html and
http://www.opensource.org/licenses/lgpl-2.1.php and
http://www.gnu.org/licenses/old-licenses/library.txt.
Software
  BIOS: version 08.38
  NXOS: version 9.3(4)
  BIOS compile time:  05/29/2020
  NXOS image file is: bootflash:///nxos.9.3.4.bin
  NXOS compile time:  4/28/2020 21:00:00 [04/29/2020 02:28:31]
Hardware
  cisco Nexus9000 C9336C-FX2 Chassis
  Intel(R) Xeon(R) CPU E5-2403 v2 @ 1.80GHz with 8154432 kB of memory.
  Processor Board ID FOC20291J6K
  Device name: cs2
  bootflash:   53298520 kB
Kernel uptime is 0 day(s), 0 hour(s), 3 minute(s), 42 second(s)
Last reset at 157524 usecs after Mon Nov  2 18:32:06 2020
  Reason: Reset Requested by CLI command reload
  System version: 9.3(4)
  Service:
plugin
  Core Plugin, Ethernet Plugin
Active Package(s):

cs2#

  1. Install the NX-OS image.

Note Installing the image file causes it to be loaded every time the switch is rebooted. 
cs2# install all nxos bootflash:nxos.9.3.5.bin
Installer will perform compatibility check first. Please wait.
Installer is forced disruptive
Verifying image bootflash:/nxos.9.3.5.bin for boot variable "nxos".
[####################] 100% -- SUCCESS
Verifying image type.
[####################] 100% -- SUCCESS
Preparing "nxos" version info using image bootflash:/nxos.9.3.5.bin.
[####################] 100% -- SUCCESS
Preparing "bios" version info using image bootflash:/nxos.9.3.5.bin.
[####################] 100% -- SUCCESS
Performing module support checks.
[####################] 100% -- SUCCESS
Notifying services about system upgrade.
[####################] 100% -- SUCCESS
Compatibility check is done:
Module  bootable       Impact     Install-type  Reason
------  --------  --------------- ------------  ------
  1       yes      disruptive         reset     default upgrade is not hitless
Images will be upgraded according to following table:
Module   Image    Running-Version(pri:alt                New-Version         Upg-
                                                                             Required
------- --------- -------------------------------------- ------------------- ------------
  1      nxos     9.3(4)                                 9.3(5)              yes
  1      bios     v08.37(01/28/2020):v08.23(09/23/2015)  v08.38(05/29/2020)  yes
Switch will be reloaded for disruptive upgrade.
Do you want to continue with the installation (y/n)?  [n] y
Install is in progress, please wait.
Performing runtime checks.
[####################] 100% -- SUCCESS
Setting boot variables.
[####################] 100% -- SUCCESS
Performing configuration copy.
[####################] 100% -- SUCCESS
Module 1: Refreshing compact flash and upgrading bios/loader/bootrom.
Warning: please do not remove or power off the module at this time.
[####################] 100% -- SUCCESS
Finishing the upgrade, switch will reboot in 10 seconds.

  1. Verify the new version of NX-OS software after the switch has rebooted:
    show version

cs2# show version
Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Copyright (C) 2002-2020, Cisco and/or its affiliates.
All rights reserved.
The copyrights to certain works contained in this software are
owned by other third parties and used and distributed under their own
licenses, such as open source.  This software is provided "as is," and unless
otherwise stated, there is no warranty, express or implied, including but not
limited to warranties of merchantability and fitness for a particular purpose.
Certain components of this software are licensed under
the GNU General Public License (GPL) version 2.0 or
GNU General Public License (GPL) version 3.0  or the GNU
Lesser General Public License (LGPL) Version 2.1 or
Lesser General Public License (LGPL) Version 2.0.
A copy of each such license is available at
http://www.opensource.org/licenses/gpl-2.0.php and
http://opensource.org/licenses/gpl-3.0.html and
http://www.opensource.org/licenses/lgpl-2.1.php and
http://www.gnu.org/licenses/old-licenses/library.txt.
Software
  BIOS: version 05.33
  NXOS: version 9.3(5)
  BIOS compile time:  09/08/2018
  NXOS image file is: bootflash:///nxos.9.3.5.bin
  NXOS compile time:  11/4/2018 21:00:00 [11/05/2018 06:11:06]
Hardware
  cisco Nexus9000 C9336C-FX2 Chassis
  Intel(R) Xeon(R) CPU E5-2403 v2 @ 1.80GHz with 8154432 kB of memory.
  Processor Board ID FOC20291J6K
  Device name: cs2
  bootflash:   53298520 kB
Kernel uptime is 0 day(s), 0 hour(s), 3 minute(s), 42 second(s)
Last reset at 277524 usecs after Mon Nov  2 22:45:12 2020
  Reason: Reset due to upgrade
  System version: 9.3(4)
  Service:
plugin
  Core Plugin, Ethernet Plugin
Active Package(s):

  1. Upgrade the EPLD image and reboot the switch.

cs2# show version module 1 epld
EPLD Device                     Version
---- -------------------------- -------
MI   FPGA                        0x7
IO   FPGA                        0x17
MI   FPGA2                       0x2
GEM  FPGA                        0x2
GEM  FPGA                        0x2
GEM  FPGA                        0x2
GEM  FPGA                        0x2
cs2# install epld bootflash:n9000-epld.9.3.5.img module 1
Compatibility check:
Module      Type        Upgradable  Impact      Reason
------  -------------- ------------ ----------- -------
     1      SUP        Yes          disruptive  Module Upgradable
Retrieving EPLD versions... Please wait.
Images will be upgraded according to following table:
Module  Type   EPLD              Running-Version   New-Version  Upg-Required
------- ------ ----------------- ----------------- ------------ ------------
     1  SUP    MI FPGA           0x07              0x07         No
     1  SUP    IO FPGA           0x17              0x19         Yes
     1  SUP    MI FPGA2          0x02              0x02         No
The above modules require upgrade.
The switch will be reloaded at the end of the upgrade
Do you want to continue (y/n) ?  [n] y
Proceeding to upgrade Modules.
Starting Module 1 EPLD Upgrade
Module 1 : IO FPGA [Programming] : 100.00% (     64 of      64 sectors)
Module 1 EPLD upgrade is successful.
Module   Type  Upgrade-Result
-------- ----- --------------
     1   SUP   Success
EPLDs upgraded.
Module 1 EPLD upgrade is successful.

  1. After the switch reboot, log in again and verify that the new version of EPLD loaded successfully.

cs2# show version module 1 epld
EPLD Device                     Version
---- -------------------------- --------
MI   FPGA                        0x7
IO   FPGA                        0x19
MI   FPGA2                       0x2
GEM  FPGA                        0x2
GEM  FPGA                        0x2
GEM  FPGA                        0x2
GEM  FPGA                        0x2

Install the RCF on a Cisco Nexus 9336C- FX2 cluster switch

You can install the RCF after setting up the Nexus 9336C-FX2 cluster switch for the first time. You can also use this procedure to upgrade your RCF version on your cluster switch.

About this task

The examples in this procedure use the following switch and node nomenclature:

  • The names of the two Cisco switches are cs1 and cs2.

  • The node names are cluster1-01, cluster1-02, cluster1-03, and cluster1-04.

  • The cluster LIF names are cluster1-01_clus1, cluster1-01_clus2, cluster1-02_clus1, cluster1-02_clus2, cluster1-03_clus1, cluster1-03_clus2, cluster1-04_clus1, and cluster1-04_clus2.

  • The cluster1::*> prompt indicates the name of the cluster.

Note The procedure requires the use of both ONTAP commands and Cisco Nexus 9000 Series Switches commands; ONTAP commands are used unless otherwise indicated.
Steps

  1. Display the cluster ports on each node that are connected to the cluster switches:
    network device-discovery show

cluster1::*> network device-discovery show
Node/       Local  Discovered
Protocol    Port   Device (LLDP: ChassisID)  Interface         Platform
----------- ------ ------------------------- ----------------  --------
cluster1-01/cdp
            e3a    cs1                       Ethernet1/7       N9K-C9336C
            e0d    cs2                       Ethernet1/7       N9K-C9336C
cluster1-02/cdp
            e3a    cs1                       Ethernet1/8       N9K-C9336C
            e0d    cs2                       Ethernet1/8       N9K-C9336C
cluster1-03/cdp
            e3a    cs1                       Ethernet1/1/1     N9K-C9336C
            e3b    cs2                       Ethernet1/1/1     N9K-C9336C
cluster1-04/cdp
            e3a    cs1                       Ethernet1/1/2     N9K-C9336C
            e3b    cs2                       Ethernet1/1/2     N9K-C9336C
cluster1::*>

  1. Check the administrative and operational status of each cluster port.

  2. Verify that all the cluster ports are up with a healthy status:
    network port show –role cluster

cluster1::*> network port show -role cluster
Node: cluster1-01
                                                                       Ignore
                                                  Speed(Mbps) Health   Health
Port      IPspace      Broadcast Domain Link MTU  Admin/Oper  Status   Status
--------- ------------ ---------------- ---- ---- ----------- -------- ------
e3a       Cluster      Cluster          up   9000  auto/100000 healthy false
e0d       Cluster      Cluster          up   9000  auto/100000 healthy false
Node: cluster1-02
                                                                       Ignore
                                                  Speed(Mbps) Health   Health
Port      IPspace      Broadcast Domain Link MTU  Admin/Oper  Status   Status
--------- ------------ ---------------- ---- ---- ----------- -------- ------
e3a       Cluster      Cluster          up   9000  auto/100000 healthy false
e0d       Cluster      Cluster          up   9000  auto/100000 healthy false
8 entries were displayed.
Node: cluster1-03

                                                                        Ignore
                                                  Speed(Mbps)  Health   Health
Port      IPspace      Broadcast Domain Link MTU  Admin/Oper   Status   Status
--------- ------------ ---------------- ---- ---- ------------ -------- ------
e3a       Cluster      Cluster          up   9000  auto/100000 healthy  false
e3b       Cluster      Cluster          up   9000  auto/100000 healthy  false
Node: cluster1-04
                                                                        Ignore
                                                  Speed(Mbps)  Health   Health
Port      IPspace      Broadcast Domain Link MTU  Admin/Oper   Status   Status
--------- ------------ ---------------- ---- ---- ------------ -------- ------
e0a       Cluster      Cluster          up   9000  auto/100000 healthy  false
e0b       Cluster      Cluster          up   9000  auto/100000 healthy  false
cluster1::*>

  1. Verify that all the cluster interfaces (LIFs) are on the home port:
    network interface show -role cluster

cluster1::*> network interface show -role cluster
        Logical            Status     Network         Current      Current Is
Vserver Interface          Admin/Oper Address/Mask    Node         Port    Home
------- ------------------ ---------- --------------- ------------ ------- ----
Cluster
        cluster1-01_clus1  up/up     169.254.3.4/23   cluster1-01  e3a     true
        cluster1-01_clus2  up/up     169.254.3.5/23   cluster1-01  e0d     true
        cluster1-02_clus1  up/up     169.254.3.8/23   cluster1-02  e3a     true
        cluster1-02_clus2  up/up     169.254.3.9/23   cluster1-02  e0d     true
        cluster1-03_clus1  up/up     169.254.1.3/23   cluster1-03  e3a     true
        cluster1-03_clus2  up/up     169.254.1.1/23   cluster1-03  e3b     true
        cluster1-04_clus1  up/up     169.254.1.6/23   cluster1-04  e3a     true
        cluster1-04_clus2  up/up     169.254.1.7/23   cluster1-04  e3b     true
8 entries were displayed.
cluster1::*>

  1. Verify that the cluster displays information for both cluster switches:
    system cluster-switch show -is-monitoring-enabled-operational true

cluster1::*> system cluster-switch show -is-monitoring-enabled-operational true
Switch                      Type               Address          Model
--------------------------- ------------------ ---------------- -----
cs1                         cluster-network    10.233.205.90    N9K-C9336C
     Serial Number: FOCXXXXXXGD
      Is Monitored: true
            Reason: None
  Software Version: Cisco Nexus Operating System (NX-OS) Software, Version
                    9.3(5)
    Version Source: CDP
cs2                         cluster-network    10.233.205.91    N9K-C9336C
     Serial Number: FOCXXXXXXGS
      Is Monitored: true
            Reason: None
  Software Version: Cisco Nexus Operating System (NX-OS) Software, Version
                    9.3(5)
    Version Source: CDP
cluster1::*>

  1. Disable auto-revert on the cluster LIFs.

cluster1::*> network interface modify -vserver Cluster -lif * -auto-revert false

  1. On cluster switch cs2, shut down the ports connected to the cluster ports of the nodes.

cs2(config)# interface eth1/1/1-2,eth1/7-8
cs2(config-if-range)# shutdown

  1. Verify that the cluster LIFs have migrated to the ports hosted on cluster switch cs1. This might take a few seconds:
    network interface show -role cluster

cluster1::*> network interface show -role cluster
        Logical           Status     Network          Current      Current  Is
Vserver Interface         Admin/Oper Address/Mask     Node         Port     Home
------- ----------------- --------- ---------------- ------------- ------- ----
Cluster
        cluster1-01_clus1 up/up     169.254.3.4/23   cluster1-01   e3a     true
        cluster1-01_clus2 up/up     169.254.3.5/23   cluster1-01   e3a     false
        cluster1-02_clus1 up/up     169.254.3.8/23   cluster1-02   e3a     true
        cluster1-02_clus2 up/up     169.254.3.9/23   cluster1-02   e3a     false
        cluster1-03_clus1 up/up     169.254.1.3/23   cluster1-03   e3a     true
        cluster1-03_clus2 up/up     169.254.1.1/23   cluster1-03   e3a     false
        cluster1-04_clus1 up/up     169.254.1.6/23   cluster1-04   e3a     true
        cluster1-04_clus2 up/up     169.254.1.7/23   cluster1-04   e3a     false
8 entries were displayed.
cluster1::*>

  1. Verify that the cluster is healthy:
    cluster show

cluster1::*> cluster show
Node                 Health  Eligibility   Epsilon
-------------------- ------- ------------  -------
cluster1-01          true    true          false
cluster1-02          true    true          false
cluster1-03          true    true          true
cluster1-04          true    true          false
4 entries were displayed.
cluster1::*>

  1. Clean the configuration on switch cs2 and perform a basic setup.

    1. Clean the configuration. This step requires a console connection to the switch.

cs2# write erase
Warning: This command will erase the startup-configuration.
Do you wish to proceed anyway? (y/n)  [n] y
cs2# reload
This command will reboot the system. (y/n)?  [n] y
cs2#

  1. Perform a basic setup of the switch.

  1. Copy the RCF to the bootflash of switch cs2 using one of the following transfer protocols: FTP, TFTP, SFTP, or SCP. For more information about Cisco commands, see the appropriate guide in the Cisco Nexus 9000 Series NX-OS Command Reference guides.

    This example shows TFTP being used to copy an RCF to the bootflash on switch cs2.

cs2# copy tftp: bootflash: vrf management
Enter source filename: Nexus_9336C_RCF_v1.6-Cluster-HA-Breakout.txt
Enter hostname for the tftp server: 172.22.201.50
Trying to connect to tftp server...Connection to Server Established.
TFTP get operation was successful
Copy complete, now saving to disk (please wait)...

  1. Apply the RCF previously downloaded to the bootflash.
    For more information about Cisco commands, see the appropriate guide in the Cisco Nexus 9000 Series NX-OS Command Reference guides.

    This example shows the RCF file Nexus_9336C_RCF_v1.6-Cluster-HA-Breakout.txt being installed on switch cs2.

cs2# copy Nexus_9336C_RCF_v1.6-Cluster-HA-Breakout.txt running-config echo-commands

  1. Examine the banner output from the show banner motd command. You must read and follow these instructions to ensure the proper configuration and operation of the switch.

cs2# show banner motd
***************************************************************************
* NetApp Reference Configuration File (RCF)
*
* Switch   : Nexus N9K-C9336C-FX2
* Filename : Nexus_9336C_RCF_v1.6-Cluster-HA-Breakout.txt
* Date     : 10-23-2020
* Version  : v1.6
*
* Port Usage:
* Ports  1- 3: Breakout mode (4x10G) Intra-Cluster Ports, int e1/1/1-4,
* e1/2/1-4, e1/3/1-4
* Ports  4- 6: Breakout mode (4x25G) Intra-Cluster/HA Ports, int e1/4/1-4,
* e1/5/1-4, e1/6/1-4
* Ports  7-34: 40/100GbE Intra-Cluster/HA Ports, int e1/7-34
* Ports 35-36: Intra-Cluster ISL Ports, int e1/35-36
*
* Dynamic breakout commands:
* 10G: interface breakout module 1 port <range> map 10g-4x
* 25G: interface breakout module 1 port <range> map 25g-4x
*
* Undo breakout commands and return interfaces to 40/100G configuration in
* config mode:
* no interface breakout module 1 port <range> map 10g-4x
* no interface breakout module 1 port <range> map 25g-4x
* interface Ethernet <interfaces taken out of breakout mode>
* inherit port-profile 40-100G
* priority-flow-control mode auto
* service-policy input HA
* exit
*
***************************************************************************

  1. Verify that the RCF file is the correct newer version:
    show running-config

    When you check the output to verify you have the correct RCF, make sure that the following information is correct:

    • The RCF banner

    • The node and port settings

    • Customizations

      The output varies according to your site configuration. Check the port settings and refer to the release notes for any changes specific to the RCF that you have installed.

  2. After you verify the RCF versions and switch settings are correct, copy the running-config file to the startup-config file.

    For more information about Cisco commands, see the appropriate guide in the Cisco Nexus 9000 Series NX-OS Command Reference guides.

cs2# copy running-config startup-config [########################################] 100% Copy complete

  1. Reboot switch cs2. You can ignore the “cluster ports down” events reported on the nodes while the switch reboots.

cs2# reload
This command will reboot the system. (y/n)?  [n] y

  1. Apply the same RCF and save the running configuration for a second time.

cs2# copy Nexus_9336C_RCF_v1.6-Cluster-HA-Breakout.txt running-config echo-commands
cs2# copy running-config startup-config [########################################] 100% Copy complete

  1. Verify the health of cluster ports on the cluster.

    1. Verify that e0d ports are up and healthy across all nodes in the cluster:
      network port show -role cluster

cluster1::*> network port show -role cluster
Node: cluster1-01
                                                                   Ignore
                                             Speed(Mbps)  Health   Health
Port    IPspace   Broadcast Domain Link MTU  Admin/Oper   Status   Status
------- --------- ---------------- ---- ---- ------------ -------- ------
e3a     Cluster   Cluster          up   9000 auto/100000  healthy  false
e3b     Cluster   Cluster          up   9000 auto/100000  healthy  false

Node: cluster1-02
                                                                   Ignore
                                              Speed(Mbps)  Health  Health
Port    IPspace   Broadcast Domain Link MTU   Admin/Oper   Status  Status
------- --------- ---------------- ---- ----- ------------ -------- ------
e3a    Cluster   Cluster          up   9000  auto/100000  healthy  false
e3b    Cluster   Cluster          up   9000  auto/100000  healthy  false

Node: cluster1-03
                                                                   Ignore
                                              Speed(Mbps) Health   Health
Port   IPspace    Broadcast Domain Link MTU   Admin/Oper  Status   Status
------ ---------- ---------------- ---- ----- ----------- -------- ------
e3a    Cluster    Cluster          up   9000  auto/100000 healthy  false
e0d    Cluster    Cluster          up   9000  auto/100000 healthy  false

Node: cluster1-04
                                                                   Ignore
                                              Speed(Mbps) Health   Health
Port   IPspace    Broadcast Domain Link MTU   Admin/Oper  Status   Status
------ ---------- ---------------- ---- ----- ----------- -------- ------
e3a    Cluster      Cluster        up   9000  auto/100000 healthy  false
e0d    Cluster      Cluster        up   9000  auto/100000 healthy  false
8 entries were displayed.

  1. Verify the switch health from the cluster (this might not show switch cs2, since LIFs are not homed on e0d).

cluster1::*> network device-discovery show -protocol cdp
Node/       Local  Discovered
Protocol    Port   Device (LLDP: ChassisID)  Interface         Platform
----------- ------ ------------------------- ----------------- --------
cluster1-01/cdp
            e3a    cs1                       Ethernet1/7       N9K-C9336C
            e0d    cs2                       Ethernet1/7       N9K-C9336C
cluster01-2/cdp
            e3a    cs1                       Ethernet1/8       N9K-C9336C
            e0d    cs2                       Ethernet1/8       N9K-C9336C
cluster01-3/cdp
            e3a    cs1                       Ethernet1/1/1     N9K-C9336C
            e3b    cs2                       Ethernet1/1/1     N9K-C9336C
cluster1-04/cdp
            e3a    cs1                       Ethernet1/1/2     N9K-C9336C
            e3b    cs2                       Ethernet1/1/2     N9K-C9336C
cluster1::*> system cluster-switch show -is-monitoring-enabled-operational true
Switch                      Type               Address          Model
--------------------------- ------------------ ---------------- -----
cs1                         cluster-network    10.233.205.90    NX9-C9336C
     Serial Number: FOCXXXXXXGD
      Is Monitored: true
            Reason: None
  Software Version: Cisco Nexus Operating System (NX-OS) Software, Version
                    9.3(5)
    Version Source: CDP
cs2                         cluster-network    10.233.205.91    NX9-C9336C
     Serial Number: FOCXXXXXXGS
      Is Monitored: true
            Reason: None
  Software Version: Cisco Nexus Operating System (NX-OS) Software, Version
                    9.3(5)
    Version Source: CDP
2 entries were displayed.
Note You might observe the following output on the cs1 switch console depending on the RCF version previously loaded on the switch. 
2020 Nov 17 16:07:18 cs1 %$ VDC-1 %$ %STP-2-UNBLOCK_CONSIST_PORT: Unblocking port port-channel1 on VLAN0092. Port consistency restored.
2020 Nov 17 16:07:23 cs1 %$ VDC-1 %$ %STP-2-BLOCK_PVID_PEER: Blocking port-channel1 on VLAN0001. Inconsistent peer vlan.
2020 Nov 17 16:07:23 cs1 %$ VDC-1 %$ %STP-2-BLOCK_PVID_LOCAL: Blocking port-channel1 on VLAN0092. Inconsistent local vlan.

  1. On cluster switch cs1, shut down the ports connected to the cluster ports of the nodes. The following example uses the interface example output from step 1:

cs1(config)# interface eth1/1/1-2,eth1/7-8
cs1(config-if-range)# shutdown

  1. Verify that the cluster LIFs have migrated to the ports hosted on switch cs2. This might take a few seconds:
    network interface show -role cluster

cluster1::*> network interface show -role cluster
        Logical            Status      Network         Current      Current Is
Vserver Interface          Admin/Oper  Address/Mask    Node         Port    Home
------- ------------------ ----------- --------------- ------------ ------- ----
Cluster
        cluster1-01_clus1  up/up       169.254.3.4/23   cluster1-01   e0d  false
        cluster1-01_clus2  up/up       169.254.3.5/23   cluster1-01   e0d   true
        cluster1-02_clus1  up/up       169.254.3.8/23   cluster1-02   e0d  false
        cluster1-02_clus2  up/up       169.254.3.9/23   cluster1-02   e0d   true
        cluster1-03_clus1  up/up       169.254.1.3/23   cluster1-03   e3b  false
        cluster1-03_clus2  up/up       169.254.1.1/23   cluster1-03   e3b   true
        cluster1-04_clus1  up/up       169.254.1.6/23   cluster1-04   e3b  false
        cluster1-04_clus2  up/up       169.254.1.7/23   cluster1-04   e3b   true
8 entries were displayed.
cluster1::*>

  1. Verify that the cluster is healthy:
    cluster show

cluster1::*> cluster show
Node                 Health   Eligibility   Epsilon
-------------------- -------- ------------- -------
cluster1-01          true     true          false
cluster1-02          true     true          false
cluster1-03          true     true          true
cluster1-04          true     true          false
4 entries were displayed.
cluster1::*>

  1. Repeat Steps 7 to 14 on switch cs1.

  2. Enable auto-revert on the cluster LIFs.

cluster1::*> network interface modify -vserver Cluster -lif * -auto-revert True

  1. Reboot switch cs1. You do this to trigger the cluster LIFs to revert to their home ports. You can ignore the “cluster ports down” events reported on the nodes while the switch reboots.

cs1# reload
This command will reboot the system. (y/n)?  [n] y

  1. Verify that the switch ports connected to the cluster ports are up.

cs1# show interface brief | grep up
.
.
Eth1/1/1      1       eth  access up      none                   100G(D) --
Eth1/1/2      1       eth  access up      none                   100G(D) --
Eth1/7        1       eth  trunk  up      none                   100G(D) --
Eth1/8        1       eth  trunk  up      none                   100G(D) --
.
.

  1. Verify that the ISL between cs1 and cs2 is functional:
    show port-channel summary

cs1# show port-channel summary
Flags:  D - Down        P - Up in port-channel (members)
        I - Individual  H - Hot-standby (LACP only)
        s - Suspended   r - Module-removed
        b - BFD Session Wait
        S - Switched    R - Routed
        U - Up (port-channel)
        p - Up in delay-lacp mode (member)
        M - Not in use. Min-links not met
--------------------------------------------------------------------------------
Group Port-       Type     Protocol  Member Ports      Channel
--------------------------------------------------------------------------------
1     Po1(SU)     Eth      LACP      Eth1/35(P)        Eth1/36(P)
cs1#

  1. Verify that the cluster LIFs have reverted to their home port:
    network interface show -role cluster

cluster1::*> network interface show -role cluster
        Logical            Status     Network           Current     Current Is
Vserver Interface          Admin/Oper Address/Mask      Node        Port    Home
------- ------------------ ---------- ----------------- ----------- ------- ----
Cluster
        cluster1-01_clus1  up/up      169.254.3.4/23    cluster1-01  e0d   true
        cluster1-01_clus2  up/up      169.254.3.5/23    cluster1-01  e0d   true
        cluster1-02_clus1  up/up      169.254.3.8/23    cluster1-02  e0d   true
        cluster1-02_clus2  up/up      169.254.3.9/23    cluster1-02  e0d   true
        cluster1-03_clus1  up/up      169.254.1.3/23    cluster1-03  e3b   true
        cluster1-03_clus2  up/up      169.254.1.1/23    cluster1-03  e3b   true
        cluster1-04_clus1  up/up      169.254.1.6/23    cluster1-04  e3b   true
        cluster1-04_clus2  up/up      169.254.1.7/23    cluster1-04  e3b   true
8 entries were displayed.
cluster1::*>

  1. Verify that the cluster is healthy:
    cluster show

cluster1::*> cluster show
Node                 Health  Eligibility   Epsilon
-------------------- ------- ------------- -------
cluster1-01          true    true          false
cluster1-02          true    true          false
cluster1-03          true    true          true
cluster1-04          true    true          false
4 entries were displayed.
cluster1::*>

  1. Ping the remote cluster interfaces to verify connectivity:
    cluster ping-cluster -node local

cluster1::*> cluster ping-cluster -node local
Host is cluster1-03
Getting addresses from network interface table...
Cluster cluster1-03_clus1 169.254.1.3 cluster1-03 e3a
Cluster cluster1-03_clus2 169.254.1.1 cluster1-03 e3b
Cluster cluster1-04_clus1 169.254.1.6 cluster1-04 e3a
Cluster cluster1-04_clus2 169.254.1.7 cluster1-04 e3b
Cluster cluster1-01_clus1 169.254.3.4 cluster1-01 e3a
Cluster cluster1-01_clus2 169.254.3.5 cluster1-01 e0d
Cluster cluster1-02_clus1 169.254.3.8 cluster1-02 e3a
Cluster cluster1-02_clus2 169.254.3.9 cluster1-02 e0d
Local = 169.254.1.3 169.254.1.1
Remote = 169.254.1.6 169.254.1.7 169.254.3.4 169.254.3.5 169.254.3.8 169.254.3.9
Cluster Vserver Id = 4294967293
Ping status:

Basic connectivity succeeds on 12 path(s)
Basic connectivity fails on 0 path(s)

Detected 9000 byte MTU on 12 path(s):
    Local 169.254.1.3 to Remote 169.254.1.6
    Local 169.254.1.3 to Remote 169.254.1.7
    Local 169.254.1.3 to Remote 169.254.3.4
    Local 169.254.1.3 to Remote 169.254.3.5
    Local 169.254.1.3 to Remote 169.254.3.8
    Local 169.254.1.3 to Remote 169.254.3.9
    Local 169.254.1.1 to Remote 169.254.1.6
    Local 169.254.1.1 to Remote 169.254.1.7
    Local 169.254.1.1 to Remote 169.254.3.4
    Local 169.254.1.1 to Remote 169.254.3.5
    Local 169.254.1.1 to Remote 169.254.3.8
    Local 169.254.1.1 to Remote 169.254.3.9
Larger than PMTU communication succeeds on 12 path(s)
RPC status:
6 paths up, 0 paths down (tcp check)
6 paths up, 0 paths down (udp check)

Install the RCF on a Cisco Nexus 9336C-FX2 storage switch

The reference configuration files (RCFs) can be upgraded on Cisco Nexus 9336C-FX2 storage switches.

Before you begin

The following conditions must exist before you upgrade the RCF on the storage switch:

  • The switch must be fully functioning (there should be no errors in the logs or similar issues).

  • You must have checked or set your desired boot variables in the RCF to reflect the desired boot images if you are installing only NX-OS and keeping your current RCF version.

  • If you need to change the boot variables to reflect the current boot images, you must do so before reapplying the RCF so that the correct version is instantiated on future reboots.

  • You must have referred to the appropriate software and upgrade guides available on the Cisco web site for complete documentation on the Cisco storage upgrade and downgrade procedures. See Cisco Nexus 9000 Series Switches for more information.

  • The number of 100 GbE ports are defined in the reference configuration files (RCFs) available on the Cisco Ethernet switches page.

Procedure summary

  1. Check the health status of switches and ports (steps 1-4)

  2. Download the NX-OS image to Cisco switch st2 and reboot (steps 5-8)

  3. Copy the RCF to Cisco switch st2 (steps 9-12)

  4. Recheck the health status of switches and ports (steps 13-15)

  5. Repeat steps 1-15 for Cisco switch st1.

Note The command outputs might vary depending on different releases of ONTAP.

The examples in this procedure use the following switch and node nomenclature:

  • The names of the two storage switches are st1 and st2.

  • The nodes are node1 and node2.

Note The procedure requires the use of both ONTAP commands and Cisco Nexus 9000 Series Switches commands; ONTAP commands are used unless otherwise indicated.
Steps

  1. If AutoSupport is enabled on this cluster, suppress automatic case creation by invoking an AutoSupport message: system node autosupport invoke -node * -type all - message MAINT=xh

    Where x is the duration of the maintenance window in hours.

  2. Check that the storage switches are available:
    system switch ethernet show

storage::*> system switch ethernet show
Switch                    Type               Address          Model
------------------------- ------------------ ---------------- ---------------
st1
                          storage-network    172.17.227.5     NX9-C9336C
      Serial Number: FOC221206C2
       Is Monitored: true
             Reason: None
   Software Version: Cisco Nexus Operating System (NX-OS) Software, Version
                     9.3(5)
     Version Source: CDP
st2
                          storage-network    172.17.227.6     NX9-C9336C
      Serial Number: FOC220443LZ
       Is Monitored: true
             Reason: None
   Software Version: Cisco Nexus Operating System (NX-OS) Software, Version
                     9.3(5)
     Version Source: CDP
2 entries were displayed.
storage::*>

  1. Verify that the node ports are healthy and operational:
    storage port show -port-type ENET

storage::*> storage port show -port-type ENET
                                   Speed                            VLAN
Node     Port   Type    Mode       (Gb/s)     State     Status        ID
------- ------- ------- ---------- ---------- --------- ----------- -----
node1
        e3a     ENET    storage    100        enabled   online         30
        e3b     ENET    storage      0        enabled   offline        30
        e7a     ENET    storage      0        enabled   offline        30
        e7b     ENET    storage    100        enabled   online         30
node2
        e3a     ENET    storage    100        enabled   online         30
        e3b     ENET    storage      0        enabled   offline        30
        e7a     ENET    storage      0        enabled   offline        30
        e7b     ENET    storage    100        enabled   online         30

  1. Check that there are no storage switch or cabling issues with the cluster:
    system health alert show -instance

storage::*> system health alert show -instance
There are no entries matching your query.

  1. Download the NX-OS image to switch st2.

  2. Install the system image so that the new version will be loaded the next time switch st2 is rebooted. The switch will be reboot in 10 seconds with the new image as shown in the following output:

st2# install all nxos bootflash:nxos.9.3. 5.bin
Installer will perform compatibility check first. Please wait.
Installer is forced disruptive
Verifying image bootflash:/nxos.9.3.4.bin for boot variable "nxos".
[####################] 100% -- SUCCESS
Verifying image type.
[[####################] 100% -- SUCCESS
Preparing "nxos" version info using image bootflash:/nxos.9.3.4.bin.
[####################] 100% -- SUCCESS
Preparing "bios" version info using image bootflash:/nxos.9.3.4.bin.
[####################] 100% -- SUCCESS
Performing module support checks.
[####################] 100% -- SUCCESS
Notifying services about system upgrade.
[####################] 100% -- SUCCESS
Compatibility check is done:
Module  bootable  Impact  Install-type  Reason
------    --------   ----- --------   ------------   ---- --
     1        yes      disruptive         reset  default upgrade is not hitless
Images will be upgraded according to following table:
Module Image        Running-Version(pri:alt)               New-Version  Upg
                                                                        Required
------ --------  ---------------------------------------  ------------  --------
 1     nxos                                     9.3(3)          9.3(4)       yes
 1     bios      v08.37(01/28/2020):v08.23(09/23/2015)   v08.38(05/29/2020)   no
Switch will be reloaded for disruptive upgrade.
Do you want to continue with the installation (y/n)? [n] y
input string too long
Do you want to continue with the installation (y/n)? [n] y
Install is in progress, please wait.
Performing runtime checks.
[####################] 100% -- SUCCESS
Setting boot variables.
[####################] 100% -- SUCCESS
Performing configuration copy.
[####################] 100% -- SUCCESS
Module 1: Refreshing compact flash and upgrading bios/loader/bootrom.
Warning: please do not remove or power off the module at this time.
[####################] 100% -- SUCCESS
Finishing the upgrade, switch will reboot in 10 seconds.
st2#

  1. Save the configuration.

    You are prompted to reboot the system as shown in the following example:

st2# copy running-config startup-config
[########################################] 100% Copy complete.
st2# reload
This command will reboot the system. (y/n)? [n] y

  1. Confirm that the new NX-OS version number is on the switch.

st2# show version
Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Upgrading a Cisco Nexus 9336C Storage Switch 6
Upgrading a Cisco Nexus 9336C storage switch
Copyright (C) 2002-2020, Cisco and/or its affiliates.
All rights reserved.
The copyrights to certain works contained in this software are
owned by other third parties and used and distributed under their own
licenses, such as open source. This software is provided "as is," and unless otherwise stated, there is no warranty, express or implied, including but not limited to warranties of merchantability and fitness for a particular purpose.
Certain components of this software are licensed under
the GNU General Public License (GPL) version 2.0 or
GNU General Public License (GPL) version 3.0 or the GNU
Lesser General Public License (LGPL) Version 2.1 or
Lesser General Public License (LGPL) Version 2.0.
A copy of each such license is available at
http://www.opensource.org/licenses/gpl-2.0.php and
http://opensource.org/licenses/gpl-3.0.html and
http://www.opensource.org/licenses/lgpl-2.1.php and
.
Software
 BIOS: version 08.38
 NXOS: version 9.3(5)
 BIOS compile time: 05/29/2020
 NXOS image file is: bootflash:///nxos.9.3. 5.bin
 NXOS compile time: 4/28/2020 21:00:00 [04/29/2020 02:28:31]
Hardware
 cisco Nexus9000 C9336C Chassis (Nexus 9000 Series)
 Intel(R) Xeon(R) CPU E5-2403 v2 @ 1.80GHz with 8154432 kB of memory.
 Processor Board ID FOC20291J6K
 Device name: S2
 bootflash: 53298520 kB
Kernel uptime is 0 day(s), 0 hour(s), 3 minute(s), 42 second(s)
Last reset at 157524 usecs after Mon Nov 2 18:32:06 2020
           Reason: Reset due to upgrade
   System version: 9.3(5)
   Service:
plugin
   Core Plugin, Ethernet Plugin
   Active Package(s):
st2#

  1. Copy the RCF on switch st2 to the switch bootflash using one of the following transfer protocols: FTP, HTTP, TFTP, SFTP, or SCP.

    For more information about Cisco commands, see the appropriate guide in the Cisco Nexus 9000 Series NX-OS Command Reference guides.

    The following example shows HTTP being used to copy an RCF to the bootflash on switch st2:

st2# copy http://172.16.10.1//cfg/Nexus_9336C_RCF_v1.6-Storage.txt bootflash: vrf management
% Total % Received % Xferd   Average   Speed  Time   Time   Time
Current
   Dload     Upload  Total Spent   Left
Speed
 100    3254          100     3254      0       0     8175    0 --:--:-- --:--:-- --:--:–
8301
Copy complete, now saving to disk (please wait)...
Copy complete.
st2#

  1. Apply the RCF previously downloaded to the bootflash:
    copy bootflash.

    The following example shows the RCF file Nexus_9336C_RCF_v1.6-Storage.txt being installed on switch st2:

st2# copy Nexus_9336C_RCF_v1.6-Storage.txt running-config echo-commands

  1. Verify that the RCF file is the correct newer version:
    show running-config

    When you check the output to verify you have the correct RCF, make sure that the following information is correct:

    • The RCF banner

    • The node and port settings

    • Customizations

      The output varies according to your site configuration. Check the port settings and refer to the release notes for any changes specific to the RCF that you have installed.

      Important: In the banner output from the show banner motd command, you must read and follow the instructions in the *IMPORTANT NOTES *section to ensure the proper configuration and operation of the switch.

st2# show banner motd
******************************************************************************
*NetApp Reference Configuration File (RCF)
*
*Switch : Nexus N9K-C9336C-FX2
*Filename : Nexus_9336C_RCF_v1.6-Storage.txt
* Date : 10-23-2020
*Version  : v1.6
*
*Port Usage: Storage configuration
*Ports 1-36: 100GbE Controller and Shelf Storage Ports
*
*IMPORTANT NOTES*
*- This RCF utilizes QoS and requires TCAM re-configuration, requiring RCF
*to be loaded twice with the Storage Switch rebooted in between.
*
*- Perform the following 4 steps to ensure proper RCF installation:
*
*(1) Apply RCF first time, expect following messages:
*- Please save config and reload the system...
*- Edge port type (portfast) should only be enabled on ports...
*- TCAM region is not configured for feature QoS class IPv4 ingress...
*
*(2) Save running-configuration and reboot Cluster Switch
*
*(3) After reboot, apply same RCF second time and expect following messages:
*- % Invalid command at '^' marker
*- Syntax error while parsing...
*
*(4) Save running-configuration again
******************************************************************************
st2#

  1. After you verify that the software versions and switch settings are correct, copy the running-config file to the startup-config file on switch st2.

    For more information on Cisco commands, see the appropriate guide in the Cisco Nexus 9000 Series NX-OS Command Reference guides.

    The following example shows the running-config file successfully copied to the startup-config file:

st2# copy running-config startup-config
[########################################] 100% Copy complete.

  1. Recheck that the storage switches are available after the reboot:
    system switch ethernet show

storage::*> system switch ethernet show
Switch                       Type               Address          Model
---------------------------- ------------------ ---------------- ---------------
st1
                            storage-network     172.17.227.5     NX9-C9336C
     Serial Number: FOC221206C2
      Is Monitored: true
            Reason: None
  Software Version: Cisco Nexus Operating System (NX-OS) Software, Version
                    9.3(5)
    Version Source: CDP
st2
                            storage-network      172.17.227.6    NX9-C9336C
     Serial Number: FOC220443LZ
      Is Monitored: true
            Reason: None
  Software Version: Cisco Nexus Operating System (NX-OS) Software, Version
                    9.3(5)
    Version Source: CDP
2 entries were displayed.
storage::*

  1. Verify that the switch ports are healthy and operational after the reboot:
    storage port show -port-type ENET

storage::*> storage port show -port-type ENET
                                   Speed                            VLAN
Node    Port    Type    Mode       (Gb/s)      State     Status       ID
------- ------- ------- ---------- ----------- --------- --------- -----
node1
        e3a     ENET    storage           100   enabled   online       30
        e3b     ENET    storage             0   enabled   offline      30
        e7a     ENET    storage             0   enabled   offline      30
        e7b     ENET    storage           100   enabled   online       30
node2
        e3a     ENET    storage           100   enabled   online       30
        e3b     ENET    storage             0   enabled   offline      30
        e7a     ENET    storage             0   enabled   offline      30
        e7b     ENET    storage           100   enabled   online       30

  1. Recheck that there is no storage switch or cabling issues with the cluster:
    system health alert show -instance

storage::*> system health alert show -instance
There are no entries matching your query.

  1. Repeat this procedure for the RCF on switch st1.

  2. If you suppressed automatic case creation, re-enable it by invoking an AutoSupport message:
    system node autosupport invoke -node * -type all -message MAINT=END

Install the RCF on a Cisco Nexus 9336C-FX2 shared switch

From ONTAP 9.9.1, you can use Cisco Nexus 9336C-FX2 switches to combine storage and cluster functionality into a shared switch scenario.

Before you begin

  • The cluster switches must be fully functioning (there should be no errors in the logs or similar issues).

  • The storage switches must be fully functioning (there should be no errors in the logs or similar issues).

  • The names of the two storage switches are sh1 and sh2.

  • The example used here loads the shared RCF on to the new switch.

Steps

  1. Copy the RCF on switch sh2 to the switch bootflash using one of the following transfer protocols: FTP, HTTP, TFTP, SFTP, or SCP.

    For more information on Cisco commands, see the appropriate guide in the Cisco Nexus 9000 Series NX-OS Command Reference guides.

    The following example shows HTTP being used to copy an RCF to the bootflash on switch sh2:

sh2# copy http://172.16.10.1//cfg/Nexus_9336C_RCF_v1.7-Cluster-Ha-Storage.txt bootflash: vrf management
% Total % Received % Xferd   Average   Speed  Time   Time   Time
Current
   Dload     Upload  Total Spent   Left
Speed
 100    5143          100     5143      0       0     11300    0 --:--:-- --:--:-- --:--:–
11300
Copy complete, now saving to disk (please wait)...
Copy complete.
sh2#

  1. Apply the RCF previously downloaded to the bootflash:
    copy bootflash.

    The following example shows the RCF file `Nexus_9336C_RCF_v1.7-Cluster-HA-Storage.txt ` being installed on switch sh2:

sh2# copy Nexus_9336C_RCF_v1.7-Cluster-HA-Storage.txt running-config echo-commands

  1. Verify that the RCF file is the correct newer version: show running-config

    When you check the output to verify you have the correct RCF, make sure that the following information is correct:

    • The RCF banner

    • The node and port settings

    • Customizations

      The output varies according to your site configuration. Check the port settings and refer to the release notes for any changes specific to the RCF that you have installed.

      Important: In the banner output from the show banner motd command, you must read and follow the instructions in the *IMPORTANT NOTES *section to ensure the proper configuration and operation of the switch.

sh2# show banner motd
******************************************************************************
*NetApp Reference Configuration File (RCF)
*
*Switch : Nexus N9K-C9336C-FX2
*Filename : Nexus_9336C_RCF_v1.7-Cluster-HA-Storage.txt
* Date :  Jan-08-2021
*Version  : v1.7
*
*Port Usage:
*Ports  1-8: 40/100GbE Intra-Cluster/HA Ports, int e1/1-8
*Port     9: 10GbE breakout Intra-Cluster Ports, int e1/9/1-4
*Port    10: 25GbE breakout Intra-Cluster/HA Ports, int e1/10/1-4
*Ports 11-22: First HA-pair Controller and Shelf Storage Ports, int e1/11-22
*Ports 23-34: Second HA-pair Controller and Shelf Storage Ports, int e1/23-34
*Ports 35-36: Intra-Cluster ISL Ports, int e1/35-36
*
* Undo breakout commands and return interfaces to 40/100G configuration in
* config mode:
* no interface breakout module 1 port 9 map 10g-4x
* no interface breakout module 1 port 10 map 25g-4x
* interface Ethernet 1/9-10
* inherit port-profile CLUSTER_HA
* priority-flow-control mode auto
* service-policy type qos input HA_POLICY
* exit
*
*IMPORTANT NOTES*
* In certain conditions, N9K-C9336C-FX2 may not be able to auto-negotiate port
* speed correctly, and port speed must be manually set, in config mode, e.g.
* int e1/1
* speed 40000
* int e1/3
* speed 100000
*
******************************************************************************
sh2#

  1. After you verify that the software versions and switch settings are correct, copy the running-config file to the startup-config file on switch sh2.

    For more information on Cisco commands, see the appropriate guide in the Cisco Nexus 9000 Series NX-OS Command Reference guides.

    The following example shows the running-config file successfully copied to the startup-config file:

sh2# copy running-config startup-config
[########################################] 100% Copy complete.

  1. Repeat this procedure for the RCF on switch sh1.