Install NX-OS software and Reference Configuration Files (RCFs)
Install NX-OS software and RCF on a Cisco Nexus 9336C- FX2 cluster switch
The Cisco NX-OS software and reference configuration file (RCF) must be installed on Cisco Nexus 9336C-FX2 cluster switches.
The following conditions must exist before you install the NX-OS software and RCF on the cluster switch:
-
The cluster must be fully functioning (there should be no errors in the logs or similar issues).
-
You must have checked or set your desired boot configuration in the RCF to reflect the desired boot images if you are installing only NX-OS and keeping your current RCF version.
-
If you need to change the boot configuration to reflect the current boot images, you must do so before reapplying the RCF so that the correct version is instantiated on future reboots.
-
You must have a console connection to the switch, required when installing the RCF.
-
You must have consulted the switch compatibility table on the Cisco Ethernet switch page for the supported ONTAP, NX-OS, and RCF versions.
See Cisco Ethernet Switches for more information. -
There can be command dependencies between the command syntax in the RCF and that found in versions of NX-OS.
-
You must have referred to the appropriate software and upgrade guides available on the Cisco web site for complete documentation on the Cisco switch upgrade and downgrade procedures on Cisco Nexus 9000 Series Switches.
See Cisco Nexus 9000 Series Switches for more information. -
You must have the current RCF.
Initial setup
The examples in this procedure use two nodes. These nodes use two 100GbE cluster interconnect ports e3a and e3b, as per the A400 controller.
See the Hardware Universe to verify the correct cluster ports on your platforms.
The command outputs might vary depending on different releases of ONTAP. |
The examples in this procedure use the following switch and node nomenclature:
-
The names of the two Cisco switches are cs1 and cs2.
-
The node names are cluster1-01 and cluster1-02.
-
The cluster LIF names are cluster1-01_clus1 and cluster1-01_clus2 for cluster1-01 and cluster1-02_clus1 and cluster1-02_clus2 for cluster1-02.
-
The cluster1::*> prompt indicates the name of the cluster.
The procedure requires the use of both ONTAP commands and Cisco Nexus 9000 Series Switches commands; ONTAP commands are used unless otherwise indicated. |
cluster1:: network device-discovery show -protocol cdp Node/ Local Discovered Protocol Port Device (LLDP: ChassisID) Interface Platform ----------- ------ ------------------------- ----------------- -------- cluster1-02/cdp e3a cs1 Eth1/2 N9K-C9336C e3b cs2 Eth1/2 N9K-C9336C cluster1-01/cdp e3a cs1 Eth1/1 N9K-C9336C e3b cs2 Eth1/1 N9K-C9336C 4 entries were displayed.
cluster1::*> network port show -ipspace Cluster Node: cluster1-02 Speed(Mbps) Health Port IPspace Broadcast Domain Link MTU Admin/Oper Status --------- ------------ ---------------- ---- ---- ------------ ------ e3a Cluster Cluster up 9000 auto/100000 healthy e3b Cluster Cluster up 9000 auto/100000 healthy Node: cluster1-01 Speed(Mbps) Health Port IPspace Broadcast Domain Link MTU Admin/Oper Status --------- ------------ ---------------- ---- ---- ------------ ------ e3a Cluster Cluster up 9000 auto/100000 healthy e3b Cluster Cluster up 9000 auto/100000 healthy 4 entries were displayed.
-
Display information about the LIFs:
network interface show - vserver Cluster
cluster1::*> network interface show -vserver Cluster Logical Status Network Current Current Is Vserver Interface Admin/Oper Address/Mask Node Port Home ------- ------------------ ---------- ------------------ ----------- ------ ---- Cluster cluster1-01_clus1 up/up 169.254.209.69/16 cluster1-01 e3a true cluster1-01_clus2 up/up 169.254.49.125/16 cluster1-01 e3b true cluster1-02_clus1 up/up 169.254.47.194/16 cluster1-02 e3a true cluster1-02_clus2 up/up 169.254.19.183/16 cluster1-02 e3b true 4 entries were displayed.
cluster1::*> cluster ping-cluster -node cluster1-02 Host is cluster1-02 Getting addresses from network interface table... Cluster cluster1-01_clus1 169.254.209.69 cluster1-01 e3a Cluster cluster1-01_clus2 169.254.49.125 cluster1-01 e3b Cluster cluster1-02_clus1 169.254.47.194 cluster1-02 e3a Cluster cluster1-02_clus2 169.254.19.183 cluster1-02 e3b Local = 169.254.47.194 169.254.19.183 Remote = 169.254.209.69 169.254.49.125 Cluster Vserver Id = 4294967293 Ping status: Basic connectivity succeeds on 4 path(s) Basic connectivity fails on 0 path(s) Detected 9000 byte MTU on 4 path(s): Local 169.254.19.183 to Remote 169.254.209.69 Local 169.254.19.183 to Remote 169.254.49.125 Local 169.254.47.194 to Remote 169.254.209.69 Local 169.254.47.194 to Remote 169.254.49.125 Larger than PMTU communication succeeds on 4 path(s) RPC status: 2 paths up, 0 paths down (tcp check) 2 paths up, 0 paths down (udp check)
cluster1::*> network interface show -vserver Cluster -fields auto-revert Logical Vserver Interface Auto-revert --------- ––––––-------------- ------------ Cluster cluster1-01_clus1 true cluster1-01_clus2 true cluster1-02_clus1 true cluster1-02_clus2 true 4 entries were displayed.
cluster1::*> system switch ethernet log setup password Enter the switch name: <return> The switch name entered is not recognized. Choose from the following list: cs1 cs2 cluster1::*> system switch ethernet log setup-password Enter the switch name: cs1 RSA key fingerprint is e5:8b:c6:dc:e2:18:18:09:36:63:d9:63:dd:03:d9:cc Do you want to continue? {y|n}::[n] y Enter the password: <enter switch password> Enter the password again: <enter switch password> cluster1::*> system switch ethernet log setup-password Enter the switch name: cs2 RSA key fingerprint is 57:49:86:a1:b9:80:6a:61:9a:86:8e:3c:e3:b7:1f:b1 Do you want to continue? {y|n}:: [n] y Enter the password: <enter switch password> Enter the password again: <enter switch password> cluster1::*> system switch ethernet log enable-collection Do you want to enable cluster log collection for all nodes in the cluster? {y|n}: [n] y Enabling cluster switch log collection. cluster1::*>
If any of these commands return an error, contact NetApp support. |
Install the NX-OS software on a Cisco Nexus 9336C- FX2 cluster switch
You can use this procedure to install the NX-OS software on the Cisco Nexus 9336C-FX2 cluster switch.
cs2# ping 172.19.2.1 Pinging 172.19.2.1 with 0 bytes of data: Reply From 172.19.2.1: icmp_seq = 0. time= 5910 usec.
cs2# copy sftp: bootflash: vrf management Enter source filename: /code/nxos.9.3.5.bin Enter hostname for the sftp server: 172.19.2.1 Enter username: user1 Outbound-ReKey for 172.19.2.1:22 Inbound-ReKey for 172.19.2.1:22 user1@172.19.2.1's password: sftp> progress Progress meter enabled sftp> get /code/nxos.9.3.5.bin /bootflash/nxos.9.3.5.bin /code/nxos.9.3.5.bin 100% 1261MB 9.3MB/s 02:15 sftp> exit Copy complete, now saving to disk (please wait)... Copy complete. cs2# copy sftp: bootflash: vrf management Enter source filename: /code/n9000-epld.9.3.5.img Enter hostname for the sftp server: 172.19.2.1 Enter username: user1 Outbound-ReKey for 172.19.2.1:22 Inbound-ReKey for 172.19.2.1:22 user1@172.19.2.1's password: sftp> progress Progress meter enabled sftp> get /code/n9000-epld.9.3.5.img /bootflash/n9000-epld.9.3.5.img /code/n9000-epld.9.3.5.img 100% 161MB 9.5MB/s 00:16 sftp> exit Copy complete, now saving to disk (please wait)... Copy complete.
cs2# show version Cisco Nexus Operating System (NX-OS) Software TAC support: http://www.cisco.com/tac Copyright (C) 2002-2020, Cisco and/or its affiliates. All rights reserved. The copyrights to certain works contained in this software are owned by other third parties and used and distributed under their own licenses, such as open source. This software is provided "as is," and unless otherwise stated, there is no warranty, express or implied, including but not limited to warranties of merchantability and fitness for a particular purpose. Certain components of this software are licensed under the GNU General Public License (GPL) version 2.0 or GNU General Public License (GPL) version 3.0 or the GNU Lesser General Public License (LGPL) Version 2.1 or Lesser General Public License (LGPL) Version 2.0. A copy of each such license is available at http://www.opensource.org/licenses/gpl-2.0.php and http://opensource.org/licenses/gpl-3.0.html and http://www.opensource.org/licenses/lgpl-2.1.php and http://www.gnu.org/licenses/old-licenses/library.txt. Software BIOS: version 08.38 NXOS: version 9.3(4) BIOS compile time: 05/29/2020 NXOS image file is: bootflash:///nxos.9.3.4.bin NXOS compile time: 4/28/2020 21:00:00 [04/29/2020 02:28:31] Hardware cisco Nexus9000 C9336C-FX2 Chassis Intel(R) Xeon(R) CPU E5-2403 v2 @ 1.80GHz with 8154432 kB of memory. Processor Board ID FOC20291J6K Device name: cs2 bootflash: 53298520 kB Kernel uptime is 0 day(s), 0 hour(s), 3 minute(s), 42 second(s) Last reset at 157524 usecs after Mon Nov 2 18:32:06 2020 Reason: Reset Requested by CLI command reload System version: 9.3(4) Service: plugin Core Plugin, Ethernet Plugin Active Package(s): cs2#
Installing the image file causes it to be loaded every time the switch is rebooted. |
cs2# install all nxos bootflash:nxos.9.3.5.bin Installer will perform compatibility check first. Please wait. Installer is forced disruptive Verifying image bootflash:/nxos.9.3.5.bin for boot variable "nxos". [####################] 100% -- SUCCESS Verifying image type. [####################] 100% -- SUCCESS Preparing "nxos" version info using image bootflash:/nxos.9.3.5.bin. [####################] 100% -- SUCCESS Preparing "bios" version info using image bootflash:/nxos.9.3.5.bin. [####################] 100% -- SUCCESS Performing module support checks. [####################] 100% -- SUCCESS Notifying services about system upgrade. [####################] 100% -- SUCCESS Compatibility check is done: Module bootable Impact Install-type Reason ------ -------- --------------- ------------ ------ 1 yes disruptive reset default upgrade is not hitless Images will be upgraded according to following table: Module Image Running-Version(pri:alt New-Version Upg- Required ------- --------- -------------------------------------- ------------------- ------------ 1 nxos 9.3(4) 9.3(5) yes 1 bios v08.37(01/28/2020):v08.23(09/23/2015) v08.38(05/29/2020) yes Switch will be reloaded for disruptive upgrade. Do you want to continue with the installation (y/n)? [n] y Install is in progress, please wait. Performing runtime checks. [####################] 100% -- SUCCESS Setting boot variables. [####################] 100% -- SUCCESS Performing configuration copy. [####################] 100% -- SUCCESS Module 1: Refreshing compact flash and upgrading bios/loader/bootrom. Warning: please do not remove or power off the module at this time. [####################] 100% -- SUCCESS Finishing the upgrade, switch will reboot in 10 seconds.
cs2# show version Cisco Nexus Operating System (NX-OS) Software TAC support: http://www.cisco.com/tac Copyright (C) 2002-2020, Cisco and/or its affiliates. All rights reserved. The copyrights to certain works contained in this software are owned by other third parties and used and distributed under their own licenses, such as open source. This software is provided "as is," and unless otherwise stated, there is no warranty, express or implied, including but not limited to warranties of merchantability and fitness for a particular purpose. Certain components of this software are licensed under the GNU General Public License (GPL) version 2.0 or GNU General Public License (GPL) version 3.0 or the GNU Lesser General Public License (LGPL) Version 2.1 or Lesser General Public License (LGPL) Version 2.0. A copy of each such license is available at http://www.opensource.org/licenses/gpl-2.0.php and http://opensource.org/licenses/gpl-3.0.html and http://www.opensource.org/licenses/lgpl-2.1.php and http://www.gnu.org/licenses/old-licenses/library.txt. Software BIOS: version 05.33 NXOS: version 9.3(5) BIOS compile time: 09/08/2018 NXOS image file is: bootflash:///nxos.9.3.5.bin NXOS compile time: 11/4/2018 21:00:00 [11/05/2018 06:11:06] Hardware cisco Nexus9000 C9336C-FX2 Chassis Intel(R) Xeon(R) CPU E5-2403 v2 @ 1.80GHz with 8154432 kB of memory. Processor Board ID FOC20291J6K Device name: cs2 bootflash: 53298520 kB Kernel uptime is 0 day(s), 0 hour(s), 3 minute(s), 42 second(s) Last reset at 277524 usecs after Mon Nov 2 22:45:12 2020 Reason: Reset due to upgrade System version: 9.3(4) Service: plugin Core Plugin, Ethernet Plugin Active Package(s):
cs2# show version module 1 epld EPLD Device Version ---- -------------------------- ------- MI FPGA 0x7 IO FPGA 0x17 MI FPGA2 0x2 GEM FPGA 0x2 GEM FPGA 0x2 GEM FPGA 0x2 GEM FPGA 0x2 cs2# install epld bootflash:n9000-epld.9.3.5.img module 1 Compatibility check: Module Type Upgradable Impact Reason ------ -------------- ------------ ----------- ------- 1 SUP Yes disruptive Module Upgradable Retrieving EPLD versions... Please wait. Images will be upgraded according to following table: Module Type EPLD Running-Version New-Version Upg-Required ------- ------ ----------------- ----------------- ------------ ------------ 1 SUP MI FPGA 0x07 0x07 No 1 SUP IO FPGA 0x17 0x19 Yes 1 SUP MI FPGA2 0x02 0x02 No The above modules require upgrade. The switch will be reloaded at the end of the upgrade Do you want to continue (y/n) ? [n] y Proceeding to upgrade Modules. Starting Module 1 EPLD Upgrade Module 1 : IO FPGA [Programming] : 100.00% ( 64 of 64 sectors) Module 1 EPLD upgrade is successful. Module Type Upgrade-Result -------- ----- -------------- 1 SUP Success EPLDs upgraded. Module 1 EPLD upgrade is successful.
cs2# show version module 1 epld EPLD Device Version ---- -------------------------- -------- MI FPGA 0x7 IO FPGA 0x19 MI FPGA2 0x2 GEM FPGA 0x2 GEM FPGA 0x2 GEM FPGA 0x2 GEM FPGA 0x2
Install the RCF on a Cisco Nexus 9336C- FX2 cluster switch
You can install the RCF after setting up the Nexus 9336C-FX2 cluster switch for the first time. You can also use this procedure to upgrade your RCF version on your cluster switch.
The examples in this procedure use the following switch and node nomenclature:
-
The names of the two Cisco switches are
cs1
andcs2
. -
The node names are
cluster1-01
,cluster1-02
,cluster1-03
, andcluster1-04
. -
The cluster LIF names are
cluster1-01_clus1
,cluster1-01_clus2
,cluster1-02_clus1
,cluster1-02_clus2
,cluster1-03_clus1
,cluster1-03_clus2
,cluster1-04_clus1
, andcluster1-04_clus2
. -
The cluster1::*> prompt indicates the name of the cluster.
The procedure requires the use of both ONTAP commands and Cisco Nexus 9000 Series Switches commands; ONTAP commands are used unless otherwise indicated. |
cluster1::*> network device-discovery show Node/ Local Discovered Protocol Port Device (LLDP: ChassisID) Interface Platform ----------- ------ ------------------------- ---------------- -------- cluster1-01/cdp e3a cs1 Ethernet1/7 N9K-C9336C e0d cs2 Ethernet1/7 N9K-C9336C cluster1-02/cdp e3a cs1 Ethernet1/8 N9K-C9336C e0d cs2 Ethernet1/8 N9K-C9336C cluster1-03/cdp e3a cs1 Ethernet1/1/1 N9K-C9336C e3b cs2 Ethernet1/1/1 N9K-C9336C cluster1-04/cdp e3a cs1 Ethernet1/1/2 N9K-C9336C e3b cs2 Ethernet1/1/2 N9K-C9336C cluster1::*>
cluster1::*> network port show -role cluster Node: cluster1-01 Ignore Speed(Mbps) Health Health Port IPspace Broadcast Domain Link MTU Admin/Oper Status Status --------- ------------ ---------------- ---- ---- ----------- -------- ------ e3a Cluster Cluster up 9000 auto/100000 healthy false e0d Cluster Cluster up 9000 auto/100000 healthy false Node: cluster1-02 Ignore Speed(Mbps) Health Health Port IPspace Broadcast Domain Link MTU Admin/Oper Status Status --------- ------------ ---------------- ---- ---- ----------- -------- ------ e3a Cluster Cluster up 9000 auto/100000 healthy false e0d Cluster Cluster up 9000 auto/100000 healthy false 8 entries were displayed. Node: cluster1-03 Ignore Speed(Mbps) Health Health Port IPspace Broadcast Domain Link MTU Admin/Oper Status Status --------- ------------ ---------------- ---- ---- ------------ -------- ------ e3a Cluster Cluster up 9000 auto/100000 healthy false e3b Cluster Cluster up 9000 auto/100000 healthy false Node: cluster1-04 Ignore Speed(Mbps) Health Health Port IPspace Broadcast Domain Link MTU Admin/Oper Status Status --------- ------------ ---------------- ---- ---- ------------ -------- ------ e0a Cluster Cluster up 9000 auto/100000 healthy false e0b Cluster Cluster up 9000 auto/100000 healthy false cluster1::*>
cluster1::*> network interface show -role cluster Logical Status Network Current Current Is Vserver Interface Admin/Oper Address/Mask Node Port Home ------- ------------------ ---------- --------------- ------------ ------- ---- Cluster cluster1-01_clus1 up/up 169.254.3.4/23 cluster1-01 e3a true cluster1-01_clus2 up/up 169.254.3.5/23 cluster1-01 e0d true cluster1-02_clus1 up/up 169.254.3.8/23 cluster1-02 e3a true cluster1-02_clus2 up/up 169.254.3.9/23 cluster1-02 e0d true cluster1-03_clus1 up/up 169.254.1.3/23 cluster1-03 e3a true cluster1-03_clus2 up/up 169.254.1.1/23 cluster1-03 e3b true cluster1-04_clus1 up/up 169.254.1.6/23 cluster1-04 e3a true cluster1-04_clus2 up/up 169.254.1.7/23 cluster1-04 e3b true 8 entries were displayed. cluster1::*>
cluster1::*> system cluster-switch show -is-monitoring-enabled-operational true Switch Type Address Model --------------------------- ------------------ ---------------- ----- cs1 cluster-network 10.233.205.90 N9K-C9336C Serial Number: FOCXXXXXXGD Is Monitored: true Reason: None Software Version: Cisco Nexus Operating System (NX-OS) Software, Version 9.3(5) Version Source: CDP cs2 cluster-network 10.233.205.91 N9K-C9336C Serial Number: FOCXXXXXXGS Is Monitored: true Reason: None Software Version: Cisco Nexus Operating System (NX-OS) Software, Version 9.3(5) Version Source: CDP cluster1::*>
cluster1::*> network interface modify -vserver Cluster -lif * -auto-revert false
cs2(config)# interface eth1/1/1-2,eth1/7-8 cs2(config-if-range)# shutdown
cluster1::*> network interface show -role cluster Logical Status Network Current Current Is Vserver Interface Admin/Oper Address/Mask Node Port Home ------- ----------------- --------- ---------------- ------------- ------- ---- Cluster cluster1-01_clus1 up/up 169.254.3.4/23 cluster1-01 e3a true cluster1-01_clus2 up/up 169.254.3.5/23 cluster1-01 e3a false cluster1-02_clus1 up/up 169.254.3.8/23 cluster1-02 e3a true cluster1-02_clus2 up/up 169.254.3.9/23 cluster1-02 e3a false cluster1-03_clus1 up/up 169.254.1.3/23 cluster1-03 e3a true cluster1-03_clus2 up/up 169.254.1.1/23 cluster1-03 e3a false cluster1-04_clus1 up/up 169.254.1.6/23 cluster1-04 e3a true cluster1-04_clus2 up/up 169.254.1.7/23 cluster1-04 e3a false 8 entries were displayed. cluster1::*>
cluster1::*> cluster show Node Health Eligibility Epsilon -------------------- ------- ------------ ------- cluster1-01 true true false cluster1-02 true true false cluster1-03 true true true cluster1-04 true true false 4 entries were displayed. cluster1::*>
cs2# write erase Warning: This command will erase the startup-configuration. Do you wish to proceed anyway? (y/n) [n] y cs2# reload This command will reboot the system. (y/n)? [n] y cs2#
-
Perform a basic setup of the switch.
-
Copy the RCF to the bootflash of switch cs2 using one of the following transfer protocols: FTP, TFTP, SFTP, or SCP. For more information about Cisco commands, see the appropriate guide in the Cisco Nexus 9000 Series NX-OS Command Reference guides.
This example shows TFTP being used to copy an RCF to the bootflash on switch cs2.
cs2# copy tftp: bootflash: vrf management Enter source filename: Nexus_9336C_RCF_v1.6-Cluster-HA-Breakout.txt Enter hostname for the tftp server: 172.22.201.50 Trying to connect to tftp server...Connection to Server Established. TFTP get operation was successful Copy complete, now saving to disk (please wait)...
-
Apply the RCF previously downloaded to the bootflash.
For more information about Cisco commands, see the appropriate guide in the Cisco Nexus 9000 Series NX-OS Command Reference guides.This example shows the RCF file
Nexus_9336C_RCF_v1.6-Cluster-HA-Breakout.txt
being installed on switch cs2.
cs2# copy Nexus_9336C_RCF_v1.6-Cluster-HA-Breakout.txt running-config echo-commands
cs2# show banner motd *************************************************************************** * NetApp Reference Configuration File (RCF) * * Switch : Nexus N9K-C9336C-FX2 * Filename : Nexus_9336C_RCF_v1.6-Cluster-HA-Breakout.txt * Date : 10-23-2020 * Version : v1.6 * * Port Usage: * Ports 1- 3: Breakout mode (4x10G) Intra-Cluster Ports, int e1/1/1-4, * e1/2/1-4, e1/3/1-4 * Ports 4- 6: Breakout mode (4x25G) Intra-Cluster/HA Ports, int e1/4/1-4, * e1/5/1-4, e1/6/1-4 * Ports 7-34: 40/100GbE Intra-Cluster/HA Ports, int e1/7-34 * Ports 35-36: Intra-Cluster ISL Ports, int e1/35-36 * * Dynamic breakout commands: * 10G: interface breakout module 1 port <range> map 10g-4x * 25G: interface breakout module 1 port <range> map 25g-4x * * Undo breakout commands and return interfaces to 40/100G configuration in * config mode: * no interface breakout module 1 port <range> map 10g-4x * no interface breakout module 1 port <range> map 25g-4x * interface Ethernet <interfaces taken out of breakout mode> * inherit port-profile 40-100G * priority-flow-control mode auto * service-policy input HA * exit * ***************************************************************************
-
Verify that the RCF file is the correct newer version:
show running-config
When you check the output to verify you have the correct RCF, make sure that the following information is correct:
-
The RCF banner
-
The node and port settings
-
Customizations
The output varies according to your site configuration. Check the port settings and refer to the release notes for any changes specific to the RCF that you have installed.
-
-
After you verify the RCF versions and switch settings are correct, copy the running-config file to the startup-config file.
For more information about Cisco commands, see the appropriate guide in the Cisco Nexus 9000 Series NX-OS Command Reference guides.
cs2# copy running-config startup-config [########################################] 100% Copy complete
cs2# reload This command will reboot the system. (y/n)? [n] y
cs2# copy Nexus_9336C_RCF_v1.6-Cluster-HA-Breakout.txt running-config echo-commands cs2# copy running-config startup-config [########################################] 100% Copy complete
cluster1::*> network port show -role cluster Node: cluster1-01 Ignore Speed(Mbps) Health Health Port IPspace Broadcast Domain Link MTU Admin/Oper Status Status ------- --------- ---------------- ---- ---- ------------ -------- ------ e3a Cluster Cluster up 9000 auto/100000 healthy false e3b Cluster Cluster up 9000 auto/100000 healthy false Node: cluster1-02 Ignore Speed(Mbps) Health Health Port IPspace Broadcast Domain Link MTU Admin/Oper Status Status ------- --------- ---------------- ---- ----- ------------ -------- ------ e3a Cluster Cluster up 9000 auto/100000 healthy false e3b Cluster Cluster up 9000 auto/100000 healthy false Node: cluster1-03 Ignore Speed(Mbps) Health Health Port IPspace Broadcast Domain Link MTU Admin/Oper Status Status ------ ---------- ---------------- ---- ----- ----------- -------- ------ e3a Cluster Cluster up 9000 auto/100000 healthy false e0d Cluster Cluster up 9000 auto/100000 healthy false Node: cluster1-04 Ignore Speed(Mbps) Health Health Port IPspace Broadcast Domain Link MTU Admin/Oper Status Status ------ ---------- ---------------- ---- ----- ----------- -------- ------ e3a Cluster Cluster up 9000 auto/100000 healthy false e0d Cluster Cluster up 9000 auto/100000 healthy false 8 entries were displayed.
-
Verify the switch health from the cluster (this might not show switch cs2, since LIFs are not homed on e0d).
cluster1::*> network device-discovery show -protocol cdp Node/ Local Discovered Protocol Port Device (LLDP: ChassisID) Interface Platform ----------- ------ ------------------------- ----------------- -------- cluster1-01/cdp e3a cs1 Ethernet1/7 N9K-C9336C e0d cs2 Ethernet1/7 N9K-C9336C cluster01-2/cdp e3a cs1 Ethernet1/8 N9K-C9336C e0d cs2 Ethernet1/8 N9K-C9336C cluster01-3/cdp e3a cs1 Ethernet1/1/1 N9K-C9336C e3b cs2 Ethernet1/1/1 N9K-C9336C cluster1-04/cdp e3a cs1 Ethernet1/1/2 N9K-C9336C e3b cs2 Ethernet1/1/2 N9K-C9336C cluster1::*> system cluster-switch show -is-monitoring-enabled-operational true Switch Type Address Model --------------------------- ------------------ ---------------- ----- cs1 cluster-network 10.233.205.90 NX9-C9336C Serial Number: FOCXXXXXXGD Is Monitored: true Reason: None Software Version: Cisco Nexus Operating System (NX-OS) Software, Version 9.3(5) Version Source: CDP cs2 cluster-network 10.233.205.91 NX9-C9336C Serial Number: FOCXXXXXXGS Is Monitored: true Reason: None Software Version: Cisco Nexus Operating System (NX-OS) Software, Version 9.3(5) Version Source: CDP 2 entries were displayed.
You might observe the following output on the cs1 switch console depending on the RCF version previously loaded on the switch. |
2020 Nov 17 16:07:18 cs1 %$ VDC-1 %$ %STP-2-UNBLOCK_CONSIST_PORT: Unblocking port port-channel1 on VLAN0092. Port consistency restored. 2020 Nov 17 16:07:23 cs1 %$ VDC-1 %$ %STP-2-BLOCK_PVID_PEER: Blocking port-channel1 on VLAN0001. Inconsistent peer vlan. 2020 Nov 17 16:07:23 cs1 %$ VDC-1 %$ %STP-2-BLOCK_PVID_LOCAL: Blocking port-channel1 on VLAN0092. Inconsistent local vlan.
cs1(config)# interface eth1/1/1-2,eth1/7-8 cs1(config-if-range)# shutdown
cluster1::*> network interface show -role cluster Logical Status Network Current Current Is Vserver Interface Admin/Oper Address/Mask Node Port Home ------- ------------------ ----------- --------------- ------------ ------- ---- Cluster cluster1-01_clus1 up/up 169.254.3.4/23 cluster1-01 e0d false cluster1-01_clus2 up/up 169.254.3.5/23 cluster1-01 e0d true cluster1-02_clus1 up/up 169.254.3.8/23 cluster1-02 e0d false cluster1-02_clus2 up/up 169.254.3.9/23 cluster1-02 e0d true cluster1-03_clus1 up/up 169.254.1.3/23 cluster1-03 e3b false cluster1-03_clus2 up/up 169.254.1.1/23 cluster1-03 e3b true cluster1-04_clus1 up/up 169.254.1.6/23 cluster1-04 e3b false cluster1-04_clus2 up/up 169.254.1.7/23 cluster1-04 e3b true 8 entries were displayed. cluster1::*>
cluster1::*> cluster show Node Health Eligibility Epsilon -------------------- -------- ------------- ------- cluster1-01 true true false cluster1-02 true true false cluster1-03 true true true cluster1-04 true true false 4 entries were displayed. cluster1::*>
cluster1::*> network interface modify -vserver Cluster -lif * -auto-revert True
cs1# reload This command will reboot the system. (y/n)? [n] y
cs1# show interface brief | grep up . . Eth1/1/1 1 eth access up none 100G(D) -- Eth1/1/2 1 eth access up none 100G(D) -- Eth1/7 1 eth trunk up none 100G(D) -- Eth1/8 1 eth trunk up none 100G(D) -- . .
cs1# show port-channel summary Flags: D - Down P - Up in port-channel (members) I - Individual H - Hot-standby (LACP only) s - Suspended r - Module-removed b - BFD Session Wait S - Switched R - Routed U - Up (port-channel) p - Up in delay-lacp mode (member) M - Not in use. Min-links not met -------------------------------------------------------------------------------- Group Port- Type Protocol Member Ports Channel -------------------------------------------------------------------------------- 1 Po1(SU) Eth LACP Eth1/35(P) Eth1/36(P) cs1#
cluster1::*> network interface show -role cluster Logical Status Network Current Current Is Vserver Interface Admin/Oper Address/Mask Node Port Home ------- ------------------ ---------- ----------------- ----------- ------- ---- Cluster cluster1-01_clus1 up/up 169.254.3.4/23 cluster1-01 e0d true cluster1-01_clus2 up/up 169.254.3.5/23 cluster1-01 e0d true cluster1-02_clus1 up/up 169.254.3.8/23 cluster1-02 e0d true cluster1-02_clus2 up/up 169.254.3.9/23 cluster1-02 e0d true cluster1-03_clus1 up/up 169.254.1.3/23 cluster1-03 e3b true cluster1-03_clus2 up/up 169.254.1.1/23 cluster1-03 e3b true cluster1-04_clus1 up/up 169.254.1.6/23 cluster1-04 e3b true cluster1-04_clus2 up/up 169.254.1.7/23 cluster1-04 e3b true 8 entries were displayed. cluster1::*>
cluster1::*> cluster show Node Health Eligibility Epsilon -------------------- ------- ------------- ------- cluster1-01 true true false cluster1-02 true true false cluster1-03 true true true cluster1-04 true true false 4 entries were displayed. cluster1::*>
cluster1::*> cluster ping-cluster -node local Host is cluster1-03 Getting addresses from network interface table... Cluster cluster1-03_clus1 169.254.1.3 cluster1-03 e3a Cluster cluster1-03_clus2 169.254.1.1 cluster1-03 e3b Cluster cluster1-04_clus1 169.254.1.6 cluster1-04 e3a Cluster cluster1-04_clus2 169.254.1.7 cluster1-04 e3b Cluster cluster1-01_clus1 169.254.3.4 cluster1-01 e3a Cluster cluster1-01_clus2 169.254.3.5 cluster1-01 e0d Cluster cluster1-02_clus1 169.254.3.8 cluster1-02 e3a Cluster cluster1-02_clus2 169.254.3.9 cluster1-02 e0d Local = 169.254.1.3 169.254.1.1 Remote = 169.254.1.6 169.254.1.7 169.254.3.4 169.254.3.5 169.254.3.8 169.254.3.9 Cluster Vserver Id = 4294967293 Ping status: Basic connectivity succeeds on 12 path(s) Basic connectivity fails on 0 path(s) Detected 9000 byte MTU on 12 path(s): Local 169.254.1.3 to Remote 169.254.1.6 Local 169.254.1.3 to Remote 169.254.1.7 Local 169.254.1.3 to Remote 169.254.3.4 Local 169.254.1.3 to Remote 169.254.3.5 Local 169.254.1.3 to Remote 169.254.3.8 Local 169.254.1.3 to Remote 169.254.3.9 Local 169.254.1.1 to Remote 169.254.1.6 Local 169.254.1.1 to Remote 169.254.1.7 Local 169.254.1.1 to Remote 169.254.3.4 Local 169.254.1.1 to Remote 169.254.3.5 Local 169.254.1.1 to Remote 169.254.3.8 Local 169.254.1.1 to Remote 169.254.3.9 Larger than PMTU communication succeeds on 12 path(s) RPC status: 6 paths up, 0 paths down (tcp check) 6 paths up, 0 paths down (udp check)
Install the RCF on a Cisco Nexus 9336C-FX2 storage switch
The reference configuration files (RCFs) can be upgraded on Cisco Nexus 9336C-FX2 storage switches.
The following conditions must exist before you upgrade the RCF on the storage switch:
-
The switch must be fully functioning (there should be no errors in the logs or similar issues).
-
You must have checked or set your desired boot variables in the RCF to reflect the desired boot images if you are installing only NX-OS and keeping your current RCF version.
-
If you need to change the boot variables to reflect the current boot images, you must do so before reapplying the RCF so that the correct version is instantiated on future reboots.
-
You must have referred to the appropriate software and upgrade guides available on the Cisco web site for complete documentation on the Cisco storage upgrade and downgrade procedures. See Cisco Nexus 9000 Series Switches for more information.
-
The number of 100 GbE ports are defined in the reference configuration files (RCFs) available on the Cisco Ethernet switches page.
Procedure summary
-
Check the health status of switches and ports (steps 1-4)
-
Download the NX-OS image to Cisco switch st2 and reboot (steps 5-8)
-
Copy the RCF to Cisco switch st2 (steps 9-12)
-
Recheck the health status of switches and ports (steps 13-15)
-
Repeat steps 1-15 for Cisco switch st1.
The command outputs might vary depending on different releases of ONTAP. |
The examples in this procedure use the following switch and node nomenclature:
-
The names of the two storage switches are st1 and st2.
-
The nodes are node1 and node2.
The procedure requires the use of both ONTAP commands and Cisco Nexus 9000 Series Switches commands; ONTAP commands are used unless otherwise indicated. |
-
If AutoSupport is enabled on this cluster, suppress automatic case creation by invoking an AutoSupport message:
system node autosupport invoke -node * -type all - message MAINT=xh
Where x is the duration of the maintenance window in hours.
-
Check that the storage switches are available:
system switch ethernet show
storage::*> system switch ethernet show Switch Type Address Model ------------------------- ------------------ ---------------- --------------- st1 storage-network 172.17.227.5 NX9-C9336C Serial Number: FOC221206C2 Is Monitored: true Reason: None Software Version: Cisco Nexus Operating System (NX-OS) Software, Version 9.3(5) Version Source: CDP st2 storage-network 172.17.227.6 NX9-C9336C Serial Number: FOC220443LZ Is Monitored: true Reason: None Software Version: Cisco Nexus Operating System (NX-OS) Software, Version 9.3(5) Version Source: CDP 2 entries were displayed. storage::*>
storage::*> storage port show -port-type ENET Speed VLAN Node Port Type Mode (Gb/s) State Status ID ------- ------- ------- ---------- ---------- --------- ----------- ----- node1 e3a ENET storage 100 enabled online 30 e3b ENET storage 0 enabled offline 30 e7a ENET storage 0 enabled offline 30 e7b ENET storage 100 enabled online 30 node2 e3a ENET storage 100 enabled online 30 e3b ENET storage 0 enabled offline 30 e7a ENET storage 0 enabled offline 30 e7b ENET storage 100 enabled online 30
storage::*> system health alert show -instance There are no entries matching your query.
st2# install all nxos bootflash:nxos.9.3. 5.bin Installer will perform compatibility check first. Please wait. Installer is forced disruptive Verifying image bootflash:/nxos.9.3.4.bin for boot variable "nxos". [####################] 100% -- SUCCESS Verifying image type. [[####################] 100% -- SUCCESS Preparing "nxos" version info using image bootflash:/nxos.9.3.4.bin. [####################] 100% -- SUCCESS Preparing "bios" version info using image bootflash:/nxos.9.3.4.bin. [####################] 100% -- SUCCESS Performing module support checks. [####################] 100% -- SUCCESS Notifying services about system upgrade. [####################] 100% -- SUCCESS Compatibility check is done: Module bootable Impact Install-type Reason ------ -------- ----- -------- ------------ ---- -- 1 yes disruptive reset default upgrade is not hitless Images will be upgraded according to following table: Module Image Running-Version(pri:alt) New-Version Upg Required ------ -------- --------------------------------------- ------------ -------- 1 nxos 9.3(3) 9.3(4) yes 1 bios v08.37(01/28/2020):v08.23(09/23/2015) v08.38(05/29/2020) no Switch will be reloaded for disruptive upgrade. Do you want to continue with the installation (y/n)? [n] y input string too long Do you want to continue with the installation (y/n)? [n] y Install is in progress, please wait. Performing runtime checks. [####################] 100% -- SUCCESS Setting boot variables. [####################] 100% -- SUCCESS Performing configuration copy. [####################] 100% -- SUCCESS Module 1: Refreshing compact flash and upgrading bios/loader/bootrom. Warning: please do not remove or power off the module at this time. [####################] 100% -- SUCCESS Finishing the upgrade, switch will reboot in 10 seconds. st2#
st2# copy running-config startup-config [########################################] 100% Copy complete. st2# reload This command will reboot the system. (y/n)? [n] y
st2# show version Cisco Nexus Operating System (NX-OS) Software TAC support: http://www.cisco.com/tac Upgrading a Cisco Nexus 9336C Storage Switch 6 Upgrading a Cisco Nexus 9336C storage switch Copyright (C) 2002-2020, Cisco and/or its affiliates. All rights reserved. The copyrights to certain works contained in this software are owned by other third parties and used and distributed under their own licenses, such as open source. This software is provided "as is," and unless otherwise stated, there is no warranty, express or implied, including but not limited to warranties of merchantability and fitness for a particular purpose. Certain components of this software are licensed under the GNU General Public License (GPL) version 2.0 or GNU General Public License (GPL) version 3.0 or the GNU Lesser General Public License (LGPL) Version 2.1 or Lesser General Public License (LGPL) Version 2.0. A copy of each such license is available at http://www.opensource.org/licenses/gpl-2.0.php and http://opensource.org/licenses/gpl-3.0.html and http://www.opensource.org/licenses/lgpl-2.1.php and . Software BIOS: version 08.38 NXOS: version 9.3(5) BIOS compile time: 05/29/2020 NXOS image file is: bootflash:///nxos.9.3. 5.bin NXOS compile time: 4/28/2020 21:00:00 [04/29/2020 02:28:31] Hardware cisco Nexus9000 C9336C Chassis (Nexus 9000 Series) Intel(R) Xeon(R) CPU E5-2403 v2 @ 1.80GHz with 8154432 kB of memory. Processor Board ID FOC20291J6K Device name: S2 bootflash: 53298520 kB Kernel uptime is 0 day(s), 0 hour(s), 3 minute(s), 42 second(s) Last reset at 157524 usecs after Mon Nov 2 18:32:06 2020 Reason: Reset due to upgrade System version: 9.3(5) Service: plugin Core Plugin, Ethernet Plugin Active Package(s): st2#
-
Copy the RCF on switch st2 to the switch bootflash using one of the following transfer protocols: FTP, HTTP, TFTP, SFTP, or SCP.
For more information about Cisco commands, see the appropriate guide in the Cisco Nexus 9000 Series NX-OS Command Reference guides.
The following example shows HTTP being used to copy an RCF to the bootflash on switch st2:
st2# copy http://172.16.10.1//cfg/Nexus_9336C_RCF_v1.6-Storage.txt bootflash: vrf management % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 3254 100 3254 0 0 8175 0 --:--:-- --:--:-- --:--:– 8301 Copy complete, now saving to disk (please wait)... Copy complete. st2#
st2# copy Nexus_9336C_RCF_v1.6-Storage.txt running-config echo-commands
-
Verify that the RCF file is the correct newer version:
show running-config
When you check the output to verify you have the correct RCF, make sure that the following information is correct:
-
The RCF banner
-
The node and port settings
-
Customizations
The output varies according to your site configuration. Check the port settings and refer to the release notes for any changes specific to the RCF that you have installed.
Important: In the banner output from the
show banner motd
command, you must read and follow the instructions in the *IMPORTANT NOTES *section to ensure the proper configuration and operation of the switch.
-
st2# show banner motd ****************************************************************************** *NetApp Reference Configuration File (RCF) * *Switch : Nexus N9K-C9336C-FX2 *Filename : Nexus_9336C_RCF_v1.6-Storage.txt * Date : 10-23-2020 *Version : v1.6 * *Port Usage: Storage configuration *Ports 1-36: 100GbE Controller and Shelf Storage Ports * *IMPORTANT NOTES* *- This RCF utilizes QoS and requires TCAM re-configuration, requiring RCF *to be loaded twice with the Storage Switch rebooted in between. * *- Perform the following 4 steps to ensure proper RCF installation: * *(1) Apply RCF first time, expect following messages: *- Please save config and reload the system... *- Edge port type (portfast) should only be enabled on ports... *- TCAM region is not configured for feature QoS class IPv4 ingress... * *(2) Save running-configuration and reboot Cluster Switch * *(3) After reboot, apply same RCF second time and expect following messages: *- % Invalid command at '^' marker *- Syntax error while parsing... * *(4) Save running-configuration again ****************************************************************************** st2#
-
After you verify that the software versions and switch settings are correct, copy the running-config file to the startup-config file on switch st2.
For more information on Cisco commands, see the appropriate guide in the Cisco Nexus 9000 Series NX-OS Command Reference guides.
The following example shows the
running-config
file successfully copied to thestartup-config
file:
st2# copy running-config startup-config [########################################] 100% Copy complete.
storage::*> system switch ethernet show Switch Type Address Model ---------------------------- ------------------ ---------------- --------------- st1 storage-network 172.17.227.5 NX9-C9336C Serial Number: FOC221206C2 Is Monitored: true Reason: None Software Version: Cisco Nexus Operating System (NX-OS) Software, Version 9.3(5) Version Source: CDP st2 storage-network 172.17.227.6 NX9-C9336C Serial Number: FOC220443LZ Is Monitored: true Reason: None Software Version: Cisco Nexus Operating System (NX-OS) Software, Version 9.3(5) Version Source: CDP 2 entries were displayed. storage::*
storage::*> storage port show -port-type ENET Speed VLAN Node Port Type Mode (Gb/s) State Status ID ------- ------- ------- ---------- ----------- --------- --------- ----- node1 e3a ENET storage 100 enabled online 30 e3b ENET storage 0 enabled offline 30 e7a ENET storage 0 enabled offline 30 e7b ENET storage 100 enabled online 30 node2 e3a ENET storage 100 enabled online 30 e3b ENET storage 0 enabled offline 30 e7a ENET storage 0 enabled offline 30 e7b ENET storage 100 enabled online 30
storage::*> system health alert show -instance There are no entries matching your query.
Install the RCF on a Cisco Nexus 9336C-FX2 shared switch
From ONTAP 9.9.1, you can use Cisco Nexus 9336C-FX2 switches to combine storage and cluster functionality into a shared switch scenario.
-
The cluster switches must be fully functioning (there should be no errors in the logs or similar issues).
-
The storage switches must be fully functioning (there should be no errors in the logs or similar issues).
-
The names of the two storage switches are sh1 and sh2.
-
The example used here loads the shared RCF on to the new switch.
-
Copy the RCF on switch sh2 to the switch bootflash using one of the following transfer protocols: FTP, HTTP, TFTP, SFTP, or SCP.
For more information on Cisco commands, see the appropriate guide in the Cisco Nexus 9000 Series NX-OS Command Reference guides.
The following example shows HTTP being used to copy an RCF to the bootflash on switch sh2:
sh2# copy http://172.16.10.1//cfg/Nexus_9336C_RCF_v1.7-Cluster-Ha-Storage.txt bootflash: vrf management % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 5143 100 5143 0 0 11300 0 --:--:-- --:--:-- --:--:– 11300 Copy complete, now saving to disk (please wait)... Copy complete. sh2#
sh2# copy Nexus_9336C_RCF_v1.7-Cluster-HA-Storage.txt running-config echo-commands
-
Verify that the RCF file is the correct newer version:
show running-config
When you check the output to verify you have the correct RCF, make sure that the following information is correct:
-
The RCF banner
-
The node and port settings
-
Customizations
The output varies according to your site configuration. Check the port settings and refer to the release notes for any changes specific to the RCF that you have installed.
Important: In the banner output from the
show banner motd
command, you must read and follow the instructions in the *IMPORTANT NOTES *section to ensure the proper configuration and operation of the switch.
-
sh2# show banner motd ****************************************************************************** *NetApp Reference Configuration File (RCF) * *Switch : Nexus N9K-C9336C-FX2 *Filename : Nexus_9336C_RCF_v1.7-Cluster-HA-Storage.txt * Date : Jan-08-2021 *Version : v1.7 * *Port Usage: *Ports 1-8: 40/100GbE Intra-Cluster/HA Ports, int e1/1-8 *Port 9: 10GbE breakout Intra-Cluster Ports, int e1/9/1-4 *Port 10: 25GbE breakout Intra-Cluster/HA Ports, int e1/10/1-4 *Ports 11-22: First HA-pair Controller and Shelf Storage Ports, int e1/11-22 *Ports 23-34: Second HA-pair Controller and Shelf Storage Ports, int e1/23-34 *Ports 35-36: Intra-Cluster ISL Ports, int e1/35-36 * * Undo breakout commands and return interfaces to 40/100G configuration in * config mode: * no interface breakout module 1 port 9 map 10g-4x * no interface breakout module 1 port 10 map 25g-4x * interface Ethernet 1/9-10 * inherit port-profile CLUSTER_HA * priority-flow-control mode auto * service-policy type qos input HA_POLICY * exit * *IMPORTANT NOTES* * In certain conditions, N9K-C9336C-FX2 may not be able to auto-negotiate port * speed correctly, and port speed must be manually set, in config mode, e.g. * int e1/1 * speed 40000 * int e1/3 * speed 100000 * ****************************************************************************** sh2#
-
After you verify that the software versions and switch settings are correct, copy the
running-config
file to the startup-config file on switch sh2.For more information on Cisco commands, see the appropriate guide in the Cisco Nexus 9000 Series NX-OS Command Reference guides.
The following example shows the
running-config
file successfully copied to thestartup-config
file:
sh2# copy running-config startup-config [########################################] 100% Copy complete.