Skip to main content
ONTAP MetroCluster

Configuring Broadcom IP switches

Contributors netapp-aoife netapp-aaron-holt netapp-folivia netapp-pcarriga
PDFs

You must configure the Broadcom IP switches for use as the cluster interconnect and for backend MetroCluster IP connectivity.

Note Your configuration requires additional licenses (6 x 100-Gb port license) in the following scenarios:

  • You use ports 53 and 54 as a 40-Gbps or 100-Gbps MetroCluster ISL.

  • You use a platform that connects the local cluster and MetroCluster interfaces to ports 49 - 52.

Resetting the Broadcom IP switch to factory defaults

Before installing a new switch software version and RCFs, you must erase the Broadcom switch settings and perform basic configuration.

About this task

  • You must repeat these steps on each of the IP switches in the MetroCluster IP configuration.

  • You must be connected to the switch using the serial console.

  • This task resets the configuration of the management network.

Steps

  1. Change to the elevated command prompt (#): enable

    (IP_switch_A_1)> enable
(IP_switch_A_1) #

  2. Erase the startup configuration and remove the banner

    1. Erase the startup configuration:

      erase startup-config

      (IP_switch_A_1) #erase startup-config

Are you sure you want to clear the configuration? (y/n) y

(IP_switch_A_1) #

      This command does not erase the banner.

    2. Remove the banner:

      no set clibanner

      (IP_switch_A_1) #configure
(IP_switch_A_1)(Config) # no set clibanner
(IP_switch_A_1)(Config) #

  3. Reboot the switch: (IP_switch_A_1) #reload

    Are you sure you would like to reset the system? (y/n) y
    Note If the system asks whether to save the unsaved or changed configuration before reloading the switch, select No.

  4. Wait for the switch to reload, and then log in to the switch.

    The default user is “admin”, and no password is set. A prompt similar to the following is displayed:

    (Routing)>

  5. Change to the elevated command prompt:

    enable

    Routing)> enable
(Routing) #

  6. Set the service port protocol to none:

    serviceport protocol none

    (Routing) #serviceport protocol none
Changing protocol mode will reset ip configuration.
Are you sure you want to continue? (y/n) y

(Routing) #

  7. Assign the IP address to the service port:

    serviceport ip ip-address netmask gateway

    The following example shows a service port assigned IP address "10.10.10.10" with subnet "255.255.255.0" and gateway "10.10.10.1":

    (Routing) #serviceport ip 10.10.10.10 255.255.255.0 10.10.10.1

  8. Verify that the service port is correctly configured:

    show serviceport

    The following example shows that the port is up and the correct addresses have been assigned:

    (Routing) #show serviceport

Interface Status............................... Up
IP Address..................................... 10.10.10.10
Subnet Mask.................................... 255.255.255.0
Default Gateway................................ 10.10.10.1
IPv6 Administrative Mode....................... Enabled
IPv6 Prefix is ................................ fe80::dac4:97ff:fe56:87d7/64
IPv6 Default Router............................ fe80::222:bdff:fef8:19ff
Configured IPv4 Protocol....................... None
Configured IPv6 Protocol....................... None
IPv6 AutoConfig Mode........................... Disabled
Burned In MAC Address.......................... D8:C4:97:56:87:D7

(Routing) #

  9. If desired, configure the SSH server.

    Note The RCF file disables the Telnet protocol. If you do not configure the SSH server, you can only access the bridge using the serial port connection.

    1. Generate RSA keys.

      (Routing) #configure
(Routing) (Config)#crypto key generate rsa

    2. Generate DSA keys (optional)

      (Routing) #configure
(Routing) (Config)#crypto key generate dsa

    3. If you are using the FIPS compliant version of EFOS, generate the ECDSA keys. The following example creates the keys with a length of 521. Valid values are 256, 384 or 521.

      (Routing) #configure
(Routing) (Config)#crypto key generate ecdsa 521

    4. Enable the SSH server.

      If necessary, exit the configuration context.

      (Routing) (Config)#end
(Routing) #ip ssh server enable
      Note If keys already exist, then you might be asked to overwrite them.

  10. If desired, configure the domain and name server:

    configure

    The following example shows the ip domain and ip name server commands:

    (Routing) # configure
(Routing) (Config)#ip domain name lab.netapp.com
(Routing) (Config)#ip name server 10.99.99.1 10.99.99.2
(Routing) (Config)#exit
(Routing) (Config)#

  11. If desired, configure the time zone and time synchronization (SNTP).

    The following example shows the sntp commands, specifying the IP address of the SNTP server and the relative time zone.

    (Routing) #
(Routing) (Config)#sntp client mode unicast
(Routing) (Config)#sntp server 10.99.99.5
(Routing) (Config)#clock timezone -7
(Routing) (Config)#exit
(Routing) (Config)#

    For EFOS version 3.10.0.3 and later, use the ntp command, as shown in the following example:

    > (Config)# ntp ?

authenticate             Enables NTP authentication.
authentication-key       Configure NTP authentication key.
broadcast                Enables NTP broadcast mode.
broadcastdelay           Configure NTP broadcast delay in microseconds.
server                   Configure NTP server.
source-interface         Configure the NTP source-interface.
trusted-key              Configure NTP authentication key number for trusted time source.
vrf                      Configure the NTP VRF.

>(Config)# ntp server ?

ip-address|ipv6-address|hostname  Enter a valid IPv4/IPv6 address or hostname.

>(Config)# ntp server 10.99.99.5

  12. Configure the switch name:

    hostname IP_switch_A_1

    The switch prompt will display the new name:

    (Routing) # hostname IP_switch_A_1

(IP_switch_A_1) #

  13. Save the configuration:

    write memory

    You receive prompts and output similar to the following example:

    (IP_switch_A_1) #write memory

This operation may take a few minutes.
Management interfaces will not be available during this time.

Are you sure you want to save? (y/n) y

Config file 'startup-config' created successfully .


Configuration Saved!

(IP_switch_A_1) #

  14. Repeat the previous steps on the other three switches in the MetroCluster IP configuration.

Downloading and installing the Broadcom switch EFOS software

You must download the switch operating system file and RCF file to each switch in the MetroCluster IP configuration.

About this task

This task must be repeated on each switch in the MetroCluster IP configuration.

Note the following:

  • When upgrading from EFOS 3.4.x.x to EFOS 3.7.x.x or later, the switch must be running EFOS 3.4.4.6 (or later 3.4.x.x release). If you are running a release prior to that, then upgrade the switch to EFOS 3.4.4.6 (or later 3.4.x.x release) first, then upgrade the switch to EFOS 3.7.x.x or later.

  • The configuration for EFOS 3.4.x.x and 3.7.x.x or later are different. Changing the EFOS version from 3.4.x.x to 3.7.x.x or later, or vice versa, requires the switch to be reset to factory defaults and the RCF files for the corresponding EFOS version to be (re)applied. This procedure requires access through the serial console port.

  • Beginning with EFOS version 3.7.x.x or later, a non-FIPS compliant and a FIPS compliant version is available. Different steps apply when moving to from a non-FIPS compliant to a FIPS compliant version or vice versa. Changing EFOS from a non-FIPS compliant to a FIPS compliant version or vice versa will reset the switch to factory defaults. This procedure requires access through the serial console port.

Steps

  1. Download the switch firmware from the Broadcom support site.

  2. Check if your version of EFOS is FIPS compliant or non-FIPS compliant by using the show fips status command. In the following examples, IP_switch_A_1 is using FIPS compliant EFOS and IP_switch_A_2 is using non-FIPS compliant EFOS.

    Example 1

    IP_switch_A_1 #show fips status

System running in FIPS mode

IP_switch_A_1 #

    Example 2

    IP_switch_A_2 #show fips status
                     ^
% Invalid input detected at `^` marker.

IP_switch_A_2 #

  3. Use the following table to determine which method you must follow:

    Procedure

    Current EFOS version

    New EFOS version

    High level steps

    Steps to upgrade EFOS between two (non) FIPS compliant versions

    3.4.x.x

    3.4.x.x

    Install the new EFOS image using method 1) The configuration and license information is retained

    3.4.4.6 (or later 3.4.x.x)

    3.7.x.x or later non-FIPS compliant

    Upgrade EFOS using method 1. Reset the switch to factory defaults and apply the RCF file for EFOS 3.7.x.x or later

    3.7.x.x or later non-FIPS compliant

    3.4.4.6 (or later 3.4.x.x)

    Downgrade EFOS using method 1. Reset the switch to factory defaults and apply the RCF file for EFOS 3.4.x.x

    3.7.x.x or later non-FIPS compliant

    Install the new EFOS image using method 1. The configuration and license information is retained

    3.7.x.x or later FIPS compliant

    3.7.x.x or later FIPS compliant

    Install the new EFOS image using method 1. The configuration and license information is retained

    Steps to upgrade to/from a FIPS compliant EFOS version

    Non-FIPS compliant

    FIPS compliant

    Installation of the EFOS image using method 2. The switch configuration and license information will be lost.

    FIPS compliant

    Non-FIPS compliant

Steps to upgrade EFOS with downloading the software image to the backup boot partition

You can perform the following steps only if both EFOS versions are non-FIPS compliant or both EFOS versions are FIPS compliant.

Note Do not use these steps if one version is FIPS compliant and the other version is non-FIPS compliant.
Steps

  1. Copy the switch software to the switch: copy sftp://user@50.50.50.50/switchsoftware/efos-3.4.4.6.stk backup

    In this example, the efos-3.4.4.6.stk operating system file is copied from the SFTP server at 50.50.50.50 to the backup partition. You need to use the IP address of your TFTP/SFTP server and the file name of the RCF file that you need to install.

    (IP_switch_A_1) #copy sftp://user@50.50.50.50/switchsoftware/efos-3.4.4.6.stk backup
Remote Password:*************

Mode........................................... SFTP
Set Server IP.................................. 50.50.50.50
Path........................................... /switchsoftware/
Filename....................................... efos-3.4.4.6.stk
Data Type...................................... Code
Destination Filename........................... backup

Management access will be blocked for the duration of the transfer
Are you sure you want to start? (y/n) y

File transfer in progress. Management access will be blocked for the duration of the transfer. Please wait...
SFTP Code transfer starting...


File transfer operation completed successfully.

(IP_switch_A_1) #

  2. Set the switch to boot from the backup partition on the next switch reboot:

    boot system backup

    (IP_switch_A_1) #boot system backup
Activating image backup ..

(IP_switch_A_1) #

  3. Verify that the new boot image will be active on the next boot:

    show bootvar

    (IP_switch_A_1) #show bootvar

Image Descriptions

 active :
 backup :


 Images currently available on Flash

 ----  -----------  --------  ---------------  ------------
 unit       active    backup   current-active   next-active
 ----  -----------  --------  ---------------  ------------

	1       3.4.4.2    3.4.4.6      3.4.4.2        3.4.4.6

(IP_switch_A_1) #

  4. Save the configuration:

    write memory

    (IP_switch_A_1) #write memory

This operation may take a few minutes.
Management interfaces will not be available during this time.

Are you sure you want to save? (y/n) y


Configuration Saved!

(IP_switch_A_1) #

  5. Reboot the switch:

    reload

    (IP_switch_A_1) #reload

Are you sure you would like to reset the system? (y/n) y

  6. Wait for the switch to reboot.

    Note In rare scenarios the switch may fail to boot. Follow the Steps to upgrade EFOS using the ONIE OS installation to install the new image.

  7. If you change the switch from EFOS 3.4.x.x to EFOS 3.7.x.x or vice versa then follow the following two procedures to apply the correct configuration (RCF):

    1. Resetting the Broadcom IP switch to factory defaults

    2. Downloading and installing the Broadcom RCF files

  8. Repeat these steps on the remaining three IP switches in the MetroCluster IP configuration.

Steps to upgrade EFOS using the ONIE OS installation

You can perform the following steps if one EFOS version is FIPS compliant and the other EFOS version is non-FIPS compliant. These steps can be used to install the non-FIPS or FIPS compliant EFOS 3.7.x.x image from ONIE if the switch fails to boot.

Steps

  1. Boot the switch into ONIE installation mode.

    During boot, select ONIE when the following screen appears:

     +--------------------------------------------------------------------+
 |EFOS                                                                |
 |*ONIE                                                               |
 |                                                                    |
 |                                                                    |
 |                                                                    |
 |                                                                    |
 |                                                                    |
 |                                                                    |
 |                                                                    |
 |                                                                    |
 |                                                                    |
 |                                                                    |
 +--------------------------------------------------------------------+

    After selecting "ONIE", the switch will then load and present you with the following choices:

     +--------------------------------------------------------------------+
 |*ONIE: Install OS                                                   |
 | ONIE: Rescue                                                       |
 | ONIE: Uninstall OS                                                 |
 | ONIE: Update ONIE                                                  |
 | ONIE: Embed ONIE                                                   |
 | DIAG: Diagnostic Mode                                              |
 | DIAG: Burn-In Mode                                                 |
 |                                                                    |
 |                                                                    |
 |                                                                    |
 |                                                                    |
 |                                                                    |
 +--------------------------------------------------------------------+

    The switch now will boot into ONIE installation mode.

  2. Stop the ONIE discovery and configure the ethernet interface

    Once the following message appears press <enter> to invoke the ONIE console:

     Please press Enter to activate this console. Info: eth0:  Checking link... up.
 ONIE:/ #
    Note The ONIE discovery will continue and messages will be printed to the console. 
    Stop the ONIE discovery
ONIE:/ # onie-discovery-stop
discover: installer mode detected.
Stopping: discover... done.
ONIE:/ #

  3. Configure the ethernet interface and add the route using ifconfig eth0 <ipAddress> netmask <netmask> up and route add default gw <gatewayAddress>

    ONIE:/ # ifconfig eth0 10.10.10.10 netmask 255.255.255.0 up
ONIE:/ # route add default gw 10.10.10.1

  4. Verify that the server hosting the ONIE installation file is reachable:

    ONIE:/ # ping 50.50.50.50
PING 50.50.50.50 (50.50.50.50): 56 data bytes
64 bytes from 50.50.50.50: seq=0 ttl=255 time=0.429 ms
64 bytes from 50.50.50.50: seq=1 ttl=255 time=0.595 ms
64 bytes from 50.50.50.50: seq=2 ttl=255 time=0.369 ms
^C
--- 50.50.50.50 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.369/0.464/0.595 ms
ONIE:/ #

  5. Install the new switch software

    ONIE:/ # onie-nos-install http:// 50.50.50.50/Software/onie-installer-x86_64
discover: installer mode detected.
Stopping: discover... done.
Info: Fetching http:// 50.50.50.50/Software/onie-installer-3.7.0.4 ...
Connecting to 50.50.50.50 (50.50.50.50:80)
installer            100% |*******************************| 48841k  0:00:00 ETA
ONIE: Executing installer: http:// 50.50.50.50/Software/onie-installer-3.7.0.4
Verifying image checksum ... OK.
Preparing image archive ... OK.

    The software will install and then reboot the switch. Let the switch reboot normally into the new EFOS version.

  6. Verify that the new switch software is installed

    show bootvar

    (Routing) #show bootvar
Image Descriptions
active :
backup :
Images currently available on Flash
---- 	----------- -------- --------------- ------------
unit 	active 	   backup   current-active  next-active
---- 	----------- -------- --------------- ------------
1 	3.7.0.4     3.7.0.4  3.7.0.4         3.7.0.4
(Routing) #

  7. Complete the installation

    The switch will reboot with no configuration applied and reset to factory defaults. Follow the two procedures to configure the switch basic settings and apply the RCF file as outlined in the following two documents:

    1. Configure the switch basic settings. Follow step 4 and later: Resetting the Broadcom IP switch to factory defaults

    2. Create and apply the RCF file as outlined in Downloading and installing the Broadcom RCF files

Downloading and installing the Broadcom RCF files

You must generate and install the switch RCF file to each switch in the MetroCluster IP configuration.

Before you begin

This task requires file transfer software, such as FTP, TFTP, SFTP, or SCP, to copy the files to the switches.

About this task

These steps must be repeated on each of the IP switches in the MetroCluster IP configuration.

There are four RCF files, one for each of the four switches in the MetroCluster IP configuration. You must use the correct RCF files for the switch model you are using.

Switch

RCF file

IP_switch_A_1

v1.32_Switch-A1.txt

IP_switch_A_2

v1.32_Switch-A2.txt

IP_switch_B_1

v1.32_Switch-B1.txt

IP_switch_B_2

v1.32_Switch-B2.txt
Note The RCF files for EFOS version 3.4.4.6 or later 3.4.x.x. release and EFOS version 3.7.0.4 are different. You need to make sure that you have created the correct RCF files for the EFOS version that the switch is running.

EFOS version

RCF file version

3.4.x.x

v1.3x, v1.4x

3.7.x.x

v2.x
Steps

  1. Generate the Broadcom RCF files for MetroCluster IP.

    1. Download the RcfFileGenerator for MetroCluster IP

    2. Generate the RCF file for your configuration using the RcfFileGenerator for MetroCluster IP.

      Note Modifications to the RCF files after download are not supported.

  2. Copy the RCF files to the switches:

    1. Copy the RCF files to the first switch: copy sftp://user@FTP-server-IP-address/RcfFiles/switch-specific-RCF/BES-53248_v1.32_Switch-A1.txt nvram:script BES-53248_v1.32_Switch-A1.scr

      In this example, the "BES-53248_v1.32_Switch-A1.txt" RCF file is copied from the SFTP server at "50.50.50.50" to the local bootflash. You need to use the IP address of your TFTP/SFTP server and the file name of the RCF file that you need to install.

      (IP_switch_A_1) #copy sftp://user@50.50.50.50/RcfFiles/BES-53248_v1.32_Switch-A1.txt nvram:script BES-53248_v1.32_Switch-A1.scr

Remote Password:*************

Mode........................................... SFTP
Set Server IP.................................. 50.50.50.50
Path........................................... /RcfFiles/
Filename....................................... BES-53248_v1.32_Switch-A1.txt
Data Type...................................... Config Script
Destination Filename........................... BES-53248_v1.32_Switch-A1.scr

Management access will be blocked for the duration of the transfer
Are you sure you want to start? (y/n) y

File transfer in progress. Management access will be blocked for the duration of the transfer. Please wait...
File transfer operation completed successfully.


Validating configuration script...

config

set clibanner "***************************************************************************

* NetApp Reference Configuration File (RCF)

*

* Switch    : BES-53248


...
The downloaded RCF is validated. Some output is being logged here.
...


Configuration script validated.
File transfer operation completed successfully.

(IP_switch_A_1) #

    2. Verify that the RCF file is saved as a script:

      script list

      (IP_switch_A_1) #script list

Configuration Script Name        Size(Bytes)  Date of Modification
-------------------------------  -----------  --------------------
BES-53248_v1.32_Switch-A1.scr             852   2019 01 29 18:41:25

1 configuration script(s) found.
2046 Kbytes free.
(IP_switch_A_1) #

    3. Apply the RCF script:

      script apply BES-53248_v1.32_Switch-A1.scr

      (IP_switch_A_1) #script apply BES-53248_v1.32_Switch-A1.scr

Are you sure you want to apply the configuration script? (y/n) y


config

set clibanner "********************************************************************************

* NetApp Reference Configuration File (RCF)

*

* Switch    : BES-53248

...
The downloaded RCF is validated. Some output is being logged here.
...

Configuration script 'BES-53248_v1.32_Switch-A1.scr' applied.

(IP_switch_A_1) #

    4. Save the configuration:

      write memory

      (IP_switch_A_1) #write memory

This operation may take a few minutes.
Management interfaces will not be available during this time.

Are you sure you want to save? (y/n) y


Configuration Saved!

(IP_switch_A_1) #

    5. Reboot the switch:

      reload

      (IP_switch_A_1) #reload

Are you sure you would like to reset the system? (y/n) y

    6. Repeat the previous steps for each of the other three switches, being sure to copy the matching RCF file to the corresponding switch.

  3. Reload the switch:

    reload

    IP_switch_A_1# reload

  4. Repeat the previous steps on the other three switches in the MetroCluster IP configuration.

Disable unused ISL ports and port channels

NetApp recommends disabling unused ISL ports and port channels to avoid unnecessary health alerts.

  1. Identify the unused ISL ports and port channels using the RCF file banner:

    Note If the port is in breakout mode, the port name you specify in the command might be different than the name stated in the RCF banner. You can also use the RCF cabling files to find the port name.
    For ISL port details

    Run the command show port all.

    For port channel details

    Run the command show port-channel all.

  2. Disable the unused ISL ports and port channels.

    You must run the following commands for each identified unused port or port channel.

    (SwtichA_1)> enable
(SwtichA_1)# configure
(SwtichA_1)(Config)# <port_name>
(SwtichA_1)(Interface 0/15)# shutdown
(SwtichA_1)(Interface 0/15)# end
(SwtichA_1)# write memory