Skip to main content
Cluster and storage switches

Migrate to a switched NetApp cluster environment

Contributors netapp-yvonneo netapp-jolieg
PDFs

If you have an existing two-node switchless cluster environment, you can migrate to a two-node switched cluster environment using Broadcom-supported BES-53248 cluster switches, which enables you to scale beyond two nodes in the cluster.

The migration process works for all cluster node ports using optical or Twinax ports, but it is not supported on this switch if nodes are using onboard 10GBASE-T RJ45 ports for the cluster network ports.

Review requirements

Review the following requirements for the cluster environment.

  • Be aware that most systems require two dedicated cluster-network ports on each controller.

  • Make sure that the BES-53248 cluster switch is set up as described in Replace requirements before starting this migration process.

  • For the two-node switchless configuration, ensure that:

    • The two-node switchless configuration is properly set up and functioning.

    • The nodes are running ONTAP 9.5P8 and later. Support for 40/100 GbE cluster ports starts with EFOS firmware version 3.4.4.6 and later.

    • All cluster ports are in the up state.

    • All cluster logical interfaces (LIFs) are in the up state and on their home ports.

  • For the Broadcom-supported BES-53248 cluster switch configuration, ensure that:

    • The BES-53248 cluster switch is fully functional on both switches.

    • Both switches have management network connectivity.

    • There is console access to the cluster switches.

    • BES-53248 node-to-node switch and switch-to-switch connections are using Twinax or fiber cables.

      The NetApp Hardware Universe contains information about ONTAP compatibility, supported EFOS firmware, and cabling to BES-53248 switches.

  • Inter-Switch Link (ISL) cables are connected to ports 0/55 and 0/56 on both BES-53248 switches.

  • Initial customization of both the BES-53248 switches is complete, so that:

    • BES-53248 switches are running the latest version of software.

    • BES-53248 switches have optional port licenses installed, if purchased.

    • Reference Configuration Files (RCFs) are applied to the switches.

  • Any site customization (SMTP, SNMP, and SSH) are configured on the new switches.

Port group speed constraints

  • The 48 10/25GbE (SFP28/SFP+) ports are combined into 12 x 4-port groups as follows: Ports 1-4, 5-8, 9-12, 13-16, 17-20, 21-24, 25-28, 29-32, 33-36, 37-40, 41-44, and 45-48.

  • The SFP28/SFP+ port speed must be the same (10GbE or 25GbE) across all ports in the 4-port group.

  • If speeds in a 4-port group are different, the switch ports will not operate correctly.

Migrate to the cluster environment

About the examples

The examples in this procedure use the following cluster switch and node nomenclature:

  • The names of the BES-53248 switches are cs1 and cs2.

  • The names of the cluster SVMs are node1 and node2.

  • The names of the LIFs are node1_clus1 and node1_clus2 on node 1, and node2_clus1 and node2_clus2 on node 2 respectively.

  • The cluster1::*> prompt indicates the name of the cluster.

  • The cluster ports used in this procedure are e0a and e0b.

    The NetApp Hardware Universe contains the latest information about the actual cluster ports for your platforms.

Step 1: Prepare for migration

  1. If AutoSupport is enabled on this cluster, suppress automatic case creation by invoking an AutoSupport message:

    system node autosupport invoke -node * -type all -message MAINT=xh

    where x is the duration of the maintenance window in hours.

    Note The AutoSupport message notifies technical support of this maintenance task so that automatic case creation is suppressed during the maintenance window.

    The following command suppresses automatic case creation for two hours:

    cluster1::*> system node autosupport invoke -node \* -type all -message MAINT=2h

  2. Change the privilege level to advanced, entering y when prompted to continue:

    set -privilege advanced

    The advanced prompt (*>) appears.

Step 2: Configure ports and cabling

  1. Disable all activated node-facing ports (not ISL ports) on both the new cluster switches cs1 and cs2.

    Note You must not disable the ISL ports.

    The following example shows that node-facing ports 1 through 16 are disabled on switch cs1:

    (cs1)# configure
(cs1)(Config)# interface 0/1-0/16
(cs1)(Interface 0/1-0/16)# shutdown
(cs1)(Interface 0/1-0/16)# exit
(cs1)(Config)# exit

  2. Verify that the ISL and the physical ports on the ISL between the two BES-53248 switches cs1 and cs2 are up:

    show port-channel

    Show example

    The following example shows that the ISL ports are up on switch cs1:

    (cs1)# show port-channel 1/1
Local Interface................................ 1/1
Channel Name................................... Cluster-ISL
Link State..................................... Up
Admin Mode..................................... Enabled
Type........................................... Dynamic
Port channel Min-links......................... 1
Load Balance Option............................ 7
(Enhanced hashing mode)

Mbr    Device/       Port       Port
Ports  Timeout       Speed      Active
------ ------------- ---------  -------
0/55   actor/long    100G Full  True
       partner/long
0/56   actor/long    100G Full  True
       partner/long
(cs1) #

    The following example shows that the ISL ports are up on switch cs2:

    (cs2)# show port-channel 1/1
Local Interface................................ 1/1
Channel Name................................... Cluster-ISL
Link State..................................... Up
Admin Mode..................................... Enabled
Type........................................... Dynamic
Port channel Min-links......................... 1
Load Balance Option............................ 7
(Enhanced hashing mode)

Mbr    Device/       Port       Port
Ports  Timeout       Speed      Active
------ ------------- ---------  -------
0/55   actor/long    100G Full  True
       partner/long
0/56   actor/long    100G Full  True
       partner/long

  3. Display the list of neighboring devices:

    show isdp neighbors

    This command provides information about the devices that are connected to the system.

    Show example

    The following example lists the neighboring devices on switch cs1:

    (cs1)# show isdp neighbors

Capability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge,
                  S - Switch, H - Host, I - IGMP, r - Repeater
Device ID      Intf     Holdtime  Capability   Platform    Port ID
-------------- -------- --------- ------------ ----------- ---------
cs2            0/55     176       R            BES-53248   0/55
cs2            0/56     176       R            BES-53248   0/56

    The following example lists the neighboring devices on switch cs2:

    (cs2)# show isdp neighbors

Capability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge,
                  S - Switch, H - Host, I - IGMP, r - Repeater
Device ID      Intf     Holdtime  Capability   Platform    Port ID
-------------- -------- --------- ------------ ----------- ---------
cs2            0/55     176       R            BES-53248   0/55
cs2            0/56     176       R            BES-53248   0/56

  4. Verify that all cluster ports are up:

    network port show -ipspace Cluster

    Show example 
    cluster1::*> network port show -ipspace Cluster

Node: node1

                                                  Speed(Mbps) Health
Port      IPspace      Broadcast Domain Link MTU  Admin/Oper  Status
--------- ------------ ---------------- ---- ---- ----------- --------
e0a       Cluster      Cluster          up   9000  auto/10000 healthy
e0b       Cluster      Cluster          up   9000  auto/10000 healthy

Node: node2

                                                  Speed(Mbps) Health
Port      IPspace      Broadcast Domain Link MTU  Admin/Oper  Status
--------- ------------ ---------------- ---- ---- ----------- --------
e0a       Cluster      Cluster          up   9000  auto/10000 healthy
e0b       Cluster      Cluster          up   9000  auto/10000 healthy

  5. Verify that all cluster LIFs are up and operational:

    network interface show -vserver Cluster

    Show example 
    cluster1::*> network interface show -vserver Cluster

            Logical      Status     Network            Current       Current Is
Vserver     Interface    Admin/Oper Address/Mask       Node          Port    Home
----------- ------------ ---------- ------------------ ------------- ------- -----
Cluster
            node1_clus1  up/up      169.254.209.69/16  node1         e0a     true
            node1_clus2  up/up      169.254.49.125/16  node1         e0b     true
            node2_clus1  up/up      169.254.47.194/16  node2         e0a     true
            node2_clus2  up/up      169.254.19.183/16  node2         e0b     true

  6. Disable auto-revert on the cluster LIFs.

    cluster1::*> network interface modify -vserver Cluster -lif * -auto-revert false

  7. Disconnect the cable from cluster port e0a on node1, and then connect e0a to port 1 on cluster switch cs1, using the appropriate cabling supported by the BES-53248 switches.

    The NetApp Hardware Universe contains more information about cabling.

  8. Disconnect the cable from cluster port e0a on node2, and then connect e0a to port 2 on cluster switch cs1, using the appropriate cabling supported by the BES-53248 switches.

  9. Enable all node-facing ports on cluster switch cs1.

    The following example shows that ports 1 through 16 are enabled on switch cs1:

    (cs1)# configure
(cs1)(Config)# interface 0/1-0/16
(cs1)(Interface 0/1-0/16)# no shutdown
(cs1)(Interface 0/1-0/16)# exit
(cs1)(Config)# exit

  10. Verify that all cluster ports are up:

    network port show -ipspace Cluster

    Show example 
    cluster1::*> network port show -ipspace Cluster

Node: node1
                                                                       Ignore
                                                  Speed(Mbps) Health   Health
Port      IPspace      Broadcast Domain Link MTU  Admin/Oper  Status   Status
--------- ------------ ---------------- ---- ---- ----------- -------- ------
e0a       Cluster      Cluster          up   9000  auto/10000 healthy  false
e0b       Cluster      Cluster          up   9000  auto/10000 healthy  false

Node: node2
                                                                       Ignore
                                                  Speed(Mbps) Health   Health
Port      IPspace      Broadcast Domain Link MTU  Admin/Oper  Status   Status
--------- ------------ ---------------- ---- ---- ----------- -------- ------
e0a       Cluster      Cluster          up   9000  auto/10000 healthy  false
e0b       Cluster      Cluster          up   9000  auto/10000 healthy  false

  11. Verify that all cluster LIFs are up and operational:

    network interface show -vserver Cluster

    Show example 
    cluster1::*> network interface show -vserver Cluster

         Logical      Status     Network            Current     Current Is
Vserver  Interface    Admin/Oper Address/Mask       Node        Port    Home
-------- ------------ ---------- ------------------ ----------- ------- ----
Cluster
         node1_clus1  up/up      169.254.209.69/16  node1       e0a     false
         node1_clus2  up/up      169.254.49.125/16  node1       e0b     true
         node2_clus1  up/up      169.254.47.194/16  node2       e0a     false
         node2_clus2  up/up      169.254.19.183/16  node2       e0b     true

  12. Display information about the status of the nodes in the cluster:

    cluster show

    Show example

    The following example displays information about the health and eligibility of the nodes in the cluster:

    cluster1::*> cluster show

Node                 Health  Eligibility   Epsilon
-------------------- ------- ------------  ------------
node1                true    true          false
node2                true    true          false

  13. Disconnect the cable from cluster port e0b on node1, and then connect e0b to port 1 on cluster switch cs2, using the appropriate cabling supported by the BES-53248 switches.

  14. Disconnect the cable from cluster port e0b on node2, and then connect e0b to port 2 on cluster switch cs2, using the appropriate cabling supported by the BES-53248 switches.

  15. Enable all node-facing ports on cluster switch cs2.

    The following example shows that ports 1 through 16 are enabled on switch cs2:

    (cs2)# configure
(cs2)(Config)# interface 0/1-0/16
(cs2)(Interface 0/1-0/16)# no shutdown
(cs2)(Interface 0/1-0/16)# exit
(cs2)(Config)# exit

  16. Verify that all cluster ports are up:

    network port show -ipspace Cluster

    Show example 
    cluster1::*> network port show -ipspace Cluster

Node: node1
                                                                       Ignore
                                                  Speed(Mbps) Health   Health
Port      IPspace      Broadcast Domain Link MTU  Admin/Oper  Status   Status
--------- ------------ ---------------- ---- ---- ----------- -------- ------
e0a       Cluster      Cluster          up   9000  auto/10000 healthy  false
e0b       Cluster      Cluster          up   9000  auto/10000 healthy  false

Node: node2
                                                                       Ignore
                                                  Speed(Mbps) Health   Health
Port      IPspace      Broadcast Domain Link MTU  Admin/Oper  Status   Status
--------- ------------ ---------------- ---- ---- ----------- -------- ------
e0a       Cluster      Cluster          up   9000  auto/10000 healthy  false
e0b       Cluster      Cluster          up   9000  auto/10000 healthy  false

Step 3: Verify the configuration

  1. Enable auto-revert on the cluster LIFs.

    cluster1::*> network interface modify -vserver Cluster -lif * -auto-revert true

  2. Verify that the cluster LIFs have reverted to their home ports (this might take a minute):

    network interface show -vserver Cluster

    If the cluster LIFs have not reverted to their home port, manually revert them:

    network interface revert -vserver Cluster -lif *

  3. Verify that all interfaces display true for Is Home:

    network interface show -vserver Cluster

    Note This might take several minutes to complete.
    Show example 
    cluster1::*> network interface show -vserver Cluster

          Logical      Status     Network            Current    Current Is
Vserver   Interface    Admin/Oper Address/Mask       Node       Port    Home
--------- ------------ ---------- ------------------ ---------- ------- ----
Cluster
          node1_clus1  up/up      169.254.209.69/16  node1      e0a     true
          node1_clus2  up/up      169.254.49.125/16  node1      e0b     true
          node2_clus1  up/up      169.254.47.194/16  node2      e0a     true
          node2_clus2  up/up      169.254.19.183/16  node2      e0b     true

  4. Verify that both nodes each have one connection to each switch:

    show isdp neighbors

    Show example

    The following example shows the appropriate results for both switches:

    (cs1)# show isdp neighbors

Capability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge,
                  S - Switch, H - Host, I - IGMP, r - Repeater
Device ID      Intf         Holdtime  Capability   Platform -- Port ID
-------------- ------------ --------- ------------ ----------- ----------
node1          0/1          175       H            FAS2750     e0a
node2          0/2          157       H            FAS2750     e0a
cs2            0/55         178       R            BES-53248   0/55
cs2            0/56         178       R            BES-53248   0/56


(cs2)# show isdp neighbors

Capability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge,
                  S - Switch, H - Host, I - IGMP, r - Repeater
Device ID      Intf         Holdtime  Capability   Platform    Port ID
-------------- ------------ --------- ------------ ----------- ------------
node1          0/1          137       H            FAS2750     e0b
node2          0/2          179       H            FAS2750     e0b
cs1            0/55         175       R            BES-53248   0/55
cs1            0/56         175       R            BES-53248   0/56

  5. Display information about the discovered network devices in your cluster:

    network device-discovery show -protocol cdp

    Show example 
    cluster1::*> network device-discovery show -protocol cdp
Node/       Local  Discovered
Protocol    Port   Device (LLDP: ChassisID)  Interface         Platform
----------- ------ ------------------------- ----------------  ----------------
node2      /cdp
            e0a    cs1                       0/2               BES-53248
            e0b    cs2                       0/2               BES-53248
node1      /cdp
            e0a    cs1                       0/1               BES-53248
            e0b    cs2                       0/1               BES-53248

  6. Verify that the settings are disabled:

    network options switchless-cluster show

    Note It might take several minutes for the command to complete. Wait for the '3 minute lifetime to expire' announcement.

    The false output in the following example shows that the configuration settings are disabled:

    cluster1::*> network options switchless-cluster show
Enable Switchless Cluster: false

  7. Verify the status of the node members in the cluster:

    cluster show

    Show example

    The following example shows information about the health and eligibility of the nodes in the cluster:

    cluster1::*> cluster show

Node                 Health  Eligibility   Epsilon
-------------------- ------- ------------  --------
node1                true    true          false
node2                true    true          false

  8. Verify the connectivity of the remote cluster interfaces:

ONTAP 9.9.1 and later

You can use the network interface check cluster-connectivity command to start an accessibility check for cluster connectivity and then display the details:

network interface check cluster-connectivity start and network interface check cluster-connectivity show

cluster1::*> network interface check cluster-connectivity start

NOTE: Wait for a number of seconds before running the show command to display the details.

cluster1::*> network interface check cluster-connectivity show
                                  Source          Destination       Packet
Node   Date                       LIF             LIF               Loss
------ -------------------------- --------------- ----------------- -----------
node1
       3/5/2022 19:21:18 -06:00   node1_clus2      node2_clus1      none
       3/5/2022 19:21:20 -06:00   node1_clus2      node2_clus2      none

node2
       3/5/2022 19:21:18 -06:00   node2_clus2      node1_clus1      none
       3/5/2022 19:21:20 -06:00   node2_clus2      node1_clus2      none
All ONTAP releases

For all ONTAP releases, you can also use the cluster ping-cluster -node <name> command to check the connectivity:

cluster ping-cluster -node <name>

cluster1::*> cluster ping-cluster -node node2
Host is node2
Getting addresses from network interface table...
Cluster node1_clus1 169.254.209.69 node1     e0a
Cluster node1_clus2 169.254.49.125 node1     e0b
Cluster node2_clus1 169.254.47.194 node2     e0a
Cluster node2_clus2 169.254.19.183 node2     e0b
Local = 169.254.47.194 169.254.19.183
Remote = 169.254.209.69 169.254.49.125
Cluster Vserver Id = 4294967293
Ping status:
....
Basic connectivity succeeds on 4 path(s)
Basic connectivity fails on 0 path(s)
................
Detected 9000 byte MTU on 4 path(s):
    Local 169.254.19.183 to Remote 169.254.209.69
    Local 169.254.19.183 to Remote 169.254.49.125
    Local 169.254.47.194 to Remote 169.254.209.69
    Local 169.254.47.194 to Remote 169.254.49.125
Larger than PMTU communication succeeds on 4 path(s)
RPC status:
2 paths up, 0 paths down (tcp check)
2 paths up, 0 paths down (udp check)

  1. Change the privilege level back to admin:

    set -privilege admin

  2. If you suppressed automatic case creation, reenable it by invoking an AutoSupport message:

    system node autosupport invoke -node * -type all -message MAINT=END

    Show example 
    cluster1::*> system node autosupport invoke -node \* -type all -message MAINT=END

    For more information, see: NetApp KB Article: How to suppress automatic case creation during scheduled maintenance windows

What's next?

Configure switch health monitoring.