Skip to main content
Cluster and storage switches

Migrate from a switchless cluster with direct-attached storage

Contributors netapp-karunesh netapp-jolieg

You can migrate from a switchless cluster with direct-attached storage by adding two new shared switches.

The procedure you use depends on whether you have two dedicated cluster-network ports on each controller or a single cluster port on each controller. The process documented works for all nodes using optical or Twinax ports, but is not supported on this switch if nodes are using onboard 10Gb BASE-T RJ45 ports for the cluster-network ports.

Most systems require two dedicated cluster-network ports on each controller. See Cisco Ethernet Switches for more information.

If you have an existing two-node switchless cluster environment, you can migrate to a two-node switched cluster environment using Cisco Nexus 9336C-FX2 switches to enable you to scale beyond two nodes in the cluster.

Review requirements

Ensure that:

  • For the two-node switchless configuration:

    • The two-node switchless configuration is properly set up and functioning.

    • The nodes are running ONTAP 9.8 and later.

    • All cluster ports are in the up state.

    • All cluster logical interfaces (LIFs) are in the up state and on their home ports.

  • For the Cisco Nexus 9336C-FX2 switch configuration:

    • Both switches have management network connectivity.

    • There is console access to the cluster switches.

    • Nexus 9336C-FX2 node-to-node switch and switch-to-switch connections use Twinax or fiber cables.

    • The NetApp Hardware Universe contains more information about cabling.

    • Inter-Switch Link (ISL) cables are connected to ports 1/35 and 1/36 on both 9336C-FX2 switches.

  • Initial customization of the 9336C-FX2 switches are completed. So that the:

    • 9336C-FX2 switches are running the latest version of software

    • Reference Configuration Files (RCFs) have been applied to the switches

    • Any site customization, such as SMTP, SNMP, and SSH is configured on the new switches.

Migrate the switches

About the examples

The examples in this procedure use the following cluster switch and node nomenclature:

  • The names of the 9336C-FX2 switches are cs1 and cs2.

  • The names of the cluster SVMs are node1 and node2.

  • The names of the LIFs are node1_clus1 and node1_clus2 on node 1, and node2_clus1 and node2_clus2 on node 2 respectively.

  • The cluster1::*> prompt indicates the name of the cluster.

  • The cluster ports used in this procedure are e3a and e3b, as per the AFF A400 controller. The Hardware Universe contains the latest information about the actual cluster ports for your platforms.

Step 1: Migrate from a switchless cluster with direct-attached

  1. If AutoSupport is enabled on this cluster, suppress automatic case creation by invoking an AutoSupport message: system node autosupport invoke -node * -type all -message MAINT=xh.

    where x is the duration of the maintenance window in hours.

Note The AutoSupport message notifies technical support of this maintenance task so that automatic case creation is suppressed during the maintenance window.
  1. Change the privilege level to advanced, entering y when prompted to continue:

    set -privilege advanced

    The advanced prompt (*>) appears.

  2. Disable all node-facing ports (not ISL ports) on both the new cluster switches cs1 and cs2. You must not disable the ISL ports.

    Show example

    The following example shows that node-facing ports 1 through 34 are disabled on switch cs1:

    cs1# config
    Enter configuration commands, one per line. End with CNTL/Z.
    cs1(config)# interface e1/1-34
    cs1(config-if-range)# shutdown
  1. Verify that the ISL and the physical ports on the ISL between the two 9336C-FX2 switches cs1 and cs2 are up on ports 1/35 and 1/36:

    show port-channel summary

    Show example

    The following example shows that the ISL ports are up on switch cs1:

    cs1# show port-channel summary
    Flags:  D - Down        P - Up in port-channel (members)
            I - Individual  H - Hot-standby (LACP only)
            s - Suspended   r - Module-removed
            b - BFD Session Wait
            S - Switched    R - Routed
            U - Up (port-channel)
            p - Up in delay-lacp mode (member)
            M - Not in use. Min-links not met
    --------------------------------------------------------------------------------
    Group Port-       Type     Protocol  Member Ports
          Channel
    --------------------------------------------------------------------------------
    1     Po1(SU)     Eth      LACP      Eth1/35(P)   Eth1/36(P)

    The following example shows that the ISL ports are up on switch cs2:

           cs2# show port-channel summary
            Flags:  D - Down        P - Up in port-channel (members)
            I - Individual  H - Hot-standby (LACP only)
            s - Suspended   r - Module-removed
            b - BFD Session Wait
            S - Switched    R - Routed
            U - Up (port-channel)
            p - Up in delay-lacp mode (member)
            M - Not in use. Min-links not met
    --------------------------------------------------------------------------------
    Group Port-       Type     Protocol  Member Ports
          Channel
    --------------------------------------------------------------------------------
    1     Po1(SU)     Eth      LACP      Eth1/35(P)   Eth1/36(P)
  1. Display the list of neighboring devices:

    show cdp neighbors

    This command provides information about the devices that are connected to the system.

    Show example

    The following example lists the neighboring devices on switch cs1:

    cs1# show cdp neighbors
    Capability Codes: R - Router, T - Trans-Bridge, B - Source-Route-Bridge
                      S - Switch, H - Host, I - IGMP, r - Repeater,
                      V - VoIP-Phone, D - Remotely-Managed-Device,
                      s - Supports-STP-Dispute
    Device-ID          Local Intrfce  Hldtme Capability  Platform      Port ID
    cs2                Eth1/35        175    R S I s     N9K-C9336C    Eth1/35
    cs2                Eth1/36        175    R S I s     N9K-C9336C    Eth1/36
    Total entries displayed: 2

    The following example lists the neighboring devices on switch cs2:

    cs2# show cdp neighbors
    Capability Codes: R - Router, T - Trans-Bridge, B - Source-Route-Bridge
                      S - Switch, H - Host, I - IGMP, r - Repeater,
                      V - VoIP-Phone, D - Remotely-Managed-Device,
                      s - Supports-STP-Dispute
    Device-ID          Local Intrfce  Hldtme Capability  Platform      Port ID
    cs1                Eth1/35        177    R S I s     N9K-C9336C    Eth1/35
    cs1           )    Eth1/36        177    R S I s     N9K-C9336C    Eth1/36
    
    Total entries displayed: 2
  1. Verify that all cluster ports are up:

    network port show - ipspace Cluster

    Each port should display up for Link and healthy for Health Status.

    Show example
    cluster1::*> network port show -ipspace Cluster
    
    Node: node1
                                                      Speed(Mbps)  Health
    Port      IPspace      Broadcast Domain Link MTU  Admin/Oper   Status
    --------- ------------ ---------------- ---- ---- ------------ ---------
    e3a       Cluster      Cluster          up   9000  auto/100000 healthy
    e3b       Cluster      Cluster          up   9000  auto/100000 healthy
    
    Node: node2
                                                      Speed(Mbps)  Health
    Port      IPspace      Broadcast Domain Link MTU  Admin/Oper   Status
    --------- ------------ ---------------- ---- ---- ------------ ---------
    e3a       Cluster      Cluster          up   9000  auto/100000 healthy
    e3b       Cluster      Cluster          up   9000  auto/100000 healthy
    4 entries were displayed.
  1. Verify that all cluster LIFs are up and operational:

    network interface show - vserver Cluster

    Each cluster LIF should display true for Is Home and have a Status Admin/Oper of up/up.

    Show example
    cluster1::*> network interface show -vserver Cluster
                Logical     Status     Network            Current       Current Is
    Vserver     Interface   Admin/Oper Address/Mask       Node          Port    Home
    ----------- ---------- ---------- ------------------ ------------- ------- -----
    Cluster
                node1_clus1  up/up    169.254.209.69/16  node1         e3a     true
                node1_clus2  up/up    169.254.49.125/16  node1         e3b     true
                node2_clus1  up/up    169.254.47.194/16  node2         e3a     true
                node2_clus2  up/up    169.254.19.183/16  node2         e3b     true
    4 entries were displayed.
  1. Verify that auto-revert is enabled on all cluster LIFs:

    network interface show - vserver Cluster -fields auto-revert

    Show example
    cluster1::*> network interface show -vserver Cluster -fields auto-revert
           Logical
    Vserver   Interface     Auto-revert
    --------- ------------- ------------
    Cluster
              node1_clus1   true
              node1_clus2   true
              node2_clus1   true
              node2_clus2   true
    4 entries were displayed.
  1. Disconnect the cable from cluster port e3a on node1, and then connect e3a to port 1 on cluster switch cs1, using the appropriate cabling supported by the 9336C-FX2 switches.

    The NetApp Hardware Universe contains more information about cabling.

  2. Disconnect the cable from cluster port e3a on node2, and then connect e3a to port 2 on cluster switch cs1, using the appropriate cabling supported by the 9336C-FX2 switches.

  3. Enable all node-facing ports on cluster switch cs1.

    Show example

    The following example shows that ports 1/1 through 1/34 are enabled on switch cs1:

    cs1# config
    Enter configuration commands, one per line. End with CNTL/Z.
    cs1(config)# interface e1/1-34
    cs1(config-if-range)# no shutdown
  1. Verify that all cluster LIFs are up, operational, and display as true for Is Home:

    network interface show - vserver Cluster

    Show example

    The following example shows that all the LIFs are up on node1 and node2 and that Is Home results are true:

    cluster1::*> network interface show -vserver Cluster
              Logical      Status     Network            Current     Current Is
    Vserver   Interface    Admin/Oper Address/Mask       Node        Port    Home
    --------- ------------ ---------- ------------------ ----------- ------- ----
    Cluster
              node1_clus1  up/up      169.254.209.69/16  node1       e3a     true
              node1_clus2  up/up      169.254.49.125/16  node1       e3b     true
              node2_clus1  up/up      169.254.47.194/16  node2       e3a     true
              node2_clus2  up/up      169.254.19.183/16  node2       e3b     true
    4 entries were displayed.
  1. Display information about the status of the nodes in the cluster:

    cluster show

    Show example

    The following example displays information about the health and eligibility of the nodes in the cluster:

    cluster1::*> cluster show
    Node                 Health  Eligibility   Epsilon
    -------------------- ------- ------------  ------------
    node1                true    true          false
    node2                true    true          false
    2 entries were displayed.
  1. Disconnect the cable from cluster port e3b on node1, and then connect e3b to port 1 on cluster switch cs2, using the appropriate cabling supported by the 9336C-FX2 switches.

  2. Disconnect the cable from cluster port e3b on node2, and then connect e3b to port 2 on cluster switch cs2, using the appropriate cabling supported by the 9336C-FX2 switches.

  3. Enable all node-facing ports on cluster switch cs2.

    Show example

    The following example shows that ports 1/1 through 1/34 are enabled on switch cs2:

    cs2# config
    Enter configuration commands, one per line. End with CNTL/Z.
    cs2(config)# interface e1/1-34
    cs2(config-if-range)# no shutdown
  1. Verify that all cluster ports are up:

    network port show - ipspace Cluster

    Show example

    The following example shows that all the cluster ports are up on node1 and node2:

    cluster1::*> network port show -ipspace Cluster
    
    Node: node1
                                                                            Ignore
                                                      Speed(Mbps)  Health   Health
    Port      IPspace      Broadcast Domain Link MTU  Admin/Oper   Status   Status
    --------- ------------ ---------------- ---- ---- ------------ -------- ------
    e3a       Cluster      Cluster          up   9000  auto/100000 healthy  false
    e3b       Cluster      Cluster          up   9000  auto/100000 healthy  false
    
    Node: node2
                                                                            Ignore
                                                      Speed(Mbps)  Health   Health
    Port      IPspace      Broadcast Domain Link MTU  Admin/Oper   Status   Status
    --------- ------------ ---------------- ---- ---- ------------ -------- ------
    e3a       Cluster      Cluster          up   9000  auto/100000 healthy  false
    e3b       Cluster      Cluster          up   9000  auto/100000 healthy  false
    4 entries were displayed.
  1. Verify that all interfaces display true for Is Home:

    network interface show - vserver Cluster

    Note This might take several minutes to complete.
    Show example

    The following example shows that all LIFs are up on node1 and node2 and that Is Home results are true:

    cluster1::*> network interface show -vserver Cluster
              Logical      Status     Network            Current    Current Is
    Vserver   Interface    Admin/Oper Address/Mask       Node       Port    Home
    --------- ------------ ---------- ------------------ ---------- ------- ----
    Cluster
              node1_clus1  up/up      169.254.209.69/16  node1      e3a     true
              node1_clus2  up/up      169.254.49.125/16  node1      e3b     true
              node2_clus1  up/up      169.254.47.194/16  node2      e3a     true
              node2_clus2  up/up      169.254.19.183/16  node2      e3b     true
    4 entries were displayed.
  1. Verify that both nodes each have one connection to each switch:

    show cdp neighbors

    Show example

    The following example shows the appropriate results for both switches:

    cs1# show cdp neighbors
    Capability Codes: R - Router, T - Trans-Bridge, B - Source-Route-Bridge
                      S - Switch, H - Host, I - IGMP, r - Repeater,
                      V - VoIP-Phone, D - Remotely-Managed-Device,
                      s - Supports-STP-Dispute
    Device-ID          Local Intrfce  Hldtme Capability  Platform      Port ID
    node1              Eth1/1         133    H           AFFA400       e3a
    node2              Eth1/2         133    H           AFFA400       e3a
    cs2                Eth1/35        175    R S I s     N9K-C9336C    Eth1/35
    cs2                Eth1/36        175    R S I s     N9K-C9336C    Eth1/36
    Total entries displayed: 4
    cs2# show cdp neighbors
    Capability Codes: R - Router, T - Trans-Bridge, B - Source-Route-Bridge
                      S - Switch, H - Host, I - IGMP, r - Repeater,
                      V - VoIP-Phone, D - Remotely-Managed-Device,
                      s - Supports-STP-Dispute
    Device-ID          Local Intrfce  Hldtme Capability  Platform      Port ID
    node1              Eth1/1         133    H           AFFA400       e3b
    node2              Eth1/2         133    H           AFFA400       e3b
    cs1                Eth1/35        175    R S I s     N9K-C9336C    Eth1/35
    cs1                Eth1/36        175    R S I s     N9K-C9336C    Eth1/36
    Total entries displayed: 4
  1. Display information about the discovered network devices in your cluster:

    network device-discovery show -protocol cdp

    Show example
    cluster1::*> network device-discovery show -protocol cdp
    Node/       Local  Discovered
    Protocol    Port   Device (LLDP: ChassisID)  Interface         Platform
    ----------- ------ ------------------------- ----------------  ----------------
    node2       /cdp
                e3a    cs1                       0/2               N9K-C9336C
                e3b    cs2                       0/2               N9K-C9336C
    
    node1       /cdp
                e3a    cs1                       0/1               N9K-C9336C
                e3b    cs2                       0/1               N9K-C9336C
    4 entries were displayed.
  1. Verify that the storage configuration of HA pair 1 (and HA pair 2) is correct and error free:

    system switch ethernet show

    Show example
    storage::*> system switch ethernet show
    Switch                    Type                   Address         Model
    ------------------------- ---------------------- --------------- ----------
    sh1
                              storage-network        172.17.227.5    C9336C
    
           Serial Number: FOC221206C2
            Is Monitored: true
                  Reason: None
        Software Version: Cisco Nexus Operating System (NX-OS) Software, Version
                          9.3(5)
          Version Source: CDP
    sh2
                              storage-network        172.17.227.6    C9336C
           Serial Number: FOC220443LZ
            Is Monitored: true
                  Reason: None
        Software Version: Cisco Nexus Operating System (NX-OS) Software, Version
                          9.3(5)
          Version Source: CDP
    2 entries were displayed.
    storage::*>
  1. Verify that the settings are disabled:

    network options switchless-cluster show

    Note It might take several minutes for the command to complete. Wait for the '3-minute lifetime to expire' announcement.

    The false output in the following example shows that the configuration settings are disabled:

    Show example
    cluster1::*> network options switchless-cluster show
    Enable Switchless Cluster: false
  1. Verify the status of the node members in the cluster:

    cluster show

    Show example

    The following example shows information about the health and eligibility of the nodes in the cluster:

    cluster1::*> cluster show
    Node                 Health  Eligibility   Epsilon
    -------------------- ------- ------------  --------
    node1                true    true          false
    node2                true    true          false
  1. Ensure that the cluster network has full connectivity:

    cluster ping-cluster -node node-name

    Show example
    cluster1::*> cluster ping-cluster -node node2
    Host is node2
    Getting addresses from network interface table...
    Cluster node1_clus1 169.254.209.69 node1 e3a
    Cluster node1_clus2 169.254.49.125 node1 e3b
    Cluster node2_clus1 169.254.47.194 node2 e3a
    Cluster node2_clus2 169.254.19.183 node2 e3b
    Local = 169.254.47.194 169.254.19.183
    Remote = 169.254.209.69 169.254.49.125
    Cluster Vserver Id = 4294967293
    Ping status:
    ....
    Basic connectivity succeeds on 4 path(s)
    Basic connectivity fails on 0 path(s)
    ................
    Detected 9000 byte MTU on 4 path(s):
    Local 169.254.47.194 to Remote 169.254.209.69
    Local 169.254.47.194 to Remote 169.254.49.125
    Local 169.254.19.183 to Remote 169.254.209.69
    Local 169.254.19.183 to Remote 169.254.49.125
    Larger than PMTU communication succeeds on 4 path(s)
    RPC status:
    2 paths up, 0 paths down (tcp check)
    2 paths up, 0 paths down (udp check)
  1. Change the privilege level back to admin:

    set -privilege admin

  2. Enable the Ethernet switch health monitor log collection feature for collecting switch-related log files, using the commands:

    • system switch ethernet log setup-password

    • system switch ethernet log enable-collection

      Show example
      cluster1::*> system switch ethernet log setup-password
      Enter the switch name: <return>
      The switch name entered is not recognized.
      
      Choose from the following list:
      cs1
      cs2
      cluster1::*> system switch ethernet log setup-password
      Enter the switch name: cs1
      RSA key fingerprint is e5:8b:c6:dc:e2:18:18:09:36:63:d9:63:dd:03:d9:cc
      Do you want to continue? {y|n}::[n] y
      Enter the password: <enter switch password>
      Enter the password again: <enter switch password>
      cluster1::*> system switch ethernet log setup-password
      Enter the switch name: cs2
      RSA key fingerprint is 57:49:86:a1:b9:80:6a:61:9a:86:8e:3c:e3:b7:1f:b1
      Do you want to continue? {y|n}:: [n] y
      Enter the password: <enter switch password>
      Enter the password again: <enter switch password>
      cluster1::*> system  switch ethernet log enable-collection
      Do you want to enable cluster log collection for all nodes in the cluster? {y|n}: [n] y
      Enabling cluster switch log collection.
      cluster1::*>

Step 2: Set up the shared switch

The examples in this procedure use the following switch and node nomenclature:

  • The names of the two shared switches are sh1 and sh2.

  • The nodes are node1 and node2.

Note The procedure requires the use of both ONTAP commands and Cisco Nexus 9000 Series Switches commands, ONTAP commands are used unless otherwise indicated.
  1. Verify that the storage configuration of HA pair 1 (and HA pair 2) is correct and error free:

    system switch ethernet show

    Show example
    storage::*> system switch ethernet show
    Switch                    Type                   Address         Model
    ------------------------- ---------------------  --------------- -------
    sh1
                              storage-network        172.17.227.5    C9336C
    
          Serial Number: FOC221206C2
           Is Monitored: true
                 Reason: None
       Software Version: Cisco Nexus Operating System (NX-OS) Software, Version
                         9.3(5)
         Version Source: CDP
    sh2
                              storage-network        172.17.227.6    C9336C
           Serial Number: FOC220443LZ
            Is Monitored: true
                  Reason: None
        Software Version: Cisco Nexus Operating System (NX-OS) Software, Version
                          9.3(5)
          Version Source: CDP
    2 entries were displayed.
    storage::*>
  2. Verify that the storage node ports are healthy and operational:

    storage port show -port-type ENET

    Show example
    storage::*> storage port show -port-type ENET
                                       Speed                             VLAN
    Node    Port    Type    Mode       (Gb/s)      State      Status       ID
    ------- ------- ------- ---------- ----------- ---------- ---------- -----
    node1
            e0c     ENET   storage          100      enabled  online        30
            e0d     ENET   storage          100      enabled  online        30
            e5a     ENET   storage          100      enabled  online        30
            e5b     ENET   storage          100      enabled  online        30
    
    node2
            e0c     ENET  storage           100      enabled  online        30
            e0d     ENET  storage           100      enabled  online        30
            e5a     ENET  storage           100      enabled  online        30
            e5b     ENET  storage           100      enabled  online        30
  1. Move the HA pair 1, NSM224 path A ports to sh1 port range 11-22.

  2. Install a cable from HA pair 1, node1, path A to sh1 port range 11-22. For example, the path A storage port on an AFF A400 is e0c.

  3. Install a cable from HA pair 1, node2, path A to sh1 port range 11-22.

  4. Verify that the node ports are healthy and operational:

    storage port show -port-type ENET

    Show example
    storage::*> storage port show -port-type ENET
                                       Speed                             VLAN
    Node    Port    Type    Mode       (Gb/s)      State      Status       ID
    ------- ------- ------- ---------- ----------- ---------- ---------- -----
    node1
            e0c     ENET   storage          100      enabled  online        30
            e0d     ENET   storage            0      enabled  offline       30
            e5a     ENET   storage            0      enabled  offline       30
            e5b     ENET   storage          100      enabled  online        30
    
    node2
            e0c     ENET  storage           100      enabled  online        30
            e0d     ENET  storage             0      enabled  offline       30
            e5a     ENET  storage             0      enabled  offline       30
            e5b     ENET  storage           100      enabled  online        30
  5. Check that there are no storage switch or cabling issues with the cluster:

    system health alert show -instance

    Show example
    storage::*> system health alert show -instance
    There are no entries matching your query.
  6. Move the HA pair 1, NSM224 path B ports to sh2 port range 11-22.

  7. Install a cable from HA pair 1, node1, path B to sh2 port range 11-22. For example, the path B storage port on an AFF A400 is e5b.

  8. Install a cable from HA pair 1, node2, path B to sh2 port range 11-22.

  9. Verify that the node ports are healthy and operational:

    storage port show -port-type ENET

    Show example
    storage::*> storage port show -port-type ENET
                                       Speed                             VLAN
    Node    Port    Type    Mode       (Gb/s)      State      Status       ID
    ------- ------- ------- ---------- ----------- ---------- ---------- -----
    node1
            e0c     ENET   storage          100      enabled  online        30
            e0d     ENET   storage            0      enabled  offline       30
            e5a     ENET   storage            0      enabled  offline       30
            e5b     ENET   storage          100      enabled  online        30
    
    node2
            e0c     ENET  storage           100      enabled  online        30
            e0d     ENET  storage             0      enabled  offline       30
            e5a     ENET  storage             0      enabled  offline       30
            e5b     ENET  storage           100      enabled  online        30
  10. Verify that the storage configuration of HA pair 1 is correct and error free:

    system switch ethernet show

    Show example
    storage::*> system switch ethernet show
    Switch                    Type                   Address          Model
    ------------------------- ---------------------- ---------------- ----------
    sh1
                              storage-network        172.17.227.5     C9336C
    
          Serial Number: FOC221206C2
           Is Monitored: true
                 Reason: None
       Software Version: Cisco Nexus Operating System (NX-OS) Software, Version
                         9.3(5)
         Version Source: CDP
    sh2
                              storage-network        172.17.227.6     C9336C
          Serial Number: FOC220443LZ
           Is Monitored: true
                 Reason: None
       Software Version: Cisco Nexus Operating System (NX-OS) Software, Version
                         9.3(5)
         Version Source: CDP
    2 entries were displayed.
    storage::*>
  11. Reconfigure the unused (controller) secondary storage ports on HA pair 1 from storage to networking. If more than one NS224 was direct attached, there will be ports that should be reconfigured.

    Show example
    storage port modify –node [node name] –port [port name] –mode network

    To place storage ports into a broadcast domain:

    • network port broadcast-domain create (to create a new domain, if needed)

    • network port broadcast-domain add-ports (to add ports to an existing domain)

  12. If you suppressed automatic case creation, re-enable it by invoking an AutoSupport message:

    system node autosupport invoke -node * -type all -message MAINT=END