Skip to main content
Enterprise applications

Network configuration

Contributors netapp-bingen jfsinmsp netapp-revathid
PDFs

Configuring network settings when using vSphere with systems running ONTAP is straightforward and similar to other network configuration.

Here are some things to consider:

  • Separate storage network traffic from other networks. A separate network can be achieved by using a dedicated VLAN or separate switches for storage. If the storage network shares physical paths such as uplinks, you might need QoS or additional uplink ports to make sure of sufficient bandwidth. Don't connect hosts directly to storage unless your solution guide specifically calls for it; use switches to have redundant paths and allow VMware HA to work without intervention.

  • Jumbo frames should be used if supported by your network. If they are used, make sure they are configured identically on all network devices, VLANs, and so on in the path between storage and the ESXi host. Otherwise, you might see performance or connection problems. The MTU must also be set identically on the ESXi virtual switch, the VMkernel port, and also on the physical ports or interface groups of each ONTAP node.

  • NetApp only recommends disabling network flow control on the cluster-interconnect ports within an ONTAP cluster. NetApp makes no other recommendations for best practices regarding flow control for the remaining network ports used for data traffic. You should enable or disable it as necessary. See TR-4182 for more background on flow control.

  • When ESXi and ONTAP storage arrays are connected to Ethernet storage networks, NetApp recommends configuring the Ethernet ports to which these systems connect as Rapid Spanning Tree Protocol (RSTP) edge ports or by using the Cisco PortFast feature. NetApp recommends enabling the Spanning-Tree PortFast trunk feature in environments that use the Cisco PortFast feature and that have 802.1Q VLAN trunking enabled to either the ESXi server or the ONTAP storage arrays.

  • NetApp recommends the following best practices for link aggregation:

    • Use switches that support link aggregation of ports on two separate switch chassis using a multi-chassis link aggregation group approach such as Cisco's Virtual PortChannel (vPC).

    • Disable LACP for switch ports connected to ESXi unless you are using dvSwitches 5.1 or later with LACP configured.

    • Use LACP to create link aggregates for ONTAP storage systems with dynamic multimode interface groups with IP hash.

    • Use an IP hash teaming policy on ESXi.

The following table provides a summary of network configuration items and indicates where the settings are applied.

Item ESXi Switch Node SVM

IP address

VMkernel

No**

No**

Yes

Link aggregation

Virtual switch

Yes

Yes

No*

VLAN

VMkernel and VM port groups

Yes

Yes

No*

Flow control

NIC

Yes

Yes

No*

Spanning tree

No

Yes

No

No

MTU (for jumbo frames)

Virtual switch and VMkernel port (9000)

Yes (set to max)

Yes (9000)

No*

Failover groups

No

No

Yes (create)

Yes (select)

*SVM LIFs connect to ports, interface groups, or VLAN interfaces that have VLAN, MTU, and other settings. However, the settings are not managed at the SVM level.

**These devices have IP addresses of their own for management, but these addresses are not used in the context of ESXi storage networking.

SAN (FC, NVMe/FC, iSCSI, NVMe/TCP), RDM

ONTAP offers enterprise-class block storage for VMware vSphere using traditional iSCSI and Fibre Channel Protocol (FCP) as well as the highly efficient and performant next-generation block protocol, NVMe over Fabrics (NVMe-oF), with support for both NVMe/FC and NVMe/TCP.

For detailed best practices for implementing block protocols for VM storage with vSphere and ONTAP refer to Datastores and Protocols - SAN

NFS

vSphere allows customers to use enterprise-class NFS arrays to provide concurrent access to datastores to all the nodes in an ESXi cluster. As mentioned in the datastores section, there are some ease of use and storage efficiency visibility benefits when using NFS with vSphere.

For recommended best practices refer to Datastores and Protocols - NFS

Direct connect networking

Virtualization administrators sometimes prefer to simplify their infrastructures by removing network switches from the configuration. This allows creation of a more self-contained solution with fewer components and less space required in the rack. There are some important restrictions that depend on the protocol in use.

Fiber Channel

Starting with ONTAP 9.19.1, direct-connect FC is supported with both traditional FC-SAN (SCSI) and NVMe/FC with select storage system FC adapters. The HBAs will establish a standard point-to-point FC connection.

Prior to ONTAP 9.19.1, ONTAP storage systems require NPIV, which means the storage system must be connected to an FC switch that also supports NPIV.

iSCSI and NVMe/TCP

A host using iSCSI or NVMe/TCP can be directly connected to a storage system and operate normally. The reason is pathing. Direct connections to two different storage controllers results in two independent paths for data flow. The loss of path, port, or controller does not prevent the other path from being used.

NFS

Direct-connected NFS storage can be used, but with a significant limitation - failover will not work without a significant scripting effort, which would be the responsibility of the customer.

The reason nondisruptive failover is complicated with direct-connected NFS storage is the routing that occurs on the local OS. For example, assume a host has an IP address of 192.168.1.1/24 and is directly connected to an ONTAP controller with an IP address of 192.168.1.50/24. During failover, that 192.168.1.50 address can fail over to the other controller, and it will be available to the host, but how does the host detect its presence? The original 192.168.1.1 address still exists on the host NIC that no longer connects to an operational system. Traffic destined for 192.168.1.50 would continue to be sent to an inoperable network port.

The second OS NIC could be configured as 192.168.1.2 and would be capable of communicating with the failed over 192.168.1.50 address, but the local routing tables would have a default of using one and only one address to communicate with the 192.168.1.0/24 subnet. A sysadmin could create a scripting framework that would detect a failed network connection and alter the local routing tables or bring interfaces up and down. The exact procedure would depend on the OS in use.

In practice, NetApp customers do have direct-connected NFS, but normally only for workloads where IO pauses during failovers are acceptable. When hard mounts are used, there should not be any IO errors during such pauses. The IO should hang until services are restored, either by a failback or manual intervention to move IP addresses between NICs on the host.