Skip to main content
NetApp Solutions

Anthos Clusters on VMware

Contributors kevin-hoke banum-netapp
PDFs

Anthos clusters on VMware is an extension of Google Kubernetes Engine that is deployed in an end user’s private data center. An organization can deploy the same applications designed to run in containers in Google Cloud in Kubernetes clusters on-premises.
Anthos clusters on VMware can be deployed into an existing VMware vSphere environment in your data center, which can save on capital expenses and enable more rapid deployment and scaling operations.

The deployment of Anthos clusters on VMware includes the following components:

  • Anthos admin workstation. A deployment host from which gkectl and kubectl commands can be run to deploy and interact with Anthos deployments.

  • Admin cluster. The initial cluster deployed when setting up Anthos clusters on VMware. This cluster manages all subordinate user cluster actions, including deployment, scaling, and upgrade.

  • User cluster. Each user cluster is deployed with it's own load balancer instance or partition, allowing it to act as a standalone Kubernetes cluster for individual users or groups, helping to achieve full multitenancy.

The following graphic is a description of an Anthos-clusters-on-VMware deployment.

Figure showing input/output dialog or representing written content

Benefits

Anthos clusters on VMware offers the following benefits:

  • Advanced multitenancy. Each end user can be assigned their own user cluster, deployed with the virtual resources necessary for their own development environment.

  • Cost savings. End users can realize significant cost savings by deploying multiple user clusters to the same physical environment and utilizing their own physical resources for their application deployments instead of provisioning resources in their Google Cloud environment or on large bare-metal clusters.

  • Develop then publish. On-premises deployments can be used while applications are in development, which allows for testing of applications in the privacy of a local data center before being made publicly available in the cloud.

  • Security requirements. Customers with increased security concerns or sensitive data sets that cannot be stored in the public cloud are able to run their applications from the security of their own data centers, thereby meeting organizational requirements.

VMware vSphere

VMware vSphere is a virtualization platform for centrally managing a large number of virtualized servers and networks running on the ESXi hypervisor.

For more information about VMware vSphere, see the VMware vSphere website.

VMware vSphere provides the following features:

  • VMware vCenter Server. VMware vCenter Server provides unified management of all hosts and VMs from a single console and aggregates performance monitoring of clusters, hosts, and VMs.

  • VMware vSphere vMotion. VMware vCenter allows you to hot migrate VMs between nodes in the cluster upon request in a non-disruptive manner.

  • vSphere High Availability. To avoid disruption in the event of host failures, VMware vSphere allows hosts to be clustered and configured for high availability. VMs that are disrupted by host failure are rebooted shortly on other hosts in the cluster, restoring services.

  • Distributed Resource Scheduler (DRS). A VMware vSphere cluster can be configured to load balance the resource needs of the VMs it is hosting. VMs with resource contentions can be hot migrated to other nodes in the cluster to make sure that enough resources are available.

Hardware requirements

Compute

Google Cloud periodically requests updated validation of partner server platforms with new releases of Anthos through their Anthos Ready platform partner program. A listing of currently validated server platforms and the versions of Anthos supported can be found here.

Operating system

Anthos clusters on VMware can be deployed to both vSphere 7 and 8 environments as chosen by the customer to help match their current datacenter infrastructure.

The following table contains a list vSphere versions that have been used by NetApp and our partners to validate the solution.

Operating System

Release

Anthos Versions

VCenter

8.0.1

1.28

Additional hardware

To complete the deployment of Anthos with NetApp as a fully validated solution, additional data center components for networking and storage have been tested by NetApp and our partner engineers.

The following table includes information about these additional infrastructure components.

Manufacturer Hardware Component

Mellanox

switch (data network)

Cisco

switch (management network)

NetApp

AFF Storage System

Additional software

The following table includes a list of software versions deployed in the validation environment.

Manufacturer

Software Name

Version

NetApp

ONTAP

9.12.1

NetApp

Trident

24.02.0

During the Anthos Ready platform validation performed by NetApp, the lab environment was built based on the following diagram, which allowed us to test multiple scenarios using various NetApp ONTAP storage backends.

Figure showing input/output dialog or representing written content

Network infrastructure support resources

The following infrastructure should be in place prior to the deployment of Anthos:

  • At least one DNS server providing full host-name resolution that is accessible from the in-band management network and the VM network.

  • At least one NTP server that is accessible from the in-band management network and the VM network.

  • A DHCP server available to provide network address leases on demand should clusters need to scale dynamically.

  • (Optional) Outbound internet connectivity for both the in-band management network and the VM network.

Best practices for production deployments

This section lists several best practices that an organization should take into consideration before deploying this solution into production.

Deploy Anthos to an ESXi cluster of at least three nodes

Although it is possible to install Anthos in a vSphere cluster of less than three nodes for demonstration or evaluation purposes, this is not recommended for production workloads. Although two nodes allow for basic HA and fault tolerance, an Anthos cluster configuration must be modified to disable default host affinity, and this deployment method is not supported by Google Cloud.

Configure virtual machine and host affinity

Distributing Anthos cluster nodes across multiple hypervisor nodes can be achieved by enabling VM and host affinity.

Affinity or anti-affinity is a way to define rules for a set of VMs and/or hosts that determine whether the VMs run together on the same host or hosts in the group or on different hosts. It is applied to VMs by creating affinity groups that consist of VMs and/or hosts with a set of identical parameters and conditions. Depending on whether the VMs in an affinity group run on the same host or hosts in the group or separately on different hosts, the parameters of the affinity group can define either positive affinity or negative affinity.

To configure affinity groups, see the appropriate link below for your version of VMWare vSphere.

vSphere 6.7 Documentation: Using DRS Affinity Rules.
vSphere 7.0 Documentation: Using DRS Affinity Rules.

Note Anthos has a config option in each individual cluster.yaml file to automatically create node affinity rules that can be enabled or disabled based on the number of ESXi hosts in your environment.