Managing Ingresses on Kubernetes
Table of Contents
Almost everyone who is deploying an application on NetApp Kubernetes Service (NKS) would like the app to be accessible to other people on the public internet.
If you are an independent developer working with a cloud provider, then you’ll ask, what’s my public IP address, and what ports are exposed?
If you are a developer in a corporation with private data center resources, you’ll ask, how do I get the load-balancer rules configured for my set of internal hosts (and whom do I need to talk to)?
If you are working an application deployed inside a Kubernetes cluster, you’ll ask, what’s going on here at all? How do I let people get n?
If you need to expose your Kubernetes services to the world, Ingresses are the way to go. At the time of this writing Ingress is only available in beta, so let’s see the alternatives first.
Exposing services to the world
Your application in a Kubernetes cluster probably contains these items:
You have already put your carefully-designed set of applications into a set of containers-sharing-resources as a Kubernetes pod. Now how do those processes communicate with others?
The pod exposes endpoints usually at the overlay network. That network is not reachable from outside the cluster. You can also use
hostPort to expose the endpoint at the host IP, or set
hostNetwork: true to use the host’s network interface from your pod. However, in both scenarios, you should take extra care to avoid port conflicts at the host, and possibly some issues with packet routing and name resolutions. So let’s assume that we want a risk-free and scalable kubernetes cluster, and we are correctly using pods exposing their functions to the container ports.
Kubernetes services primarily work to interconnect different pod-based applications. They’re designed to help make a microservices system scale well internally. They are not primarily intended for external access, but there are some accepted ways to expose services to external clients.
Let’s simplify services as a routing, balancing and discovery mechanism for the pod’s endpoints. Services target pods using selectors, and can map container ports to service ports. A service exposes one or more ports, although you usually find that only one is defined.
A service can be exposed using 3
ClusterIP: The default, only exposed within the cluster.
NodePort: Uses a port from a cluster defined range (usually 30000-32767) at every cluster node to expose the service.
LoadBalancer: Setting your cluster kubelet with the parameter
--cloud-provider=the-cloud-providerbrings some goodies, including the ability to use the provider’s load balancer.
If we choose
NodePort to expose our services, we still need an external proxy that uses DNAT to expose more friendly ports. That element also needs to balance on the cluster nodes, which in turn balance again through the service to the internal pods. There is no easy way of adding TLS or simple host header routing rules to the external service.
LoadBalancer is probably the easiest of all methods to get your service exposed to the internet. The problem is that there is no standard way of telling a Kubernetes service about the elements that a balancer requires, again TLS and host headers are left out.
Endpoints are usually automatically created by services unless you are using headless services and adding the endpoints manually. An endpoint is a host:port tuple registered at Kubernetes, and in the service context it is used to route traffic. The service tracks the endpoints as pods that match the selector are created, deleted and modified. Individually, endpoints are not useful to expose services, since they are to some extent ephemeral objects.
If you can rely on your cloud provider to correctly implement the LoadBalancer for their API, to keep up-to-date with Kubernetes releases, and you are happy with their management interfaces for DNS and certificates, then setting up your services as type
LoadBalanceris quite acceptable. On the other hand, if you want to manage load balancing systems manually and set up port mappings yourself,
NodePortis a low-complexity solution. If you are directly using
Endpointsto expose external traffic, perhaps you already know what you are doing (but consider that you might have made a mistake, there could be another option).
Given that none of these elements has been designed to expose services to the internet, their functionality may seem limited for this purpose.
The Ingress resource is a set of rules that map to Kubernetes services. Sure, there is a more complex definition, and I’ll be missing some points, but I still think that "rules to services mapping" is the best way to understand what an Ingress does.
Ingress resources are defined purely within Kubernetes as an object that other entities can watch and respond to.
Supported rules at this beta stage are:
host header: So we can forward traffic based on domain names.
paths: Looks for a match at the beginning of the path.
TLS: If the ingress adds TLS, it uses HTTPS and a certificate configured through a secret.
When an ingress does not include host header rules, requests without a match use that Ingress and are mapped to the backend service. A typical example is using a 404 page to respond to requests for unmatched sites/paths.
Ingress tries to match requests to rules and forwards them to backends, which are composed of a service and a port (remember that a service can contain multiple ports.
To summarize: An Ingress defines how to take a request and (based on host/path/tls) send it to a backend.
Introducing Ingress Controllers
To grant (or remove) access, an entity must be watching and responding to changes in the services, pods, and Ingresses. That entity is the Ingress controller. While the Ingress controller does work directly with the Kubernetes API, watching for state changes, it is not coupled as tightly to the Kubernetes source code as the cloud-provider LoadBalancer implementations.
Ingress controllers are applications that watch Ingresses in the cluster and configure a balancer to apply those rules. Typically you use an existing Ingress controller that controls a third party balancer like HAProxy, NGINX, Vulcand or Traefik, updating configuration as the Ingress and their underlying elements change.
Ingress controllers usually track and communicate with endpoints behind services instead of using services directly. This way some network plumbing is avoided, and we can also manage the balancing strategy from the balancer.
Ingress Controller extensions
As of this writing, Ingress doesn’t support TCP balancing, balancing policies, rewriting, SNI, and many other common balancer configuration parameters. Ingress controller developers have extended the Ingress definition using annotations, but be aware that those annotations are bound to the controller implementation. The Ingress resource is continually evolving, which might make some of these annotations obsolete in the future.
This example demonstrates Ingress usage. In the cluster, create:
A backend to receive requests for
A pair of backends to receive requests for
One whose path begins with
One whose path begins with
A default backend that shows a 404 page.
You can use any website or service in a container as a backend. Create three backends.
echoheaders: pod/service which contains a webpage that shows information about the received request.
--- apiVersion: v1 kind: ReplicationController metadata: name: echoheaders spec: replicas: 1 template: metadata: labels: app: echoheaders spec: containers: - name: echoheaders image: gcr.io/google_containers/echoserver:1.4 ports: - containerPort: 8080 readinessProbe: httpGet: path: /healthz port: 8080 periodSeconds: 1 timeoutSeconds: 1 successThreshold: 1 failureThreshold: 10 --- apiVersion: v1 kind: Service metadata: name: echoheaders labels: app: echoheaders spec: ports: - port: 80 targetPort: 8080 protocol: TCP name: http selector: app: echoheaders
default-http-backend: pod/service which is a simple 404 static page.
--- apiVersion: v1 kind: ReplicationController metadata: name: default-http-backend spec: replicas: 2 selector: app: default-http-backend template: metadata: labels: app: default-http-backend spec: terminationGracePeriodSeconds: 60 containers: - name: default-http-backend # Any image is permissable as long as: # 1. It serves a 404 page at / # 2. It serves 200 on a /healthz endpoint image: gcr.io/google_containers/defaultbackend:1.0 livenessProbe: httpGet: path: /healthz port: 8080 scheme: HTTP initialDelaySeconds: 30 timeoutSeconds: 5 ports: - containerPort: 8080 resources: limits: cpu: 10m memory: 20Mi requests: cpu: 10m memory: 20Mi --- apiVersion: v1 kind: Service metadata: name: default-http-backend labels: app: default-http-backend spec: ports: - port: 80 targetPort: 8080 protocol: TCP name: http selector: app: default-http-backend
game2048: A pod/service which contains a static web page game
--- apiVersion: v1 kind: ReplicationController metadata: name: game2048 labels: name: game2048 spec: replicas: 2 selector: name: game2048 template: metadata: labels: name: game2048 version: stable spec: containers: - name: game2048 image: alexwhen/docker-2048 ports: - containerPort: 80 --- apiVersion: v1 kind: Service metadata: name: game2048 labels: name: game2048 spec: ports: - port: 80 targetPort: 80 selector: name: game2048
Create the backends of your choice with
kubectl create -f …
We only use standard functionality for this example, so you should be able to use any Ingress controller. I’m using the HAProxy Ingress controller, which is included by default with every NKS cluster build at http://netapp.io.
The HAProxy Ingress controller at NKS is already configured and listening to changes in your Kubernetes cluster. If you prefer to get NGINX, you can also deploy it in any cluster following these instructions.
Let’s start by creating the
game2048 service at the balancer.
--- apiVersion: extensions/v1beta1 kind: Ingress metadata: name: game-ingress spec: rules: - host: game.domain1.io http: paths: - path: backend: serviceName: game2048 servicePort: 80
That’s it. We need now to reach HAPRoxy node using the host header
Use a domain name. Point the A record to the HAProxy node’s IP address.
curl -H "Host: game.domain1.io" real.server.address
Install a browser plugin to add host header, like Virtual-Hosts for Chrome.
If you use a browser, it shows the 2048 game. However, if you try the IP address without that host header, you get a 503 error.
We’ll be fixing that with our next Ingress, the default backend:
apiVersion: extensions/v1beta1 kind: Ingress metadata: name: default-http-backend spec: backend: serviceName: default-http-backend servicePort: 80
default-http-backend serves every request that doesn’t match an existing rule. Now if you try the IP address without the game host header, you should get a simple 404 page.
Finally, let’s use the flexible
echoheaders service to add some path matching:
--- apiVersion: extensions/v1beta1 kind: Ingress metadata: name: echoheaders spec: rules: - host: domain2.io http: paths: - path: /path1 backend: serviceName: echoheaders servicePort: 80 - path: /path2 backend: serviceName: echoheaders servicePort: 80
In a real-world scenario, you use different services for each path, but for us to test it’s ok to use the echoheaders service.
To test all Ingresses try these requests:
balancer node IP -→ will be sent to
domain1.io -→ will be sent to
domain1.io/path1 -→ will be sent to
game2048(and it will fail with 404, since game2048 has no resource at that path)
domain2.io/path1 -→ will be sent to
domain2.io/path2 -→ will be sent to
domain2.io/path3 -→ will be sent to
domain2.io -→ will be sent to
Ingresses are simple and very easy to deploy, and really fun to play with. However, when you plan your Ingresses for production some other factors arise: - High Availability - SSL - Custom configuration - Troubleshooting
However, we leave those questions for a future article.