Platform9 Blog

Kubernetes Service Discovery: Principles in Practice

One of the reasons why Kubernetes is the leading Container Orchestration platform today is because it offers a nice separation of concerns in terms of describing workloads. Essentially, Kubernetes consists of a collection of API-oriented, very fast binaries, that are well-documented and easy to contribute to and build applications upon (more on Kubernetes Architecture here).

The basic building block starts with the Pod, which is just a resource that can be created and destroyed on demand. Because a Pod can be moved or rescheduled to another Node, any internal IPs that this Pod is assigned can change over time.

If we were to connect to this Pod to access our application, it would not work on the next re-deployment. To make a Pod reachable to external networks or clusters without relying on any internal IPs, we need another layer of abstraction. K8s offers that abstraction with what we call a Service Deployment.

Services provide network connectivity to Pods that work uniformly across clusters. Service discovery is the actual process of figuring out how to connect to a service.

In this article, we are going to explore some of the fundamental operations of service discovery in K8s along with some common troubleshooting issues and gotchas that appear in large-scale apps.

Kubernetes Service Discovery Operations

For the following examples, we assume you have a configured Kubernetes cluster. I created one using DigitalOcean with one node. For testing we are using hello-kubernetes image. Let’s get started.

Defining and deploying Services

Let’s first create an initial infrastructure so that we can observe the concepts of services in Kubernetes and how they work in practice/

Create two namespaces, one for develop and one for production:


$ cat << ENDL > develop-namespace.yml
apiVersion: v1
kind: Namespace
metadata:
  name: develop
ENDL

$ cat << ENDL > production-namespace.yml
apiVersion: v1
kind: Namespace
metadata:
  name: production
ENDL

$ kubectl apply -f develop-namespace.yml
namespace/develop created

$ kubectl apply -f production-namespace.yml
namespace/production created

Deploy our example application on both namespaces:


$ cat << ENDL > app-deployment-develop.yml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello
  namespace: develop
spec:
  replicas: 2
  selector:
    matchLabels:
      app: hello
  template:
    metadata:
      labels:
        app: hello
    spec:
      containers:
      - name: hello-kubernetes
        image: paulbouwer/hello-kubernetes:1.5
        ports:
        - containerPort: 8080
ENDL

cat << ENDL > app-deployment-production.yml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello
  namespace: production
spec:
  replicas: 2
  selector:
    matchLabels:
      app: hello
  template:
    metadata:
      labels:
        app: hello
    spec:
      containers:
      - name: hello-kubernetes
        image: paulbouwer/hello-kubernetes:1.5
        ports:
        - containerPort: 8080
ENDL

$ kubectl apply -f app-deployment-develop.yml
$ kubectl apply -f app-deployment-production.yml

Now we can describe the Pods and gather their IP addresses:



$ kubectl describe pods --all-namespaces

...
Name:               hello-f878d7d65-qjmbp
Namespace:          production
IP:                 10.244.0.112
...

In the output above, the IPs for each Pod are internal and specific to each instance. If we were to redeploy the application, it would be assigned a new IP each time.

We now verify that we can ping a Pod within the cluster. Create a temporary Pod in the default namespace and ping that IP address:



$ cat << ENDL >> jumpod.yml
apiVersion: v1
kind: Pod
metadata:
  name: jumpod
  namespace: default
spec:
  containers:
  - name: busybox
    image: busybox:1.28
    command:
      - sleep
      - "3600"
    imagePullPolicy: IfNotPresent
  restartPolicy: Always
ENDL

$ kubectl apply -f jumpod.yml
pod/jumpod created

$ kubectl exec -it jumpod ping 10.244.0.149
PING 10.244.0.149 (10.244.0.149): 56 data bytes

Using the nslookup tool, we can get the FQDN of the Pod:



$ kubectl exec -it jumpod nslookup 10.244.0.149
Server:    10.245.0.10
Address 1: 10.245.0.10 kube-dns.kube-system.svc.cluster.local

Name:      10.244.0.149
Address 1: 10.244.0.149 10-244-0-149.hello.production.svc.cluster.local

Next, we use a service deployment to expose our application to the Internet. Here are examples for each namespace:



$ cat << ENDL > app-service-develop.yml
apiVersion: v1
kind: Service (1)
metadata:
  name: hello
  namespace: develop 
spec:
  type: LoadBalancer (2)
  ports:
  - port: 80 (3)
    targetPort: 8080 
  selector:
    app: hello (4)
ENDL

$ cat << ENDL > app-service-production.yml
apiVersion: v1
kind: Service (1)
metadata:
  name: hello
  namespace: production
spec:
  type: LoadBalancer (2)
  ports:
  - port: 80 (3)
    targetPort: 8080
  selector:
    app: hello (4)
ENDL

$ kubectl apply -f app-service-develop.yml
$ kubectl apply -f app-service-production.yml

Where:

  • (1) We specify the kind of workload to be a Service.
  • (2) We specify the Service type to be a LoadBalancer. This means that we expose the service externally using a cloud provider’s load balancer.  Other options include ClusterIP, which makes the Service only reachable from within the cluster, NodePort, which uses the Node IP and a static port to expose the service outside the cluster, and finally, ExternalName, which maps a Service to a DNS name that we configure elsewhere.
  • (3) This is the port to expose to the external network.
  • (4) This is the name of the deployment that we want to connect to.

Verify that we can visit the public IPs of the services that we deployed:



$ kubectl get services --all-namespaces
NAMESPACE     NAME         TYPE           CLUSTER-IP       EXTERNAL-IP      PORT(S)                  AGE
default       kubernetes   ClusterIP      10.245.0.1                  443/TCP                  150m
develop       hello        LoadBalancer   10.245.150.103   159.65.213.28    80:31129/TCP             64m
kube-system   kube-dns     ClusterIP      10.245.0.10                 53/UDP,53/TCP,9153/TCP   150m
production    hello        LoadBalancer   10.245.143.177   159.65.214.225   80:31705/TCP             58m

If we visit either http://159.65.213.28 or http://159.65.214.225, we should be able to see the application running:

K8s Service Discovery demo

Service Discovery Demo: Access public IPs of deployed service

Note: If we were to use a NodePort Service type instead, then Kubernetes would expose the service in a random port within the range 30000-32767 using the Node’s primary IP address. For example, let’s look at the Droplet that was created when we provisioned the Kubernetes Cluster:

Service Discovery Demo: Random port assignment with NodePort Service

Here, the Node IP is 167.71.141.161, and when we create a NodePort Service, we get the following assignment:



$ kubectl get services --all-namespaces
NAMESPACE     NAME         TYPE           CLUSTER-IP       EXTERNAL-IP      PORT(S)                  AGE
default       kubernetes   ClusterIP      10.245.0.1                  443/TCP                  4h16m
develop       hello        NodePort       10.245.209.176              80:31519/TCP             17s
kube-system   kube-dns     ClusterIP      10.245.0.10                 53/UDP,53/TCP,9153/TCP   4h16m
production    hello        LoadBalancer   10.245.143.177   159.65.214.225   80:31705/TCP             164m

So, based on that information, we can visit http://167.71.141.161:31519/ and access the application again.

Also, note that the ClusterIP that was assigned here for both the develop and production namespaces is the LoadBalancer address. This means that it will balance the requests based on an algorithm (default is Round-robin) and it is only accessible within the cluster. We can verify this using the following commands:



$ kubectl exec -it jumpod nslookup 10.245.150.103
Server:    10.245.0.10
Address 1: 10.245.0.10 kube-dns.kube-system.svc.cluster.local

Name:      10.245.150.103
Address 1: 10.245.150.103 hello.develop.svc.cluster.local

$ nslookup 10.245.150.103
Server:		8.8.8.8
Address:	8.8.8.8#53

** server can't find 103.150.245.10.in-addr.arpa: NXDOMAIN

Additionally, we can access the applications from within the cluster using the following HTTP urls: http://hello.develop or http://hello.production



$ kubectl exec -it jumpod wget -- -O - http://hello.develop
Connecting to hello.develop (10.245.150.103:80)
    
    


…

The previous examples worked because this Kubernetes cluster automatically configures an internal DNS service to provide a lightweight mechanism for service discovery by default.

This is not, however, the only way to do it. We can use environment variables as a last resort, but they are not ideal in cases where the Pod might fail to reach a service that was created after the Pod.

In our example application, Kubernetes exports a set of environment variables on the node where the Pod gets created:



$ kubectl exec -it hello-f878d7d65-2qctb env --namespace=develop
...
HELLO_SERVICE_PORT=80
HELLO_PORT=tcp://10.245.150.103:80
HELLO_SERVICE_HOST=10.245.150.103
HELLO_PORT_80_TCP=tcp://10.245.150.103:80
HELLO_PORT_80_TCP_PROTO=tcp
HELLO_PORT_80_TCP_ADDR=10.245.150.103
HELLO_PORT_80_TCP_PORT=80
NPM_CONFIG_LOGLEVEL=info
NODE_VERSION=8.1.0
YARN_VERSION=0.24.6
HOME=/root

Below we can see the final schematic of the Node and ClusterIP layout:

Service Discovery Demo: Node and ClusterIP layout

Service Discovery Troubleshooting: Common Gotchas and Things to Keep in Mind

Now that you understand the basics of Service Discovery with Kubernetes, let’s look at some of the most common troubleshooting scenarios.

We cannot find the service Environmental Variables

You have created a Pod that needs to access a Service, and you discover that there are no environment variables defined other than the default ones. In this case, you may have to redeploy the Pods again, because the Service will not inject the variables after the Pod is created.

The quickest way to redeploy them is to scale them down and up again:



$ kubectl scale deployment hello --replicas=0 --namespace=develop
$ kubectl scale deployment hello --replicas=2 --namespace=develop

If you inspect the Pod environment again, you should be able to read the service variables from the environment.

DNS Service Discovery is not working

First, make sure that the DNS addon service is installed:



$ kubectl get pods --namespace=kube-system -l k8s-app=kube-dns
NAME                      READY   STATUS    RESTARTS   AGE
coredns-9d6bf9876-lnk5w   1/1     Running   0          174m
coredns-9d6bf9876-mshvs   1/1     Running   0          174m

Then, verify that the service IP address is assigned:



$ kubectl get services --namespace=kube-system -l k8s-app=kube-dns
NAME       TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)                  AGE
kube-dns   ClusterIP   10.245.0.10           53/UDP,53/TCP,9153/TCP   176m

Since DNS is an addon to Kubernetes, it may not have been installed or configured correctly.

Now, when a new Pod is created, Kubernetes adds the following entry in /etc/resolv.conf with the nameserver option to the cluster IP of the kube-dns service and appropriate search options to allow for shorter hostnames:



$ cat /etc/resolv.conf 
nameserver 
search .svc. svc.  
options ndots:5

If we try to nslookup the kubernetes.default name and it fails, then there is a problem with the coredns/kube-dns add-on:

$ nslookup kubernetes.default
Server:    10.32.0.10
Address 1: 10.32.0.10

nslookup: can’t resolve ‘kubernetes.default’

We also need to make sure that the server address matches the kube-dns service IP that we found earlier (for our example 10.245.0.10).

Note: DNS Configuration for multi-cloud Kubernetes (federated) environments is not yet GA. You can look at the relevant kubefed project for more information.

Conclusion

In this article, we have seen how easy it is to expose an application publicly using Kubernetes Services. We also troubleshooted a couple of common scenarios that we may encounter. While the basic concepts are easy to grasp, there is a lot more than that behind the scenes.

The road to migrate an enterprise infrastructure to using Kubernetes is long and full of surprises. If we want a painless alternative, consider using our Managed Kubernetes solution which comes built-in with fully automated day-2 operations, 99.9 SLA for bare-metal or VMs or Kubernetes running on any infrastructure, and Managed Apps for seamless integration with monitoring, logging and other critical Kubernetes services. To test drive the service try the sandbox environment here.

Platform9

Platform9 delivers a SaaS-managed hybrid cloud solution that turns existing infrastructure into a cloud, instantly. We help enterprises drive digital transformation by enabling them to manage VMs, Containers and Serverless Functions on ANY infrastructure — on-premises, in public clouds, or at the edge – with a self-service, simple and unified experience. Customers such as Cadence, Autodesk, Splunk, EBSCO, Bitly, LogMeIn, and Aruba see upwards of 300 percent improvement in IT efficiency, 33 percent faster time to market, and 50-80 percent improvement in data center utilization and cost reduction. The company is headquartered in Sunnyvale, CA, and is backed by Redpoint Ventures, Menlo Ventures, Canvas Ventures, and HPE.
Platform9

You may also enjoy

Kubernetes in the Enterprise – Chapter 7: What You Need to Know About Platform9 Managed Kubernetes

By Vamsi Chemitiganti

Kubernetes Multi-Tenancy Best Practices

By Platform9

The browser you are using is outdated. For the best experience please download or update your browser to one of the following:

Enterprise Kubernetes Guide

Kubernetes

Gorilla Guide to Enterprise Kubernetes

Download Now

You have Successfully Subscribed!