How to improve DNS resolution performance inside a kubernetes cluster?

Follow
Table of Contents

Task

This article gives a quick overview of DNS resolution inside a Kubernetes cluster. It also explains how performance can be improved.

Pre-requisites

  • A working Kubernetes cluster with CoreDNS or kube-dns installed.

Overview

When a workload is deployed on a Kubernetes cluster, the nameserver (DNS server) is set to the Service IP address of CoreDNS, along with the search options.

Example:

# cat /etc/resolv.conf
nameserver 10.43.0.10
search cattle-system.svc.cluster.local svc.cluster.local cluster.local members.linode.com
options ndots:5
  • The first search option is generally <namespace-where-workload-is-deployed>.svc.<cluster domain>
  • Second option is svc.<cluster domain>
  • Third one being just <cluster domain>
  • The rest of the options are copied over from what's configured on the node. (options from /etc/resolv.conf on the node)

The above options are helpful in the following ways: - When a workload tries to connect to a service in the same namespace, it can simply reference it using the destination service name. Ex: hello-world-service - Similarly, when it tries to connect to a service in a different namespace, it's enough to append the namespace name after the service name. Example: database-service.finance-app

The above conveniences come at a cost, though! Suppose the workload tries to resolve a service name outside of the cluster. In that case, the DNS client first exhausts appending all the options to this name before finally resolving to the correct IP, this is known as search path expansion.

Say the workload needs to resolve login.mycompany.com. The DNS client sends out a query for login.mycompany.com.<namespace-where-workload-is-deployed>.svc.<cluster domain>. The DNS server responds correctly with a nonexistent domain message (NXDOMAIN) as it's an invalid domain.

Next, the DNS client tries to resolve login.mycompany.com.svc.<cluster domain>, followed by login.mycompany.com.<cluster domain>.

As you can see, there are a lot of lookups, on top of this there is an A (IPv4) query, and a quad A AAAA (IPv6) query sent out for each of the above.

All these unnecessary queries add to the network traffic and increase the load on the CoreDNS service.

Resolution

To avoid DNS client cycling through each of the options specified in /etc/resolv.conf, add a trailing dot . to the name that's being resolved outside of the cluster, this ensures the DNS client performs an absolute lookup. Example: login.mycompany.com.

Similarly, when possible, use the full service name (<destination-service-name>.<destination-namespace>.svc.<cluster domain>). As a further recommendation, some clusters can benefit from deploying NodeLocal DNS. This is particularly useful in a production cluster or a cluster with a high frequency of DNS queries.

There are a number of ways that NodeLocal DNS improves on the reliability and scalability of CoreDNS, please see the documentation for details.

Further reading

  • https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/
  • https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/
Was this article helpful?
0 out of 0 found this helpful

Comments

0 comments

Please sign in to leave a comment.