How does Kubernetes DNS service discovery work?

A Kubernetes Service is a stable name for a changing set of Pods. You define a label selector, and the control plane continuously records which Pods match it in EndpointSlice objects. The cluster DNS server, usually CoreDNS, watches the API server and turns those objects into DNS records. Applications never track Pods directly. They resolve a name.

Each Service gets a name of the form service.namespace.svc.cluster.local, where the cluster domain is configurable. The kubelet writes search domains into every Pod's /etc/resolv.conf, so a client in the same namespace can resolve just service, and one in another namespace can use service.namespace.

ClusterIP vs headless Services

What the name resolves to depends on the Service type:

  • ClusterIP Service: DNS returns a single virtual IP. No interface owns that address. kube-proxy on each node programs iptables or IPVS rules that rewrite connections to a backend Pod. DNS answers stay stable, and per-connection load balancing happens in the kernel.
  • Headless Service (clusterIP: None): DNS returns the individual Pod IPs as multiple A records. Use this when clients must address specific instances, such as StatefulSet members or client-side load balancers.

How readiness decides what gets published

The kubelet runs each container's readiness probe and reports the result as the Pod's Ready condition. The endpoint controller marks endpoints for unready Pods as not ready. Those Pods then drop out of headless DNS answers and kube-proxy routing. A Pod can be running but not receive traffic until its probe passes.

This is the default because it needs nothing beyond standard cluster components. There are no agents to deploy and no separate registry to operate.

# ClusterIP Service: DNS name orders.shop.svc.cluster.local resolves to one virtual IP.
# kube-proxy rewrites connections to that IP into a ready backend Pod.
apiVersion: v1
kind: Service
metadata:
  name: orders
  namespace: shop
spec:
  selector:
    app: orders          # EndpointSlices track Pods carrying this label
  ports:
    - port: 80
      targetPort: 8080
---
# Headless Service: same selector, but DNS returns one A record per ready Pod IP.
apiVersion: v1
kind: Service
metadata:
  name: orders-headless
  namespace: shop
spec:
  clusterIP: None        # no virtual IP; clients see individual Pods
  selector:
    app: orders
  ports:
    - port: 8080
---
# Pod template fragment (inside a Deployment): the kubelet runs this probe.
# Until it succeeds, the Pod is not Ready and is excluded from both Services' traffic.
containers:
  - name: orders
    image: registry.example.com/orders:1.0
    ports:
      - containerPort: 8080
    readinessProbe:
      httpGet:
        path: /healthz
        port: 8080
      periodSeconds: 5
      failureThreshold: 3

A ClusterIP Service and a headless Service selecting the same Pods, plus the readiness probe that decides which Pods are published as ready endpoints.

What does Kubernetes DNS not give you out of the box?

Kubernetes DNS only knows about objects in the API server of the cluster it runs in. A VM running a legacy billing service, a managed database, or a service in another cluster has no record unless someone creates one. The common workarounds all have the same weakness.

Selector-less Services with hand-maintained Endpoints. You create a Service without a selector and write the Endpoints or EndpointSlice objects yourself, listing the VM IPs. DNS then resolves the name as if it were a normal Service. No controller watches those addresses, though. When a VM is replaced, its IP changes, or it fails, the record keeps pointing at the old address until a person or a script updates it. Nothing probes the VM, so a dead backend stays in rotation.

ExternalName Services. These return a CNAME to an outside hostname. That is useful for aliasing, but it adds a second layer of DNS with its own TTLs. It carries no port mapping and has no health information at all. Kubernetes passes the name through and has no knowledge of what sits behind it.

Cross-cluster resolution. Each cluster has its own cluster.local namespace. A Pod in cluster A cannot resolve a Service in cluster B without an extra multi-cluster layer that standard installs do not include.

The underlying reason is architectural. Location and health are derived from Pod state that the kubelet reports to one API server, and readiness probes only run inside that cluster. Anything outside it is static configuration with no health checking behind it.

# Selector-less Service: Kubernetes will not create endpoints for it.
apiVersion: v1
kind: Service
metadata:
  name: billing
  namespace: legacy
spec:
  ports:
    - name: http
      port: 80
      targetPort: 8080
---
# Hand-written EndpointSlice listing the VM IPs.
# No controller watches these VMs. If one is replaced or dies,
# this list stays stale until a person or script edits it.
apiVersion: discovery.k8s.io/v1
kind: EndpointSlice
metadata:
  name: billing-vms
  namespace: legacy
  labels:
    kubernetes.io/service-name: billing             # links the slice to the Service
    endpointslice.kubernetes.io/managed-by: ops-team  # not the built-in controller
addressType: IPv4
ports:
  - name: http
    port: 8080
    protocol: TCP
endpoints:
  - addresses: ["10.20.0.11"]
    conditions:
      ready: true   # static value; nothing probes the VM to change it
  - addresses: ["10.20.0.12"]
    conditions:
      ready: true
---
# ExternalName: DNS returns a CNAME and nothing more.
# No port mapping and no health data. A second DNS layer with its own TTLs.
apiVersion: v1
kind: Service
metadata:
  name: orders-db
  namespace: legacy
spec:
  type: ExternalName
  externalName: orders-db.internal.example.com

A selector-less Service backed by a hand-maintained EndpointSlice, plus an ExternalName Service, showing that both workarounds publish static locations with no health checking.

How does the Consul catalog model services and health?

Consul approaches the problem from the other direction. A Consul agent runs on every node, whether that node is a VM, a bare-metal host, or a Kubernetes worker. Each service instance is registered with its local agent, either through a configuration file or the agent's HTTP API, along with one or more health checks: an HTTP endpoint, a TCP connect, a script, or a TTL that the application must refresh. The agent runs those checks locally and syncs results to the Consul servers. The servers replicate the catalog through Raft, so every server answers from the same agreed state.

Because checks belong to service instances rather than to Pods, a payment service on a VM and the same service in a cluster are governed by identical rules. A failing check marks that instance critical. If gossip detects that a node has stopped responding, every service on that node is marked unhealthy, even if no individual check has reported a failure.

Aspect Kubernetes readiness Consul health check
Who runs it kubelet on the Pod's node Consul agent next to the instance
Scope Pods in one cluster Any registered host or Pod
Effect on ClusterIP DNS Virtual IP is unchanged; the unready Pod is dropped from the endpoints behind it Not applicable
Effect on direct lookups Headless Service records omit unready Pods Critical instances are omitted from DNS and filtered API results

Callers resolve a name such as web.service.consul through Consul's DNS interface, or they query the HTTP API for health-filtered results with metadata and tags. Either way, they get one registry, regardless of where the workload runs.

# Register a service instance on a VM with its local Consul agent.
# The same definition works for an instance on a Kubernetes node.
cat > payments.json <<'EOF'
{
  "service": {
    "name": "payments",
    "id": "payments-vm-01",
    "port": 8080,
    "tags": ["vm", "v2"],
    "check": {
      "http": "",
      "interval": "10s",
      "timeout": "2s"
    }
  }
}
EOF

# The local agent runs the check and syncs results to the servers.
consul services register payments.json

# DNS interface: only instances whose checks are passing are returned.
# SRV records include the port as well as the address.
dig @127.0.0.1 -p 8600 payments.service.consul SRV

# HTTP API: ?passing filters out instances with failing checks.
# The response includes node, tags, metadata and check status.
curl -s ''

# Compare with the unfiltered view, which also lists critical instances.
curl -s ''

Registers a VM-hosted service with an HTTP health check through the local Consul agent, then resolves it through Consul DNS and the health-filtered HTTP API.

How do you integrate Kubernetes DNS and Consul in a mixed environment?

There are three common integration patterns. Each one makes a different system the source of truth.

Kubernetes Services into the Consul catalog

A sync process watches the Kubernetes API and registers each Service in Consul. Kubernetes is the source of truth. VMs and bare-metal workloads can then find in-cluster services through the same Consul lookups they already use. Health for these entries comes from Kubernetes endpoint readiness, not from Consul checks. A synced entry is only as current as the sync loop that produced it.

Consul services into the cluster

The reverse sync creates Kubernetes Services that point at Consul-registered workloads. Pods can then resolve them through cluster DNS like any other Service. Consul is the source of truth. The Kubernetes object is a read-only projection, and anyone who edits it will see their change overwritten.

Forwarding the Consul domain

Cluster DNS gets a stub-domain rule that forwards the Consul domain to Consul's DNS interface. Nothing is copied between systems, so there is nothing to drift. The cost is that every pod lookup for a Consul name now depends on Consul being available and responding quickly. When Consul is slow, application latency rises. When Consul is down, lookups for its names fail, even though cluster-local names still resolve.

Two-way sync failure modes

Running both syncs at once creates three problems:

  • Name collisions. Two systems can claim the same service name. Without a prefix or namespace convention, it is unclear which one wins.
  • Sync lag. A pod can be terminated in Kubernetes while its Consul entry still looks healthy, so traffic keeps going to an endpoint that no longer exists.
  • Unclear ownership. It is not always obvious which side created a registration. A loop can form where each side re-creates entries the other side deletes.

Choose one owner per service and mark synced entries so their origin is visible.

# values.yaml for the Consul Helm chart: bidirectional catalog sync
# with naming rules that keep each side's entries distinguishable.
syncCatalog:
  enabled: true

  # K8s -> Consul: Kubernetes is the source of truth for these entries.
  toConsul: true
  # Prefix every K8s-originated name in Consul so it cannot collide
  # with a VM service that already uses the same name.
  consulPrefix: "k8s-"
  # Append the K8s namespace (e.g. k8s-api-payments) so identically
  # named Services in different namespaces stay distinct.
  addK8SNamespaceSuffix: true
  # Tag synced entries so operators can tell who owns a registration.
  k8sTag: "synced-from-k8s"
  # Only export namespaces that are meant to be reachable from VMs.
  k8sAllowNamespaces: ["payments", "orders"]
  k8sDenyNamespaces: ["kube-system", "kube-public"]

  # Consul -> K8s: Consul is the source of truth; the generated
  # Service objects are projections and manual edits get overwritten.
  toK8S: true
  # Prefix Consul-originated Services in the cluster so they never
  # shadow a native Kubernetes Service of the same name.
  k8sPrefix: "consul-"

Consul Helm chart values that enable sync in both directions and use prefixes, namespace suffixes, tags and namespace filters to avoid name collisions and make ownership of each registration visible.

Why do DNS caching and TTLs break failover in both systems?

DNS was designed for records that change rarely. Service discovery needs records that change within seconds. That mismatch affects both systems.

TTLs are a ceiling only if clients respect them

Both CoreDNS and Consul's DNS interface set a TTL on each answer. Every cache between the server and the application can hold that answer at least that long, and some caches hold it longer. Node-level caches such as systemd-resolved or nscd can add their own caching. Language runtimes do too. The JVM caches lookups according to networkaddress.cache.ttl, and under some configurations it caches them indefinitely. The DNS server can't tell how stale the clients are.

Connections outlive resolution

Most stale-endpoint incidents come from pooled connections, not from DNS caches. HTTP keep-alive pools and gRPC channels resolve once and then reuse the socket. When a Consul health check fails or a headless Service's pod disappears, an established connection doesn't notice until it errors or times out.

No load awareness, and SRV ignored

A DNS answer lists addresses. It carries no information about load or latency. Many clients use the first record, so resolver ordering determines traffic distribution. Most HTTP clients and database drivers also ignore SRV records. Any port information you publish there is lost unless the client explicitly asks for it.

Test re-resolution before an incident

In staging, run the application under steady load, then deregister or kill one endpoint. Capture DNS traffic on the node to confirm the application queries again. Measure how long requests keep hitting the dead address. If the gap exceeds the TTL, look for caching in the runtime or connection pool. Bound connection lifetime with a maximum connection age, and record the measured failover time in your runbook.

What does Consul add to operational cost and on-call load?

Running Consul is a second distributed system with its own failure modes, not a DNS plugin. Before adopting it, count what you will have to page someone for.

Illustrative estimate (assumed numbers, not a benchmark). Assume 3 Consul servers, 40 VM hosts running client agents, and 2 Kubernetes clusters, each with one catalog sync process and one CoreDNS forwarding rule for the .consul domain.

Components = S + H + 2K = 3 + 40 + (2 × 2) = 47, plus one gossip key and roughly S + H + K = 45 ACL tokens to issue and rotate.

Component Count (assumed) On-call responsibility
Server quorum 3 Raft health, snapshots, ordered upgrades
Client agents 40 Agent crashes, version drift, local health checks
Catalog sync 2 Stale or duplicated registrations
DNS forwarding rules 2 Resolution failures when servers are unreachable
ACL tokens + gossip key ~45 + 1 Distribution, rotation, expiry-driven outages

Losing quorum blocks catalog writes, so new instances can't register and health changes stop propagating. Reads may still return a frozen view, depending on consistency settings. Because the catalog spans every environment, one failure degrades discovery everywhere at once. You also own snapshot backups and tested restores, rolling server upgrades, and a security surface that includes the gossip port, the HTTP API and the token store. Scope creep compounds all of this. Mesh, intentions and KV each turn Consul into a dependency for more critical paths.

Kubernetes DNS costs almost nothing inside a cluster, since CoreDNS ships with it. The cost moves to workarounds. Selectorless Services with hand-maintained EndpointSlices, and ExternalName records for VMs, have no health checks. They go stale quietly, and nobody is paged until traffic fails.

When should you use Kubernetes DNS, Consul, or a simpler alternative?

The choice comes down to where your callers and callees live, and how many of them cross a boundary.

Situation Use Who it suits Long-term risk
One cluster, or several clusters that never call each other Kubernetes DNS Teams that are already fully on Kubernetes Low. Nothing extra to run.
VMs and pods call each other in both directions, or multiple clusters share services Consul Platform teams with capacity to operate a stateful control plane Consul becomes load-bearing and hard to remove
A few static external dependencies, such as a managed database or a legacy API Selector-less Service or ExternalName Small teams with a short, stable list of external endpoints Stale entries if the list grows or the endpoints change

The third row is a deliberate trade. With a handful of dependencies, one person can keep the list correct by hand, and that costs less than running a consensus cluster. The approach breaks down once the list changes often enough that manual updates start to lag.

The choice also shapes your migration path. If you adopt Kubernetes DNS alone while important services still run on VMs, teams tend to move those services into clusters early, just to make them reachable. Workloads that aren't ready get migrated under pressure.

Consul creates the opposite pull. VMs become first-class catalog citizens, so the urgency to move them disappears. Legacy hosts can then stay in place for years.

Decide which outcome you want before choosing. If VMs are meant to be temporary, set a retirement date for Consul when you adopt it.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to kubernetes dns vs consul, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.