What service discovery is and why every microservice system needs it

In a microservice system, the address of any given service is temporary. Autoscalers add and remove instances, rolling deploys replace every pod or VM in a fleet, and failed hosts disappear without warning. Hardcoded IPs or static config files go stale on the first deploy. Service discovery is the mechanism that keeps a current list of healthy endpoints for each service, so a caller can reach "payments" without knowing which machines are running it right now.

Every discovery mechanism has three parts:

  • Registration: Something records that an instance exists and where it listens. The instance can register itself, or the platform can observe it.
  • Health determination: Something decides whether a registered instance should receive traffic. Without this step, the list fills with dead or still-starting instances.
  • Lookup: A caller turns a logical name into one or more concrete addresses.

The main design choice is where lookup happens. In server-side discovery, the caller sends traffic to a stable address, such as a load balancer, proxy or virtual IP. That component resolves the endpoints and picks one. In client-side discovery, the caller queries a service registry, gets the endpoint list, and chooses an instance itself. This gives the caller control over load balancing and retries, but every client library has to implement that logic correctly.

The two approaches split ownership differently. Platform engineers run the registry, DNS or orchestrator, such as Kubernetes, which provides discovery and load balancing natively. Backend engineers write the callers, and callers decide how often addresses are refreshed, how failures are retried and whether stale endpoints are ever dropped.

  SERVER-SIDE DISCOVERY                 CLIENT-SIDE DISCOVERY
  +------------------+                  +----------------------------+
  | Caller           |                  | Caller + client library    |
  +------------------+                  | (picks instance, retries)  |
           |                            +----------------------------+
           | "payments"                    |       ^              |
           v                     3. lookup |       | [A, B]       |
  +------------------+                     v       |              |
  | LB / proxy / VIP | 3. resolve +--------------------+          |
  |                  |----------->|  Service registry  |          |
  |                  |<-----------| payments: A, B     |          |
  +------------------+   [A, B]   | C failing: removed |          |
           |                      +--------------------+          |
           | 4. forward to A          ^           |               |
           |              1. register |  2. health|               | 4. call B
           v                          |     checks v               v directly
  +----------------------------------------------------------------------+
  |  payments instances                                                  |
  |  +-------------+      +-------------+      +-------------+           |
  |  | A  healthy  |      | B  healthy  |      | C  failing  |           |
  |  +-------------+      +-------------+      +-------------+           |
  +----------------------------------------------------------------------+
  Owners: backend engineers write the callers; platform engineers run
  the registry, DNS or orchestrator (e.g. Kubernetes) and the LB/proxy.

Instances register and are health-checked so the registry lists only A and B. In server-side discovery a load balancer, proxy or virtual IP resolves the endpoints and forwards the request, while in client-side discovery the caller looks up the endpoint list and calls an instance directly.

How service discovery works, step by step

Whatever the implementation, the lifecycle of one instance and one request follows the same sequence:

  1. An instance starts and begins listening on an address and port.
  2. Registration happens. Either the instance announces itself to a registry, or a platform component that launched it (a scheduler, orchestrator or deploy agent) records the address on its behalf.
  3. Health checks begin. The registry or platform probes the instance, or waits for heartbeats, and marks it eligible for traffic only once it passes.
  4. A caller looks up a logical name, such as payments. Depending on the design, the answer is a stable virtual address (server-side) or a list of healthy instance addresses (client-side).
  5. A target is chosen. A load balancer, proxy or client library picks one instance, typically by round-robin, random choice or a weighted policy.
  6. The caller caches the result. DNS TTLs, client caches and open connections all mean the caller does not repeat the lookup for every request.
  7. The instance leaves. On a clean shutdown it is deregistered or marked unready before it stops serving. On a crash, a failed health check or an expired lease removes it.

Most production problems with discovery come from steps 6 and 7: callers holding on to addresses after they stop being valid, and instances disappearing before callers notice. The rest of this article looks at how DNS, registries and Kubernetes handle each step, and where each one leaks.

Who should use it

Service discovery matters most to teams in these situations:

  • Teams running more than a handful of services that call each other. Once services call each other by name and instances change on every deploy, manually maintained address lists stop scaling.
  • Systems with autoscaling or frequent deploys. If instance addresses change daily or hourly, something has to track them automatically.
  • Platform and infrastructure engineers, who choose and operate the mechanism: the cluster DNS, the registry, the proxies and the health-check conventions.
  • Backend engineers writing service-to-service clients, who need to understand how their HTTP, gRPC or database clients resolve names and reuse connections, because that determines whether discovery changes actually reach their code.
  • Organisations spanning several clusters, VMs or environments, where services in one place need to find services in another.

Teams that can usually skip a dedicated mechanism: a single application behind one load balancer, a small set of services on fixed hosts that rarely change, or a monolith with a few external dependencies. For them, a static config file or a stable DNS name pointing at a load balancer is simpler and does the job. If you already run on Kubernetes, you are using service discovery whether you planned to or not. The question then is whether its in-cluster behaviour is enough, not whether to adopt discovery at all.

How DNS discovery works and where it goes stale

DNS is the oldest discovery mechanism and still the most common. An A record maps a name to IPv4 addresses and an AAAA record to IPv6 addresses. Neither carries a port, so the caller must already know it. An SRV record fills that gap: it returns a priority, a weight, a port and a target hostname, which then resolves through A/AAAA. That lets you run several instances on one host on different ports. The catch is that many HTTP clients never query SRV at all.

Caching happens in layers

Each record has a TTL, and every layer between the authoritative server and your code may cache it. Recursive resolvers honour the TTL. A local cache such as systemd-resolved or nscd may add its own. Language runtimes can cache too: the JVM, for example, keeps its own lookup cache controlled by the networkaddress.cache.ttl security property.

The bigger problem sits above the resolver. A connection pool resolves a name when it opens a connection and then reuses that connection. A long-lived HTTP/2 or gRPC channel may never resolve again. A 5-second TTL has no effect if the socket stays open for days.

Choosing a TTL

Short TTLs shrink the staleness window but raise query volume on resolvers. Long TTLs cut that load but keep dead addresses in circulation longer. Use dig +noall +answer orders.internal A to see the remaining TTL on cached answers. Use dig SRV _http._tcp.orders.internal to check ports and weights.

#!/usr/bin/env bash
NAME=orders.internal
RESOLVER=10.0.0.2

dig @"$RESOLVER" +noall +answer "$NAME" A
dig @"$RESOLVER" +noall +answer "$NAME" AAAA

sleep 5
dig @"$RESOLVER" +noall +answer "$NAME" A

dig @"$RESOLVER" +noall +answer SRV "_http._tcp.$NAME"

getent ahosts "$NAME"

ss -tn state established '( dport = :8080 )'

Inspects each caching layer in turn: the recursive resolver's TTL, SRV port data, the OS resolver path, and open connections that keep using old addresses whatever the TTL is.

Service registries: registration models, leases and health checks

A registry replaces DNS records with an explicit, queryable list of live instances. Each design decision is about who writes to that list and how quickly bad entries are removed.

Registration models

In self-registration, each instance registers itself on startup and then renews a lease through heartbeats. In third-party registration, a platform component that already watches process lifecycle writes the entries. A scheduler or deploy agent is a typical example. The application stays unaware of the registry.

Aspect Self-registration Third-party registration
Who writes entries The instance Platform component
App code required Registry client in every service None
Crash without deregistering Entry lives until lease expires Removed when the watcher notices
Hung process Can keep renewing its lease Depends on the health check used

Leases bound staleness, not correctness

An instance that is killed or loses power never deregisters. Lease expiry limits the damage. With a 30-second lease, a dead entry survives for at most about 30 seconds plus client cache time. That figure is an illustrative estimate. However, a lease only proves that the heartbeat thread is alive. A process that is deadlocked on a database pool can still renew forever. Tie renewal to a real readiness check, not a timer.

The registry is critical infrastructure

If clients query the registry on every call, an outage becomes a total outage. Clients should keep a last-known-good cache and keep using it when the registry is unreachable. During a long outage or network partition, that cache drifts from reality. Pair it with retries and outlier ejection.

Client-side discovery also costs money and effort. Every language needs a maintained library or a sidecar. Load-balancing bugs get fixed in one runtime but not the others.

How Kubernetes Services implement discovery

A Kubernetes Service gives a group of Pods one stable identity. Its label selector, such as app: orders, defines which Pods belong to it. The cluster gives the Service a virtual IP (the ClusterIP) and a DNS name of the form orders.payments.svc.cluster.local. Clients resolve that name once and get an address that never changes, even as Pods are replaced. This avoids the stale-cache problem that plain DNS discovery has.

The control plane keeps the Service's backends current by watching Pods that match the selector. It records only the Ready ones in EndpointSlice objects. On every node, kube-proxy watches those slices and programs packet-forwarding rules. Each new connection to the ClusterIP goes to one ready Pod. Balancing happens per connection, not per request.

A headless Service sets clusterIP: None. DNS then returns the Pod IPs directly, which lets a client library or gRPC resolver do its own client-side balancing.

Liveness and readiness probes do different jobs. A failed liveness probe restarts the container. A failed readiness probe keeps the container running but removes it from the endpoints, so the Pod is not advertised until it can serve (Kubernetes overview).

Probe design decides whether this mechanism helps you or hurts you:

  • Too strict: If a readiness check also checks a shared database, one database slowdown marks every replica unready at the same moment. The Service is left with zero endpoints, and a partial degradation becomes a full outage.
  • Too lenient: If a check only confirms that the process answers HTTP, a Pod with exhausted connection pools or failed config stays in rotation and keeps receiving traffic it can't handle.

The best readiness check tests what the instance can control: whether it has finished warming up, has spare capacity and has its local dependencies ready.

apiVersion: v1
kind: Service
metadata:
  name: orders               # DNS: orders.payments.svc.cluster.local
  namespace: payments
spec:
  selector:
    app: orders              # Ready Pods with this label become backends
  ports:
    - port: 80               # port clients use on the ClusterIP
      targetPort: 8080       # port the container listens on
---
apiVersion: v1
kind: Service
metadata:
  name: orders-headless
  namespace: payments
spec:
  clusterIP: None            # no virtual IP; DNS returns ready Pod IPs
  selector:
    app: orders
  ports:
    - port: 8080
---
containers:
  - name: orders
    image: registry.example.com/orders:1.4.2
    readinessProbe:          # failure removes the Pod from EndpointSlices
      httpGet:
        path: /ready         # checks local state (pools, config), not the shared DB
        port: 8080
      periodSeconds: 5
      failureThreshold: 3    # tolerate brief blips before pulling the Pod
    livenessProbe:           # failure restarts the container
      httpGet:
        path: /healthz       # only "is the process stuck?"
        port: 8080

A ClusterIP Service, a headless Service for client-side balancing, and a container with separate readiness and liveness probes, where the readiness check tests only what the instance controls.

Long-lived connections, rolling deploys and graceful shutdown

A ClusterIP Service balances connections, not requests. kube-proxy picks a backend when a TCP connection opens. After that, every request on that connection goes to the same Pod. gRPC and HTTP/2 multiplex many requests over a single long-lived connection, and database pools keep connections open for hours. So a client that connected before a scale-out never reaches the new Pods. Load stays skewed until connections happen to reset.

Mitigations

  • Client-side balancing over a headless Service. A headless Service (clusterIP: None) returns the Pod IPs directly in DNS. A gRPC client using a round-robin policy can then open a connection to each Pod and spread requests across them. The client must re-resolve DNS periodically to see endpoint changes.
  • A per-request proxy. A sidecar or node-level L7 proxy terminates HTTP/2 and balances each request, so the application needs no balancing logic.
  • Bounded connection lifetime. Set a maximum connection age on servers and pool lifetimes on clients. Connections then recycle and rebalance over time.

Draining without dropped requests

When a Pod is deleted, two things start at the same time. The kubelet sends SIGTERM, and the control plane removes the Pod from the Service's endpoints. Endpoint removal must reach kube-proxy on every node, plus any client caches, so it often finishes after the process has already exited. Requests that arrive in that gap get connection errors.

The usual fix is a short sleep in a preStop hook, so the process keeps serving while the endpoint removal propagates. terminationGracePeriodSeconds must cover both that delay and the application's own drain time, because the grace period includes the preStop hook.

The draining sequence:

  1. The preStop sleep runs, and the Pod is removed from endpoints during it.
  2. SIGTERM arrives.
  3. The server stops accepting new connections.
  4. HTTP/2 servers send GOAWAY so clients move to other Pods.
  5. The server finishes in-flight requests and exits.
  6. If the process is still running when the grace period ends, the kubelet sends SIGKILL.
apiVersion: apps/v1
kind: Deployment
metadata:
  name: orders-api
spec:
  replicas: 3
  selector:
    matchLabels:
      app: orders-api
  template:
    metadata:
      labels:
        app: orders-api
    spec:
      # Must cover the preStop sleep (10s) plus the app's own drain time.
      # The countdown starts before preStop runs, so the hook consumes part of it.
      terminationGracePeriodSeconds: 45
      containers:
        - name: orders-api
          image: registry.example.com/orders-api:1.4.2
          ports:
            - containerPort: 8080
          readinessProbe:
            httpGet:
              path: /healthz/ready
              port: 8080
            periodSeconds: 5
          lifecycle:
            preStop:
              exec:
                # Keep serving while endpoint removal reaches kube-proxy and
                # client caches. SIGTERM is sent only after this hook returns.
                # Requires a sleep binary in the image.
                command: ["sleep", "10"]
          # On SIGTERM the app should stop accepting new connections, send
          # HTTP/2 GOAWAY, finish in-flight requests, close DB pools, then exit.
          # Anything still running when the grace period ends gets SIGKILL.

A Deployment that delays SIGTERM with a 10-second preStop sleep so endpoint removal can propagate, and sets terminationGracePeriodSeconds to 45 so the sleep and the application's drain both finish before the kubelet sends SIGKILL.

Discovery across clusters, VMs and environments

A name like orders.payments.svc.cluster.local only means something inside the cluster that issued it. Pods resolve it because the kubelet points their resolver at the cluster DNS server and adds search domains. A VM or a Pod in another cluster queries a different resolver, which has no authority for that zone. Even if the name did resolve, a ClusterIP is a virtual address that node-level proxy rules implement, so it is not routable from outside the cluster's nodes.

There are three common ways to bridge that gap.

Shared registry

Every cluster and VM registers its endpoints in one registry, and clients query that registry directly. You get a single view of all instances, including Pod IPs if the network is flat enough to reach them.

The cost is that every workload now depends on the registry's availability and on a sync agent in each cluster. A registry outage, or a partition between a site and the registry, affects every environment at once. You have made the failure domain global.

DNS delegation

You give each cluster or site its own subzone, such as east.internal, and have the corporate resolvers forward or delegate queries for that subzone to a DNS endpoint the cluster exposes. Clients keep using plain DNS, and each cluster stays authoritative for its own names, so failures stay local to that cluster. The weakness is the one covered earlier: answers are only as fresh as their TTLs and caches allow.

Gateways

You expose selected services through an ingress or east-west gateway with a stable address. Consumers then discover only the gateway, not individual instances. This gives the loosest coupling and a clear policy boundary. The gateway, however, becomes a shared hop that adds latency and failure risk for all cross-boundary traffic.

Alternatives to dynamic service discovery

DNS, registries and Kubernetes Services are the main discovery mechanisms, but some systems avoid per-instance discovery altogether or push it into a different layer. These are the realistic alternatives:

  • Static configuration. Addresses live in config files or environment variables and change only through a config change and redeploy. This works when the set of hosts is small and stable. It fails as soon as instances are replaced automatically, because every replacement needs a config update.
  • A fixed load balancer per service. Each service gets one long-lived load balancer address, and callers only ever know that address. The load balancer's own target registration still has to track instances, so discovery has moved rather than disappeared. It costs an extra hop and a component to run for each service.
  • A service mesh or sidecar proxy layer. Each instance gets a local proxy that handles lookup, per-request load balancing, retries and often mutual TLS. The application calls localhost or a logical name and the proxy does the rest. This removes balancing logic from every language's client library, but adds a control plane to operate and a proxy hop on every call.
  • Message brokers and queues. If services communicate asynchronously through a broker, producers and consumers only need to find the broker, not each other. This removes point-to-point discovery for those flows but changes the communication model, so it is not a drop-in replacement for request/response calls.
  • Gateways at boundaries. As described in the previous section, exposing services through a gateway means consumers outside a cluster discover one stable address instead of individual instances.

None of these removes the underlying need to track which instances are alive. Each one moves that responsibility into a component other than the caller, which is often the right trade when caller code is spread across many languages and teams.

Choosing an approach: when to use, when to avoid, and cost impact

That trade-off applies to every option here: each one gives up some freshness, reach or simplicity to get something else.

Approach Use when Avoid when Development cost Operational cost
Plain DNS The instance set is stable and clients re-resolve Clients cache resolved addresses indefinitely Low Low, but staleness is hard to see
Kubernetes Services Traffic stays inside one cluster Cross-cluster calls, or multiplexed connections that pin to one backend Low Included in the cluster
Registry / client-side discovery You need metadata, reach across environments or per-request control Nobody has time to run and upgrade it High: client libraries in every language High: the registry needs its own availability plan

DNS and Services cost little because the hard parts are in infrastructure you already run. A registry moves load balancing logic into every client. You pay for that in each language you support, and the registry can take down every caller when it fails.

How service discovery saves engineering time

The time savings come from work that stops being manual and from failures that become easier to diagnose:

  • No address bookkeeping. Without discovery, every scale-out, host replacement or new environment means editing config and redeploying callers. With discovery, new instances become reachable once they register and pass health checks, and nobody files a ticket to update an IP list.
  • Deploys without coordination. Readiness-based endpoint management and graceful drain let a rolling deploy replace every instance without callers changing anything. Teams can ship independently instead of scheduling deploys around each other.
  • Automatic removal of failed instances. Health checks and lease expiry take broken instances out of rotation without an on-call engineer editing load balancer pools by hand.
  • One naming convention across environments. When orders resolves correctly in development, staging and production, the same code and config run everywhere. That removes a class of environment-specific bugs and the time spent chasing them.
  • Faster diagnosis of deploy errors. Most discovery-related incidents are staleness problems: cached DNS answers, pinned connections or missing drain time. The measurements below turn those into a specific, measurable gap instead of an open-ended investigation.

The savings depend on choosing the mechanism that fits the system. Running a registry that a single-cluster deployment doesn't need adds operational work rather than removing it.

Whichever option you choose, three checks show whether it behaves as expected:

  • How often clients actually re-resolve.
  • How long terminated instances keep receiving traffic during a deploy.
  • Whether in-flight requests finish during graceful drain.

These checks turn vague "errors during deploys" tickets into a specific, measured gap.

Illustrative estimate (assumed numbers, not a benchmark): time saved = (incidents per quarter × hours per investigation) − hours to build the checks. With 3 deploy-related incidents, 8 engineer-hours each and 10 hours of setup, the result is (3 × 8) − 10 = 14 hours saved in the first quarter. Every later quarter saves the full 24 hours.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to service discovery, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.