How a ClusterIP Service routes gRPC traffic, and why it sticks to one pod

A ClusterIP Service is a virtual IP, not a proxy. kube-proxy programs the node's kernel, through iptables rules or IPVS virtual servers, to rewrite packets addressed to that IP. The backend pod is chosen when the first packet of a new TCP connection arrives. After that, connection tracking sends every later packet on that connection to the same pod. The kernel balances connections. It never sees requests.

gRPC runs over HTTP/2, which multiplexes many concurrent streams over a single TCP connection. A gRPC channel dialed to the Service's DNS name resolves to one address, the ClusterIP. Under the default pick_first policy, the client opens one connection to that address and keeps it for the life of the process. Every RPC from that client, whether ten per second or ten thousand, rides that one connection to whichever pod kube-proxy selected at dial time. New pods added by a scale-up receive nothing from existing clients.

HTTP/1.1 hides this. A connection carries one in-flight request, so clients keep pools of many connections and recycle them often. Each new connection is a fresh backend selection, which spreads load reasonably well by accident.

Connection-level (ClusterIP + kube-proxy)   Per-RPC (client-side or L7)

client --1 TCP/HTTP2 conn--> VIP            client --RPC 1--> pod-a
          |                                         --RPC 2--> pod-b
          v  (chosen once at dial)                  --RPC 3--> pod-c
        pod-a <- RPC 1, 2, 3 ... N                  --RPC 4--> pod-a
        pod-b    idle
        pod-c    idle

 (A) Connection-level: ClusterIP Service + kube-proxy, pick_first channel

 +-------------+   one TCP/HTTP2 conn    +----------------------------+
 | gRPC client |========================>| ClusterIP (virtual IP)     |
 | RPC 1..N    |   RPC streams muxed     | iptables / IPVS rewrite    |
 +-------------+                         | pod chosen once at dial,   |
                                         | conntrack pins the rest    |
                                         +-------------+--------------+
                                                       |
          +--------------------------------------------+
          v
    +-----------+     +-----------+     +-----------+
    |   pod-a   |     |   pod-b   |     |   pod-c   |
    |  RPC 1..N |     |   idle    |     |   idle    |
    +-----------+     +-----------+     +-----------+

 (B) Per-RPC: client-side round_robin or an L7 proxy

 +-------------+   conn 1: RPC 1, 4, 7 ...    +-----------+
 | gRPC client |----------------------------->|   pod-a   |
 | or L7 proxy |                              +-----------+
 |             |   conn 2: RPC 2, 5, 8 ...    +-----------+
 |             |----------------------------->|   pod-b   |
 |             |                              +-----------+
 |             |   conn 3: RPC 3, 6, 9 ...    +-----------+
 |             |----------------------------->|   pod-c   |
 +-------------+                              +-----------+

Through a ClusterIP Service, kube-proxy picks a pod once per TCP connection, so a gRPC client that multiplexes every RPC over one HTTP/2 connection sends everything to one pod; per-RPC balancing from the client or an L7 proxy spreads RPCs across all pods.

Symptoms and how to confirm the imbalance

The pattern in that diagram shows up in production as uneven load across replicas. One pod runs near its CPU limit while its siblings sit mostly idle, and per-replica request rates differ by multiples, not percentages. Two situations make it obvious:

  • After scale-up or a rolling deploy. New pods receive almost nothing. Existing clients already hold open HTTP/2 connections and have no reason to dial again, so added capacity goes unused until those connections break.
  • With few clients. If ten client pods each hold one connection to a six-replica backend, the best case is still uneven, and random placement often stacks several clients on one pod. Hundreds of clients average the skew out, which is why the problem often appears only on low-fan-out internal services.

To confirm it, compare per-pod numbers. kubectl top pods with a label selector gives a quick view of CPU spread. Request metrics are better evidence: break RPS down by pod name, not by Service, because a Service-level total looks healthy even when one replica handles most of the traffic.

Then count connections directly. Run kubectl exec into each backend pod and use ss to list established TCP connections on the gRPC port. Minimal images often lack ss, so you may need an ephemeral debug container that shares the pod's network namespace. If each pod holds only a handful of long-lived connections and the counts match the RPS skew, you have connection-level balancing. Adding more replicas will not fix it.

#!/usr/bin/env bash
# Compare per-pod CPU and established gRPC connections for one backend.
# Placeholders: adjust namespace, label selector, container name and port.
set -euo pipefail

NS=payments
SELECTOR=app=orders-api
CONTAINER=orders-api
GRPC_PORT=50051

# 1. CPU spread per replica. A Service-level view would hide this.
kubectl top pods -n "$NS" -l "$SELECTOR"

# 2. Established inbound connections on the gRPC port, per pod.
#    Look for a few long-lived connections and counts that match the RPS skew.
for pod in $(kubectl get pods -n "$NS" -l "$SELECTOR" \
    -o jsonpath='{.items[*].metadata.name}'); do
  # The state filter keeps only ESTABLISHED sockets. sport matches the
  # server side of each connection. tail drops the header line.
  count=$(kubectl exec -n "$NS" "$pod" -c "$CONTAINER" -- \
    ss -tn state established "( sport = :$GRPC_PORT )" | tail -n +2 | wc -l)
  echo "$pod $count"
done

# If the image has no ss, attach an ephemeral debug container instead.
# Containers in a pod share its network namespace, so ss there sees the
# app's sockets. Use any image you trust that ships iproute2:
#   kubectl debug -n "$NS" -it <pod> --image=<image-with-iproute2> \
#     --target="$CONTAINER" -- ss -tn state established '( sport = :50051 )'

A shell script that shows CPU per replica with kubectl top pods and counts established gRPC connections in each backend pod with ss, so you can check whether a few long-lived connections explain the load skew.

Client-side balancing with a headless Service and round_robin

The standard fix is to let the client see every pod and spread RPCs across them itself.

Expose pod IPs with a headless Service

Set clusterIP: None on a Service that selects your gRPC pods and exposes the gRPC port. Kubernetes allocates no virtual IP for it, so kube-proxy does no connection-level balancing. Cluster DNS answers a query for the Service name with one A record per ready pod. By default, pods that fail readiness are left out of the answer, which keeps unready backends away from clients.

Point the client at DNS and choose round_robin

The client target must use the DNS resolver, for example dns:///orders.prod.svc.cluster.local:50051, so the gRPC library receives the full address list. In Go, grpc.Dial defaults to the passthrough scheme. That scheme hands the name to the system dialer, which connects to a single address, so the gRPC balancer only ever knows about one backend.

Getting the address list is not enough. The default policy, pick_first, tries addresses in order, keeps the first one that connects, and sends every RPC down that single HTTP/2 connection. The other pods sit idle no matter how many records DNS returns.

To spread load, pass a default service config such as {"loadBalancingConfig": [{"round_robin": {}}]}. In Go, you do this with grpc.WithDefaultServiceConfig. round_robin opens one subchannel per resolved address and rotates RPCs across the subchannels that are ready. Load now spreads per RPC rather than per connection.

# Headless Service for the orders gRPC backend.
# clusterIP: None means no virtual IP and no kube-proxy balancing.
# Cluster DNS returns one A record per ready pod instead.
apiVersion: v1
kind: Service
metadata:
  name: orders
  namespace: prod
spec:
  clusterIP: None
  # Must match the labels on the gRPC server pods.
  selector:
    app: orders
  ports:
    - name: grpc
      port: 50051
      targetPort: 50051
      protocol: TCP
  # This is the default, shown explicitly: pods that fail readiness are
  # left out of DNS answers, so clients never resolve unready backends.
  publishNotReadyAddresses: false

# Client side (Go), shown as a comment because the manifest is YAML:
#
#   conn, err := grpc.Dial(
#       // dns:/// makes grpc-go resolve every A record.
#       // A bare name would use passthrough and dial only one address.
#       "dns:///orders.prod.svc.cluster.local:50051",
#       grpc.WithTransportCredentials(insecure.NewCredentials()),
#       // Override pick_first so each address gets its own subchannel
#       // and RPCs rotate across all ready pods.
#       grpc.WithDefaultServiceConfig(
#           `{"loadBalancingConfig": [{"round_robin": {}}]}`),
#   )

A headless Service that publishes one DNS record per ready pod, with the matching Go client that dials a dns:/// target and selects round_robin through the default service config.

Keeping clients current: DNS re-resolution, max connection age and keepalive

The catch is that round_robin only spreads load across the addresses the resolver last returned. gRPC's DNS resolver does not poll on a timer. It re-resolves mainly when a subchannel fails or a connection closes. During a scale-up nothing fails, so new pods sit idle until something disturbs the existing connections.

Forcing periodic reconnection

The usual fix is on the server. In Go, keepalive.ServerParameters has a MaxConnectionAge field. When a connection reaches that age, the server sends GOAWAY, and the client re-resolves and reconnects. grpc-go adds jitter to this age so that clients do not all reconnect at once. MaxConnectionAgeGrace then sets how long in-flight RPCs may run before the connection is closed forcibly. If the grace period is shorter than your longest streaming RPC, those streams are cut off. Size the grace period for the streams, or have clients resume after an interruption.

The same setting is a cheap partial fix even behind a plain ClusterIP Service. Each reconnect lands on a pod chosen by kube-proxy, so load evens out statistically over time. With only a handful of clients, the spread stays uneven.

Keepalive on both sides

Client keepalive pings detect dead connections. Servers enforce a minimum interval between pings through EnforcementPolicy.MinTime. A client that pings more often than that, or pings without active streams when PermitWithoutStream is false, receives a GOAWAY with "too_many_pings" and is disconnected. Configure the client and server values together. Also check the idle timeouts of any intermediate load balancers or NAT gateways. Keep the client ping interval below those timeouts, or those devices may drop idle connections without telling either side.

import grpc
from concurrent import futures

# Server: recycle connections so clients re-resolve DNS and discover new pods.
server_options = [
    # Send GOAWAY after ~5 minutes. gRPC adds jitter so clients do not reconnect together.
    ("grpc.max_connection_age_ms", 5 * 60 * 1000),
    # Let in-flight RPCs finish before closing. Must exceed your longest stream.
    ("grpc.max_connection_age_grace_ms", 10 * 60 * 1000),
    # Reject client pings closer together than 20s ('too_many_pings' otherwise).
    ("grpc.http2.min_ping_interval_without_data_ms", 20 * 1000),
    # Allow keepalive pings on connections with no active RPCs.
    ("grpc.keepalive_permit_without_calls", 1),
]
server = grpc.server(futures.ThreadPoolExecutor(max_workers=16), options=server_options)

# Client: ping interval must be >= server minimum (20s) and below any
# intermediate LB or NAT idle timeout (assumed here to be 60s).
client_options = [
    ("grpc.keepalive_time_ms", 30 * 1000),
    ("grpc.keepalive_timeout_ms", 10 * 1000),
    # Matches the server's permit_without_calls setting.
    ("grpc.keepalive_permit_without_calls", 1),
    # Spread RPCs across every address the resolver returns.
    ("grpc.service_config", '{"loadBalancingConfig": [{"round_robin": {}}]}'),
]
# A headless Service makes DNS return one A record per pod.
channel = grpc.insecure_channel(
    "dns:///orders-headless.shop.svc.cluster.local:50051",
    options=client_options,
)

Python gRPC server and client options that cap connection age to force periodic GOAWAY and DNS re-resolution, size the grace period for long streams, and pair client keepalive with the server's minimum ping interval.

Alternatives: L7 proxies, service mesh sidecars and xDS-based balancing

If you would rather not push balancing logic into every client, put an HTTP/2-aware proxy between client and server. The proxy terminates the client's connection, keeps its own pool of connections to each backend pod, and makes a routing decision per RPC rather than per connection. The client can keep a single long-lived connection to a plain ClusterIP Service and still get an even spread. This works for any language and any gRPC library, and it can add retries, outlier ejection and per-route policy. The proxy can run as a sidecar next to each pod in a service mesh or as a shared standalone tier. The cost is an extra network hop on every call, more CPU and memory, and one more component that can fail or be misconfigured.

The second alternative is lookaside balancing, usually done today through xDS. Clients subscribe to a control plane that pushes endpoint lists, weights and load-balancing policy. Updates arrive when pods change, not when a DNS TTL expires, so the staleness problems from the previous section mostly go away, and there is still no proxy in the data path. The trade-off is that the control plane becomes a critical dependency. Decide and test what clients do when it is unreachable at startup or stops sending updates. They may keep the last known endpoints, fall back to DNS, or fail.

None of these approaches rebalances a streaming RPC. The balancer picks a backend once, when the stream opens, and that stream stays on that backend until it ends.

When to use each approach and when to avoid it

The right choice depends on three things: how many clients and backends you run, how many languages your clients are written in, and how uneven your traffic is.

Approach Use when Avoid when Main cost
Headless Service + round_robin + max connection age Few client deployments, one or two languages, requests of roughly equal cost Many clients × many pods; request cost or node capacity varies widely Per-client configuration; connection count grows as clients × pods
L7 proxy or mesh sidecar Polyglot clients, or a mesh already running for mTLS and telemetry Small, latency-sensitive services with no other reason to run a mesh Extra hop, CPU and memory per pod, operational surface
xDS-based balancing Large fleets that need weighting, locality or outlier ejection Small teams without a control plane to run Running and securing a control plane

The headless option runs out of room because each client opens a connection to every pod. Take 200 client pods and 50 server pods: that is 10,000 HTTP/2 connections, each with its own keepalive traffic and memory overhead (illustrative estimate). round_robin also counts requests and nothing else. A cheap health check and a ten-second report query count the same, and so do a pod on a saturated node and one on an idle node. If your load looks like that, you need least-request or weighted balancing, which means a proxy or xDS.

Sidecars add a network hop to every call in both directions. For a service whose budget is a few milliseconds, that hop costs too much if the mesh is only there to fix balancing.

Impact on cost and engineering time

Each option shifts cost to a different place, and you should pick deliberately.

Client-side balancing adds no runtime cost, but its configuration is spread across every service, language and team that calls you. If one caller ships without the round_robin policy or with DNS re-resolution disabled, the imbalance comes back and nothing alerts. To prevent this, put the balancing policy, keepalive and retry settings in a shared client library or config template so teams inherit them. Then add dashboards that show requests and CPU per pod. Service-level averages hide a single hot pod.

Proxies and sidecars centralize the policy, but each one consumes CPU and memory in every pod and adds a component that can fail. They also complicate latency debugging, because a slow call now has to be traced through two extra hops.

Max connection age causes reconnection churn, and with TLS every reconnect is a full handshake.

Illustrative estimate (assumed numbers, not a benchmark): handshakes per second ≈ (client pods × connections per client) ÷ max connection age in seconds. With 200 client pods, 4 connections each and a 300-second age, that is 800 ÷ 300 ≈ 2.7 handshakes per second across the fleet. This is usually affordable, but it grows quickly if you shorten the age aggressively.

The biggest time saving comes from confirming the root cause first. Teams that see one saturated pod often raise resource requests or tune HPA thresholds. Adding replicas does not help here, because existing connections stay pinned to the pods they already reach, so the new pods receive little traffic.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to grpc load balancing kubernetes, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.