WorksOnMyCluster

The Silent DNS Saturation

Intermittent Payment 502s With Every Pod Reporting Healthy

SEV-1 Kubernetes ~35m

The situation you’re stepping into

You’re on the Platform Reliability rotation for a multi-tenant, cloud-native payments platform on AWS EKS. Incident INC-2026-022 was just raised as SEV-1 (revenue impact).

The platform is a handful of services in the core-banking namespace, all talking to each other over internal cluster DNS:

  • checkout-gateway (public-facing, 3 replicas)
  • payment-reconciliation (batch)
  • risk-engine (2 replicas)
  • notification-service (2 replicas)
  • internal-auth-api

Every service resolves its peers through service-name.namespace.svc.cluster.local. The cluster is deliberately vanilla: 2 × t3.small nodes, CoreDNS with 2 replicas on default resources, kube-proxy on all nodes, VPC CNI. Crucially — no HPA on CoreDNS, and no NodeLocal DNS cache. At peak it does ~3,000 transactions/minute.

What the pager and dashboards say

At 14:05 IST:

  • checkout-gateway starts returning intermittent HTTP 502/504.
  • Payment processing latency jumps from ~90 ms to over 1.8 s.
  • External load-balancer health checks intermittently flap.
  • Some curls from inside pods time out.

And yet every “is it the usual suspect?” check comes back clean:

kubectl get pods -n core-banking     # all Running
                                     # no OOMKilled, no Evicted
kubectl get nodes                    # all Ready
# AWS: no NLB/ALB issues, no EC2 status-check failures
# No application deployment occurred. Some pods show NO errors in logs at all.
🔥 Stakes

The blast radius is cross-service: public API 502s, internal API timeouts, background jobs piling up retries — while node health, control plane, and application pods all read healthy. That combination (“everything is broken, nothing is unhealthy”) is the signature of a systemic fault, not an isolated one.

?
Decision Point 1

Multiple unrelated services degrade at the same instant, but no single service, node, or pod looks unhealthy and nothing deployed. What class of dependency can fail 'everywhere at once' without any one component showing an error?

Ask what sits in the request path of EVERY service-to-service call, before the call even reaches the target service.

Commit to your answer, then reveal the responder’s move →
✓What the responder did & why

A simultaneous, cross-service degradation with healthy components points at a shared dependency in the common path — not at any individual service. In this architecture that dependency is cluster DNS:

  • checkout-gateway → risk-engine requires a DNS lookup first.
  • checkout-gateway → payment-reconciliation requires a DNS lookup first.
  • risk-engine → internal-auth-api requires a DNS lookup first.

If DNS slows or intermittently fails, every service slows together, calls hang before they ever reach the target, and — critically — the target app logs nothing, because the request never arrived. That’s exactly why “some pods show no errors in logs.” You stop treating this as five separate service incidents and start treating it as one DNS incident.

🧭 Why the LB health checks flap too

The external load balancer health-checks the gateway; the gateway’s readiness/health path itself often triggers a name resolution. When DNS is intermittently slow, health checks cross their timeout threshold sometimes, so targets flap in and out of the pool — producing the 502/504s at the edge without any pod actually crashing.

Proving it’s DNS, not the app

Theories are cheap; you isolate. From a debug pod (or an existing pod) you compare name resolution against a direct IP call — because if the latency is in resolution, a direct-IP call will be fast while the named call hangs.

kubectl -n core-banking exec -it deploy/checkout-gateway -- sh

# Time a cluster DNS lookup — is resolution itself slow / failing?
time nslookup risk-engine.core-banking.svc.cluster.local
dig +stats risk-engine.core-banking.svc.cluster.local

# Named call vs. direct pod-IP call
time curl -s http://risk-engine.core-banking.svc.cluster.local/health   # hangs / slow
time curl -s http://10.0.3.47/health                                    # fast
?
Decision Point 2

The named curl hangs but the direct pod-IP curl returns instantly, and nslookup sometimes takes seconds or returns SERVFAIL. What does that prove, and where do you go next?

If the connection is fast once you skip resolution, the fault is upstream of the app — in whatever answers the name query.

Commit to your answer, then reveal the responder’s move →
✓What the responder did & why

It proves the latency is in resolution, before the TCP connection — the application and the network path to the pod are fine. The component that answers those queries is CoreDNS in kube-system, so that’s the next stop:

kubectl -n kube-system get pods -l k8s-app=kube-dns -o wide
kubectl -n kube-system top pods -l k8s-app=kube-dns     # CPU pegged / throttled
kubectl -n kube-system logs -l k8s-app=kube-dns --tail=100 | grep -Ei 'timeout|SERVFAIL|i/o'

You find 2 CoreDNS pods pinned at their CPU limit and being throttled, logs peppered with i/o timeout and SERVFAIL. Two replicas on default resources, no autoscaling, no node-local cache — against thousands of transactions a minute, each fanning out to multiple downstream lookups — is simply under-provisioned for the query volume.

📟 Reinforce · Interactive War Room Kubernetes Memory Eviction Cascade & CoreDNS Outage → The interactive War Room that pairs DNS saturation with the memory-eviction cascade it often travels with.
?
Decision Point 3

Why is the failure intermittent rather than a clean, total DNS outage — and what common Kubernetes default quietly multiplies the query load?

Think about queueing under saturation, and about how a short name like 'risk-engine' actually gets resolved inside a pod.

Commit to your answer, then reveal the responder’s move →
✓What the responder did & why

Two mechanisms:

  • Saturation is probabilistic. Once CoreDNS is at capacity, some queries are answered from cache or served quickly, while others queue past the client’s timeout. So the same call succeeds one second and 502s the next — the “intermittent” signature.
  • ndots:5 search-path expansion. The default pod resolv.conf has options ndots:5 and a search list. A lookup for a short name like risk-engine is first tried as risk-engine.core-banking.svc.cluster.local, then risk-engine.svc.cluster.local, then risk-engine.cluster.local, and so on — several queries per logical lookup, multiplying load on an already-saturated CoreDNS.
📖 Reinforce · Guide Live Structured Debugging Demo (CPU / Thread / DNS / FD) → Its DNS-isolation sub-procedure — dig, ss, and resolver checks — is the exact toolkit used above.

Root cause

CoreDNS was saturated. Two replicas on default resources, with no HPA and no NodeLocal DNS cache, could not keep up with peak service-to-service query volume (amplified by ndots:5 search expansion). Queries queued and timed out intermittently, so every service that resolves a name before calling a peer slowed or 502’d — while the services, pods, nodes, and control plane all stayed “healthy,” because the failure lived one layer below them in service discovery.

Containment and durable fix

# Immediate relief: give DNS more capacity
kubectl -n kube-system scale deploy/coredns --replicas=4
# and/or raise CoreDNS CPU/memory requests+limits so it stops getting throttled

# Durable:
#  - Add the cluster-proportional autoscaler (or HPA) for CoreDNS so replicas track node/core count
#  - Deploy NodeLocal DNSCache (a per-node caching agent) to absorb most lookups locally
#  - Tune ndots / enable autopath to collapse the search-domain query fan-out
#  - Add response caching in CoreDNS's Corefile

Containment is a single high-leverage move — scale CoreDNS — which drains the queue and restores resolution latency almost immediately; the autoscaler + NodeLocal cache are the prevention so peak traffic never re-saturates two lonely replicas again.

🎯 Transferable principle

When many services fail together but each component is individually healthy, suspect the shared dependency in the common path — and for Kubernetes that path almost always starts with DNS. Isolate it in one move (named call vs. direct-IP call), then treat cluster DNS as the tier-0 dependency it is: give CoreDNS autoscaling, a node-local cache, and query-fan-out tuning before peak, not during the incident.

Telling this story to a recruiter

The 30-second version:

“Our payments platform started throwing intermittent 502s at 3,000 transactions a minute — checkout latency went from 90 milliseconds to nearly two seconds — and yet every pod, node, and load balancer reported healthy. I proved the failure lived one layer below the services, in cluster DNS: CoreDNS was CPU-throttled and silently dropping queries. Scaling it stopped the bleeding in minutes, and the durable fix — DNS autoscaling and node-local caching — removed the platform’s single quietest point of failure.”

The detailed telling:

Situation. A SEV-1 with revenue impact: the checkout API began returning intermittent 502s and 504s at peak, latency jumped twentyfold, and load-balancer health checks were flapping. The confusing part was that every standard check came back clean — all pods Running, nothing OOMKilled or evicted, nodes Ready, no deploy had gone out, and some affected services had no errors in their logs at all.

Task. Identify which layer was actually failing, explain why it was intermittent and invisible, contain it, and produce a structured RCA.

Action. The shape of the blast radius was my first clue: five unrelated services degrading simultaneously, with healthy components, means a shared dependency in the common path — and in Kubernetes, every service-to-service call starts with a DNS lookup. I proved it in one move from inside a pod: a curl against the service name hung, while the same request against the pod IP returned instantly. Resolution itself was the bottleneck. That took me to CoreDNS in kube-system, where both replicas were pinned at their CPU limits, throttled, and logging query timeouts. The math explained everything: two default-sized replicas, no autoscaling, no node-local cache, against thousands of transactions a minute — each one fanning out into multiple lookups thanks to Kubernetes’ default ndots search-path expansion. Saturation is probabilistic, so some queries were served from cache and some queued past their timeout — the exact intermittent signature we were seeing. And the apps logged nothing because failed calls hung before ever reaching the target service. I scaled CoreDNS immediately, then landed the durable fixes: proportional autoscaling for DNS, NodeLocal DNSCache to absorb lookups on each node, and ndots tuning to collapse the query fan-out.

Result. Resolution latency recovered within minutes of scaling; checkout latency returned to ~90ms and 502s stopped. The RCA reframed DNS as a tier-0 dependency with capacity planning, autoscaling, and dashboards of its own — it had previously been invisible in our monitoring precisely because nothing “owned” it.

What this story demonstrates. Reasoning from blast-radius shape to a shared dependency, designing a single decisive isolation test instead of ten weak ones, and following an incident through to structural prevention rather than stopping at “we scaled it.”

Interview deep-dive: the full case study

How the issue happened (the mechanism)

Every service-to-service call in the platform started with a DNS lookup against cluster DNS — service.namespace.svc.cluster.local. That made CoreDNS a tier-0 dependency sitting in the request path of every call, but it was provisioned as an afterthought: two default-sized replicas, no HPA, and no NodeLocal DNS cache.

At peak (~3,000 transactions/minute), each transaction fanned out into multiple downstream calls, and each call was multiplied again by the default pod resolver setting options ndots:5. With ndots:5, a short name like risk-engine is tried first as risk-engine.<namespace>.svc.cluster.local, then .svc.cluster.local, then .cluster.local, and so on through the search list — several queries per logical lookup. Two small CoreDNS pods simply couldn’t keep up: they hit their CPU limits, got CPU-throttled by the kernel, and their query queue backed up.

Under that saturation the failure became probabilistic: some queries were served from cache or answered quickly, while others queued past the client’s timeout and failed. That produced the exact intermittent signature — the same call 502’ing one second and succeeding the next. Two things made it nearly invisible: (1) the calls hung during resolution, before ever reaching the target service, so the target app logged nothing; and (2) every individual component (pods, nodes, control plane, load balancer) was genuinely healthy, because the fault lived one layer below them in service discovery. The edge 502/504s and the flapping load-balancer health checks were downstream effects — the gateway’s own health path also resolves names, so it crossed its timeout threshold intermittently and flapped in and out of the target pool.

Impact

  • Latency: checkout p50 jumped from ~90ms to over 1.8s — a ~20x regression.
  • Errors: intermittent HTTP 502/504 at the public API, internal API timeouts, and background jobs piling up retries.
  • Classification: SEV-1, revenue-impacting, during peak business hours on a payments platform.
  • Diagnostic cost: the “healthy everything” picture burned real minutes as the team checked pods, nodes, and AWS before reframing to a shared dependency.

Steps taken to resolve

  1. Read the blast-radius shape: many unrelated services degrading together with every component healthy → a shared dependency in the common path, which in Kubernetes begins with DNS.
  2. One decisive isolation test from inside a pod: curl to a service name hung while curl to the pod IP returned instantly, and time nslookup/dig showed multi-second or SERVFAIL responses — proving the latency was in resolution, before the connection.
  3. Went to the resolver tier: kubectl -n kube-system get pods -l k8s-app=kube-dns, top pods (CoreDNS pinned/throttled at CPU limit), and logs full of i/o timeout / SERVFAIL.
  4. Contained with one high-leverage move: scaled CoreDNS replicas (and raised its CPU/memory requests+limits so it stopped being throttled), which drained the queue and restored resolution latency within minutes.

Outcomes

  • Resolution latency recovered within minutes of scaling; checkout latency returned to ~90ms and the 502/504s stopped.
  • The RCA reclassified cluster DNS as a tier-0 dependency with its own capacity plan, autoscaling, and dashboards — previously it had no owner and no monitoring, which is exactly why it failed silently.

What we learned

  • When many services fail together but each component is individually healthy, suspect the shared dependency in the common path — and in Kubernetes that path almost always starts with DNS.
  • Saturation of a shared service looks like intermittency, not a clean outage, because queuing past a timeout is probabilistic.
  • Absence of application errors is itself evidence: a call that fails before reaching the app leaves no app log, which points upstream, not at the service.
  • Defaults have a cost at scale: ndots:5 search expansion quietly multiplies DNS load.

Prevention — what we changed so it won’t recur

  • Autoscale CoreDNS (cluster-proportional autoscaler or HPA) so replica count tracks node/core count instead of being a fixed two.
  • Deploy NodeLocal DNSCache — a per-node caching agent that absorbs the vast majority of lookups locally and removes the central round trips.
  • Tune the query fan-out — lower ndots where appropriate and/or enable autopath, plus response caching in the CoreDNS Corefile.
  • Treat DNS as tier-0 in monitoring — dashboards and alerts on CoreDNS CPU throttling, query latency, and SERVFAIL rate, so saturation pages before it becomes customer-visible.