Probes: Liveness, Readiness & Startup

Probes are the only way Kubernetes knows anything about your application’s health. Get them right and rollouts and self-healing work. Get them wrong and you build an outage generator that fires under exactly the conditions you most need stability.

Topic 1: Three Probes, Three Consequences

That table is the lesson. Readiness controls traffic; liveness controls lifecycle. Confusing them is how healthy services get restarted into an outage.

Topic 2: Probe Mechanics

readinessProbe:
  httpGet:
    path: /readyz
    port: http                  # by NAME — survives a port change
    httpHeaders:
      - name: X-Probe
        value: readiness
  initialDelaySeconds: 0
  periodSeconds: 5
  timeoutSeconds: 2
  successThreshold: 1
  failureThreshold: 3

Time to act = periodSeconds × failureThreshold (plus up to one timeoutSeconds). With defaults, a liveness probe kills a container after roughly 30 seconds of failure.

timeoutSeconds: 1 is the default and it is the single most common cause of spurious probe failures: an app under GC pause or heavy load takes 1.2s to answer /healthz and gets declared dead.

The four probe handlers:

httpGet:  { path: /healthz, port: 8080, scheme: HTTP }   # 200–399 = pass
tcpSocket: { port: 8080 }                                 # connection opens = pass
exec:     { command: ["/bin/sh","-c","pg_isready -U app"] }  # exit 0 = pass
grpc:     { port: 9090, service: "" }                     # gRPC health protocol, GA in 1.27

exec is the expensive one — it forks a process on every probe, on every pod, forever. A periodSeconds: 1 exec probe across 500 pods is 500 processes per second of pure overhead. Prefer httpGet.

tcpSocket only proves something is listening. A process that has accepted the socket but deadlocked its worker pool passes a TCP probe indefinitely.

Topic 3: The Liveness Probe Is Dangerous

A liveness probe is a standing instruction to kill your container. It should only ever detect states that a restart genuinely fixes — a deadlock, an unrecoverable internal error.

Failure mode 1: the dependency check

# WRONG — checks the database
livenessProbe:
  httpGet: { path: /healthz }     # this handler queries Postgres

The database has a blip. Every replica’s liveness probe fails simultaneously. Kubernetes restarts every pod at once. The restarts hammer the recovering database, and a 30-second database hiccup becomes a full outage that outlives it.

Rule: a liveness probe must never check anything outside the container. No database, no cache, no downstream API. Those belong in a readiness probe, where the consequence is “stop sending traffic” rather than “kill everything”.

Failure mode 2: too aggressive under load

An app that slows down under load fails its probe, gets killed, and its traffic shifts to the remaining pods — which are now more loaded, so they fail too. A cascading restart that only occurs at peak.

Failure mode 3: no startup allowance

A JVM taking 60s to warm up with initialDelaySeconds: 10 never survives long enough to start. It is killed, restarted, killed again — a crash loop with no application error anywhere in the logs.

The defensible default: do not set a liveness probe.

Seriously. If your app exits on unrecoverable errors — which it should — the restartPolicy already handles it. Add a liveness probe only when you have a specific, known hang that a restart demonstrably fixes, and make it check strictly in-process state:

livenessProbe:
  httpGet: { path: /livez, port: http }    # a handler that returns 200 unconditionally
  periodSeconds: 10
  timeoutSeconds: 5
  failureThreshold: 6                       # 60s of failure before killing

/livez returning a static 200 sounds pointless. It is not: it proves the HTTP server loop is still scheduling requests. That is exactly the deadlock a restart fixes, and nothing more.

Topic 4: The Readiness Probe Is the Important One

Readiness controls Service membership, which means it controls:

• Whether traffic reaches this pod at all.
• Whether a rolling update proceeds (a pod counts as available only when ready).
• Whether a pod is removed from endpoints during shutdown.

readinessProbe:
  httpGet: { path: /readyz, port: http }
  periodSeconds: 5
  timeoutSeconds: 3
  failureThreshold: 2         # react quickly — the cost is only "no traffic"

Readiness should check dependencies — this is the inverse of liveness. If the database is unreachable, this pod cannot serve, so take it out of rotation. Nothing gets killed; when the dependency recovers, the probe passes and traffic returns.

One caveat: if every replica depends on the same broken thing, they all go unready and the Service has no endpoints — clients get connection failures rather than a useful error. For that reason many teams have /readyz check only hard dependencies (the primary datastore) and degrade gracefully on soft ones (a cache, a recommendations API).

/readyz  → is my DB pool healthy? are my caches warm? am I shutting down?
/livez   → am I still able to serve an HTTP request at all?

Using readiness for graceful shutdown:

// on SIGTERM
shuttingDown.Store(true)   // /readyz starts returning 503 immediately
// keep serving in-flight requests

Flipping readiness to false on SIGTERM removes the pod from endpoints before it stops accepting connections, which closes the race described in the rollouts lesson.

Topic 5: startupProbe Solves the Slow-Boot Problem

Before startupProbe, a slow-starting app forced you to set initialDelaySeconds high on the liveness probe — which then meant a genuinely wedged container took that long to be detected for its entire life.

startupProbe:
  httpGet: { path: /healthz, port: http }
  periodSeconds: 10
  failureThreshold: 30        # up to 300s to start
livenessProbe:
  httpGet: { path: /livez, port: http }
  periodSeconds: 10
  failureThreshold: 3         # once started, react in 30s

While the startup probe is running, liveness and readiness are suspended entirely. The container gets up to periodSeconds × failureThreshold = 300 seconds to boot; after that, the fast liveness settings apply. You get a generous startup budget and quick detection later.

Set failureThreshold generously — the cost of being wrong in the tight direction is a crash loop; the cost of being generous is a slightly slower detection of a genuinely broken start.

Topic 6: Reading Probe Failures

kubectl describe pod api-7d9f-x2k4 | grep -A5 Events
# Warning  Unhealthy  kubelet  Liveness probe failed: HTTP probe failed with statuscode: 500
# Warning  Unhealthy  kubelet  Readiness probe failed: Get "http://10.1.2.9:8080/readyz":
#                              context deadline exceeded (Client.Timeout exceeded ...)
# Normal   Killing    kubelet  Container api failed liveness probe, will be restarted

Three signatures worth recognising:

• context deadline exceeded → timeoutSeconds too low, or the app is genuinely slow.
• connection refused → nothing is listening yet. Wrong port, or the app has not bound.
• statuscode: 500 → the app answered and said it is unhealthy. Read its logs; the probe is working correctly.

kubectl get pod api-7d9f-x2k4 -o jsonpath='{.status.containerStatuses[0].restartCount}{"\n"}'
kubectl get events --field-selector reason=Unhealthy --sort-by=.lastTimestamp

# Probe it yourself, exactly as the kubelet does:
kubectl run -it --rm t --image=nicolaka/netshoot --restart=Never -- \
  curl -sS -m2 -o /dev/null -w '%{http_code} %{time_total}s\n' http://10.1.2.9:8080/readyz

That last command settles arguments: it reproduces the probe from inside the cluster, with a timeout, and prints the response time. If it takes 1.4s and your timeoutSeconds is 1, you have your answer.

A defensible starting point:

startupProbe:
  httpGet: { path: /healthz, port: http }
  periodSeconds: 5
  failureThreshold: 30           # 150s to boot
readinessProbe:
  httpGet: { path: /readyz, port: http }
  periodSeconds: 5
  timeoutSeconds: 3
  failureThreshold: 2            # out of rotation within ~10s
# livenessProbe: omitted unless you have a specific hang to detect

Try it yourself: Point a liveness probe at an endpoint that queries a dependency, then stop the dependency. Watch every replica restart simultaneously. That is the failure mode in a controlled setting — it is considerably less pleasant in production.

Common mistake: Using the same /healthz endpoint for liveness and readiness. They answer different questions, and sharing them means either your liveness probe checks dependencies (restart storms) or your readiness probe does not (traffic to pods that cannot serve). Two endpoints, two meanings.