The Production Debugging Playbook

Everything in this module converges here. The point of a playbook is not to memorise commands — it is to have a fixed order so that under pressure you eliminate layers instead of guessing.

Topic 1: The 60-Second Triage

When you do not yet know what is wrong, this sweep narrows it:

# 1. What is not Running?
kubectl get pods -A --field-selector status.phase!=Running

# 2. What is Running but NOT READY? (the ones people miss)
kubectl get pods -A -o json | jq -r '.items[] |
  select(.status.phase=="Running") |
  select([.status.containerStatuses[]?.ready] | index(false)) |
  "\(.metadata.namespace)/\(.metadata.name)"'

# 3. What is restarting?
kubectl get pods -A --sort-by=.status.containerStatuses[0].restartCount | tail -10

# 4. What did Kubernetes complain about recently?
kubectl get events -A --field-selector type=Warning --sort-by=.lastTimestamp | tail -20

# 5. Are the nodes healthy?
kubectl get nodes
kubectl top nodes

# 6. Is the control plane healthy?
kubectl get --raw='/readyz?verbose' | grep -v ok

Step 2 matters more than it looks: kubectl get pods shows Running for a pod failing every readiness probe, so a service can be entirely down while every pod looks fine at a glance. That jq filter finds them.

Topic 2: Pod Pending

Meaning: no node has been assigned. This is the scheduler’s domain — the kubelet has never heard of this pod.

kubectl describe pod api-xyz | tail -15

The scheduler prints a complete audit of every node:

0/9 nodes are available: 3 Insufficient cpu,
                         2 node(s) had untolerated taint {dedicated: gpu},
                         2 node(s) didn't match Pod's node affinity/selector,
                         2 node(s) had volume node affinity conflict.

kubectl describe node <n> | grep -A8 'Allocated resources'   # REQUESTS, not usage
kubectl get pvc -n <ns>
kubectl get pods -n kube-system -l component=kube-scheduler

Topic 3: CrashLoopBackOff

Meaning: the container starts, exits, and Kubernetes is backing off before retrying (10s, 20s, 40s… capped at 5 min). The backoff is the symptom; the exit is the cause.

kubectl logs api-xyz --previous          # THE instance that died — always start here
kubectl get pod api-xyz -o jsonpath='{.status.containerStatuses[0].lastState.terminated}' | jq

{"exitCode": 137, "reason": "OOMKilled", "startedAt": "...", "finishedAt": "..."}

# No logs at all? The container never started. Check config refs and the image.
kubectl describe pod api-xyz | grep -A20 Events
kubectl get pod api-xyz -o jsonpath='{.status.containerStatuses[0].state.waiting.reason}{"\n"}'

CreateContainerConfigError means a referenced ConfigMap or Secret does not exist — the container was never created, so there are no logs to read.

Topic 4: ImagePullBackOff and ContainerCreating

kubectl describe pod api-xyz | grep -A10 Events

kubectl get sa default -o jsonpath='{.imagePullSecrets}{"\n"}'
kubectl get secret regcred -o jsonpath='{.data.\.dockerconfigjson}' | base64 -d | jq

ContainerCreating for more than a minute is a different problem — the kubelet is stuck on setup:

kubectl describe pod api-xyz | grep -A10 Events
# FailedMount        Unable to attach or mount volumes: timed out
# FailedCreatePodSandBox  ... failed to setup network for sandbox

That means volumes (CSI) or networking (CNI). Check the CNI DaemonSet and the node’s kubelet.

Topic 5: Service Unreachable

The ladder from the Services lesson, in order. Each rung eliminates a layer.

# 1. Does the Service have endpoints?    ← ALWAYS FIRST
kubectl get endpointslices -l kubernetes.io/service-name=api

# Empty? Only two causes:
kubectl get svc api -o jsonpath='{.spec.selector}{"\n"}'    # a) selector matches nothing
kubectl get pods -l app=api                                  # b) pods exist but NOT READY

# 2. Does the POD answer directly? (bypasses Service entirely)
kubectl port-forward pod/api-xyz 8080:8080
curl -sS localhost:8080/healthz

# 3. Does the SERVICE answer from inside the cluster?
kubectl run -it --rm t --image=nicolaka/netshoot --restart=Never -- \
  curl -sS -m5 http://api.default.svc.cluster.local

# 4. Does DNS work at all?
kubectl run -it --rm t --image=nicolaka/netshoot --restart=Never -- \
  nslookup kubernetes.default

# 5. Is a NetworkPolicy blocking it?
kubectl get networkpolicy -A

The decisive split is 2 vs 3. Pod answers, Service does not → Service, endpoints, kube-proxy or NetworkPolicy. Neither answers → it is the application, and Kubernetes is not your problem.

Timeout vs connection refused is the other high-value signal: refused means something answered (wrong port, or the app is not listening); timeout means the packet vanished (NetworkPolicy, security group, or routing).

Topic 6: Node Problems and Cluster-Wide Slowness

kubectl get nodes
kubectl describe node ip-10-0-1-42 | grep -A6 Conditions

Ready=False with everything else fine is usually the kubelet, the container runtime, or the network:

ssh node; systemctl status kubelet; journalctl -u kubelet -n 100 --no-pager
crictl ps -a | head
df -h /var/lib/kubelet /var/lib/containerd

Evicted pods are a node problem, not a pod problem:

kubectl get pods -A --field-selector status.phase=Failed
kubectl get events -A --field-selector reason=Evicted

Everything is slow but nothing is broken:

This is the one people chase for hours in the wrong place.

# etcd disk latency — the usual culprit
kubectl get --raw /metrics | grep etcd_disk_wal_fsync_duration_seconds
# API server latency
kubectl get --raw /metrics | grep apiserver_request_duration_seconds
# Are you being throttled by API priority and fairness?
kubectl get --raw /metrics | grep apiserver_flowcontrol_rejected_requests_total

If etcd fsync p99 is above ~10ms, that is your answer, and no amount of application investigation will find it.

Similarly, CPU throttling produces latency with no error anywhere:

kubectl exec api-xyz -- cat /sys/fs/cgroup/cpu.stat | grep -E 'nr_throttled|throttled_usec'

Topic 7: The Emergency Toolkit

# A shell in the cluster with real network tools
kubectl run -it --rm netshoot --image=nicolaka/netshoot --restart=Never -- bash

# Debug a distroless pod that has no shell
kubectl debug -it api-xyz --image=nicolaka/netshoot --target=api

# Debug a NODE
kubectl debug node/ip-10-0-1-42 -it --image=busybox

# Copy the pod with a modified spec, leaving the original running
kubectl debug api-xyz -it --copy-to=api-debug --container=api -- sh

# Everything about one object
kubectl get pod api-xyz -o yaml
kubectl describe pod api-xyz

# What changed, and when?
kubectl rollout history deployment/api
kubectl get events -A --sort-by=.lastTimestamp | tail -40

# Reproduce a probe exactly as the kubelet does
kubectl run -it --rm t --image=nicolaka/netshoot --restart=Never -- \
  curl -sS -m2 -o /dev/null -w '%{http_code} %{time_total}s\n' http://10.1.2.9:8080/readyz

The questions, in order, for any incident:

1. What changed? kubectl rollout history, your deploy log, kubectl get events. Most incidents are a change.
2. Is it one pod, one node, or everything? That single distinction eliminates most of the search space.
3. What does the object’s status say versus its spec? The gap is the problem.
4. What do the Events say? Kubernetes narrating its own failure, and it expires in an hour.
5. Is the app broken, or the platform? Port-forward directly to the pod — that answers it.

Try it yourself: Break all four failures from the hands-on task in a scratch namespace and time yourself diagnosing each. The goal is one command per failure, not five.

Common mistake: Reading application logs first. If the pod is Pending, ImagePullBackOff or CreateContainerConfigError, the container never ran and there are no logs — the answer was in kubectl describe the whole time. Check the pod’s state before you read its output.