Every previous lesson taught one layer. Real incidents cross all of them, and the evidence expires — events are gone in an hour. This capstone builds the tool that captures and correlates everything while it still exists.
The Requirements
| # | Requirement | Lesson |
|---|---|---|
| 1 | Capture events before they expire | 22 |
| 2 | Snapshot object state at incident time | 3 |
| 3 | Pull logs including --previous from crashed containers | 27 |
| 4 | Collect node conditions and pressure | 2 |
| 5 | Query metrics for the window | 22 |
| 6 | Correlate everything into one time-ordered timeline | — |
| 7 | Classify known failure signatures automatically | 27 |
| 8 | Emit a Markdown report ready for a postmortem | — |
Step 1: Interface and Capture
#!/usr/bin/env bash
# k8s-forensics — capture and correlate cluster state for an incident window
set -Eeuo pipefail
readonly SCRIPT_NAME=${0##*/}
NAMESPACE=""; SINCE="1h"; OUTDIR=""; PROM_URL="${PROM_URL:-}"
usage() {
cat <<EOF
${SCRIPT_NAME} — capture cluster forensics for an incident
USAGE
${SCRIPT_NAME} -n NAMESPACE [-s SINCE] [-o OUTDIR]
OPTIONS
-n NS namespace to investigate (required)
-s SINCE how far back (default 1h) — accepts 30m, 2h, 1d
-o DIR output directory (default ./forensics-<ts>)
-p URL Prometheus base URL (or set PROM_URL)
EOF
}
while getopts ":n:s:o:p:h" opt; do
case $opt in
n) NAMESPACE=$OPTARG ;; s) SINCE=$OPTARG ;;
o) OUTDIR=$OPTARG ;; p) PROM_URL=$OPTARG ;;
h) usage; exit 0 ;; \?) usage >&2; exit 2 ;;
esac
done
[[ -n $NAMESPACE ]] || { usage >&2; exit 2; }
OUTDIR="${OUTDIR:-./forensics-$(date +%Y%m%d-%H%M%S)}"
mkdir -p "$OUTDIR"/{objects,logs,events,nodes,metrics}
log() { printf '%s [%-5s] %s\n' "$(date -Iseconds)" "$1" "${*:2}" >&2; }
capture_objects() {
log INFO "capturing object state"
local kind
for kind in pods deployments replicasets statefulsets daemonsets services \
endpointslices configmaps pvc hpa pdb networkpolicies jobs; do
kubectl get "$kind" -n "$NAMESPACE" -o yaml > "$OUTDIR/objects/${kind}.yaml" 2>/dev/null || true
done
kubectl get pods -n "$NAMESPACE" -o wide > "$OUTDIR/objects/pods-wide.txt" 2>/dev/null || true
# describe captures Events, which live nowhere else
local pod
while read -r pod; do
[[ -n $pod ]] || continue
kubectl describe pod "$pod" -n "$NAMESPACE" > "$OUTDIR/objects/describe-${pod}.txt" 2>/dev/null || true
done < <(kubectl get pods -n "$NAMESPACE" -o name 2>/dev/null | cut -d/ -f2)
}
capture_events() {
log INFO "capturing events (these expire in ~1h)"
kubectl get events -n "$NAMESPACE" --sort-by=.lastTimestamp \
-o json > "$OUTDIR/events/events.json" 2>/dev/null || true
kubectl get events -A --field-selector type=Warning --sort-by=.lastTimestamp \
> "$OUTDIR/events/cluster-warnings.txt" 2>/dev/null || true
}
capture_logs() {
log INFO "capturing logs"
local pod c
while read -r pod; do
[[ -n $pod ]] || continue
while read -r c; do
[[ -n $c ]] || continue
kubectl logs "$pod" -n "$NAMESPACE" -c "$c" --since="$SINCE" --timestamps \
> "$OUTDIR/logs/${pod}_${c}.log" 2>/dev/null || true
# --previous is where the CAUSE of a crash loop lives
kubectl logs "$pod" -n "$NAMESPACE" -c "$c" --previous --timestamps \
> "$OUTDIR/logs/${pod}_${c}_PREVIOUS.log" 2>/dev/null || true
done < <(kubectl get pod "$pod" -n "$NAMESPACE" \
-o jsonpath='{range .spec.containers[*]}{.name}{"\n"}{end}' 2>/dev/null)
done < <(kubectl get pods -n "$NAMESPACE" -o name 2>/dev/null | cut -d/ -f2)
find "$OUTDIR/logs" -size 0 -delete
}
capture_nodes() {
log INFO "capturing node state"
kubectl get nodes -o wide > "$OUTDIR/nodes/nodes.txt"
kubectl top nodes > "$OUTDIR/nodes/top.txt" 2>/dev/null || true
local n
while read -r n; do
kubectl describe node "$n" > "$OUTDIR/nodes/describe-${n}.txt" 2>/dev/null || true
done < <(kubectl get nodes -o name | cut -d/ -f2)
}
Two capture decisions carry the project: kubectl describe per pod (the only place events attach to an object) and --previous logs (the only place a crashed container’s output survives).
Step 2: Normalise Into One Timeline
The correlation trick from any log-analysis work: reduce every source to the same shape, then sort.
epoch <TAB> source <TAB> severity <TAB> object <TAB> message
build_timeline() {
log INFO "building timeline"
{
# events
jq -r '.items[] |
[(.lastTimestamp // .eventTime | fromdateiso8601),
"event",
(if .type=="Warning" then "WARN" else "INFO" end),
(.involvedObject.kind + "/" + .involvedObject.name),
(.reason + ": " + (.message | gsub("\t";" ")))] | @tsv' \
"$OUTDIR/events/events.json" 2>/dev/null || true
# container restarts and terminations
jq -r '.items[] as $p | ($p.status.containerStatuses // [])[] |
select(.lastState.terminated != null) |
[(.lastState.terminated.finishedAt | fromdateiso8601),
"container", "ERROR",
($p.metadata.name + "/" + .name),
("terminated exit=" + (.lastState.terminated.exitCode|tostring)
+ " reason=" + (.lastState.terminated.reason // "unknown"))] | @tsv' \
"$OUTDIR/objects/pods.yaml.json" 2>/dev/null || true
# log lines carrying an RFC3339 timestamp (from --timestamps)
grep -rhE '^[0-9]{4}-[0-9]{2}-[0-9]{2}T' "$OUTDIR/logs/" 2>/dev/null \
| awk '{ ts=$1; $1=""; sub(/^ /,"");
sev = /(?i)error|fatal|panic/ ? "ERROR" : (/(?i)warn/ ? "WARN" : "INFO");
print ts "\t" "log" "\t" sev "\t-\t" $0 }' \
| while IFS=$'\t' read -r ts src sev obj msg; do
e=$(date -d "$ts" +%s 2>/dev/null || echo 0)
(( e > 0 )) && printf '%s\t%s\t%s\t%s\t%s\n' "$e" "$src" "$sev" "$obj" "$msg"
done
} | sort -n -k1,1 > "$OUTDIR/timeline.tsv"
wc -l < "$OUTDIR/timeline.tsv"
}
Convert pods.yaml to JSON once up front (kubectl get pods -o json) rather than parsing YAML — simpler and faster.
Step 3: Classify Known Signatures
This is where the debugging playbook becomes code.
classify() {
log INFO "classifying failure signatures"
local out="$OUTDIR/findings.md"
: > "$out"
# 1. CrashLoopBackOff — with the exit code, which names the cause
jq -r '.items[] as $p | ($p.status.containerStatuses // [])[] |
select(.state.waiting.reason == "CrashLoopBackOff") |
"- **CrashLoopBackOff** `\($p.metadata.name)/\(.name)` — restarts: \(.restartCount), " +
"last exit: \(.lastState.terminated.exitCode // "?") " +
"(\(.lastState.terminated.reason // "unknown"))"' \
"$OUTDIR/objects/pods.json" 2>/dev/null >> "$out" || true
# 2. OOMKilled — memory limit too low, or a leak
jq -r '.items[] as $p | ($p.status.containerStatuses // [])[] |
select(.lastState.terminated.reason == "OOMKilled") |
"- **OOMKilled** `\($p.metadata.name)/\(.name)` — exceeded its memory limit"' \
"$OUTDIR/objects/pods.json" 2>/dev/null >> "$out" || true
# 3. Pending — the scheduler already explained why, in the event
jq -r '.items[] | select(.reason == "FailedScheduling") |
"- **Unschedulable** `\(.involvedObject.name)` — \(.message)"' \
"$OUTDIR/events/events.json" 2>/dev/null | sort -u >> "$out" || true
# 4. Service with no endpoints — selector or readiness
jq -r '.items[] | select((.endpoints // []) | length == 0) |
"- **Service has no endpoints** `\(.metadata.labels["kubernetes.io/service-name"])` " +
"— selector matches nothing, or no pod is Ready"' \
"$OUTDIR/objects/endpointslices.json" 2>/dev/null >> "$out" || true
# 5. Rollout not complete
jq -r '.items[] |
select(.status.readyReplicas // 0 < (.spec.replicas // 0)) |
"- **Deployment degraded** `\(.metadata.name)` — \(.status.readyReplicas // 0)/\(.spec.replicas) ready"' \
"$OUTDIR/objects/deployments.json" 2>/dev/null >> "$out" || true
# 6. Node pressure — a NODE problem masquerading as a pod problem
grep -l 'Pressure.*True' "$OUTDIR"/nodes/describe-*.txt 2>/dev/null | while read -r f; do
echo "- **Node under pressure** \`$(basename "$f" .txt | sed 's/describe-//')\` — see node detail" >> "$out"
done
# 7. Evicted pods
jq -r '.items[] | select(.status.reason == "Evicted") |
"- **Evicted** `\(.metadata.name)` — \(.status.message // "node pressure")"' \
"$OUTDIR/objects/pods.json" 2>/dev/null >> "$out" || true
# 8. PDBs blocking disruption
jq -r '.items[] | select(.status.disruptionsAllowed == 0) |
"- **PDB blocks all disruption** `\(.metadata.name)` — drains and scale-down will hang"' \
"$OUTDIR/objects/pdb.json" 2>/dev/null >> "$out" || true
sort -u "$out" -o "$out"
wc -l < "$out"
}
Each rule is one failure mode from lesson 27, expressed as a query. That is the whole design: the playbook, automated.
Step 4: Metrics for the Window
capture_metrics() {
[[ -n $PROM_URL ]] || { log WARN "PROM_URL unset — skipping metrics"; return 0; }
log INFO "querying metrics"
local end start
end=$(date +%s)
start=$(( end - $(printf '%s' "$SINCE" | awk '/h$/{print substr($0,1,length-1)*3600} /m$/{print substr($0,1,length-1)*60} /d$/{print substr($0,1,length-1)*86400}') ))
q() { # name, promql
curl -fsS --max-time 20 --get "$PROM_URL/api/v1/query_range" \
--data-urlencode "query=$2" \
--data-urlencode "start=$start" --data-urlencode "end=$end" \
--data-urlencode "step=30s" \
> "$OUTDIR/metrics/$1.json" 2>/dev/null || log WARN "metric query failed: $1"
}
q restarts "sum by (pod) (increase(kube_pod_container_status_restarts_total{namespace=\"$NAMESPACE\"}[5m]))"
q cpu "sum by (pod) (rate(container_cpu_usage_seconds_total{namespace=\"$NAMESPACE\"}[5m]))"
q memory "sum by (pod) (container_memory_working_set_bytes{namespace=\"$NAMESPACE\"})"
q throttling "sum by (pod) (rate(container_cpu_cfs_throttled_periods_total{namespace=\"$NAMESPACE\"}[5m]) / rate(container_cpu_cfs_periods_total{namespace=\"$NAMESPACE\"}[5m]))"
q notready "sum(kube_deployment_spec_replicas{namespace=\"$NAMESPACE\"}) - sum(kube_deployment_status_replicas_available{namespace=\"$NAMESPACE\"})"
}
CPU throttling is included deliberately: it is invisible in every other signal and is a frequent cause of “slow but nothing is wrong”.
Step 5: The Report
generate_report() {
local ts_count finding_count
ts_count=$(wc -l < "$OUTDIR/timeline.tsv" 2>/dev/null || echo 0)
finding_count=$(wc -l < "$OUTDIR/findings.md" 2>/dev/null || echo 0)
cat > "$OUTDIR/REPORT.md" <<EOF
# Incident Forensics — ${NAMESPACE}
**Captured:** $(date -Iseconds)
**Cluster:** $(kubectl config current-context)
**Window:** last ${SINCE}
**Events correlated:** ${ts_count}
## Findings (${finding_count})
$(cat "$OUTDIR/findings.md" 2>/dev/null || echo "_No known signatures matched._")
## First Error
\`\`\`
$(awk -F'\t' '$3=="ERROR" {print strftime("%Y-%m-%d %H:%M:%S",$1), "["$2"]", $4, $5; exit}' "$OUTDIR/timeline.tsv" 2>/dev/null || echo "none")
\`\`\`
## Timeline — errors and warnings only
\`\`\`
$(awk -F'\t' '$3!="INFO" {print strftime("%H:%M:%S",$1), "["$2"]", $4, substr($5,1,110)}' "$OUTDIR/timeline.tsv" 2>/dev/null | head -60)
\`\`\`
## Workload State
\`\`\`
$(cat "$OUTDIR/objects/pods-wide.txt" 2>/dev/null)
\`\`\`
## Node Conditions
\`\`\`
$(cat "$OUTDIR/nodes/nodes.txt" 2>/dev/null)
$(cat "$OUTDIR/nodes/top.txt" 2>/dev/null)
\`\`\`
## Suggested Next Steps
$(if grep -q 'CrashLoopBackOff' "$OUTDIR/findings.md" 2>/dev/null; then
echo "- Read \`logs/*_PREVIOUS.log\` — the crashed instance, not the current one"; fi)
$(if grep -q 'OOMKilled' "$OUTDIR/findings.md" 2>/dev/null; then
echo "- Compare \`metrics/memory.json\` against the container's memory limit"; fi)
$(if grep -q 'no endpoints' "$OUTDIR/findings.md" 2>/dev/null; then
echo "- Compare the Service selector against actual pod labels, then check readiness"; fi)
$(if grep -q 'Node under pressure' "$OUTDIR/findings.md" 2>/dev/null; then
echo "- This is a NODE problem — pod-level investigation will mislead you"; fi)
---
*Generated by ${SCRIPT_NAME}. Raw evidence in \`$(basename "$OUTDIR")/\`.*
EOF
log INFO "report written: $OUTDIR/REPORT.md"
}
main() {
log INFO "capturing forensics for namespace=${NAMESPACE} window=${SINCE}"
kubectl get ns "$NAMESPACE" >/dev/null || { log ERROR "no such namespace"; exit 3; }
kubectl get pods -n "$NAMESPACE" -o json > "$OUTDIR/objects/pods.json"
kubectl get deployments -n "$NAMESPACE" -o json > "$OUTDIR/objects/deployments.json"
kubectl get endpointslices -n "$NAMESPACE" -o json > "$OUTDIR/objects/endpointslices.json"
kubectl get pdb -n "$NAMESPACE" -o json > "$OUTDIR/objects/pdb.json"
capture_objects; capture_events; capture_logs; capture_nodes; capture_metrics
build_timeline; classify; generate_report
tar czf "${OUTDIR}.tar.gz" -C "$(dirname "$OUTDIR")" "$(basename "$OUTDIR")"
log INFO "archive: ${OUTDIR}.tar.gz"
}
main
Step 6: Verification
Create three unrelated, simultaneous failures — the realistic case, and the one a single-purpose script gets wrong.
kubectl create namespace forensics-test
# 1. CrashLoopBackOff
kubectl create deploy crasher --image=busybox -n forensics-test -- /bin/sh -c 'exit 1'
# 2. Service with no endpoints (selector typo)
kubectl create deploy web --image=nginx -n forensics-test
kubectl expose deploy web --port=80 -n forensics-test --selector=app=wrong-label
# 3. OOMKilled
kubectl run hog --image=polinux/stress -n forensics-test --restart=Never \
--overrides='{"spec":{"containers":[{"name":"hog","image":"polinux/stress",
"resources":{"limits":{"memory":"32Mi"}},
"command":["stress","--vm","1","--vm-bytes","256M"]}]}}'
sleep 120
./k8s-forensics -n forensics-test -s 30m
less forensics-*/REPORT.md
The report must identify all three, correctly timestamped, with no further commands from you. If it finds two, the classifier is incomplete — which is exactly the gap the exercise exists to expose.
kubectl delete namespace forensics-test
Extensions worth building: attach the output to a Slack thread automatically; run it from a CronJob so events are captured continuously rather than reactively; add the deploy history from kubectl rollout history so “what changed” is answered in the same report; emit JSON for ingestion into an incident-management tool.
The lesson to take away: the value is not in any single query — it is in capturing before the evidence expires and normalising so correlation is a sort. Events are gone in an hour; a tool that runs in thirty seconds during the incident is worth more than perfect analysis performed the next morning against data that no longer exists.