Observability: Metrics, Logs, Events & Traces

Four signals, and the common mistake is treating them as interchangeable. Metrics tell you something is wrong. Logs tell you what the code said. Events tell you what Kubernetes did. Traces tell you where the time went.

Topic 1: The Metrics Pipeline

Two distinct paths, and confusing them wastes hours:

metrics.k8s.io — the resource metrics API, served by metrics-server. Powers kubectl top and the HPA. In-memory only, roughly the last minute, no history.

Prometheus — scrapes and stores time series. Everything else.

kubectl top nodes
kubectl top pods --containers -A --sort-by=memory
kubectl get --raw /apis/metrics.k8s.io/v1beta1/nodes | jq '.items[0]'

If kubectl top fails, metrics-server is missing or broken — and your HPAs are not working either, silently. It is not installed by default on many distributions.

Where the numbers come from:

cAdvisor and kube-state-metrics answer different questions and you need both. cAdvisor says “this container is using 300Mi”; kube-state-metrics says “this Deployment wants 5 replicas and has 3”. A rollout stuck at 3/5 is invisible in cAdvisor and obvious in kube-state-metrics.

Topic 2: The Metrics That Predict Incidents

Workload health

# Replicas wanted vs available — a persistent gap IS the incident
kube_deployment_spec_replicas - kube_deployment_status_replicas_available > 0

# Restarts — the strongest early signal there is
increase(kube_pod_container_status_restarts_total[15m]) > 3

# CrashLoopBackOff and other waiting reasons
kube_pod_container_status_waiting_reason{reason!="ContainerCreating"} == 1

# Pods stuck Pending
kube_pod_status_phase{phase="Pending"} == 1

Resource pressure

# Memory usage against the LIMIT — approaching 1 means an OOMKill is coming
container_memory_working_set_bytes / on(pod,container)
  kube_pod_container_resource_limits{resource="memory"} > 0.9

# CPU THROTTLING — the invisible latency killer from the resources lesson
rate(container_cpu_cfs_throttled_periods_total[5m])
  / rate(container_cpu_cfs_periods_total[5m]) > 0.25

# Cluster-level: requested vs allocatable
sum(kube_pod_container_resource_requests{resource="cpu"})
  / sum(kube_node_status_allocatable{resource="cpu"})

Use container_memory_working_set_bytes, not container_memory_usage_bytes — the latter includes reclaimable page cache and will have you provisioning memory you do not need.

Control plane

histogram_quantile(0.99, rate(apiserver_request_duration_seconds_bucket[5m]))
rate(apiserver_request_total{code=~"5.."}[5m])
histogram_quantile(0.99, rate(etcd_disk_wal_fsync_duration_seconds_bucket[5m]))  # etcd disk!
etcd_server_has_leader == 0

etcd fsync latency is the leading indicator of a slow cluster. When it rises, every API call slows, controllers lag, and the whole cluster feels broken for reasons no application metric explains. Alert on it.

Node-level

kube_node_status_condition{condition="Ready",status="true"} == 0
kube_node_status_condition{condition="DiskPressure",status="true"} == 1
node_nf_conntrack_entries / node_nf_conntrack_entries_limit > 0.8   # from the network lesson

The four golden signals, per service:

# Latency
histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket{job="api"}[5m])))
# Traffic
sum(rate(http_requests_total{job="api"}[5m]))
# Errors
sum(rate(http_requests_total{job="api",code=~"5.."}[5m])) / sum(rate(http_requests_total{job="api"}[5m]))
# Saturation
rate(container_cpu_cfs_throttled_periods_total{pod=~"api-.*"}[5m])

Alert on symptoms, not causes. “Error rate above 1% for 5 minutes” is worth waking someone. “CPU above 80%” is not — it may be entirely healthy, and it fires constantly. Page on what users experience; graph everything else.

Topic 3: Events — the Signal People Forget

Events are Kubernetes narrating its own decisions, and they answer questions no other signal can.

kubectl get events -A --sort-by=.lastTimestamp | tail -30
kubectl get events --field-selector type=Warning -A
kubectl get events --field-selector involvedObject.name=api-7d9f-x2k4
kubectl describe pod api-7d9f-x2k4 | tail -20     # events for one object

Events expire, by default after one hour (--event-ttl on the API server). Investigating an incident from yesterday means the events are gone. This surprises people repeatedly and it is worth fixing before you need it:

• Ship events to your logging backend — the Prometheus event-exporter, Fluent Bit’s Kubernetes events input, or a similar collector.
• Or raise --event-ttl, at the cost of etcd space.

Events worth alerting on:

Warning  FailedScheduling     no nodes available
Warning  Unhealthy            probe failed
Warning  BackOff              CrashLoopBackOff
Warning  Failed               ImagePullBackOff
Warning  Evicted              node under pressure
Warning  FailedMount          volume problems
Warning  NodeNotReady
Normal   Killing              container failed liveness probe

Note events are also deduplicated and counted rather than repeated — a count: 847 on one event means it happened 847 times, which is easy to miss when skimming.

Topic 4: Logs

kubectl logs api-7d9f-x2k4
kubectl logs api-7d9f-x2k4 --previous              # the instance that CRASHED
kubectl logs -l app=api --all-containers --tail=100 --prefix
kubectl logs -f deploy/api --since=15m
kubectl logs api-7d9f-x2k4 -c sidecar --timestamps

--previous is the one that matters in a crash loop: kubectl logs shows the current attempt, which usually has not produced output yet. The error is in the instance that died.

How logs actually get collected:

Containers write to stdout/stderr → the runtime writes to /var/log/pods/... on the node → a DaemonSet collector (Fluent Bit, Vector, Promtail) reads and ships them.

That is why logging to stdout is the correct pattern. An application writing to a file inside the container needs a sidecar to ship it — one extra container per pod instead of one agent per node.

Node log rotation is silent data loss. The kubelet rotates container logs (default 10Mi × 5 files). A pod logging heavily can lose its own history within minutes, and kubectl logs will happily show you what remains without telling you anything was dropped. If you need the logs, ship them off-node.

Structured logging pays for itself:

{"ts":"2026-08-07T14:22:07Z","level":"error","msg":"payment failed",
 "trace_id":"a3f9c1","order_id":"ORD-4471","duration_ms":1240}

Grepping unstructured logs across 200 pods does not scale. Structured logs let you query by field, and — critically — carry a trace_id that links a log line to a trace.

Topic 5: Traces

Metrics tell you the p99 is 2 seconds. Traces tell you which of the eleven services consumed it.

OpenTelemetry is the vendor-neutral standard and is where the ecosystem has converged. Instrument once; export to Jaeger, Tempo, Honeycomb or Datadog.

env:
  - name: OTEL_EXPORTER_OTLP_ENDPOINT
    value: "http://otel-collector.observability:4317"
  - name: OTEL_SERVICE_NAME
    value: "api"
  - name: OTEL_RESOURCE_ATTRIBUTES
    value: "deployment.environment=production,k8s.namespace.name=$(POD_NAMESPACE)"

Two things make traces useful rather than decorative:

Context propagation. Every service must forward the traceparent header. Miss one and the trace breaks in half at exactly the point you needed to see.

Correlation with logs. Put the trace_id in every log line. Then a slow trace links directly to the log output of the specific request, and an error log links to the full request path. That connection is where most of the value lives.

A service mesh gives you traces for free at the network layer — but without application spans you see service-to-service hops and nothing about what happened inside a service.

Topic 6: A Practical Stack, and What to Alert On

Metrics:  Prometheus (or Mimir/Thanos for long-term) + kube-state-metrics + node-exporter
Logs:     Fluent Bit / Vector DaemonSet → Loki / Elasticsearch / cloud provider
Traces:   OpenTelemetry SDK → OTel Collector → Tempo / Jaeger
Events:   event-exporter → your log backend
Dashboards + alerting: Grafana + Alertmanager

kube-prometheus-stack (Helm) gives you Prometheus, Alertmanager, Grafana, kube-state-metrics, node-exporter and a large set of well-tested default rules in one install. Starting there and pruning is far faster than assembling it yourself.

The alerts worth having on day one:

- alert: KubePodCrashLooping
  expr: rate(kube_pod_container_status_restarts_total[15m]) * 60 * 15 > 3
  for: 15m
- alert: KubePodNotReady
  expr: kube_pod_status_phase{phase=~"Pending|Unknown"} == 1
  for: 15m
- alert: KubeDeploymentReplicasMismatch
  expr: kube_deployment_spec_replicas != kube_deployment_status_replicas_available
  for: 15m
- alert: NodeNotReady
  expr: kube_node_status_condition{condition="Ready",status="true"} == 0
  for: 5m
- alert: PVCAlmostFull
  expr: kubelet_volume_stats_available_bytes / kubelet_volume_stats_capacity_bytes < 0.1
  for: 10m
- alert: CPUThrottlingHigh
  expr: rate(container_cpu_cfs_throttled_periods_total[5m]) / rate(container_cpu_cfs_periods_total[5m]) > 0.25
  for: 15m
- alert: EtcdHighFsyncDuration
  expr: histogram_quantile(0.99, rate(etcd_disk_wal_fsync_duration_seconds_bucket[5m])) > 0.5
  for: 10m

The for: clause is what separates a useful alert from noise — it requires the condition to hold, which filters transient blips that resolve themselves.

PVC utilisation is the one people forget entirely. Nothing in Kubernetes warns you that a volume is filling, and a full volume takes down a database with no prior signal.

Try it yourself: Delete a pod and immediately run kubectl get events --sort-by=.lastTimestamp | tail. Watch the Killing, Scheduled, Pulled, Created, Started sequence — that is the whole reconciliation loop narrated in real time. Then wait an hour and confirm those events are gone.

Common mistake: Alerting on resource utilisation instead of user-visible symptoms. “Memory above 80%” fires on every healthy long-running pod, teaches everyone to ignore alerts, and misses the actual incident. Alert on error rate, latency, and replicas-not-available; keep utilisation on dashboards where it belongs.