Pods: The Atomic Unit

A pod is not “a container with extra steps”. It is a shared execution context — and the reason it exists is that some containers genuinely need to be scheduled together, on one machine, sharing an address.

Topic 1: What a Pod Actually Is

A pod is one or more containers that share:

They do not share a filesystem root or a PID namespace by default (shareProcessNamespace: true opts in).

The pause container:

Every pod has an invisible extra container — pause — that does nothing but hold the namespaces open. It is the parent that owns the network namespace, so application containers can restart without the pod losing its IP. You will see it on the node with crictl ps and never in kubectl.

That detail explains a real behaviour: a container restarting keeps the pod’s IP, but a pod being replaced gets a new one. Anything caching a pod IP is broken by design.

Topic 2: Why Group Containers At All

The honest default is one container per pod. Multi-container pods exist for a specific shape: a helper that is useless without its main container and must be on the same machine.

Three legitimate patterns:

Sidecar — augments the main container. A log shipper reading a shared emptyDir, a service-mesh proxy intercepting traffic, a credential refresher writing to a shared volume. Since 1.33 sidecars have first-class support, covered in its own lesson.

Ambassador — proxies outbound connections. The app connects to localhost:6379 and the ambassador handles sharding, TLS or failover to the real backend.

Adapter — reshapes output. The app writes its own log format; the adapter transforms it into what your platform expects.

apiVersion: v1
kind: Pod
metadata:
  name: web-with-shipper
spec:
  volumes:
    - name: logs
      emptyDir: {}
  containers:
    - name: web
      image: nginx:1.27
      volumeMounts:
        - name: logs
          mountPath: /var/log/nginx
    - name: log-shipper
      image: fluent-bit:3.1
      volumeMounts:
        - name: logs
          mountPath: /logs
          readOnly: true

When NOT to group: if the two things can scale independently, be released independently, or run on different machines, they are two pods and a Service. Putting a frontend and a backend in one pod is the classic beginner error — you can no longer scale them separately, and a change to either forces a restart of both.

Topic 3: Pod Phases

Running does not mean healthy. It means a container process exists. A pod can be Running and failing every readiness probe, serving nothing.

kubectl get pods -o custom-columns='NAME:.metadata.name,PHASE:.status.phase,READY:.status.containerStatuses[*].ready,RESTARTS:.status.containerStatuses[*].restartCount'

That READY column is the one that matters. Running + false is a service that is up but taking no traffic.

Topic 4: The Waiting Reasons — a Diagnostic Table

Pending and Running are too coarse. The useful signal is status.containerStatuses[].state.waiting.reason:

And for Pending specifically, the answer is always in Events:

kubectl describe pod api-xyz | tail -20
# 0/6 nodes are available: 3 Insufficient cpu, 2 node(s) had taint
#   {node-role.kubernetes.io/control-plane: }, 1 node(s) had volume node affinity conflict.

That single line is a complete diagnosis: it enumerates every node and why each was rejected. Reading it carefully replaces an hour of speculation.

CrashLoopBackOff is not an error — it is a backoff:

The container is exiting and Kubernetes is waiting before retrying, with exponential delay (10s, 20s, 40s… capped at 5 minutes). The cause is whatever made the process exit.

kubectl logs api-xyz --previous          # the instance that DIED — this is the one you want
kubectl logs api-xyz -c sidecar --previous
kubectl get pod api-xyz -o jsonpath='{.status.containerStatuses[0].lastState.terminated}' | jq

That last command gives you the exit code and reason directly:

• exit 0 in a Deployment — the process completed. Your entrypoint is not a long-running server.
• exit 1 — application error. Read the logs.
• exit 137 = 128 + 9 (SIGKILL) — almost always OOMKilled. Check reason.
• exit 143 = 128 + 15 (SIGTERM) — orderly shutdown; something asked it to stop.

Topic 5: The Pod Spec Fields That Matter Early

apiVersion: v1
kind: Pod
metadata:
  name: api
  labels:
    app: api
spec:
  serviceAccountName: api-sa
  securityContext:                  # pod-level: applies to all containers
    runAsNonRoot: true
    runAsUser: 10001
    fsGroup: 10001
    seccompProfile:
      type: RuntimeDefault
  containers:
    - name: api
      image: registry.example.com/api:1.4.2       # NEVER :latest
      imagePullPolicy: IfNotPresent
      command: ["/app/server"]                     # overrides ENTRYPOINT
      args: ["--port=8080"]                        # overrides CMD
      ports:
        - name: http
          containerPort: 8080
      env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name             # downward API
      securityContext:
        allowPrivilegeEscalation: false
        readOnlyRootFilesystem: true
        capabilities:
          drop: ["ALL"]
      resources:
        requests: { cpu: 100m, memory: 128Mi }
        limits:   { memory: 256Mi }
  terminationGracePeriodSeconds: 30
  restartPolicy: Always

Three things worth arguing about:

Never :latest. The tag is mutable, so two pods from the same Deployment can run different code, and a rollback has nothing to roll back to. Pin a tag; pin a digest (@sha256:...) if you need certainty. :latest also silently sets imagePullPolicy: Always.

command/args map to Docker’s ENTRYPOINT/CMD, and the names not matching is a genuine trap. command replaces the entrypoint entirely.

containerPort is documentation. It does not open or restrict anything — the container listens on whatever it listens on. Its value is naming the port so a Service can reference it by name.

restartPolicy applies to the whole pod:

Restarts always happen on the same node, in the same pod. A pod is never rescheduled elsewhere because of a container crash — that only happens if the pod itself is deleted or evicted.

Topic 6: Debugging a Pod You Cannot Exec Into

kubectl exec -it api-xyz -- /bin/sh          # works only if the image HAS a shell

Distroless and scratch images have no shell, which is good for security and inconvenient at 3am. Ephemeral debug containers (stable since 1.25) solve it by attaching a new container to a running pod:

# Attach a debug container sharing the target's namespaces
kubectl debug -it api-xyz --image=nicolaka/netshoot --target=api

# Copy the pod with a modified spec — leaves the original running
kubectl debug api-xyz -it --copy-to=api-debug --container=api -- /bin/sh

# Debug a NODE by starting a pod in its host namespaces
kubectl debug node/ip-10-0-1-42 -it --image=busybox

--target=api puts the debug container in the same process namespace as the app container, so you can see its processes and /proc. nicolaka/netshoot bundles dig, tcpdump, ss, curl and iperf and is worth remembering by name.

Other things worth knowing:

kubectl cp api-xyz:/app/config.yaml ./config.yaml     # pull a file out
kubectl port-forward pod/api-xyz 8080:8080            # reach it without a Service
kubectl attach -it api-xyz                            # stdin/stdout of PID 1

kubectl port-forward is the fastest way to bisect a networking problem: if the app answers on a port-forward but not through the Service, the app is fine and the problem is Service/endpoints/network policy.

Try it yourself: Deploy a pod using a distroless image with no shell. Confirm kubectl exec fails, then get a working shell in its namespace with kubectl debug --target.

Common mistake: Creating bare pods in production. A pod created directly has nothing watching it — when its node dies, it is simply gone, because the thing that would recreate it is the controller you did not use. Always create pods through a Deployment, StatefulSet, Job or DaemonSet; bare pods are for debugging only.