Most tools do what you tell them. Kubernetes does not — you tell it what you want, and a set of processes work continuously to make that true. Every confusing behaviour in this module resolves once that distinction is solid.
Topic 1: The Problem Orchestration Solves
Containers gave us a reliable unit of packaging. They did not answer any of the operational questions:
- A container exited at 3am. Who restarts it?
- The host died. Who moves its twelve containers, and where?
- Traffic tripled. Who starts more, and who tells the load balancer they exist?
- You want to ship a new version without dropping requests. Who sequences that?
- A config value changed. Who rolls it out, and who rolls it back when it is wrong?
Before orchestration, the answer to each was a bespoke script, a runbook, and a person. Kubernetes exists to make all of them properties of a declared specification rather than a sequence of actions somebody has to perform.
Its lineage matters:
Kubernetes descends from Borg, the system Google ran internally for a decade before open-sourcing the ideas in 2014. That heritage explains its shape: it assumes machines fail routinely, that scheduling is a bin-packing problem, and that the operator’s job is to describe intent rather than drive machinery. It has been a CNCF project since 2016.
This module is pinned to Kubernetes 1.36, the current stable release. Where a feature depends on a version, the lesson says so explicitly — the supported window is 1.34–1.36, and material written even two years ago is frequently wrong about the specifics.
Topic 2: Imperative vs Declarative — the Real Difference
# Imperative: a sequence of actions you are responsible for
docker run -d --name api --restart=always myapp:1.2
docker run -d --name api2 --restart=always myapp:1.2
# ...and when the host dies, you run them somewhere else. Personally.
# Declarative: a statement of fact that something else keeps true
apiVersion: apps/v1
kind: Deployment
metadata:
name: api
spec:
replicas: 3
template:
spec:
containers:
- name: api
image: myapp:1.2
The YAML is not a script. It is a claim about how the world should look. Nothing in it says “start a container”. It says three of these should exist, and a controller spends the rest of its life ensuring that.
This is why kubectl apply on an unchanged file does nothing, why deleting a pod is pointless if a Deployment owns it, and why there is no “run this once” verb — the model has no concept of an action, only of a state that is or is not currently true.
Topic 3: The Reconciliation Loop
A controller does three things, forever:
- Observe the desired state (the
spec, from etcd) and the actual state (thestatus, reported by whatever is running it). - Diff them.
- Act to close the gap — then loop, and check again.
Two properties fall out of this, and both matter operationally:
It is level-triggered, not edge-triggered. A controller does not react to events (“a pod was deleted”); it reacts to the current difference between spec and status. If a controller is down for ten minutes and comes back, it does not need a replay of what it missed — it just looks at the world and fixes whatever is wrong. This is why Kubernetes tolerates its own components restarting.
It never assumes success. After creating a pod, the controller does not mark the job done. It checks again on the next iteration, and the one after. Re-running a reconcile that has nothing to do is cheap and safe by design — that property is called idempotence, and it is the reason kubectl apply can be run a thousand times with the same result.
# Watch a loop work. Delete a pod a Deployment owns:
kubectl delete pod api-7d9f-x2k4
# pod "api-7d9f-x2k4" deleted
kubectl get pods -l app=api
# a new pod, with a new name, already Running
Nothing “handled the deletion”. The ReplicaSet controller observed that it wanted 3 and had 2, and created one. It would have done exactly the same if the node had caught fire.
Common mistake: Reading this as “Kubernetes restarted my pod”. It did not restart anything — the old pod is gone permanently. A new pod was created to satisfy the replica count. That distinction matters the moment you have local state on disk.
Topic 4: Spec and Status
Every Kubernetes object has the same two-part shape, and once you see it you see it everywhere:
apiVersion: apps/v1
kind: Deployment
metadata:
name: api
spec: # ← WHAT YOU WANT. You own this.
replicas: 3
status: # ← WHAT IS. The system owns this.
replicas: 3
readyReplicas: 2
unavailableReplicas: 1
specis written by you (or by a controller acting as a user). It is intent.statusis written by the controller. It is observation. Editing it by hand achieves nothing — the next reconcile overwrites it.
Debugging is very often just comparing the two:
kubectl get deploy api -o jsonpath='{.spec.replicas}{"\n"}{.status.readyReplicas}{"\n"}'
# 3
# 2 ← the gap IS the problem. Now find out why.
That gap is the single most useful thing to look at first. 3 desired, 2 ready immediately tells you the system is trying and failing, rather than not trying — which rules out half the possible causes before you have run a second command.
Conditions — the structured version:
Most objects also carry a conditions array, which is where controllers record why the gap exists:
kubectl get deploy api -o jsonpath='{range .status.conditions[*]}{.type}={.status} {.reason}{"\n"}{end}'
# Available=False MinimumReplicasUnavailable
# Progressing=False ProgressDeadlineExceeded
ProgressDeadlineExceeded is a controller telling you, in one word, that it gave up waiting. Reading conditions is faster than reading logs and is usually the second command of any investigation.
Topic 5: Controllers Are Not Special
The controllers that ship with Kubernetes — ReplicaSet, Deployment, Job, Node, EndpointSlice — have no privileged status. They are ordinary clients of the API doing the loop above. This is not a design detail; it is the extension mechanism.
When you write an operator (covered later in this module), you write the same loop against your own resource type, and it is a first-class citizen because there was never a second class.
built-in controller: watch Deployments → create/delete ReplicaSets
your operator: watch PostgresDBs → create StatefulSets, Secrets, backups
The system does not distinguish between these. That symmetry is why the ecosystem grew the way it did: cert-manager, Prometheus Operator, ArgoCD and Karpenter are all just loops watching resources and acting.
Topic 6: What This Means When Things Break
Because the model is declarative and looping, the diagnostic questions are always the same three, in order:
1. What is the desired state? — kubectl get <kind> <name> -o yaml, read spec. Is it what you think you asked for? A surprising share of incidents end here, at a typo’d image tag or a selector that matches nothing.
2. What is the actual state, and what is the gap? — read status and conditions.
3. Why can the controller not close the gap? — kubectl describe and read the Events at the bottom. Events are the controller narrating its own failures: FailedScheduling, ImagePullBackOff, FailedMount.
kubectl describe pod api-7d9f-x2k4 | tail -20
kubectl get events --sort-by=.lastTimestamp | tail -20
Events are not logs. They are short-lived (about an hour by default) and they belong to the object, not to your application. kubectl logs shows you what your code said; events show you what Kubernetes said about your code. Beginners reach for logs first and miss that the pod never started, so there are no logs to read.
The corollary that trips people up:
Because controllers loop forever, manual changes get reverted. Edit a pod that a Deployment owns and the change survives until the next reconcile. Scale a Deployment by hand while GitOps watches the repo and your change is undone within seconds. The system is not fighting you — it is doing exactly what you asked, which was to keep the declared state true.
The fix is always to change the declaration, never the instance.
Try it yourself: Scale a Deployment to 5 with kubectl scale, then kubectl edit one of its pods and change a label the selector depends on. Watch what happens to the replica count, and work out why the number went up.
Common mistake: Treating kubectl apply as “deploy” and assuming that when it returns, the change is live. apply returns as soon as the spec is durably written to etcd — step 3 of 6. Everything after that is asynchronous. A green CI step that runs kubectl apply and exits has verified nothing at all; use kubectl rollout status to actually wait.