Cluster Anatomy: Control Plane & Node Components

A cluster is about seven processes. Knowing which one owns which decision turns “Kubernetes is broken” into a specific, answerable question.

Topic 1: The Map

Topic 2: Control Plane Components

kube-apiserver — the front door

Every read and every write, from kubectl, from controllers, from the kubelet, goes through the API server. It handles authentication, authorisation, admission, validation and persistence.

It is deliberately the only component that talks to etcd. That single choice buys the entire system:

• One place to enforce authn/authz/admission — there is no back channel.
• One place to validate schemas, so no component can write malformed state.
• One place to implement watches, so controllers get pushed changes rather than polling.
• etcd’s client protocol never leaks into components, so it can be swapped or upgraded independently.

It is stateless, so you run several behind a load balancer for HA.

kubectl get --raw='/readyz?verbose'      # every readiness check, individually
kubectl get --raw='/livez?verbose'

etcd — the only source of truth

A distributed key-value store holding the entire cluster state: every object’s spec and status. It uses Raft for consensus, which has two operational consequences people learn the hard way:

• You need an odd number of members (3 or 5). Raft needs a majority quorum; 4 members tolerate the same single failure as 3 while costing more to write.
• It is extremely sensitive to disk latency. Every write is fsynced to a majority. Put etcd on slow or shared disks and the whole API server becomes slow, which makes the entire cluster feel broken for reasons no application-level metric explains.

ETCDCTL_API=3 etcdctl endpoint status --write-out=table \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

Ports 2379 (client) and 2380 (peer). Back it up — this lesson’s follow-up on etcd operations covers snapshots and restore, and it is the one component whose loss is unrecoverable.

kube-scheduler — placement only

Watches for pods with no nodeName set, picks a node, and writes the binding. That is its entire job. It does not start containers and it does not monitor them.

It runs two phases, covered fully in the scheduling lesson: filter (which nodes could take this pod) then score (which is best).

If the scheduler is down, existing pods keep running untouched; new pods simply sit in Pending forever.

kube-controller-manager — the loops

A single binary running dozens of controllers in goroutines: ReplicaSet, Deployment, Job, CronJob, Node, EndpointSlice, ServiceAccount, PersistentVolume and more. Each is the loop from lesson 1.

It uses leader election — in an HA setup, several run but only one is active, because two controllers both creating pods for the same ReplicaSet would double everything.

cloud-controller-manager — the provider boundary

Split out of the controller manager so that cloud-specific code lives on its own release cycle. It owns node lifecycle (does this VM still exist?), route configuration, and Service type: LoadBalancer provisioning. On EKS/GKE/AKS your provider runs this; on bare metal you may have none, which is why type: LoadBalancer hangs in Pending on a kind cluster.

Topic 3: Node Components

kubelet — the node’s agent

The kubelet owns everything on its node. It watches the API server for pods bound to it, tells the container runtime to start them, mounts volumes, runs probes, and reports status back.

Two behaviours worth internalising:

• The kubelet is the only thing that can create a container. Nothing else touches the runtime. If pods are stuck ContainerCreating, that is the kubelet or something it depends on (CNI, CSI, image pull) — never the scheduler.
• A kubelet that loses contact with the API server keeps running its pods. It cannot report status, so after node-monitor-grace-period (40s by default) the node controller marks the node NotReady and, after a further timeout, evicts its pods on paper. The containers may still be running on the isolated node — which is why network partitions produce duplicate workloads unless you fence properly.

kube-proxy — service routing

Programs the node’s iptables or IPVS rules so that a Service’s ClusterIP is DNAT’d to a real pod IP. It moves no packets itself; the kernel does. Covered in depth in the pod networking lesson.

Some CNIs (Cilium in kube-proxy replacement mode) remove it entirely and implement Services in eBPF.

Container runtime — CRI

containerd or CRI-O, spoken to over the Container Runtime Interface. Docker’s direct integration (dockershim) was removed in 1.24 — images built with Docker still run fine, because they are OCI images; only the runtime shim changed.

CNI and CSI plugins

Not “components” in the same sense, but the kubelet cannot function without them:

• CNI gives each pod its network identity. No CNI, and every pod stays ContainerCreating.
• CSI attaches and mounts volumes.

Topic 4: What Breaks When Each Dies

This table is the reason to learn the components at all.

The pattern is worth stating: the control plane going down does not take your application down. Running pods keep serving because the data path — kubelet, runtime, kernel routing — is independent of the control path. What you lose is the ability to change anything or recover from anything. A cluster with a dead control plane is a cluster serving traffic on borrowed time.

Topic 5: Reading a Node

kubectl get nodes -o wide
kubectl describe node ip-10-0-1-42

Three blocks in that output do most of the work:

Conditions — the node’s own health report:

Type                 Status
MemoryPressure       False
DiskPressure         False
PIDPressure          False
Ready                True

Ready=False means the kubelet has stopped reporting or has declared itself unhealthy. DiskPressure=True triggers image garbage collection and then pod eviction, and is a common cause of pods being evicted from a node that has plenty of memory.

Capacity vs Allocatable — the difference matters:

Capacity:            cpu: 4      memory: 16305468Ki
Allocatable:         cpu: 3920m  memory: 15154892Ki

Allocatable is what the scheduler may hand out. The gap is reserved for the kubelet, the OS and eviction thresholds (--kube-reserved, --system-reserved, --eviction-hard). Capacity planning against Capacity rather than Allocatable overcommits every node by a few percent, which is exactly enough to cause eviction under load.

Allocated resources — the sum of requests on the node, not actual usage:

Resource   Requests      Limits
cpu        2100m (53%)   4 (102%)
memory     3Gi (20%)     6Gi (40%)

This is the number the scheduler cares about. A node showing 53% CPU requested may be at 5% real utilisation, and it will still refuse a pod requesting 2 cores. Scheduling is arithmetic on requests; it never looks at actual usage.

kubectl top nodes                       # ACTUAL usage (needs metrics-server)
kubectl describe node <n> | grep -A6 'Allocated resources'   # REQUESTED

Comparing those two numbers is how you find a cluster that is “full” while idle.

Topic 6: Static Pods and the Bootstrap Chicken-and-Egg

A reasonable question: if the API server runs as a pod, and pods are created by the API server, how does the first one start?

Static pods. The kubelet watches a directory on disk — /etc/kubernetes/manifests/ — and runs anything it finds there directly, with no API server involved. On a kubeadm cluster the control plane itself is four static pods:

ls /etc/kubernetes/manifests/
# etcd.yaml  kube-apiserver.yaml  kube-controller-manager.yaml  kube-scheduler.yaml

The kubelet then creates read-only mirror pods in the API so you can see them with kubectl. You cannot delete them that way — deleting the mirror just makes the kubelet recreate it. To change a control-plane component on a self-managed cluster you edit the file on disk, and the kubelet restarts it within seconds.

On managed clusters (EKS, GKE, AKS) the control plane is the provider’s problem and you will not see these at all — which is a real difference in what you can debug.

Try it yourself: Run kubectl get pods -n kube-system -o wide and classify each pod: control plane, node agent (a DaemonSet on every node), or add-on. Then find which of them are mirror pods with kubectl get pod <name> -n kube-system -o jsonpath='{.metadata.annotations}'.

Common mistake: Debugging a Pending pod by looking at the kubelet. Pending means no node has been assigned, which is the scheduler’s domain — the kubelet has never heard of the pod. ContainerCreating is the kubelet’s domain. Getting that boundary right saves you from reading the wrong logs entirely.