Kubernetes
From the reconciliation loop to production operations β workloads, scheduling, the pod network, RBAC, autoscaling, upgrades and etcd. Pinned to Kubernetes 1.36.
Stage 1 β Fundamentals
6 lessonsThe single idea the whole system is built on: you declare a desired state, and controllers loop forever making reality match. Everything else in Kubernetes is a consequence of this.
Every process that makes a cluster work, what breaks when each one dies, and why the API server being the only writer to etcd is the most important design decision in the system.
What actually happens between typing kubectl apply and a container starting, why apply differs from create, and how to read any object in the cluster without memorising its schema.
Why the smallest deployable thing is a group of containers rather than one, what they actually share, and how to read every phase and waiting reason a pod can get stuck in.
The loose coupling that holds the whole system together: how objects find each other without ever naming each other, and the selector mistakes that silently orphan workloads.
How a stable name reaches a shifting set of pod IPs, what every Service type actually provisions, and why 'no endpoints' is nearly always one of two problems.
Stage 2 β Workloads & Configuration
6 lessonsWhat actually happens during a rolling update, why maxSurge and maxUnavailable decide your blast radius, and the three reasons a rollout stalls forever.
The four controllers beyond Deployment: when stable identity is worth the operational cost, why a DaemonSet ignores the scheduler, and the CronJob settings that quietly pile up work.
How configuration reaches a container, why a mounted ConfigMap updates but an env var never does, and why Secrets are not encrypted unless you make them so.
Three probes with three different consequences, why a liveness probe is the most dangerous thing in a pod spec, and how a misconfigured probe takes down a healthy service.
Ordered setup before your app starts, and the 1.33 change that finally fixed sidecars β why a Job with an Istio proxy used to hang forever, and what replaced the workarounds.
Why requests and limits do completely different things, how QoS class decides who dies first, and the CPU throttling that destroys latency while every memory metric looks fine.
Stage 3 β Storage
1 lessonThe abstraction from a pod's mount path down to a real disk, why access modes are not what most people assume, and the multi-zone scheduling deadlock that leaves a pod Pending forever.
Stage 4 β Scheduling & Scaling
2 lessonsHow the scheduler actually picks a node, the difference between a preference and a requirement, and why anti-affinity that looks correct still puts every replica in one zone.
Three axes of scaling that operate on different things and interact badly if you let them, plus the reason your HPA does nothing at all.
Stage 5 β Networking
3 lessonsFollow one packet from client pod to server pod, name every hop that can drop it, and understand the two silent killers β conntrack exhaustion and MTU mismatch.
One entry point for many Services, why Ingress stalled as a spec, and how Gateway API's role-oriented model replaces annotation soup β with a migration path that does not require a flag day.
Why every pod can reach every other pod by default, how policies compose additively, and the two mistakes that make a policy either useless or an outage.
Stage 6 β Security
3 lessonsWho may do what to which resources, why RBAC is purely additive, and how to debug a permission error in one command instead of guessing at YAML.
PodSecurityPolicy is gone. What replaced it, what the three built-in standards actually enforce, and how to harden a pod spec so a container compromise stays inside the container.
Take an app from a bare Deployment to something that survives a node drain, a bad deploy, a slow dependency and a hostile pod β verified by breaking each one deliberately.
Stage 7 β Operations & Production
8 lessonsTwo ways to stop copy-pasting YAML across environments, why they solve different problems, and the deployment model that makes the cluster match a Git repository rather than whoever ran kubectl last.
Four signals with four different jobs, the metrics that actually predict incidents, and why events disappear an hour after the thing you needed to investigate.
How to replace every node in a cluster without dropping traffic, why your drain hangs forever, and the API deprecation that breaks a controller three versions later.
The one component whose loss is unrecoverable: how to snapshot it, how to restore it, and why its disk latency decides whether your whole cluster feels healthy.
Sharing a cluster safely: what namespace isolation does and does not buy you, the quota interaction that breaks every deployment in a namespace, and where the money actually goes.
Sizing a cluster from real numbers rather than guesses, the limits you will hit before CPU, and a checklist that separates a workload that survives a bad Tuesday from one that does not.
A fixed sequence for the six failures you will actually meet, ordered so each command eliminates a layer β and the triage that works when you do not yet know what is wrong.
Plan and execute a full minor-version upgrade with a written runbook, a pre-flight that catches removed APIs, node-by-node draining and a tested rollback β measured by requests dropped.
Stage 8 β Extending Kubernetes
3 lessonsHow to extend Kubernetes itself: intercept every write, define your own resource types, and write the controller that reconciles them β plus the webhook mistake that can lock a cluster.
Extend Kubernetes with your own resource type and the reconciliation loop behind it β including the finalizer, the status conditions and the idempotence that separate an operator from a script.
Build the tool that answers 'what happened' across every layer at once β objects, events, metrics, logs and node state β correlated into one timeline you can paste into a postmortem.
πΊοΈ Beginner β Expert Roadmap
8 stages with prerequisites and a concrete mastery check at each.
π― What You'll Learn
- β’ Explain every Kubernetes behaviour as a reconciliation loop rather than a command.
- β’ Trace a kubectl apply through admission, etcd, the controller and the scheduler to a running container.
- β’ Diagnose a Pending pod, a CrashLoopBackOff and a Service with no endpoints without guessing.
- β’ Size requests and limits deliberately, and recognise CPU throttling that never appears as an OOMKill.
- β’ Follow a packet from client pod to server pod and name every hop that can drop it.
- β’ Write RBAC and Pod Security Admission that is least-privilege rather than cargo-culted.
- β’ Choose between HPA, VPA, Cluster Autoscaler and Karpenter on their real trade-offs.
- β’ Upgrade a cluster without a PDB blocking the drain, and restore etcd from a snapshot.
- β’ Build a CRD and controller, and know when an operator is the wrong answer.
- β’ Ship four production projects: a hardened workload, an operator, an upgrade runbook and an incident forensics capstone.
πΌ Interview Readiness
Once you reach the end of this path, test your engineering knowledge against real questions asked by top technical teams:
A production pod shows status Evicted. What should you check first, and why?
KubernetesDesign a systematic diagnostic sequence for a pod stuck in Pending state in an EKS cluster where all standard component health checks (nodes, CNI, kube-proxy) report healthy.
KubernetesA pod is in Pending state. Where do you look first?
KubernetesExplain the "networking stack alive for TCP but dead for ICMP/ARP" scenario. What causes it and what does it indicate?
Linux & NetworkingIf a TCP handshake to port 22 succeeds via netcat, but ping to the same host fails completely, what does that tell you?
Linux & NetworkingWalk through a structured approach to diagnosing an ambiguous "intermittent connectivity issue" on a Linux server, from the outside in.
Structured Debugging