Capacity Planning & Production Readiness

Capacity is arithmetic on requests, not on usage — and the limits that actually stop you are rarely CPU. This lesson closes the module with the numbers and the checklist.

Topic 1: The Three Numbers

# Allocatable — what the scheduler may hand out
kubectl get nodes -o json | jq -r '[.items[].status.allocatable.cpu] | @tsv'
kubectl describe nodes | grep -A5 'Allocatable'

# Requested — what the scheduler has committed
kubectl describe nodes | grep -A5 'Allocated resources'

# Used — what is actually happening
kubectl top nodes

Two ratios tell you almost everything:

Requested ÷ Allocatable — how full the scheduler thinks you are. At 100% nothing new schedules, regardless of real load.

Used ÷ Requested — how honest your requests are. Below ~40% means you are paying for capacity nobody uses; above 100% means you are relying on burst and will be evicted under pressure.

# Cluster-wide request commitment
kubectl get pods -A -o json | jq -r '
  [.items[].spec.containers[].resources.requests.cpu // "0"] | length as $n |
  "containers: \($n)"'

The common finding is requests at 2–4× real usage. That gap is the largest single cost lever in most clusters, and VPA in recommender mode (updateMode: "Off") is the tool that closes it with data rather than opinion.

Remember Allocatable is not Capacity. The difference is reserved for the kubelet, the OS and eviction thresholds. Planning against Capacity overcommits every node by a few percent — exactly enough to cause eviction at the worst moment.

Topic 2: The Limits You Hit Before CPU

Nodes run out of things other than cores, and each has its own ceiling:

The AWS VPC CNI case is worth planning for specifically: an m5.large supports 29 pods because of ENI limits, not because of CPU. A cluster sized on cores will be surprised. Prefix delegation raises this substantially and should usually be enabled.

kubectl get nodes -o custom-columns='NAME:.metadata.name,PODS:.status.allocatable.pods'
kubectl get csinode <node> -o jsonpath='{.spec.drivers[0].allocatable.count}{"\n"}'

Sizing nodes:

A reasonable default is medium nodes (8–16 vCPU), with a rule that no single node holds more than ~10% of any critical workload — which is really a statement about replica count and topology spread.

DaemonSet cost multiplies. A 500Mi request on 200 nodes reserves 100Gi before any application schedules. DaemonSet resources deserve more scrutiny than almost anything else.

Topic 3: Headroom and Growth

Three kinds of headroom, and they are not the same:

1. Failure headroom — survive losing a node (or a zone) without becoming unschedulable. For 3 zones, that means running at ~66% so one zone’s loss still fits.
2. Burst headroom — absorb a traffic spike while the autoscaler provisions. Node provisioning takes 1–5 minutes; you need capacity to cover that window.
3. Growth headroom — normal organic growth before the next capacity review.

Overprovisioning with pause pods is the standard trick for burst headroom: schedule low-priority placeholder pods that hold capacity and get preempted instantly when real work arrives.

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata: { name: overprovisioning }
value: -10                       # NEGATIVE — preempted before anything real
globalDefault: false
---
apiVersion: apps/v1
kind: Deployment
metadata: { name: overprovisioning }
spec:
  replicas: 5
  template:
    spec:
      priorityClassName: overprovisioning
      terminationGracePeriodSeconds: 0
      containers:
        - name: pause
          image: registry.k8s.io/pause:3.10
          resources:
            requests: { cpu: "2", memory: 4Gi }

Those pods do nothing, hold real capacity, and vanish the instant a real pod needs the room — turning a 3-minute node provisioning wait into an instant schedule.

Forecast from trend, not threshold:

predict_linear(
  sum(kube_pod_container_resource_requests{resource="cpu"})[7d:1h], 14*24*3600
) / sum(kube_node_status_allocatable{resource="cpu"})

“Will cross 90% in 14 days” is an alert you can act on during working hours. “Is at 90%” is an alert that arrives at 2am.

Topic 4: Cost, Briefly

The levers, in rough order of impact:

1. Right-size requests. Usually the single largest win. VPA recommender.
2. Consolidate nodes. Karpenter’s WhenEmptyOrUnderutilized repacks continuously.
3. Spot instances for interruptible work, with PDBs and interruption handling.
4. One ingress, not N load balancers. Each type: LoadBalancer is a recurring bill.
5. Scale non-production to zero overnight and at weekends.
6. Reduce cross-AZ traffic — trafficDistribution: PreferClose.
7. Clean up orphaned PVCs left by deleted StatefulSets.

OpenCost or Kubecost give per-namespace and per-label attribution, which is what turns “the cluster costs too much” into “this workload costs too much”.

Topic 5: The Production Readiness Checklist

Run any workload through this before it takes real traffic.

Availability

• replicas >= 2 (3 if a PDB requires 2 available)
• topologySpreadConstraints across zones, not just hostnames
• A PDB with ALLOWED DISRUPTIONS >= 1 — verified, not assumed
• maxUnavailable: 0 on the rollout strategy if capacity must not dip
• minReadySeconds set so a briefly-ready pod does not advance a bad rollout

Health

• readinessProbe that reflects real ability to serve
• startupProbe if the app is slow to boot
• No livenessProbe unless a specific hang exists that a restart fixes
• No dependency checks in the liveness probe
• preStop sleep plus a grace period longer than the slowest request
• The app handles SIGTERM by draining

Resources

• Requests set on every container, including sidecars
• Memory request == limit (Guaranteed, predictable)
• CPU limits generous or absent; throttling monitored
• Numbers derived from measurement, not guessed

Security

• runAsNonRoot, allowPrivilegeEscalation: false, capabilities.drop: [ALL]
• readOnlyRootFilesystem with emptyDir for writable paths
• seccompProfile: RuntimeDefault
• automountServiceAccountToken: false unless the app calls the API
• A dedicated ServiceAccount with least-privilege RBAC
• NetworkPolicy: default-deny plus explicit allows (including DNS)
• Image pinned by tag or digest — never :latest
• Secrets from a secret manager, never committed

Observability

• Structured logs to stdout, with a trace_id
• Metrics exposed and scraped
• Alerts on symptoms (error rate, latency, replicas unavailable)
• Dashboard showing the four golden signals
• A runbook link in an annotation on the object

Data

• Volumes on a WaitForFirstConsumer StorageClass
• reclaimPolicy: Retain for anything you would miss
• Backups running and a restore tested
• PVC utilisation alerted on

Operations

• Deployed from Git, not kubectl apply by hand
• kubectl rollout status gates the pipeline
• Rollback tested, not assumed
• kubernetes.io/change-cause populated
• Owner labels (team, cost-center) present

Topic 6: What “Ready” Actually Means

A workload is production-ready when you can answer these without checking:

• What happens when a node dies? (Replicas elsewhere, PDB permits the drain, no local state lost.)
• What happens when a zone dies? (Spread constraints, not just anti-affinity on hostname.)
• What happens when the database is slow? (Readiness fails, traffic stops, no restart storm.)
• What happens on a bad deploy? (Rollout halts on failing probes; rollback is one command.)
• How do you know it is broken before a user tells you? (Symptom alerts.)
• Who gets paged, and what do they read first? (Owner labels, runbook annotation.)

If any answer is “I am not sure”, that is the next thing to fix — and it will be cheaper to fix now than during the incident that asks the question for you.

Try it yourself: Take your most important workload and answer the six questions above out loud. Then run it through the checklist and count the unticked boxes. That count is your real production-readiness score.

Common mistake: Planning capacity on CPU and memory alone. Pods-per-node, IPs-per-node and volume attachment limits stop clusters far more often, and each produces a FailedScheduling message that names it explicitly — which is only useful if you know those ceilings exist.