Requests, Limits, QoS & CPU Throttling

Requests and limits look like a pair. They are not — they are consumed by two different systems for two different purposes, and almost every capacity problem traces back to conflating them.

Topic 1: Requests Are for the Scheduler. Limits Are for the Kernel.

resources:
  requests:              # ← THE SCHEDULER reads this. Reserved capacity.
    cpu: 100m
    memory: 128Mi
  limits:                # ← THE KERNEL enforces this. A ceiling.
    cpu: 500m
    memory: 256Mi

Two consequences follow immediately:

Scheduling is arithmetic on requests, never on usage. A node with 4 cores whose pods request 3.9 cores will refuse a pod requesting 200m — even if actual CPU usage is 3%. Conversely, a node can be at 95% real CPU and still accept pods, because nothing requested that capacity.

kubectl describe node ip-10-0-1-42 | grep -A8 'Allocated resources'
#   cpu     3900m (99%)   ← what the SCHEDULER sees
kubectl top node ip-10-0-1-42
#   cpu     140m (3%)     ← what is actually happening

That gap is the most common cause of “my cluster is full but idle”. The fix is right-sizing requests, not adding nodes.

Units matter and are easy to get wrong:

cpu: 1        = 1 core = 1000m
cpu: 500m     = half a core   (m = millicores)
memory: 128Mi = 128 × 1024²   (mebibytes)
memory: 128M  = 128 × 1000²   (megabytes — 7% smaller!)

Always use Mi/Gi. Writing memory: 1G when you meant 1Gi costs you 74 MiB, which is exactly enough to cause an OOM you cannot explain.

Topic 2: QoS Classes and Who Dies First

kubectl get pods -o custom-columns='NAME:.metadata.name,QOS:.status.qosClass'

Guaranteed — requests == limits for both CPU and memory, on every container in the pod. Evicted last, and gets the strongest CPU guarantees.

Burstable — requests set, limits higher or absent. The common case.

BestEffort — nothing set at all. First to be evicted; scheduled anywhere because it reserves nothing.

The subtlety that catches people: a single container without resources makes the whole pod Burstable, even if the other five are perfectly specified. A sidecar with no resources silently downgrades your Guaranteed pod.

Eviction order under node memory pressure:

1. BestEffort pods first.
2. Burstable pods exceeding their requests, ordered by how far over they are.
3. Guaranteed pods last — and only if the node is still under pressure.

Within a tier, higher priorityClass is evicted later. This is why production workloads should be Guaranteed or well-specified Burstable, and why leaving resources unset is not a neutral choice — it is opting into being killed first.

Topic 3: Memory Limits Kill; CPU Limits Throttle

This asymmetry is the most important thing in the lesson.

Memory is incompressible:

Exceed limits.memory and the kernel’s OOM killer terminates the container immediately. There is no graceful degradation, because memory cannot be reclaimed from a process that is using it.

kubectl get pod api-xyz -o jsonpath='{.status.containerStatuses[0].lastState.terminated}' | jq
# { "exitCode": 137, "reason": "OOMKilled", "startedAt": "...", "finishedAt": "..." }

Exit 137 = 128 + 9 (SIGKILL). reason: OOMKilled is definitive.

Note there are two different OOM kills, and telling them apart matters:

• Container exceeded its own limit → only that container dies. reason: OOMKilled on the container.
• The node ran out of memory → the kubelet evicts whole pods by QoS class before the kernel has to act. Pod status becomes Failed with reason: Evicted.

kubectl get events --field-selector reason=Evicted -A
kubectl describe node <n> | grep -i pressure

Adding node memory fixes the second; it does nothing for the first.

CPU is compressible — and this is where the trap is:

Exceed limits.cpu and nothing is killed. The kernel’s CFS bandwidth control simply stops scheduling the container until the next 100ms period.

limits.cpu: 100m  →  10ms of CPU per 100ms period

A request needing 30ms of CPU now takes three periods — 200ms of wall-clock time added, purely from waiting. Your p99 latency triples. Memory is fine. CPU utilisation looks low (you are using exactly your limit). Nothing restarts. No event is emitted.

Proving it:

kubectl exec -it api-xyz -- cat /sys/fs/cgroup/cpu.stat
# nr_periods 84213
# nr_throttled 39187        ← throttled in 46% of periods
# throttled_usec 892341000  ← ~892 seconds of pure waiting

(cgroup v1 path: /sys/fs/cgroup/cpu/cpu.cfs_throttled_us.)

The Prometheus metric is container_cpu_cfs_throttled_periods_total / container_cpu_cfs_periods_total. Anything sustained above a few percent on a latency-sensitive service is worth investigating; above 25% it is almost certainly your latency problem.

The multi-threaded amplifier:

A JVM or Go runtime seeing 8 cores will start 8 worker threads. With limits.cpu: 1, those 8 threads share 100ms of quota per period and can burn it in 12.5ms of wall time, then stall for 87.5ms. The application is more throttled than a single-threaded one at the same limit.

Fixes: set GOMAXPROCS from the limit (automaxprocs), size JVM thread pools explicitly, and give latency-sensitive services generous CPU limits — or none.

Should you set CPU limits at all?

A genuine, live debate:

Against: throttling causes latency that no metric obviously explains. Requests already guarantee a share; without a limit a pod can use idle capacity for free. Many large operators set CPU requests and omit CPU limits entirely.

For: limits make behaviour predictable and stop one workload starving others; they are required for Guaranteed QoS; multi-tenant clusters usually need them.

A reasonable position: always set memory requests and limits equal (memory is incompressible; predictability beats bursting). For CPU, always set requests; set limits only where you need Guaranteed QoS or hard multi-tenant isolation, and if you do, set them generously and watch the throttle counters.

Topic 4: LimitRange and ResourceQuota

LimitRange — per-namespace defaults and bounds, applied at admission:

apiVersion: v1
kind: LimitRange
metadata:
  name: defaults
  namespace: payments
spec:
  limits:
    - type: Container
      default:              { cpu: 500m, memory: 512Mi }   # limit if unset
      defaultRequest:       { cpu: 100m, memory: 128Mi }   # request if unset
      max:                  { cpu: "4",  memory: 8Gi }
      min:                  { cpu: 10m,  memory: 32Mi }
      maxLimitRequestRatio: { cpu: "10" }                  # limit ≤ 10× request

This eliminates BestEffort pods in the namespace by construction — the defaults are injected into anything that omits resources.

ResourceQuota — a namespace-wide cap:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: payments-quota
  namespace: payments
spec:
  hard:
    requests.cpu: "20"
    requests.memory: 40Gi
    limits.cpu: "40"
    limits.memory: 80Gi
    persistentvolumeclaims: "10"
    count/deployments.apps: "20"
    pods: "100"

kubectl describe quota -n payments

The interaction people trip over: once a ResourceQuota specifies requests.cpu or limits.cpu, every pod in that namespace must set them, or creation is rejected. Combine a quota with a LimitRange so existing manifests keep working, otherwise you break every deployment in the namespace the moment the quota lands.

Topic 5: Choosing the Numbers

Guessing produces either waste or throttling. Measure.

kubectl top pods --containers -n payments

# p95 CPU over a week → a defensible request
quantile_over_time(0.95, rate(container_cpu_usage_seconds_total{pod=~"api-.*"}[5m])[7d:5m])

# max memory over a week → the limit (memory does not burst safely)
max_over_time(container_memory_working_set_bytes{pod=~"api-.*"}[7d])

# throttling — the number that tells you the limit is too low
rate(container_cpu_cfs_throttled_periods_total[5m])
  / rate(container_cpu_cfs_periods_total[5m])

Rules of thumb worth starting from:

• CPU request ≈ p95 usage. CPU limit ≈ 2–4× request, or omit.
• Memory request ≈ p95 working set + headroom. Memory limit = request (Guaranteed, predictable).
• Use container_memory_working_set_bytes, not container_memory_usage_bytes — the latter includes reclaimable page cache and will lead you to over-provision substantially.
• Re-measure after real traffic. Initial numbers are always wrong.

VPA in recommender mode (updateMode: "Off") will do this analysis continuously and emit suggestions without changing anything — the safest way to right-size a fleet.

Topic 6: Diagnosing “The Pod Is Slow”

# 1. Is it throttled? (the invisible one)
kubectl exec api-xyz -- cat /sys/fs/cgroup/cpu.stat | grep -E 'nr_throttled|throttled_usec'

# 2. Was it OOMKilled, and by which mechanism?
kubectl get pod api-xyz -o jsonpath='{.status.containerStatuses[0].lastState.terminated.reason}{"\n"}'
kubectl get events --field-selector reason=Evicted -A

# 3. Actual vs requested
kubectl top pod api-xyz --containers
kubectl get pod api-xyz -o jsonpath='{.spec.containers[*].resources}' | jq

# 4. Is the NODE the problem?
kubectl describe node $(kubectl get pod api-xyz -o jsonpath='{.spec.nodeName}') | grep -A8 'Allocated resources'

Try it yourself: Run a busy-loop container with limits.cpu: 100m and watch nr_throttled climb in cpu.stat while kubectl top shows it pinned at exactly 100m. That pairing — usage exactly at the limit, throttle counter rising — is the signature to recognise.

Common mistake: Responding to a latency problem by adding replicas. If each pod is CPU-throttled, more pods means more throttled pods; per-request latency is unchanged because the bottleneck is the per-container quota, not total capacity. Check nr_throttled before scaling.