Scheduling: Affinity, Taints & Topology Spread

The scheduler answers one question per pod: which node? Understanding how it answers turns Pending from a mystery into a readable explanation the cluster hands you for free.

Topic 1: Filter, Then Score

The scheduler runs two phases:

1. Filtering (predicates) — eliminate every node that cannot run this pod:

• Insufficient allocatable CPU/memory for the pod’s requests
• Node has a taint the pod does not tolerate
• nodeSelector / required nodeAffinity does not match
• Port conflict (hostPort already in use)
• Volume cannot be attached, or its zone does not match
• Node is unschedulable (cordoned) or not Ready

2. Scoring (priorities) — rank the survivors:

• LeastAllocated — prefer emptier nodes (the default; spreads load)
• BalancedAllocation — prefer nodes where CPU and memory usage stay proportionate
• ImageLocality — prefer nodes that already have the image
• Affinity and topology-spread preferences
• Taint toleration preferences

Highest score wins; ties are broken randomly. Then the scheduler binds the pod by writing spec.nodeName, and the kubelet on that node takes over.

If filtering leaves zero nodes, the pod stays Pending and the scheduler tells you exactly why:

kubectl describe pod api-xyz | tail -10
# 0/9 nodes are available: 3 Insufficient cpu,
#                          2 node(s) had untolerated taint {dedicated: gpu},
#                          2 node(s) didn't match Pod's node affinity/selector,
#                          2 node(s) had volume node affinity conflict.

That message is a complete audit of all nine nodes. Read it before doing anything else — it usually is the diagnosis.

Topic 2: nodeSelector and Node Affinity

# Simplest form — exact label match, all-or-nothing
spec:
  nodeSelector:
    disktype: ssd

Node affinity is the expressive version:

spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:      # a HARD requirement
        nodeSelectorTerms:
          - matchExpressions:
              - key: topology.kubernetes.io/zone
                operator: In
                values: [eu-west-1a, eu-west-1b]
              - key: node.kubernetes.io/instance-type
                operator: NotIn
                values: [t3.micro]
      preferredDuringSchedulingIgnoredDuringExecution:      # a SOFT preference
        - weight: 100
          preference:
            matchExpressions:
              - key: disktype
                operator: In
                values: [ssd]

The field names are long but they parse:

That last part is not a detail. Relabel a node so a running pod no longer matches its own required affinity and nothing happens — the pod keeps running. A RequiredDuringExecution variant has been proposed for years and does not exist.

Operators: In, NotIn, Exists, DoesNotExist, Gt, Lt.

Useful built-in node labels:

kubernetes.io/hostname
topology.kubernetes.io/zone
topology.kubernetes.io/region
node.kubernetes.io/instance-type
kubernetes.io/arch          amd64 | arm64
kubernetes.io/os            linux | windows

Topic 3: Pod Affinity and Anti-Affinity

These schedule pods relative to other pods, not to nodes.

affinity:
  podAntiAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchLabels: { app: api }
        topologyKey: kubernetes.io/hostname     # ← "one per NODE"
  podAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        podAffinityTerm:
          labelSelector:
            matchLabels: { app: redis }
          topologyKey: topology.kubernetes.io/zone   # ← "same ZONE as redis"

topologyKey is the field that decides what “together” means. The scheduler groups nodes by the value of that label and applies the rule per group:

The trap: anti-affinity on hostname guarantees one replica per node — and says nothing about zones. Three replicas can sit on three different nodes that are all in eu-west-1a. When that zone fails you lose everything, having “correctly” configured anti-affinity. For zone-level resilience the topologyKey must be zone.

The cost: required pod anti-affinity is expensive to evaluate — the scheduler must check every candidate node against every existing pod matching the selector. On large clusters this measurably slows scheduling, which is one reason topology spread constraints were introduced.

The other trap: required anti-affinity on hostname with replicas greater than your node count means the surplus pods can never schedule. A 5-replica Deployment on a 3-node cluster leaves 2 pods Pending forever. Use preferred unless you genuinely mean “refuse to run rather than co-locate”.

Topic 4: Topology Spread Constraints — the Better Tool

spec:
  topologySpreadConstraints:
    - maxSkew: 1
      topologyKey: topology.kubernetes.io/zone
      whenUnsatisfiable: DoNotSchedule
      labelSelector:
        matchLabels: { app: api }
      matchLabelKeys: [pod-template-hash]      # 1.27+; ignore other rollouts' pods
    - maxSkew: 1
      topologyKey: kubernetes.io/hostname
      whenUnsatisfiable: ScheduleAnyway
      labelSelector:
        matchLabels: { app: api }

maxSkew is the maximum allowed difference between the most and least populated topology domain. With maxSkew: 1 across three zones and six replicas you get 2/2/2; with four replicas you get 2/1/1.

Why this beats anti-affinity: anti-affinity is binary (“never together”); spread constraints express balance, they compose across multiple topology levels, and they are cheaper for the scheduler to evaluate.

matchLabelKeys: [pod-template-hash] is worth knowing: without it, during a rolling update the constraint counts old pods too, so the new pods are spread against a population that is about to disappear — producing strange placement. Adding the hash scopes the constraint to the current ReplicaSet.

There are also cluster-wide defaults in the scheduler config, so you may see spreading behaviour you did not ask for.

Topic 5: Taints and Tolerations — the Inverse

Affinity is the pod choosing nodes. Taints are the node rejecting pods.

kubectl taint nodes gpu-node-1 dedicated=gpu:NoSchedule
kubectl taint nodes gpu-node-1 dedicated=gpu:NoSchedule-      # trailing dash removes
kubectl get nodes -o custom-columns='NAME:.metadata.name,TAINTS:.spec.taints'

tolerations:
  - key: dedicated
    operator: Equal
    value: gpu
    effect: NoSchedule
  - operator: Exists                # tolerate EVERYTHING — DaemonSets only
  - key: node.kubernetes.io/not-ready
    operator: Exists
    effect: NoExecute
    tolerationSeconds: 300          # tolerate for 5 min, then evict

A toleration is permission, not preference. Tolerating a taint does not attract a pod to that node — it merely removes the barrier. To require the GPU node you need a toleration and a nodeSelector/affinity. Tolerating without selecting means your pod may land on the expensive GPU node or anywhere else, at random.

Built-in taints you will meet:

node-role.kubernetes.io/control-plane:NoSchedule    keeps workloads off the control plane
node.kubernetes.io/not-ready:NoExecute              added when a node goes NotReady
node.kubernetes.io/unreachable:NoExecute            added on network partition
node.kubernetes.io/memory-pressure:NoSchedule
node.kubernetes.io/disk-pressure:NoSchedule
node.kubernetes.io/unschedulable:NoSchedule         added by kubectl cordon

Every pod gets an automatic 300-second toleration for not-ready and unreachable. That is why a pod does not move for five minutes after its node dies — the delay is deliberate, to avoid mass rescheduling on a transient network blip. For latency-critical workloads you can lower tolerationSeconds, at the cost of more churn on flaps.

Topic 6: Priority, Preemption and Debugging

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: critical
value: 1000000
globalDefault: false
preemptionPolicy: PreemptLowerPriority
description: "Payment path. Preempts everything else."

spec:
  priorityClassName: critical

When a high-priority pod cannot schedule, the scheduler looks for lower-priority pods it could evict to make room, and preempts them. The victims are deleted with their grace period and rescheduled elsewhere if they can be.

Two consequences worth planning for: preemption ignores PodDisruptionBudgets (it tries to respect them, but will violate them if necessary), and a preempted pod may not be reschedulable at all, so it just disappears from where it was. Reserve high priority for genuinely critical paths, and set preemptionPolicy: Never for high-priority-but-not-urgent work that should queue rather than evict.

Built-in classes system-cluster-critical and system-node-critical exist for control-plane components.

The debugging ladder:

# 1. The scheduler's own explanation — usually sufficient
kubectl describe pod api-xyz | tail -15

# 2. What is on each node, and what is free
kubectl get pods -o wide --field-selector spec.nodeName=ip-10-0-1-42
kubectl describe node ip-10-0-1-42 | grep -A8 'Allocated resources'

# 3. Taints, labels, and readiness across the fleet
kubectl get nodes -o custom-columns='NAME:.metadata.name,TAINTS:.spec.taints,ZONE:.metadata.labels.topology\.kubernetes\.io/zone'
kubectl get nodes --show-labels

# 4. Zone distribution of what IS scheduled
kubectl get pods -l app=api -o custom-columns='NAME:.metadata.name,NODE:.spec.nodeName' 

# 5. Scheduler logs, if the message is genuinely unclear
kubectl logs -n kube-system -l component=kube-scheduler --tail=100

Try it yourself: Create a 4-replica Deployment with required pod anti-affinity on hostname in a 3-node cluster. Confirm the fourth pod is Pending forever, and read the message that says so. Then switch to preferred and watch it schedule.

Common mistake: Using pod anti-affinity on kubernetes.io/hostname and believing you have zone redundancy. You have node redundancy. If all your nodes are in one zone — which is common in a small cluster or a single-AZ node group — a zone failure takes every replica. Spread on topology.kubernetes.io/zone for that, and verify with kubectl get pods -o wide against the actual node zone labels.