Project 1: Ship a Production-Ready Workload

Everything in the first fourteen lessons, applied to one workload. The deliverable is not the YAML — it is the six broken states you deliberately create and survive at the end.

The Requirements

Step 1: Namespace, Identity and Policy Baseline

apiVersion: v1
kind: Namespace
metadata:
  name: shop
  labels:
    team: payments
    cost-center: "4471"
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/enforce-version: v1.36
    pod-security.kubernetes.io/warn: restricted
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: api
  namespace: shop
automountServiceAccountToken: false
---
apiVersion: v1
kind: LimitRange
metadata:
  name: defaults
  namespace: shop
spec:
  limits:
    - type: Container
      defaultRequest: { cpu: 100m, memory: 128Mi }
      default:        { cpu: 500m, memory: 256Mi }
      max:            { cpu: "4",  memory: 8Gi }

The LimitRange goes in before any quota, for the reason in lesson 26: a quota without defaults rejects every pod that omits resources.

Step 2: Config, With a Rollout Trigger

apiVersion: v1
kind: ConfigMap
metadata:
  name: api-config
  namespace: shop
data:
  LOG_LEVEL: "info"
  UPSTREAM_TIMEOUT: "5s"

Mounted config does not restart pods on its own. Either hash it into the pod template (Helm/Kustomize both do this), or accept that you must kubectl rollout restart after a config change. Kustomize’s configMapGenerator handles it automatically:

configMapGenerator:
  - name: api-config
    literals: [LOG_LEVEL=info, UPSTREAM_TIMEOUT=5s]

Step 3: The Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
  namespace: shop
  annotations:
    kubernetes.io/change-cause: "initial deploy 1.4.2"
spec:
  replicas: 3
  revisionHistoryLimit: 5
  minReadySeconds: 15
  progressDeadlineSeconds: 300
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  selector:
    matchLabels:
      app.kubernetes.io/name: api
      app.kubernetes.io/instance: api-prod
  template:
    metadata:
      labels:
        app.kubernetes.io/name: api
        app.kubernetes.io/instance: api-prod
        app.kubernetes.io/version: "1.4.2"
        team: payments
      annotations:
        runbook: "https://wiki.example.com/runbooks/api"
    spec:
      serviceAccountName: api
      automountServiceAccountToken: false
      terminationGracePeriodSeconds: 45

      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone      # ZONE, not hostname
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels: { app.kubernetes.io/name: api }
          matchLabelKeys: [pod-template-hash]
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels: { app.kubernetes.io/name: api }

      securityContext:
        runAsNonRoot: true
        runAsUser: 10001
        runAsGroup: 10001
        fsGroup: 10001
        seccompProfile: { type: RuntimeDefault }

      containers:
        - name: api
          image: registry.example.com/api:1.4.2
          imagePullPolicy: IfNotPresent
          ports:
            - { name: http, containerPort: 8080 }
            - { name: metrics, containerPort: 9090 }

          envFrom:
            - configMapRef: { name: api-config }
          env:
            - name: POD_NAME
              valueFrom: { fieldRef: { fieldPath: metadata.name } }
            - name: POD_NAMESPACE
              valueFrom: { fieldRef: { fieldPath: metadata.namespace } }

          resources:
            requests: { cpu: 200m, memory: 256Mi }
            limits:   { memory: 256Mi }          # memory == request; NO cpu limit

          startupProbe:
            httpGet: { path: /healthz, port: http }
            periodSeconds: 5
            failureThreshold: 30                  # 150s to boot
          readinessProbe:
            httpGet: { path: /readyz, port: http }
            periodSeconds: 5
            timeoutSeconds: 3
            failureThreshold: 2
          # NO livenessProbe — see lesson 9

          lifecycle:
            preStop:
              exec: { command: ["sleep", "10"] }

          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities: { drop: ["ALL"] }

          volumeMounts:
            - { name: tmp, mountPath: /tmp }

      volumes:
        - name: tmp
          emptyDir: {}

Five decisions worth defending:

maxUnavailable: 0 — capacity never dips during a rollout. Costs one extra pod’s headroom.

Memory request == limit, no CPU limit. Guaranteed-ish memory behaviour without CFS throttling on latency. Watch nr_throttled to confirm you do not need one.

No liveness probe. Lesson 9’s argument: it is a standing instruction to kill your container, and most apps do not need one. Add it only for a specific known hang.

preStop: sleep 10 with a 45s grace period — closes the endpoint-removal race that produces connection-refused errors on every deploy.

Zone spread with DoNotSchedule — refusing to schedule is better than silently losing zone redundancy.

Step 4: Service, PDB and Autoscaling

apiVersion: v1
kind: Service
metadata:
  name: api
  namespace: shop
spec:
  selector:
    app.kubernetes.io/name: api
    app.kubernetes.io/instance: api-prod      # NOT version — see lesson 5
  ports:
    - { name: http, port: 80, targetPort: http }
    - { name: metrics, port: 9090, targetPort: metrics }
  trafficDistribution: PreferClose
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: api
  namespace: shop
spec:
  maxUnavailable: 1                            # scales with replicas
  unhealthyPodEvictionPolicy: AlwaysAllow
  selector:
    matchLabels:
      app.kubernetes.io/name: api
      app.kubernetes.io/instance: api-prod
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: api
  namespace: shop
spec:
  scaleTargetRef: { apiVersion: apps/v1, kind: Deployment, name: api }
  minReplicas: 3
  maxReplicas: 20
  metrics:
    - type: Resource
      resource:
        name: cpu
        target: { type: Utilization, averageUtilization: 65 }
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0
      policies: [{ type: Percent, value: 100, periodSeconds: 30 }]
    scaleDown:
      stabilizationWindowSeconds: 300
      policies: [{ type: Percent, value: 25, periodSeconds: 60 }]

maxUnavailable: 1 not minAvailable: 3. The latter equals your replica count and permanently blocks drains — lesson 23’s trap.

Remove replicas from the Deployment once the HPA owns it, or every apply fights the autoscaler.

Step 5: Network Policy

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: default-deny, namespace: shop }
spec:
  podSelector: {}
  policyTypes: [Ingress, Egress]
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: api-allow, namespace: shop }
spec:
  podSelector:
    matchLabels: { app.kubernetes.io/name: api }
  policyTypes: [Ingress, Egress]
  ingress:
    - from:
        - namespaceSelector:
            matchLabels: { kubernetes.io/metadata.name: ingress-nginx }
      ports: [{ protocol: TCP, port: 8080 }]
    - from:
        - namespaceSelector:
            matchLabels: { kubernetes.io/metadata.name: monitoring }
      ports: [{ protocol: TCP, port: 9090 }]
  egress:
    # DNS — WITHOUT THIS, EVERYTHING BREAKS
    - to:
        - namespaceSelector:
            matchLabels: { kubernetes.io/metadata.name: kube-system }
          podSelector:
            matchLabels: { k8s-app: kube-dns }
      ports: [{ protocol: UDP, port: 53 }, { protocol: TCP, port: 53 }]
    - to:
        - podSelector: { matchLabels: { app.kubernetes.io/name: postgres } }
      ports: [{ protocol: TCP, port: 5432 }]
    # Outbound internet, but NOT the metadata endpoint
    - to:
        - ipBlock:
            cidr: 0.0.0.0/0
            except: [169.254.169.254/32, 10.0.0.0/8]
      ports: [{ protocol: TCP, port: 443 }]

Step 6: The Verification — This Is the Project

Manifests are the easy half. Run all six with a load generator producing steady traffic and counting failures.

# Baseline: steady load, count non-200s
kubectl run -it --rm load --image=williamyeh/hey --restart=Never -- \
  -z 10m -c 20 http://api.shop.svc.cluster.local/

1. Node drain — expect ZERO failures

NODE=$(kubectl get pods -n shop -l app.kubernetes.io/name=api -o jsonpath='{.items[0].spec.nodeName}')
kubectl drain "$NODE" --ignore-daemonsets --delete-emptydir-data --timeout=5m
kubectl uncordon "$NODE"

Failures here mean preStop, the grace period, or readiness is wrong.

2. Bad deploy — expect it to STALL, not to break

kubectl set image deploy/api api=registry.example.com/api:does-not-exist -n shop
kubectl rollout status deploy/api -n shop --timeout=2m     # should FAIL
kubectl get pods -n shop                                    # 3 old Running, 1 new ImagePullBackOff
kubectl rollout undo deploy/api -n shop

With maxUnavailable: 0 the old pods are never removed. Traffic is unaffected throughout.

3. Bad readiness — expect the rollout to halt

kubectl set image deploy/api api=registry.example.com/api:1.4.3-broken -n shop
# new pod starts, fails /readyz, never becomes available, rollout stops at 1
kubectl rollout undo deploy/api -n shop

4. Dependency failure — expect no restart storm

kubectl scale deploy/postgres --replicas=0 -n shop
kubectl get pods -n shop -w
# Expect: READY 0/1, RESTARTS stays at 0.
# If restarts climb, you have a liveness probe checking a dependency — remove it.
kubectl scale deploy/postgres --replicas=1 -n shop

5. Hostile pod — expect a timeout

kubectl run -it --rm attacker --image=nicolaka/netshoot -n default --restart=Never -- \
  curl -m5 http://api.shop.svc.cluster.local/
# must TIME OUT (not "refused") — that is NetworkPolicy working

6. Security posture — expect rejections

kubectl run bad --image=nginx -n shop --restart=Never       # rejected by PSA restricted
kubectl exec -n shop deploy/api -- id                       # uid=10001, not 0
kubectl exec -n shop deploy/api -- touch /test              # read-only filesystem
kubectl exec -n shop deploy/api -- ls /var/run/secrets/kubernetes.io  # should not exist

What Good Looks Like

kubectl get pdb -n shop
# ALLOWED DISRUPTIONS: 1      ← not 0

kubectl get pods -n shop -o custom-columns='NAME:.metadata.name,ZONE:.spec.nodeName,QOS:.status.qosClass'
# spread across at least 2 zones

kubectl get hpa -n shop
# TARGETS: 40%/65%            ← not <unknown>

kubectl get endpointslices -n shop -l kubernetes.io/service-name=api
# 3 addresses

Extensions worth building: add a ServiceMonitor and the four golden-signal alerts from lesson 22; move Secrets to External Secrets Operator; express the whole thing as a Kustomize base with dev/prod overlays; add a Gateway API HTTPRoute with a 90/10 canary split.

The lesson to take away: every item in the manifest exists because of a specific failure, and you have now caused all six. A workload nobody has deliberately broken is a workload whose resilience is a hypothesis.