Multi-Tenancy, Quotas & Cost Control

Multiple teams on one cluster is the normal case, and it is a spectrum of isolation rather than a binary. Getting it right means knowing precisely what each mechanism does — and what none of them do.

Topic 1: Namespaces Are a Name Scope, Not a Boundary

Repeating the point from lesson 5 because everything here depends on it. A namespace gives you:

• Name uniqueness and a DNS segment
• A scope for RBAC, ResourceQuota and LimitRange
• A target for NetworkPolicy

It does not give you:

• Network isolation (every pod can reach every pod by default)
• Node isolation (tenants share machines and their kernels)
• Kernel isolation (a container escape reaches every workload on that node)
• Control-plane fairness (one tenant hammering the API affects everyone)

Soft multi-tenancy — trusted teams in one organisation — is what namespaces are designed for, and with quotas, RBAC and NetworkPolicy it works well.

Hard multi-tenancy — untrusted or hostile tenants — needs more: separate clusters, or at minimum node isolation plus a sandboxed runtime (gVisor, Kata Containers). If tenants could be adversarial, separate clusters are the honest answer, and the cost of that is lower than the cost of being wrong.

Topic 2: The Layered Setup

apiVersion: v1
kind: Namespace
metadata:
  name: team-payments
  labels:
    team: payments
    cost-center: "4471"
    pod-security.kubernetes.io/enforce: baseline
    pod-security.kubernetes.io/warn: restricted
---
apiVersion: v1
kind: ResourceQuota
metadata:
  name: compute
  namespace: team-payments
spec:
  hard:
    requests.cpu: "40"
    requests.memory: 80Gi
    limits.cpu: "80"
    limits.memory: 160Gi
    persistentvolumeclaims: "20"
    requests.storage: 2Ti
    count/deployments.apps: "50"
    count/services: "30"
    services.loadbalancers: "2"          # ← each one is a cloud bill
    pods: "200"
---
apiVersion: v1
kind: LimitRange
metadata:
  name: defaults
  namespace: team-payments
spec:
  limits:
    - type: Container
      default:        { cpu: 500m, memory: 512Mi }
      defaultRequest: { cpu: 100m, memory: 128Mi }
      max:            { cpu: "8",  memory: 16Gi }
      min:            { cpu: 10m,  memory: 32Mi }
    - type: PersistentVolumeClaim
      max: { storage: 500Gi }

The interaction that breaks deployments:

Once a ResourceQuota constrains requests.cpu or limits.memory, every pod in that namespace must specify them. A pod without them is rejected outright:

Error from server (Forbidden): pods "api-xyz" is forbidden:
  failed quota: compute: must specify limits.cpu for: api; requests.cpu for: api

If you apply a quota to a namespace with existing workloads that omit resources, every subsequent rollout fails. Existing pods keep running, so nothing breaks immediately — it breaks on the next deploy, in an unrelated change, and the error names the quota rather than the missing field.

Always apply the LimitRange first, or in the same change. The LimitRange injects defaults at admission, so manifests without resources still work and the quota still counts them.

kubectl describe quota -n team-payments
# Resource         Used   Hard
# requests.cpu     12500m 40
# requests.memory  24Gi   80Gi
# pods             47     200

Watch services.loadbalancers in particular — it is the only quota field that maps directly to a recurring cloud invoice.

Topic 3: RBAC and Network Boundaries

# Bind the built-in `edit` ClusterRole into ONE namespace
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: payments-team
  namespace: team-payments
subjects:
  - kind: Group
    name: payments-engineers
    apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: edit                # can manage workloads; CANNOT touch RBAC
  apiGroup: rbac.authorization.k8s.io

edit rather than admin is the right default: admin includes managing Roles and RoleBindings, which lets a tenant grant themselves more. Grant admin only where a team genuinely self-manages access.

Deny cross-namespace traffic by default, from the NetworkPolicy lesson:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-ingress
  namespace: team-payments
spec:
  podSelector: {}
  policyTypes: [Ingress]
---
# Then allow same-namespace traffic explicitly
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-same-namespace
  namespace: team-payments
spec:
  podSelector: {}
  policyTypes: [Ingress]
  ingress:
    - from:
        - podSelector: {}       # any pod in THIS namespace

Remember to allow DNS egress and monitoring ingress, or you will spend an afternoon on it.

Topic 4: Node Isolation

When tenants must not share kernels, separate them physically:

kubectl taint nodes payments-node-1 tenant=payments:NoSchedule
kubectl label nodes payments-node-1 tenant=payments

spec:
  tolerations:
    - { key: tenant, operator: Equal, value: payments, effect: NoSchedule }
  nodeSelector:
    tenant: payments

Both are required, and this is the point from the scheduling lesson worth repeating: a toleration is permission, not preference. Without the nodeSelector the pod may land anywhere; without the toleration it cannot land on the dedicated nodes.

Enforce it with policy rather than trusting manifests — Kyverno can mutate every pod in a namespace to add the right nodeSelector and toleration automatically, which removes the possibility of someone forgetting.

Priority classes prevent one tenant starving another:

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata: { name: tenant-standard }
value: 1000
preemptionPolicy: Never          # queue rather than evict others

preemptionPolicy: Never matters in multi-tenancy: without it, a tenant can set a high priority and evict another tenant’s workloads.

Topic 5: The API Server Is Shared Too

An overlooked shared resource. One tenant’s badly-written controller listing every pod in a tight loop degrades the API server for everyone.

API Priority and Fairness (GA since 1.29) partitions API server capacity:

kubectl get flowschemas
kubectl get prioritylevelconfigurations
kubectl get --raw /metrics | grep apiserver_flowcontrol_rejected_requests_total

Requests are classified into flow schemas and given a share of concurrency, so one noisy client is throttled rather than starving the others. It is on by default; knowing it exists means you can diagnose “why are my requests being throttled” (429 with Retry-After).

etcd is also shared: a tenant creating hundreds of thousands of objects grows the database toward its quota, which affects the whole cluster. count/ quotas are the control for that.

Topic 6: Cost Attribution and Control

You cannot control what you cannot attribute.

Label everything, from the start:

metadata:
  labels:
    team: payments
    cost-center: "4471"
    environment: production
    app.kubernetes.io/part-of: checkout

Retrofitting labels across a live cluster is significantly harder than deciding them on day one, and every cost tool depends on them.

# Requested CPU per namespace — what you are BILLED for, effectively
kubectl get pods -A -o json | jq -r '
  .items[] | .metadata.namespace as $ns |
  (.spec.containers[].resources.requests.cpu // "0") | "\($ns)"' \
  | sort | uniq -c | sort -rn

Kubecost or OpenCost (CNCF) map node cost onto pods by request share and produce per-namespace, per-label breakdowns. Worth installing before the finance conversation rather than after.

Where the money actually goes:

The first row dominates. A cluster where requests are set at twice real usage costs twice what it needs to, and that is the normal state of an unmeasured cluster.

# Orphaned PVCs — bound to nothing
kubectl get pvc -A -o json | jq -r '.items[] |
  select(.status.phase=="Bound") | "\(.metadata.namespace)/\(.metadata.name)"'
# cross-reference against pods that mount them

Topic 7: Choosing an Isolation Model

vCluster is worth knowing: it runs a real API server and controller manager per tenant inside a namespace of the host cluster, so tenants can install their own CRDs and cluster-scoped resources without affecting anyone else — while sharing the host’s nodes. It sits neatly between namespaces and full clusters.

Try it yourself: Apply a ResourceQuota with requests.cpu to a namespace containing a Deployment whose pods have no resources set. Confirm the existing pods keep running, then trigger a rollout and watch it fail. That delay between the change and the breakage is what makes this a production trap.

Common mistake: Assuming namespaces provide isolation because the name suggests it. Without NetworkPolicy, a pod in team-a can reach every Service in team-b and read its metrics endpoints. Without quotas, one namespace can consume the entire cluster. The namespace is where you attach isolation, not the isolation itself.