Storage: Volumes, PVs, PVCs & CSI

Storage is where Kubernetes leaks the most detail about the infrastructure underneath. The abstraction is good, but the failure modes are all about physical reality — zones, attachment limits, and disks that can only be mounted in one place.

Topic 1: The Layers

• Volume — what the pod mounts. Defined in the pod spec.
• PersistentVolumeClaim (PVC) — a request for storage. Namespaced. What developers write.
• PersistentVolume (PV) — the actual provisioned volume. Cluster-scoped.
• StorageClass — the recipe: which provisioner, what parameters, what reclaim policy.
• CSI driver — the code that talks to EBS, Ceph, NFS or a local disk.

Topic 2: Ephemeral Volumes First

Not everything needs persistence, and reaching for a PVC when you do not need one adds a whole category of problems.

volumes:
  # Scratch space. Lives and dies with the POD (survives container restarts).
  - name: cache
    emptyDir:
      sizeLimit: 1Gi

  # Same, but in RAM — counts against the pod's memory limit
  - name: fast-cache
    emptyDir:
      medium: Memory
      sizeLimit: 256Mi

  # Config and secrets
  - name: config
    configMap: { name: api-config }
  - name: creds
    secret: { secretName: api-secrets, defaultMode: 0400 }

  # Combine several sources into one directory tree
  - name: combined
    projected:
      sources:
        - configMap: { name: api-config }
        - secret: { name: api-secrets }
        - serviceAccountToken:
            path: token
            expirationSeconds: 3600
            audience: vault

emptyDir survives container restarts but not pod rescheduling. That distinction matters: a crash-looping container keeps its scratch data; a pod moved to another node starts empty.

medium: Memory counts against your memory limit. A tmpfs emptyDir filling up is indistinguishable from an application memory leak in every dashboard, and it will get the container OOMKilled.

Avoid hostPath in production. It mounts a path from the node, which ties the pod to that node’s contents, breaks portability, and is a straightforward container escape if writable. Legitimate uses are node-level agents (a log collector reading /var/log) in a DaemonSet — and those should be readOnly: true.

Topic 3: Static vs Dynamic Provisioning

Static — an admin creates the PV; a PVC binds to a matching one. Rare, and mostly for pre-existing NFS exports.

Dynamic — the PVC names a StorageClass and a volume is created on demand. The normal case.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gp3
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
  iops: "3000"
  throughput: "125"
  encrypted: "true"
reclaimPolicy: Delete            # or Retain
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data
spec:
  accessModes: ["ReadWriteOnce"]
  storageClassName: gp3
  resources:
    requests: { storage: 100Gi }

volumeBindingMode — the most important field here:

Immediate in a multi-zone cluster is a bug waiting to happen. The volume is created in some zone — say eu-west-1a — before anyone knows where the pod will run. The scheduler then discovers the pod can only run in eu-west-1a, and if that zone has no capacity, the pod is Pending forever with volume node affinity conflict. The volume and the pod each hold the other hostage.

WaitForFirstConsumer inverts the order: schedule first, then provision in the right place. Use it for any zonal storage, which is every cloud block device.

kubectl get storageclass
kubectl get sc gp3 -o jsonpath='{.volumeBindingMode}{"\n"}'

reclaimPolicy:

• Delete — deleting the PVC deletes the PV and the underlying disk. Good for scratch; catastrophic for a database.
• Retain — the PV survives, Released, and must be manually cleaned up. Data is safe; you pay for it until someone acts.

For anything you would be upset to lose, use Retain and accept the cleanup burden.

Topic 4: Access Modes Are Not What They Look Like

RWO is per-node, not per-pod. Two pods on the same node can both mount an RWO volume. Two pods on different nodes cannot. This is the single most misunderstood line in Kubernetes storage, and it explains why a Deployment with an RWO volume works fine at 1 replica, works fine at 3 replicas if they land on one node, and then breaks the day the scheduler spreads them.

Cloud block storage is RWO. Full stop. EBS, GCE PD and Azure Disk cannot be attached to multiple nodes. If you need RWX you need a filesystem: EFS, Filestore, Azure Files, CephFS or an NFS server. They are slower and cost more, which is the trade.

ReadWriteOncePod is the mode you actually want for a database — it guarantees a single writer even if two pods land on the same node.

The access mode is not enforced by the storage. It is metadata the scheduler and attach/detach controller respect. Setting RWX on an EBS volume does not make it work; it makes the failure happen later and more confusingly.

Topic 5: The Failure Modes

PVC stuck Pending

kubectl get pvc
kubectl describe pvc data

Pod stuck Pending with a volume conflict

0/6 nodes are available: 3 node(s) had volume node affinity conflict,
                         3 Insufficient cpu.

The volume exists in a zone whose nodes are full or absent. Fix the binding mode for next time; for now, either free capacity in that zone or recreate the volume elsewhere.

Pod stuck ContainerCreating on a mount

kubectl describe pod api-0 | tail -20
# Warning FailedAttachVolume  Multi-Attach error for volume "pvc-abc123"
#         Volume is already exclusively attached to one node and can't be attached to another

This is the RWO rule biting. Common when a node dies and its pod is rescheduled while the old attachment is still registered — the volume must be detached first, which can take minutes, and a genuinely stuck one needs the volumeattachment cleared.

kubectl get volumeattachment | grep pvc-abc123

Volume full

kubectl exec api-0 -- df -h /data

Nothing in Kubernetes alerts on this by default. If allowVolumeExpansion: true, you can grow it in place:

kubectl patch pvc data -p '{"spec":{"resources":{"requests":{"storage":"200Gi"}}}}'

The PV expands, then the filesystem is resized — online for most CSI drivers, though some still need a pod restart. You cannot shrink a PVC. Ever.

Topic 6: CSI, Snapshots and Practical Advice

The Container Storage Interface is why storage vendors ship their own drivers instead of being merged into Kubernetes. In-tree cloud volume plugins have been removed in favour of CSI, so kubernetes.io/aws-ebs is gone and ebs.csi.aws.com is the provisioner you use.

kubectl get csidrivers
kubectl get csinodes -o custom-columns='NODE:.metadata.name,DRIVERS:.spec.drivers[*].name'

Attachment limits are a real scheduling constraint. Each node can only attach so many volumes — commonly 25–39 on AWS depending on instance type, and it counts against the same limit as the root volume. A node can be nowhere near its CPU or memory limits and still refuse a pod because it cannot attach another disk.

kubectl get csinode <node> -o jsonpath='{.spec.drivers[0].allocatable.count}{"\n"}'

Snapshots:

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: data-backup-20260807
spec:
  volumeSnapshotClassName: ebs-snapshot
  source:
    persistentVolumeClaimName: data

Restore by creating a PVC with dataSource pointing at the snapshot. Note that a snapshot is crash-consistent, not application-consistent — for a database you want its own backup tooling, or at minimum a filesystem freeze around the snapshot.

Advice that saves incidents:

1. WaitForFirstConsumer on every zonal StorageClass.
2. Retain for anything you would miss. The default Delete means kubectl delete pvc destroys production data with no confirmation.
3. Alert on volume utilisation — kubelet_volume_stats_available_bytes. Nothing does this for you.
4. Do not run a database on Kubernetes because you can. Managed services remove attachment races, zone pinning and backup correctness from your problem list. If you do run one, use a mature operator.
5. Test your restore. A snapshot you have never restored from is a hypothesis, not a backup.

Try it yourself: Create a StorageClass with volumeBindingMode: Immediate in a multi-zone cluster, provision a PVC, and then try to schedule its pod onto a node in a different zone with a nodeSelector. Read the resulting volume node affinity conflict message.

Common mistake: Setting replicas: 3 on a Deployment whose pods mount the same RWO PVC. One pod runs; the other two sit in ContainerCreating with multi-attach errors — unless all three happen to land on one node, in which case it works until it does not. Per-pod storage means StatefulSet with volumeClaimTemplates.