etcd Operations, Backup & Disaster Recovery

Every other component is replaceable. Lose etcd without a backup and you lose the cluster’s entire declared state — every Deployment, Secret, RBAC rule and CRD.

Topic 1: What etcd Is Doing

A distributed key-value store using the Raft consensus algorithm. Every object in Kubernetes is a key:

/registry/pods/payments/api-7d9f-x2k4
/registry/deployments/payments/api
/registry/secrets/payments/db-credentials

Two operational facts drive everything below.

Quorum requires a majority. Cluster sizes are odd for a reason:

Lose quorum and etcd goes read-only — the API server can still serve reads for a while, then everything fails. Nothing can be created, changed or deleted.

Every write is fsynced to a majority of members. etcd’s throughput is bounded by disk latency, not CPU. This is the single most important thing to know about running it: put etcd on fast local SSD/NVMe, never on shared or network storage, and never co-located with anything I/O-heavy.

histogram_quantile(0.99, rate(etcd_disk_wal_fsync_duration_seconds_bucket[5m]))   # want < 10ms
histogram_quantile(0.99, rate(etcd_disk_backend_commit_duration_seconds_bucket[5m]))
etcd_server_leader_changes_seen_total          # frequent changes = instability
etcd_server_has_leader                         # 0 = no quorum
etcd_mvcc_db_total_size_in_bytes               # against the quota

When fsync latency rises, the entire cluster feels broken. API calls slow, controllers lag, kubectl hangs, and every application-level metric looks fine. It is the most misdiagnosed cluster-wide slowdown there is — teams look at their applications for hours while the answer is one disk.

Topic 2: Taking Backups

export ETCDCTL_API=3
ETCD_EP=https://127.0.0.1:2379
CERTS="--cacert=/etc/kubernetes/pki/etcd/ca.crt \
       --cert=/etc/kubernetes/pki/etcd/server.crt \
       --key=/etc/kubernetes/pki/etcd/server.key"

etcdctl --endpoints=$ETCD_EP $CERTS snapshot save /backup/etcd-$(date +%Y%m%d-%H%M%S).db

# ALWAYS verify — a snapshot you have not checked is a hypothesis
etcdctl --write-out=table snapshot status /backup/etcd-20260807-143000.db
# +----------+----------+------------+------------+
# |   HASH   | REVISION | TOTAL KEYS | TOTAL SIZE |
# +----------+----------+------------+------------+
# | 8f4kd2m9 |  4471928 |      12847 |     148 MB |

snapshot save is a point-in-time copy taken through the API, safe to run on a live member.

Health and membership:

etcdctl --endpoints=$ETCD_EP $CERTS endpoint health --cluster
etcdctl --endpoints=$ETCD_EP $CERTS endpoint status --write-out=table --cluster
etcdctl --endpoints=$ETCD_EP $CERTS member list --write-out=table

Automate it, and test the restore:

#!/usr/bin/env bash
set -Eeuo pipefail
SNAP="/backup/etcd-$(date +%Y%m%d-%H%M%S).db"
etcdctl --endpoints=$ETCD_EP $CERTS snapshot save "$SNAP"
etcdctl snapshot status "$SNAP" >/dev/null || { echo "snapshot verification FAILED" >&2; exit 1; }
aws s3 cp "$SNAP" "s3://cluster-backups/prod/" --sse aws:kms
find /backup -name 'etcd-*.db' -mtime +7 -delete

Run it hourly via a CronJob or systemd timer, ship it off-cluster, and encrypt it — an etcd snapshot contains every Secret in the cluster in plaintext unless encryption at rest is enabled. Treat the backup with the same care as the Secrets themselves.

On managed clusters (EKS/GKE/AKS) you cannot reach etcd. The provider backs up the control plane, and etcdctl is not available to you. Your backup story there is Velero plus GitOps, covered below.

Topic 3: Restoring

A restore is disruptive and must be done deliberately.

# 1. STOP the API server on every control-plane node
sudo mv /etc/kubernetes/manifests/kube-apiserver.yaml /tmp/
# (static pod — moving the manifest stops it)

# 2. Restore into a NEW data directory — never over the existing one
sudo etcdctl snapshot restore /backup/etcd-20260807-143000.db \
  --data-dir=/var/lib/etcd-restored \
  --name=master-1 \
  --initial-cluster=master-1=https://10.0.1.10:2380 \
  --initial-advertise-peer-urls=https://10.0.1.10:2380

# 3. Point etcd at the restored directory
sudo mv /var/lib/etcd /var/lib/etcd-old
sudo mv /var/lib/etcd-restored /var/lib/etcd

# 4. Bring the API server back
sudo mv /tmp/kube-apiserver.yaml /etc/kubernetes/manifests/

# 5. Verify
kubectl get nodes
kubectl get pods -A

Things that go wrong:

• Restoring over a live data directory corrupts it. Always restore to a new path.
• Multi-member restore: every member must be restored from the same snapshot with matching --initial-cluster flags, then started together. Restoring one member and hoping it syncs does not work.
• State reverts to the snapshot time. Objects created after it are gone; objects deleted after it come back. Workloads created since the snapshot are unknown to the control plane while their containers may still be running on nodes — a genuinely confusing state that needs reconciling.

Topic 4: Database Size and Defragmentation

etcd has a default 2 GiB quota (--quota-backend-bytes). Exceed it and etcd enters a read-only alarm state: no writes, cluster-wide, until you clear it. This is a real production incident and it arrives without warning.

etcdctl --endpoints=$ETCD_EP $CERTS endpoint status --write-out=table   # DB SIZE
etcdctl --endpoints=$ETCD_EP $CERTS alarm list

Because etcd keeps a full revision history, the on-disk size grows even when the number of objects does not. Two operations manage it:

# 1. Compact — discard revisions older than the current one
REV=$(etcdctl --endpoints=$ETCD_EP $CERTS endpoint status --write-out=json | jq -r '.[0].Status.header.revision')
etcdctl --endpoints=$ETCD_EP $CERTS compact "$REV"

# 2. Defragment — actually release the freed space to the filesystem
etcdctl --endpoints=$ETCD_EP $CERTS defrag --cluster

# 3. Clear the alarm once size is back under quota
etcdctl --endpoints=$ETCD_EP $CERTS alarm disarm

Defrag blocks the member it runs on. Do it one member at a time, never all at once, or you lose quorum during the operation.

Kubernetes runs auto-compaction by default (--etcd-compaction-interval, 5 minutes), but defrag is not automatic — the space is freed logically and never returned to the filesystem without it.

What actually fills etcd:

• Completed Jobs and their pods with no ttlSecondsAfterFinished
• Events, if --event-ttl was raised
• Large ConfigMaps and Secrets
• High-churn CRDs from a badly-behaved operator
• Helm release history (--history-max)

# Which resource types dominate?
etcdctl --endpoints=$ETCD_EP $CERTS get / --prefix --keys-only \
  | awk -F/ '/registry/ {print $3}' | sort | uniq -c | sort -rn | head -20

Topic 5: Velero — Application-Level DR

etcd snapshots restore a whole cluster. They cannot restore one namespace, and they are useless for migrating between clusters. Velero covers that gap, and it is the answer on managed clusters where etcd is inaccessible.

velero backup create nightly --include-namespaces payments,shop --wait
velero backup create full --exclude-namespaces kube-system --snapshot-volumes
velero schedule create daily --schedule="0 2 * * *" --ttl 720h
velero restore create --from-backup nightly --include-namespaces payments
velero backup describe nightly --details

Velero backs up API objects and, optionally, volume contents (via CSI snapshots or file-level copy). It can restore into a different cluster, which makes it the practical tool for migration as well as recovery.

Run both where you can. And note the third leg: GitOps is a backup. If every manifest is in Git and Argo/Flux reconciles it, rebuilding a cluster is terraform apply plus pointing the controller at the repo. That covers everything except data in volumes and objects created outside Git.

Topic 6: A DR Plan You Have Actually Tested

Decide these numbers before an incident, not during one:

• RPO (recovery point objective) — how much data may you lose? Hourly snapshots means up to an hour.
• RTO (recovery time objective) — how long may recovery take? This is the number people never measure.

# Recovering a single failed member (no restore needed)
etcdctl member remove <member-id>
etcdctl member add master-2 --peer-urls=https://10.0.1.11:2380
# then start etcd on that node with --initial-cluster-state=existing

The rules that make this real:

1. Test the restore quarterly, on a real cluster. A snapshot you have never restored from is not a backup.
2. Store backups off-cluster and encrypted. A backup on the cluster you lost is not a backup.
3. Monitor backup success, not just the schedule. A CronJob that has been failing for three weeks looks identical to one that is working, unless you alert on it.
4. Write down the restore procedure with real commands, and make sure someone other than its author has followed it.
5. Alert on etcd fsync latency, DB size and leader changes before they become incidents.

Try it yourself: On a kind cluster, snapshot etcd, create a ConfigMap, then restore. Watch the ConfigMap disappear. That is what “restores to the snapshot point” means, made concrete.

Common mistake: Treating etcd snapshots as sufficient DR on a managed cluster where you cannot access etcd at all. On EKS/GKE/AKS your recovery story is Velero plus GitOps plus infrastructure-as-code — and if you have never rebuilt a cluster from those three, you do not know how long it takes.