Cloud KMS, CMEK and Secret Manager

Encryption at rest is on by default in GCP and needs no configuration. Everything in this lesson is about the cases where “Google holds the key” is not an acceptable answer — and about the operational weight that choice adds.

Topic 1: Three Levels of Key Control

Google-managed (default) — every byte at rest is already encrypted. You cannot see, rotate or disable the key, and there is nothing to operate. For most workloads this is the correct answer, and saying so out loud saves a lot of unnecessary work.

CMEK — customer-managed encryption keys — the key lives in Cloud KMS, in your project, under your IAM. You control rotation, you can disable it, and every use appears in your audit log.

CSEK / Cloud EKM — customer-supplied or external — the key material lives outside Google, in your HSM or an external manager. Strongest separation, and you now own the availability of the key service: if it is unreachable, your data is unreadable.

What CMEK actually buys, stated precisely because it is often oversold:

• A kill switch. Disabling the key makes the data unreadable, immediately, including to Google. That is the control auditors ask about.
• An audit trail of key use in your own logs.
• Rotation on your schedule, and the ability to prove it.

What it does not buy: protection from someone with IAM access to the data itself. A principal who can read the bucket can read the bucket — CMEK does not change that.

Topic 2: How CMEK Works, and What It Costs

gcloud kms keyrings create prod --location=europe-west1

gcloud kms keys create checkout-data \
  --location=europe-west1 --keyring=prod --purpose=encryption \
  --rotation-period=90d --next-rotation-time=2026-09-01T00:00:00Z

The service generates a data encryption key, encrypts your bytes with it locally, and asks KMS to wrap that DEK. Your key never leaves KMS; your data never enters it. The same envelope model as every other cloud, and the reason CMEK adds no measurable throughput cost.

Grant the service agent, not your users. Each managed service has its own service agent identity that must be allowed to use the key:

# GCS
gcloud kms keys add-iam-policy-binding checkout-data \
  --location=europe-west1 --keyring=prod \
  --member="serviceAccount:service-${PROJECT_NUMBER}@gs-project-accounts.iam.gserviceaccount.com" \
  --role=roles/cloudkms.cryptoKeyEncrypterDecrypter

Forgetting this produces a resource creation failure naming a service account you have never seen, and it is the most common first CMEK error.

Three constraints that shape the design:

• The key must be in the same location as the resource. A europe-west1 key cannot protect a us-central1 disk. Multi-region resources need a matching multi-region key ring.
• Rotation re-keys new data only. Existing ciphertext keeps its old key version, which is why old versions must stay enabled — destroying one destroys the data it wrapped.
• Key destruction has a 24-hour minimum delay (configurable up to 120 days). That window is your protection against a mistake, and it is the only one.

gcloud kms keys versions list --key=checkout-data --keyring=prod --location=europe-west1
gcloud kms keys versions disable 3 --key=checkout-data --keyring=prod --location=europe-west1

Topic 3: Where CMEK Applies

”At creation only” appears three times in that table, and it is the planning point: deciding CMEK after a Cloud SQL instance exists means creating a new instance, migrating the data, and cutting over. Decide before you create, or accept the migration.

The organisation policy that makes it non-optional:

gcloud resource-manager org-policies enable-enforce \
  gcp.restrictNonCmekServices --folder=987654321

That constraint refuses creation of resources without CMEK for the listed services — the guardrail equivalent of a code review that never gets skipped.

Topic 4: Secret Manager

KMS protects data at rest inside a service. Secret Manager stores the credential your application reads — a different problem with a different tool.

echo -n "s3cr3t" | gcloud secrets create db-password \
  --replication-policy=user-managed --locations=europe-west1 --data-file=-

gcloud secrets versions add db-password --data-file=./new-password.txt

gcloud secrets add-iam-policy-binding db-password \
  --member=serviceAccount:checkout@acme.iam.gserviceaccount.com \
  --role=roles/secretmanager.secretAccessor

The properties that matter operationally:

• Versions are immutable and numbered. latest is an alias; pin a version where a rotation should be a deliberate deploy.
• IAM is per secret, so secretAccessor on one secret is not access to the others. Grant it per secret, never at the project level.
• Every access is an audit log entry with the caller identity — which is the trail Cloud KMS gives you for keys and a CI system’s own secret store usually does not.
• Replication is a choice. automatic is simplest; user-managed with explicit locations is what a data-residency requirement needs.

Consuming it without a key file, which is the whole point:

# Cloud Run — mounted as an environment variable, resolved at start
gcloud run deploy checkout \
  --set-secrets=DB_PASSWORD=db-password:latest \
  --service-account=checkout@acme.iam.gserviceaccount.com

# GKE, via the Secret Manager CSI driver — mounted as a file, refreshed
volumes:
  - name: secrets
    csi:
      driver: secrets-store.csi.k8s.io
      readOnly: true
      volumeAttributes: { secretProviderClass: checkout-secrets }

Rotation only works if the application cooperates — the same rule as everywhere else in this course. An application that reads the secret once at startup and caches it forever keeps working until the old version is disabled, then fails at the next reconnect with no deployment to correlate against. Either re-read on authentication failure, or use a client with a TTL shorter than your rotation interval.

Topic 5: Encryption in Transit, Briefly

At rest is the easy half; in transit is where a requirement is usually actually failed.

• Google-managed certificates on a load balancer are free and auto-renew. Use them unless something specifically prevents it.
• Cloud SQL — enforce SSL (--require-ssl) or, better, connect through the Auth Proxy / Connector, which encrypts and authenticates with IAM and removes the authorised-network list entirely.
• Internal traffic between Google services stays on Google’s network and is encrypted at the physical layer; for service-to-service mTLS inside GKE, that is a service mesh decision, not a GCP one.
• VPC Service Controls is not encryption and is often confused with it — it stops data leaving a perimeter, which is a separate control covered in the private-connectivity lesson.

Topic 6: An Audit You Can Run Today

# Every key, its rotation schedule, and whether it has one
gcloud kms keys list --location=europe-west1 --keyring=prod \
  --format='table(name, purpose, rotationPeriod, nextRotationTime)'

# Who can use each key — this list should be short and be service agents
gcloud kms keys get-iam-policy checkout-data --keyring=prod --location=europe-west1

# Secrets with no access policy of their own (inheriting project-level access)
for s in $(gcloud secrets list --format='value(name)'); do
  n=$(gcloud secrets get-iam-policy "$s" --format='value(bindings.role)' | wc -l)
  [ "$n" -eq 0 ] && echo "no per-secret IAM: $s"
done

# Service-account keys, which should not exist at all
gcloud iam service-accounts list --format='value(email)' | while read sa; do
  gcloud iam service-accounts keys list --iam-account="$sa" \
    --managed-by=user --format='value(name)' | sed "s|^|$sa |"
done

The questions those answer, and they are usually uncomfortable: which keys have no rotation, which secrets are readable by the whole project, and how many downloadable service-account keys exist despite the org policy that was supposed to prevent them.

Try it yourself: disable a key version on a CMEK-protected bucket and try to read an object. The failure is immediate and total, and it is the clearest possible demonstration that the key is a live dependency rather than a checkbox.

Common mistake: enabling CMEK across the estate because a compliance document asked for “customer-managed keys”, without deciding who operates the keys. The key becomes a single point of failure nobody owns: rotation stalls, an old version gets destroyed during a clean-up, and data becomes unrecoverable. Decide the owner, the rotation schedule and the destruction policy before the first key protects anything real.