Secret Manager in Depth

Secret Manager is small enough to learn completely in one lesson, and the parts people get wrong are the version model and the assumption that “rotation” means something is rotated for you.

Topic 1: Secrets, Versions and Aliases

gcloud secrets create db-password \
  --replication-policy=user-managed --locations=europe-west1,europe-west4

echo -n 's3cr3t' | gcloud secrets versions add db-password --data-file=-
gcloud secrets versions list db-password
gcloud secrets versions access latest --secret=db-password

A version is immutable. You never edit a secret; you add a version. Every version has a state:

latest is an alias, not a version. It resolves to the highest-numbered enabled version at the moment of the call, which means a consumer using latest picks up a rotation without a deploy — and also picks up a mistake without a deploy.

Version aliases give you a stable name for a moving target, which is useful when several consumers must move together:

gcloud secrets update db-password --update-version-aliases=current=7
gcloud secrets versions access current --secret=db-password

Topic 2: Disable Before You Destroy

The safe rotation sequence, and the reason for each step:

1. Add the new version.
2. Deploy consumers to the new version (or let `latest` carry them).
3. DISABLE the old version — reversible.
4. Wait. Anything still reading it now fails loudly, and you can undo it in one command.
5. Only then destroy.

Step 3 is how you discover the consumer nobody documented. Destroying instead of disabling turns that discovery into an outage with no undo — there is no backup of a destroyed version.

gcloud secrets versions disable 5 --secret=db-password
gcloud secrets versions enable 5 --secret=db-password     # the undo
gcloud secrets versions destroy 5 --secret=db-password    # no undo

Topic 3: Replication and Residency

# Automatic — Google chooses, simplest, no residency guarantee
gcloud secrets create api-key --replication-policy=automatic

# User-managed — you name the regions
gcloud secrets create api-key \
  --replication-policy=user-managed --locations=europe-west1,europe-west4

The replication policy is fixed at creation. Changing it means creating a new secret and migrating consumers, so decide when the residency requirement is known rather than afterwards.

CMEK on a secret encrypts the payload with your own key, and the key must exist in every replica location:

gcloud secrets create db-password --replication-policy=user-managed \
  --locations=europe-west1 \
  --kms-key-name=projects/acme/locations/europe-west1/keyRings/prod/cryptoKeys/secrets

That gives you the kill switch from the CMEK lesson applied to secrets — disable the key and every payload becomes unreadable.

Topic 4: Rotation Is a Reminder

This is the most common misunderstanding. Secret Manager can publish a message to Pub/Sub on a schedule; it does not generate a new value, change the downstream system, or add a version.

gcloud secrets update db-password \
  --next-rotation-time="2026-09-01T02:00:00Z" \
  --rotation-period="90d" \
  --add-topics=projects/acme/topics/secret-rotation

The rotation you actually want is a small Cloud Run job or Cloud Function subscribed to that topic which:

1. generates or requests a new credential from the source system
2. applies it there (a new database password, a new API key)
3. adds it as a new secret version
4. verifies a consumer can authenticate with it
5. disables the previous version after a grace period

Steps 4 and 5 are what make it safe. Without them a rotation is a deploy that fails hours later.

Consumers must cooperate, exactly as in the Helm and Jenkins modules: read on start and re-read on authentication failure, or use a client library with a cache TTL shorter than the grace period.

Topic 5: Consuming Without a Key File

Cloud Run — mounted as an environment variable or a file, resolved by the service’s own identity:

gcloud run deploy checkout \
  --set-secrets=DB_PASSWORD=db-password:latest \
  --set-secrets=/etc/certs/tls.key=tls-key:3 \
  --service-account=checkout@acme.iam.gserviceaccount.com

GKE — the Secret Manager CSI driver mounts it as a file and refreshes it, using Workload Identity:

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata: { name: checkout-secrets }
spec:
  provider: gcp
  parameters:
    secrets: |
      - resourceName: "projects/acme/secrets/db-password/versions/latest"
        path: "db-password"

Compute Engine — read it in a startup script with the instance’s service account:

gcloud secrets versions access latest --secret=db-password > /run/db-password

Cloud Build — availableSecrets plus secretEnv, covered in the CI/CD module.

In every case the identity doing the reading is a service account with secretAccessor on that one secret. No key file exists anywhere in that chain, which is the whole point and is what the iam.disableServiceAccountKeyCreation org policy enforces.

Topic 6: Access, Audit and Cost

# Per secret, never at the project
gcloud secrets add-iam-policy-binding db-password \
  --member=serviceAccount:checkout@acme.iam.gserviceaccount.com \
  --role=roles/secretmanager.secretAccessor

# Who accessed what, and when
gcloud logging read \
  'protoPayload.serviceName="secretmanager.googleapis.com"
   AND protoPayload.methodName:"AccessSecretVersion"' \
  --freshness=24h \
  --format='table(timestamp, protoPayload.authenticationInfo.principalEmail, protoPayload.resourceName)'

roles/secretmanager.secretAccessor at the project level grants access to every secret in it, including ones created next year. That binding is the most common finding in a Secret Manager audit, and the fix is per-secret grants.

Cost is per secret version per location per month, plus per access operation. Two consequences:

• Access is billed, so a consumer that reads the secret on every request rather than caching it is a line item. Read at startup, cache, and re-read on failure.
• Old versions cost money. Destroy versions you have finished with — after disabling them first.

Try it yourself: pin a consumer to version 2, add version 3, and confirm nothing changes. Then switch it to latest and restart. That contrast is the whole pin-versus-follow decision, and it takes two minutes.

Common mistake: granting secretAccessor at the project level because per-secret bindings felt tedious, then adding a secret months later and not realising every existing workload can already read it. Grant on the secret, and if that is genuinely too many bindings, that is a signal to split the project — which is the argument the hierarchy lesson already made.