KMS, Encryption and Secrets

Encryption on AWS is mostly a key management question, and key management is mostly a policy question. The cryptography is done for you; what is left is deciding who can use which key, and making sure the answer survives the day someone leaves the company.

Topic 1: Envelope Encryption

A KMS key never leaves KMS and can encrypt at most 4 KB directly. So for real data, services use envelope encryption:

1. Call GenerateDataKey. KMS returns the same data key twice — once in plaintext, once encrypted under your KMS key.
2. Encrypt your data locally with the plaintext data key. This is fast, local AES, at any size.
3. Store the encrypted data key next to the ciphertext, and wipe the plaintext one from memory.
4. To decrypt: send the encrypted data key to KMS, get the plaintext back, decrypt locally.

Everything AWS encrypts for you — EBS volumes, S3 objects with SSE-KMS, RDS storage, Secrets Manager values — works exactly this way. Understanding it explains the operational properties that follow: KMS is in the authorization path but not the data path, which is why encryption adds no throughput penalty, and why a KMS problem stops new decryptions rather than corrupting anything.

Every KMS call is a CloudTrail event. That is the audit story: you cannot prove nobody copied a file, but you can prove which principal asked to decrypt which key, when, and from where.

Topic 2: Key Types and the Key Policy

The decisive column is the last one. AWS-managed keys cannot be shared across accounts, so a snapshot encrypted with aws/ebs cannot be shared with another account, and a bucket encrypted with aws/s3 cannot be read cross-account. Every one of those turns into an unplanned migration at the moment somebody needs the data elsewhere. If there is any chance of cross-account access, use a customer-managed key from the start.

The key policy is authoritative. This is the one place in AWS where IAM alone is not enough:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "EnableIAMPolicies",
      "Effect": "Allow",
      "Principal": { "AWS": "arn:aws:iam::111122223333:root" },
      "Action": "kms:*",
      "Resource": "*"
    },
    {
      "Sid": "AppDecryptOnly",
      "Effect": "Allow",
      "Principal": { "AWS": "arn:aws:iam::111122223333:role/app" },
      "Action": ["kms:Decrypt", "kms:DescribeKey"],
      "Resource": "*"
    }
  ]
}

Without the first statement, IAM policies granting kms:Decrypt have no effect whatsoever — an administrator with AdministratorAccess is locked out, and the only recovery is an AWS support case. Write the key policy first, and review it as carefully as you would a bucket policy.

Key rotation with --enable-key-rotation generates new key material yearly while keeping the same key ID and ARN. Old material is retained so old ciphertext still decrypts; nothing is re-encrypted. That means rotation is genuinely free of operational risk, and there is no reason to leave it off.

Deleting a key enforces a 7–30 day waiting period, and after deletion everything it wrapped is unreadable, permanently. Before scheduling deletion, check the key’s CloudTrail usage over the last 90 days — a key with recent Decrypt calls is in use by something, whether or not anyone remembers what.

Grants are the mechanism for temporary, programmatic delegation — an AWS service that needs to use your key on your behalf for the duration of an operation. You will mostly encounter them as entries you did not create; list-grants explains them.

Topic 3: Where Encryption Applies

S3      SSE-S3    AES-256, AWS-managed, free, no key control
        SSE-KMS   your key, per-request KMS charges, full audit trail
        SSE-C     you supply the key on every request; you own losing it
        DSSE-KMS  dual-layer, for specific compliance regimes

EBS     KMS-based, on by default if you enable account-level default.
        Cannot encrypt in place — snapshot, copy encrypted, restore.

RDS     Set at creation only. Same snapshot dance to add it later.

Secrets Manager / Parameter Store SecureString
        KMS envelope encryption, transparently.

S3 Bucket Keys deserve a specific mention: with SSE-KMS on a high-traffic bucket, every object operation is a KMS request, and the KMS bill can exceed the S3 bill. A bucket key generates a short-lived key at the bucket level and reduces KMS requests by up to 99%. It is a checkbox, it is nearly free, and it should be on for any bucket with meaningful request volume.

Encryption in transit is the other half, and it is enforced with policy rather than configuration: an aws:SecureTransport deny on buckets, rds.force_ssl on databases, HTTPS-only listeners on load balancers, and TLS certificates from ACM — which are free, and auto-renew as long as validation stays in place. ACM certificates cannot be exported, which is why an appliance needing the private key means either a certificate from elsewhere or ACM Private CA.

Topic 4: Secrets Manager vs Parameter Store

The honest split: rotation is the feature you are paying for. A database credential that should rotate every 30 days belongs in Secrets Manager, because the rotation Lambda changes the password in the database and updates the secret atomically, which is the part that is difficult to build correctly. A log level, a feature flag, or an endpoint URL belongs in Parameter Store, free, in a path hierarchy:

aws ssm get-parameters-by-path --path /app/prod/ --recursive --with-decryption \
  --query 'Parameters[].[Name,Value]' --output text

Rotation only works if the application cooperates. The failure mode is an application that fetches the secret once at startup and caches it forever: rotation succeeds, the old password is invalidated, and the application fails at the next reconnect — often hours later, with no deployment to correlate against. Applications must either re-fetch on authentication failure and retry, or use a caching client with a TTL shorter than the rotation interval. Test this by forcing a rotation while the application runs, which is exactly what the hands-on task asks for.

Never put secrets in: environment variables in a task definition (visible in describe-task-definition to anyone with read access), Lambda environment variables in plaintext, user data, AMIs, container images, or Terraform state — remembering that Terraform state holds every value it manages in plaintext, which is why the state backend deserves the same protection as the secrets themselves.

Topic 5: The Wider Security Services, Briefly

You should know what these do so you can recognise which one belongs in a given conversation:

• GuardDuty — threat detection from VPC flow logs, DNS logs and CloudTrail. Turn it on organization-wide; it needs no agents and the findings are usually specific enough to act on.
• Security Hub — aggregates findings from GuardDuty, Inspector, Config and Macie against standards like CIS and AWS Foundational Security Best Practices. The single pane, and the place a compliance conversation should start.
• Macie — finds sensitive data in S3 by sampling and classifying. Answers “do we have card numbers in a bucket” rather than guessing.
• Inspector — continuous vulnerability scanning of EC2, ECR images and Lambda. Replaces a scheduled scanning job.
• WAF — layer 7 filtering in front of CloudFront, ALB or API Gateway. Managed rule groups handle the common cases; rate-based rules handle the noisy ones.
• Shield Standard — DDoS protection, free, always on. Shield Advanced adds cost protection and a response team for a serious monthly fee.
• CloudHSM — dedicated hardware you control, for the specific regulatory requirement that KMS’s shared model does not satisfy. Significantly more operational burden; choose it because a rule demands it.

The default posture that costs little and catches a lot: GuardDuty and Security Hub on across the organization, Inspector on your registries, WAF with managed rules on anything public, and Config recording the resources you care about.

Try it yourself: create a key whose policy grants decrypt to exactly one role. From a second role with kms:* in IAM, attempt a decrypt and read the error. That single experiment is what makes “the key policy is authoritative” stick permanently.

Common mistake: encrypting an EBS volume or an RDS instance with an AWS-managed key, then needing to share the snapshot with another account during a migration or an incident. It cannot be done — you must copy the snapshot, re-encrypt it with a customer-managed key, and only then share. Doing that under time pressure, with terabytes, is a bad afternoon that a five-minute decision at creation time would have prevented.