Remote Backends & Locking

Local state works until the moment a second person runs Terraform. Then two people hold two different records of the same infrastructure, and whoever applies last wins — silently.

Topic 1: The Problem Remote State Solves

With local state, every operator needs the latest state file before running anything, and nobody may run Terraform at the same time as anyone else. Neither is enforceable by convention.

Remote state puts the file in shared storage. Everyone reads the same record, and the backend enforces mutual exclusion so two applies cannot interleave.

Topic 2: Backend Options

Object storage is the usual default: it is a managed service, supports encryption at rest, supports versioning so you can roll back a corrupted state, and pairs with a lock table.

terraform {
  backend "s3" {
    bucket         = "org-terraform-state-prod"
    key            = "platform/network/terraform.tfstate"
    region         = "us-east-1"
    dynamodb_table = "terraform-state-locks"
    encrypt        = true
  }
}

The key is the path within the bucket. Give each state file a distinct key — one bucket can hold every state file in the organisation, separated by key prefix.

Note that a plain source-control repository is not a supported backend type.

The lock table is ordinary infrastructure:

resource "aws_dynamodb_table" "tf_locks" {
  name         = "terraform-state-locks"
  billing_mode = "PAY_PER_REQUEST"
  hash_key     = "LockID"

  attribute {
    name = "LockID"
    type = "S"
  }
}

This creates a bootstrapping question — the thing that stores state must exist before state exists. The usual answer is a small, separately-managed bootstrap project whose own state lives locally and is committed once, or resources created by hand and then imported.

Topic 3: Locking

State locking happens automatically on any operation that could write state. There is no message — it is silent. If the lock cannot be acquired, Terraform refuses to continue.

terraform apply -lock=false          # disable — not recommended
terraform force-unlock <LOCK_ID>     # clear a stale lock

force-unlock is for the case where a run was killed mid-apply and left the lock held. Confirm nothing is actually running before you use it; clearing a live lock lets two applies interleave, which is the exact failure locking exists to prevent.

While state is locked:

• Blocked: apply, destroy
• Not blocked: fmt, validate, state list

Topic 4: Partial Configuration

You can deliberately omit backend arguments from committed code so credentials and environment-specific values are never shared, then supply them at init time:

terraform {
  backend "s3" {}      # deliberately empty
}

terraform init -backend-config=backend.hcl
terraform init -backend-config=cfg/s3.prod.tf -reconfigure

# backend.hcl — not committed
bucket         = "org-terraform-state-prod"
key            = "platform/network/terraform.tfstate"
region         = "us-east-1"
dynamodb_table = "terraform-state-locks"

This is also how one configuration targets several backends — one backend.hcl per environment, selected at init.

-reconfigure tells Terraform not to copy existing state to the new location. Without it, Terraform detects a local state file and offers to migrate. Which behaviour you want depends entirely on whether you are switching backends or setting one up fresh, and getting it wrong either duplicates state or leaves it behind.

Topic 5: Migrating an Existing Project

The sequence that works, and is worth following in order:

1. Start with the backend block commented out. State is local; iterate quickly.
2. Create the bucket and lock table.
3. Uncomment the backend block and run terraform init. Terraform detects local state and offers to copy it.
4. Answer yes. State moves to the backend.
5. Verify: terraform.tfstate no longer exists locally; the object exists in the bucket.
6. Run terraform plan. It must report no changes. That is the proof the remote backend sees the same infrastructure — if it proposes creating everything, the migration did not happen and you are about to duplicate your estate.

Step six is not optional. It is the only check that distinguishes a successful migration from an empty state file pointing at the wrong place.

Topic 6: Protecting the Backend

• Enable bucket versioning. This is your rollback mechanism for a corrupted state file, and it costs almost nothing.
• Enable encryption at rest (encrypt = true).
• Scope IAM narrowly — the pipeline needs read/write on exactly its state prefix and the lock table, nothing else.

{
  "Effect": "Allow",
  "Action": ["s3:GetObject", "s3:PutObject", "s3:ListBucket"],
  "Resource": "arn:aws:s3:::org-terraform-state-prod/*"
}

Anyone with read access to the state bucket can read every secret in every managed environment. Treat that permission as equivalent to production admin, because it effectively is.

Try it yourself: With remote state configured, start an apply and — while it is running — start a second one from another terminal. Read the lock error. That message is what protects your state from two concurrent writers.

Common mistake: One state file for the entire estate. It works until the plan takes fifteen minutes, the lock is permanently contended, and a mistake in one team’s code can destroy another team’s resources. Split state along team and lifecycle boundaries early — retrofitting is far more expensive.