Terraform on AWS — the watchtower
cloud state, locks, and the deploy role

Infrastructure as code starts with an uncomfortable question: what deploys the deployer? Several laptops and a CI runner need to apply Terraform into three AWS accounts — without two applies ever colliding, and without anyone holding permanent admin keys. The answer is one small pattern with a name that stuck: the watchtower — a stack that stands above all other stacks and owns exactly three things: where state lives, how locks work, and who may deploy.

The problem

Terraform's failure modes at a growing startup are boringly predictable. State on laptops: whoever ran apply last owns reality, and their disk is now production infrastructure. Two concurrent applies: state corruption, or two people creating the same subnet with different opinions. Credentials: everyone who deploys carries god-mode access keys in a dotfile, forever, for every account. Each problem has a known fix — remote state, state locking, role assumption — but the fixes have a chicken-and-egg core: the bucket that holds state, the table that holds locks and the role that deploys are themselves infrastructure. What deploys them?

The watchtower idea

The watchtower is the answer to the recursion. It is a deliberately tiny Terraform stack — a few dozen resources — that is bootstrapped with a local backend, its little state file committed to git alongside the code. It creates the things every other stack will depend on and then, essentially, never changes again:

• The state bucket — one S3 bucket for all stacks' states: versioned (every apply is a recoverable snapshot), KMS-encrypted at rest, all four public-access blocks on.
• The lock tables — DynamoDB tables with a single LockID hash key, one per stack/environment, so a lock queue in dev never delays prod.
• The deployment identities — a CrossAccountDeploy role in every workload account, and per-environment IAM policies that decide who may assume which of them.

The name earns itself: every other stack looks up to it, nothing works without it, and it watches every deployment happen — every state write lands in its bucket, every lock passes through its tables, every apply assumes a role it defined. It is also the only state that lives in git, which is acceptable precisely because it is small, secret-free by construction, and changes a few times a year with two pairs of eyes.

One resource even the watchtower cannot conjure: on a fresh workload account, the CrossAccountDeploy role is created manually, once, by that account's admin — Terraform cannot assume into an account before something exists to assume. It is the single hand-made resource in the estate; from then on everything, including later changes to that very role, flows through Terraform. In order:

# backend: LOCAL, deliberately — this stack is the recursion's base case.
terraform {
  backend "local" {
    workspace_dir = "workspaces"   # committed to git, next to this code
  }
}

# 1. the state bucket every other stack will use
resource "aws_s3_bucket" "tfstates" {
  bucket = "devops-tfstates"
  versioning { enabled = true }                     # every apply recoverable
  server_side_encryption_configuration {
    rule { apply_server_side_encryption_by_default { sse_algorithm = "aws:kms" } }
  }
}
resource "aws_s3_bucket_public_access_block" "tfstates" {
  bucket                  = aws_s3_bucket.tfstates.id
  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

# 2. one lock table per stack/environment — LockID is the whole schema
resource "aws_dynamodb_table" "core_dev_lock" {
  name         = "terraform-core-dev-statelock"
  hash_key     = "LockID"
  read_capacity  = 20
  write_capacity = 20
  attribute { name = "LockID"  type = "S" }
}
# ...same shape for core-prod, core-stage, devops, ...

Cloud state and locks: how a collision becomes impossible

Every real stack then opens with the same eleven lines — its state keyed into the shared bucket, its lock pointed at its own table:

terraform {
  backend "s3" {
    bucket         = "devops-tfstates"
    key            = "core/dev.tfstate"                # this stack's slot
    region         = "ap-southeast-1"
    dynamodb_table = "terraform-core-dev-statelock"    # this stack's lock
    encrypt        = true
  }
}

The lock mechanics are worth spelling out once, because they're what makes the collision structurally impossible rather than unlikely. On every operation Terraform writes a lock item into the DynamoDB table with a conditional put — "create this LockID only if it doesn't exist". DynamoDB's conditional writes are atomic, so exactly one of two concurrent applies wins; the other receives the lock holder's name, operation and timestamp and refuses to run. The state itself, meanwhile, is versioned in S3 — a bad apply is a rollback to the previous object version, not an archaeology dig.

The deploy role: super-admin, but nobody holds it

No engineer and no CI runner has standing admin access to any workload account. Instead, each workload account contains one CrossAccountDeploy role — the super-admin that Terraform's providers assume for the duration of an apply. What a given human or pipeline may do is decided entirely by which of the per-environment deployment policies they carry, and those policies grant exactly three things, all scoped to one environment:

# carried by dev deployers (humans and CI alike)
resource "aws_iam_policy" "terraform_deployment_dev" {
  name = "DevDeploymentTerraform"
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      { Effect = "Allow", Action = ["sts:AssumeRole"],
        Resource = "arn:aws:iam::<dev-account>:role/CrossAccountDeploy" },
      { Effect = "Allow",
        Action = ["dynamodb:GetItem","dynamodb:PutItem","dynamodb:DeleteItem"],
        Resource = ["arn:aws:dynamodb:…:table/terraform-core-dev-statelock"] },
      { Effect = "Allow", Action = "s3:ListBucket",
        Resource = "arn:aws:s3:::devops-tfstates" },
      { Effect = "Allow",
        Action = ["s3:GetObject","s3:PutObject","s3:DeleteObject"],
        Resource = ["arn:aws:s3:::devops-tfstates/core/dev",
                    "arn:aws:s3:::devops-tfstates/core/dev/*"] },
    ]
  })
}

# and in the tenant stack: every provider deploys AS the assumed role
provider "aws" {
  region = "ap-southeast-1"
  assume_role {
    role_arn = var.deploy_role     # CrossAccountDeploy in the target account
  }
}

The consequence chain is the point: identities are cheap and weak, the powerful role is assumed for minutes via STS and never held, and the environment partition runs through all three grants at once — role, locks, state. Add that every apply is a role-assumption event in CloudTrail, and any future compliance audit gets its who-deployed-what answer for free.

Strengthening measures: SSO-only humans, two break-glass keys

The deploy role solves what a session may do; the remaining question is where sessions come from. The answer here is absolute: all accounts live in one AWS Organization, and the only human door is SSO — AWS IAM Identity Center federated to JumpCloud as the identity provider, with user and group provisioning synced (SCIM) from the directory. There are no IAM users with passwords, no long-lived access keys in anyone's dotfiles. Offboarding someone in JumpCloud removes their AWS access in every account at once — the joiner/leaver process and the cloud access model are the same system. And because the directory's users and groups are themselves managed by a small Terraform stack, even who can log in is reviewed code.

The everyday path, end to end: an engineer signs into their SSO session (JumpCloud, MFA) → opens the AWS SSO app and picks an account + permission set — the permission set carrying that environment's deployment policy from the previous section → receives short-lived session credentials → runs Terraform, whose providers then assume CrossAccountDeploy in the target account. Two hops of temporary credentials, zero static secrets anywhere in the chain.

The absolutism needs its escape hatch: if the IdP or Identity Center is down, nobody can log in — including the people who would fix it. Hence exactly two break-glass accounts: plain IAM users outside SSO, hardware MFA, credentials sealed in a vault, never used for daily work. Their entire job is the day SSO cannot open the door. Any authentication by either of them fires an alert — a break-glass login that nobody is expecting is the incident — and the seal is test-broken on a schedule so the day it's needed isn't the day you learn the password was rotated wrong. Two, not one: the second exists for when the first is unavailable — lost token, departed holder, or the account itself impaired.

The whole setup

An apply, end to end

Caveats

• The watchtower's local state is a real trade. It lives in git — reviewable, but state files can carry more than you'd commit knowingly. Keep the stack minimal and secret-free (buckets, tables, roles only), and treat any resource with sensitive attributes as automatically disqualified from living there.
• State-prefix IAM drifts as stacks multiply. The deployment policy enumerates state keys; every new stack means editing the policy. In the real repo the dev-2, dev-data-team suffixes tell the story of that growth — it's fine at this scale, but the enumeration is the security boundary, so review it like one.
• Stale locks are a feature that feels like a bug. A killed run leaves its lock behind, and the fix is a human decision (force-unlock), not a timeout. Resist automating it — an auto-expiring lock is just a slow collision.
• Provisioned lock-table capacity is a fossil. 20 RCU/WCU for a table that sees one item per apply; on-demand billing is the right answer today and costs effectively nothing.
• One bucket, many stacks means the key convention is law. core/<env>.tfstate, devops.tfstate — the prefix structure is what the IAM partition grips. A stack that names its key creatively escapes the permission model without anyone noticing.
• The assumed role is still a super-admin. Scoping who can assume it is the protection; the role itself remains broad because Terraform touches everything. Pair it with CloudTrail alerts on assumption from unexpected principals — the role's power is fine, unaudited use of it is not.

Before you bootstrap: the decision checklist

• Where the base case lives. The watchtower's own state: local + git, a manually-created bucket, or Terraform Cloud? (my choice — local backend committed to git: smallest moving part, fully reviewable, and the stack is secret-free by design)
• Bucket topology. One bucket with key prefixes, or bucket-per-team? (my choice — one versioned bucket, prefix-per-stack, IAM partitioned on prefixes; fewer things to secure identically)
• Lock granularity. One table or per-environment? (my choice — a table per stack/environment: lock waits stay local, and the lock table name in a backend block documents ownership)
• Role model. Role-per-service, or one deploy role per account? (my choice — one CrossAccountDeploy per workload account; the partitioning lives in who may assume it, which is one policy per environment instead of a role matrix)
• Human vs CI paths. Same identity path or separate? (my choice — identical: humans and the runner carry the same per-env policies and assume the same roles, so there is exactly one access model to audit)
• The human door. IAM users, or SSO-only with an IdP? (my choice — Organization-wide SSO: Identity Center federated to JumpCloud with SCIM-synced users and groups, themselves Terraform-managed; zero IAM users except exactly two sealed, alarmed break-glass accounts)
• Non-negotiables on the bucket. (my choice — versioning, KMS, all four public-access blocks, and encrypt=true in every backend block; none of these are optional and all are enforced in the watchtower, once)

Every stack trusts the watchtower; the watchtower trusts only git. That one asymmetry is what lets three accounts, five regions and a dozen engineers deploy without ever meeting in the same state file.