Prerequisites
- Terraform installed
- An AWS account
- Basic Terraform syntax (resource, provider blocks)
What You'll Learn
- Explain what Terraform state is and why it must never be edited by hand
- Configure an S3 backend with a DynamoDB table for state locking
- Recover from a stuck lock and from state that has drifted from real infrastructure
- Use terraform state mv / import safely without recreating existing resources
Theory
Terraform state (terraform.tfstate) is a JSON file mapping every resource in your configuration to the real infrastructure object it corresponds to (an actual AWS resource ID, for example). Without it, Terraform would have no way to know that the aws_instance.webblock in your config is the same EC2 instance it created yesterday — it's the only thing that lets terraform plan compute a diff instead of trying to recreate everything on every run.
Why local state breaks on a team
With state stored as a local file, only one person has the current truth at any moment. A second engineer running apply with their own stale local copy can create duplicate resources, silently drop resources from tracking, or apply changes computed against an out-of-date diff — none of which produce an obvious error until something is already broken.
Remote backends and state locking
A remote backend (commonly S3) stores state centrally so every engineer and CI job reads the same current state. Locking (via a DynamoDB table, one item per state file) prevents two apply operations from running concurrently against the same state — the second one blocks until the first releases the lock, rather than both racing to write conflicting changes.
Architecture
DynamoDB's lock table serializes concurrent applies against the same S3-backed state file
Hands-On Lab
# 1. Create the S3 bucket and DynamoDB lock table (one-time, can be its own
# small Terraform config applied with local state, since it's the bootstrap)
aws s3api create-bucket --bucket my-org-tfstate --region us-east-1
aws s3api put-bucket-versioning --bucket my-org-tfstate \
--versioning-configuration Status=Enabled
aws dynamodb create-table \
--table-name terraform-locks \
--attribute-definitions AttributeName=LockID,AttributeType=S \
--key-schema AttributeName=LockID,KeyType=HASH \
--billing-mode PAY_PER_REQUESTterraform {
backend "s3" {
bucket = "my-org-tfstate"
key = "app/prod/terraform.tfstate"
region = "us-east-1"
dynamodb_table = "terraform-locks"
encrypt = true
}
}terraform init # migrates existing local state into the new backend, with a prompt
terraform plan # confirm no unexpected diff after migration
terraform state list # see every resource currently tracked in stateBest Practices
Enable S3 bucket versioning on the state bucket
Versioning means a corrupted or bad state write can be rolled back to the previous version — treat it as a mandatory safety net, not optional.Split state by environment/blast radius
One giant state file for an entire organization means a single mistaken apply can touch everything. Separate state per environment (and often per major component) so a mistake in one state file can't cascade into another.Common Mistakes
Manually editing terraform.tfstate
State files use a structured, version-specific internal format — hand-editing it easily produces a file Terraform can no longer parse, or a subtly wrong mapping. Useterraform state subcommands (mv, rm, import) instead, which validate as they go.Force-unlocking a lock that's actually still in use
terraform force-unlock exists for a genuinely stuck lock (a crashed CI job that never released it) — running it while another apply is legitimately in progress lets two applies write to state concurrently, which is exactly what locking exists to prevent. Confirm no other apply is actually running first.Troubleshooting
"Error acquiring the state lock": first confirm no other apply/planis genuinely running (check with your team, check CI job status) — if it's confirmed stuck from a crashed process, the error output includes the Lock ID; run terraform force-unlock <lock-id>.
State has drifted from real infrastructure (someone changed a resource manually in the console): terraform plan will show the drift as a diff on the next run. To reconcile without applying unwanted changes, use terraform apply -refresh-only to update state to match reality, then decide separately whether the manual change should be codified into the configuration.
Renaming a resource in code would recreate it: use terraform state mv aws_instance.old aws_instance.new to re-point the existing state entry at the renamed resource block, avoiding a destroy/recreate cycle for a change that's purely cosmetic in the config.
Frequently Asked Questions
Summary
State is the map between your configuration and real infrastructure — never edit it by hand. A remote backend with locking (S3 + DynamoDB) lets a whole team and CI share the same current state safely, and the terraform state subcommands (mv, rm, import, refresh-only applies) are the safe way to reconcile drift or reorganize configuration without destroying and recreating real resources.