How We Use Terraform

What Terraform actually is

Terraform is an open-source infrastructure-as-code (IaC) tool from HashiCorp. You declare the cloud resources you want in .tf files written in HCL (HashiCorp Configuration Language), and Terraform talks to each cloud's API to create, update, or delete resources until the real world matches your code.

The loop has three verbs you'll use constantly:

• terraform init - download the provider plugins your code needs and connect to where your state is stored. Run it once per stack, and again whenever you add a provider.
• terraform plan - compare your code to reality and print exactly what would change. Nothing is applied. This is your safety net; read it every time.
• terraform apply - make the changes from the plan. Terraform records what it created in a state file so it knows what it owns next time.

The thing that makes it declarative: you don't write "create a bucket." You write "a bucket named X should exist," and Terraform decides whether that means create it, change it, or do nothing. Here's the absolute minimum - a single AWS S3 bucket:

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
}

provider "aws" {
  region = "us-east-1"
}

resource "aws_s3_bucket" "example" {
  bucket = "my-globally-unique-bucket-name"
}

terraform init && terraform apply, and the bucket exists. Change the code and apply again, and only the difference is applied. Delete the block and apply, and Terraform deletes the bucket. That shape - declare, plan, apply - stays the same whether you're managing one bucket or an entire multi-cloud edge.

Why it's great (and where it bites)

What's good

• Your infrastructure is reviewable. Every change is a diff in a pull request. You can see exactly what's about to change to production before it happens - and so can a reviewer. (See Git & Version Control and how we wire that into CI.)
• It's reproducible. The same code builds the same infra in a new account or region. Disaster recovery becomes "re-run apply" instead of "remember what we clicked two years ago."
• One language, many clouds. AWS, GCP, Azure, Cloudflare, GitHub, Datadog - 4,000+ providers, all driven from the same HCL and the same plan/apply workflow. Our site spans four providers in one repo.
• It documents itself. The .tf files are the source of truth for what exists. No stale wiki describing a console nobody remembers configuring.
• Drift is visible. terraform plan tells you if someone hand-edited a resource in the console. That's a config-management feature and a security feature - unexpected drift can be the first sign of a problem.

Where it bites

• State is the whole ballgame. Terraform tracks what it owns in a state file. Lose it, corrupt it, or let two people apply at once, and you get a bad day. Remote, locked, versioned state is not optional (we'll show ours).
• State can contain secrets. Generated passwords, keys, and connection strings land in state in plaintext. State must be treated as sensitive and access-controlled.
• The plan is a promise, not a guarantee. Most of the time apply matches plan. Occasionally an API rejects a change mid-apply and you're left in a partial state. Small, frequent applies hurt less than big-bang ones.
• Provider upgrades can surprise you. A major-version provider bump can rename or deprecate arguments. Pin versions and read the changelog (we use ~> constraints and commit the lock file).
• Destroy is real. terraform destroy - or removing a resource block and applying - deletes live infrastructure. plan first, every time, and read the "destroy" lines carefully.

A note on licensing & OpenTofu

Anatomy of a Terraform stack

Open infra/terraform/aws/versions.tf in another tab and follow along - it's short and shows the three pieces that configure Terraform itself.

The terraform {} block - configuring the tool

This block configures Terraform itself, not any cloud. It pins versions and says where state lives, so every laptop and every CI run behaves identically:

terraform {
  required_version = ">= 1.6.0"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"     # any 5.x, never 6.0
    }
  }

  backend "gcs" {            # where state is stored
    bucket = "csoh-org-495800-tfstate"
    prefix = "csoh/aws"
  }
}

Three things to notice: required_version refuses to run on an old CLI; the ~> ("pessimistic") constraint takes bug-fix releases but never a breaking major bump; and the backend says state lives in a Google Cloud Storage bucket. Yes - our AWS stack keeps its state in GCS. That's deliberate: one state store to secure instead of one per cloud (the rationale is commented at the top of the file).

Providers - the plugins that talk to each cloud

A provider is the plugin that translates HCL into a specific platform's API calls. You declare which ones you need (above) and configure them (below):

provider "aws" {
  region              = var.aws_region
  allowed_account_ids = [var.aws_account_id]   # safety tripwire
}

Notice what's not here: credentials. They come from the environment - your local AWS profile, or in CI a short-lived OIDC token. That's why no secrets appear in the file. allowed_account_ids is a guardrail we love: if the credentials Terraform happens to be using point at the wrong account, the apply aborts before touching anything.

Variables, resources, and outputs

The other three building blocks, each in its own file by convention:

• variable (in variables.tf) - named inputs with a type and an optional default, read elsewhere as var.name. Ours all have defaults, so the stack runs with zero manual input.
• resource (in s3.tf, etc.) - one real cloud object to create and manage. resource "TYPE" "NAME" { ... }; refer to it elsewhere as TYPE.NAME.attribute.
• output (in outputs.tf) - values to surface after apply (a bucket name, a role ARN). We feed these into GitHub repo Variables that the deploy workflow reads.

References between blocks (var.bucket_name, aws_s3_bucket.site.arn) do double duty: they pass values around and they tell Terraform the dependency order, so it builds the bucket before the role that points at it. You never sequence things by hand.

A tour of CSOH's four stacks

The entire site is static, served active/active from three clouds behind Cloudflare. That shape is described in How We Deploy; here we're reading the Terraform that builds it. The code lives in four independent stacks under infra/terraform/:

infra/terraform/
  aws/          S3 (private) + CloudFront/OAC + GitHub OIDC role
  azure/        Storage account + $web static website + federated cred
  gcp/          Cloud Run + Artifact Registry + Workload Identity Federation
  cloudflare/   Load Balancer + pool/monitor + header/redirect/cache rules

Each directory is its own stack with its own state - you cd in (or use -chdir) and run init/plan/apply there. All four keep state in the same GCS bucket under separate prefixes (csoh/aws, csoh/azure, csoh/prod, csoh/cloudflare): one secured store, no per-cloud bootstrapping.

Why split into four stacks instead of one?

A smaller blast radius. A bad apply to the Cloudflare stack can't accidentally delete the AWS bucket, because AWS isn't in that state file. Each cloud also authenticates differently, plans faster on its own, and can be destroyed independently. The trade-off is coordinating outputs between stacks by hand - which for a four-stack, solo-maintained repo is a fine price.

Keyless deploys: the security centerpiece

The file worth reading twice is aws/oidc.tf. It sets up deploys with no long-lived cloud secret in the repo at all. Instead, GitHub Actions proves its identity to AWS with a short-lived, signed OIDC token, and AWS hands back temporary credentials. The trust is scoped tighter than most people's first attempt:

condition {
  test     = "StringEquals"
  variable = "token.actions.githubusercontent.com:sub"
  values   = ["repo:${var.github_owner}/${var.github_repo}:environment:production"]
}

Read that condition out loud: only a workflow in this repo, running in the production environment, may assume the deploy role. A token from any other repo, any fork, or any other environment is rejected. The role's permissions are equally tight - write to one bucket and invalidate one CloudFront distribution, nothing else. So even if a token leaked, it could touch exactly one site's files and nothing else in the account. GCP does the same thing with Workload Identity Federation in gcp/wif.tf; Azure with a federated credential in azure/identity.tf.

Mapping a claim is not the same as checking it

All three clouds pin that same subject string, and the reason is worth saying out loud: trusting the repository alone would let any workflow in the repo mint deploy credentials - including scheduled jobs that read untrusted web pages and never go near the production environment. "Only this repo" is not the same statement as "only this deploy job." GCP's attribute condition therefore requires two claims, not one:

attribute_condition = "assertion.repository == '${var.github_owner}/${var.github_repo}' && assertion.sub == 'repo:${var.github_owner}/${var.github_repo}:environment:production'"

and the IAM binding underneath it names that one identity - principal://.../subject/repo:<owner>/<repo>:environment:production - instead of matching a group with principalSet://.../attribute.repository/<owner>/<repo>. The difference is the whole point of the two URL forms: principalSet:// matches every federated identity sharing a mapped attribute, principal:// names exactly one subject. The attribute condition is the hard gate that rejects a bad token outright, so the member is defense in depth - but having both say the same thing is what makes the file honest to read.

Two lessons here generalize well past this one file, and Terraform will warn you about neither:

• Mapping an attribute does nothing on its own. The attribute_mapping block a few lines above maps attribute.ref from the token's ref claim, right next to repository and workflow. Mapping makes a claim available to match on later; it is not a check. A condition that never mentions assertion.ref restricts nothing about the branch, however prominently the mapping sits above it.
• A declared-but-unused variable reads like a control. var.github_branch lives in gcp/variables.tf and is referenced nowhere in the trust. Terraform does not flag an unused variable, so no plan, no lint, and no diff points at it - it simply makes the config look like branch enforcement exists. The variable stays, because it does document the intended branch, but its comment now states plainly that the WIF trust does not read it.

So why pin the environment rather than the branch? Because the GitHub production environment is itself restricted to main by a deployment branch policy. Pinning the environment enforces the branch transitively and adds a gate a ref check alone would miss: a workflow can run on main without ever entering the environment. One string, two controls. As with any change to a stack, it reaches the cloud when someone runs terraform -chdir=infra/terraform/gcp apply, not when the file is merged.

A resource, start to finish

From aws/s3.tf, the private origin bucket and one of its settings:

resource "aws_s3_bucket" "site" {
  bucket = var.bucket_name
}

resource "aws_s3_bucket_versioning" "site" {
  bucket = aws_s3_bucket.site.id      # depends on the bucket above
  versioning_configuration {
    status = "Enabled"                # cheap rollback / forensics trail
  }
}

Two idioms worth internalizing. First, AWS bucket settings are split across several small resources (versioning, encryption, public-access block, policy) that each point back at the one bucket via bucket = aws_s3_bucket.site.id - that reference both targets the bucket and orders the work. Second, versioning here is a deliberate security choice: a bad publish can be rolled back object-by-object, and accidental deletes are recoverable.

Headers at the origin, not only at the edge

The site's security headers - CSP, HSTS, X-Frame-Options and the rest - are declared once at Cloudflare in cloudflare/rules.tf and added to every response on the way out. That covers normal traffic. It leaves a different question open: what does an origin serve when it is reached directly? That is a real path, not a hypothetical, because a CloudFront distribution has its own public *.cloudfront.net hostname.

aws/cloudfront.tf answers it with an aws_cloudfront_response_headers_policy, referenced from the distribution's default_cache_behavior so the origin sets its own headers:

resource "aws_cloudfront_response_headers_policy" "security" {
  name = "csoh-security-headers"

  security_headers_config {
    strict_transport_security {
      access_control_max_age_sec = 31536000
      include_subdomains         = true
      preload                    = true
      override                   = true
    }
    frame_options {
      frame_option = "DENY"
      override     = true
    }
    # plus content_type_options, referrer_policy,
    # and content_security_policy
  }

  # Permissions-Policy, COOP and CORP have no first-class
  # argument in this resource, so they go through here.
  custom_headers_config {
    items {
      header   = "Cross-Origin-Opener-Policy"
      value    = "same-origin"
      override = true
    }
    # plus Permissions-Policy and Cross-Origin-Resource-Policy
  }
}

The GCP origin never needed this: its nginx image already sets the same headers itself from nginx-security-headers.conf. Azure is the honest gap - Blob static websites cannot emit custom response headers at all, so that origin depends on the edge and no amount of Terraform changes it. The cost of the pattern is duplication: the same values now live in three files (cloudflare/rules.tf, aws/cloudfront.tf, nginx-security-headers.conf) and have to move together. That is the usual defense-in-depth trade - maintenance you take on in exchange for a control that survives any one layer being wrong. Like the WIF change above, it is live on AWS once terraform -chdir=infra/terraform/aws apply runs.

The edge stack: build redirect targets, don't rewrite them

The Cloudflare stack is the odd one out - no servers, just rules - and cloudflare/rules.tf holds the one that sends www.csoh.org to the apex domain. The destination is computed, and what it is computed from is the entire lesson:

target_url {
  expression = "concat(\"https://csoh.org\", http.request.uri.path)"
}

Scheme and host are constants. Only the path comes from the request, and preserve_query_string = true just above carries any ?query=string across. The tidier-looking alternative - take the inbound URL and pattern-replace the www. out of it - is a trap, because the destination then inherits whatever the client supplied, including the scheme. Redirect rules run in the dynamic-redirect phase, before "Always Use HTTPS" gets a say, and the matching expression (http.host eq "www.csoh.org") matches plaintext HTTP exactly as happily as HTTPS. A replacement pattern anchored on https:// would not match an http:// request at all, and a replace that matches nothing returns its input unchanged - which is a 301 pointing at the request's own URL. Assembling the target from constants plus one specific field cannot produce that class of bug. This one lands at the edge after terraform -chdir=infra/terraform/cloudflare apply.

Concepts that bite newcomers

1. State is shared, so it must be remote and locked

By default Terraform writes terraform.tfstate to your local disk. That's fine for a tutorial and a disaster for a team or CI: two applies at once race and corrupt it. The fix is a remote backend (our gcs backend) so state is centralized, and state locking so a second apply waits for the first to finish. Never commit a state file to git, and never let two people run apply against the same state simultaneously.

2. State can hold secrets - treat it as sensitive

Any secret a resource generates (a database password, an access key, a TLS private key) is stored in state in plaintext, even if you marked the output sensitive (that only hides it from the console). So the bucket holding state must be encrypted, access-controlled, and versioned - exactly the properties our GCS state bucket is configured with at bootstrap.

3. Always plan before apply

Read the plan. + is create, ~ is change-in-place, -/+ is destroy and recreate (downtime!), and - is destroy. The recreate and destroy lines are where production incidents come from. In CI, save the plan and apply that exact plan so what you reviewed is what runs.

4. Pin providers and commit the lock file

terraform init writes .terraform.lock.hcl, which records the exact provider versions and their checksums. Commit it. It's the supply-chain equivalent of a package-lock.json: it guarantees every machine and every CI run uses byte-identical provider binaries, and it'll catch a tampered provider via checksum mismatch. Ours are committed in each stack (e.g. aws/.terraform.lock.hcl).

5. Drift: the console fights your code

If someone changes a resource by hand in the cloud console, your code and reality "drift." The next plan shows it - Terraform wants to put things back the way your code says. The discipline: change infrastructure in the code, not the console. We even tag every AWS resource managedBy = "terraform" as a visible reminder.

6. count and for_each - don't copy-paste resources

Need three near-identical buckets? Don't write three blocks. for_each over a map (or count over a number) makes one block manage many instances. The gotcha: count indexes by position, so removing the middle item renumbers everything after it and Terraform plans destroy/recreate. Prefer for_each with stable keys when items can be added or removed.

7. terraform -chdir and per-directory state

Because our four stacks are separate directories, we drive them without cd-ing around: terraform -chdir=infra/terraform/aws plan. Each directory is an isolated root module with its own state and its own init. Forgetting which directory you're in is the most common "why is the plan empty?" moment.

8. import existing resources instead of recreating them

Already built something by hand and want Terraform to manage it? terraform import (or an import {} block) brings an existing cloud object under management without destroying it. The alternative - letting Terraform create a duplicate - is how people accidentally end up with two load balancers.

9. lifecycle { ignore_changes } can quietly unmanage a resource

ignore_changes tells Terraform to stop diffing specific attributes. It is the standard escape hatch for a value something else legitimately owns - an autoscaler's instance count, a tag another tool writes, an argument a provider bug reports back wrong. Used narrowly it is fine. The sharp edge shows up when the attribute you ignore is the only one that matters.

Our Cloudflare security-header ruleset carries ignore_changes = [rules], a deliberate workaround for a provider bug that returns the multi-header block in non-deterministic order and so produces a permanent phantom diff. But rules is the only meaningful attribute of a cloudflare_ruleset - everything else is a name, a phase, and a zone. Ignoring it makes the resource inert after creation. You tighten the CSP in Git, run apply, get a clean plan, and ship nothing. Terraform reports success. The commit looks right, the reviewer has no signal, and the edge keeps serving the old policy.

Generalize it: ignore_changes on a resource's only meaningful attribute is equivalent to deleting the resource from your config, except the config is still sitting there describing it convincingly. So when you have to reach for it, pair it with a check that asserts reality from outside Terraform. Ours is tools/check_edge_headers.py: it parses the header name/value pairs back out of rules.tf, requests the live site, and exits non-zero if any header is missing or has drifted. The deploy workflow runs it and fails the build on a mismatch, right alongside the equivalent check for our subresource-integrity hashes. Same shape in both cases: the repo states a claim, and CI verifies it against production rather than against the plan. The check gets deleted when the provider upgrade lets the ignore_changes go.

Security baseline

Terraform is a security tool when you use it well and a footgun when you don't. From day one:

1. Remote, encrypted, versioned, access-controlled state. No local state for anything real. Lock it. Assume it contains secrets, because it can.
2. No secrets in .tf or .tfvars files. Pull credentials from the environment, a secrets manager, or - best of all - don't use long-lived credentials at all. Add *.tfvars and .terraform/ to .gitignore.
3. Prefer keyless / OIDC for CI. Our deploys carry zero static cloud keys; GitHub federates short-lived tokens to each cloud. If you take one idea from this page, take this one. See oidc.tf.
4. Pin the OIDC subject, not just the repository. All three of our clouds require the full subject repo:<owner>/<repo>:environment:production, because trusting the repository alone lets any workflow in it - a scheduled job, a job that reads untrusted input, a job on someone's branch - obtain deploy credentials. And remember that mapping a claim is not checking it: only the condition you actually write is enforced.
5. Verify the control from outside Terraform when Terraform can't. Anything you exclude from management (ignore_changes, a resource applied by hand, a setting the provider does not model) needs a check that reads the live system and fails a build on drift. A clean plan is evidence about your config, not about production.
6. Least-privilege everything. The deploy role can write one bucket and invalidate one CDN - not "PowerUserAccess." Scope provider credentials and resource permissions to exactly the job.
7. Pin provider versions and commit .terraform.lock.hcl. Supply-chain hygiene; checksums catch tampering.
8. Scan your IaC. Static analyzers - Trivy, Checkov, tfsec - catch "this bucket is public" or "this SG is 0.0.0.0/0" before apply. Our deploy pipeline already runs Trivy on the GCP container image; IaC scanning is the same idea, one step earlier.
9. Guardrails against the wrong account. allowed_account_ids (AWS) and equivalent checks abort an apply pointed at the wrong place. Cheap insurance.
10. Review the plan like a code review. Treat a surprising -/+ or - the way you'd treat a surprising rm -rf in a PR.

Using our repo as a learning resource

The point of this page is that our Terraform is real, in-production, and heavily commented for beginners - every non-obvious line has a one-to-three-line explanation. A suggested reading order:

1. Start with the tool config: aws/versions.tf. The terraform{} block, provider pinning, and the remote backend in one short file.
2. Then the inputs: aws/variables.tf. How values are declared once and reused as var.x.
3. Then a real resource: aws/s3.tf and cloudfront.tf. Resources, references, and how settings split across small blocks.
4. Then the security set-piece: aws/oidc.tf. Trust policy vs permissions policy, scoped to one repo and environment.
5. Finally the edge: the cloudflare/ stack - a different provider entirely (Load Balancer, pools, header/redirect/cache rules) driven by the same workflow. Proof that the shape transfers across clouds.

The infra/README.md ties it together: the architecture diagram, the bootstrap commands, the cutover/rollback plan, and the cost table (~$8-12/mo across three clouds). Fork the repo, swap in your own account IDs and bucket names, and you have a working multi-cloud static-hosting setup to learn from.

Further reading

• Terraform documentation - the official source of truth.
• State and backends - read these early; state is where the sharp edges are.
• Terraform Registry - providers and reusable modules. Check the source before you trust a module.
• OpenTofu - the open-source, Linux Foundation fork. Drop-in for everything here.
• Trivy / Checkov - scan IaC for misconfigurations before apply.
• TFLint and terraform-docs - linting and auto-generated docs.
• On this site: How We Use GitHub Actions (the CI that runs this Terraform), How We Deploy (the architecture it builds), and Git & Version Control (the workflow that reviews every change).