How We Deploy csoh.org Across AWS, GCP & Azure

The big picture: defense in depth

The shape, in one paragraph. One site exists in two environments. Changes land on the qa branch and deploy to qa.csoh.org, a staging copy behind a Cloudflare Access login. When it looks right, a gated one-button promotion fast-forwards main, and production ships the same container image staging already ran to three cloud origins - AWS, GCP, and Azure - which Cloudflare spreads live traffic across, active/active. Two environments, three production origins, one edge in front of all of it, and no stored cloud password anywhere in the path.

The single most important idea on this page is called defense in depth: instead of relying on one strong wall, we stack many imperfect ones. If an attacker bypasses one layer, the next one is still in their way. None of the individual controls below are unbreakable - but together, an attacker has to be lucky on every layer at once, while we only have to be lucky on one.

Our stack has eight layers. Each one is a section on this page:

1. Cloudflare in front of everything - terminates TLS, caches at the edge, runs the WAF, sets security headers, applies legacy redirects, and load-balances across our three origins with health checks. This one edge does all of it, in front of three interchangeable origins.
2. Three cloud origins, active/active - the same site lives on AWS (S3 + CloudFront), GCP (Cloud Run), and Azure (Blob static website). Cloudflare spreads live traffic across all three and pulls any unhealthy one out of rotation automatically.
3. TLS end-to-end (browser → Cloudflare, Cloudflare → each origin at Full strict) - encrypted, modern ciphers, certificates that auto-renew, no unauthenticated hop anywhere.
4. The origins themselves - each locked down: S3 is private (reachable only via CloudFront), Cloud Run runs as a zero-permission identity, Azure serves only its public $web container.
5. The bytes we ship - the GCP container is pinned to known-good bytes and vulnerability-scanned; the object-storage origins get only an allowlisted public file set (sensitive files are never uploaded).
6. The deploy identity - GitHub Actions deploys to all three clouds without a single stored password, using keyless OIDC federation that grants ~1 hour of narrowly-scoped access per workflow run, per cloud. Staging and production authenticate as separate identities, and the staging one cannot reach production.
7. The pipeline itself - Code Owners review on the deploy workflow, branch protection on main, secret scanning, push protection, and a promotion step that refuses to run unless staging actually built and deployed the exact commit being promoted.
8. Logging - Cloudflare zone analytics at the edge, plus per-cloud origin + IAM/audit logs (GCP's kept for 400 days).

Read on for the plain-English version of each layer, what it's defending against, and what the actual config looks like.

What attacks are we actually defending against?

Before designing controls, you need a threat model - a list of "what could go wrong, and roughly how likely is it?" For a public, static site, the surface is smaller than you might think:

• There's no database to inject into.
• There's no login form to brute-force.
• There are no user accounts, sessions, or cookies to hijack.
• There's no per-user data to steal.

That eliminates most of the OWASP Top 10 right off the bat. What's actually left, ranked from most-to-least likely:

1. Someone tampers with the build. An attacker compromises the base image we use, a GitHub Action we depend on, or a CI token, and slips malicious bytes into our deploy. The site visibly looks normal but ships malware to readers. Mitigated by: pinning the base image to its content hash, pinning every GitHub Action to a specific commit, scanning the built image for vulnerabilities, and refusing to overwrite image tags after they're published.
2. Someone steals the deploy credentials. Historically the worst single way to compromise a website: leak the CI's deploy password and now anyone with that password can publish whatever they want. Mitigated by: not having a deploy password at all (see Layer 6 - keyless deploys).
3. Someone messes with our DNS or TLS. Misissued certificate, an on-path attacker downgrading HTTPS to HTTP, DNS hijack. Mitigated by: two-factor on Cloudflare, registrar lock on the domain, HSTS preload (browsers refuse plain HTTP for our domain, period), modern TLS only.
4. Someone defaces the site. Got into the build pipeline somehow and pushed an embarrassing change. Mitigated by: everything in (1) and (2), plus a one-command rollback (every Cloud Run revision is pinned to a specific image hash; "go back to yesterday's deploy" is a single CLI call).
5. Volumetric attack (DDoS) or an origin/region outage. Someone tries to take the site offline by sending an enormous amount of traffic - or one cloud simply has a bad day. Mitigated by: Cloudflare absorbing the bulk of traffic at its edge, edge rate limiting, the CDN serving cached content even if an origin goes down, and three independent origins behind a health-checked load balancer, so an entire cloud can fail and the site keeps serving from the other two.
6. Bot scraping and probing. Bots constantly throw classic attack patterns (?id=1' OR 1=1--) at every endpoint on the public internet. Mitigated by: Cloudflare's WAF (the free Managed Ruleset) plus a rate-limit rule, silently dropping those requests at the edge so they never reach any of our three origins.

What we explicitly do not defend against, because none of it applies to a static site: authenticated session theft, broken access control, business-logic abuse, privilege escalation from an application server. If you're reading this page to copy the design for a site that does have logged-in users - you need more than what's here.

Architecture diagram

Here's the whole system in one picture: two environments on the left, the release path that feeds them on the right. Don't worry if some of the labels are unfamiliar - every box is explained in its own section below.

Everything in that diagram - every origin setting, every Cloudflare rule, every IAM permission across all three clouds - is defined as code in infra/terraform/, which has one directory per cloud (aws/, gcp/, azure/, cloudflare/). We never click around in any cloud console to make changes; the consoles are read-only for normal operation. This is called infrastructure as code, and it's how you keep a real multi-cloud setup from drifting into three different snowflakes nobody can rebuild.

Layer 1 - Cloudflare: the one edge in front of everything

What it stops: volumetric attacks (DDoS), known-bad bots, exposing our origins to the public internet - and a whole cloud going down.

The big idea. Cloudflare isn't just a CDN sitting in front of a cloud load balancer - it is the load balancer, the WAF, the TLS terminator, the redirect engine, and the security-header layer, all at once. Running those same controls again on a per-cloud load balancer would be paying twice for them, so we don't: one edge (Cloudflare's free plan plus the ~$5/mo Load Balancing add-on) does all of it, in front of three interchangeable origins.

MITRE ATT&CK mitigated: T1498 (Network Denial of Service), T1499 (Endpoint Denial of Service), T1595 (Active Scanning).

How it works in plain English. When you type csoh.org in your browser, the DNS lookup returns a Cloudflare IP - not ours. Your browser opens a TLS connection to Cloudflare. Cloudflare looks at the request and one of three things happens:

1. The page is cached at Cloudflare's edge. Cloudflare returns the cached response directly. We never see this request. (For a static site like ours, this is most traffic.)
2. The page isn't cached. Cloudflare opens its own connection to our load balancer and fetches it on your behalf, then caches the result so the next reader gets the cached version.
3. The request is bad. Cloudflare's bot mitigation, rate limiting, or threat intelligence flags it; the request is blocked before it ever reaches us.

This pattern is called a reverse proxy or CDN. The security wins are big:

• Our origin IP is hidden. Public DNS only ever points to Cloudflare. An attacker who wants to attack us has to attack Cloudflare first.
• Floods get absorbed at the edge. Cloudflare's network is much bigger than ours. A DDoS that would take us offline is unnoticed at their scale.
• The browser sees Cloudflare's certificate. Cloudflare manages its own TLS cert with its own auto-renewal. We don't have to hand-feed it our domain.
• One origin failing doesn't take the site down. Cloudflare's Load Balancer (Layer 2) health-checks all three origins and routes only to healthy ones. AWS, GCP, and Azure would all have to be down at once for the site to go dark.

The trade-off, worth being honest about: Cloudflare's free-plan WAF is a lighter rule set than a tunable OWASP Core Rule Set. For a static site with no database or login that's an easy trade (see the "What we didn't do" section for how we'd restore parity). Our origins only ever see requests from Cloudflare IPs, not real readers - which is exactly why per-IP rate limiting belongs at Cloudflare's edge, where the real client IP is visible, rather than at the origin.

Layer 2 - Three cloud origins, active/active

What it stops: a single cloud (or region) outage taking the site down; vendor lock-in; the cost of running a dedicated cloud load balancer just to get an HTTPS front door.

MITRE ATT&CK mitigated: T1499 (Endpoint Denial of Service, via failover), T1498 (Network Denial of Service), T1195 (Supply Chain Compromise, via not depending on one vendor's pipeline).

The shape. The exact same static site lives on three clouds at once. Cloudflare's Load Balancer holds all three in one pool and uses random origin steering to spread live requests across every healthy origin - this is what "active/active" means: they all serve real traffic simultaneously, not "one live, two on standby." A health monitor probes each origin every five minutes; any that fails is pulled out of rotation automatically and slipped back in when it recovers.

The pool holds three members, and the staging origin is deliberately not one of them. qa.csoh.org is a fourth origin - a second Cloud Run service - and it reaches its readers through the same Cloudflare zone, but as a plain proxied DNS record rather than a pool member. Pool membership isn't free: every member is health-probed around the clock. Until 13 September that meant from every Cloudflare data center, roughly 200,000 requests per origin per day even at a five-minute interval (a million a day at the one-minute interval it used until late August), and it had a real price tag (see what this costs to run). Probes now come from three data centers in one region, about one every hundred seconds, which is still often enough to keep a scale-to-zero service permanently warm. Staging wants neither - it should cost nothing while nobody is looking at it - and it has no availability requirement worth failing over for.

Why each origin is shaped the way it is

The one hard requirement: every origin must answer over HTTPS with a valid certificate, so the Cloudflare→origin leg can run at Full (strict) - no unencrypted or unauthenticated hop anywhere. That requirement quietly drives each choice:

• AWS - private S3 bucket behind CloudFront (with Origin Access Control). The cheap, obvious option - the S3 "static website" endpoint - is HTTP-only, which would force an unencrypted origin hop. So instead the bucket stays fully private and CloudFront serves it over HTTPS with a valid *.cloudfront.net cert. CloudFront's free tier covers our egress; the bucket has no public access at all.
• GCP - Cloud Run (scale-to-zero). Its *.run.app URL is already HTTPS with a Google-managed cert and costs ~nothing when idle. That's why this origin is Cloud Run with no cloud load balancer in front of it - the run.app URL is a perfectly good HTTPS origin on its own. (Google Cloud Storage's website endpoint, like S3's, is HTTP-only, so Cloud Run is actually the cheaper path to an HTTPS origin on GCP.)
• Azure - Storage Account "static website" ($web). Azure serves the special $web container over a built-in *.web.core.windows.net HTTPS endpoint with a managed cert - no load balancer, no CDN, just static hosting. The simplest of the three.

Notice the pattern: none of the three needs a cloud load balancer, a managed-cert dance, or a WAF product, because Cloudflare does all of that once at the edge. Each origin is reduced to "the cheapest way this vendor will hand me an HTTPS URL for a folder of files."

The three stacks, side by side

Each cloud ends up with a deliberately different shape, because each vendor's cheapest path to a valid-HTTPS origin is different. Here is the full stack on each, end to end - what serves the bytes, what's exposed, how it gets a cert, how CI publishes to it, and what (if anything) runs code:

The throughline: we let each cloud do the one thing it does cheapest, and pushed everything else (TLS to the browser, caching, WAF, redirects, headers, failover) up to the single Cloudflare edge. That's why two origins run zero code and the third runs a zero-permission container - the less each origin is trusted to do, the smaller the blast radius if any one of them is ever compromised.

One subtlety: the Host header

Each origin answers on its own hostname (…cloudfront.net, …run.app, …web.core.windows.net). If Cloudflare forwarded the public Host: csoh.org to them, each would reject the request - it doesn't recognize that name. So every origin in the Cloudflare pool sets a Host-header override to its own hostname. Small detail, but it's the thing that most often trips people up the first time they put object storage behind a proxy.

WAF, rate limiting, redirects, and caching - set once at the edge

All of it lives at Cloudflare, set once and applied no matter which origin serves the response:

• WAF - Cloudflare's free Managed Ruleset plus a rate-limit rule. It's a light rule set, but a static site has no SQL to inject or login to brute-force, so the rules mostly just eat bot-probe noise (see "What we didn't do" for restoring full-CRS parity).
• Legacy redirects - the /conc8/* and /csoh/* 301 maps are Cloudflare Redirect Rules, so they fire at the edge for every origin identically.
• www to the apex - another Redirect Rule sends www.csoh.org to csoh.org. It builds the destination as concat("https://csoh.org", http.request.uri.path): scheme and host written out in full, and only the path carried over from the request (the rule's preserve_query_string setting brings any ?query=string along).
• Caching - Cloudflare Cache Rules set the edge and browser TTLs (HTML 1h, assets 1y immutable, search.html 60s). The object-storage origins don't emit consistent Cache-Control headers, so setting it at the edge gives uniform caching regardless of which cloud answered.
• HTTP → HTTPS - Cloudflare's "Always Use HTTPS" handles the port-80 redirect; there's no plain-HTTP path to any origin.

Why that www redirect is written the boring way

The clever-looking version of that rule rewrites the whole request URL: match https://www.* against http.request.full_uri and hand back https:// plus whatever the * captured. It reads as equivalent, and it isn't - because the edge runs its features in a fixed order, and dynamic redirects run before "Always Use HTTPS."

Follow a plaintext request through that ordering. Someone (or something - an old bookmark, a mail scanner, a link in a PDF) requests http://www.csoh.org/about.html. The rule's trigger condition is http.host eq "www.csoh.org", which is true on port 80 just as much as on 443, so the rule fires. But the target pattern says https://www.*, and this URI starts with http://. No match. A pattern rewrite that doesn't match returns its input unchanged - so the rule emits a 301 whose Location is the exact URL that was just requested. The browser follows it, arrives at the same rule, gets the same answer, and loops until it gives up. In cleartext the whole time, which is the part that stings: the reader never reaches a response that could have carried an HSTS header for the www hostname, so the browser never learns to stop trying plain HTTP for it.

Deriving the target from http.request.uri.path with the scheme hardcoded removes the failure mode rather than patching it. The rule now has no way to produce a destination that isn't https://csoh.org/…, whatever scheme the request arrived on. The transferable lesson: at a CDN edge, know which phase your rule runs in, and never rebuild a redirect target out of a part of the request you were hoping something earlier had already normalized. State the scheme and host outright; take only the path from the caller.

Layer 3 - TLS (the lock icon in the browser)

What it stops: someone reading or modifying the page in transit between the user's browser and us.

MITRE ATT&CK mitigated: T1557 (Adversary-in-the-Middle), T1040 (Network Sniffing), T1565.002 (Transmitted Data Manipulation).

What's TLS? TLS (Transport Layer Security) is the modern name for what people called SSL - the encryption layer that makes URLs https:// instead of http://. Two computers establish a TLS connection, prove identity to each other with certificates, agree on a shared secret, and from there everything is encrypted. The lock icon in the browser is the user-facing signal.

Our setup has TLS at two separate layers, which often confuses people the first time they see it:

1. Browser ↔ Cloudflare. Cloudflare's "Universal SSL" certificate, valid for csoh.org and www.csoh.org. This is the cert your browser actually validates and shows the lock icon for. It auto-renews on Cloudflare's normal cadence; we don't manage it.
2. Cloudflare ↔ each origin, at Full (strict). A separate TLS connection on the back side of Cloudflare to whichever origin it picked. Each origin presents its own provider-managed cert (CloudFront's *.cloudfront.net, Cloud Run's *.run.app, Azure's *.web.core.windows.net), and Cloudflare's SSL/TLS mode is set to Full (strict), meaning it validates that origin cert rather than blindly trusting it. There is no unencrypted or unauthenticated hop anywhere in the path.

Why two layers and not just one? Because Cloudflare doesn't have your domain's private key. They generated their own cert that the browser trusts (Cloudflare is a public Certificate Authority); each origin presents a cert its own cloud provider issued and renews. Each cert covers what its owner can prove they control, and neither side has to share secrets.

The hardening details

• Modern TLS floor. Cloudflare is configured to refuse TLS 1.0 and 1.1 - only 1.2 and 1.3. Old TLS versions have known weaknesses; refusing them is the easiest "free" hardening you can do.
• HSTS with preload. Every response from our site includes an HTTP header telling the browser "always use HTTPS for csoh.org for the next year." With preload we get added to a list browsers ship with by default, so the protection is active on the very first visit too - even before any of our HTTPS responses have been seen.
• Auto-renewing certificates, everywhere. The edge cert (Cloudflare) and all three origin certs (CloudFront, Cloud Run, Azure) renew themselves on their own provider's schedule. We don't have a calendar reminder for any of them. Expired certs cause more outages than they prevent attacks; eliminating manual renewal eliminates that risk - and with three origins, that's three fewer certs to forget about.

Other security headers - at the edge, and at the origins that can set them

HSTS is the highest-impact header but not the only one. Cloudflare attaches the whole set on the way out (a response-header Transform Rule), so whichever cloud served the bytes, the response a reader actually receives carries the same headers:

• Content Security Policy (CSP) - a strict policy: only first-party scripts, no inline JS, no eval(), only specific image and frame sources allowed. The single highest-impact defense against XSS, even if an attacker were able to inject a <script> tag into our HTML.
• X-Frame-Options: DENY + frame-ancestors 'none' in CSP - prevents anyone from embedding our pages in an iframe on their site. Stops clickjacking, where an attacker invisibly overlays our page under their own UI.
• X-Content-Type-Options: nosniff - tells the browser to trust the content-type we declared, instead of guessing from the first bytes. Closes some old MIME-confusion attacks.
• Referrer-Policy: strict-origin-when-cross-origin - when a reader clicks an external link from our site, the destination only sees that they came "from csoh.org," not the specific page or query parameters.
• Permissions-Policy - explicitly disables camera, microphone, geolocation, payment, USB, and motion-sensor APIs. We don't use any of them, so we deny them.
• Cross-Origin-Opener-Policy + Cross-Origin-Resource-Policy - limit how other origins can interact with windows or load resources from ours. Defends against newer-class side-channel attacks.

You can see all of these by running curl -I https://csoh.org/ from any terminal. They're public; that's the point.

The edge is not the only place they're set

"Set it once at the edge" is a tidy story, and taken alone it's a slightly optimistic one. Each origin has its own public hostname - …cloudfront.net, …run.app, …web.core.windows.net - and each of those answers anyone who finds it, not just Cloudflare. A request that goes straight to an origin skips the edge, and skips everything the edge would have added on the way out. If headers exist only at the edge, then the bare origin serves a fully working copy of the site with no CSP, no HSTS, and no X-Frame-Options on it. So where the platform allows it, each origin sets the headers for itself as well:

• GCP - the nginx container has always carried them, in nginx-security-headers.conf, baked into the image. This origin was never dependent on the edge.
• AWS - a CloudFront response headers policy (aws_cloudfront_response_headers_policy in aws/cloudfront.tf, attached to the distribution's default cache behavior) emits the same set. HSTS, X-Content-Type-Options, X-Frame-Options, Referrer-Policy and the CSP each have a first-class argument in that resource; Permissions-Policy, COOP and CORP have none, so those three are declared as custom headers on the same policy. Same values, slightly different spelling.
• Azure - can't. A Storage Account static website has no mechanism for custom response headers at all: there is no config surface to put them in. The workaround would be putting Azure Front Door in front of it - a second CDN, in front of our CDN, to add a header set the first CDN already adds - which is real money and real complexity for one origin. So this origin genuinely does depend on the edge, and we'd rather write that down than let "we set headers at the edge" imply the coverage is uniform. Honest gaps you can point at are cheaper than surprises.

The bill for that redundancy is three files that have to agree: the Cloudflare ruleset (cloudflare/rules.tf), the CloudFront policy (aws/cloudfront.tf), and the nginx config (nginx-security-headers.conf). Change a header in one and it has to change in all three, or the site starts answering differently depending on which door you came through. Each of the three says so in a comment at the top, naming the other two. Duplication you can't design away is duplication you label.

A control that only exists in Git is not a control

There's a subtler problem with the Cloudflare copy, and it's a good illustration of why "it's in the repo" is not the same as "it's in production." That ruleset carries lifecycle { ignore_changes = [rules] } - a deliberate workaround for a provider bug that returns the multi-header block in a non-deterministic order and so produces a permanent phantom diff. The catch is that rules is the only attribute of a cloudflare_ruleset that means anything. Ignoring it makes the resource inert after it's created: you can tighten the CSP in Git, run terraform apply, get a clean plan and a green pipeline, and change nothing at all at the edge. The commit, the diff, and the reviewer would all be satisfied. The browser would still be getting the old policy.

Since Terraform can't enforce it, the deploy asserts it from the outside instead. check_edge_headers.py parses the header names and values straight out of rules.tf and compares them against what csoh.org actually returns over the wire; any header that's missing or has drifted fails the deploy. It runs in the same post-publish job that already re-derives every published asset's SHA-384 hash from what the edge is serving and checks it against the integrity attribute our pages declare - so the pipeline now verifies both the bytes we ship and the headers we ship them with, against production, after the fact. You can run the header check yourself: clone the repo and python3 tools/check_edge_headers.py, or point it at a single origin with --url. The script is meant to be deleted, incidentally: it exists to compensate for the ignore_changes workaround, and goes away with it when the provider upgrade lands.

Layer 4 - The origins themselves (S3, Cloud Run, Blob)

What it stops: reaching data an origin shouldn't expose; over-permissive cloud access if an origin were ever compromised.

MITRE ATT&CK mitigated: T1530 (Data from Cloud Storage), T1078.004 (Valid Accounts: Cloud Accounts), T1098.003 (Account Manipulation: Additional Cloud Roles).

Each origin is just "the cheapest HTTPS front door this cloud offers for a folder of static files" - but each is locked down so the only thing reachable is the site itself.

Each origin exposes only the site, nothing else

• AWS - the bucket is private. The S3 bucket blocks all public access; only this CloudFront distribution can read it, enforced by an Origin Access Control policy keyed to the distribution's ARN. There's no public S3 URL to find and poke at - the bucket simply isn't reachable except through the front door we built.
• GCP - public URL, zero-permission identity (below). Cloud Run's ingress is all (Cloudflare reaches the run.app URL directly). Public reachability is fine because the container only serves static files and its identity can touch nothing else in the project.
• Azure - only $web is public. The static-website feature exposes exactly one container ($web); every other blob in the storage account stays private. Deploys write to $web through a data-plane role (below), not by making the account public.

The compute origin's identity has zero permissions

The GCP origin is the one that runs code (nginx in a container), so it's the one that needs an identity. Every workload in GCP runs as a service account - an identity that holds the cloud permissions for whatever code is using it. A misconfigured workload SA is one of the most common cloud security mistakes: people grant "Editor" to the application's identity "to make it work," and now every CVE in the application is potentially also a path to "rewrite all the GCP resources in this project."

Our application is static nginx that makes no GCP API calls. So we created a dedicated service account (csoh-run-runtime) with zero IAM roles and run the container as that identity. If the container were ever compromised - RCE in nginx, malicious bytes in the image, anything - the attacker gets a foothold in a process that can't talk to anything else in the cloud project. Its blast radius is the container itself; that's the whole point. (The object-storage origins, AWS and Azure, run no code at all, so there's no runtime identity to abuse there - the attack surface is just "static files served read-only.")

This is a practical example of zero trust applied to your own application: don't grant your code anything you can't justify, and "I might need it later" is not a justification.

Layer 5 - The bytes we ship to each origin

What it stops: shipping malicious or vulnerable bytes to production by accident - or accidentally publishing a file that should never be public.

MITRE ATT&CK mitigated: T1195.002 (Compromise Software Supply Chain), T1525 (Implant Internal Image), T1552.001 (Unsecured Credentials in Files).

Two origin types, two flavors of "what we ship." The GCP origin ships a container image (nginx + the site), so it gets the full container supply-chain treatment below. The object-storage origins (AWS, Azure) ship a folder of files - so for them, "supply chain" means making sure that folder contains only what's meant to be public.

0. The object-storage origins: an allowlist, not request-time blocking

An nginx origin can keep sensitive files (dotfiles, .py scripts, internal .json, anything with a key in it) present in the container but blocked at request time by nginx rules. Object storage has no request-time rules - whatever you upload is world-readable. So for the object-storage origins we flip the model: a single build step (stage_site.sh) stages a dist/ directory containing only the public file set, and that's what gets synced to S3 and Azure. The allowlist (site-publish.filter) mirrors the nginx block rules exactly, and the build fails loudly if a secret-shaped file ever slips into dist/. Not uploading a file is a stronger guarantee than serving it and hoping a deny rule catches every request for it.

What's a container image?

A container image is a packaged-up filesystem snapshot - your application code, plus the operating system files it needs to run, frozen as one shippable unit. For the GCP origin we push the image to a registry (Google's Artifact Registry); Cloud Run pulls it from there and runs it. The three controls below protect that image.

What's pinning? Pinning is naming a dependency by something the publisher cannot quietly redefine. A version like nginx:1.27-alpine or actions/checkout@v4 looks specific, but it's just a label - whoever owns it can repoint that label at different bytes tomorrow, and your build will pull the new bytes the next time it runs. Pinning means replacing that label with an immutable identifier - for container images, the SHA-256 digest of the exact bytes (@sha256:65645c…); for GitHub Actions, the full commit SHA (@a1b2c3d…). The label can move; the hash can't. If anyone tampers with the artifact upstream, the hash no longer matches, and the build fails closed instead of silently shipping the new bytes. We pin every external thing our deploy depends on (base image, every GitHub Action, the Cloud Run revision we route traffic to) for exactly this reason: it makes our deploy tamper-evident and reproducible. The trade-off is friction - somebody has to manually update the pin when we want a newer version - but that friction is the feature: an automated supply-chain attack can't propagate to us silently.

The supply chain for our container has three places where an attacker could substitute "what we meant to ship" with "what they preferred to ship." Each one gets a control:

1. The base image we start from

A typical Dockerfile starts with a line like FROM nginx:1.27-alpine, meaning "use whatever the nginx 1.27-alpine image is right now." But "right now" is whatever the registry returns. If someone compromised the registry, or the image owner's account, or any link in their build chain - that FROM line ships a malicious base layer into your image, and you'd never know unless you bought tooling specifically to detect it.

We pin to the content hash instead:

FROM nginx:1.27-alpine@sha256:65645c7bb6a0661892a8b03b89d0743208a18dd2f3f17a54ef4b76fb8e2f2a10

That long string after the @ is the cryptographic hash of the exact bytes we expect. Docker downloads the image, computes the hash, and refuses to use it if the hash doesn't match. The registry can't substitute different bytes without changing the hash, and the changed hash would make our build fail. Tamper-evident.

The trade-off: we have to manually update the hash when we want a newer base image. That's friction by design - it means an automated supply-chain attack doesn't propagate to us silently.

2. Stale packages on top of the pinned base

Pinning the hash freezes the base image's bytes. But the OS packages inside that base image (libssl, libxml2, libpng, etc.) keep getting new security fixes upstream. A digest pinned 6 months ago has 6 months of accumulated CVEs baked in.

We solve this by running apk upgrade immediately after the pinned base, in our Dockerfile:

RUN apk upgrade --no-cache && \
    rm -rf /var/cache/apk/*

That tells the package manager: "fetch the current versions of every installed package, in this build." We start from a known good snapshot (the digest pin) and end with current security patches (the upgrade). The next layer (Trivy scanning) verifies we haven't missed anything.

3. The CI build artifact

Even with both controls above, a clever attacker might find a CVE in a newly-disclosed package that we just included. So every container we build gets scanned, in CI, by Trivy - an open-source vulnerability scanner. The scan walks every package in the image, cross-references it against public CVE databases, and the build fails if anything HIGH or CRITICAL shows up:

trivy image \
  --exit-code 1 \
  --ignore-unfixed \
  --severity HIGH,CRITICAL \
  "${IMAGE}"

The --ignore-unfixed flag filters CVEs that don't have a fix available yet - those are noise we can't act on, and including them would just train people to ignore the scan output.

If the scan passes, the image is pushed to Artifact Registry (Google's container registry) with two important properties:

• Deployed by digest, not by tag. Every deploy resolves csoh-site:abc123 to its sha256: digest and hands Cloud Run that. A tag is a label and labels can move; a digest is the content itself, so nobody - not an attacker who got into our deploy account, not a careless engineer - can change the bytes a revision runs by moving a tag afterwards. This repo used to buy the same property with immutable_tags, which forbids moving a tag at all. That had to go, and the reason is the most useful thing on this page: Artifact Registry will not delete a tagged artifact while immutable tags are enabled, so immutability and any deletion-based retention are mutually exclusive. Keeping it meant a registry that grows forever. The honest trade is that two workflows racing the same tag now end in a silent overwrite rather than a loud rejection.
• Hash-based, not :latest. Each Cloud Run deploy points to a specific image hash. Rollback is one CLI command: gcloud run services update-traffic --to-revisions <old-revision>=100, and there's zero ambiguity about what bytes the rolled-back revision is running. The registry keeps only the newest ten images, so plan on rolling back that far at most; further back, redeploy the older commit from git.

The Artifact Registry repo has a retention policy: keep the 10 most recent images, and delete any other image, tagged or untagged, once it is a day old. An old image has no use here beyond a quick rollback, and redeploying an older commit rebuilds it anyway. Both delete rules were dead on arrival when they were first written, for two different reasons, and the pair is worth more than the policy. The untagged rule could never match: immutable tags meant an image was tagged at push and could never reach the untagged state. The tagged rule, added to fix exactly that, could never execute: Artifact Registry refuses to delete a tagged artifact while immutable tags are on. So the fix for a rule that could not match was a rule that could not run, and both reported success the whole time. Neither failure is visible in the policy; the only tell was that the inventory never moved. Turning immutability off is what makes retention possible at all, which is why the bullet above trades it for digest pinning.

4. What Cloud Run will actually run

Everything above is about making the image we build trustworthy: a pinned base, upgraded packages, a scan that fails the build, a tag that can't be moved afterwards. None of it answers a different question - will Cloud Run run an image we didn't build at all? For a long time the answer was yes, and nothing on this page stopped it.

The gap is in the IAM role. roles/run.admin grants "create revisions of this service." There is no variant that grants "create revisions from approved images," because that isn't the shape IAM has: it authorizes the verb, not the argument. So an identity that can deploy can name any image it can pull - something off Docker Hub, something built on a laptop and pushed to a public registry. Keyless federation (Layer 6) governs who may deploy. Immutable tags govern whether a tag can change underneath you. Neither is the control that says this image and not that one.

Binary Authorization is that control: a policy Cloud Run consults before it will create a revision. Ours is about as small as this feature gets - allowlist the one repository our CI pushes to, deny everything else:

admission_whitelist_patterns {
  name_pattern = "us-central1-docker.pkg.dev/csoh-org-495800/csoh-containers/**"
}

default_admission_rule {
  evaluation_mode  = "ALWAYS_DENY"
  enforcement_mode = "ENFORCED_BLOCK_AND_AUDIT_LOG"
}

The trailing ** is load-bearing and is not interchangeable with *: a single star matches any run of characters except a slash, so it would stop covering anything at a nested path. Getting that wrong doesn't weaken the policy, it over-denies - which surfaces as a deploy that fails rather than as a control that quietly isn't there. That's the better direction for a mistake to point.

Both Cloud Run services - production and staging - opt in with binary_authorization { use_default = true }, so one project policy governs both. That is the point rather than a shortcut: staging builds the image production later runs, so a staging service allowed to run images production would refuse would quietly make a staging pass mean less than it appears to.

Two things are worth stating precisely here, because "Binary Authorization" is usually shorthand for something stronger than what we've turned on.

• It checks provenance, not a signature. The policy asks where an image came from and accepts "our own registry" as the answer. It does not verify a cryptographic attestation - that's Binary Authorization's other mode, REQUIRE_ATTESTATION, and it's still on the list below. The reason it isn't here yet is more interesting than "we didn't get to it," and it's in that entry.
• It fails at deploy time, not at request time. The policy is evaluated when a revision is created. A rejection breaks the next deploy with an explicit error and leaves the revision currently serving traffic running and untouched. That asymmetry is why we enabled it enforcing rather than in Binary Authorization's dry-run mode: the cost of getting the policy wrong is a failed workflow run, and a dry-run policy is one more instrument that reports success while enforcing nothing.

What this buys is narrower than it first sounds, and the narrow version is the honest one. Both deploy identities also hold registry write, so an attacker who compromised one could push a malicious image into our own repository, and the policy would admit it - it came from the allowlisted repo. What the policy removes is the ability to run an image that never passed through our registry at all: a public image named directly at deploy time, whether by a mistake in a workflow file or by any identity that acquires Cloud Run deploy rights without registry write. And it forces the remaining path to leave evidence, because an attacker's image now has to exist in our registry, under an immutable tag, in a repository we can list. The policy is gcp/binary_authorization.tf.

Layer 6 - How we deploy to three clouds without a saved password

What it stops: credential theft from the deploy pipeline. The most common single vector of website compromise - and with three clouds, three times the credentials that don't exist to steal.

MITRE ATT&CK mitigated: T1552.001 (Credentials In Files), T1552.004 (Private Keys), T1528 (Steal Application Access Token).

This is the most consequential design choice on this page. Read it carefully. The naïve way to deploy to three clouds would be to store three sets of long-lived credentials (an AWS access key, a GCP service-account JSON, an Azure client secret) in GitHub Secrets - tripling the blast radius of a leaked secrets store. We store none of them. Every cloud is reached with keyless OIDC federation: the same idea, implemented three times. We'll walk through the GCP version in detail because it's representative, then show how AWS and Azure do the identical dance.

The traditional approach (don't do this)

Most CI/CD pipelines deploy by storing a long-lived credential in a secret manager. For GCP, that means a service account key - a JSON file with cryptographic material that proves "I am this service account." Workflow runs read the JSON from secrets, presents it to GCP, and uses the resulting access. This works. It's also the source of countless real breaches, because:

• The JSON key never expires unless someone manually rotates it.
• Anyone who reads the secrets store gets it.
• It survives the engineer who created it leaving the company, until someone notices.
• If a leaked secret ends up on GitHub or a Pastebin, it's still useful to attackers months later.

What we do instead: Workload Identity Federation

Workload Identity Federation (WIF) replaces "stored credential" with "prove who you are at the moment you ask for access."

Walking the diagram step-by-step:

1. Our GitHub Actions workflow runs. As part of starting up, GitHub mints a short-lived OIDC token for that specific workflow run. The token is signed by GitHub's identity service and includes verifiable claims describing the run: this is repo CloudSecurityOfficeHours/csoh.org, on branch main, in workflow deploy.yml, run #12345, executing in the production environment. The workflow doesn't get to write those claims; GitHub fills them in from what the job actually is.
2. The workflow hands that token to Google Cloud's STS (Security Token Service), saying "exchange this for an access token, please."
3. Google Cloud's STS checks the token against a policy. Do I trust GitHub's identity service as a token issuer? Yes (we configured it to). Does the token's repository claim equal exactly CloudSecurityOfficeHours/csoh.org? And does its sub claim equal exactly repo:CloudSecurityOfficeHours/csoh.org:environment:production? Both have to be true, or the exchange is refused. (The next section explains why the second one carries most of the weight.)
4. STS returns a 1-hour Google Cloud access token, scoped to impersonating one specific service account - csoh-deployer, our deploy-only identity.
5. The workflow uses that token to push containers and deploy revisions. After 1 hour, the token expires.

Crucially: there is no JSON key anywhere in this flow. There's nothing for a leaked GitHub secret to reveal - the deploy auth is created on-demand, scoped to one workflow run, and discarded. If a workflow log somehow leaked, an attacker would get an access token that's valid for at most one hour, scoped to "deploy to this one project," and they'd have to use it before it expired. There's nothing to rotate, because there's nothing stored.

What the trust policy pins - and why "the repository" isn't enough

Step 3 is where nearly all the security of this design lives, and it is the step most tutorials get wrong. Once you've decided to trust GitHub's OIDC issuer, the only thing standing between "a token exists" and "these credentials are mine" is the condition you wrote about the token's claims. So it's worth being precise about which claim, and why.

The obvious condition is on the repository claim: this token came from CloudSecurityOfficeHours/csoh.org. That reads like a tight rule, and it is a lot better than a stored key. But sit with what it actually authorizes. It trusts every workflow in the repository, on every branch, whether or not it was doing anything to do with deploying. A repo like ours has more than one workflow: link checkers, an SEO audit, a news updater, a scheduled job that reads pages from around the web to propose new resources. None of those has any business minting deploy credentials for a cloud project - and under a repository-only condition, every one of them could, simply by asking. The blast radius of a bug in the least important scheduled job becomes the permissions of the most important one.

So all three clouds pin the sub claim instead. A deploy job's subject is repo:CloudSecurityOfficeHours/csoh.org:environment:<environment>, and that exact string - environment name included - is what each cloud's trust condition names. GitHub composes the subject itself, and only stamps the environment: form onto a token when the job that requested it declares one. That turns entering the environment into the gate. A workflow that doesn't declare one gets a token with a different subject and is refused, no matter what else it holds. We run two environments, production and qa, and the next subsection is why that distinction does real work rather than just naming two copies of the same permission.

Two things follow that are easy to miss:

• Branch enforcement comes along for free, and comes along stronger. Each environment carries a deployment branch policy with exactly one entry - production allows only main, qa only qa. So pinning the environment transitively pins the branch, without a second condition to keep in sync. It's also the better of the two checks: a job can run on main trivially, but it can only enter production by declaring it in the workflow file - which is a Code Owners-protected path (Layer 7).
• Belt and braces on the GCP side. GCP expresses this in two places, and they have to agree: the pool provider's attribute_condition, which is the hard gate that rejects the token outright, and the IAM member on the service account, which is a principal://…/subject/… naming one exact identity rather than a principalSet://…/attribute.repository/… matching a whole group. Either alone would do the job; agreeing with each other means a future edit to one doesn't silently widen the other. There are two of each, one pair per environment.

The generalizable rule, if you take one thing from this section: federate on the narrowest claim that describes the job you actually mean to authorize, not the broadest one that happens to be true. "This is my repo" is true of everything in your repo. That's the problem with it.

The Terraform that wires this up is in gcp/wif.tf - about 30 lines, plus a lot of comments.

Two environments, two subjects, two service accounts

The staging copy is why there are two of these strings on GCP rather than one. qa.csoh.org is built from the qa branch onto the csoh-site-qa Cloud Run service, and a gated workflow promotes a tested commit by fast-forwarding main - Layer 7 walks that path. What matters in this section is the credential: the two branches deploy under two different identities, and the trust conditions are what make that true rather than a convention.

So GCP's trust condition names two subjects, one per environment:

assertion.repository == 'CloudSecurityOfficeHours/csoh.org' &&
(assertion.sub == 'repo:CloudSecurityOfficeHours/csoh.org:environment:production' ||
 assertion.sub == 'repo:CloudSecurityOfficeHours/csoh.org:environment:qa')

Two subjects is a wider gate than one, and it's worth being exact that wider is not the same as looser - because the two don't lead to the same place. Each has its own principal:// IAM member pointing at its own service account, and the two accounts are not equivalent. csoh-deployer holds roles/run.admin at the project level. csoh-deployer-qa holds it on the csoh-site-qa service only - a resource-level binding, not a project-level one - plus read-only run.viewer and registry write. A token minted by the staging workflow cannot deploy production, even though it arrives through the same pool, in the same project, from the same repository, at the same moment.

AWS and Azure name exactly one subject each, the production one, and that's the direct reason staging is one origin rather than three. It costs less than it sounds: Cloud Run is the only one of the three origins that actually executes nginx.conf and the security-header config, so the single staging origin exercises strictly more of the serving path than the two it leaves out.

The corollary is the part to actually remember. Two workflows in this repo read input nobody here controls: the weekly docs review, which reads pages from around the open web, and the pull-request security review, which reads a fork's diff. Both hold id-token: write. Both are safe for exactly one reason, which is that they declare no environment: at all, so the token they can mint carries a subject no cloud trusts and satisfies nothing anywhere. Note the exact form of the warning those two files carry - don't add any environment line, rather than "don't add environment: production." With two environments federated, qa reaches a real service account too, and a warning written against the name of one environment silently stops covering the case the moment a second one exists. A condition is only as narrow as the widest thing that satisfies it - the same lesson as the section above, arriving from the other direction.

The same pattern, on AWS and Azure

The exchange above isn't a GCP feature - it's the OIDC federation standard, and every major cloud speaks it. So the AWS and Azure publish jobs do the identical dance, just with each cloud's nouns:

• AWS - GitHub's OIDC token is presented to AWS STS via sts:AssumeRoleWithWebIdentity. An IAM role (csoh-site-publisher) trusts GitHub's issuer, with a StringEquals condition requiring the token's sub claim to equal repo:CloudSecurityOfficeHours/csoh.org:environment:production. The role can write the S3 bucket and invalidate the one CloudFront distribution, nothing more. (aws/oidc.tf)
• Azure - an Entra ID app registration carries a federated credential whose subject is that same string and whose issuer is GitHub. The app's service principal holds one data-plane role, "Storage Blob Data Contributor," scoped to the one storage account. No client secret is ever created. (azure/identity.tf)

Three clouds, three short-lived tokens minted on demand, zero stored credentials. A leaked GitHub secrets store would reveal nothing useful, because the deploy auth for every cloud is created per-run and discarded.

The detail worth copying is that all three pin a subject string, not a repository, in three different config languages: an IAM trust-policy condition, an Entra federated credential's subject field, and a GCP attribute condition plus IAM member. For AWS and Azure that string is exactly one sentence - "the production deploy job in this repository, and nothing else." GCP's names two, each routed to its own service account, for the reason described above. Writing these the same way everywhere is not tidiness for its own sake. When one cloud's rule is looser than the others, it's the loose one that decides your actual security, and it's the one nobody re-reads because the other two look fine. That is exactly why GCP's second subject is spelled out here rather than left as a detail in the Terraform: it is the one place where the three configs don't say the same thing.

The deploy identities have narrow permissions

Each cloud's deploy identity can do exactly what it needs to publish, and nothing else. The GCP csoh-deployer service account can do exactly three things:

• Push container images to our Artifact Registry repo.
• Create Cloud Run revisions and shift traffic between them.
• Set the runtime identity on a Cloud Run revision (so deploys can specify which service account the running container will use).

It can't read other GCP projects, disable logging, or escalate to admin. The AWS role and Azure principal are scoped just as tightly: write-one-bucket-and-invalidate-one-distribution, and write-one-storage-account, respectively.

The staging deployer, csoh-deployer-qa, is a fourth identity and a deliberately smaller one. It could have been the same account - it does the same job against a different service - and it isn't, because csoh-deployer's run.admin is granted at the project level and therefore covers every Cloud Run service in the project, production included. Reusing it would hand a qa-branch workflow the ability to deploy production, which is the entire thing the two-environment split exists to prevent. So staging gets its own account with run.admin bound to the one service, and the branch that can reach it is fixed by that environment's own deployment branch policy.

And remember from Layer 4: the GCP runtime identity (csoh-run-runtime) the deployer sets on the running container has zero permissions. So even an attacker who compromises the deploy identity AND uses it to ship a malicious container to production… ends up with a malicious container that has no cloud access. Since the Binary Authorization policy in Layer 5, that attacker also can't reach for an image off the public internet: whatever they ship has to be pushed into our own registry first, under an immutable tag, where it is listed and retained. The blast radius is bounded at every layer.

Layer 7 - Protecting the deploy pipeline itself

What it stops: a malicious or accidental change to the workflow that does the deploying.

MITRE ATT&CK mitigated: T1195.001 (Compromise Software Dependencies and Development Tools), T1199 (Trusted Relationship), T1078 (Valid Accounts).

The deploy workflow file can ship code to three production clouds. That makes the file itself as sensitive as a production secret - anyone who can change it can change what gets shipped, everywhere. We protect it with multiple layers on the GitHub side:

• CODEOWNERS - a special file that says "any change to .github/workflows/, infra/, the Dockerfile, or security docs requires review from @Nunley." A pull request touching those paths can't merge without that explicit approval.
• Branch protection on main - the Main ruleset blocks branch deletion, requires a pull request with at least 1 approving review from a code owner, and requires five status checks (ruff, actionlint, yamllint, validate-html, check-urls) to pass before a merge is allowed. None of the five carries a paths: filter on its pull-request trigger, and that's deliberate: a filtered-out workflow never starts, so it never reports, so a required check waiting on it blocks the pull request indefinitely with no failure to look at. The filtering lives inside each job instead, which reports green in seconds when there's nothing to check. Three entries are on its bypass list, and each can push straight to main: the OrganizationAdmin role - which is not one person but every owner of the GitHub organization, currently two - the csoh-ci GitHub App our own automation runs as, and the site owner's own account, listed a second time in its own right. That is a deliberate trade, not an oversight - the housekeeping workflows commit their own fixes (re-stamped SRI hashes, refreshed link caches) and cannot review their own pull requests, and the owner keeps the bypass so a broken deploy can be repaired without waiting on a second reviewer. It costs something real, so it's worth stating plainly: the honest version of this control is "everyone else needs a reviewed pull request," not "nothing reaches main unreviewed." Which is also why the App's token is treated as a production credential everywhere else on this page. The rules are public - curl https://api.github.com/repos/CloudSecurityOfficeHours/csoh.org/rulesets/12819332 - but the anonymous view omits bypass_actors entirely, which is precisely why it is written out here rather than left for you to discover.
• Environment gates - every job that talks to a cloud declares an environment: production for the three publish jobs, qa for the staging deploy. GitHub restricts each to one branch, main and qa respectively, and a pull request from a fork can enter neither - so it can't run those jobs even if the fork's author is sneaky about it.
• Secret scanning + push protection are on. If anyone ever commits a credential-shaped string (an AWS key, a GitHub token, anything Git knows the pattern of), the push is blocked at the moment of git push.
• Dependabot security updates are on. Anything we depend on that ships a CVE generates an automatic PR for us to review.
• Every third-party GitHub Action is pinned to a specific commit SHA, not a version tag. Tags can be moved silently. SHAs can't. This isn't left to reviewer discipline: the repository has GitHub's sha_pinning_required policy enabled, so an unpinned uses: reference is rejected at run time rather than depending on someone spotting @v4 in a diff.

Three workflows, and the gate between them

A change reaches a reader through three workflow files, and CODEOWNERS protects all three the same way.

• deploy-qa.yml - fires on every push to qa, with no paths: filter. That looks like a missing safeguard and is the opposite. GitHub's * doesn't match /, so a pattern list is very easy to write narrower than the set of files actually published, and a filtered-out change deploys nothing while looking entirely successful - which costs precisely the confidence a staging environment exists to provide. A redundant staging deploy costs a couple of minutes. The workflow itself runs the same validation gates, the same build and the same Trivy scan as production, then deploys to csoh-site-qa as csoh-deployer-qa.
• promote-qa.yml - manual, and designed to refuse. Two gates: main must be an ancestor of qa, so the fast-forward can't discard housekeeping commits that landed on main while you were testing; and the exact commit being promoted must have a successful Deploy QA run behind it. That second gate is only trustworthy because of the first bullet - every push to qa produces a run, so "no run for this commit" always means "never tested" and never "filtered out." The two files are coupled, and each says so.
• deploy.yml - fires on the resulting push to main, and publishes to all three origins. Walked step by step below.

The property that makes this worth the extra branch is in the container tag. Both build workflows derive csoh-site:<short-sha> from the commit and push it to the same Artifact Registry repository, and production's publish step skips the push when the tag is already there, then deploys the digest that tag resolves to. So a promoted commit is never rebuilt - production runs the identical bytes staging ran, with no artifact-passing machinery between the two workflows. That holds only while the two produce identical images, which is why there are no staging-only container settings and why Terraform keeps the two Cloud Run services configuration-identical. Anything genuinely staging-specific - the Access login, the cache bypass - lives at the Cloudflare edge, outside the image.

The general shape is worth separating from our particular branch names: a staging environment is only evidence if the thing it tested is the thing that ships. Rebuilding at promotion time gives you a second artifact that you have to argue is equivalent to the first. Reusing the tag removes the argument.

Walking the production workflow

deploy.yml builds once, then fans out to three publish jobs in parallel:

1. Triggers on pushes to main matching specific paths (HTML, CSS, JS, Dockerfile, nginx.conf, the staging script, the workflow file itself), plus workflow_dispatch for manual runs.
2. Concurrency group deploy, one fixed lock for every run of the workflow, with cancel-in-progress: false. A newer commit pends rather than cancelling the run in flight, because a publish job uploads assets first and HTML second across three origins: cancel between those two passes and an origin serves the new style.css beside HTML naming the old hash, which SRI then blocks. Serializing still gives the newest commit the last word, and GitHub keeps at most one pending run per group, so a burst of pushes costs one extra deploy rather than N. Staging sets this the other way round - a Cloud Run rollout is one atomic revision switch, so there's nothing to interrupt, and while iterating you want the newest push to win.
3. Permissions block scopes the auto-injected GITHUB_TOKEN to contents: read + id-token: write (the latter is what lets each cloud's OIDC exchange happen). Nothing else.
4. build job - regenerates the search index, runs stage_site.sh to produce the public dist/ folder, and uploads it as an artifact. One build, so all three origins serve byte-identical content.
5. publish-aws - assumes the IAM role via OIDC, aws s3 sync --delete of dist/ to the bucket, then a CloudFront invalidation.
6. publish-azure - logs in via the Entra federated credential, az storage blob sync of dist/ into $web (sync handles deletions too).
7. publish-gcp - the container path: WIF auth, docker build with org.opencontainers.image.* labels, Trivy scan (fails on HIGH/CRITICAL), push to Artifact Registry under a single immutable hash-based tag, then gcloud run deploy. The push is idempotent - it skips when the tag is already present - which is both what keeps a rerun from tripping over immutable_tags=true and what makes a promoted commit deploy the image staging already built instead of a fresh one.
8. purge-cloudflare - waits for all three publishes, then clears the edge cache (with an API token scoped to exactly one permission, cache purge, on one zone) and verifies what production is now serving. Two checks, both against the live site rather than against the build: the SHA-384 hash of every versioned asset the home page references (anything with a ?v= cache-bust key - in practice style.css, main.js and the analytics loader, the three that every page shares) is re-derived from what the edge returns and compared with the integrity attribute the page declares, and the security headers are compared with the values in the Terraform (Layer 3). Both fail the deploy. Worth stating the edge of that scope rather than rounding it up: the gate fetches one page, so a per-page script like search-init.js or glossary.js is not covered, and neither is the un-versioned vendored MiniSearch bundle. The reason they're worth the extra minute is that everything before this point verifies our intent; only these two verify the result, and the gap between those is where stale caches and unapplied config live.
9. Every publish job declares environment: production - publish-aws, publish-azure, publish-gcp and purge-cloudflare - so a fork PR can't run any of them even if it could mint an OIDC token, because protected-environment rules only apply on main. This is the same declaration each cloud's OIDC trust policy is pinned to (Layer 6), so the environment gate and the credential gate are the same gate. build deliberately does not declare it, and doesn't need to: it checks out the repo, regenerates the search index, stages dist/ and uploads it as an artifact. It never touches a cloud, holds no credential worth gating, and adding the environment there would only add a deployment record for a job that deploys nothing.

The automation that reads the open web

Not everything in .github/workflows/ is the deploy. One scheduled job drafts candidate links for the resources page, which means its whole purpose is to consume pages written by people we've never met. That is the classic shape of a CI compromise: untrusted input meeting a job that holds credentials. Three things keep it boring.

The tool allowlist contains no shell. The job runs with an explicit list of the only things it may do, and that list is exactly Read,Edit,Glob,Grep,WebSearch,WebFetch: read files, edit files, glob, search the workspace, fetch and search the web. Every entry is an in-process tool, and there is no Bash(...) pattern on it at all. That exclusion is the whole design, so it's worth stating flatly: an allowlist that contains a shell or an interpreter is not an allowlist. Permitting python3 looks like one narrow entry next to a dozen others, but python3 -c '…' executes anything, which means that single entry silently re-grants every capability the other entries were carefully written to exclude. A list of eleven safe things and one interpreter is a list of one interpreter.

Nor does it carry a "harmless" utility. Bash(grep:*) reads like a tightly bounded entry and is not one. grep takes a path, like nearly every Unix utility, so that pattern is a general read primitive over the entire runner filesystem - grep . /proc/self/environ returns that step's own environment, secrets and all - and restricting the command name changes nothing about it. The built-in Grep tool on the list instead searches the checked-out workspace and is not a shell. The rule generalizes past interpreters: an allowlist entry naming a command that accepts a path is a file-read capability, whatever the command is for. If a future version of that job genuinely needs Python, it gets a checked-in script allowlisted by its exact path - reviewable, diffable, and unable to become something else after review.

The repo-write credential is minted after the model runs, not before. The csoh-ci App installation token is the credential that could actually change this repository, and because that App sits on the Main ruleset's bypass list, it is a production credential. It is issued in a step deliberately placed below the model step, immediately before the create-PR step that uses it, so during the research pass it does not exist on the runner at all. What is reachable during that pass is the model's own OAuth token, which buys inference and nothing else - not this repo, not any of the three clouds. That is the residual risk, and it is priced deliberately: WebFetch is unqualified and Read takes absolute paths, because researching the open web is the job. Step order is the control that keeps a successful injection cheap.

The checkout doesn't leave a credential behind. This is a default that surprises people: actions/checkout writes the token it authenticated with into .git/config as an http.extraheader, and leaves it there for the rest of the job, so that later git commands keep working. Convenient, and it means any subsequent step in that job can read a live credential out of a plain file. In a job whose later steps process web pages, that's a gift you don't want to be giving. Setting persist-credentials: false turns it off; nothing there needs it, because the step that opens the pull request is handed its token explicitly. The same flag is on the deploy pipeline's own verification checkout, for the same reason. General principle: a credential should live as briefly and in as few places as the work allows, and "the default put it there" is not a reason for it to be somewhere.

Worth noting what this job can and can't do even so: it opens a pull request. It doesn't publish. Everything it proposes still has to clear Code Owners review and land on main before any of it reaches a reader.

Layer 8 - Logging and what we'd see during an attack

What it stops: nothing directly - but it's how we'd notice if any of the layers above failed.

MITRE ATT&CK detection coverage: T1190 (Exploit Public-Facing Application, via WAF block logs), T1098 (Account Manipulation, via IAM change logs), T1078.004 (Valid Accounts: Cloud Accounts, via the audit log stream).

Even with everything above, you should assume something will eventually go wrong. A vulnerability in nginx, a leaked credential we didn't anticipate, a configuration drift no one caught - there's always a possibility. Logging is what turns "an attack happened" into "an attack happened, here's exactly when, here's exactly what they did, and here's what we need to fix." Without logs, you have no idea.

With three clouds, logging lives in two places. The edge - where total traffic is visible - is Cloudflare's zone analytics and Load Balancer health dashboards: requests, cache hit ratio, WAF blocks, and which origins are healthy. The origins log only what got past Cloudflare's cache and actually reached them. The GCP origin keeps the deepest forensic trail, because that's where the IAM and audit story lives.

By default, Google Cloud Logging keeps logs for 30 days. That's not enough for security work - supply-chain attacks specifically are often discovered months after the fact, and the logs you'd need for forensics are gone. We define a custom 400-day retention bucket and a log sink that routes the security-relevant events into it (see gcp/logging.tf). The filter captures three categories:

(resource.type="cloud_run_revision" AND httpRequest.status>=400)
OR protoPayload.serviceName="iam.googleapis.com"
OR protoPayload.@type="type.googleapis.com/google.cloud.audit.AuditLog"

1. Every 4xx and 5xx from the Cloud Run origins. The filter keys on resource.type rather than a service name, so it captures the staging service as well as production - errors and probes that reached either, rather than being served from cache or handled at the edge. Helpful for both performance triage and abuse detection.
2. Every IAM change. If anyone modifies a permission, grants a role, creates a service account - we have it. The single highest-leverage admin event in any cloud project; you almost always want to know about it before the audit happens.
3. The full audit log stream. Every API call against this project, who made it, when, with what outcome.

WAF blocks and per-request edge logs live in Cloudflare, at the layer that does the blocking. (Cloudflare's free tier keeps less log history than a paid plan or our GCP sink would, which is part of the trade noted in "What we didn't do.")

What we don't have (yet): a SIEM, real-time alerting, or anomaly detection. For a community site the cost/benefit doesn't justify it; for a production SaaS workload, you'd want this same sink plus an export to BigQuery for long-term analytics or Pub/Sub for streaming detection. We also keep a Cloud Monitoring dashboard for the GCP origin's day-to-day metrics - request rate, latency percentiles, instance count (defined in gcp/monitoring.tf, "csoh.org Origin" in the GCP console) - and watch Cloudflare's analytics for the whole-site view.

What we didn't do (and why)

Listing controls we considered and rejected is more honest than pretending the design is finished. Each of these is a defensible choice for a small static site and a less-defensible choice as the threat surface grows. If you're copying this design for something bigger, this list is your homework.

• Signed-image enforcement (REQUIRE_ATTESTATION). Binary Authorization itself is on - Layer 5 describes the policy, which requires every image Cloud Run runs to have come from our own Artifact Registry repository. What's absent is the stronger mode, where an image must additionally carry a cryptographic attestation from an approved signer. Two things stand in the way, and neither is effort. First, Cloud Run accepts only the project's default Binary Authorization policy - the flag must literally be set to default, and a project has exactly one - so a single rule governs production and staging together. Requiring an attestation would therefore require it on staging too, and staging is where images are born, before anything has approved them. Routing around that means a separate registry repository for staging plus a digest-preserving copy at promotion, which is a lot of moving parts to buy the second problem: the attestation would be gated on the same identity boundary that already gates registry write. Both are reachable only by a job that can enter the production or qa environment, so signing adds a second lock that opens with the same key. That calculus changes the day the builder and the deployer stop being the same job. Until then the honest version of this control is the durable one below, not the admission-time one.
• Real client IP visibility through Cloudflare. Cloudflare proxy hides the real reader's IP from us. There's a standard way to surface it (Cloudflare adds an X-Forwarded-For header; you configure your origin to trust that header from Cloudflare's IP ranges). We haven't wired this up yet because it requires keeping a current allowlist of Cloudflare egress IPs in Terraform, which is a non-zero maintenance ask. On the to-do list.
• Restricting each origin to Cloudflare's IP ranges. All three origin hostnames answer anyone on the internet who discovers them, not only the edge. We could lock each down to Cloudflare's published egress ranges, and for anything with private data behind it you should - but that's a current IP allowlist maintained in Terraform for three vendors, the same standing maintenance cost that keeps real-client-IP off this list too. What direct reachability costs us is narrow, because the bytes on the origins are the same public bytes the edge serves: what you'd miss is the edge's added behavior, the WAF, the rate limit, the redirects, and the security headers. We've closed most of that gap from the other end by having the origins set the headers themselves wherever the platform permits it (Layer 3), which leaves Azure Blob - which has no way to set response headers - as the one origin whose responses are only fully hardened via the edge. That's a known gap, not an oversight, and it's the reason the header set is duplicated into two origin configs rather than living only at Cloudflare.
• SLSA provenance attestation. A standard for cryptographically attesting "this image was built by this pipeline from this source commit." The slsa-github-generator action makes this fairly easy to add. On the to-do list; would deepen the supply-chain story above.
• Image signing with cosign. Same shape as SLSA, and best done with it. Note what it is not: a prerequisite for the admission entry above. A signature is worth having because it can be checked later, by someone else, without our deploy path being involved - not because it would let Cloud Run reject anything it doesn't already reject.
• Distroless or scratch base image. A truly minimal container has only the application binary, with no shell, no package manager, no system utilities. nginx-on-alpine is bigger than that - but it gives us the URL-rewriting, header-injection, and config flexibility we need today. The Trivy scan + apk upgrade keeps the alpine package surface honest.
• Full OWASP CRS at the edge. Cloudflare's free-plan WAF is the lighter free Managed Ruleset rather than a tunable OWASP Core Rule Set. For a static site with no database or login, a full CRS would be mostly demonstrative anyway. To restore parity you'd move to Cloudflare's paid WAF, or attach AWS WAF to the CloudFront origin - both real options if the threat surface grows.
• Edge-level WebP conversion (Cloudflare Polish). We do serve WebP, the origin-agnostic way: every <img> with a generated .webp sibling is wrapped in a <picture> with a WebP <source> (see wrap_img_webp.py), so capable browsers fetch the smaller file and everything else falls back to the original. What we don't do is Cloudflare Polish - transparent edge conversion with no markup - because that's a Pro-plan feature and we run the free plan plus the Load Balancing add-on.
• Per-region failover within a cloud. Each origin is single-region (one S3 region, one Cloud Run region, one Azure region). We don't need per-cloud multi-region because the cross-cloud failover already covers the realistic outage: Cloudflare health-checks all three and routes around any that's down. A region outage in one cloud just shifts traffic to the other two. Adding multi-region inside each cloud on top of that is cost + complexity that doesn't pay off at our scale.
• SIEM integration / real-time alerting. Logs are retained; we'd notice an attack on the next dashboard check. We don't get woken up at 3am. For a community site, that's the right call. For something with real users and revenue, you'd want at least PagerDuty integration on the most-critical filters.
• Custom VPC-SC service perimeter. GCP's heaviest network-isolation feature - useful when you have services holding sensitive data and want to forbid all data egress outside a defined boundary. Our runtime SA has access to nothing, so there's nothing to perimeterize.
• An enterprise-style landing zone / cloud foundation. Everything on this page lives in one GCP project, one AWS account, and one Azure subscription - flat, single-tenant. An enterprise foundation is a different shape: a folder hierarchy of dozens of projects separated by environment and business unit, a Shared VPC owned by a platform team, Cloud Identity sync from your IdP, aggregated org-wide log sinks into a dedicated security project, Cloud KMS and Secret Manager owned by separate teams, org-policy constraints that pre-block risky configs everywhere, and a Cloud Build / Terraform pipeline that's the only thing allowed to write to prod. We're a single static site with one deployer (Shawn) - we don't need any of that. If you're standing up GCP at company scale, the patterns we did not use are documented in the landing zones & cloud foundations guide.

What this costs to run

Every cloud line below is read from the provider's billing API, and the window each figure comes from is stated rather than implied. That third column used to say "estimated" on two rows. Both were wrong, one of them by about seventy times, and the reason is worth more than the number: an estimate cannot see a credit expiring.

Figures are the daily cost over 14-19 September 2026 scaled to a 30.44-day month, with each provider's monthly free allowance applied once, taken from the GCP BigQuery billing export, the Azure Cost Management API, and AWS Cost Explorer. The "was" figures are the 1-12 September window this table showed until 20 September, and the August figures beside them are the window before that. Cloud Run at $0.02 is not a rounding: at 281 CPU-seconds and 2,700 requests a day, a whole month of it now fits inside the free allowance, and the two cents are network egress. The Cloudflare line is the subscription price, confirmed in the dashboard on 13 September: neither API token this deployment holds is allowed to read billing, and widening one just to fill in a table would be the wrong trade.

A few honest notes on cost:

• The GCP row once said "~$0-1", and the first measurement put it at $67. The estimate was not careless; it was stale. GCP billed exactly $0.00 a day through 27 July 2026, because promotional credits were absorbing everything. They ran out mid-day on the 28th, and from the 29th it billed about $2.25 a day until the fixes below landed, with nothing in the deployment changing on that date - no new service, no traffic spike, no config edit. This is the failure mode that makes cost estimates dangerous rather than merely imprecise: the estimate was right when it was written, the architecture never changed, and the bill went from nothing to sixty-seven dollars a month on a calendar date. A cost note without a date is a claim about a moment someone has already forgotten.
• AWS still reads $0.00 for precisely the same reason, and that is still not reassuring. A credit line cancels the account's genuine usage to the cent every month: $27.31 of it in August, and a $7.90 monthly rate in September, almost all of it S3 (see below). That is the identical shape GCP had in June, one month before the drop. It is in the table as $0.00 because $0.00 is what is billed, but the honest planning number is about $16 rather than $13. There was $17.55 of credit left on 13 September and roughly $16.70 a week later, and the account now bills $2.63 a month, so the credits should last into about March 2027. That date is an estimate, the kind this page has learned to distrust, so what this needs is an alert on the first dollar actually billed rather than a date in a calendar. Since 13 September it has one: every cloud here carries a $10 monthly budget that emails on actual spend and on the provider's forecast, free on all three, because every problem on this list was found by a person reading a bill, weeks late. If you copy this design, find out when your credits expire before you quote the running cost to anyone.
• Cloud Run does scale to zero. It did not help, and the bill explains why. Over 25 days in August the production service billed 25.6 million requests and 1.18 million CPU-seconds, while minimum-instance CPU - the charge you would see if an instance were pinned warm - came to three cents. So the container genuinely is idle-free; the problem is that it is never idle. The Cloudflare load balancer health-checked every origin from every one of its data centers, which at the one-minute interval it used then came to roughly a million probes a day per origin, and on a request-billed platform each probe pays for a request and the CPU-milliseconds to answer it. Scale-to-zero prices idleness, and a health monitor's whole job is to make sure you never have any. The interval went from 60 seconds to 300 on 25 August, and the bill followed almost exactly: requests fell from about 1.03 million a day to 209,000 from the 26th, and this line from $47.64 to $9.95. CPU-seconds fell by less, about 2.8 times, because Cloud Run bills the time an instance is busy rather than each request, and a probe landing every 0.4 seconds on average still kept one busy for about 4.7 hours a day. On 13 September the probes were confined to a single region, three data centers instead of every one, and the service's request log went from about 146 probes a minute to one or two within the minute. A week later this line is $0.02. The monitor now sends about 860 probes a day per origin, the whole month fits inside Cloud Run's free allowance, and what reaches the origin is mostly readers, bots and our own deploy checks: 36 of the 323 requests in a sample hour were the health monitor, against 97.8% of them in August.
• Artifact Registry was the line that grew while nobody was looking, and it has stopped. On 24 August it held 1,074 container images and 219 GB, growing about 2.4 GB a day, because every deploy pushes one and nothing had ever deleted one. The fix took two attempts and neither reported a problem. A delete rule for untagged images had been live and never in dry-run since the repo was created, and could not match a single image, because immutable tags mean nothing here is ever untagged. A second rule targeting tagged images went live on 2026-08-28 and deleted nothing either, because Artifact Registry will not delete a tagged artifact while immutable tags are enabled. A rule that never fires, a rule that fires and finds nothing, and a rule that cannot run at all are indistinguishable from outside. All three report success forever, and the only instrument that ever disagreed was the image count. Retention needed immutable tags turned off, which happened on 30 August, and billed storage fell from 232 GiB on the 29th to 4.8 GiB on the 31st. Even that was not the rule. It was a hand-run delete, and the image count gave it away: the rule's keep-the-newest-50 floor alone should have spared several dozen of the images that went. So the 30-day rule was never once seen deleting anything, and it has now been replaced rather than tested: keep the newest ten images and delete everything else once it is a day old, because an old image has no use here beyond a quick rollback. That landed on 13 September with 99 images in the registry. A week later it holds 13, about 3 GiB, and the line is $0.25 - the first time any rule here has been observed deleting anything.
• The line that grew while nobody was looking moved to AWS, where credits hide it. The S3 origin bucket kept object versioning on, as a rollback and forensics trail, with no lifecycle rule to expire old versions. Every deploy re-uploads the whole site, and each upload turned the previous copy of a file into a noncurrent version that was kept forever. So on 13 September the live site was 255 MB and the bucket was 233 GB: 2.77 million object versions, growing by about 20,000 versions and 1.4 GB a day, which is one full copy of the site per deploy. That was about $5 a month of storage, rising every month, plus about $3 of upload requests, and the bill showed $0.00 for all of it because credits cancel it. It is the Artifact Registry story again in a different cloud: a setting that is individually correct, with nothing to bound it. The usual fix is a noncurrent-version expiry, which keeps a rollback window and stops the slope. This bucket does not need the window, because every deploy rebuilds it from git and reverting a commit is the rollback, so versioning is now suspended and a lifecycle rule keeps only the current copy of each file. The rule ran the same day: the bucket went from 233 GB to 2.3 GB overnight and has sat at 0.51 GB since, favicon.png's 957 old copies are down to the one the last deploy made, and AWS usage fell from $7.90 a month to $2.63. What is left is the uploads, because every deploy still re-uploads every file.
• The health monitor issues a HEAD, and that one word is worth about $100 a month - confirmed, not predicted. A GET / pulls the full 52 KB homepage on every probe, because Azure Blob can't gzip. The daily Azure transfer line ran $4.35-$4.42 every day through 9 August, dropped through the 10th as the change rolled out, and has stayed at a few cents a day or less since the 11th. That is the whole prediction, visible in the billing data, two weeks after the fact. It is only safe because the monitor sets no expected_body - set one and it has to go back to GET, and the bandwidth comes back with it. Note what shrinking the payload does not fix: a HEAD is still a billable storage operation, so Azure's probe charge only fell when the number of probes did.
• Azure is a transaction bill, not a storage bill. Since mid-September it runs at $2.59 a month of write operations and $0.22 of "All Other Operations" - the meter Azure bills a HEAD under - against three cents of data transfer, and a few hundred megabytes of stored data too cheap to register at all. The two halves fell for different reasons: the probe half 4.7-fold when the interval went to five minutes and another twelvefold when the probes moved to one region, and the write half by roughly half on its own, because deploys did, from about 11 a day in August to about 6. Every deploy re-uploads the whole site and every probe is an operation. The rule applies to operation counts exactly as it does to bytes: ask what the fan-out is before asking whether the payload is small.
• Staging really is free, and now that is measured rather than argued. Every dollar of Cloud Run cost belongs to the production service; csoh-site-qa billed $0.00 in August and again in September. Four things have to hold for that: Cloud Run scales to zero, so an idle staging service bills nothing; the image is shared with production, so registry storage doesn't grow on its account; the origin is outside the load balancer pool, so nothing probes it; and Workers (100k requests/day) and Cloudflare Zero Trust (50 users) both sit inside their free tiers. Put a staging origin in the pool and the first and third stop being true at once - which, given what the probes do to Cloud Run above, would roughly double the Cloud Run line.
• For a security community the deployment is the value proposition, and this one is genuinely multi-cloud: the full security story plus active/active failover across three providers, for about $13 a month today and about $16 once the AWS credits lapse. On 25 August this page said $97, and before that it said $30. The architecture did not change at any point. What changed was first what we could see, and then four settings: how often a health check runs, where it runs from, whether a registry may delete anything, and whether a bucket keeps every version of every deploy. The single largest line left is the Cloudflare load balancer that makes the whole thing multi-cloud.

If you want to copy this for your own site

This isn't a step-by-step tutorial - it's a checklist. Each item links to the actual file we use, so you can read the working version. If you want to copy this for your own static site, the path is roughly:

1. Pick how many clouds you actually want. The design degrades gracefully: one origin is a normal "static site behind Cloudflare," two gives you failover, three is what we run. Start with one and add origins later - the Cloudflare pool just grows.
2. Stand up each origin from its Terraform directory. infra/terraform/ has one dir per cloud (aws/, gcp/, azure/, cloudflare/). Each is self-contained: terraform apply in aws/ builds the private bucket + CloudFront + the OIDC role, and so on. See infra/README.md for the per-cloud bootstrap.
3. Wire the origins into the Cloudflare dir. Feed each origin's hostname (from its terraform output) into cloudflare/ and apply - that creates the Load Balancer, pool, health monitor, security-header + redirect + cache rules.
4. Copy .github/workflows/deploy.yml and tools/stage_site.sh, and set the per-cloud resource IDs as repo Variables (the README lists exactly which terraform output feeds each one). Every cloud authenticates keyless via OIDC - no secrets to paste.
5. On GitHub: create a production environment scoped to main; create a CODEOWNERS file requiring your review on workflow + infra paths; turn on branch protection, secret scanning, and Dependabot.
6. Add the staging environment once one origin runs containers. A second environment (qa, scoped to a qa branch), a second service account trusted by that subject and scoped to the one staging service, and a promotion workflow that fast-forwards main. Two properties are what make it worth having, and both are easy to lose: the staging deploy has no paths: filter, so every push produces a run the promotion gate can check for; and both workflows derive the same image tag from the commit, so promotion redeploys the tested image rather than rebuilding it. If your staging origin is object storage rather than a container, you get the first property and not the second.
7. Cut over safely. Verify each origin directly, add them to the Cloudflare LB with the old origin kept as a fallback, then flip DNS - the README has the staged cutover + rollback runbook.

Realistic time investment if you've never used Terraform before: a weekend to read everything carefully and a day to stand up all three clouds. If you only want one origin to start: an evening. If you hit a wall, bring it to Friday Zoom.

Further reading

• The Terraform itself - infra/terraform/, one directory per cloud. Start with cloudflare/load_balancer.tf for the active/active pool; gcp/wif.tf, aws/oidc.tf, and azure/identity.tf for the three keyless-deploy stories side by side.
• The release path - deploy-qa.yml (build, scan, deploy to staging), promote-qa.yml (the two gates and the fast-forward), and deploy.yml (build once, fan out to three clouds). All three heavily commented; QA_PIPELINE_README.md is the design note for the first two.
• Our broader security writeup - SECURITY.md. Covers the security headers, the CI auth model, every secret in the repo, and rotation cadence.
• How CSOH uses GitHub Actions - the companion learning page walks through every workflow file in the repo.
• Glossary - every acronym we used here is in the CSOH glossary with cross-links.
• External documentation:
  • GCP - Workload Identity Federation overview
  • google-github-actions/auth - the action that does the OIDC exchange
  • Cloudflare - Load Balancing - the active/active pool, health monitors, and origin steering this site's edge runs on
  • GCP - Cloud Run ingress controls
  • GCP - Enterprise foundations blueprint - what this same defense-in-depth model looks like at organization scale: folder hierarchy, Shared VPC, aggregated log sinks, Cloud KMS, Security Command Center, and the org-policy constraints we don't need for a single static site
  • OWASP Top 10 - the canonical list of common web app vulnerabilities
  • SLSA - the supply-chain provenance framework we'll add next