2021-2022 Cloud Breach Review

Where it started: Log4Shell broke dependency visibility, Codecov and Kaseya established the pipeline and the management plane, and MFA became the battleground.

· · Analysis · View source on GitHub

These two years are treated together because they tell one story. Between them, the software supply chain and the identity layer both became primary targets for the first time, and the industry acquired two capabilities it had largely been managing without: knowing what is inside its own software, and having a second factor that cannot be phished.

Seven kill chains on this site fall in this period. None of them are the largest breaches of the era by record count. All of them are here because they established something that every later year assumes.

On this page

  1. Overview: the shape of 2021-2022
  2. The incidents that defined the period
  3. Theme 1: You cannot patch what you cannot find
  4. Theme 2: The build pipeline and the management plane
  5. Theme 3: Tenant isolation is not free
  6. Theme 4: MFA became the battleground
  7. What this set up

Overview: the shape of 2021-2022

December 2021 is the closest thing cloud security has to a shared memory. Log4Shell was not the most damaging event of the period by any measurable standard - most organizations were never confirmed compromised - but it was the one where a simple question from leadership turned out to have no answer. "Are we affected?" met with "we will have to go and find out", and finding out took weeks of grepping filesystems, scanning images and emailing vendors.

The rest of the period established the two structural targets that every later year builds on. Codecov and Kaseya showed that build pipelines and management planes are worth attacking precisely because everyone downstream trusts them. ChaosDB showed that tenant isolation - the property every multi-tenant cloud service rests on - can fail, and that no customer control sits between one tenant and another.

And 2022 was when multi-factor authentication stopped being a checkbox and became a battleground, with three distinct outcomes: defeated by fatigue at Uber, defeated by real-time relay across 130 organizations, and - at exactly one company - not defeated at all.

The incidents that defined the period

IncidentWhenRoot cause in one line
Codecov Bash UploaderJan-Apr 2021A credential recoverable from a Docker image layer allowed edits to a script everyone curls
Kaseya VSA / REvilJul 2021Zero-days in an internet-facing RMM server, then ransomware down the agent update channel
ChaosDB / Cosmos DBAug 2021A notebook feature allowed escape to other tenants' database credentials
Log4Shell (CVE-2021-44228)Dec 2021A logging library evaluated its own input; nobody knew where it was installed
Okta / LAPSUS$Jan-Mar 2022A subcontractor's support engineer, and two months before customers were told
UberSep 2022Purchased credentials plus MFA push fatigue, then a hardcoded PAM secret
0ktapus / TwilioAug 2022SMS phishing with live one-time codes relayed in real time, across 130+ organizations

Theme 1: You cannot patch what you cannot find

Log4Shell earns its place on this site despite not being a breach, because of what it changed rather than what it did. The flaw itself is a category error - a logging library evaluating expressions inside the strings it was asked to log, so a crafted value in a username field or an HTTP header caused the server to fetch and execute remote code.

What made December 2021 memorable is that Log4j was overwhelmingly a transitive dependency. Almost nobody chose it. It arrived bundled inside application servers, monitoring agents, network appliances, build tools and commercial products whose vendors themselves took days or weeks to answer. Teams found it in fat JARs, in container images they had not rebuilt in a year, and in hardware they could not patch at all.

Organizations with an accurate software bill of materials and a real asset inventory identified their exposure in hours. Everyone else spent weeks, and some never reached certainty. That contrast is why SBOM adoption moved from a compliance idea to something security teams actively wanted, and why software composition analysis stopped being optional in mature pipelines. It is also the period's clearest demonstration of the maintainer-sustainability problem: a library underpinning a vast share of enterprise Java was maintained by a handful of unpaid volunteers, who spent that December fixing it under extraordinary pressure.

Theme 2: The build pipeline and the management plane

Two 2021 incidents established target classes that recur in every subsequent year.

Codecov is the ancestor of the modern CI supply-chain attack. A credential recoverable from a Docker image layer gave write access to the Bash Uploader - a script tens of thousands of pipelines download and execute on every build - and the payload was a single added line exporting the CI environment to an attacker server. It ran undetected for two months and ended only because one customer compared the downloaded file's SHA-256 against the published value. HashiCorp had to rotate the GPG key used to sign its product releases, which is to say a CI-tool compromise reached another vendor's release-signing key.

Kaseya established the management plane. REvil exploited zero-days in internet-facing VSA servers and used the product's own agent update mechanism to push ransomware, so the encryptor arrived on victim machines as a trusted update from the company managing their IT. Roughly 50 to 60 managed service providers produced about 1,500 encrypted businesses, and Coop Sweden closed around 800 stores despite having no relationship with Kaseya at all. Timed for the July 4 weekend, deliberately.

Theme 3: Tenant isolation is not free

ChaosDB is research rather than a breach, and it belongs here for the same reason Log4Shell does. Wiz researchers found that the Jupyter Notebook feature added to Azure Cosmos DB could be used to escalate out of a customer's own notebook container, rewrite firewall rules, reach the backend, and retrieve the primary keys of other customers' databases. Several thousand Azure customers were exposed, including many in the Fortune 500.

What makes it worth documenting is that every customer-side control the industry recommends was irrelevant. Network restrictions, least privilege, MFA, encryption at rest, private endpoints - none of them sit between one tenant's notebook and another tenant's keys, because that boundary belongs to the provider and customers have no visibility into it. Microsoft disabled the feature within about 48 hours and advised every Cosmos DB customer to regenerate their primary keys.

The lever customers actually hold is credential lifetime. Cosmos DB primary keys are long-lived and rarely rotated, so a key obtained during the exposure window kept working until someone changed it. That is the transferable control, and it is the same one the Entra ID Actor token flaw would demand four years later.

Theme 4: MFA became the battleground

2022 produced three distinct answers to the same question, and reading them together is the most useful thing this period offers.

Defeated by fatigue. At Uber, an attacker who had purchased a contractor's credentials still faced MFA, and defeated it with a barrage of push prompts plus a message claiming to be from Uber IT. Push-approve MFA asks a tired human to make a security decision with one tap, repeatedly, until they make it wrong.

Defeated by relay. The 0ktapus campaign sent SMS messages impersonating IT to employees at scores of companies, captured credentials and live one-time codes on convincing lookalike Okta pages, and relayed them to the real login inside the code's validity window. Roughly 9,931 accounts across more than 130 organizations, including Twilio - and through Twilio, phone numbers and SMS verification codes for around 1,900 Signal users.

Not defeated. Cloudflare received the same messages, on the same day, from the same actor. At least three employees entered their credentials and their one-time codes into the phishing page. The attack failed anyway, because Cloudflare required FIDO2 hardware security keys, and a hardware key performs a cryptographic challenge scoped to the origin it was registered against - it will not respond to a lookalike domain no matter how convincing the page or how cooperative the human.

That last one is the only chain on this entire site where a control demonstrably held. It is worth noticing that Cloudflare's account does not claim its people were smarter. They were phished, they fell for it, and the technical control absorbed the failure completely.

What this set up

Third-party and fourth-party risk became unavoidable. The Okta / LAPSUS$ compromise reached customers three relationships away - through a subcontractor of a support provider to their identity provider - and the two-month disclosure delay meant the organizations who might have checked their own logs were never asked to. Both halves of that recur constantly afterwards.

The build pipeline became a permanent target. Codecov's descendants run through tj-actions, Nx s1ngularity, Shai-Hulud, LiteLLM, Mini Shai-Hulud and Megalodon, and the mechanism gets more sophisticated every year while the underlying exposure - a build step is a principal holding every secret in the job - does not change.

Phishing-resistant MFA became the recommendation it still is. Twilio's own response was to issue FIDO2 hardware keys to all employees. Every subsequent year on this site contains at least one chain that a hardware key would have broken, and several that it would not - which is why the later years are about what happens after authentication rather than at it.

More in this series