Cloud Security Office Hours Banner

ChaosDB / Cosmos DB 2021

Step-by-step kill chain mapped to MITRE ATT&CK Cloud, sourced from official post-mortems and primary technical analyses.

August 2021 Critical Azure Cosmos DB

ChaosDB - A Notebook Feature in a Managed Database → Local Privilege Escalation → Firewall Rules Rewritten → Full Admin on Other Tenants' Databases

In August 2021 Wiz researchers found that the Jupyter Notebook feature added to Azure Cosmos DB in 2019 could be used to escalate out of a customer's own notebook container, reach the Cosmos DB backend, and retrieve the primary keys of other customers' databases. The result was complete unrestricted read, write and delete access to the accounts and databases of several thousand Azure customers, including many in the Fortune 500. Microsoft disabled the feature within roughly 48 hours of the report and advised every Cosmos DB customer to regenerate their primary keys. Like the Entra ID Actor token flaw, this was research rather than a breach - but it is the more fundamental one. Actor tokens broke authentication. ChaosDB broke tenant isolation, the property every multi-tenant cloud service is built on, and the one customers have no ability to verify or compensate for.

Several thousand Azure customers exposed, many Fortune 500
Tenant isolation - the property you cannot verify or compensate for
~48 hours from report to Microsoft disabling the feature
The keys were long-lived, so exposure predated discovery by months
📄 Wiz - ChaosDB: unauthorized privileged access to Azure Cosmos DB ↗ 📄 Wiz - ChaosDB explained: a walkthrough ↗
🔬 The Setup - A Convenience Feature With Its Own Runtime
01
Cosmos DB gained an embedded Jupyter Notebook feature in 2019, giving every customer a code-execution environment inside the managed service
T1580 - Cloud Infrastructure Discovery

The feature is genuinely useful - query your data, visualize it, run Python next to it, without moving anything. It is also, structurally, the provider offering to run arbitrary customer code inside its own managed service, which changes the isolation problem completely. A managed database normally exposes a query interface; adding notebooks means the provider now has to contain a general-purpose runtime, per tenant, correctly. Wiz's later PEACH framework grew out of exactly this class of finding: the more capability a managed service offers inside the tenant boundary, the more surface there is for that boundary to fail on.

Feature: Embedded Jupyter Notebooks, added to Cosmos DB in 2019
Enabled: By default for Cosmos DB accounts
What it changes: The provider must now contain a general-purpose runtime per tenant
Customer visibility: None into how that containment is implemented
Later formalized as: Wiz's PEACH tenant isolation framework
Jupyter NotebooksManaged Service RuntimeTenant Isolation
⬆️ Privilege Escalation - Out of the Notebook Container
02
A local privilege escalation inside the notebook environment, followed by rewriting firewall rules, gave unrestricted network access to the Cosmos DB backend
T1611 - Escape to Host T1562.007 - Disable or Modify Cloud Firewall

The chain ran from inside the customer's own notebook: escalate locally within that environment, then modify the firewall rules constraining it, and the network restrictions that were supposed to keep a tenant's notebook talking only to that tenant's data no longer applied. From there the backend was reachable directly. Each individual step is a familiar container-security problem - a local escalation, a network control that could be changed from inside the thing it was containing - but the consequence is not, because the blast radius of a container escape in a multi-tenant managed service is every other tenant on it.

Step 1: Local privilege escalation within the notebook environment
Step 2: Modify firewall rules to remove network restrictions
Step 3: Reach and authenticate to the Cosmos DB backend directly
Why it is different from a normal escape: The blast radius is every other tenant
Control that failed: A network boundary changeable from inside the thing it contained
Container EscapeFirewall ModificationBackend AccessT1611
🔑 Credential Access - Other Tenants' Primary Keys
03
From the backend the researchers retrieved and decrypted credentials belonging to other customers, obtaining full read, write and delete access to their databases
T1552 - Unsecured Credentials

The prize was the primary keys - Cosmos DB's long-lived, full-privilege credentials. With them, an attacker has complete administrative control of a victim's database from anywhere on the internet, independent of the vulnerability that produced them. Two consequences follow. First, the access is not bounded by the flaw's lifetime: a key obtained in August 2021 keeps working until it is rotated, which is why Microsoft's advice was to regenerate keys rather than simply to apply a fix. Second, because these keys are long-lived by design and rarely rotated in practice, the honest position for any customer was to assume exposure over an extended window rather than a moment.

Obtained: Other customers' Cosmos DB primary keys, retrieved and decrypted
Access granted: Full read, write and delete, from anywhere
Key property: Long-lived and rarely rotated, so the access outlives the vulnerability
Microsoft's advice: Regenerate primary keys - not merely wait for the fix
Honest customer position: Assume exposure across months, not a moment
Primary KeysCross-TenantLong-Lived CredentialsT1552
🏢 Impact - Thousands of Tenants, and Nothing a Customer Could Have Done
04
Several thousand Azure customers were exposed, including many Fortune 500 companies, with no customer-side configuration that would have prevented it

This is the point of including a research finding on a page of breaches. Every customer-side control the industry recommends was irrelevant here: network restrictions, least privilege, MFA, encryption at rest, private endpoints. None of them sit between one tenant's notebook and another tenant's keys, because that boundary is the provider's to enforce and the customer has no visibility into it. It maps directly onto the shared responsibility model - isolation is unambiguously the provider's side of the line - and it is the clearest available demonstration that the provider's side is not a formality. Microsoft disabled the feature within about 48 hours, which is a genuinely fast response, and notified affected customers.

Exposed: Several thousand Azure customers, many Fortune 500
Customer controls that would have helped: None
Whose responsibility: Tenant isolation is unambiguously the provider's
Microsoft response: Feature disabled within ~48 hours; affected customers notified
The uncomfortable part: You cannot audit, test or compensate for this boundary
Multi-TenancyShared ResponsibilityProvider-Side Only

🛡 How to Defend Against This Chain

Rotate long-lived service keys on a schedule, because that is the one lever you hold. You could not have prevented ChaosDB, but you control how long a stolen key stays useful. Cosmos DB primary keys, storage account keys and their equivalents elsewhere should be rotated routinely and automatically, and the rotation should be tested - most organizations discover during an incident that something breaks when a key changes.
Prefer identity-based access over long-lived keys wherever the service allows it. Cosmos DB supports Entra ID authentication and role-based access control; storage accounts support the same. Identity-based access can be revoked centrally, is subject to conditional access, and produces better logs. A long-lived key is a bearer token with no context and no expiry, and this incident is what that costs.
Disable managed-service features you do not use. The vulnerable surface was a notebook feature many customers had never touched. Every optional capability a managed service offers is attack surface you inherit by default, so review what is enabled on your data services and turn off what you are not using - the smallest version of the service is the safest version.
Read your provider's isolation posture as part of vendor assessment, not as a given. Ask how tenant isolation is implemented for the specific services you rely on, whether the provider publishes research or attestations about it, and how they have handled past isolation findings. Frameworks like Wiz's PEACH exist to make that a structured conversation rather than a vague one.
Keep your own data-plane logs, so "were we accessed" is answerable by you. If the provider's control plane is what failed, provider-side assurances are exactly what you cannot rely on. Stream Cosmos DB and equivalent diagnostic logs into your own SIEM with independent retention, and baseline normal access so anomalous reads stand out even when the credential used was technically valid.
Include "the provider had a cross-tenant flaw" in your risk register and your IR plan. It is rare and it is not zero. Decide in advance what you would do on such an advisory: which keys you rotate, in what order, how fast, and what you tell customers. The organizations that handled ChaosDB well were the ones that could rotate quickly, and that is a capability built in advance.

Related defense topics