Logging & Telemetry

Which plane the log lives on

Cloud logging arguments go sideways because people use "the logs" for five different things. Name the plane before you name the product.

• Control plane. Who called the cloud API, from where, and whether it succeeded. AWS CloudTrail management events, Azure Activity Log, GCP Admin Activity audit logs. This is the record of the account itself changing.
• Identity. Who signed in, with what factor, and which application they consented to. The IdP (Entra ID, Okta, Google Workspace) knows this before the cloud account does. CloudTrail records the use of a credential. The IdP records the issuance of it.
• Data plane. What happened to the data after the API allowed it. S3 data events, GCS Data Access logs, Azure Storage diagnostics, database audit logs. High volume, high value, easy to turn on "for everything" and regret.
• Network. VPC flow logs, DNS query logs, load balancer access logs, firewall logs. Useful when you are reconstructing a path. Expensive and low-signal as a standing detection source if you ingest every byte into a SIEM.
• Workload. Process, file, and container runtime telemetry from an agent or from the orchestrator. This is not the same thing as application stdout. A container printing "request received" is an application log. It is rarely a security log, and it is almost never the right place to look first.

GuardDuty, Defender for Cloud, and Security Command Center are detectors. They read some of the logs above and emit findings. If you only keep the finding and drop the underlying event, you cannot answer "what else did this identity do in the hour before." Keep the source log. Treat the detector as a second opinion.

Control plane, first

If you do nothing else, do this. A trail, a diagnostic setting, or an audit-log sink that covers every account in the organization, lands in a logging account the application teams cannot administer, and cannot be stopped by someone who has compromised a single workload.

On AWS that is an organization CloudTrail, written to a bucket in a separate account, with the trail's own StopLogging and the bucket's DeleteObject outside the permissions of the accounts being watched. On Azure it is Activity Log diagnostic settings at the subscription, preferably collected by an Azure Policy initiative so a new subscription does not arrive silent. On GCP it is Admin Activity audit logs, which are on by default, exported from the organization to a sink the project owners do not control. Data Access audit logs are not on by default. That distinction matters, and it is the next section's problem, not this one's.

Two properties matter more than the brand of the log store. The log must exist before you need it, and the person who can change the resource must not be the person who can erase the record of that change. Object lock, a retention policy, or a bucket policy that denies delete from the source account is the control. A dashboard that "monitors whether logging is on" is a consolation prize for not having the control.

Identity, second

Control-plane logs tell you an access key was used. They do not tell you that the human behind the federated role had their session stolen at the IdP an hour earlier, or that someone consented an OAuth application with mail-read and then never signed into the cloud at all. IAM abuse that never touches the cloud API is invisible to CloudTrail.

Collect, at the IdP: interactive and non-interactive sign-ins, MFA challenges and their result, legacy-protocol authentication, new application consents, credential registration (a new FIDO key, a new authenticator), and role or group changes. Keep the cloud side too: AssumeRole, service-account key creation, managed-identity assignment, and any event that mints a long-lived credential. The pair is the story. Either half is a fragment.

Workload identity belongs in the same bucket of attention even though it is not a person. A new role trust that allows a GitHub repository, a Kubernetes service account, or an external OIDC issuer is a standing authentication decision. Log the change when the trust is created. You will not get a useful "sign-in" for it later, because the workload presents a token and the cloud says yes.

Data plane and network, on purpose

Data events answer "who read this object." You want that answer for the buckets, tables, and secrets that would hurt if they left. You do not want it for every build artifact, every temporary object, and every health-check GET. The failure mode is flipping data events on at the organization and discovering the bill, then flipping them off everywhere, including the one bucket that mattered.

Pick the crown jewels by name. Production data stores, the secrets manager, the logging bucket itself, and anything holding customer data or credentials. Sample or exclude the rest. Revisit the list when a new store is created, which is an argument for a tagging convention (data-class=restricted turns data events on) rather than a wiki page nobody updates.

Network logs are the same shape of decision. Flow logs are how you prove a path existed, or that it did not. Full-fidelity flow logs from every interface, ingested into a SIEM priced per gigabyte, are how a logging program gets cancelled. Keep flow logs for the sensitive subnets in the SIEM or in the detection pipeline. Keep the rest, sampled, in cheap object storage for the investigation you do not know you will have. DNS query logs are worth more per byte than flow logs for command-and-callback questions, and they are still not free. Turn them on where the workloads can make arbitrary outbound calls, not on a network that already denies egress.

What not to collect

A logging program fails as often by collecting the wrong thing as by collecting nothing.

• Secrets, tokens, and query strings. Application logs that print the Authorization header, the cloud metadata response, or the full URL with a signed parameter are a second copy of the credential, sitting in a system more people can read than the secret store. Redact at the source. Do not rely on the SIEM to do it later.
• Debug-level application logs as a security control. They are huge, they rotate fast, and the security question is almost never in them. If a service needs request logs for forensics, log the request id, the caller identity, the route, and the status. Leave the body out.
• Every flow record, as a detection. Use them to investigate. Alert on the control-plane change that opened the path (0.0.0.0/0 on an admin port, a new public IP on a private subnet), not on the million flows that followed.
• Duplicate copies of the same event in three tools. One durable copy in object storage, one hot copy of the fields the detections need. A third copy "just in case" is how retention math quietly triples.
• The detector's finding, and nothing else. Findings without the underlying event cannot be tuned, cannot be re-investigated when the detector was wrong, and disappear when the vendor changes the rule.

Retention and what it costs

There are two clocks, and they should not share a price.

Hot is the window you detect in. Days to a few weeks, in whatever system runs the rules. You pay for ingest and for query speed. Put the control plane and the identity logs here in full. Put a filtered slice of data and network logs here.

Cold is the window you investigate and comply in. Many programs land on a year because a framework, a contract, or an insurer asked for a year. Object storage in the logging account is the cheap place for that copy. You do not need to ask a SIEM a question about February in order to still have February. When you do need it, you replay the slice into the hot store or query it in place.

The number that surprises people is not storage. Storage of compressed logs is close to free at the scale of a single organization trail. The number is ingest into a tool that charges per gigabyte analyzed. Data events and flow logs dominate that number. If a vendor quote assumes "all logs," ask which plane. A quote that does not separate control plane from data plane is not a quote you can compare.

Retention also has an integrity problem. An attacker who can delete the trail has deleted the evidence of deleting the trail, unless the trail's storage refuses the delete. Set the retention on the bucket or the sink, not as a hope inside the SIEM. And log the logging: an event that stops the trail, deletes a sink, or changes a diagnostic setting is itself a control-plane event, and it belongs in the first-alert list.

The first events worth alerting on

Start with events that are rare, high impact, and hard for an attacker to avoid if they want to stay. Do not start with a content pack of four hundred rules. The detection engineering page is the lifecycle after you have a source. These are the sources' first questions.

• The organization trail, diagnostic setting, or audit sink was stopped, deleted, or pointed somewhere new.
• A long-lived credential was created: an access key, a service-account key, a client secret, a new root or break-glass use.
• A human signed in without MFA, or registered a new factor, or consented an application with broad scopes.
• A trust policy, workload identity binding, or federated issuer changed. Someone just decided who is allowed to become this role.
• A network path opened that was closed: 0.0.0.0/0 or a public IP on a port that speaks administration, or a bucket or storage account that became anonymous.
• The logging bucket, the secrets store, or a production data store was read by an identity that does not usually read it. This one needs the data-plane log, which is why the crown-jewel list exists.

Each of these fails closed in a useful way: if the log is missing, the absence is the alert, because you already decided the control plane is non-optional. A detection that only fires when a particular field is present will sit quiet through the week someone turned the trail off.

What you do with the alert is a different page. A finding that says "this identity can read production" is not, by itself, an incident. Working a finding is the four questions to ask before you open the ticket.