Time: ~2 hours · Difficulty: Beginner · You need: a Sumo Logic organisation that accepts your own data (step 1 explains which trials do), and a terminal with curl and jq.
How this page was checked: Sumo Logic runs only as a hosted service and this page was written without an account, so none of the Sumo Logic queries here were run. Every operator, option, limit and menu path was checked against Sumo Logic's documentation, on its live site on 27 September 2026 and in its open-source repository, and the results described are what those documented rules give for the lab file, worked out by a local script rather than captured from Sumo Logic. The shell steps were run as printed, in bash and zsh on macOS (curl 8.7.1, jq 1.8.2), against a local stand-in for the upload address. The monitor in step 9 passed terraform validate with Terraform 1.15.7 and Sumo Logic's provider 3.3.2 from the Terraform registry, and was never applied.
A search language that parses as it reads
Sumo Logic is a hosted log analytics and security platform. Its search query language is what you type into Log Search, and the same queries drive dashboards, scheduled searches and monitors, Sumo Logic's alerting feature. Cloud SIEM (security information and event management), Sumo Logic's security product, adds a separate rule language; step 11 shows how it differs. Analysts write these queries to investigate, and detection engineers turn the ones worth repeating into monitors. A query is plain text, so it can live in git and be reviewed like code, which step 9 does.
The one idea to hold on to: Sumo Logic stores each event as raw text plus a few metadata tags added when the data arrived. Named fields such as eventName or sourceIPAddress are pulled out of the text while your query runs, by a parse step you write (or at ingest, by a field extraction rule someone set up in advance). Until something parses a field, it does not exist, and most of the surprises on this page trace back to that one fact.
On this page
Scope first, then a pipe of operators
A query has two parts. The scope comes first: keywords and metadata that choose which raw messages to read. After it come operators, each introduced by a pipe character, |, and each working on whatever the previous one handed it: parse, filter, count, sort. Keywords match whole words in the raw text and ignore case. _sourceCategory is metadata attached when the data was sent; step 3 sets it to lab/aws/cloudtrail.
_sourceCategory=lab/aws/cloudtrail StopLogging
That returns the one raw event containing the word StopLogging, with no fields yet. Now ask for a field with where, which keeps only the messages for which a comparison is true, and see what happens:
_sourceCategory=lab/aws/cloudtrail | where eventName = "StopLogging"
Log Search has two search modes. In Manual mode this stops with an error of the form Field eventName not found, please check the spelling and try again: nothing has created eventName. In Auto Parse mode (a toggle under the gear icon), which extracts JSON (JavaScript Object Notation) fields for you, the same query works. The queries on this page parse explicitly instead, so they behave the same in either mode and when saved or scheduled. json reads the key eventName from each message and creates a field of that name, so where has something to compare:
_sourceCategory=lab/aws/cloudtrail | json "eventName" | where eventName = "StopLogging"
The parse step filters too. By default json drops every message that lacks the key you asked for. Keys are case-sensitive, so json "sourceIpAddress" against CloudTrail, which spells it sourceIPAddress, drops all 24 events. Log Search shows a warning that messages lacked the key, but the result is the same empty table a quiet account gives, and a monitor, which counts result rows, sees only the empty table. Add nodrop to keep messages that lack a key, with the field left empty.
Walk-through
The lab is 24 Amazon Web Services (AWS) CloudTrail events. A continuous integration user's access key, build-bot, has leaked. From 203.0.113.50 someone checks who the key belongs to, tries to list Identity and Access Management (IAM) users, Amazon Elastic Compute Cloud (EC2) resources and secrets, and is mostly refused. They then create a user, backup-svc, give it administrator rights and an access key, and use it to stop CloudTrail logging. Around that sits ordinary activity, including a console login without multi-factor authentication (MFA) and an application repeatedly denied one file. The walk-through builds five detections from it:
- D1, trail tampering: CloudTrail logging stopped, deleted or changed.
- D2, denied-call burst: one source address refused five or more times.
- D3, new user given a key: a user created and handed an access key within 10 minutes.
- D4, console login without MFA: a successful console sign-in without multi-factor authentication.
- D5, administrator policy attached:
AdministratorAccessattached to a user.
1. Get an organisation you can send data to
Sumo Logic runs only as a hosted service, so there is nothing to install. What matters is whether your organisation can receive your own data:
- The 14-day Agentic Free Trial from the Sumo Logic website runs in a shared environment on preloaded example data and cannot ingest your own. You can practise searching there, but not load this lab.
- The 30-day trial through AWS Marketplace can ingest your data. It is not open to existing Sumo Logic customers, needs a business email address (not a personal one such as Gmail), asks you to accept Sumo Logic's service licence agreement, and at the end of the 30 days converts to a paid subscription, billed monthly through AWS for actual usage. Note the date, and cancel in AWS Marketplace under Manage subscriptions if you do not want that.
- A non-production organisation at work also works, if your role has the Manage Collectors and Manage Monitors capabilities.
Then set the interface to UTC (Coordinated Universal Time), so the times you see match this page: open your account settings from the person icon in the top menu (in the Classic interface, your username at the bottom of the main menu), and on the Preferences tab set Default Timezone to UTC.
2. Create an HTTP source
Data arrives through a source. This lab uses an HTTP (Hypertext Transfer Protocol) source: a unique web address that stores whatever you upload to it. Open Data Management, then Collection under Data Collection (Classic interface: Manage Data > Collection > Collection). Click Add Collector, choose Hosted Collector, give it a name and save it. Then click Add Source next to it, choose HTTP Logs & Metrics, set Name to cloudtrail-lab and Source Category to lab/aws/cloudtrail, and under the advanced options for logs set:
- Timestamp Format: Specify a format, then + Add Timestamp Format with format
yyyy-MM-dd'T'HH:mm:ss'Z'and timestamp locator"eventTime":"(.*?)". - Default Time Zone: UTC. The format treats the trailing
Zas a plain letter, so the zone comes from this setting. - Multiline Processing: off. Each line of the lab file is one complete event.
Save, choose Presigned URL, and copy the address. Treat it like a password: anyone who has it can send data into your account.
3. Load the lab data and check what arrived
mkdir -p ~/sumo-lab && cd ~/sumo-lab
cat > cloudtrail-lab.jsonl <<'EOF'
{"eventTime":"2026-09-15T08:55:12Z","eventSource":"signin.amazonaws.com","eventName":"ConsoleLogin","awsRegion":"us-east-1","sourceIPAddress":"198.51.100.20","userAgent":"Mozilla/5.0","userIdentity":{"type":"IAMUser","arn":"arn:aws:iam::111122223333:user/alice","userName":"alice"},"responseElements":{"ConsoleLogin":"Success"},"additionalEventData":{"MFAUsed":"Yes"}}
{"eventTime":"2026-09-15T09:02:40Z","eventSource":"iam.amazonaws.com","eventName":"CreateUser","awsRegion":"us-east-1","sourceIPAddress":"198.51.100.20","userAgent":"AWS Internal","userIdentity":{"type":"IAMUser","arn":"arn:aws:iam::111122223333:user/alice","userName":"alice"},"requestParameters":{"userName":"svc-reporting"},"readOnly":false}
{"eventTime":"2026-09-15T09:03:05Z","eventSource":"s3.amazonaws.com","eventName":"ListBuckets","awsRegion":"us-east-1","sourceIPAddress":"198.51.100.20","userAgent":"AWS Internal","userIdentity":{"type":"IAMUser","arn":"arn:aws:iam::111122223333:user/alice","userName":"alice"},"readOnly":true}
{"eventTime":"2026-09-15T09:10:22Z","eventSource":"ec2.amazonaws.com","eventName":"DescribeInstances","awsRegion":"us-east-1","sourceIPAddress":"192.0.2.10","userAgent":"aws-cli/2.17.40","userIdentity":{"type":"AssumedRole","arn":"arn:aws:sts::111122223333:assumed-role/ci-deploy/runner-4821"},"readOnly":true}
{"eventTime":"2026-09-15T09:15:47Z","eventSource":"signin.amazonaws.com","eventName":"ConsoleLogin","awsRegion":"us-east-1","sourceIPAddress":"198.51.100.21","userAgent":"Mozilla/5.0","userIdentity":{"type":"IAMUser","arn":"arn:aws:iam::111122223333:user/bob","userName":"bob"},"responseElements":{"ConsoleLogin":"Success"},"additionalEventData":{"MFAUsed":"No"}}
{"eventTime":"2026-09-15T09:20:03Z","eventSource":"signin.amazonaws.com","eventName":"ConsoleLogin","awsRegion":"us-east-1","sourceIPAddress":"192.0.2.77","userAgent":"Mozilla/5.0","userIdentity":{"type":"IAMUser","arn":"arn:aws:iam::111122223333:user/alice","userName":"alice"},"errorMessage":"Failed authentication","responseElements":{"ConsoleLogin":"Failure"},"additionalEventData":{"MFAUsed":"No"}}
{"eventTime":"2026-09-15T09:31:10Z","eventSource":"s3.amazonaws.com","eventName":"GetObject","awsRegion":"us-east-1","sourceIPAddress":"192.0.2.20","userAgent":"Boto3/1.35.10","userIdentity":{"type":"AssumedRole","arn":"arn:aws:sts::111122223333:assumed-role/reports-app/i-0a12b34c56d78e90f"},"errorCode":"AccessDenied","requestParameters":{"bucketName":"finance-reports","key":"2026/09/summary.csv"},"readOnly":true}
{"eventTime":"2026-09-15T09:47:55Z","eventSource":"s3.amazonaws.com","eventName":"GetObject","awsRegion":"us-east-1","sourceIPAddress":"192.0.2.20","userAgent":"Boto3/1.35.10","userIdentity":{"type":"AssumedRole","arn":"arn:aws:sts::111122223333:assumed-role/reports-app/i-0a12b34c56d78e90f"},"errorCode":"AccessDenied","requestParameters":{"bucketName":"finance-reports","key":"2026/09/summary.csv"},"readOnly":true}
{"eventTime":"2026-09-15T10:01:30Z","eventSource":"s3.amazonaws.com","eventName":"GetObject","awsRegion":"us-east-1","sourceIPAddress":"192.0.2.20","userAgent":"Boto3/1.35.10","userIdentity":{"type":"AssumedRole","arn":"arn:aws:sts::111122223333:assumed-role/reports-app/i-0a12b34c56d78e90f"},"errorCode":"AccessDenied","requestParameters":{"bucketName":"finance-reports","key":"2026/09/summary.csv"},"readOnly":true}
{"eventTime":"2026-09-15T10:02:05Z","eventSource":"sts.amazonaws.com","eventName":"GetCallerIdentity","awsRegion":"us-east-1","sourceIPAddress":"203.0.113.50","userAgent":"aws-cli/2.17.40","userIdentity":{"type":"IAMUser","arn":"arn:aws:iam::111122223333:user/build-bot","userName":"build-bot"},"readOnly":true}
{"eventTime":"2026-09-15T10:02:31Z","eventSource":"iam.amazonaws.com","eventName":"ListUsers","awsRegion":"us-east-1","sourceIPAddress":"203.0.113.50","userAgent":"aws-cli/2.17.40","userIdentity":{"type":"IAMUser","arn":"arn:aws:iam::111122223333:user/build-bot","userName":"build-bot"},"errorCode":"AccessDenied","readOnly":true}
{"eventTime":"2026-09-15T10:02:38Z","eventSource":"iam.amazonaws.com","eventName":"ListRoles","awsRegion":"us-east-1","sourceIPAddress":"203.0.113.50","userAgent":"aws-cli/2.17.40","userIdentity":{"type":"IAMUser","arn":"arn:aws:iam::111122223333:user/build-bot","userName":"build-bot"},"errorCode":"AccessDenied","readOnly":true}
{"eventTime":"2026-09-15T10:02:52Z","eventSource":"iam.amazonaws.com","eventName":"GetAccountAuthorizationDetails","awsRegion":"us-east-1","sourceIPAddress":"203.0.113.50","userAgent":"aws-cli/2.17.40","userIdentity":{"type":"IAMUser","arn":"arn:aws:iam::111122223333:user/build-bot","userName":"build-bot"},"errorCode":"AccessDenied","readOnly":true}
{"eventTime":"2026-09-15T10:03:05Z","eventSource":"ec2.amazonaws.com","eventName":"DescribeInstances","awsRegion":"us-east-1","sourceIPAddress":"203.0.113.50","userAgent":"aws-cli/2.17.40","userIdentity":{"type":"IAMUser","arn":"arn:aws:iam::111122223333:user/build-bot","userName":"build-bot"},"errorCode":"Client.UnauthorizedOperation","readOnly":true}
{"eventTime":"2026-09-15T10:03:09Z","eventSource":"ec2.amazonaws.com","eventName":"DescribeVpcs","awsRegion":"us-east-1","sourceIPAddress":"203.0.113.50","userAgent":"aws-cli/2.17.40","userIdentity":{"type":"IAMUser","arn":"arn:aws:iam::111122223333:user/build-bot","userName":"build-bot"},"errorCode":"Client.UnauthorizedOperation","readOnly":true}
{"eventTime":"2026-09-15T10:03:14Z","eventSource":"ec2.amazonaws.com","eventName":"DescribeSecurityGroups","awsRegion":"us-east-1","sourceIPAddress":"203.0.113.50","userAgent":"aws-cli/2.17.40","userIdentity":{"type":"IAMUser","arn":"arn:aws:iam::111122223333:user/build-bot","userName":"build-bot"},"errorCode":"Client.UnauthorizedOperation","readOnly":true}
{"eventTime":"2026-09-15T10:03:22Z","eventSource":"secretsmanager.amazonaws.com","eventName":"ListSecrets","awsRegion":"us-east-1","sourceIPAddress":"203.0.113.50","userAgent":"aws-cli/2.17.40","userIdentity":{"type":"IAMUser","arn":"arn:aws:iam::111122223333:user/build-bot","userName":"build-bot"},"errorCode":"AccessDenied","readOnly":true}
{"eventTime":"2026-09-15T10:03:40Z","eventSource":"s3.amazonaws.com","eventName":"ListBuckets","awsRegion":"us-east-1","sourceIPAddress":"203.0.113.50","userAgent":"aws-cli/2.17.40","userIdentity":{"type":"IAMUser","arn":"arn:aws:iam::111122223333:user/build-bot","userName":"build-bot"},"readOnly":true}
{"eventTime":"2026-09-15T10:06:12Z","eventSource":"iam.amazonaws.com","eventName":"CreateUser","awsRegion":"us-east-1","sourceIPAddress":"203.0.113.50","userAgent":"aws-cli/2.17.40","userIdentity":{"type":"IAMUser","arn":"arn:aws:iam::111122223333:user/build-bot","userName":"build-bot"},"requestParameters":{"userName":"backup-svc"},"readOnly":false}
{"eventTime":"2026-09-15T10:06:30Z","eventSource":"iam.amazonaws.com","eventName":"AttachUserPolicy","awsRegion":"us-east-1","sourceIPAddress":"203.0.113.50","userAgent":"aws-cli/2.17.40","userIdentity":{"type":"IAMUser","arn":"arn:aws:iam::111122223333:user/build-bot","userName":"build-bot"},"requestParameters":{"userName":"backup-svc","policyArn":"arn:aws:iam::aws:policy/AdministratorAccess"},"readOnly":false}
{"eventTime":"2026-09-15T10:06:51Z","eventSource":"iam.amazonaws.com","eventName":"CreateAccessKey","awsRegion":"us-east-1","sourceIPAddress":"203.0.113.50","userAgent":"aws-cli/2.17.40","userIdentity":{"type":"IAMUser","arn":"arn:aws:iam::111122223333:user/build-bot","userName":"build-bot"},"requestParameters":{"userName":"backup-svc"},"responseElements":{"accessKey":{"userName":"backup-svc","accessKeyId":"AKIAIOSFODNN7EXAMPLE","status":"Active"}},"readOnly":false}
{"eventTime":"2026-09-15T10:09:03Z","eventSource":"cloudtrail.amazonaws.com","eventName":"StopLogging","awsRegion":"us-east-1","sourceIPAddress":"203.0.113.50","userAgent":"aws-cli/2.17.40","userIdentity":{"type":"IAMUser","arn":"arn:aws:iam::111122223333:user/backup-svc","userName":"backup-svc"},"requestParameters":{"name":"arn:aws:cloudtrail:us-east-1:111122223333:trail/org-trail"},"readOnly":false}
{"eventTime":"2026-09-15T10:40:18Z","eventSource":"ec2.amazonaws.com","eventName":"DescribeInstances","awsRegion":"us-east-1","sourceIPAddress":"192.0.2.10","userAgent":"aws-cli/2.17.40","userIdentity":{"type":"AssumedRole","arn":"arn:aws:sts::111122223333:assumed-role/ci-deploy/runner-4821"},"readOnly":true}
{"eventTime":"2026-09-15T11:40:26Z","eventSource":"iam.amazonaws.com","eventName":"CreateAccessKey","awsRegion":"us-east-1","sourceIPAddress":"198.51.100.20","userAgent":"AWS Internal","userIdentity":{"type":"IAMUser","arn":"arn:aws:iam::111122223333:user/alice","userName":"alice"},"requestParameters":{"userName":"svc-reporting"},"responseElements":{"accessKey":{"userName":"svc-reporting","accessKeyId":"AKIAI44QH8DHBEXAMPLE","status":"Active"}},"readOnly":false}
EOF
shasum -a 256 cloudtrail-lab.jsonl
jq -c '(now - 86400 | strftime("%Y-%m-%d")) as $d | .eventTime |= ($d + .[10:])' \
cloudtrail-lab.jsonl > upload.jsonl
SUMO_URL='https://paste-your-presigned-url-here'
curl -sS -o /dev/null -w '%{http_code}\n' -X POST \
-H 'X-Sumo-Category:lab/aws/cloudtrail' -T upload.jsonl "$SUMO_URL"The cat line writes everything up to EOF into cloudtrail-lab.jsonl, and shasum prints the file's hash so you can check the copy is exact. Expect the hash 35b7c80582d0650ec4850acf84f84fa4b8e5d37169ac3f7276c76f29b67c3319, then 200. The jq line moves the lab to yesterday (the warning below explains why); only the date changes, so every time of day, and every gap between events, stays the same. Use -T, as Sumo Logic's own examples do: -d strips the newlines, and all 24 events arrive as one line. The X-Sumo-Category header sets _sourceCategory for everything in the request, overriding the source's own setting.
A 200 only means the request was received (if an ingest budget, a cap on how much data your organisation takes in, has stopped collection, the source discards the data and still answers 200), so count what arrived, and how each event's time was set. In Log Search, set the time range to -2d (widen it if you come back later), because the default, Last 15 Minutes, misses yesterday:
_sourceCategory=lab/aws/cloudtrail | _format as ts_format | count by ts_format
The counts should add up to 24: fewer means lines were merged or dropped, 48 means you uploaded twice. _format records how each timestamp was found, and a value starting t:fail, t:none, t:ac1 or t:ac2 means the event's own time was not used (not found, not parsed, or auto-corrected), which would make every time window on this page wrong.
Old timestamps are replaced without an error. Sumo Logic expects log times to be recent. Events more than 30 days old need a timestamp format and locator on the source; events more than a year old get the current time unless Sumo Logic Support changes that for your organisation; and a message more than a day away from the source's recent messages is auto-corrected unless the source has a custom timestamp format. The events still arrive and count correctly, only their times are wrong, and that silently breaks every time window. Hence the move to yesterday, and the explicit format in step 2.
4. Filter on a parsed field: trail tampering (D1)
// D1: CloudTrail logging stopped, deleted or changed
_sourceCategory=lab/aws/cloudtrail
| json "eventName", "sourceIPAddress", "userIdentity.userName" as eventName, sourceIPAddress, userName nodrop
| where eventName in ("StopLogging", "DeleteTrail", "UpdateTrail", "PutEventSelectors")
| fields eventName, userName, sourceIPAddressExpect one message, at 10:09:03: StopLogging by backup-svc from 203.0.113.50. The other 23 events are the negative case, and none appears.
A dot in a key means "go inside": userIdentity.userName is the key userName inside the object userIdentity. as names the extracted fields in order. nodrop is there because not every event has userIdentity.userName (roles do not), and without it those events would vanish before where saw them. in tests a value against a list, and fields picks what to display. One trap: keywords in the scope ignore case, but string comparisons in where do not, so "stoplogging" would match nothing.
5. Nested fields and the naive rule: console logins without MFA (D4)
The login result and the MFA flag sit one level down, in responseElements.ConsoleLogin and additionalEventData.MFAUsed, the same paths Sumo Logic's PCI (Payment Card Industry) Compliance for AWS CloudTrail app parses. Start with the obvious rule:
// D4, naive: console logins that did not use MFA _sourceCategory=lab/aws/cloudtrail | json "eventName", "responseElements.ConsoleLogin", "additionalEventData.MFAUsed", "userIdentity.userName", "sourceIPAddress" as eventName, loginResult, mfaUsed, userName, sourceIPAddress nodrop | where eventName = "ConsoleLogin" and mfaUsed = "No" | fields userName, sourceIPAddress, loginResult, mfaUsed
Expect two messages: bob at 09:15:47 from 198.51.100.21, and alice at 09:20:03 from 192.0.2.77 with loginResult Failure. The second is someone failing to sign in as alice: nobody got in, and the failed login carries MFAUsed No as well. The detection needs the result too:
// D4: successful console logins without MFA _sourceCategory=lab/aws/cloudtrail | json "eventName", "responseElements.ConsoleLogin", "additionalEventData.MFAUsed", "userIdentity.userName", "sourceIPAddress" as eventName, loginResult, mfaUsed, userName, sourceIPAddress nodrop | where eventName = "ConsoleLogin" and loginResult = "Success" and mfaUsed = "No" | fields userName, sourceIPAddress, loginResult, mfaUsed
Expect bob only. The negative cases are alice's 08:55:12 login, which used MFA, and the failed attempt the naive rule reported.
6. Count, then threshold: the burst that never fires (D2)
// D2, naive: AccessDenied per source address, 5 or more _sourceCategory=lab/aws/cloudtrail | json "errorCode", "sourceIPAddress" nodrop | where errorCode = "AccessDenied" | count by sourceIPAddress | where _count >= 5
count by makes one row per source address, with the number of its events in a field called _count, and the last where keeps rows of 5 or more. No results. Delete the last line and you see why: 203.0.113.50 has 4 and 192.0.2.20 has 3, so nobody reaches 5. Did nothing happen, or can the query not see what happened? Run the control: drop the filter and ask which error codes exist at all.
// Control: which error codes are in the data at all? _sourceCategory=lab/aws/cloudtrail | json "errorCode" nodrop | if(isBlank(errorCode), "(none)", errorCode) as errorCode | count by errorCode
Expect (none) 14, AccessDenied 7 and Client.UnauthorizedOperation 3 (the if gives events without a code a visible label). EC2 reports a refusal as Client.UnauthorizedOperation, so three of the attacker's seven denials were invisible to the rule; Sumo Logic's own PCI app alert for excessive failed API (application programming interface) calls counts both codes. Other AWS services use AccessDeniedException, and UnauthorizedOperation also appears without the prefix; AWS's own Security Hub control for unauthorized API calls matches both families, AccessDenied* and *UnauthorizedOperation. So the fix lists all four:
// D2: every denial code, per source address, 5 or more
_sourceCategory=lab/aws/cloudtrail
| json "errorCode", "sourceIPAddress" nodrop
| where errorCode in ("AccessDenied", "AccessDeniedException", "Client.UnauthorizedOperation", "UnauthorizedOperation")
| count by sourceIPAddress
| where _count >= 5Expect one row: 203.0.113.50 with 7. Without the last line, 192.0.2.20 shows 3 and stays under the threshold, which is the negative case.
A query that returns nothing and a query that cannot work return the same thing. Before you trust an empty result, run a control that must return something: the same query without its threshold, or a count of the values that actually exist. The nodrop in the control matters too: without it, json would have dropped the 14 events that have no errorCode, and you would never have seen them.
7. Time buckets
timeslice cuts time into fixed buckets and puts each event's bucket start in a field called _timeslice, which count can then group by:
// D2 in 10-minute buckets
_sourceCategory=lab/aws/cloudtrail
| json "errorCode", "sourceIPAddress" nodrop
| where errorCode in ("AccessDenied", "AccessDeniedException", "Client.UnauthorizedOperation", "UnauthorizedOperation")
| timeslice 10m
| count by _timeslice, sourceIPAddressExpect 192.0.2.20 once each in the 09:30, 09:40 and 10:00 buckets, and 203.0.113.50 seven times in 10:00. Buckets start on the clock (10:00, 10:10), not at the first event, so a burst straddling 10:10 would be split in two and could stay under the threshold in both halves. The monitor in step 9 avoids that, because its window moves forward at every evaluation.
8. One user, two events, ten minutes (D3)
"A new user was given an access key within 10 minutes" is about two events. transaction groups events by a key, counts how many fall into each state you name, and records the first and last time in each group as _start_time and _end_time, in milliseconds:
// D3: a new user given an access key within 10 minutes
_sourceCategory=lab/aws/cloudtrail
| json "eventName", "requestParameters.userName" as eventName, newUser nodrop
| where eventName in ("CreateUser", "CreateAccessKey")
| transaction on newUser
with states CreateUser as created, CreateAccessKey as keyed in eventName
results by transactions
| where created > 0 and keyed > 0
| _end_time - _start_time as gap_ms
| where gap_ms <= 10m
| gap_ms / 1000 as gap_secondsExpect backup-svc only, with gap_seconds 39. Now run the control: delete the last three lines. Two rows come back, backup-svc (10:06:12 to 10:06:51) and svc-reporting (09:02:40 to 11:40:26), alice's routine work, keyed 2 hours 37 minutes after she created the user. The window is what keeps her out. 10m is a time literal that Sumo Logic reads as 600,000 milliseconds, the same unit as the transaction times.
Two notes on the choice. join also relates two sets of events and accepts a time window, but its documentation warns that results can be incomplete and differ from run to run, which you do not want in a detection. And the span does not check which event came first; if order matters, results by flow reports each transition between states with its latency.
9. Turn the search into a monitor
Monitors are Sumo Logic's alerting feature, which its documentation sets apart from scheduled searches by automatic resolution and notification to several channels. You can build one under Monitoring > Monitors > Add > New Monitor (Classic interface: Manage Data > Monitoring > Monitors), or as code with Sumo Logic's provider for Terraform, a tool that creates resources from text files, which gives you a file to review and version like any other detection. This is D2:
resource "sumologic_monitor" "cloudtrail_denied_burst" {
name = "CloudTrail denied API calls from one source"
description = "5 or more denied AWS API calls from one source address in 10 minutes, counting every denial code."
monitor_type = "Logs"
tags = { "mitre_tactic" = "TA0007", "mitre_techniques" = "T1087.004 T1580" }
queries {
row_id = "A"
query = <<-EOT
_sourceCategory=lab/aws/cloudtrail
| json "errorCode", "sourceIPAddress" nodrop
| where errorCode in ("AccessDenied", "AccessDeniedException", "Client.UnauthorizedOperation", "UnauthorizedOperation")
| count by sourceIPAddress
| where _count >= 5
EOT
}
trigger_conditions {
logs_static_condition {
critical {
time_range = "10m" # look at the last 10 minutes,
frequency = "1m" # every minute
alert {
threshold = 0
threshold_type = "GreaterThan" # any row at all
}
resolution {
threshold = 0
threshold_type = "LessThanOrEqual"
}
}
}
}
notification_group_fields = ["sourceIPAddress"] # one alert per address
}The threshold lives in the query and the trigger fires on any row, the pattern Sumo Logic's documentation suggests for scheduled searches too; it also means the query alone, run in Log Search, tells you exactly what would alert. With no field set, the trigger counts rows. In the web interface the same settings are Monitor Type Logs, Detection Method Static, a trigger that alerts when the result is greater than 0 within 10 minutes and is evaluated every minute, and alert grouping by sourceIPAddress. Add a notifications block (the Notifications step in the web interface) to send each alert to email, a webhook or another connection; it is left out here for length. Log monitors run two minutes behind real time, and search with their creator's role search filter, which limits the data that role can see: a creator who cannot see CloudTrail gets a monitor that never fires.
The tags are free-form key-value pairs whose values may not contain commas, so the two techniques share one value, separated by a space. They carry MITRE ATT&CK (Adversarial Tactics, Techniques, and Common Knowledge) identifiers as numbered in ATT&CK v19.2: T1087.004 Account Discovery: Cloud Account and T1580 Cloud Infrastructure Discovery, both under the Discovery tactic (TA0007). Tag the others the same way: D1 with T1685.002 Disable or Modify Cloud Log, under the Defense Impairment tactic (TA0112); D3 with T1136.003 and T1098.001; D4 with T1078.004; D5 with T1098.003.
10. Test the monitor both ways
This monitor only looks at the last 10 minutes, and the lab is dated yesterday, so replay the relevant events stamped with the current time. First the negative case, the reports-app role's three denials, which must leave the monitor Normal:
jq -c --arg ip 192.0.2.20 '(now | strftime("%Y-%m-%dT%H:%M:%SZ")) as $t
| select(.sourceIPAddress == $ip and has("errorCode")) | .eventTime = $t' \
cloudtrail-lab.jsonl > replay.jsonl
curl -sS -o /dev/null -w '%{http_code}\n' -X POST \
-H 'X-Sumo-Category:lab/aws/cloudtrail' -T replay.jsonl "$SUMO_URL"Give it five minutes and confirm the monitor is still Normal. Then run the same two commands with --arg ip 203.0.113.50, which replays the attacker's seven denials. Within a few minutes the monitor should go Critical, with one alert, for 203.0.113.50. For the full picture, run a second monitor with the naive query from step 6 through the same replay: it stays Normal, which is exactly how the bug hides in production. The replays stay in lab/aws/cloudtrail with today's date, so for the exercises below set the time range to the day of the lab upload only: two dates, such as 09/26/2026 09/27/2026 in your date format, mean midnight to midnight.
11. Cloud SIEM rules are a different language
A Cloud SIEM rule is not a search query. Cloud SIEM parses and normalises events into records with one shared set of field names, and its rules are expressions over those records. This is the expression of the built-in rule AWS CloudTrail - Logging Configuration Change Observed, from Sumo Logic's published catalogue of Cloud SIEM content:
metadata_vendor = 'Amazon AWS'
and metadata_product = 'CloudTrail'
and metadata_deviceEventId IN ('AwsApiCall-CreateTrail',
'AwsApiCall-UpdateTrail',
'AwsApiCall-DeleteTrail',
'AwsApiCall-StartLogging',
'AwsApiCall-StopLogging',
'AwsApiCall-DeleteLogGroup',
'AwsApiCall-DeleteLogStream',
'AwsApiCall-DeleteDestination')There is no scope, no pipe and no parse step, and the field names are Cloud SIEM's: metadata_deviceEventId holds the event ID that Cloud SIEM's CloudTrail parser builds from eventType and eventName, which is why StopLogging appears as AwsApiCall-StopLogging. Even quoting differs: this rule writes its strings in single quotes, which in a search query do not make a string at all. The catalogue tags this rule with T1685.002 and the Defense Impairment tactic, the same ATT&CK v19 mapping this page uses for D1. A search you have proved in Log Search does not paste into a Cloud SIEM rule; read the Cloud SIEM rules syntax reference before writing one.
Hands-on exercises
Write D5: find every time the AdministratorAccess managed policy was attached to a user, showing who attached it, to whom, and from where.
Show the answer
// D5: AdministratorAccess attached to a user _sourceCategory=lab/aws/cloudtrail | json "eventName", "requestParameters.policyArn", "requestParameters.userName", "userIdentity.userName", "sourceIPAddress" as eventName, policyArn, targetUser, actor, sourceIPAddress nodrop | where eventName = "AttachUserPolicy" and policyArn = "arn:aws:iam::aws:policy/AdministratorAccess" | fields actor, targetUser, policyArn, sourceIPAddress
Expect one message at 10:06:30: build-bot attached AdministratorAccess to backup-svc from 203.0.113.50. The aliases matter more than usual here, because both userIdentity and requestParameters contain a key called userName: one is the actor, the other the target, and as keeps them apart. For real coverage, also consider AttachRolePolicy, AttachGroupPolicy and inline policies written with PutUserPolicy, each of which grants the same power by another route.
This query returns nothing against the lab. Why, and what is the quickest way to prove it rather than guess?
_sourceCategory=lab/aws/cloudtrail | json "sourceIpAddress" as ip | count by ip
Show the answer
json keys are case-sensitive and CloudTrail spells the key sourceIPAddress. No event has sourceIpAddress, so json drops all 24 and there is nothing left to count.
_sourceCategory=lab/aws/cloudtrail | json "sourceIpAddress" as ip nodrop | if(isBlank(ip), "(missing)", ip) as ip | count by ip
With nodrop the events survive, and the query returns one row: (missing) with a count of 24. That single row proves the scope found the events and the key did not, which moves the problem from "is there data?" to "what is the key really called?". Then copy the key name from a message instead of typing it: right-click the key in the Messages tab and choose Parse selected key, which writes the json line for you.
A colleague speeds up the fixed D2 query by adding the keyword AccessDenied to its scope, _sourceCategory=lab/aws/cloudtrail AccessDenied, and leaves the rest alone. It returns nothing again. Explain, and suggest a keyword that is safe.
Show the answer
Keywords filter the raw text before the first pipe. The three EC2 denials contain Client.UnauthorizedOperation and never the word AccessDenied, so they are gone before where sees them, and 203.0.113.50 is back to 4. The naive rule has moved from the where into the scope, where it is even easier to miss.
A safe keyword is errorCode: only events with an error contain that word, so it narrows the search without deciding which codes count. Naming the key you parse in the scope is also what Sumo Logic's json documentation suggests, and its cheat sheet scopes a CloudTrail error query the same way.
Which source addresses were used by more than one identity? Use count_distinct, which counts different values rather than events. Then explain what goes wrong if you count userIdentity.userName instead of the identity's Amazon Resource Name (ARN).
Show the answer
// How many identities used each source address? _sourceCategory=lab/aws/cloudtrail | json "sourceIPAddress", "userIdentity.arn" as sourceIPAddress, arn | count_distinct(arn) as identities by sourceIPAddress | sort by identities
Expect six rows, 203.0.113.50 first with 2 (the leaked build-bot key and the backup-svc user it created) and the other five addresses with 1 each. sort by puts the largest value first unless you add asc. Every event has an ARN, so nothing is dropped.
Roles have no userName. Parse it without nodrop and the ci-deploy events from 192.0.2.10 and the reports-app events from 192.0.2.20 are dropped: both addresses vanish from a table that still looks complete, four rows instead of six. With nodrop they come back with 1 each, because count_distinct counts an empty value as a value. That is the right number for the wrong reason: an address used by one user and one role would show 2 while naming only one of them.
Common mistakes
- Believing an empty result. A dropped key, a mis-cased key, a narrow scope and the default 15-minute time range all return the same nothing as a quiet account. Remove the threshold, or add
nodrop, and see what comes back. - Leaving out
nodropwhen a key is not on every event.jsondrops the messages that lack it before anything else in the query runs. - Filtering in the scope by accident. Keywords before the first pipe are a filter.
AccessDeniedin the scope hides every EC2 denial. - Mixing up the two case rules. Keywords and metadata ignore case;
jsonkeys and string comparisons inwheredo not. - Depending on Auto Parse mode. It is a per-search setting, saved searches store their own search mode, and scheduled searches do not support it for non-aggregate queries. An explicit
jsonbehaves the same everywhere. - Loading old events and not checking
_format. Sumo Logic replaces implausible timestamps with the arrival time, silently, and every time window after that is wrong. - Treating a timeslice bucket as a sliding window. Buckets are clock-aligned, so a burst across a boundary is split in two. A monitor's window moves.
- Trusting a 200 from the HTTP source. It means received, not stored; an ingest budget can drop the data and still answer 200. Count what arrived.
Where next
- Splunk's Search Processing Language (SPL) and Kusto Query Language (KQL) run the same lab and the same five detections, a good way to see which ideas are Sumo Logic's and which are universal.
- Sigma and YARA for writing a detection once and converting it, and jq and JMESPath for the
jqused in the load steps. - The CloudTrail to SIEM lab for a real CloudTrail feed to point these queries at, and detection engineering for testing, tuning and retiring them.
- Sumo Logic's search query language reference and its guide to creating monitors.
- Sumo Logic's Cloud SIEM content repository, which publishes every built-in rule with its expression, and the documentation's own source, whose app pages carry many worked CloudTrail queries.