Get the Zoom link
Cloud Security Office Hours Banner

Splunk SPL, hands-on

Splunk's Search Processing Language (SPL) chooses events, then pipes them through commands that count, filter and reshape. Learn it from zero on a local Splunk Enterprise with a small CloudTrail lab, and learn to spot the search that returns nothing because it cannot work.

Jump to the walk-through All how-to guides

· · Vendor-neutral

Time: ~2 hours  ·  Difficulty: Beginner  ·  You need: Docker with about 3 GB of memory and 3 GB of disk to spare, and a terminal. No Splunk account and no cloud account.

How this page was checked: every command and search was run as printed, in order, against Splunk Enterprise 10.4.3 in Docker (image splunk/splunk:10.4.3, under Rosetta emulation on an Apple silicon Mac), by a script that copies each block from this page and compares the results with the output shown here. The public rule in step 9 was read from Splunk's repository and built with Splunk's own tool, contentctl-ng 1.1.1. Not run: Splunk Cloud Platform, Splunk Enterprise Security, the Splunk Add-on for AWS, and the Splunk Free licence.

The language of Splunk searches

SPL is how you ask questions of data in Splunk Enterprise, which you run yourself, and Splunk Cloud Platform, Splunk's hosted service. Splunk Enterprise Security, Splunk's security information and event management (SIEM) app, runs its detections as SPL searches too. Analysts in a security operations centre (SOC) type SPL to investigate; detection engineers save searches that run on a schedule and raise alerts.

The distinction that matters: Splunk keeps each event's raw text and works out fields when you search. A source type tells Splunk how to split the text into events, where each event's time is, and how to find the fields. When it is wrong, every search is quietly wrong, so this page checks the load before anything else.

Splunk also publishes SPL2, a newer version of the language, and the search bar in Splunk Enterprise 10.4 offers to convert a search to it. Saved searches, alerts and Splunk's public detection library, all used on this page, are SPL.

On this page

  1. A search is a pipeline
  2. Walk-through: one leaked key, five detections
  3. Hands-on exercises
  4. Common mistakes
  5. Where next

A search is a pipeline

The part before the first pipe character, |, is the base search: it chooses events. index= says where to look, and every term after it must match (AND is implied; write OR and NOT in capitals). field=value matches one field. After that, each | hands the results so far to one command, which reshapes them and passes them on. Once step 2 has loaded the lab, try:

index=cloudtrail readOnly=false
| stats count BY userIdentity.userName
userIdentity.userName count
--------------------- -----
alice                     2
backup-svc                1
build-bot                 3

Read it left to right: the base search keeps the six events that changed something (readOnly=false), and stats turns them into one row per identity. A command sees only what the command before it produced, so every prefix of a search is a search you can run. To debug one, delete commands from the end until the output stops surprising you.

Walk-through: one leaked key, five detections

1. Start Splunk Enterprise in Docker

Splunk publishes Splunk Enterprise as the splunk/splunk image, for x86-64 processors only. On an Apple silicon Mac, Docker Desktop runs it under Rosetta emulation when that option is on in its settings; this page was checked that way, and the container took about two minutes to start and one to restart. The two SPLUNK_ settings accept the licence and the Splunk General Terms for you, so read them first. The first line creates a throwaway admin password for this container only.

mkdir -p ~/splunk-lab && cd ~/splunk-lab
openssl rand -hex 12 > .splunk-password
docker run -d --name splunk-lab --platform linux/amd64 \
  -p 8000:8000 -p 8089:8089 \
  -e SPLUNK_START_ARGS=--accept-license \
  -e SPLUNK_GENERAL_TERMS=--accept-sgt-current-at-splunk-com \
  -e SPLUNK_PASSWORD="$(cat .splunk-password)" \
  splunk/splunk:10.4.3
until [ "$(docker inspect -f '{{.State.Health.Status}}' splunk-lab)" = healthy ]; do sleep 10; done

Port 8000 is Splunk Web, the browser interface. Port 8089 is the management port, where the command-line interface (CLI) and the REST API (Splunk's HTTP application programming interface) reach Splunk. A new install runs on a 60-day Enterprise Trial licence: 500 MB of new data a day, alerting included, which step 7 needs. Now save a helper that runs one search from the terminal and prints exactly the output shown on this page:

cat > spl <<'EOF'
#!/bin/sh
# spl: run the search on standard input against the lab. Drop only the CLI's
# certificate notice; keep errors, so a broken search never looks like an empty one.
docker exec -u splunk splunk-lab /opt/splunk/bin/splunk search "$(cat)" \
  -auth "admin:$(cat ~/splunk-lab/.splunk-password)" 2>&1 |
  grep -v '^WARNING: Server Certificate Hostname Validation is disabled'
EOF
chmod +x spl

To run a search, type ./spl, paste the search, press Enter, then Ctrl-D. Or use Splunk Web: open http://localhost:8000, sign in as admin with the password in .splunk-password, choose Search & Reporting, set the time range picker to All time (it starts at Last 24 hours, which shows none of the lab), and paste the search. Splunk Web shows the same rows as a table, in UTC (Coordinated Universal Time), the container's time zone. The first sign-in also tells you that sharing of product usage data with Splunk is on: to stop it, open Settings > Instrumentation, click the gear next to Usage Data and switch Usage Data off (licence and software version data have no switch).

2. Load the lab data, then check what arrived

The lab is 24 Amazon Web Services (AWS) CloudTrail events from one morning, trimmed to the fields this page uses but keeping CloudTrail's real field names and nesting. The access key of build-bot, a continuous integration (CI) user, has leaked. From 203.0.113.50 someone checks whose key it is, tries to list Identity and Access Management (IAM) users, Elastic Compute Cloud (EC2) instances and Secrets Manager secrets (and is mostly refused), lists the Simple Storage Service (S3) buckets, then creates a user backup-svc, attaches AdministratorAccess to it, gives it an access key and uses it to stop CloudTrail logging. Around that runs an ordinary morning: sign-ins, one of them without multi-factor authentication (MFA) and one failed; an administrator creating a service account; an application refused the same S3 object three times; a CI role describing instances. The page builds five detections from it:

cat > cloudtrail-lab.jsonl <<'EOF'
{"eventTime":"2026-09-15T08:55:12Z","eventSource":"signin.amazonaws.com","eventName":"ConsoleLogin","awsRegion":"us-east-1","sourceIPAddress":"198.51.100.20","userAgent":"Mozilla/5.0","userIdentity":{"type":"IAMUser","arn":"arn:aws:iam::111122223333:user/alice","userName":"alice"},"responseElements":{"ConsoleLogin":"Success"},"additionalEventData":{"MFAUsed":"Yes"}}
{"eventTime":"2026-09-15T09:02:40Z","eventSource":"iam.amazonaws.com","eventName":"CreateUser","awsRegion":"us-east-1","sourceIPAddress":"198.51.100.20","userAgent":"AWS Internal","userIdentity":{"type":"IAMUser","arn":"arn:aws:iam::111122223333:user/alice","userName":"alice"},"requestParameters":{"userName":"svc-reporting"},"readOnly":false}
{"eventTime":"2026-09-15T09:03:05Z","eventSource":"s3.amazonaws.com","eventName":"ListBuckets","awsRegion":"us-east-1","sourceIPAddress":"198.51.100.20","userAgent":"AWS Internal","userIdentity":{"type":"IAMUser","arn":"arn:aws:iam::111122223333:user/alice","userName":"alice"},"readOnly":true}
{"eventTime":"2026-09-15T09:10:22Z","eventSource":"ec2.amazonaws.com","eventName":"DescribeInstances","awsRegion":"us-east-1","sourceIPAddress":"192.0.2.10","userAgent":"aws-cli/2.17.40","userIdentity":{"type":"AssumedRole","arn":"arn:aws:sts::111122223333:assumed-role/ci-deploy/runner-4821"},"readOnly":true}
{"eventTime":"2026-09-15T09:15:47Z","eventSource":"signin.amazonaws.com","eventName":"ConsoleLogin","awsRegion":"us-east-1","sourceIPAddress":"198.51.100.21","userAgent":"Mozilla/5.0","userIdentity":{"type":"IAMUser","arn":"arn:aws:iam::111122223333:user/bob","userName":"bob"},"responseElements":{"ConsoleLogin":"Success"},"additionalEventData":{"MFAUsed":"No"}}
{"eventTime":"2026-09-15T09:20:03Z","eventSource":"signin.amazonaws.com","eventName":"ConsoleLogin","awsRegion":"us-east-1","sourceIPAddress":"192.0.2.77","userAgent":"Mozilla/5.0","userIdentity":{"type":"IAMUser","arn":"arn:aws:iam::111122223333:user/alice","userName":"alice"},"errorMessage":"Failed authentication","responseElements":{"ConsoleLogin":"Failure"},"additionalEventData":{"MFAUsed":"No"}}
{"eventTime":"2026-09-15T09:31:10Z","eventSource":"s3.amazonaws.com","eventName":"GetObject","awsRegion":"us-east-1","sourceIPAddress":"192.0.2.20","userAgent":"Boto3/1.35.10","userIdentity":{"type":"AssumedRole","arn":"arn:aws:sts::111122223333:assumed-role/reports-app/i-0a12b34c56d78e90f"},"errorCode":"AccessDenied","requestParameters":{"bucketName":"finance-reports","key":"2026/09/summary.csv"},"readOnly":true}
{"eventTime":"2026-09-15T09:47:55Z","eventSource":"s3.amazonaws.com","eventName":"GetObject","awsRegion":"us-east-1","sourceIPAddress":"192.0.2.20","userAgent":"Boto3/1.35.10","userIdentity":{"type":"AssumedRole","arn":"arn:aws:sts::111122223333:assumed-role/reports-app/i-0a12b34c56d78e90f"},"errorCode":"AccessDenied","requestParameters":{"bucketName":"finance-reports","key":"2026/09/summary.csv"},"readOnly":true}
{"eventTime":"2026-09-15T10:01:30Z","eventSource":"s3.amazonaws.com","eventName":"GetObject","awsRegion":"us-east-1","sourceIPAddress":"192.0.2.20","userAgent":"Boto3/1.35.10","userIdentity":{"type":"AssumedRole","arn":"arn:aws:sts::111122223333:assumed-role/reports-app/i-0a12b34c56d78e90f"},"errorCode":"AccessDenied","requestParameters":{"bucketName":"finance-reports","key":"2026/09/summary.csv"},"readOnly":true}
{"eventTime":"2026-09-15T10:02:05Z","eventSource":"sts.amazonaws.com","eventName":"GetCallerIdentity","awsRegion":"us-east-1","sourceIPAddress":"203.0.113.50","userAgent":"aws-cli/2.17.40","userIdentity":{"type":"IAMUser","arn":"arn:aws:iam::111122223333:user/build-bot","userName":"build-bot"},"readOnly":true}
{"eventTime":"2026-09-15T10:02:31Z","eventSource":"iam.amazonaws.com","eventName":"ListUsers","awsRegion":"us-east-1","sourceIPAddress":"203.0.113.50","userAgent":"aws-cli/2.17.40","userIdentity":{"type":"IAMUser","arn":"arn:aws:iam::111122223333:user/build-bot","userName":"build-bot"},"errorCode":"AccessDenied","readOnly":true}
{"eventTime":"2026-09-15T10:02:38Z","eventSource":"iam.amazonaws.com","eventName":"ListRoles","awsRegion":"us-east-1","sourceIPAddress":"203.0.113.50","userAgent":"aws-cli/2.17.40","userIdentity":{"type":"IAMUser","arn":"arn:aws:iam::111122223333:user/build-bot","userName":"build-bot"},"errorCode":"AccessDenied","readOnly":true}
{"eventTime":"2026-09-15T10:02:52Z","eventSource":"iam.amazonaws.com","eventName":"GetAccountAuthorizationDetails","awsRegion":"us-east-1","sourceIPAddress":"203.0.113.50","userAgent":"aws-cli/2.17.40","userIdentity":{"type":"IAMUser","arn":"arn:aws:iam::111122223333:user/build-bot","userName":"build-bot"},"errorCode":"AccessDenied","readOnly":true}
{"eventTime":"2026-09-15T10:03:05Z","eventSource":"ec2.amazonaws.com","eventName":"DescribeInstances","awsRegion":"us-east-1","sourceIPAddress":"203.0.113.50","userAgent":"aws-cli/2.17.40","userIdentity":{"type":"IAMUser","arn":"arn:aws:iam::111122223333:user/build-bot","userName":"build-bot"},"errorCode":"Client.UnauthorizedOperation","readOnly":true}
{"eventTime":"2026-09-15T10:03:09Z","eventSource":"ec2.amazonaws.com","eventName":"DescribeVpcs","awsRegion":"us-east-1","sourceIPAddress":"203.0.113.50","userAgent":"aws-cli/2.17.40","userIdentity":{"type":"IAMUser","arn":"arn:aws:iam::111122223333:user/build-bot","userName":"build-bot"},"errorCode":"Client.UnauthorizedOperation","readOnly":true}
{"eventTime":"2026-09-15T10:03:14Z","eventSource":"ec2.amazonaws.com","eventName":"DescribeSecurityGroups","awsRegion":"us-east-1","sourceIPAddress":"203.0.113.50","userAgent":"aws-cli/2.17.40","userIdentity":{"type":"IAMUser","arn":"arn:aws:iam::111122223333:user/build-bot","userName":"build-bot"},"errorCode":"Client.UnauthorizedOperation","readOnly":true}
{"eventTime":"2026-09-15T10:03:22Z","eventSource":"secretsmanager.amazonaws.com","eventName":"ListSecrets","awsRegion":"us-east-1","sourceIPAddress":"203.0.113.50","userAgent":"aws-cli/2.17.40","userIdentity":{"type":"IAMUser","arn":"arn:aws:iam::111122223333:user/build-bot","userName":"build-bot"},"errorCode":"AccessDenied","readOnly":true}
{"eventTime":"2026-09-15T10:03:40Z","eventSource":"s3.amazonaws.com","eventName":"ListBuckets","awsRegion":"us-east-1","sourceIPAddress":"203.0.113.50","userAgent":"aws-cli/2.17.40","userIdentity":{"type":"IAMUser","arn":"arn:aws:iam::111122223333:user/build-bot","userName":"build-bot"},"readOnly":true}
{"eventTime":"2026-09-15T10:06:12Z","eventSource":"iam.amazonaws.com","eventName":"CreateUser","awsRegion":"us-east-1","sourceIPAddress":"203.0.113.50","userAgent":"aws-cli/2.17.40","userIdentity":{"type":"IAMUser","arn":"arn:aws:iam::111122223333:user/build-bot","userName":"build-bot"},"requestParameters":{"userName":"backup-svc"},"readOnly":false}
{"eventTime":"2026-09-15T10:06:30Z","eventSource":"iam.amazonaws.com","eventName":"AttachUserPolicy","awsRegion":"us-east-1","sourceIPAddress":"203.0.113.50","userAgent":"aws-cli/2.17.40","userIdentity":{"type":"IAMUser","arn":"arn:aws:iam::111122223333:user/build-bot","userName":"build-bot"},"requestParameters":{"userName":"backup-svc","policyArn":"arn:aws:iam::aws:policy/AdministratorAccess"},"readOnly":false}
{"eventTime":"2026-09-15T10:06:51Z","eventSource":"iam.amazonaws.com","eventName":"CreateAccessKey","awsRegion":"us-east-1","sourceIPAddress":"203.0.113.50","userAgent":"aws-cli/2.17.40","userIdentity":{"type":"IAMUser","arn":"arn:aws:iam::111122223333:user/build-bot","userName":"build-bot"},"requestParameters":{"userName":"backup-svc"},"responseElements":{"accessKey":{"userName":"backup-svc","accessKeyId":"AKIAIOSFODNN7EXAMPLE","status":"Active"}},"readOnly":false}
{"eventTime":"2026-09-15T10:09:03Z","eventSource":"cloudtrail.amazonaws.com","eventName":"StopLogging","awsRegion":"us-east-1","sourceIPAddress":"203.0.113.50","userAgent":"aws-cli/2.17.40","userIdentity":{"type":"IAMUser","arn":"arn:aws:iam::111122223333:user/backup-svc","userName":"backup-svc"},"requestParameters":{"name":"arn:aws:cloudtrail:us-east-1:111122223333:trail/org-trail"},"readOnly":false}
{"eventTime":"2026-09-15T10:40:18Z","eventSource":"ec2.amazonaws.com","eventName":"DescribeInstances","awsRegion":"us-east-1","sourceIPAddress":"192.0.2.10","userAgent":"aws-cli/2.17.40","userIdentity":{"type":"AssumedRole","arn":"arn:aws:sts::111122223333:assumed-role/ci-deploy/runner-4821"},"readOnly":true}
{"eventTime":"2026-09-15T11:40:26Z","eventSource":"iam.amazonaws.com","eventName":"CreateAccessKey","awsRegion":"us-east-1","sourceIPAddress":"198.51.100.20","userAgent":"AWS Internal","userIdentity":{"type":"IAMUser","arn":"arn:aws:iam::111122223333:user/alice","userName":"alice"},"requestParameters":{"userName":"svc-reporting"},"responseElements":{"accessKey":{"userName":"svc-reporting","accessKeyId":"AKIAI44QH8DHBEXAMPLE","status":"Active"}},"readOnly":false}
EOF
shasum -a 256 cloudtrail-lab.jsonl

The hash must be 35b7c80582d0650ec4850acf84f84fa4b8e5d37169ac3f7276c76f29b67c3319; anything else means the file changed on the way. Now create an index (a named store of events) and the source type, and load the file once with oneshot:

PW=$(cat .splunk-password)
curl -sk -u "admin:$PW" -o /dev/null -w '%{http_code}\n' https://localhost:8089/services/data/indexes -d name=cloudtrail
curl -sk -u "admin:$PW" -o /dev/null -w '%{http_code}\n' https://localhost:8089/servicesNS/nobody/search/configs/conf-props \
  -d name=lab:cloudtrail -d SHOULD_LINEMERGE=false -d KV_MODE=json \
  --data-urlencode 'TIME_PREFIX="eventTime":"' --data-urlencode 'TIME_FORMAT=%Y-%m-%dT%H:%M:%S%Z'
docker cp cloudtrail-lab.jsonl splunk-lab:/tmp/cloudtrail-lab.jsonl
docker exec -u splunk splunk-lab /opt/splunk/bin/splunk add oneshot /tmp/cloudtrail-lab.jsonl -index cloudtrail -sourcetype lab:cloudtrail -auth "admin:$PW"
201
201
WARNING: Server Certificate Hostname Validation is disabled. Please see server.conf/[sslConfig]/cliVerifyServerName for details.
Oneshot '/tmp/cloudtrail-lab.jsonl' added

201 means created. -k lets curl accept the container's self-signed certificate, which is reasonable only because it is your own machine. SHOULD_LINEMERGE=false makes each line one event; TIME_PREFIX and TIME_FORMAT take each event's time from its eventTime, where %Z reads the trailing Z as UTC; KV_MODE=json extracts the JSON (JavaScript Object Notation) fields when you search, naming nested ones with dots, as in userIdentity.userName. The first three act while data arrives, so they must exist before the load. Created through the API they apply at once; typed into a props.conf file they would wait for a restart. Give Splunk a few seconds, then check that 24 events arrived, each with its own time rather than the time you loaded it:

index=cloudtrail
| eval own_time=if(_time == strptime(eventTime, "%Y-%m-%dT%H:%M:%S%Z"), "yes", "no")
| stats count BY sourcetype own_time
  sourcetype   own_time count
-------------- -------- -----
lab:cloudtrail yes         24

_time is the time Splunk gave the event. eval computes a new field for every event, here with if(test, then, else), and strptime turns the eventTime text into the same kind of number as _time. A no, or any count but 24, means the load went wrong.

A load can report success and still be wrong. Point the oneshot at an index that does not exist and it still prints Oneshot ... added, while Splunk drops all 24 events and says so only in its own log. Swap %H and %M in TIME_FORMAT and 19 events get wrong times while 5 look right, because Splunk falls back to guessing where the format fails. Only a per-event check like the one above catches both.

3. Filter and pick columns: trail tampering (D1)

index=cloudtrail eventName IN (StopLogging, DeleteTrail, UpdateTrail, PutEventSelectors)
| table _time eventName userIdentity.userName sourceIPAddress
           _time             eventName  userIdentity.userName sourceIPAddress
--------------------------- ----------- --------------------- ---------------
2026-09-15 10:09:03.000 GMT StopLogging backup-svc            203.0.113.50

IN matches any value in the list, and table shows the fields you name, in that order. One event: backup-svc stopped the trail from 203.0.113.50. The other 23 events are the negative case, and none appears. Two traps already live in this search. Leave out index=cloudtrail and it returns nothing, because Splunk then searches only your role's default indexes (main and os for this admin). And field names are case-sensitive while values in the base search are not: eventname=StopLogging finds nothing, eventName=stoplogging finds the event.

4. Nested fields, and where versus the base search: logins without MFA (D4)

The obvious first draft looks only at the MFA flag:

index=cloudtrail eventName=ConsoleLogin additionalEventData.MFAUsed=No
| table _time userIdentity.userName sourceIPAddress responseElements.ConsoleLogin
| sort _time
           _time            userIdentity.userName sourceIPAddress responseElements.ConsoleLogin
--------------------------- --------------------- --------------- -----------------------------
2026-09-15 09:15:47.000 GMT bob                   198.51.100.21   Success
2026-09-15 09:20:03.000 GMT alice                 192.0.2.77      Failure

The second row is someone failing to sign in as alice, and a failed sign-in carries MFAUsed No too. AWS's own Security Hub control for sign-in without MFA, [CloudWatch.3], requires success as well, and so must D4 (sort _time just puts rows in time order):

index=cloudtrail eventName=ConsoleLogin additionalEventData.MFAUsed=No responseElements.ConsoleLogin=Success
| table _time userIdentity.userName sourceIPAddress
           _time            userIdentity.userName sourceIPAddress
--------------------------- --------------------- ---------------
2026-09-15 09:15:47.000 GMT bob                   198.51.100.21

You can filter after a pipe too, with where, but it follows different rules. where compares case-sensitively, and a field name containing dots must go in single quotes: | where 'additionalEventData.MFAUsed'="No". Without the quotes, where reads each dot as joining two strings, both empty, and silently matches nothing. Or name the field yourself with spath, which extracts one JSON path on demand: | spath output=mfa path=additionalEventData.MFAUsed | where mfa="No". You need spath whenever a source type has no automatic extraction, or JSON arrives inside another field.

5. Count, then threshold: the burst that never fires (D2)

index=cloudtrail errorCode=AccessDenied
| stats count BY sourceIPAddress
| where count >= 5

This prints nothing. Delete the last line and you see why: 203.0.113.50 has 4 and 192.0.2.20 has 3, so nobody reaches five. Did nothing happen, or can the search not see what happened? Run the control: drop the filter and ask which error codes exist at all.

index=cloudtrail
| fillnull value="(none)" errorCode
| stats count BY errorCode
         errorCode           count
---------------------------- -----
(none)                          14
AccessDenied                     7
Client.UnauthorizedOperation     3

EC2 reports a refusal as Client.UnauthorizedOperation, so three of the attacker's seven refusals were invisible to the rule. The fillnull line matters as well: stats silently leaves out every event that lacks the field after BY, so without it the 14 events with no errorCode would not appear at all. AWS's Security Hub control for unauthorized API calls, [CloudWatch.2], matches both families: codes starting AccessDenied, such as AccessDeniedException, and codes ending UnauthorizedOperation. List all four:

index=cloudtrail errorCode IN (AccessDenied, AccessDeniedException, Client.UnauthorizedOperation, UnauthorizedOperation)
| stats count BY sourceIPAddress
| where count >= 5 ```the D2 threshold```
sourceIPAddress count
--------------- -----
203.0.113.50        7

Without the where line, 192.0.2.20 shows 3 and stays under the threshold: the negative case. Text between three backticks is a comment.

A query that returns nothing and a query that cannot work return the same thing. Before you trust an empty result, run a control that must return something: the search without its threshold, or a count of the values that really exist.

To see when the refusals happened, cut time into ten-minute buckets with bin:

index=cloudtrail errorCode IN (AccessDenied, AccessDeniedException, Client.UnauthorizedOperation, UnauthorizedOperation)
| bin _time span=10m
| stats count BY _time sourceIPAddress
           _time            sourceIPAddress count
--------------------------- --------------- -----
2026-09-15 09:30:00.000 GMT 192.0.2.20          1
2026-09-15 09:40:00.000 GMT 192.0.2.20          1
2026-09-15 10:00:00.000 GMT 192.0.2.20          1
2026-09-15 10:00:00.000 GMT 203.0.113.50        7

bin rounds each _time down to its bucket, so stats can count per address per ten minutes. The buckets start on the clock (10:00, 10:10), not at the first event, so a burst straddling 10:10 would be split in two and could stay under five in both halves. The scheduled alert in step 7 has the same edge, because each run looks at one fixed ten-minute stretch.

6. One user, two events, ten minutes (D3)

index=cloudtrail eventName IN (CreateUser, CreateAccessKey)
| transaction requestParameters.userName startswith=eval(eventName="CreateUser") endswith=eval(eventName="CreateAccessKey") maxspan=10m
| table _time requestParameters.userName userIdentity.userName sourceIPAddress duration eventcount
           _time            requestParameters.userName userIdentity.userName sourceIPAddress duration eventcount
--------------------------- -------------------------- --------------------- --------------- -------- ----------
2026-09-15 10:06:12.000 GMT backup-svc                 build-bot             203.0.113.50          39          2

transaction groups events that share the new user's name (requestParameters.userName, the target, as opposed to userIdentity.userName, the actor), starting at CreateUser and ending at CreateAccessKey, and reports the group's duration in seconds: 39. Now delete maxspan=10m and run it again. svc-reporting appears too, with a duration of 9466 seconds: alice created it at 09:02:40 and keyed it 2 hours 37 minutes later, routine work that the window exists to leave out. Exercise 3 writes the same detection with stats.

7. Turn D2 into a scheduled alert

Splunk stores alerts as stanzas in savedsearches.conf: a search, a schedule, a time window and a condition. This one runs every ten minutes over the ten minutes that ended ten minutes ago, leaving time for CloudTrail's delivery delay, and fires when the search returns any row. Save it, install it as a small app, and restart Splunk to load the new app:

cat > savedsearches.conf <<'EOF'
# D2, denied-call burst. MITRE ATT&CK v19.2: T1087.004, T1580.
[d2_denied_call_burst]
description = Five or more denied AWS API calls from one source address in 10 minutes
search = index=cloudtrail errorCode IN (AccessDenied, AccessDeniedException, Client.UnauthorizedOperation, UnauthorizedOperation) \
| stats count BY sourceIPAddress \
| where count >= 5
enableSched = 1
cron_schedule = */10 * * * *
dispatch.earliest_time = -20m@m
dispatch.latest_time = -10m@m
counttype = number of events
relation = greater than
quantity = 0
alert.track = 1
EOF
docker exec -u splunk splunk-lab mkdir -p /opt/splunk/etc/apps/lab_detections/local
docker cp savedsearches.conf splunk-lab:/opt/splunk/etc/apps/lab_detections/local/
docker exec -u splunk splunk-lab /opt/splunk/bin/splunk restart

counttype, relation and quantity together say "more than 0 results", and alert.track lists each firing under Activity > Triggered Alerts in Splunk Web. Plain Splunk Enterprise has no field for MITRE ATT&CK (Adversarial Tactics, Techniques, and Common Knowledge) techniques, so the IDs sit in a comment. Alerts need the Enterprise Trial licence the container started on. Splunk Free, which you can switch to later, still indexes 500 MB a day and runs scheduled searches, but has no alerting and no sign-in at all: anyone who reaches port 8000 is admin.

8. Test the alert both ways

The schedule only ever looks at the last half hour, and the lab is older than that, so run the alert over two stretches of the lab morning and let it act (trigger_actions=1). The window is in epoch seconds: 09:30 to 10:00 UTC holds only reports-app's refusals, and 10:00 to 10:10 holds the attack. Splunk did not honour dates written as 2026-09-15T10:00:00Z here: the job silently searched all time, so a test written that way passes for the wrong reason.

PW=$(cat .splunk-password)
URL=https://localhost:8089/servicesNS/nobody/lab_detections/saved/searches/d2_denied_call_burst/dispatch
curl -sk -u "admin:$PW" -o /dev/null -w '%{http_code}\n' "$URL" -d trigger_actions=1 -d dispatch.earliest_time=1789464600 -d dispatch.latest_time=1789466400
curl -sk -u "admin:$PW" -o /dev/null -w '%{http_code}\n' "$URL" -d trigger_actions=1 -d dispatch.earliest_time=1789466400 -d dispatch.latest_time=1789467000

Each prints 201. Wait a few seconds, then count the firings:

| rest /servicesNS/-/-/alerts/fired_alerts splunk_server=local
| table title triggered_alert_count
       title         triggered_alert_count
-------------------- ---------------------
-                                        1
d2_denied_call_burst                     1

Two runs, one firing (the - row is the total): the attack window fired, the quiet one did not. | rest reads Splunk's own REST API from inside a search.

9. Splunk's public rules: the security_content format

Splunk publishes its detections as YAML files in splunk/security_content, built into an app by its tool contentctl-ng, which writes each one out as a savedsearches.conf stanza like step 7's. This is its D1 rule, trimmed; its ATT&CK ID is already the v19 one:

name: AWS Defense Evasion Stop Logging Cloudtrail
id: 8a2f3ca2-4eb5-4389-a549-14063882e537
version: 12
status: production
search: |-
    `cloudtrail` eventName = StopLogging eventSource = cloudtrail.amazonaws.com userAgent!=console.amazonaws.com errorCode = success
      | rename user_name as user
# (trimmed: the search goes on to count by signature, dest, user, src and vendor fields)
how_to_implement: You must install Splunk AWS Add on and enable Cloudtrail logs in your AWS Environment.
references:
    - https://attack.mitre.org/techniques/T1562/008/
mitre_attack_id:
    - T1685.002
# (trimmed: tests, which points at an attack sample the rule must match)

The file carries its own metadata and, in tests, an attack sample that the rule must match when loaded into a Splunk instance. Three things to notice. ATT&CK rot: mitre_attack_id is T1685.002, and the build copies it into the stanza for Splunk Enterprise Security, but references and the test sample's folder still name T1562.008, the ID that v19 revoked; the name still says Defense Evasion, the tactic v19 split into Stealth and Defense Impairment; and contentctl-ng accepted a planted T1562.008 without complaint. Field names: `cloudtrail` is a macro for sourcetype=aws:cloudtrail, and fields such as src, user and signature come from the Splunk Add-on for AWS, which maps CloudTrail onto Splunk's Common Information Model (CIM). The silent condition: a successful call has no errorCode at all in raw CloudTrail (the rule's own test event has none), so errorCode = success matches only data the add-on has filled in. Run the rule's base search against the raw lab, with the macro replaced by the lab's index:

index=cloudtrail eventName=StopLogging eventSource=cloudtrail.amazonaws.com userAgent!=console.amazonaws.com errorCode=success
| stats count
count
-----
    0

Zero, although the lab contains exactly the event it describes; delete errorCode=success and the count is 1. A published rule is only as good as the field names it assumes, so run its control before you trust its silence. For the record, the five detections in ATT&CK v19.2: D1 T1685.002, D2 T1087.004 and T1580, D3 T1136.003 and T1098.001, D4 T1078.004, D5 T1098.003.

Hands-on exercises

Exercise 1

Write D5: every time the AdministratorAccess managed policy was attached to a user, showing who attached it, to whom, and from where.

Show the answer 
index=cloudtrail eventName=AttachUserPolicy requestParameters.policyArn="arn:aws:iam::aws:policy/AdministratorAccess"
| table _time userIdentity.userName requestParameters.userName sourceIPAddress
           _time            userIdentity.userName requestParameters.userName sourceIPAddress
--------------------------- --------------------- -------------------------- ---------------
2026-09-15 10:06:30.000 GMT build-bot             backup-svc                 203.0.113.50

Two fields are called userName: userIdentity.userName is who acted, requestParameters.userName who was acted on. The quotes around the policy ARN (Amazon Resource Name) keep its colons and slash as one value. For real coverage, add AttachRolePolicy, AttachGroupPolicy and PutUserPolicy, which grant the same power by other routes.

Exercise 2

This search returns nothing. Find both faults, and say how you would prove each one rather than guess.

eventName=StopLogging
| where userIdentity.userName="backup-svc"
| table _time eventName userIdentity.userName
Show the answer

First, no index: Splunk searched only the default indexes. Second, the unquoted dotted name in where, which joins two empty strings. Fixing either one alone still returns nothing, which is why guessing fails here. Prove it by deleting from the end: eventName=StopLogging alone returns nothing (the index), index=cloudtrail eventName=StopLogging returns the event (so the where is at fault).

index=cloudtrail eventName=StopLogging
| where 'userIdentity.userName'="backup-svc"
| table _time eventName userIdentity.userName

That returns the StopLogging event from step 3.

Exercise 3

Write D3 without transaction, using stats, so that svc-reporting stays out. Then say what each condition protects against.

Show the answer 
index=cloudtrail eventName IN (CreateUser, CreateAccessKey)
| stats earliest(eventName) AS first_step dc(eventName) AS steps range(_time) AS gap BY requestParameters.userName
| where steps=2 AND first_step="CreateUser" AND gap<=600
requestParameters.userName first_step steps gap
-------------------------- ---------- ----- ---
backup-svc                 CreateUser     2  39

dc counts distinct values, so steps=2 needs both events; earliest gives the first event's name, so the user must be created before the key; range(_time) is last minus first, in seconds, and gap<=600 is the ten-minute window. Delete the where line and svc-reporting appears with a gap of 9466. stats keeps one row per user instead of whole groups of events, which is usually lighter on a large index.

Exercise 4

Find the source addresses used by more than one identity. Then explain why counting userIdentity.userName instead reports 0 identities for two of the addresses.

Show the answer 
index=cloudtrail
| stats dc(userIdentity.arn) AS identities values(userIdentity.userName) AS users BY sourceIPAddress
| eval users=mvjoin(users, ", ")
| where identities > 1
sourceIPAddress identities         users
--------------- ---------- ---------------------
203.0.113.50             2 backup-svc, build-bot

The leaked key and the user it created, from one address. values collects distinct values into one multivalue field and mvjoin prints them on one line. Roles have no userName, so dc(userIdentity.userName) gives 0 for 192.0.2.10 (ci-deploy) and 192.0.2.20 (reports-app). Count a field every event has, such as userIdentity.arn (the ARN).

Common mistakes

Where next