Get the Zoom link
Cloud Security Office Hours Banner

Splunk SPL, hands-on

Splunk stores logs and lets you search them. Its Search Processing Language (SPL) picks the events you want, then counts, filters and reshapes them one step at a time. Learn it from zero on a trial copy of Splunk Enterprise on your own computer, with 24 events from AWS CloudTrail, the record of calls made to an AWS account, and learn to spot the search that returns nothing because it cannot work.

Jump to the walk-through All how-to guides

·

Time: ~2 hours  ·  Difficulty: Beginner  ·  You need: Docker with about 3 GB of memory and 3 GB of disk to spare, and a terminal. No Splunk account and no cloud account.

How this page was checked: every command and search was run as printed, in order, against Splunk Enterprise 10.4.3 in Docker (image splunk/splunk:10.4.3, under Rosetta emulation on an Apple silicon Mac), by a script that copies each block from this page and compares the results with the output shown here. The public rule in step 9 was read from Splunk's repository and built with Splunk's own tool, contentctl-ng 1.1.1. Not run: Splunk Cloud Platform, Splunk Enterprise Security, the Splunk Add-on for AWS, and the Splunk Free licence.

The language of Splunk searches

Splunk collects logs and lets you search them. A search can answer an incident question such as "who stopped our audit logging, and from where?", or count something, such as how many times each address was refused. SPL is how you ask those questions of data in Splunk Enterprise, which you run yourself, and Splunk Cloud Platform, Splunk's hosted service. Splunk Enterprise Security, Splunk's security information and event management (SIEM) app, runs its detections as SPL searches too; a SIEM gathers security logs in one place and raises alerts from them.

Analysts in a security operations centre (SOC), the team that watches for attacks and responds to them, type SPL to investigate. Detection engineers save searches that run on a schedule and raise alerts.

One thing about how Splunk works matters from the start. Splunk keeps each event's raw text and works out its fields, the named values such as eventName that searches filter on, when you search. A source type tells Splunk how to split the text into events, where each event's time is, and how to find the fields. When it is wrong, every search is quietly wrong, so this page checks the load before anything else (step 2).

Splunk also publishes SPL2, a newer version of the language, and the search bar in Splunk Enterprise 10.4 offers to convert a search to it. Saved searches, alerts and Splunk's public detection library, all used on this page, are SPL.

On this page

  1. How to use this guide
  2. A search is a pipeline
  3. Walk-through: one leaked key, five detections
  4. Hands-on exercises
  5. Common mistakes
  6. Where next

How to use this guide

Beginner here means new to Splunk and to query languages. You should be comfortable typing commands into a terminal and have Docker installed. You do not need to know AWS: the AWS terms the lab depends on are explained as they come up, and terms with a glossary entry link to it.

Read it in order, because each part uses the one before. The next section explains how every search is built. Its example needs the lab data, so read it first and run it once step 2 is done. In the walk-through, steps 1 and 2 start Splunk in Docker, load 24 CloudTrail events and check that they arrived intact. Steps 3 to 6 build four of the five detections listed in step 2, one per step. Steps 4 and 5 start with a search that looks right and returns the wrong answer without an error, then show why and fix it. Steps 7 and 8 turn one detection into an alert that runs on a schedule and test it, and step 9 reads one of Splunk's published rules. The exercises then ask you to write the fifth detection and repair broken searches yourself, with each answer hidden until you click Show the answer.

A code block labelled BASH is something to paste into your terminal, and each block has a copy button. A block labelled SPL is a search: type ./spl, paste the search, press Enter, then Ctrl-D, or paste it into Splunk Web, as the end of step 1 explains. The unlabelled block under it, when there is one, shows what you should see. Some BASH blocks start with a line such as cat > spl <<'EOF', which writes every line down to the one that says only EOF into a file called spl. Paste the whole block at once, the EOF line included. The YAML block in step 9 is part of one of Splunk's rule files, there to read.

Your output should match the page line for line. A script ran every command and search here in order against Splunk Enterprise 10.4.3 in Docker and compared the results with the output shown, and Splunk Web shows the same rows as a table. Splunk Cloud Platform, Splunk Enterprise Security, the Splunk Add-on for AWS and the Splunk Free licence were not run, so what the page says about them may not match what you see if you use them.

The 24 lines of lab data in step 2 can be pasted without reading them, since the hash check under the block tells you whether the paste worked. And step 9, on Splunk's published rules, can wait for a second reading: it matters most once you start running rules that other people wrote.

If your output differs from the page, check that you are in ~/splunk-lab, the folder step 1 creates, because ./spl, the password file and the lab file are all there. Then repeat step 2's two checks: the file's hash, and the count of 24 events, each with its own time. In Splunk Web, check that the time range picker says All time. And check that you started the image step 1 names, splunk/splunk:10.4.3, the version this page was checked on.

A search is a pipeline

Suppose you want to know who changed anything in the account that morning, and how many changes each of them made. You would first set aside the events that changed something, then count them per user. An SPL search is written in that order: a chain of steps, each handing its results to the next, which is why it is called a pipeline.

The part before the first pipe character, |, is the base search: it chooses events. index= says where to look, and every term after it must match (AND is implied; write OR and NOT in capitals). field=value matches one field. After that, each | hands the results so far to one command, which reshapes them and passes them on. Once step 2 has loaded the lab, try:

index=cloudtrail readOnly=false
| stats count BY userIdentity.userName
userIdentity.userName count
--------------------- -----
alice                     2
backup-svc                1
build-bot                 3

Read it left to right: the base search keeps the six events that changed something (readOnly=false), and stats turns them into one row per identity. A command sees only what the command before it produced, so every prefix of a search is a search you can run. To debug one, delete commands from the end until the output stops surprising you; step 5 does that to find out why a detection never fires.

Walk-through: one leaked key, five detections

1. Start Splunk Enterprise in Docker

Splunk publishes Splunk Enterprise as the splunk/splunk image, for x86-64 processors (the Intel and AMD kind) only. On an Apple silicon Mac, Docker Desktop runs it under Rosetta emulation when that option is on in its settings; this page was checked that way, and the container took about two minutes to start and one to restart. The two SPLUNK_ settings accept the licence and the Splunk General Terms for you, so read them first. The second line creates a throwaway admin password for this container only. The last line waits, checking every ten seconds, until Docker reports the container healthy.

mkdir -p ~/splunk-lab && cd ~/splunk-lab
openssl rand -hex 12 > .splunk-password
docker run -d --name splunk-lab --platform linux/amd64 \
  -p 8000:8000 -p 8089:8089 \
  -e SPLUNK_START_ARGS=--accept-license \
  -e SPLUNK_GENERAL_TERMS=--accept-sgt-current-at-splunk-com \
  -e SPLUNK_PASSWORD="$(cat .splunk-password)" \
  splunk/splunk:10.4.3
until [ "$(docker inspect -f '{{.State.Health.Status}}' splunk-lab)" = healthy ]; do sleep 10; done

Port 8000 is Splunk Web, the browser interface. Port 8089 is the management port, where the command-line interface (CLI) and the REST API (Splunk's HTTP application programming interface) reach Splunk. A new install runs on a 60-day Enterprise Trial licence: 500 MB of new data a day, alerting included, which step 7 needs. Now save a helper that runs one search from the terminal and prints exactly the output shown on this page:

cat > spl <<'EOF'
#!/bin/sh
# spl: run the search on standard input against the lab. Drop only the CLI's
# certificate notice; keep errors, so a broken search never looks like an empty one.
docker exec -u splunk splunk-lab /opt/splunk/bin/splunk search "$(cat)" \
  -auth "admin:$(cat ~/splunk-lab/.splunk-password)" 2>&1 |
  grep -v '^WARNING: Server Certificate Hostname Validation is disabled'
EOF
chmod +x spl

To run a search, type ./spl, paste the search, press Enter, then Ctrl-D. Or use Splunk Web: open http://localhost:8000, sign in as admin with the password in .splunk-password, choose Search & Reporting, set the time range picker to All time (it starts at Last 24 hours, which shows none of the lab), and paste the search. Splunk Web shows the same rows as a table, in UTC (Coordinated Universal Time), the container's time zone. The first sign-in also tells you that sharing of product usage data with Splunk is on: to stop it, open Settings > Instrumentation, click the gear next to Usage Data and switch Usage Data off (licence and software version data have no switch).

2. Load the lab data, then check what arrived

The lab is 24 Amazon Web Services (AWS) CloudTrail events from one morning, trimmed to the fields this page uses but keeping CloudTrail's real field names and nesting. The access key of build-bot, a continuous integration (CI) user, has leaked; an access key is the credential a program signs its AWS requests with. From 203.0.113.50 someone checks whose key it is, tries to list Identity and Access Management (IAM) users, Elastic Compute Cloud (EC2) instances and Secrets Manager secrets (and is mostly refused), lists the Simple Storage Service (S3) buckets, then creates a user backup-svc, attaches AdministratorAccess (AWS's ready-made policy that allows every action) to it, gives it an access key and uses it to stop CloudTrail logging. Around that runs an ordinary morning: sign-ins, one of them without multi-factor authentication (MFA) and one failed; an administrator creating a service account; an application refused the same S3 object three times; a CI role describing instances. The page builds five detections from it:

cat > cloudtrail-lab.jsonl <<'EOF'
{"eventTime":"2026-09-15T08:55:12Z","eventSource":"signin.amazonaws.com","eventName":"ConsoleLogin","awsRegion":"us-east-1","sourceIPAddress":"198.51.100.20","userAgent":"Mozilla/5.0","userIdentity":{"type":"IAMUser","arn":"arn:aws:iam::111122223333:user/alice","userName":"alice"},"responseElements":{"ConsoleLogin":"Success"},"additionalEventData":{"MFAUsed":"Yes"}}
{"eventTime":"2026-09-15T09:02:40Z","eventSource":"iam.amazonaws.com","eventName":"CreateUser","awsRegion":"us-east-1","sourceIPAddress":"198.51.100.20","userAgent":"AWS Internal","userIdentity":{"type":"IAMUser","arn":"arn:aws:iam::111122223333:user/alice","userName":"alice"},"requestParameters":{"userName":"svc-reporting"},"readOnly":false}
{"eventTime":"2026-09-15T09:03:05Z","eventSource":"s3.amazonaws.com","eventName":"ListBuckets","awsRegion":"us-east-1","sourceIPAddress":"198.51.100.20","userAgent":"AWS Internal","userIdentity":{"type":"IAMUser","arn":"arn:aws:iam::111122223333:user/alice","userName":"alice"},"readOnly":true}
{"eventTime":"2026-09-15T09:10:22Z","eventSource":"ec2.amazonaws.com","eventName":"DescribeInstances","awsRegion":"us-east-1","sourceIPAddress":"192.0.2.10","userAgent":"aws-cli/2.17.40","userIdentity":{"type":"AssumedRole","arn":"arn:aws:sts::111122223333:assumed-role/ci-deploy/runner-4821"},"readOnly":true}
{"eventTime":"2026-09-15T09:15:47Z","eventSource":"signin.amazonaws.com","eventName":"ConsoleLogin","awsRegion":"us-east-1","sourceIPAddress":"198.51.100.21","userAgent":"Mozilla/5.0","userIdentity":{"type":"IAMUser","arn":"arn:aws:iam::111122223333:user/bob","userName":"bob"},"responseElements":{"ConsoleLogin":"Success"},"additionalEventData":{"MFAUsed":"No"}}
{"eventTime":"2026-09-15T09:20:03Z","eventSource":"signin.amazonaws.com","eventName":"ConsoleLogin","awsRegion":"us-east-1","sourceIPAddress":"192.0.2.77","userAgent":"Mozilla/5.0","userIdentity":{"type":"IAMUser","arn":"arn:aws:iam::111122223333:user/alice","userName":"alice"},"errorMessage":"Failed authentication","responseElements":{"ConsoleLogin":"Failure"},"additionalEventData":{"MFAUsed":"No"}}
{"eventTime":"2026-09-15T09:31:10Z","eventSource":"s3.amazonaws.com","eventName":"GetObject","awsRegion":"us-east-1","sourceIPAddress":"192.0.2.20","userAgent":"Boto3/1.35.10","userIdentity":{"type":"AssumedRole","arn":"arn:aws:sts::111122223333:assumed-role/reports-app/i-0a12b34c56d78e90f"},"errorCode":"AccessDenied","requestParameters":{"bucketName":"finance-reports","key":"2026/09/summary.csv"},"readOnly":true}
{"eventTime":"2026-09-15T09:47:55Z","eventSource":"s3.amazonaws.com","eventName":"GetObject","awsRegion":"us-east-1","sourceIPAddress":"192.0.2.20","userAgent":"Boto3/1.35.10","userIdentity":{"type":"AssumedRole","arn":"arn:aws:sts::111122223333:assumed-role/reports-app/i-0a12b34c56d78e90f"},"errorCode":"AccessDenied","requestParameters":{"bucketName":"finance-reports","key":"2026/09/summary.csv"},"readOnly":true}
{"eventTime":"2026-09-15T10:01:30Z","eventSource":"s3.amazonaws.com","eventName":"GetObject","awsRegion":"us-east-1","sourceIPAddress":"192.0.2.20","userAgent":"Boto3/1.35.10","userIdentity":{"type":"AssumedRole","arn":"arn:aws:sts::111122223333:assumed-role/reports-app/i-0a12b34c56d78e90f"},"errorCode":"AccessDenied","requestParameters":{"bucketName":"finance-reports","key":"2026/09/summary.csv"},"readOnly":true}
{"eventTime":"2026-09-15T10:02:05Z","eventSource":"sts.amazonaws.com","eventName":"GetCallerIdentity","awsRegion":"us-east-1","sourceIPAddress":"203.0.113.50","userAgent":"aws-cli/2.17.40","userIdentity":{"type":"IAMUser","arn":"arn:aws:iam::111122223333:user/build-bot","userName":"build-bot"},"readOnly":true}
{"eventTime":"2026-09-15T10:02:31Z","eventSource":"iam.amazonaws.com","eventName":"ListUsers","awsRegion":"us-east-1","sourceIPAddress":"203.0.113.50","userAgent":"aws-cli/2.17.40","userIdentity":{"type":"IAMUser","arn":"arn:aws:iam::111122223333:user/build-bot","userName":"build-bot"},"errorCode":"AccessDenied","readOnly":true}
{"eventTime":"2026-09-15T10:02:38Z","eventSource":"iam.amazonaws.com","eventName":"ListRoles","awsRegion":"us-east-1","sourceIPAddress":"203.0.113.50","userAgent":"aws-cli/2.17.40","userIdentity":{"type":"IAMUser","arn":"arn:aws:iam::111122223333:user/build-bot","userName":"build-bot"},"errorCode":"AccessDenied","readOnly":true}
{"eventTime":"2026-09-15T10:02:52Z","eventSource":"iam.amazonaws.com","eventName":"GetAccountAuthorizationDetails","awsRegion":"us-east-1","sourceIPAddress":"203.0.113.50","userAgent":"aws-cli/2.17.40","userIdentity":{"type":"IAMUser","arn":"arn:aws:iam::111122223333:user/build-bot","userName":"build-bot"},"errorCode":"AccessDenied","readOnly":true}
{"eventTime":"2026-09-15T10:03:05Z","eventSource":"ec2.amazonaws.com","eventName":"DescribeInstances","awsRegion":"us-east-1","sourceIPAddress":"203.0.113.50","userAgent":"aws-cli/2.17.40","userIdentity":{"type":"IAMUser","arn":"arn:aws:iam::111122223333:user/build-bot","userName":"build-bot"},"errorCode":"Client.UnauthorizedOperation","readOnly":true}
{"eventTime":"2026-09-15T10:03:09Z","eventSource":"ec2.amazonaws.com","eventName":"DescribeVpcs","awsRegion":"us-east-1","sourceIPAddress":"203.0.113.50","userAgent":"aws-cli/2.17.40","userIdentity":{"type":"IAMUser","arn":"arn:aws:iam::111122223333:user/build-bot","userName":"build-bot"},"errorCode":"Client.UnauthorizedOperation","readOnly":true}
{"eventTime":"2026-09-15T10:03:14Z","eventSource":"ec2.amazonaws.com","eventName":"DescribeSecurityGroups","awsRegion":"us-east-1","sourceIPAddress":"203.0.113.50","userAgent":"aws-cli/2.17.40","userIdentity":{"type":"IAMUser","arn":"arn:aws:iam::111122223333:user/build-bot","userName":"build-bot"},"errorCode":"Client.UnauthorizedOperation","readOnly":true}
{"eventTime":"2026-09-15T10:03:22Z","eventSource":"secretsmanager.amazonaws.com","eventName":"ListSecrets","awsRegion":"us-east-1","sourceIPAddress":"203.0.113.50","userAgent":"aws-cli/2.17.40","userIdentity":{"type":"IAMUser","arn":"arn:aws:iam::111122223333:user/build-bot","userName":"build-bot"},"errorCode":"AccessDenied","readOnly":true}
{"eventTime":"2026-09-15T10:03:40Z","eventSource":"s3.amazonaws.com","eventName":"ListBuckets","awsRegion":"us-east-1","sourceIPAddress":"203.0.113.50","userAgent":"aws-cli/2.17.40","userIdentity":{"type":"IAMUser","arn":"arn:aws:iam::111122223333:user/build-bot","userName":"build-bot"},"readOnly":true}
{"eventTime":"2026-09-15T10:06:12Z","eventSource":"iam.amazonaws.com","eventName":"CreateUser","awsRegion":"us-east-1","sourceIPAddress":"203.0.113.50","userAgent":"aws-cli/2.17.40","userIdentity":{"type":"IAMUser","arn":"arn:aws:iam::111122223333:user/build-bot","userName":"build-bot"},"requestParameters":{"userName":"backup-svc"},"readOnly":false}
{"eventTime":"2026-09-15T10:06:30Z","eventSource":"iam.amazonaws.com","eventName":"AttachUserPolicy","awsRegion":"us-east-1","sourceIPAddress":"203.0.113.50","userAgent":"aws-cli/2.17.40","userIdentity":{"type":"IAMUser","arn":"arn:aws:iam::111122223333:user/build-bot","userName":"build-bot"},"requestParameters":{"userName":"backup-svc","policyArn":"arn:aws:iam::aws:policy/AdministratorAccess"},"readOnly":false}
{"eventTime":"2026-09-15T10:06:51Z","eventSource":"iam.amazonaws.com","eventName":"CreateAccessKey","awsRegion":"us-east-1","sourceIPAddress":"203.0.113.50","userAgent":"aws-cli/2.17.40","userIdentity":{"type":"IAMUser","arn":"arn:aws:iam::111122223333:user/build-bot","userName":"build-bot"},"requestParameters":{"userName":"backup-svc"},"responseElements":{"accessKey":{"userName":"backup-svc","accessKeyId":"AKIAIOSFODNN7EXAMPLE","status":"Active"}},"readOnly":false}
{"eventTime":"2026-09-15T10:09:03Z","eventSource":"cloudtrail.amazonaws.com","eventName":"StopLogging","awsRegion":"us-east-1","sourceIPAddress":"203.0.113.50","userAgent":"aws-cli/2.17.40","userIdentity":{"type":"IAMUser","arn":"arn:aws:iam::111122223333:user/backup-svc","userName":"backup-svc"},"requestParameters":{"name":"arn:aws:cloudtrail:us-east-1:111122223333:trail/org-trail"},"readOnly":false}
{"eventTime":"2026-09-15T10:40:18Z","eventSource":"ec2.amazonaws.com","eventName":"DescribeInstances","awsRegion":"us-east-1","sourceIPAddress":"192.0.2.10","userAgent":"aws-cli/2.17.40","userIdentity":{"type":"AssumedRole","arn":"arn:aws:sts::111122223333:assumed-role/ci-deploy/runner-4821"},"readOnly":true}
{"eventTime":"2026-09-15T11:40:26Z","eventSource":"iam.amazonaws.com","eventName":"CreateAccessKey","awsRegion":"us-east-1","sourceIPAddress":"198.51.100.20","userAgent":"AWS Internal","userIdentity":{"type":"IAMUser","arn":"arn:aws:iam::111122223333:user/alice","userName":"alice"},"requestParameters":{"userName":"svc-reporting"},"responseElements":{"accessKey":{"userName":"svc-reporting","accessKeyId":"AKIAI44QH8DHBEXAMPLE","status":"Active"}},"readOnly":false}
EOF
shasum -a 256 cloudtrail-lab.jsonl

The hash, a fingerprint of the file's exact contents, must be 35b7c80582d0650ec4850acf84f84fa4b8e5d37169ac3f7276c76f29b67c3319; anything else means the file changed on the way. Now create an index (a named store of events) and the source type, and load the file once with oneshot:

PW=$(cat .splunk-password)
curl -sk -u "admin:$PW" -o /dev/null -w '%{http_code}\n' https://localhost:8089/services/data/indexes -d name=cloudtrail
curl -sk -u "admin:$PW" -o /dev/null -w '%{http_code}\n' https://localhost:8089/servicesNS/nobody/search/configs/conf-props \
  -d name=lab:cloudtrail -d SHOULD_LINEMERGE=false -d KV_MODE=json \
  --data-urlencode 'TIME_PREFIX="eventTime":"' --data-urlencode 'TIME_FORMAT=%Y-%m-%dT%H:%M:%S%Z'
docker cp cloudtrail-lab.jsonl splunk-lab:/tmp/cloudtrail-lab.jsonl
docker exec -u splunk splunk-lab /opt/splunk/bin/splunk add oneshot /tmp/cloudtrail-lab.jsonl -index cloudtrail -sourcetype lab:cloudtrail -auth "admin:$PW"
201
201
WARNING: Server Certificate Hostname Validation is disabled. Please see server.conf/[sslConfig]/cliVerifyServerName for details.
Oneshot '/tmp/cloudtrail-lab.jsonl' added

201 means created. -k lets curl accept the container's self-signed certificate (one that no outside authority vouches for), which is reasonable only because it is your own machine. SHOULD_LINEMERGE=false makes each line one event; TIME_PREFIX and TIME_FORMAT take each event's time from its eventTime, where %Z reads the trailing Z as UTC; KV_MODE=json extracts the JSON (JavaScript Object Notation) fields when you search, naming nested ones with dots, as in userIdentity.userName. The first three act while data arrives, so they must exist before the load. Created through the API they apply at once; typed into a props.conf file they would wait for a restart. Give Splunk a few seconds, then check that 24 events arrived, each with its own time rather than the time you loaded it:

index=cloudtrail
| eval own_time=if(_time == strptime(eventTime, "%Y-%m-%dT%H:%M:%S%Z"), "yes", "no")
| stats count BY sourcetype own_time
  sourcetype   own_time count
-------------- -------- -----
lab:cloudtrail yes         24

_time is the time Splunk gave the event. eval computes a new field for every event, here with if(test, then, else), and strptime turns the eventTime text into the same kind of number as _time. A no, or any count but 24, means the load went wrong.

A load can report success and still be wrong. Point the oneshot at an index that does not exist and it still prints Oneshot ... added, while Splunk drops all 24 events and says so only in its own log. Swap %H and %M in TIME_FORMAT and 19 events get wrong times while 5 look right, because Splunk falls back to guessing where the format fails. Only a per-event check like the one above catches both.

3. Filter and pick columns: trail tampering (D1)

index=cloudtrail eventName IN (StopLogging, DeleteTrail, UpdateTrail, PutEventSelectors)
| table _time eventName userIdentity.userName sourceIPAddress
           _time             eventName  userIdentity.userName sourceIPAddress
--------------------------- ----------- --------------------- ---------------
2026-09-15 10:09:03.000 GMT StopLogging backup-svc            203.0.113.50

IN matches any value in the list, and table shows the fields you name, in that order. One event: backup-svc stopped the trail from 203.0.113.50. The other 23 events are the negative case, and none appears. Two traps already live in this search. Leave out index=cloudtrail and it returns nothing, because Splunk then searches only your role's default indexes (main and os for this admin). And field names are case-sensitive while values in the base search are not: eventname=StopLogging finds nothing, eventName=stoplogging finds the event.

4. Nested fields, and where versus the base search: logins without MFA (D4)

The obvious first draft looks only at the MFA flag:

index=cloudtrail eventName=ConsoleLogin additionalEventData.MFAUsed=No
| table _time userIdentity.userName sourceIPAddress responseElements.ConsoleLogin
| sort _time
           _time            userIdentity.userName sourceIPAddress responseElements.ConsoleLogin
--------------------------- --------------------- --------------- -----------------------------
2026-09-15 09:15:47.000 GMT bob                   198.51.100.21   Success
2026-09-15 09:20:03.000 GMT alice                 192.0.2.77      Failure

The second row is someone failing to sign in as alice, and a failed sign-in carries MFAUsed No too. AWS's own Security Hub CSPM control for sign-in without MFA, [CloudWatch.3], requires success as well, and so must D4 (sort _time just puts rows in time order):

index=cloudtrail eventName=ConsoleLogin additionalEventData.MFAUsed=No responseElements.ConsoleLogin=Success
| table _time userIdentity.userName sourceIPAddress
           _time            userIdentity.userName sourceIPAddress
--------------------------- --------------------- ---------------
2026-09-15 09:15:47.000 GMT bob                   198.51.100.21

You can filter after a pipe too, with where, but it follows different rules. where compares case-sensitively, and a field name containing dots must go in single quotes: | where 'additionalEventData.MFAUsed'="No". Without the quotes, where reads each dot as joining two strings, both empty, and silently matches nothing. Or name the field yourself with spath, which extracts one JSON path on demand: | spath output=mfa path=additionalEventData.MFAUsed | where mfa="No". You need spath whenever a source type has no automatic extraction, or JSON arrives inside another field.

5. Count, then threshold: the burst that never fires (D2)

index=cloudtrail errorCode=AccessDenied
| stats count BY sourceIPAddress
| where count >= 5

This prints nothing. Delete the last line and you see why: 203.0.113.50 has 4 and 192.0.2.20 has 3, so nobody reaches five. Did nothing happen, or can the search not see what happened? Run the control: drop the filter and ask which error codes exist at all.

index=cloudtrail
| fillnull value="(none)" errorCode
| stats count BY errorCode
         errorCode           count
---------------------------- -----
(none)                          14
AccessDenied                     7
Client.UnauthorizedOperation     3

EC2 reports a refusal as Client.UnauthorizedOperation, so three of the attacker's seven refusals were invisible to the rule. The fillnull line matters as well: stats silently leaves out every event that lacks the field after BY, so without it the 14 events with no errorCode would not appear at all. AWS's Security Hub CSPM control for unauthorized API calls, [CloudWatch.2], matches both families: codes starting AccessDenied, such as AccessDeniedException, and codes ending UnauthorizedOperation. List all four:

index=cloudtrail errorCode IN (AccessDenied, AccessDeniedException, Client.UnauthorizedOperation, UnauthorizedOperation)
| stats count BY sourceIPAddress
| where count >= 5 ```the D2 threshold```
sourceIPAddress count
--------------- -----
203.0.113.50        7

Without the where line, 192.0.2.20 shows 3 and stays under the threshold: the negative case. Text between three backticks is a comment.

A query that returns nothing and a query that cannot work return the same thing. Before you trust an empty result, run a control that must return something: the search without its threshold, or a count of the values that really exist.

To see when the refusals happened, cut time into ten-minute buckets with bin:

index=cloudtrail errorCode IN (AccessDenied, AccessDeniedException, Client.UnauthorizedOperation, UnauthorizedOperation)
| bin _time span=10m
| stats count BY _time sourceIPAddress
           _time            sourceIPAddress count
--------------------------- --------------- -----
2026-09-15 09:30:00.000 GMT 192.0.2.20          1
2026-09-15 09:40:00.000 GMT 192.0.2.20          1
2026-09-15 10:00:00.000 GMT 192.0.2.20          1
2026-09-15 10:00:00.000 GMT 203.0.113.50        7

bin rounds each _time down to its bucket, so stats can count per address per ten minutes. The buckets start on the clock (10:00, 10:10), not at the first event, so a burst straddling 10:10 would be split in two and could stay under five in both halves. The scheduled alert in step 7 has the same edge, because each run looks at one fixed ten-minute stretch.

6. One user, two events, ten minutes (D3)

index=cloudtrail eventName IN (CreateUser, CreateAccessKey)
| transaction requestParameters.userName startswith=eval(eventName="CreateUser") endswith=eval(eventName="CreateAccessKey") maxspan=10m
| table _time requestParameters.userName userIdentity.userName sourceIPAddress duration eventcount
           _time            requestParameters.userName userIdentity.userName sourceIPAddress duration eventcount
--------------------------- -------------------------- --------------------- --------------- -------- ----------
2026-09-15 10:06:12.000 GMT backup-svc                 build-bot             203.0.113.50          39          2

transaction groups events that share the new user's name (requestParameters.userName, the target, as opposed to userIdentity.userName, the actor), starting at CreateUser and ending at CreateAccessKey, and reports the group's duration in seconds: 39. Now delete maxspan=10m and run it again. svc-reporting appears too, with a duration of 9466 seconds: alice created it at 09:02:40 and keyed it 2 hours 37 minutes later, routine work that the window exists to leave out. Exercise 3 writes the same detection with stats.

7. Turn D2 into a scheduled alert

Splunk stores alerts as stanzas in savedsearches.conf: a search, a schedule, a time window and a condition. This one runs every ten minutes over the ten minutes that ended ten minutes ago, leaving time for CloudTrail's delivery delay, and fires when the search returns any row. Save it, install it as a small app, and restart Splunk to load the new app:

cat > savedsearches.conf <<'EOF'
# D2, denied-call burst. MITRE ATT&CK v19.2: T1087.004, T1580.
[d2_denied_call_burst]
description = Five or more denied AWS API calls from one source address in 10 minutes
search = index=cloudtrail errorCode IN (AccessDenied, AccessDeniedException, Client.UnauthorizedOperation, UnauthorizedOperation) \
| stats count BY sourceIPAddress \
| where count >= 5
enableSched = 1
cron_schedule = */10 * * * *
dispatch.earliest_time = -20m@m
dispatch.latest_time = -10m@m
counttype = number of events
relation = greater than
quantity = 0
alert.track = 1
EOF
docker exec -u splunk splunk-lab mkdir -p /opt/splunk/etc/apps/lab_detections/local
docker cp savedsearches.conf splunk-lab:/opt/splunk/etc/apps/lab_detections/local/
docker exec -u splunk splunk-lab /opt/splunk/bin/splunk restart

counttype, relation and quantity together say "more than 0 results", and alert.track lists each firing under Activity > Triggered Alerts in Splunk Web. Plain Splunk Enterprise has no field for MITRE ATT&CK (Adversarial Tactics, Techniques, and Common Knowledge) techniques, so the IDs sit in a comment. Alerts need the Enterprise Trial licence the container started on. Splunk Free, which you can switch to later, still indexes 500 MB a day and runs scheduled searches, but has no alerting and no sign-in at all: anyone who reaches port 8000 is admin.

8. Test the alert both ways

The schedule only ever looks at the last half hour, and the lab is older than that, so run the alert over two stretches of the lab morning and let it act (trigger_actions=1). The window is in epoch seconds: 09:30 to 10:00 UTC holds only reports-app's refusals, and 10:00 to 10:10 holds the attack. Splunk did not honour dates written as 2026-09-15T10:00:00Z here: the job silently searched all time, so a test written that way passes for the wrong reason.

PW=$(cat .splunk-password)
URL=https://localhost:8089/servicesNS/nobody/lab_detections/saved/searches/d2_denied_call_burst/dispatch
curl -sk -u "admin:$PW" -o /dev/null -w '%{http_code}\n' "$URL" -d trigger_actions=1 -d dispatch.earliest_time=1789464600 -d dispatch.latest_time=1789466400
curl -sk -u "admin:$PW" -o /dev/null -w '%{http_code}\n' "$URL" -d trigger_actions=1 -d dispatch.earliest_time=1789466400 -d dispatch.latest_time=1789467000

Each prints 201. Wait a few seconds, then count the firings:

| rest /servicesNS/-/-/alerts/fired_alerts splunk_server=local
| table title triggered_alert_count
       title         triggered_alert_count
-------------------- ---------------------
-                                        1
d2_denied_call_burst                     1

Two runs, one firing (the - row is the total): the attack window fired, the quiet one did not. | rest reads Splunk's own REST API from inside a search.

9. Splunk's public rules: the security_content format

Splunk publishes its detections as YAML files in splunk/security_content, built into an app by its tool contentctl-ng, which writes each one out as a savedsearches.conf stanza like step 7's. This is its D1 rule, trimmed; its ATT&CK ID is already the v19 one:

name: AWS Defense Evasion Stop Logging Cloudtrail
id: 8a2f3ca2-4eb5-4389-a549-14063882e537
version: 12
status: production
search: |-
    `cloudtrail` eventName = StopLogging eventSource = cloudtrail.amazonaws.com userAgent!=console.amazonaws.com errorCode = success
      | rename user_name as user
# (trimmed: the search goes on to count by signature, dest, user, src and vendor fields)
how_to_implement: You must install Splunk AWS Add on and enable Cloudtrail logs in your AWS Environment.
references:
    - https://attack.mitre.org/techniques/T1562/008/
mitre_attack_id:
    - T1685.002
# (trimmed: tests, which points at an attack sample the rule must match)

The file carries its own metadata and, in tests, an attack sample that the rule must match when loaded into a Splunk instance. Three things to notice. ATT&CK rot: mitre_attack_id is T1685.002, and the build copies it into the stanza for Splunk Enterprise Security, but references and the test sample's folder still name T1562.008, the ID that v19 revoked; the name still says Defense Evasion, the tactic v19 split into Stealth and Defense Impairment; and contentctl-ng accepted a planted T1562.008 without complaint. Field names: `cloudtrail` is a macro for sourcetype=aws:cloudtrail, and fields such as src, user and signature come from the Splunk Add-on for AWS, which maps CloudTrail onto Splunk's Common Information Model (CIM). The silent condition: a successful call has no errorCode at all in raw CloudTrail (the rule's own test event has none), so errorCode = success matches only data the add-on has filled in. Run the rule's base search against the raw lab, with the macro replaced by the lab's index:

index=cloudtrail eventName=StopLogging eventSource=cloudtrail.amazonaws.com userAgent!=console.amazonaws.com errorCode=success
| stats count
count
-----
    0

Zero, although the lab contains exactly the event it describes; delete errorCode=success and the count is 1. A published rule is only as good as the field names it assumes, so run its control before you trust its silence. For the record, the five detections in ATT&CK v19.2: D1 T1685.002, D2 T1087.004 and T1580, D3 T1136.003 and T1098.001, D4 T1078.004, D5 T1098.003.

Hands-on exercises

Exercise 1

Write D5: every time the AdministratorAccess managed policy was attached to a user, showing who attached it, to whom, and from where.

Show the answer 
index=cloudtrail eventName=AttachUserPolicy requestParameters.policyArn="arn:aws:iam::aws:policy/AdministratorAccess"
| table _time userIdentity.userName requestParameters.userName sourceIPAddress
           _time            userIdentity.userName requestParameters.userName sourceIPAddress
--------------------------- --------------------- -------------------------- ---------------
2026-09-15 10:06:30.000 GMT build-bot             backup-svc                 203.0.113.50

Two fields are called userName: userIdentity.userName is who acted, requestParameters.userName who was acted on. The quotes around the policy ARN (Amazon Resource Name) keep its colons and slash as one value. For real coverage, add AttachRolePolicy, AttachGroupPolicy and PutUserPolicy, which grant the same power by other routes.

Exercise 2

This search returns nothing. Find both faults, and say how you would prove each one rather than guess.

eventName=StopLogging
| where userIdentity.userName="backup-svc"
| table _time eventName userIdentity.userName
Show the answer

First, no index: Splunk searched only the default indexes. Second, the unquoted dotted name in where, which joins two empty strings. Fixing either one alone still returns nothing, which is why guessing fails here. Prove it by deleting from the end: eventName=StopLogging alone returns nothing (the index), index=cloudtrail eventName=StopLogging returns the event (so the where is at fault).

index=cloudtrail eventName=StopLogging
| where 'userIdentity.userName'="backup-svc"
| table _time eventName userIdentity.userName

That returns the StopLogging event from step 3.

Exercise 3

Write D3 without transaction, using stats, so that svc-reporting stays out. Then say what each condition protects against.

Show the answer 
index=cloudtrail eventName IN (CreateUser, CreateAccessKey)
| stats earliest(eventName) AS first_step dc(eventName) AS steps range(_time) AS gap BY requestParameters.userName
| where steps=2 AND first_step="CreateUser" AND gap<=600
requestParameters.userName first_step steps gap
-------------------------- ---------- ----- ---
backup-svc                 CreateUser     2  39

dc counts distinct values, so steps=2 needs both events; earliest gives the first event's name, so the user must be created before the key; range(_time) is last minus first, in seconds, and gap<=600 is the ten-minute window. Delete the where line and svc-reporting appears with a gap of 9466. stats keeps one row per user instead of whole groups of events, which is usually lighter on a large index.

Exercise 4

Find the source addresses used by more than one identity. Then explain why counting userIdentity.userName instead reports 0 identities for two of the addresses.

Show the answer 
index=cloudtrail
| stats dc(userIdentity.arn) AS identities values(userIdentity.userName) AS users BY sourceIPAddress
| eval users=mvjoin(users, ", ")
| where identities > 1
sourceIPAddress identities         users
--------------- ---------- ---------------------
203.0.113.50             2 backup-svc, build-bot

The leaked key and the user it created, from one address. values collects distinct values into one multivalue field and mvjoin prints them on one line. Roles have no userName, so dc(userIdentity.userName) gives 0 for 192.0.2.10 (ci-deploy) and 192.0.2.20 (reports-app). Count a field every event has, such as userIdentity.arn (the ARN).

Common mistakes

Where next