Time: ~2 hours · Difficulty: Intermediate · You need: Docker (Docker Desktop is fine), jq and a terminal. No cloud account: the lab is a file.
How this page was checked: every command and rule on this page was run with Falco 0.45.0 and falcoctl 0.14.2 in Docker Desktop 4.86.0 on an Apple silicon Mac, using the projects' native arm64 images (the amd64 images were not tested), and the outputs shown are what they printed. Step 8 ran Falco's kernel probe against Docker Desktop's Linux kernel (6.12.76-linuxkit). falcosidekick, named in step 6, was not run.
Rules that judge each event as it happens
Falco is an open-source (Apache 2.0) runtime security tool for Linux, created by Sysdig and now a graduated project of the Cloud Native Computing Foundation (CNCF). Its usual job is watching system calls, the requests every program makes to the Linux kernel, and raising an alert the moment one matches a rule. Plugins let the same engine read other event streams, among them Kubernetes audit logs, Okta, GitHub and, on this page, AWS (Amazon Web Services) CloudTrail.
A Falco rule is a few lines of YAML: a condition that is true or false for one event, a line of output to print when it is true, and a priority. Platform and security engineers write them, often starting from the stock rules the Falco project publishes, and keep them in git like any other code.
The one distinction that matters: Falco is not a search engine. It keeps no store of past events, and no rule can look back or count. Each event is judged alone as it arrives, then forgotten. That is what lets Falco alert in real time, and it is why two of the five detections on this page have to be finished somewhere else.
On this page
One event in, one verdict out
Everything in Falco follows from one idea: a rule is a test applied to each event on its own. Events arrive one at a time, Falco checks each against the loaded rules, and when a rule's condition is true it prints that rule's output with the event's values filled in. Then it moves on and remembers nothing. This is a complete rule for CloudTrail events:
- rule: Trail stopped desc: Someone called StopLogging. condition: ct.name = "StopLogging" output: StopLogging by %ct.user from %ct.srcip priority: CRITICAL source: aws_cloudtrail
ct.name is a field: a named value extracted from each event, here the CloudTrail eventName. In output, a field name after % is replaced by the event's value. source says which stream the rule reads. Against the lab's 24 events the condition is tested 24 times and is true once, which prints:
10:09:03.000000000: Critical StopLogging by backup-svc from 203.0.113.50
Notice what a condition cannot say. It sees one event, so there is no "five times", no "within 10 minutes" and no "after a CreateUser". Counting and correlation happen in whatever receives Falco's alerts.
Walk-through
The lab is 24 CloudTrail events. A continuous integration (CI) user's access key, build-bot, has leaked. From 203.0.113.50 someone checks whose key it is, tries to list Identity and Access Management (IAM) users, EC2 (Elastic Compute Cloud) resources and secrets, and is mostly refused. They list the storage buckets, create a user called backup-svc, make it an administrator, give it an access key and use it to stop CloudTrail logging. Around that sits ordinary activity: an administrator creating a service user, a sign-in without MFA (multi-factor authentication), a failed sign-in, and an application refused the same file three times.
Five detections run through every guide in this series:
- D1, trail tampering: CloudTrail logging stopped, deleted or changed.
- D2, denied-call burst: one source address refused five or more times.
- D3, new user given a key: a user created and handed an access key within 10 minutes.
- D4, console login without MFA: a successful console sign-in without multi-factor authentication.
- D5, administrator policy attached:
AdministratorAccessattached to a user.
D1, D4 and D5 each look at one event, so each is a Falco rule. D2 needs a count and D3 needs two events; step 6 shows what Falco does instead.
1. Get Falco, falcoctl and the plugins
Everything runs in containers, and the official images are published for both Intel/AMD and Arm processors, so an Apple silicon Mac runs them natively. falcoctl, the Falco project's tool for rules and plugins, installs the cloudtrail plugin (reads CloudTrail files), the json plugin (reads any value inside an event) and the stock CloudTrail rules used in step 7:
mkdir -p ~/falco-lab/plugins ~/falco-lab/rules && cd ~/falco-lab docker run --rm --name csoh-lab-falco falcosecurity/falco:0.45.0 falco --version docker run --rm --name csoh-lab-falcoctl -u 0 -v "$PWD":/lab \ falcosecurity/falcoctl:0.14.2 artifact install \ cloudtrail:0.17.1 json:0.7.4 cloudtrail-rules:0.17.1 \ --plugins-dir /lab/plugins --rulesfiles-dir /lab/rules
Expect Falco version: 0.45.0 and Engine: 0.65.0 among the output, then Signature successfully verified! and Artifact successfully installed once per artifact. -u 0 runs falcoctl as root, because its image's default user cannot create the configuration directory falcoctl needs. plugins/ now holds libcloudtrail.so and libjson.so, built for your machine's architecture, and rules/ holds aws_cloudtrail_rules.yaml.
2. Load the lab and give it CloudTrail's file shape
The lab file holds one JSON (JavaScript Object Notation) event per line. CloudTrail itself writes files holding one object, {"Records": [...]}, and that is what the cloudtrail plugin reads, from files whose names end in .json. One jq command converts it (the jq guide explains -s and map):
cat > cloudtrail-lab.jsonl <<'EOF'
{"eventTime":"2026-09-15T08:55:12Z","eventSource":"signin.amazonaws.com","eventName":"ConsoleLogin","awsRegion":"us-east-1","sourceIPAddress":"198.51.100.20","userAgent":"Mozilla/5.0","userIdentity":{"type":"IAMUser","arn":"arn:aws:iam::111122223333:user/alice","userName":"alice"},"responseElements":{"ConsoleLogin":"Success"},"additionalEventData":{"MFAUsed":"Yes"}}
{"eventTime":"2026-09-15T09:02:40Z","eventSource":"iam.amazonaws.com","eventName":"CreateUser","awsRegion":"us-east-1","sourceIPAddress":"198.51.100.20","userAgent":"AWS Internal","userIdentity":{"type":"IAMUser","arn":"arn:aws:iam::111122223333:user/alice","userName":"alice"},"requestParameters":{"userName":"svc-reporting"},"readOnly":false}
{"eventTime":"2026-09-15T09:03:05Z","eventSource":"s3.amazonaws.com","eventName":"ListBuckets","awsRegion":"us-east-1","sourceIPAddress":"198.51.100.20","userAgent":"AWS Internal","userIdentity":{"type":"IAMUser","arn":"arn:aws:iam::111122223333:user/alice","userName":"alice"},"readOnly":true}
{"eventTime":"2026-09-15T09:10:22Z","eventSource":"ec2.amazonaws.com","eventName":"DescribeInstances","awsRegion":"us-east-1","sourceIPAddress":"192.0.2.10","userAgent":"aws-cli/2.17.40","userIdentity":{"type":"AssumedRole","arn":"arn:aws:sts::111122223333:assumed-role/ci-deploy/runner-4821"},"readOnly":true}
{"eventTime":"2026-09-15T09:15:47Z","eventSource":"signin.amazonaws.com","eventName":"ConsoleLogin","awsRegion":"us-east-1","sourceIPAddress":"198.51.100.21","userAgent":"Mozilla/5.0","userIdentity":{"type":"IAMUser","arn":"arn:aws:iam::111122223333:user/bob","userName":"bob"},"responseElements":{"ConsoleLogin":"Success"},"additionalEventData":{"MFAUsed":"No"}}
{"eventTime":"2026-09-15T09:20:03Z","eventSource":"signin.amazonaws.com","eventName":"ConsoleLogin","awsRegion":"us-east-1","sourceIPAddress":"192.0.2.77","userAgent":"Mozilla/5.0","userIdentity":{"type":"IAMUser","arn":"arn:aws:iam::111122223333:user/alice","userName":"alice"},"errorMessage":"Failed authentication","responseElements":{"ConsoleLogin":"Failure"},"additionalEventData":{"MFAUsed":"No"}}
{"eventTime":"2026-09-15T09:31:10Z","eventSource":"s3.amazonaws.com","eventName":"GetObject","awsRegion":"us-east-1","sourceIPAddress":"192.0.2.20","userAgent":"Boto3/1.35.10","userIdentity":{"type":"AssumedRole","arn":"arn:aws:sts::111122223333:assumed-role/reports-app/i-0a12b34c56d78e90f"},"errorCode":"AccessDenied","requestParameters":{"bucketName":"finance-reports","key":"2026/09/summary.csv"},"readOnly":true}
{"eventTime":"2026-09-15T09:47:55Z","eventSource":"s3.amazonaws.com","eventName":"GetObject","awsRegion":"us-east-1","sourceIPAddress":"192.0.2.20","userAgent":"Boto3/1.35.10","userIdentity":{"type":"AssumedRole","arn":"arn:aws:sts::111122223333:assumed-role/reports-app/i-0a12b34c56d78e90f"},"errorCode":"AccessDenied","requestParameters":{"bucketName":"finance-reports","key":"2026/09/summary.csv"},"readOnly":true}
{"eventTime":"2026-09-15T10:01:30Z","eventSource":"s3.amazonaws.com","eventName":"GetObject","awsRegion":"us-east-1","sourceIPAddress":"192.0.2.20","userAgent":"Boto3/1.35.10","userIdentity":{"type":"AssumedRole","arn":"arn:aws:sts::111122223333:assumed-role/reports-app/i-0a12b34c56d78e90f"},"errorCode":"AccessDenied","requestParameters":{"bucketName":"finance-reports","key":"2026/09/summary.csv"},"readOnly":true}
{"eventTime":"2026-09-15T10:02:05Z","eventSource":"sts.amazonaws.com","eventName":"GetCallerIdentity","awsRegion":"us-east-1","sourceIPAddress":"203.0.113.50","userAgent":"aws-cli/2.17.40","userIdentity":{"type":"IAMUser","arn":"arn:aws:iam::111122223333:user/build-bot","userName":"build-bot"},"readOnly":true}
{"eventTime":"2026-09-15T10:02:31Z","eventSource":"iam.amazonaws.com","eventName":"ListUsers","awsRegion":"us-east-1","sourceIPAddress":"203.0.113.50","userAgent":"aws-cli/2.17.40","userIdentity":{"type":"IAMUser","arn":"arn:aws:iam::111122223333:user/build-bot","userName":"build-bot"},"errorCode":"AccessDenied","readOnly":true}
{"eventTime":"2026-09-15T10:02:38Z","eventSource":"iam.amazonaws.com","eventName":"ListRoles","awsRegion":"us-east-1","sourceIPAddress":"203.0.113.50","userAgent":"aws-cli/2.17.40","userIdentity":{"type":"IAMUser","arn":"arn:aws:iam::111122223333:user/build-bot","userName":"build-bot"},"errorCode":"AccessDenied","readOnly":true}
{"eventTime":"2026-09-15T10:02:52Z","eventSource":"iam.amazonaws.com","eventName":"GetAccountAuthorizationDetails","awsRegion":"us-east-1","sourceIPAddress":"203.0.113.50","userAgent":"aws-cli/2.17.40","userIdentity":{"type":"IAMUser","arn":"arn:aws:iam::111122223333:user/build-bot","userName":"build-bot"},"errorCode":"AccessDenied","readOnly":true}
{"eventTime":"2026-09-15T10:03:05Z","eventSource":"ec2.amazonaws.com","eventName":"DescribeInstances","awsRegion":"us-east-1","sourceIPAddress":"203.0.113.50","userAgent":"aws-cli/2.17.40","userIdentity":{"type":"IAMUser","arn":"arn:aws:iam::111122223333:user/build-bot","userName":"build-bot"},"errorCode":"Client.UnauthorizedOperation","readOnly":true}
{"eventTime":"2026-09-15T10:03:09Z","eventSource":"ec2.amazonaws.com","eventName":"DescribeVpcs","awsRegion":"us-east-1","sourceIPAddress":"203.0.113.50","userAgent":"aws-cli/2.17.40","userIdentity":{"type":"IAMUser","arn":"arn:aws:iam::111122223333:user/build-bot","userName":"build-bot"},"errorCode":"Client.UnauthorizedOperation","readOnly":true}
{"eventTime":"2026-09-15T10:03:14Z","eventSource":"ec2.amazonaws.com","eventName":"DescribeSecurityGroups","awsRegion":"us-east-1","sourceIPAddress":"203.0.113.50","userAgent":"aws-cli/2.17.40","userIdentity":{"type":"IAMUser","arn":"arn:aws:iam::111122223333:user/build-bot","userName":"build-bot"},"errorCode":"Client.UnauthorizedOperation","readOnly":true}
{"eventTime":"2026-09-15T10:03:22Z","eventSource":"secretsmanager.amazonaws.com","eventName":"ListSecrets","awsRegion":"us-east-1","sourceIPAddress":"203.0.113.50","userAgent":"aws-cli/2.17.40","userIdentity":{"type":"IAMUser","arn":"arn:aws:iam::111122223333:user/build-bot","userName":"build-bot"},"errorCode":"AccessDenied","readOnly":true}
{"eventTime":"2026-09-15T10:03:40Z","eventSource":"s3.amazonaws.com","eventName":"ListBuckets","awsRegion":"us-east-1","sourceIPAddress":"203.0.113.50","userAgent":"aws-cli/2.17.40","userIdentity":{"type":"IAMUser","arn":"arn:aws:iam::111122223333:user/build-bot","userName":"build-bot"},"readOnly":true}
{"eventTime":"2026-09-15T10:06:12Z","eventSource":"iam.amazonaws.com","eventName":"CreateUser","awsRegion":"us-east-1","sourceIPAddress":"203.0.113.50","userAgent":"aws-cli/2.17.40","userIdentity":{"type":"IAMUser","arn":"arn:aws:iam::111122223333:user/build-bot","userName":"build-bot"},"requestParameters":{"userName":"backup-svc"},"readOnly":false}
{"eventTime":"2026-09-15T10:06:30Z","eventSource":"iam.amazonaws.com","eventName":"AttachUserPolicy","awsRegion":"us-east-1","sourceIPAddress":"203.0.113.50","userAgent":"aws-cli/2.17.40","userIdentity":{"type":"IAMUser","arn":"arn:aws:iam::111122223333:user/build-bot","userName":"build-bot"},"requestParameters":{"userName":"backup-svc","policyArn":"arn:aws:iam::aws:policy/AdministratorAccess"},"readOnly":false}
{"eventTime":"2026-09-15T10:06:51Z","eventSource":"iam.amazonaws.com","eventName":"CreateAccessKey","awsRegion":"us-east-1","sourceIPAddress":"203.0.113.50","userAgent":"aws-cli/2.17.40","userIdentity":{"type":"IAMUser","arn":"arn:aws:iam::111122223333:user/build-bot","userName":"build-bot"},"requestParameters":{"userName":"backup-svc"},"responseElements":{"accessKey":{"userName":"backup-svc","accessKeyId":"AKIAIOSFODNN7EXAMPLE","status":"Active"}},"readOnly":false}
{"eventTime":"2026-09-15T10:09:03Z","eventSource":"cloudtrail.amazonaws.com","eventName":"StopLogging","awsRegion":"us-east-1","sourceIPAddress":"203.0.113.50","userAgent":"aws-cli/2.17.40","userIdentity":{"type":"IAMUser","arn":"arn:aws:iam::111122223333:user/backup-svc","userName":"backup-svc"},"requestParameters":{"name":"arn:aws:cloudtrail:us-east-1:111122223333:trail/org-trail"},"readOnly":false}
{"eventTime":"2026-09-15T10:40:18Z","eventSource":"ec2.amazonaws.com","eventName":"DescribeInstances","awsRegion":"us-east-1","sourceIPAddress":"192.0.2.10","userAgent":"aws-cli/2.17.40","userIdentity":{"type":"AssumedRole","arn":"arn:aws:sts::111122223333:assumed-role/ci-deploy/runner-4821"},"readOnly":true}
{"eventTime":"2026-09-15T11:40:26Z","eventSource":"iam.amazonaws.com","eventName":"CreateAccessKey","awsRegion":"us-east-1","sourceIPAddress":"198.51.100.20","userAgent":"AWS Internal","userIdentity":{"type":"IAMUser","arn":"arn:aws:iam::111122223333:user/alice","userName":"alice"},"requestParameters":{"userName":"svc-reporting"},"responseElements":{"accessKey":{"userName":"svc-reporting","accessKeyId":"AKIAI44QH8DHBEXAMPLE","status":"Active"}},"readOnly":false}
EOF
shasum -a 256 cloudtrail-lab.jsonl
jq -s '{Records: map(.eventType = (if .eventName == "ConsoleLogin" then "AwsConsoleSignIn"
else "AwsApiCall" end))}' cloudtrail-lab.jsonl > cloudtrail-lab.jsonshasum should print 35b7c80582d0650ec4850acf84f84fa4b8e5d37169ac3f7276c76f29b67c3319, the hash of the lab file every guide in this series uses. -s reads the 24 lines into one array, and map() gives each record an eventType.
Without eventType, Falco reads nothing and says nothing. AWS documents eventType as present on every CloudTrail record, and the plugin skips any record without one. The lab file was trimmed of it, so the conversion puts it back. Leave that out and step 3 reports Events detected: 0, with no error and an exit code of 0: the same thing a quiet account looks like.
3. Point Falco at the file, and look at everything first
A Falco configuration file says where events come from and where alerts go. Save this as falco-lab.yaml:
# falco-lab.yaml: read one CloudTrail file instead of the kernel
plugins:
- name: cloudtrail
library_path: /lab/plugins/libcloudtrail.so
open_params: cloudtrail-lab.json # a file path: read it, then stop
- name: json
library_path: /lab/plugins/libjson.so
load_plugins: [cloudtrail, json]
plugins_hostinfo: false # CloudTrail events need no host /proc
stdout_output:
enabled: true # print alerts in the terminalEvery Falco run from here to step 7 is the same container, so wrap it in a shell function. It mounts this folder at /lab, which is why the paths above start there, and --disable-source syscall stops Falco trying to watch the kernel as well; step 8 does that.
falco() {
docker run --rm --name csoh-lab-falco -v "$PWD":/lab -w /lab \
falcosecurity/falco:0.45.0 falco -c falco-lab.yaml --disable-source syscall "$@"
}Before any detection, write a control rule that matches every event, so you can see what Falco actually receives. Save it as everything.yaml:
# everything.yaml: a control rule that matches every CloudTrail event - rule: Every event desc: Control rule. Shows what arrived, before any detection logic. condition: evt.num > 0 output: "%ct.name from %ct.srcip error=%ct.error" priority: DEBUG source: aws_cloudtrail
falco -r everything.yaml | tail -5
Events detected: 24 Rule counts by severity: DEBUG: 24 Triggered rules by rule name: Every event: 24
All 24 arrived (tail -5 keeps only Falco's closing summary). -r names the rules file to load, in place of the default ones. evt.num is each event's sequence number, so evt.num > 0 is always true; the stock CloudTrail file uses the same condition for a rule it ships switched off. Run it again without tail to see the 24 alerts: each starts with the event's own time and the rule's priority, and the 14 events with no error code end in error=<NA>, which is how Falco prints a field the event does not have. Leave out source: aws_cloudtrail and Falco assumes syscall, the kernel, and rejects the rule: ct. fields do not exist there.
4. A list and a macro: trail tampering (D1)
Save this as d1.yaml:
# d1.yaml: D1, CloudTrail logging stopped, deleted or changed
- list: trail_tampering_calls
items: [StopLogging, DeleteTrail, UpdateTrail, PutEventSelectors]
- macro: call_succeeded
condition: (not ct.error exists)
- rule: CloudTrail logging tampered with
desc: A call that stops, deletes or changes a CloudTrail trail succeeded.
condition: ct.name in (trail_tampering_calls) and call_succeeded
output: >
CloudTrail logging tampered with (call=%ct.name user=%ct.user
ip=%ct.srcip trail=%ct.request.name)
priority: CRITICAL
source: aws_cloudtrail
tags: [aws, cloudtrail, mitre_defense_impairment, T1685.002]falco -r d1.yaml
From here on, the outputs shown leave out Falco's closing summary:
10:09:03.000000000: Critical CloudTrail logging tampered with (call=StopLogging user=backup-svc ip=203.0.113.50 trail=arn:aws:cloudtrail:us-east-1:111122223333:trail/org-trail)
One alert, backup-svc at 10:09:03; the other 23 events are the negative case and none fires. A list is a named set of values, and in is true when the field equals any of them. A macro is a named piece of condition that rules reuse. exists is true when the event has the field at all, so not ct.error exists means the call did not fail. output: > is YAML's folded style, which joins the two lines into one. The priority is a severity label, one of EMERGENCY, ALERT, CRITICAL, ERROR, WARNING, NOTICE, INFORMATIONAL and DEBUG; it does not change the order in which rules run.
Tags are free text. Falco's style guide asks rules contributed to the project to carry the MITRE ATT&CK (Adversarial Tactics, Techniques, and Common Knowledge) tactic and technique, and yours can do the same. This page uses ATT&CK v19.2, which revoked T1562.008 and files cloud-log tampering as T1685.002 Disable or Modify Cloud Log, under the Defense Impairment tactic.
5. Nested fields: console logins without MFA (D4)
The plugin's ct. fields cover the common parts of an event. For anything else, the json plugin's json.value[...] takes a JSON Pointer: a path from the top of the event, one / per level, with case-sensitive keys. Save d4.yaml, a naive rule and a correct one:
# d4.yaml: D4, a successful console sign-in without MFA, and a naive version
- rule: Console login without MFA (naive)
desc: Checks only the MFA flag.
condition: ct.name = "ConsoleLogin" and json.value[/additionalEventData/MFAUsed] = "No"
output: Naive rule (user=%ct.user ip=%ct.srcip result=%json.value[/responseElements/ConsoleLogin])
priority: WARNING
source: aws_cloudtrail
- rule: Console login without MFA
desc: A console sign-in that succeeded without multi-factor authentication.
condition: >
ct.name = "ConsoleLogin"
and json.value[/responseElements/ConsoleLogin] = "Success"
and json.value[/additionalEventData/MFAUsed] = "No"
output: Console login without MFA (user=%ct.user ip=%ct.srcip)
priority: WARNING
source: aws_cloudtrail
tags: [aws, iam, mitre_initial_access, T1078.004]falco -r d4.yaml -o rule_matching=all
09:15:47.000000000: Warning Naive rule (user=bob ip=198.51.100.21 result=Success) 09:15:47.000000000: Warning Console login without MFA (user=bob ip=198.51.100.21) 09:20:03.000000000: Warning Naive rule (user=alice ip=192.0.2.77 result=Failure)
The correct rule fires once, for bob. The naive rule also fires for alice at 09:20:03: her sign-in failed, and her failed sign-in records MFAUsed as No too. It carries no errorCode either, so the call_succeeded macro would not have caught it; the verdict is in responseElements.ConsoleLogin. -o sets one configuration option for this run, and rule_matching=all lets both rules fire on bob's event. By default only the first rule to match an event fires, which matters in step 7.
6. What Falco cannot do: count (D2) and pair (D3)
D2 asks whether one source address was refused five or more times. That is a count across events, and a condition sees one event. What Falco can do is raise one alert per refused call and hand the alerts, as JSON, to something that counts: a SIEM (security information and event management system), a log store or, here, the shell. Start with the obvious rule: the denial codes go in a list, and the list holds the one code everybody knows. Save d2.yaml:
# d2.yaml: one alert per denied call - list: denied_error_codes items: [AccessDenied] - rule: API call denied desc: A refused call. Count these per source address downstream. condition: ct.error in (denied_error_codes) output: API call denied (call=%ct.name error=%ct.error ip=%ct.srcip) priority: NOTICE source: aws_cloudtrail tags: [aws, mitre_discovery, T1087.004, T1580]
With -o json_output=true each alert is a JSON object; jq pulls out the source address, uniq -c counts, and awk keeps counts of five or more:
falco -r d2.yaml -o json_output=true \ | jq -Rr 'fromjson? | .output_fields["ct.srcip"]' | sort | uniq -c | awk '$1 >= 5'
No output. Delete the awk and the counts appear: 4 for 203.0.113.50 and 3 for 192.0.2.20, so nobody reaches five. (fromjson? keeps the JSON alert lines and skips Falco's plain-text summary.) Did nothing happen, or can the rule not see what happened?
A rule that returns nothing and a rule that cannot work return the same thing. Before you believe silence, run a control that must return something. The control rule from step 3 sees every event, so ask it which error codes exist at all.
falco -r everything.yaml -o json_output=true \ | jq -Rr 'fromjson? | .output_fields["ct.error"]' | sort | uniq -c
7 AccessDenied 3 Client.UnauthorizedOperation 14 null
EC2 reports a refusal as Client.UnauthorizedOperation, so three of the attacker's seven refusals were invisible to the rule. (null is JSON's way of showing the 14 events with no error code.) Other services use AccessDeniedException, and UnauthorizedOperation also appears without the prefix; AWS's own Security Hub control for unauthorized API calls matches both families, AccessDenied* and *UnauthorizedOperation. Change the list in d2.yaml to items: [AccessDenied, AccessDeniedException, Client.UnauthorizedOperation, UnauthorizedOperation] and run the same count:
falco -r d2.yaml -o json_output=true \ | jq -Rr 'fromjson? | .output_fields["ct.srcip"]' | sort | uniq -c | awk '$1 >= 5'
7 203.0.113.50
The rule now fires ten times, once per refusal; 203.0.113.50 has seven and crosses the threshold, and 192.0.2.20 stays at three, the negative case. In production the counting moves to a SIEM or log store: Falco sends its JSON alerts over HTTP (http_output) to falcosidekick, the project's forwarder, which passes them on to outputs such as Elasticsearch, Loki, Sumo Logic and Datadog Logs, and the threshold and the 10-minute window are that system's job.
D3, a user created and given an access key within 10 minutes, pairs two events by the new user's name, and no Falco rule can refer to an earlier event. Falco can alert on each half, as Exercise 3 does, and the system that receives both alerts does the pairing; the Splunk and Kusto guides show that part.
7. Stock rules, shadowing, and tuning by override
You rarely start from nothing. Run the stock CloudTrail rules from step 1, followed by your D1 rule:
falco -r rules/aws_cloudtrail_rules.yaml -r d1.yaml
09:02:40.000000000: Informational A new AWS user has been created (requesting user=alice, requesting IP=198.51.100.20, AWS region=us-east-1, new user created=svc-reporting) 09:15:47.000000000: Critical Detected a console login without MFA (requesting user=bob, requesting IP=198.51.100.21, AWS region=us-east-1) 10:06:12.000000000: Informational A new AWS user has been created (requesting user=build-bot, requesting IP=203.0.113.50, AWS region=us-east-1, new user created=backup-svc) 10:09:03.000000000: Warning The CloudTrail logging has been disabled. (requesting user=backup-svc, requesting IP=203.0.113.50, AWS region=us-east-1, resource name=arn:aws:cloudtrail:us-east-1:111122223333:trail/org-trail)
Your D1 rule is loaded, valid and silent. Falco's default, rule_matching: first, lets an event fire only the first rule it matches, in load order, and the stock CloudTrail Logging Disabled rule loads first. No warning says so. You could set rule_matching: all, which Falco's configuration file warns can cost performance, or switch the overlapping stock rule off; it only covers StopLogging anyway. And alice creating service users is routine, so the stock Create AWS user rule needs an exception. Never edit the stock file, because falcoctl replaces it on update: override it from a file loaded after it. Save tuning.yaml:
# tuning.yaml: change stock rules without editing their file
- rule: CloudTrail Logging Disabled # d1.yaml covers this and more
enabled: false
override:
enabled: replace
- rule: Create AWS user # alice creating svc- users is routine
exceptions:
- name: routine_service_users
fields: [ct.user, ct.request.username]
comps: [=, startswith]
values:
- [alice, svc-]
override:
exceptions: appendfalco -r rules/aws_cloudtrail_rules.yaml -r tuning.yaml -r d1.yaml
09:15:47.000000000: Critical Detected a console login without MFA (requesting user=bob, requesting IP=198.51.100.21, AWS region=us-east-1) 10:06:12.000000000: Informational A new AWS user has been created (requesting user=build-bot, requesting IP=203.0.113.50, AWS region=us-east-1, new user created=backup-svc) 10:09:03.000000000: Critical CloudTrail logging tampered with (call=StopLogging user=backup-svc ip=203.0.113.50 trail=arn:aws:cloudtrail:us-east-1:111122223333:trail/org-trail)
alice's routine user is gone, build-bot's is still there, and your D1 rule now gets its event. override says what to do with each key: replace swaps the value and append adds to it. An exception lists fields, one comparison per field (comps) and rows of values; the rule stays quiet for an event that matches a whole row. This one names the actor and the target, as Falco's documentation advises: an exception for alice alone would also hide an attacker who stole her credentials and created backup-svc. In a real deployment, list override files after the stock ones in the configuration's rules_files; the default order reads /etc/falco/falco_rules.yaml, then falco_rules.local.yaml, then the /etc/falco/rules.d folder.
8. The same language on system calls
Falco's usual source is the kernel. Its default driver, the modern eBPF (extended Berkeley Packet Filter) probe, attaches to a running Linux kernel and turns system calls into events. Falco's Docker quickstart says its command will not work on Docker Desktop. On Docker Desktop 4.86.0 for Apple silicon the commands below worked, with warnings that a safeguard against time-of-check to time-of-use (TOCTOU) tricks could not attach, which Falco says does not stop detection. If they fail for you, use a Linux machine. Either way, a privileged container can do, in Docker's words, almost everything that the host can do, so use a machine you own. Save shadow.yaml, then start Falco privileged, so the probe can load, with the Docker socket so it can name containers, and give the probe ten seconds to attach:
# shadow.yaml: a system-call rule; with no source key, the source is syscall
- rule: Shadow file read in a container
desc: A process in a container opened /etc/shadow, which holds password hashes, for reading.
condition: >
evt.type in (open, openat, openat2) and evt.is_open_read = true
and fd.name = /etc/shadow and container.id != host
output: /etc/shadow read in a container (command=%proc.cmdline)
priority: WARNING
tags: [container, filesystem, mitre_credential_access, T1003.008]docker run -d --name csoh-lab-falco-kernel --privileged \ -v /var/run/docker.sock:/host/var/run/docker.sock -v /proc:/host/proc:ro \ -v "$PWD":/lab falcosecurity/falco:0.45.0 falco -r /lab/shadow.yaml sleep 10
Now test it both ways: one container reads /etc/passwd, the negative case, and one reads /etc/shadow, the positive. Falco writes its own log to standard error, so 2> /dev/null leaves only the alerts:
docker run --rm --name csoh-lab-falco-target alpine:3.24 cat /etc/passwd > /dev/null docker run --rm --name csoh-lab-falco-target alpine:3.24 cat /etc/shadow > /dev/null sleep 2 docker logs csoh-lab-falco-kernel 2> /dev/null docker rm -f csoh-lab-falco-kernel > /dev/null
2026-09-27T18:57:33.415876851+0000: Warning /etc/shadow read in a container (command=cat /etc/shadow) container_id=bb8c4ab79ace container_name=csoh-lab-falco-target container_image_repository=alpine container_image_tag=3.24 k8s_pod_name=<NA> k8s_ns_name=<NA>
One alert, for the shadow read; the passwd read stays silent. The fields after the closing parenthesis did not come from your rule: the image's default configuration appends the container plugin's suggested fields to every system-call alert (append_output with suggested_output: true). A system-call condition should open with evt.type: Falco uses it to test each rule against only those calls, and warns (LOAD_NO_EVTTYPE) about a rule without one. The image's stock rules, loaded when you pass no -r, include a broader version of this rule, Read sensitive file untrusted.
Hands-on exercises
Write D5: alert whenever the AdministratorAccess managed policy is attached to a user, showing who attached it, to whom and from where. Check that it fires once and nothing else does.
Show the answer
# d5.yaml: D5, AdministratorAccess attached to a user
- rule: Administrator policy attached to a user
desc: The AdministratorAccess managed policy was attached to an IAM user.
condition: >
ct.name = "AttachUserPolicy" and not ct.error exists
and json.value[/requestParameters/policyArn] = "arn:aws:iam::aws:policy/AdministratorAccess"
output: Administrator policy attached (by=%ct.user to=%ct.request.username ip=%ct.srcip)
priority: CRITICAL
source: aws_cloudtrail
tags: [aws, iam, mitre_privilege_escalation, T1098.003]Run it with falco -r d5.yaml:
10:06:30.000000000: Critical Administrator policy attached (by=build-bot to=backup-svc ip=203.0.113.50)
One alert, and the summary says Events detected: 1, so the other 23 events stayed silent. The plugin has a field called iam.policy that looks made for this, and a rule on it never fires: it reads requestParameters.policyName, and AttachUserPolicy sends policyArn instead. Print a field in an output before you filter on it (%iam.policy prints <NA> here). For real coverage, add AttachRolePolicy, AttachGroupPolicy and the inline PutUserPolicy.
You want an alert for every call that succeeded. The plugin's field list says ct.error "will be <NA>" when there was no error, so you write condition: ct.error = "<NA>". Falco loads it, reads all 24 events and reports Events detected: 0. Why, and what should the condition be?
Show the answer
<NA> is how Falco prints a field that has no value. It is never the value itself, and any comparison with a missing field is false: even ct.error != "<NA>" matches only the 10 events that have an error code. Test for the field instead. not ct.error exists matches 14 events, and one of them is alice's failed sign-in, which has no error code (step 5), so "did not fail" is not the same as "succeeded" for every event type. The control rule in step 3, which prints error=%ct.error for every event, is how you would have seen <NA> for what it is before writing the condition.
This rule is meant to catch build-bot creating users or access keys, the two halves of D3. It fires three times. Which alert should not be there, and why?
# ex3.yaml: build-bot creating users or keys - rule: New users and keys by build-bot desc: CreateUser or CreateAccessKey calls made by the build-bot key. condition: ct.name = "CreateUser" or ct.name = "CreateAccessKey" and ct.user = "build-bot" output: "%ct.name by %ct.user for %ct.request.username" priority: WARNING source: aws_cloudtrail
Show the answer
alice's CreateUser at 09:02:40. and binds more tightly than or, so Falco reads the condition as ct.name = "CreateUser" or (ct.name = "CreateAccessKey" and ct.user = "build-bot"): every CreateUser matches, whoever made it. Change the condition to ct.name in (CreateUser, CreateAccessKey) and ct.user = "build-bot" and run it again:
falco -r ex3.yaml
10:06:12.000000000: Warning CreateUser by build-bot for backup-svc 10:06:51.000000000: Warning CreateAccessKey by build-bot for backup-svc
Those two alerts, 39 seconds apart and naming the same new user, are what a SIEM pairs to finish D3.
In step 7 a colleague fixes the shadowing another way: they load d1.yaml before the stock file, falco -r d1.yaml -r rules/aws_cloudtrail_rules.yaml, and leave the stock rule alone. D1 now fires. Why does that work, and why is the override in tuning.yaml the better fix?
Show the answer
First match again: your rule now loads first and claims the StopLogging event, so the stock CloudTrail Logging Disabled rule goes quiet instead. Run it and that stock alert has gone. It works by accident of load order, and it silences a rule without saying so anywhere; whoever later reorders the files, or loads another file first, changes which rule fires, and nothing warns them. enabled: false in an override states the intent in a file a reviewer reads, and survives any reordering that keeps it after the stock file.
Common mistakes
- Believing a silent run. Records without
eventType, a rule shadowed by an earlier one, and a rule below the configured minimumpriorityall end inEvents detected: 0or a missing alert, with exit code 0. Run a match-everything control first. - Adding a rule that overlaps a stock rule. With the default
rule_matching: first, whichever loads first claims the event, and the other never fires. - Trusting a field's name.
iam.policyreadspolicyName, not the policy's ARN (Amazon Resource Name), andjson.valuekeys are case-sensitive. Print a field before filtering on it. - Validating plugin rules without the plugins.
falco -V d1.yamlwith the lab configuration saysd1.yaml: Ok. Without the plugins it saysOk, with warnings, exits 0 and has skipped the rule, so a CI gate built that way checks nothing. - System-call rules without
evt.type, or withevt.dir. The first is slow and warned about.evt.dir, common in older examples, is deprecated since Falco 0.42, andevt.dir = <is now always true. - Expecting Falco to count or correlate. Thresholds, time windows and pairs of events belong to whatever receives the alerts.
Where next
- Splunk, Kusto and Elastic run the same lab and five detections in engines that count and correlate, which is where D2 and D3 are finished; Sigma and YARA cover another open rule format, and jq and JMESPath the
jqused here. - The CloudTrail to SIEM lab for a real CloudTrail feed, and detection engineering for testing, tuning and retiring rules.
- Falco's documentation on rules, macros and lists, conditions, overriding and exceptions, and the supported fields.
- The stock rules in falcosecurity/rules and the cloudtrail plugin, whose rules file and field list are worth reading in full.