jq and JMESPath, hands-on

The two query languages you will use every single day

Cloud security work is, mechanically, JSON wrangling. Every CLI returns JSON. Every API returns JSON. IAM policies are JSON, CloudTrail events are JSON, Terraform plans are JSON, and every scanner you run emits JSON. The bottleneck between "the data exists" and "I know the answer" is almost always a query you could not quite write.

There are two languages worth knowing, and confusing them is the single most common source of wasted time here, because they look similar and share nothing.

• jq is a standalone tool and a genuinely powerful language. It streams: a filter takes JSON values in and emits JSON values out, possibly more or fewer than it received. Use it anywhere you have a file or a pipe.
• JMESPath is a specification with implementations in many languages, which is exactly why AWS embedded it. It is what --query means in the AWS CLI and in the Azure CLI. It is a pure projection language: one input document, one output value, no streaming.

Google Cloud is the odd one out and uses neither: gcloud has its own --filter and --format syntax. Knowing that up front saves you trying JMESPath against gcloud and concluding you have forgotten how to write it.

Streams versus projections

Almost every jq confusion traces back to one idea. jq filters produce a stream, not a value.

echo '{"a":[1,2,3]}' | jq '.a'      # one output: the array  [1,2,3]
echo '{"a":[1,2,3]}' | jq '.a[]'    # THREE outputs: 1, then 2, then 3
echo '{"a":[1,2,3]}' | jq '[.a[]]'  # collect the stream back into one array

Those square brackets in the third line are not decoration. Any time a jq expression produces something that looks almost right but repeated, or almost right but unwrapped, the answer is usually a missing pair of collecting brackets. select(), map(), group_by() and friends all become obvious once you know which of them expect a stream and which expect an array.

JMESPath has no equivalent concept. An expression takes one document and returns one JSON value, always. That constraint is why it is safe to embed in a CLI flag and why it is considerably less capable: there is no way to express "emit a line per element" because there are no lines and no emitting.

Walk-through

1. Build the sample data

This is a trimmed CloudTrail-shaped file. Everything below runs against it, so no cloud account and no credentials are needed.

mkdir -p ~/jq-lab && cd ~/jq-lab

cat > events.json <<'EOF'
{"Records":[
 {"eventTime":"2026-08-20T14:03:11Z","eventName":"AssumeRole","errorCode":null,
  "sourceIPAddress":"203.0.113.14",
  "userIdentity":{"type":"IAMUser","arn":"arn:aws:iam::111122223333:user/alice"}},
 {"eventTime":"2026-08-20T14:03:12Z","eventName":"GetObject","errorCode":"AccessDenied",
  "sourceIPAddress":"198.51.100.7",
  "userIdentity":{"type":"IAMUser","arn":"arn:aws:iam::111122223333:user/bob"}},
 {"eventTime":"2026-08-20T14:05:40Z","eventName":"PutBucketPolicy","errorCode":null,
  "sourceIPAddress":"10.0.4.19",
  "userIdentity":{"type":"AssumedRole","arn":"arn:aws:sts::111122223333:assumed-role/deploy/i-123"}},
 {"eventTime":"2026-08-20T14:06:02Z","eventName":"ListBuckets","errorCode":"AccessDenied",
  "sourceIPAddress":"203.0.113.99",
  "userIdentity":{"type":"IAMUser","arn":"arn:aws:iam::111122223333:user/mallory"}},
 {"eventTime":"2026-08-20T14:06:03Z","eventName":"ListBuckets","errorCode":"AccessDenied",
  "sourceIPAddress":"203.0.113.99",
  "userIdentity":{"type":"IAMUser","arn":"arn:aws:iam::111122223333:user/mallory"}}
]}
EOF

2. jq: navigate, then filter

jq '.Records | length' events.json          # 5

# .Records[] streams each record; select() passes through only the
# ones where its condition is true and emits nothing for the rest.
jq '.Records[] | select(.errorCode == "AccessDenied") | .userIdentity.arn' events.json

# -r = raw output: print strings unquoted, which is what you want
# the moment the output feeds another command.
jq -r '.Records[] | select(.errorCode == "AccessDenied") | .userIdentity.arn' events.json

select() is the workhorse. Read select(cond) as a gate in the pipe: the value goes through unchanged if cond is true, and vanishes otherwise.

3. jq: build new objects

# Construct an object per record. The keys are yours to name.
jq '[.Records[]
     | select(.errorCode != null)
     | {when: .eventTime, what: .eventName, who: .userIdentity.arn, from: .sourceIPAddress}]' \
   events.json

Reshaping at the edge of a pipeline matters more than it looks. A downstream script that consumes {who, what, when} keeps working when AWS adds fields, moves things around, or you switch to a different source of the same facts.

4. jq: the query you will actually run

"Who is generating repeated access denials, and from where." This is a real triage question and it is four operators.

jq -r '
  [.Records[] | select(.errorCode != null)]      # collect failures into an array
  | group_by(.userIdentity.arn)                  # array of arrays, one per principal
  | map({arn: .[0].userIdentity.arn, denials: length})
  | sort_by(-.denials)                           # negate to sort descending
  | .[] | "\(.denials)\t\(.arn)"                 # \(...) is string interpolation
' events.json

Note the shape change at each step: stream, array, array of arrays, array of objects, then back to a stream of strings for printing. group_by requires a sorted array and sorts for you, and it hands each group to map as a list, which is why .[0] is how you get at the shared key.

5. jq: emit something another tool can read

# @csv quotes and comma-joins an array. Pair it with -r or you get
# a JSON string containing escaped quotes, which no spreadsheet wants.
jq -r '.Records[]
       | select(.errorCode != null)
       | [.eventTime, .eventName, .sourceIPAddress, .userIdentity.arn]
       | @csv' events.json

# @tsv, @sh (shell-quoted), @base64, and @uri exist too. @sh is the one
# that saves you when a value contains a space or a quote.
jq -r '[.Records[].sourceIPAddress] | unique | join(", ")' events.json

6. JMESPath: the same questions, different language

Everything below works as an AWS CLI --query value. To practise without a cloud account, evaluate against the same file:

pip3 install jmespath        # brings the reference implementation

python3 - <<'PY'
import json, jmespath
doc = json.load(open('events.json'))
print(jmespath.search("Records[?errorCode=='AccessDenied'].userIdentity.arn", doc))
print(jmespath.search("length(Records[?errorCode!=`null`])", doc))
print(jmespath.search(
    "Records[?errorCode!=`null`].{when: eventTime, what: eventName, who: userIdentity.arn}", doc))
PY

Three pieces of syntax carry most of the weight:

• [?expr] is a filter projection. It is the select() equivalent.
• {a: x, b: y} is a multiselect hash, the object-construction equivalent.
• Backticks enclose JSON literals: `null`, `true`, `100`. Single quotes enclose string literals: 'running'.

7. Against a real AWS CLI

# Public S3 buckets is not one call, so this is the shape you want:
# a client-side query per bucket, driven by a jq-produced list.
aws s3api list-buckets --query 'Buckets[].Name' --output text

# EC2 instances without IMDSv2 enforced.
aws ec2 describe-instances \
  --query 'Reservations[].Instances[?MetadataOptions.HttpTokens!=`required`].[InstanceId,PrivateIpAddress]' \
  --output text

# IAM users whose access keys are old. --filters is SERVER side; --query
# is CLIENT side. Mixing them up matters, see below.
aws ec2 describe-instances \
  --filters 'Name=instance-state-name,Values=running' \
  --query 'Reservations[].Instances[].InstanceId'

8. The combination that gets used most

In practice the strongest move is not choosing between them. It is --output json piped into jq, using JMESPath only for the trivial cases.

# --output text collapses nulls to the string "None" and flattens nested
# lists with tabs, so columns silently misalign the moment a field is
# empty. For anything you intend to parse, take JSON and use jq.
aws iam list-users --output json \
  | jq -r '.Users[] | [.UserName, .CreateDate] | @tsv'

Hands-on exercises

jq -r '
  [.Records[]]
  | group_by(.sourceIPAddress)
  | map({ip: .[0].sourceIPAddress, n: length})
  | sort_by(-.n)
  | .[] | "\(.n)\t\(.ip)"
' events.json

Records[?userIdentity.type=='IAMUser'].userIdentity.arn

aws ec2 describe-instances --query 'Reservations[].Instances[?State.Name == "running"].InstanceId'

aws ec2 describe-instances \
  --query "Reservations[].Instances[?State.Name == 'running'].InstanceId"

# The trick is to normalise first. In jq, wrapping a value in an array
# and flattening is the idiomatic "make this a list either way".
cat > policy.json <<'EOF'
{"Statement":[
  {"Effect":"Allow","Action":"*","Resource":"*"},
  {"Effect":"Allow","Action":["s3:GetObject","s3:PutObject"],"Resource":"*"},
  {"Effect":"Allow","Action":["iam:*","*"],"Resource":"*"}
]}
EOF

jq -r '.Statement[]
       | select(.Effect == "Allow")
       | select([.Action] | flatten | index("*"))
       | "wildcard action: \(.Action)"' policy.json

Common mistakes

• Double quotes in JMESPath. They mean "field name", not "string". The failure is silent and reads as an empty environment.
• Forgetting -r in jq. Without it every string is emitted with quotes, and the next command in the pipe treats those quotes as part of the value.
• Missing collecting brackets. group_by, sort_by, unique, and add want an array. Given a stream they run per element and quietly return something plausible.
• Assuming --query limits what you retrieved. It filters after the fact. It is not an access control, not a cost control, and not a way to avoid reading data you should not read.
• Parsing --output text. Nulls become the literal string None and nested structures flatten into tabs, so columns shift whenever a field is empty. Take JSON and parse it properly.
• Trying JMESPath on gcloud. Different language entirely: --filter and --format, with their own syntax.
• Believing an empty result. The strongest habit on this page: when a query returns nothing, re-run it with the filter removed. A broken query and an empty account are indistinguishable until you check.