Regex for security work, hands-on

Why regex is a security topic and not a text-editing trick

Almost everyone in this field learns just enough regular expression syntax to search a log file, and then stops. That is a reasonable place to stop if a regex is only ever a search. It is a bad place to stop the moment a regex becomes a control, and in cloud security it becomes a control constantly.

A regex decides whether a URL is on your allowlist. A regex decides whether a string in a commit is a leaked credential. A regex is what a Sigma rule compiles down to inside your SIEM, what a WAF matches request bodies against, what a CloudWatch Logs metric filter counts, and what a DLP scanner calls a Social Security number. In every one of those places a regex that is subtly too broad is an alert storm or a false block, and a regex that is subtly too narrow is a detection that silently never fires.

The skill this page is trying to build is not "write a regex". It is the habit of asking two questions about every pattern you meet: what does this match that I did not intend, and what does it fail to match that I did intend? Both of those failures are quiet. Neither one produces an error message.

Two engines, two sets of rules

Before any syntax, know which family of engine you are writing for, because it changes both what you are allowed to write and what happens under attack.

• Backtracking engines (PCRE, Perl, Python's re, Java, .NET, JavaScript, most SIEM query languages) try alternatives and reverse out of dead ends. They support lookahead, lookbehind, and backreferences. They can also take exponential time on a hostile input.
• Finite-automaton engines (RE2, Go's regexp, the Rust regex crate, and therefore ripgrep by default) match in time linear in the input length, always. The price is that lookahead, lookbehind, and backreferences do not exist, because they cannot be expressed in the model.

This is why a pattern copied from a blog post into a Go service fails to compile, and why the same pattern copied into a Python validator is a denial-of-service risk. Google built RE2 specifically so that user-supplied patterns could be run safely, and services that accept regex from users (log platforms, WAF rule builders) almost always use it for exactly that reason.

The two failure directions, concretely

# TOO BROAD - the dot is unescaped, so it matches ANY character
https://.*.example.com          # matches https://evilXexampleYcom

# TOO BROAD - no anchors, so the match can sit anywhere in the string
https://.*\.example\.com        # matches https://evil.com/?x=.example.com

# TOO NARROW - anchored and escaped, but assumes lowercase and no port
^https://[a-z]+\.example\.com$  # misses https://API.example.com:8443

All three of these are plausible-looking, and the first two are the shape of a real server-side request forgery allowlist bypass. The third is the shape of a detection rule that quietly covers less than the author thought.

Walk-through

Build a scratch directory first. Every step below runs against these files.

mkdir -p ~/regex-lab && cd ~/regex-lab

cat > sample.log <<'EOF'
2026-08-20T14:03:11Z user=alice ip=203.0.113.14 action=AssumeRole result=success
2026-08-20T14:03:12Z user=bob ip=198.51.100.7 action=GetObject result=denied
2026-08-20T14:05:40Z user=svc-deploy ip=10.0.4.19 action=PutBucketPolicy result=success
2026-08-20T14:06:02Z user=mallory ip=203.0.113.99 action=ListBuckets result=denied
2026-08-20T14:06:03Z user=mallory ip=203.0.113.99 action=ListBuckets result=denied
EOF

cat > secrets.txt <<'EOF'
aws_access_key_id = AKIAIOSFODNN7EXAMPLE
temp_key = ASIAIOSFODNN7EXAMPLE
not_a_key = XAKIAIOSFODNN7EXAMPLEX
doc_reference = "the AKIA prefix identifies long-term keys"
EOF

1. Anchors, and why they are a security control

# -E = extended regex (POSIX ERE). Without -E, grep needs backslashes
# in front of +, ?, |, (, and ) - a classic source of "why doesn't this work".
grep -E 'denied' sample.log            # substring match, anywhere on the line
grep -E '^2026-08-20T14:06' sample.log # ^ pins to start of line
grep -E 'success$' sample.log          # $ pins to end of line

In a search, an unanchored pattern is a convenience. In a validator, an unanchored pattern is a bug, because "contains" is not "is". Prove it to yourself:

python3 - <<'PY'
import re
allow = re.compile(r'https://.*\.example\.com')      # no anchors
for url in ["https://api.example.com",
            "https://evil.com/?redirect=.example.com",
            "https://api.example.com.attacker.net"]:
    print(bool(allow.search(url)), url)
PY

All three come back True. The second and third are the ones that get written up as an incident.

2. The Python anchor that is not an anchor

Anchoring both ends is necessary and, in Python specifically, not sufficient. $ matches at the end of the string or immediately before a trailing newline.

python3 - <<'PY'
import re
p = re.compile(r'^[a-z0-9-]+$')          # "a simple hostname label"
print(bool(p.match("prod-web")))          # True  - fine
print(bool(p.match("prod-web\n")))        # True  - surprising
print(bool(p.match("prod-web\nrm -rf /")))# False - the dot rule saves you here

q = re.compile(r'\A[a-z0-9-]+\Z')        # \A and \Z admit no newline at all
print(bool(q.match("prod-web\n")))        # False - what you meant
PY

Whether the trailing newline matters depends entirely on what consumes the value afterwards. If it lands in a header, a log line, or a generated config file, it matters a great deal. Use \A and \Z in Python when the regex is a validator rather than a search.

3. Character classes, quantifiers, and greed

# Extract every source IP. [0-9]{1,3} = one to three digits.
grep -oE '\bip=[0-9]{1,3}(\.[0-9]{1,3}){3}\b' sample.log

# -o prints only the matched part rather than the whole line.
# \b is a word boundary: a zero-width assertion between a word
# character and a non-word character. It is what stops the pattern
# matching the middle of a longer token.

Quantifiers are greedy by default: they take as much as they can and then give back only if the rest of the pattern fails. Adding ? makes them lazy, taking as little as possible. On a log line with more than one delimiter, the difference is the whole result:

python3 - <<'PY'
import re
line = 'user=alice action=AssumeRole result=success'
print(re.search(r'user=(.*) ',  line).group(1))   # greedy: 'alice action=AssumeRole'
print(re.search(r'user=(.*?) ', line).group(1))   # lazy:   'alice'
print(re.search(r'user=([^ ]+)', line).group(1))  # explicit: 'alice'  <-- prefer this
PY

The third form is the one to reach for. "Everything up to the next space" stated as [^ ]+ is unambiguous, is immune to a later change in the rest of the pattern, and does not depend on the reader remembering greed rules at 2am.

4. Groups, captures, and named captures

python3 - <<'PY'
import re, collections
pat = re.compile(
    r'^(?P<ts>\S+) user=(?P<user>[^ ]+) ip=(?P<ip>[^ ]+) '
    r'action=(?P<action>[^ ]+) result=(?P<result>\w+)$')
denials = collections.Counter()
for line in open('sample.log'):
    m = pat.match(line.strip())
    if m and m['result'] == 'denied':
        denials[(m['user'], m['ip'])] += 1
print(denials)      # repeated denials by principal and source IP
PY

(?P<name>...) is a named capture group. (?:...) groups without capturing, which you want whenever a group exists only so a quantifier can apply to it. Naming groups is the difference between a parser you can read next year and a wall of group(4).

5. Write a credential scanner

AWS access key IDs are 20 characters: a four-character prefix plus 16 uppercase alphanumerics. AKIA is a long-term IAM user key. ASIA is a temporary STS credential.

# First attempt - finds the keys, and also finds the two things that are not keys
grep -oE '(AKIA|ASIA)[A-Z0-9]{16}' secrets.txt

# Word boundaries reject the embedded case
grep -oE '\b(AKIA|ASIA)[A-Z0-9]{16}\b' secrets.txt

Run both. The first matches inside XAKIAIOSFODNN7EXAMPLEX, because nothing said the match had to start at a token boundary. The second does not. Neither can tell you anything about the prose line mentioning AKIA, because that line contains no 20-character key, which is the correct outcome and worth confirming rather than assuming.

Now try the other half of an AWS credential pair, and watch the approach fall apart:

# A secret access key is 40 characters of base64 alphabet. As a regex
# that is nearly content-free: it matches most git hashes, most base64
# blobs, and a good fraction of minified JavaScript.
grep -oE '\b[A-Za-z0-9/+=]{40}\b' secrets.txt

This is the boundary of what regex can do, and knowing where that boundary is matters more than another hour of syntax. Real secret scanners pair a regex with something else: Shannon entropy over the candidate, a proximity requirement (the string must sit near aws_secret_access_key), and increasingly a live validity check against the provider. Regex finds candidates. It does not decide.

6. Lookaround, and living without it

# Lookahead: match a key ID only when NOT followed by more alphanumerics.
# Lookbehind: and not preceded by any either. PCRE / Python / Java only.
python3 -c "
import re
print(re.findall(r'(?<![A-Z0-9])(AKIA|ASIA)[A-Z0-9]{16}(?![A-Z0-9])',
                 open('secrets.txt').read()))"

# ripgrep defaults to the Rust regex engine, which has no lookaround.
rg '(?<![A-Z0-9])AKIA[A-Z0-9]{16}' secrets.txt    # error: look-around not supported
rg -P '(?<![A-Z0-9])AKIA[A-Z0-9]{16}' secrets.txt # -P switches to PCRE2

When the engine has no lookaround, express the boundary instead of asserting it. \b covers most cases. Where it does not, match the surrounding character and capture only the part you want:

# Instead of  (?<=key = )\S+   which needs lookbehind,
# match the label too and take group 1.
rg -o 'key\s*=\s*(\S+)' -r '$1' secrets.txt   # -r rewrites output to the capture

7. Catastrophic backtracking

A backtracking engine can be made to explore an exponential number of paths. The trigger is nested quantifiers over overlapping alternatives, followed by something that fails.

python3 - <<'PY'
import re, time
evil = re.compile(r'^(a+)+$')          # nested quantifier, both greedy
for n in (18, 20, 22, 24):
    s = "a" * n + "!"                  # the "!" guarantees the match fails
    t = time.perf_counter()
    evil.search(s)
    print(n, f"{time.perf_counter() - t:.3f}s")
PY

Each extra character roughly doubles the time. At 24 characters you are already waiting; at 40 the process is effectively hung. An attacker who controls the input to a validator, a log parser, or a WAF rule gets a CPU exhaustion primitive for free, and it is one of the few denial-of-service bugs that survives autoscaling, because every worker is equally stuck.

Three defences, in order of preference: use a finite-automaton engine for anything touching untrusted input; avoid nesting a quantifier inside a quantified group when the inner alternatives can match the same text; and where neither is possible, impose a length cap on the input before the regex sees it.

Hands-on exercises

re.search(r'https://.*\.internal\.example\.com', url)

rg -o '\b(AKIA|ASIA)[A-Z0-9]{16}\b' secrets.txt

A:  ^(\w+\s?)*$
B:  ^\w+(\s\w+)*$
C:  ^(a|ab)+c$
D:  ^[a-z]{1,64}$

grep -E 'result=denied' sample.log \
  | grep -oE '\buser=[^ ]+' \
  | sort | uniq -c | sort -rn \
  | awk '$1 > 1'          # keep only counts above one

Common mistakes

• An unanchored pattern used as a validator. "Contains" is not "is". Anchor both ends, and in Python use \A and \Z rather than ^ and $.
• An unescaped dot inside a hostname. \. is a literal dot; a bare . is any character. In a domain allowlist that difference is the entire control.
• Assuming the engine supports lookaround. Go, Rust, ripgrep's default mode, and anything built on RE2 do not. The pattern does not degrade, it fails to compile, which is at least loud. The quieter version is a colleague deleting the lookaround to make it compile.
• Testing only strings that should match. Same failure as testing only the deny direction of a policy: a pattern that matches everything passes every positive test.
• Case sensitivity taken for granted. Hostnames are case-insensitive, HTTP header names are case-insensitive, and Windows paths are case-insensitive. A detection anchored to lowercase misses the trivially obvious evasion.
• Regex over structured data. JSON, XML, and URLs have parsers. A regex over their serialised form is correct until the first quote, escape, or nested brace.
• Forgetting . excludes newline. A multi-line payload slips past a single-line pattern, and turning on the dot-matches-newline flag to fix it often makes a greedy quantifier swallow the rest of the file.