kmail.at
← learning

shell · difficulty ◆◆

jq — slice, filter and reshape JSON

sed for JSON — one zero-dependency binary that turns every --json flag on your box into something you can address by field name.

The tool that reads JSON in your terminal is not a flag on some other program — it is a full programming language that happens to have the best command-line interface ever bolted onto a parser.

2026-10-13 · 5 min read

$ jq

A filter, not a flag

jq takes two things: a program, and a stream of JSON. The program is a filter — `.` is identity, `.State.Status` is a lookup, `.[]` explodes an array, `|` pipes. That is the whole model, and the manual's opening line is still the best description anyone has written: 'jq is like sed for JSON data — you can use it to slice and filter and map and transform structured data with the same ease that sed, awk, grep and friends let you play with text.' It is written in portable C with zero runtime dependencies, so it is one binary you can scp to a box that has nothing installed — which is why jq already exists on the server you are about to SSH into. And it is a language, not a flag set: `,` is a generator that produces several outputs, `select()` is your WHERE clause, and `map`, `to_entries`, `group_by`, `reduce` are waiting when the one-liner stops being enough. Ten minutes buys you 90% of it.

Three moves cover most real work

Navigate: `.spec.containers[0].image`, `.blockdevices[]`, `.items[]?.metadata.name` — the trailing `?` swallows errors on missing paths, which is what makes jq safe on data whose shape you do not control. Select: `select(.State == "running")` filters a stream of objects, `.[] | select(.type == "disk")` filters inside one array. Reshape: `{name: .Names, ports: .Ports}`, `[.Names, .Image] | @tsv`, and interpolation `"\(.Name) is \(.State)"`. Then two flags decide who the output is for: `-r` when a script or the next command reads it, `-c` when a log or a diff reads it, and nothing at all when a human reads it.

Where it pays for itself

Half the tooling on a modern Linux box can already print JSON, and the moment a command can print JSON it stops being interactive: `docker inspect`, `docker ps --format json`, `journalctl -o json`, `kubectl get … -o json`, `npm ls --json`, `ip -j addr`, `lsblk --json`, `sqlite3 -json`, `aws --output json`, and `curl` against any API you will ever meet. The old reflex was to grep and awk the pretty output and hope the column positions held. The new one is to ask the command for JSON and let jq do the addressing, because field names survive version upgrades and column positions do not.

Example

$ docker inspect kmail-next | jq -r '.[0] | "\(.Name)\t\(.State.Status)\t\(.Config.Image)"'
/kmail-next	running	kmail-next-app

The everyday job: four lines of shell replace a scroll through 400 lines of container metadata. `docker inspect` wraps the container in an array, so `.[0]` is the container; `-r` strips the quotes jq puts around every string; `\t` inside the interpolation keeps the columns aligned. Two habits worth stealing: an unknown field prints `null` rather than failing, so a wrong path is silent — check it once with `jq '.[0] | keys'`; and if you want the whole state object instead of one line, drop the interpolation and run `docker inspect kmail-next | jq '.[0].State'`.

$ docker ps --format json | jq -s -r '.[] | [.Names, .State, .Image] | @tsv' | head -5
kmail-next	running	kmail-next-app
kmail-app	running	kmailweb_terminal_style-app
eu-ai-act-web	running	eu-ai-act-web
eu-ai-act-server	running	eu-ai-act-server
open-webui-rebrand	running	ghcr.io/open-webui/open-webui:main

`docker ps --format json` emits one JSON object per line — that is NDJSON, not a JSON document, and plain `jq .` will just reformat each line separately and never look at the set. `-s` (slurp) reads every line into a single array, which is what makes `.[] | select(...)`, `map`, `sort_by` and `length` work on the whole list. `@tsv` then flattens each object into columns. I use this shape for the weekly 'which containers have been up forever' question: `docker ps --format json | jq -s -r '.[] | select(.Status | test("Up [0-9]+ days")) | [.Names, .Status] | @tsv'`.

$ journalctl -o json --since today | jq -r 'select(.SYSLOG_IDENTIFIER=="sshd") | .MESSAGE | capture("from (?<ip>[0-9.]+) port").ip' | sort | uniq -c | sort -rn | head -4
    915 162.225.42.218
    708 77.239.124.76
    313 109.160.32.166
    266 176.53.159.198

This is the example that converted me. journald already stores structured fields, and `-o json` hands them over one object per entry: select the unit by identifier, pull `.MESSAGE`, extract the source IP with a named capture group, then let the boring old `sort | uniq -c | sort -rn` do the counting. No grep, no column offsets, no assumption about log format — the same pipeline works on `_COMM`, `PRIORITY` or `_PID` with one word changed. The counts are today's log on this host, which today means the top offender hit 915 times before I got to it; the numbers move by the hour, the shape of the pipeline does not.

$ ip -j addr show | jq -r '.[] | select((.ifname | test("^(lo|br-|veth|docker)")) | not) | "\(.ifname)\t\(.operstate)\t\([.addr_info[]?.local] | join(", "))"'
eno1	DOWN	
wlp0s20f3	UP	192.168.1.57, fd5c:2d2b:cb4c:ef7:3e58:c2ff:fed8:5ef5, fe80::3e58:c2ff:fed8:5ef5

`ip -j` is the least-known flag in iproute2 and the reason I stopped parsing `ip a` by eye. Two details make this line work on real machines: `.[]?` with the question mark skips interfaces with no `addr_info` block instead of raising, and the `test(...) | not` filter drops the dozen Docker bridges and veth pairs that would otherwise bury the two interfaces you care about. The trailing empty column on `eno1` is real, not a bug — the interface is down, so it has no addresses to join. Widen the regex to `^(lo|br-|veth|docker|virbr|tun)` on hosts with more virtual plumbing than this one.

$ curl -s https://api.github.com/repos/jqlang/jq/releases | jq -r '.[0:3][] | [.tag_name, .published_at] | @tsv'
jq-1.8.2	2026-06-20T14:11:27Z
jq-1.8.1	2025-07-01T11:50:06Z
jq-1.8.0	2025-06-01T06:05:01Z

Checking a version means opening a browser and reading a releases page, unless you do this. The API returns the newest hundred releases as an array, `.[0:3]` slices the newest three, and `@tsv` lays them out as columns you can paste into a ticket. One change I would make in production: `curl -fsS` instead of `curl -s`, so a 404 or a rate-limit page fails the pipeline instead of being fed to jq as if it were a release list — see the warning below for what plain `-s` costs you. This is also the honest answer to 'what is the latest version of X' for any tool that has a GitHub release feed.

Common flags

-r / --raw-output
print strings without the surrounding quotes. Rule of thumb: if the output goes into another command, a variable, or a file, add -r; the default pretty printer is for humans and it will quote your strings and pretty-print your numbers.
-c / --compact-output
one line per value instead of indented blocks. `jq -c .` makes any JSON stream greppable and diffable, and it is what you want when logging a filter into a file or feeding a `while read` loop.
-s / --slurp
read the entire input as one array. This is the NDJSON fix: `docker ps --format json`, `journalctl -o json` and most log shippers emit one object per line, and -s is what makes set operations (map, select, sort_by, length) possible on them.
-e / --exit-status
set jq's exit code from the last output: false or null gives 1, an empty result gives 4, and a well-formed true value gives 0. That single flag turns jq into the test command in a shell script.
-n / --null-input
ignore stdin and build a document from nothing — used together with --arg to assemble JSON from shell variables without a single backslash.
--arg k v / --argjson k v
bind a shell value to `$k` as a string or as parsed JSON. --arg escapes it for you; --argjson refuses to invent types, so a number stays a number instead of a string that looks like one.
-S / --sort-keys
sort object keys on output, so two runs of the same pipeline produce byte-identical files. Useful the first time a config diff shows every line changed because the keys got reordered.

History

2012 — a Cambridge side project becomes the last line of every pipeline

jq arrived in 2012 from Stephen Dolan (@stedolan), released as 1.0 with a manual whose opening comparison is still the best description of it: 'jq is like sed for JSON data.' JSON was barely six years past RFC 4627 and the shell had no way to read it — grep could find a key, awk could split on a colon, and neither could follow a path through nested objects. jq's answer was to make the filter a small language: pipes, generators, and a `.[]` that walks structures. It was written in portable C with zero runtime dependencies so it could be copied to machines that have nothing, and the parser is a yacc grammar compiled at build time — which is why the project has always been one binary plus libjq for embedding. The rest of the story is a tool outliving its author's spare time: jq 1.5 shipped on 16 August 2015, and then the quiet began.

November 2018 → June 2026: the five-year gap and the security catch-up

jq 1.6 shipped on 2 November 2018 and nothing followed for almost five years. The 1.7 release notes open by admitting it: 'After a five year hiatus we're back with a GitHub organization, with new admins and new maintainers who have brought a great deal of energy to make a long-awaited and long-needed new release.' Stephen Dolan transferred the repository to the new jqlang organization; 1.7 landed on 6 September 2023, then 1.7.1 on 13 December 2023, 1.8.0 on 1 June 2025 and 1.8.2 on 20 June 2026. That last one is worth knowing about: it is a security patch that closed sixteen CVEs in the parser and the value layer — among them CVE-2026-40164, a hash-collision denial-of-service that randomises the hash seed, CVE-2026-33948, NUL truncation in the JSON parser, and a heap buffer overflow in string handling. If any part of your pipeline feeds jq JSON from outside your trust boundary, patch level is not cosmetic.

Fun facts

Pros & cons

pros

  • + One zero-dependency binary that is already installed on nearly every Linux box, container base image and CI runner you will land on — the JSON filter you write at two in the morning runs on the server, with no venv, no pip and no node_modules to provision.
  • + It reads data you have never seen before. `keys`, `to_entries`, `paths`, `.[]` and the trailing `?` let you explore a multi-megabyte API response at the prompt; the field names are the documentation, and the answer takes one line instead of 40 lines of Python.
  • + It composes with the shell rather than replacing it: -r to feed xargs, -c to hand one record per line to `while read`, -e to return an exit code, --raw-output0 with `xargs -0` when filenames contain spaces, and @sh when the shell itself is the destination.

cons

  • − The filter language has real cliffs — `,` produces several outputs rather than appending, `//` is a default operator and not a comment, `?` swallows errors at whatever granularity you attach it to, and reduce/foreach are a different afternoon entirely. The error messages are short to the point of rude: 'Cannot index object with number' never tells you which object, or which of the five objects in the stream it meant.
  • − Numbers are IEEE doubles in arithmetic. jq 1.7 made literals and comparisons precision-preserving, but `jq -n '9007199254740993 + 0'` still prints 9007199254740992, so anything that must add up to the cent — money, invoice totals, database IDs past 2^53 — belongs in Python with Decimal or in your database, not in a jq reduce.
  • − It is not a streaming processor by default: jq reads a document into memory, and -s reads the whole stream into memory, so a two-gigabyte JSON file is a two-gigabyte problem. `--stream` exists and turns the document into path/value pairs, but it changes the shape of everything downstream, which is why most people reach for a real language once the file stops fitting in RAM.

Takeaways

  1. 1Put -r on every jq whose output a script or another command will read. The default pretty printer is for humans, and it quotes strings — which is exactly what breaks the variable you assign next.
  2. 2Use `jq -c .` to make any JSON-producing command greppable and diffable, and `jq -s` when the input is already one object per line (docker ps --format json, journalctl -o json, most log shippers). Slurp first, then treat it as an array.
  3. 3Address data by field name instead of column position, and keep the `?`: `.items[]?.metadata.name` never blows up on the one record whose shape differs, and field names survive upgrades where awk column numbers do not.
  4. 4Bind shell values with --arg and --argjson instead of gluing JSON together by hand — jq escapes them, and --argjson keeps booleans and numbers from silently becoming strings.
  5. 5Let jq be the test in your scripts: `jq -e '.field' file >/dev/null && proceed` gives exit 1 on null or false, 2 on a missing or unparsable file, and 5 on a type error. Pair it with `curl -fsS`, because jq treats an empty body as success.

Related commands

← all learning