sysadmin · difficulty ◆◆◆
strace — see every system call a program makes
strace is the kernel's receipt for your program: every path it touched, every socket it opened, and the exact call the stall lives in — printed as it happens.
A command that fails, a log that says nothing, and a config file you swear you edited — strace ends that argument in one line.
fully AI-generatedThis text is fully AI-generated and was published without human editorial review.
$ straceWhat strace actually does
strace puts a process under ptrace and prints every system call as it crosses into the kernel: `syscall(args) = return`. The arguments come back decoded — real paths instead of pointers, `{sa_family=AF_INET, sin_port=htons(22), sin_addr=inet_addr("10.255.255.1")}` instead of a struct dump, flags spelled out as `O_RDONLY|O_CLOEXEC` — and a failure arrives as `-1 EACCES (Permission denied)` with the errno named. No logging to switch on, no rebuild, no restart: it watches the kernel side of the conversation, which is why it settles arguments that application logs cannot. This box runs strace 6.8, man page dated 2024-03-13, built with libunwind, so `-k` stack traces are available too.
The questions it answers in five minutes
Which config file did it *actually* read? `-e trace=%file` plus `grep ENOENT` shows every path it looked for, including the ones that do not exist. Which of a thousand calls is failing? `-Z` prints only the failures, errno attached. Where did the second go? `-T` puts each call's own duration in angle brackets, and `-c` gives you the whole bill as a table. Which host and port did it really connect to? `%network`, where you also get to watch glibc probe `/var/run/nscd/socket` before every lookup. And what happens when a disk read fails? `-e fault=` breaks the call on purpose.
How to read a trace line
Three shapes cover almost everything. A finished call: `openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3` — name, decoded arguments, return value. A failed call: `= -1 ENOENT (No such file or directory)`. An unfinished one: `= ? ERESTART_RESTARTBLOCK (Interrupted by signal)`, a `<... resumed>` continuation when the call restarts, or a bare `<detached ...>` when you interrupt strace while the call is still blocked — I hit the last one with `timeout 3 strace -p` more than once. With `-f` every line is prefixed `[pid N]`. And the trace goes to **stderr**: `strace … | head` trims nothing at all, which is why the working pattern is `-o /tmp/t.log` and then grep the file.
Example
$ strace -q -c -f -o /tmp/true.trace sh -c 'for i in $(seq 300); do /bin/true; done'
head -12 /tmp/true.trace
tail -1 /tmp/true.trace% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
60.10 0.022558 37 602 301 wait4
14.48 0.005435 18 300 vfork
7.39 0.002773 9 302 execve
4.94 0.001853 0 2417 mmap
2.90 0.001090 1 900 rt_sigprocmask
2.31 0.000866 0 906 mprotect
1.68 0.000630 2 301 rt_sigreturn
1.55 0.000582 1 302 munmap
0.77 0.000290 0 605 openat
0.53 0.000198 0 611 close
100.00 0.037536 3 10910 603 total300 runs of /bin/true look like nothing, and they cost 10,910 syscalls and 0.038 s of kernel time — wait4, vfork and execve are 10× the syscalls of the work itself. That is the fork/exec tax in one table, and the reason a shell loop over 300 small commands is slower than one command doing 300 things. -q drops the hundreds of 'Process N attached' notices that -f would otherwise print; -o captures the summary into the file, because strace writes to stderr and a pipe will not catch it.
$ strace -f -e trace=%file -o /tmp/ssh.trace ssh -V
wc -l < /tmp/ssh.trace
grep ENOENT /tmp/ssh.traceOpenSSH_9.6p1 Ubuntu-3ubuntu13.20, OpenSSL 3.0.13 30 Jan 2024
28
135682 access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
135682 statfs("/sys/fs/selinux", 0x7fffdd331830) = -1 ENOENT (No such file or directory)
135682 statfs("/selinux", 0x7fffdd331830) = -1 ENOENT (No such file or directory)
135682 access("/etc/selinux/config", F_OK) = -1 ENOENT (No such file or directory)
135682 openat(AT_FDCWD, "/proc/sys/crypto/fips_enabled", O_RDONLY) = -1 ENOENT (No such file or directory)`ssh -V` — a command whose entire job is printing a version string — makes 28 file-related syscalls, and five of them fail. The failures are the interesting part: no `/etc/ld.so.preload`, no SELinux mount, no `/etc/selinux/config`, no FIPS flag. This is the shape of the answer to 'which config did it read': everything it tried is listed, including the paths it never had. The pid prefix on every line comes from -f, and `%file` is the class that also catches lstat and statfs, which hand-written syscall lists forget.
$ strace -e trace=openat -o /tmp/shadow.trace cat /etc/shadow
grep EACCES /tmp/shadow.tracecat: /etc/shadow: Permission denied
openat(AT_FDCWD, "/etc/shadow", O_RDONLY) = -1 EACCES (Permission denied)cat tells you the outcome; the trace tells you the mechanism — which syscall, which open flags, which errno. That single line is what goes into a bug report, an AppArmor profile or a support ticket. Keep the trace in a file with -o: run the same thing to the terminal and strace's lines interleave with cat's own stderr until neither is readable.
$ strace -T -tt -e trace=clock_nanosleep sleep 109:02:26.017416 clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=1, tv_nsec=0}, 0x7ffdbe38af00) = 0 <1.000114>
09:02:27.017826 +++ exited with 0 +++`-T` appends the call's own duration in angle brackets, `-tt` prefixes an absolute timestamp. The 1.000114 s is the sleep itself, not strace overhead, and matching it against the wall-clock wait is how you prove which call owns the stall. If the number is much smaller than the wait you measured, your time is hiding in a poll/select you have not traced yet — add `-e trace=%desc`.
$ strace -Z -e trace=openat -e fault=openat:error=EACCES:when=3 -o /tmp/fault.trace cat /etc/hostname
echo "exit=$? output=$(cat /etc/hostname)"
grep INJECTED /tmp/fault.tracenuc
exit=0 output=nuc
openat(AT_FDCWD, "/usr/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC) = -1 EACCES (Permission denied) (INJECTED)I broke the third openat on purpose and glibc shrugs: the injected EACCES lands in its locale lookup, marked `(INJECTED)`, and cat still prints the hostname and exits 0. `when=` takes `first..last+step`, so `when=3+` fails the third call and everything after it, `when=1..+5` injects every fifth; the default errno for `-e fault=` is ENOSYS. This is how error paths get tested without root, without breaking real permissions and without touching source.
Common flags
- -c / -C / -w
- One table instead of a trace: % time, seconds, calls and errors per syscall, sorted by time. -C keeps the regular output as well, -w reports wall-clock time instead of kernel time, and with -f only aggregate totals are kept.
- -e trace=%file
- Syscall classes: %file, %network, %process, %signal, %desc, %memory, %ipc, %creds, %clock. The percent sign is the current syntax — the un-prefixed form is deprecated. The man page's own argument for using it: hand-listing open,stat,chmod,unlink means you forget lstat. 'Betchya woulda forgot that one.'
- -f / -ff -o
- Follow forks and threads, each line prefixed with [pid N]. -ff writes a separate file per process (trace.PID), which is how you read a multi-process daemon without losing your place; -q silences the 'Process N attached' lines.
- -p PID
- Attach to something already running instead of starting it under strace. Needs ptrace permission: on this box /proc/sys/kernel/yama/ptrace_scope is 1, which turns that into `ptrace(PTRACE_SEIZE, 133128): Operation not permitted`.
- -T / -tt / -ttt
- -T appends each call's own duration, -tt an absolute timestamp per line, -ttt a unix one. Read the two together to find the single call behind a stall instead of guessing from wall-clock time.
- -z / -Z
- Print only successful (-z) or only failing (-Z) syscalls — shorthand for -e status=successful / status=failed. -Z reduces a 40,000-line trace to the four errnos you actually care about.
- -e fault=openat:error=EACCES:when=3
- Fault injection. when= accepts first..last+step, so `when=3+` fails the third call and all later ones and `when=1..+5` injects every fifth. Default errno is ENOSYS; the general form is -e inject=…, which additionally offers :delay_enter= for latency testing.
History
1991–1992 — a syscall tracer older than Linux 1.0
The copyright block at the top of strace's man page still reads: Paul Kranenburg 1991, 1992 (pk@cs.few.eur.nl), Branko Lankester 1993, Rick Sladkey 1993–1996. Kranenburg wrote it for SunOS, where the tracing was done through a kernel driver; the Linux port in 1993 leaned on ptrace(2), which by then allowed an ordinary user to trace its own children — the whole trick strace still runs on. Linux 1.0 did not ship until March 1994, so the tool people now treat as a Linux native is older than the kernel release it is associated with.
6.8 in 2024 — still adding filters, still joking
`strace --version` on this machine prints `strace -- version 6.8` and `Copyright (c) 1991-2024 The strace developers`; the man page is dated 2024-03-13. The option list has kept growing in the 6.x line: `-e status=` (with -z/-Z as shorthand for the two useful values), `-e trace-fds=` and `-P path` for narrowing by file, `--seccomp-bpf`, `-e inject=…:delay_enter=` for adding latency instead of errors, and `--tips`, which prints a small ASCII cartoon explaining a feature — the same trick `-k` uses here, which works because this build reports `stack-trace=libunwind`.
Fun facts
Pros & cons
pros
- + The kernel's own account of what ran: no logging to enable, no rebuild, no restart, no source code — any dynamically linked binary you can execute or attach to.
- + Filters sharp enough to automate: syscall classes, single paths with -P, fd-to-path decoding, failures-only output, and a summary table you can diff between two runs of the same command.
- + Fault injection reaches error paths that permissions, hardware and test rigs cannot trigger on demand — and it costs one flag.
cons
- − ptrace overhead is real and it changes timing, so a latency bug can disappear the moment you attach; it is not something to leave running against a busy production process.
- − Permissions and sandboxes decide your day: yama ptrace_scope blocks attach, seccomp profiles and containers can refuse it outright, and a setuid binary cannot be traced by an ordinary user.
- − Unfiltered output is a firehose — megabytes per second on a threaded binary — and interleaving with the program's own stderr makes the terminal unusable unless you write to a file with -o.
Takeaways
- 1Start with `strace -c` (add -f for forks): one table tells you whether the cost is process spawning, disk or a single waiting call before you read a single trace line.
- 2`-e trace=%file -o /tmp/t.log` plus `grep ENOENT /tmp/t.log` is the fastest honest proof of which config file a program read — it lists the paths that did not exist too.
- 3When something fails, print only the failures with `-Z`; the errno in that line is the thing to paste into a bug report.
- 4`-T` together with `-tt` puts each syscall's own duration in angle brackets — the only reliable way to name the call that owns a stall.
- 5Send traces to a file with -o and grep them (strace writes to stderr, so pipes lie), and on a box where yama ptrace_scope is 1 plan for `-p` to fail: start the command under strace instead.
Related commands
archive entry — written and published by a scheduled job, never editorially reviewed. Kept for reference: out of the site’s search index, and no ads run here. the reviewed library →