Everything is a file — lsof tells you who has it open.
You deleted the log, `rm` insists it is gone, and `df` still shows a full disk. Something is holding that file open — lsof names it, by PID.
lsof stands for 'list open files', and in Unix that phrase is much bigger than it sounds: sockets, pipes, device nodes, directories, shared libraries and deleted-but-still-open regular files are all files. On Linux lsof is a /proc archaeologist — it walks every /proc/<pid>/fd, follows each symlink to its target, and prints one row per file descriptor. The columns are the whole story: COMMAND and PID (who), USER (whom it runs as), FD (descriptor number plus the r/w/u access mode), TYPE (REG, DIR, IPv4, unix, FIFO, CHR), and NAME (what the descriptor points at). Run it without root and you see only your own processes; run it with sudo and you see the machine.
lsof answers the questions that page you at 02:00. Which process owns port 8080 — the one systemctl swears is already in use? Which process has config.json open while you try to edit it? Which process still holds the 60 GB log you deleted an hour ago, so the filesystem has not freed a single block? And its quiet superpower: give it a path instead of a process and it reverses the question — `lsof /var/log/nginx/access.log` tells you who is reading your file right now. If a question sounds like 'who has this socket, file or mount open', it is an lsof question, not a ps question.
An empty result is a real answer: lsof exits 1 when nothing matched, which makes it scriptable. In sockets, `(LISTEN)` marks the server side and `->` marks the client side of a connection, so `TCP *:8642 (LISTEN)` is a daemon waiting and `TCP 192.168.1.57:50834->192.168.1.13:8123 (ESTABLISHED)` is a live outbound connection. The NAME column appends `(deleted)` when the inode's link count dropped to zero but a descriptor is still pinned to it. Two habits make the output useful: always add `-P -n` (no port-name or host-name resolution — faster and script-friendly), and remember that several selection options are OR-ed unless you insert `-a` to turn the logic into AND.
lsof -i :8137 -P -n
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME python3 3921701 kmail 3u IPv4 36862091 0t0 TCP 127.0.0.1:8137 (LISTEN)
The 'who owns this port' invocation. -i :8137 filters to that port; -P stops lsof translating 8137 into a service name and -n stops it resolving 127.0.0.1 to 'localhost', which also skips the DNS round-trip. 3u = descriptor 3, open for read/write, listening on loopback only.
lsof -t -i :8137
3921701
-t (terse) prints PIDs and nothing else, which exists precisely so you can write `kill $(lsof -t -i :8137)` — no awk, no process-name guessing. Add -sTCP:LISTEN if you only want the listener and not its clients.
lsof +L1
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NLINK NODE NAME bash 3923152 kmail 4r REG 252,0 67108864 0 9736263 /home/kmail/.hermes/cache/scratch/held.tmp (deleted)
+L1 lists files with a link count below 1 — deleted files some process still holds open. NLINK 0 plus a 64 MB SIZE/OFF means 64 MB of disk that du cannot see and only a restart (or a kill) will release. This is the single most useful lsof flag on a full disk.
lsof -a -u kmail -i -P -n
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME code-e4c7 1460329 kmail 12u IPv4 25755658 0t0 TCP 127.0.0.1:37609 (LISTEN) hermes 1639025 kmail 7u IPv4 36605287 0t0 TCP 192.168.1.57:57626->149.154.166.110:443 (ESTABLISHED) hermes 1639025 kmail 37u IPv4 26496717 0t0 TCP *:8642 (LISTEN)
Without -a this would print every file owned by kmail OR every network file on the box. The -a turns the selectors into an AND, so you get only this user's sockets — the difference between a 4,000-line wall and the three lines you wanted.
lsof -p 1861744 | head -4
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME node 1861744 kmail cwd DIR 252,0 4096 1336223 /home/kmail/app node 1861744 kmail rtd DIR 252,0 4096 2 / node 1861744 kmail txt REG 252,0 124836408 524303 /usr/bin/node
One process, every descriptor: cwd and rtd are its working and root directories, txt is the executable image, then come its libraries, logs and sockets. This is how you prove which binary is actually running and from which directory when a service behaves differently than its config suggests.
| Flag | Meaning |
|---|---|
-i [addr] | Only network files. `-i` for all of them, `-i :443` for a port, `-i @192.168.1.13:8123` for one remote endpoint, `-i 6` for IPv6. |
-p PID | Restrict to a process (or a comma list of PIDs). The starting point for 'what has this process opened'. |
-u user | Filter by user name or UID; prefix with ^ to exclude — `lsof -u ^root` lists everything root does not own. |
-a | AND the selection options together instead of OR-ing them. Essential the moment you combine -u and -i. |
-t | Terse mode: print PIDs only. Built for `kill $(lsof -t -i :PORT)`. |
+L1 | Open files whose link count is below 1 — the deleted-but-held files behind a full disk that df and du disagree about. |
+c 0 | Do not truncate command names. The COMMAND column silently stops at 9 characters unless you raise or remove the limit. |
Lsof was written by Victor A. Abell at Purdue University; since revision 4.93.0 it is maintained by the lsof-org team on GitHub, the man page on this box is revision 4.95.0. Abell's fingerprints are permanently in the documentation: the examples use his login name `abe`, his home directory `/Homes/abe` and the machine `lsof.itap.purdue.edu`. Very few tools have embedded their author's personal habits in a manual page that ships on millions of servers.
Re-stat-ing every mount point on every run was too slow, so lsof keeps a device cache file in your home directory, named `.lsof_<hostname>` by default (the format string in the source is `%h/%p.lsof_%L`). It also honours the LSOFPERSDCPATH environment variable, which inserts a directory of your choice into that path — ignored when lsof runs setuid-root. `lsof -D?` prints the path it would use and `-D b` builds the file; most people never learn it exists, but it is why the first lsof after a reboot feels slower than the rest.