tr — translate or delete characters

$ df -h | cut -d' ' -f5 | head -4
df -h | tr -s ' ' | cut -d' ' -f5 | head -4

421G
Use%
1%
36%
97%

Both halves are real, from this box. df pads its Size/Used/Avail columns to a fixed width, so with a plain single-space delimiter the fifth field is blank and the column you wanted only surfaces when the padding happens to run out — here on one line, printing 421G. Put `tr -s ' '` in front and field 5 is Use% down the whole file: 1% for /run, 36% for the EFI vars, 97% for /. This one idiom is worth more than half the flags.

$ tr -s ' ' < sys.log | cut -d' ' -f4
cut -d' ' -f4 sys.log

web1
web1
web2

08:12:03
08:12:05
web2

Three lines of syslog, and the third line is the tell: syslog pads a one-digit day to 'Oct 8' with a double space, so without -s the fourth field is the timestamp on two lines and lands on the hostname only on the one line that happens to use 'Oct 08'. The pipeline does not crash, it just quietly reports '08:12:03' as a host name. `tr -s ' ' < sys.log | cut -d' ' -f5-` then gives you the message text of every line.

$ docker logs kmail-app --tail 4000 | grep -E '^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+ ' > acc.log
tr -s ' ' < acc.log | cut -d' ' -f9 | sort | uniq -c | sort -rn

    410 404
    140 200

550 real request lines from the kmail-app container on this host. Nginx separates its fields with single spaces, but tr -s is the habit that makes the same pipeline work on an Apache log or a line with a doubled space after the timestamp. Swap -f9 for -f7 and you get the URL tally: 74 hits each on /api/trpc, /api/auth/signin, /api/auth/session and /api/auth/callback — bot traffic fishing for admin endpoints, which is why 410 of those 550 requests were 404s.

$ tr -cs '[:alpha:]' '[\n*]' < page.txt | tr '[:upper:]' '[:lower:]' | sort | uniq -c | sort -rn | head -5

      4 the
      3 is
      1 what
      1 turns
      1 tr

Word frequency in one pipe, over a two-sentence paragraph — 166 bytes, 4 occurrences of 'the'. `-cs` is the pairing to remember: complement plus delete throws away everything that is not a letter, and squeeze collapses each run of the leftover newlines into one. '[\n*]' is the ARRAY2 filler that repeats newline as many times as ARRAY1 is long, so you never have to count characters; POSIX uses this exact line as its official example. Drop -s and you get one blank line per deleted character, hundreds of them.

$ file orders.csv
cut -d, -f1 orders.csv | head -2 | xxd
tr -d '\357\273\277' < orders.csv > orders.clean.csv && file orders.clean.csv

orders.csv: CSV Unicode text, UTF-8 (with BOM) text
00000000: efbb bf6f 7264 6572 5f69 640a 3130 3031  ...order_id.1001
00000010: 0a                                       .
orders.clean.csv: CSV ASCII text

The BOM: three bytes ef bb bf sitting in front of 'order_id', so the header column is really named '\ufefforder_id' and every `if col == 'order_id'` check fails while the file looks perfect in a spreadsheet. Octal is the argument form tr wants — '\357' is 0xEF, '\273' is 0xBB, '\277' is 0xBF. file(1) confirms the cure: the copy is plain ASCII now.

$ od -c nul.txt
tr -d '\000' < nul.txt | cat -A
tr -cd '[:print:]\n' < ctrl.log | cat -v

0000000   a  \0   b  \0   c  \n
0000006
abc$
ok^Abad^Bline^G
okbadline

Two invisible-byte cases with the evidence shown. NUL bytes in a Postgres COPY or a captured dump make C string parsers stop at the first one; `tr -d '\000'` removes them and leaves the text. The second command strips every control character from a captured log — cat -v is how you see them at all (^A, ^B, ^G). If your locale is misbehaving, the byte-exact form is `tr -cd '\11\12\15\40-\176'`, which keeps tab, newline, CR and printable ASCII and nothing else.

$ find crlf2 -name '*.csv' -type f -exec sh -c 'tr -d "\r" < "$1" > "$1.tmp" && mv "$1.tmp" "$1"' _ {} \;
find crlf2 -name '*.csv' -type f -exec wc -c {} \;
printf 'before: %s bytes\n' "$(wc -c < clobber.csv)"
tr -d '\r' < clobber.csv > clobber.csv
printf 'after:  %s bytes\n' "$(wc -c < clobber.csv)"

15 crlf2/a.csv
14 crlf2/nested/b.csv
before: 17 bytes
after:  0 bytes

Top half: a small Windows-exported tree, one carriage return removed per line, 17→15 and 16→14 bytes, written to a .tmp and moved back over the original so the rename is atomic. Bottom half: the same conversion written the way it feels natural, which empties the file. `tr -d '\r' < f > f` is the single most expensive five seconds in shell: the redirection truncates before tr reads, and tr exits 0 because tr never had an error.

$ printf 'café ÜBER naïve\n' | tr 'a-z' 'A-Z'
printf 'Grüße\n' | tr '[:lower:]' '[:upper:]'

CAFé ÜBER NAïVE
GRüßE

Real output from coreutils 9.4 here, and the lesson is the shape of what did not change: the accented characters stayed exactly as they were. tr works on bytes, so it can see é as the two bytes c3 a9 but has no idea they are one letter — and case conversion, class deletion and -cs word splitting all inherit that blind spot. `LC_ALL=C tr` gives the same answer, just deliberately. If you need real Unicode case folding, tr is the wrong tool; iconv, sed or the Heirloom implementation are the ones that can see characters.

$ tr 'A-Za-z' 'N-ZA-Mn-za-m' <<< 'Pack My Box With Five Dozen Liquor Jugs'
tr 'A-Za-z' 'N-ZA-Mn-za-m' <<< 'Cnpx Zl Obk Jvgu Svir Qbmra Yvdhbe Whtf'

Cnpx Zl Obk Jvgu Svir Qbmra Yvdhbe Whtf
Pack My Box With Five Dozen Liquor Jugs

ROT13, and the only one of these you can do on any box without installing anything. Note how ARRAY2 is built: N-Z wraps the upper half of the alphabet back round to A-M, lowercase after it, which is why tr needs no wrapping logic — the rotation is baked into the two ranges. The same trick gives you ROT-N by shifting both endpoints, as long as the range stays ascending.