kmail.at
← learning

shell · difficulty ◆◆

awk — pattern scanning and text processing

The one-liner language that turns columns into answers.

You have watched someone type `awk -F, '{print $1, $3}'` and assumed it was wizardry. It is not magic — it is a whole miniature programming language hiding inside a single command, and it has been there since 1977.

2026-08-29 · 6 min read

$ awk

What it does

awk reads input line by line, splits each line into fields (by whitespace, or by any delimiter you set with -F), and runs a program against it. The program is a list of pattern { action } pairs: when the pattern matches a line, the action runs. No pattern means every line; no action means print. That tiny grammar — pattern, action, fields $1 $2 $3, and the special variables NR (line number) and NF (field count) — is enough to do real work: sum a column, filter rows, reformat output, count groups.

Why it matters

awk is the fastest way to answer 'what is the average of column 3?' or 'which rows have a value over 80000?' without opening a spreadsheet or writing a Python script. It is the middle ground between grep (which only finds lines) and a full language. Where grep stops at matching, awk computes. It is guaranteed present on every Unix-like system, and its END block lets you summarize an entire file after the last line — the pattern that turns a log dump into a report.

Example

$ awk -F, '{print $1, $3}' employees.csv
name salary
alice 95000
bob 72000
carol 88000
dave 65000
eve 101000

-F, sets the field separator to a comma. $1 and $3 are the first and third fields of each line.

$ awk -F, 'NR>1 {sum+=$3} END {print "avg:", sum/(NR-1)}' employees.csv
avg: 84200

NR is the line number. NR>1 skips the header, the END block runs once after the last line, and (NR-1) is the data-row count.

$ awk -F, '$2=="eng" {print $1}' employees.csv
alice
carol
eve

A pattern that filters: only lines whose second field equals eng trigger the print.

$ awk -F, 'NR>1 && $3>80000 {print $1, $3}' employees.csv
alice 95000
carol 88000
eve 101000

Combine conditions with &&. This is grep-plus-math: filter by a numeric threshold on one field.

$ awk -F, 'NR>1 {print $1, $3*1.1}' employees.csv
alice 104500
bob 79200
carol 96800
dave 71500
eve 111100

awk does arithmetic on fields. A 10% raise, computed across the whole file in one pass.

$ awk -F, 'NR>1 {a[$2]++} END {for (d in a) print d, a[d]}' employees.csv
eng 3
sales 2

Associative arrays count occurrences per group. This is a GROUP BY in one line — the classic awk party trick.

Common flags

-F,
set the field separator (here a comma); default is whitespace
-v var=val
assign a variable before the program runs, e.g. -v threshold=80000
-f file
read the awk program from a file instead of the command line
NR
built-in: the current line number (1-based)
NF
built-in: the number of fields in the current line
$0
the whole current line; $1, $2, ... are the individual fields
END { }
block that runs once after the last line — for summaries

History

Origin

awk was created in 1977 at Bell Labs by Alfred Aho, Peter Weinberger, and Brian Kernighan — the initials A, W, and K are the name. It was designed as a data-driven language for text processing, a step up from grep that could compute as well as match. The original was a single C program; the modern GNU version (gawk) is a full language with arrays, functions, and even a debugger.

The name that stuck

The three authors' surnames — Aho, Weinberger, Kernighan — are the whole story of the name. Kernighan (of K&R C fame) wrote the original implementation. The language was so useful that it outlived its original purpose and became a general text-processing tool, and its one-liner style directly influenced Perl, which borrowed awk's field-splitting and $1-style variables.

Fun facts

Pros & cons

pros

  • + One command does filtering, arithmetic, and grouping — no script file needed
  • + Guaranteed present on every Unix-like system since 1977
  • + The END block makes whole-file summaries trivial
  • + Associative arrays give you GROUP BY in a single line

cons

  • − The pattern { action } syntax is terse and easy to misread at a glance
  • − Field splitting on whitespace surprises people with delimited data
  • − For anything beyond a few lines, a real script (Python, Perl) is more readable
  • − GNU awk and BSD awk differ subtly in some edge behaviors

Takeaways

  1. 1Split on a delimiter with -F, and reference fields as $1, $2, $3 — $0 is the whole line
  2. 2Sum a column and skip the header: `awk -F, 'NR>1 {sum+=$3} END {print sum}' file`
  3. 3Filter with a pattern: `awk -F, '$3>80000 {print $1}' file`
  4. 4Group and count with arrays: `awk '{a[$1]++} END {for (k in a) print k, a[k]}' file`
  5. 5Use -v to pass variables in, and reach for a real script once the logic outgrows a line

Related commands

← all learning