uniq — report or omit repeated lines

What it does

uniq reads lines of input and removes adjacent duplicates, printing each unique line once. Because it only compares neighboring lines, it is almost always paired with sort first — sort groups identical lines together, then uniq collapses them. Add -c and it prefixes each line with a count of how many times it appeared.

Why it matters

uniq is the standard way to answer 'how many times does each thing occur?' in a text stream. Combined with sort, cut, and awk, it turns raw logs and lists into clean summaries: top visitors, most common error codes, most frequent words. It is a small command, but it is the difference between staring at a wall of log lines and seeing the answer.

Example

$ sort access.log | cut -d' ' -f1 | uniq -c | sort -rn | head -5

  482 203.0.113.5
  391 198.51.100.22
  210 192.0.2.77
   88 203.0.113.9
   12 198.51.100.1

$ printf 'apple\nbanana\napple\napple\ncherry\n' | sort | uniq

apple
banana
cherry

Common flags

-c

prefix each line with the number of times it occurred

-d

print only duplicated lines (one copy of each)

-u

print only lines that appear exactly once

-i

ignore case when comparing lines

-f N

skip the first N fields before comparing

-s N

skip the first N characters before comparing

-w N

compare only the first N characters of each line