The Text Toolkit: sort, uniq, cut, tr, and Friends

Small single-purpose tools, combined with pipes, solve a surprising number of data problems.

sort

printf '%s\n' "banana 3" "apple 12" "cherry 7" "apple 2" > items.txt

sort items.txt
echo "---"
sort -k2 -n items.txt
echo "---"
sort -k2 -nr items.txt | head -n 1

apple 12
apple 2
banana 3
cherry 7
---
apple 2
banana 3
cherry 7
apple 12
---
apple 12

• -n sorts numerically (otherwise 12 sorts before 3).
• -k2 sorts by the second field; -r reverses; -u removes duplicates.
• -t, sets the separator for comma-separated data.

cut, tr, and paste

echo "Ada,36,London" | cut -d, -f1,3
echo "hello world" | tr 'a-z' 'A-Z'
echo "a  lot   of    spaces" | tr -s ' '
echo "remove-the-dashes" | tr -d '-'
printf '%s\n' a b c | paste -sd,

Ada,London
HELLO WORLD
a lot of spaces
removethedashes
a,b,c

uniq and wc

printf '%s\n' red red blue red green green > colors.txt
uniq colors.txt
echo "---"
sort colors.txt | uniq -c
echo "---"
sort colors.txt | uniq -d
echo "---"
wc -l < colors.txt

red
blue
red
green
---
      1 blue
      2 green
      3 red
---
green
red
---
6

uniq only merges adjacent duplicates — which is why it's nearly always used after sort. uniq -d shows only lines that were repeated.

Putting it together

The ten most common words in a text:

cat > text.txt <<'EOF'
The quick brown fox jumps over the lazy dog.
The dog sleeps. The fox runs.
EOF

tr -cs 'A-Za-z' '\n' < text.txt | tr 'A-Z' 'a-z' | sort | uniq -c | sort -rn | head -n 3

      4 the
      2 fox
      2 dog

tr -cs 'A-Za-z' '\n' turns every run of non-letters into a newline — one word per line — and the rest is the "sort, count, sort" pattern.

Other tools worth knowing

column -t (align output into a table), comm (compare sorted lists), join (join two files on a field), diff (compare files — or use the Code Diff tool), and tee (write to a file and pass output along the pipe).