Processing Columns with awk

awk is a small programming language for processing text arranged in columns — log files, CSV files, command output. For each line, it splits the line into fields you can refer to as $1, $2, and so on.

Printing columns

cat > sales.csv <<'EOF'
region,product,amount
North,Keyboard,120
South,Monitor,250
North,Monitor,300
East,Keyboard,80
South,Keyboard,95
EOF

awk -F, '{ print $1, $3 }' sales.csv

region amount
North 120
South 250
North 300
East 80
South 95

• -F, sets the field separator to a comma (the default is whitespace).
• { print $1, $3 } runs for every line; $0 is the whole line.

Filtering and calculating

cat > sales.csv <<'EOF'
region,product,amount
North,Keyboard,120
South,Monitor,250
North,Monitor,300
East,Keyboard,80
South,Keyboard,95
EOF

awk -F, 'NR > 1 && $3 > 100 { print $2 " sold for " $3 }' sales.csv
echo "---"
awk -F, 'NR > 1 { total += $3 } END { print "Total:", total }' sales.csv
echo "---"
awk -F, 'NR > 1 { sum[$1] += $3 } END { for (r in sum) print r, sum[r] }' sales.csv | sort

Keyboard sold for 120
Monitor sold for 250
Monitor sold for 300
---
Total: 845
---
East 80
North 420
South 345

• A pattern before { } selects lines: NR > 1 skips the header (NR is the line number), $3 > 100 compares numbers.
• Variables like total start at zero; END { } runs after the last line.
• sum[$1] += $3 uses an associative array to total by region — a GROUP BY in one line.

Formatting output

printf 'Ada 36 London\nLinus 28 Helsinki\nGrace 45 New York\n' |
  awk '{ printf "%-8s %3d\n", $1, $2 }'

Ada       36
Linus     28
Grace     45

printf formats like in C: %-8s is left-aligned text 8 wide, %3d a number 3 wide.

Useful built-ins

Common mistakes

• Using double quotes around the awk program: "$1" is then expanded by the shell. Use single quotes, and pass shell variables in with awk -v name="$value".
• Parsing CSV with quoted commas ("Smith, John") — -F, splits inside the quotes. For real CSV, use a CSV-aware tool (Python, mlr, or csvkit).

Practice

Using the sales.csv data above, print the average amount of Keyboard sales.

printf 'region,product,amount\nNorth,Keyboard,120\nEast,Keyboard,80\nSouth,Keyboard,95\nSouth,Monitor,250\n' > sales.csv
awk -F, '$2 == "Keyboard" { sum += $3; n++ } END { printf "%.2f\n", sum / n }' sales.csv

98.33