Concurrency and Locking

Running tasks in parallel can make scripts much faster — as long as you control how many run at once and stop them from interfering.

xargs -P: a simple worker pool

process() { sleep 0.2; echo "processed $1"; }
export -f process

printf '%s\n' item1 item2 item3 item4 |
  xargs -P 2 -I {} bash -c 'process "$1"' _ {} | sort

processed item1
processed item2
processed item3
processed item4

• -P 2 runs up to two commands at once.
• -I {} substitutes each input line.
• export -f makes the function available to the child bash processes.
• Output from parallel jobs arrives in whatever order they finish — hence the sort.

GNU parallel is a more powerful alternative with progress bars, retries, and ordered output: parallel -j 4 process ::: item1 item2.

Limiting background jobs in plain Bash

max_jobs=2
for i in 1 2 3 4 5; do
  ( sleep 0.1; echo "job $i done" ) &
  if (( $(jobs -rp | wc -l) >= max_jobs )); then
    wait -n
  fi
done
wait
echo "all done"

wait -n (Bash 4.3+) waits for any one job to finish, so a new one can start — keeping at most two running.

Locking: one copy at a time

A scheduled script that takes longer than its interval could end up running twice at once. flock (on Linux) prevents that:

# illustrative
exec 9> /var/lock/nightly-backup.lock
if ! flock -n 9; then
  echo "Another backup is already running — exiting." >&2
  exit 0
fi
# ... the rest of the script runs with the lock held ...

flock -n takes the lock or fails immediately; the lock is released automatically when the script exits. Without flock, mkdir works as a portable lock, because creating a directory either succeeds or fails atomically:

lock=./myjob.lock
if mkdir "$lock" 2>/dev/null; then
  trap 'rmdir "$lock"' EXIT
  echo "got the lock"
else
  echo "already running"
fi

got the lock

Common mistakes

• Starting unlimited background jobs in a loop (for f in *; do ... & done) — thousands of processes at once can overwhelm the machine.
• Several jobs writing to the same file at once, producing interleaved output. Give each job its own output file, then combine them.