Distributed Computing

Threads share one process's memory. The Distributed standard library runs separate processes (workers) — on your machine or on other machines in a cluster — and moves data between them.

using Distributed

addprocs(2)                       # start two local worker processes
println("workers: ", workers())

@everywhere function collatz_len(n)
    steps = 0
    while n != 1
        n = iseven(n) ? n ÷ 2 : 3n + 1
        steps += 1
    end
    return steps
end

lengths = pmap(collatz_len, 1:10)
println(lengths)

total = @distributed (+) for i in 1:1000
    collatz_len(i)
end
println(total)

workers: [2, 3]
[0, 1, 7, 2, 5, 8, 16, 3, 19, 6]
59542

• addprocs(n) starts workers; julia -p 4 starts four at launch.
• @everywhere defines code on all processes — workers don't see functions defined only on the main process.
• pmap(f, collection) distributes calls one at a time — good for expensive, uneven tasks.
• @distributed (op) for ... splits a loop into chunks and combines the results with op — good for many cheap iterations.

Remote calls and futures

using Distributed
addprocs(2)

f = remotecall(() -> (myid(), sum(1:100)), workers()[1])
println(fetch(f))

r = @spawnat workers()[2] (myid(), rand() < 1)
println(fetch(r))
println(myid())

(2, 5050)
(3, true)
1

remotecall and @spawnat return a Future; fetch waits for and transfers the result. The main process always has ID 1.

Threads or processes?

Clusters: ClusterManagers.jl starts workers through Slurm, PBS, Kubernetes, and others. For GPUs, see CUDA.jl, AMDGPU.jl, Metal.jl, and KernelAbstractions.jl, which write one kernel for several GPU vendors.