Optimization

Optimization means finding inputs that minimize (or maximize) a function — fitting models, designing parts, planning schedules.

Gradient descent by hand

# minimize f(x, y) = (x - 3)^2 + 2(y + 1)^2
f(v) = (v[1] - 3)^2 + 2 * (v[2] + 1)^2
∇f(v) = [2 * (v[1] - 3), 4 * (v[2] + 1)]

function gradient_descent(f, ∇f, x0; lr = 0.1, tol = 1e-8, maxiter = 10_000)
    x = copy(x0)
    for iter in 1:maxiter
        g = ∇f(x)
        sqrt(sum(abs2, g)) < tol && return x, iter
        x .-= lr .* g
    end
    return x, maxiter
end

xmin, iters = gradient_descent(f, ∇f, [0.0, 0.0])
println(round.(xmin, digits = 6), " after ", iters, " iterations")

[3.0, -1.0] after 92 iterations

Newton's method for roots

function newton(f, fprime, x; tol = 1e-12)
    for i in 1:50
        step = f(x) / fprime(x)
        x -= step
        abs(step) < tol && return x, i
    end
    error("did not converge")
end

root, n = newton(x -> x^2 - 2, x -> 2x, 1.0)
println(root, " in ", n, " steps; sqrt(2) = ", sqrt(2))

1.414213562373095 in 6 steps; sqrt(2) = 1.4142135623730951

Writing derivatives by hand gets tedious. ForwardDiff.jl and Zygote.jl/Enzyme.jl compute them automatically: ForwardDiff.gradient(f, [0.0, 0.0]).

Optim.jl

# illustrative — needs the Optim package
using Optim

rosenbrock(x) = (1 - x[1])^2 + 100 * (x[2] - x[1]^2)^2
result = optimize(rosenbrock, [0.0, 0.0], BFGS())
Optim.minimizer(result)      # ≈ [1.0, 1.0]
Optim.minimum(result)        # ≈ 0.0

Optim.jl provides Nelder–Mead (no derivatives), BFGS and L-BFGS (gradients), Newton (Hessians), and box constraints. Optimization.jl wraps many optimizers behind one interface.

JuMP: mathematical programming

For linear, integer, and constrained problems, JuMP is a modeling language that reads like the math and talks to many solvers (HiGHS, Ipopt, Gurobi, and others):

# illustrative — needs the JuMP and HiGHS packages
using JuMP, HiGHS

model = Model(HiGHS.Optimizer)
@variable(model, chairs >= 0, Int)
@variable(model, tables >= 0, Int)
@constraint(model, wood, 2chairs + 5tables <= 100)
@constraint(model, labor, 3chairs + 2tables <= 60)
@objective(model, Max, 40chairs + 90tables)
optimize!(model)

value(chairs), value(tables), objective_value(model)

@variable, @constraint, and @objective are macros that turn the formulas into the solver's input format — metaprogramming at work.