Data Analysis with DataFrames

DataFrames.jl is Julia's table library, similar to pandas in Python or data frames in R, and CSV.jl reads and writes CSV files quickly. Install both with ] add DataFrames CSV (the examples use DataFrames 1.8 and CSV 1.1).

using DataFrames

df = DataFrame(name = ["Ada", "Alan", "Grace", "Edsger"],
               dept = ["math", "cs", "cs", "cs"],
               salary = [120, 95, 130, 110])
println(df)
println(size(df), " ", names(df))
println(df.salary)
println(df[2, :name], " ", df[df.salary .> 100, :name])

4×3 DataFrame
 Row │ name    dept    salary
     │ String  String  Int64
─────┼────────────────────────
   1 │ Ada     math       120
   2 │ Alan    cs          95
   3 │ Grace   cs         130
   4 │ Edsger  cs         110
(4, 3) ["name", "dept", "salary"]
[120, 95, 130, 110]
Alan ["Ada", "Grace", "Edsger"]

• Each column is an ordinary Julia vector — df.salary is a Vector{Int64}.
• df[rows, cols] indexes like a matrix, with column names as symbols.

Transforming and filtering

using DataFrames

df = DataFrame(name = ["Ada", "Alan", "Grace", "Edsger"],
               dept = ["math", "cs", "cs", "cs"],
               salary = [120, 95, 130, 110])

df.bonus = df.salary .* 0.1
high = filter(:salary => >(100), df)
println(high)
println(sort(df, :salary, rev = true)[:, [:name, :salary]])
println(select(df, :name, :salary => (s -> s .* 1000) => :salary_usd))

3×4 DataFrame
 Row │ name    dept    salary  bonus
     │ String  String  Int64   Float64
─────┼─────────────────────────────────
   1 │ Ada     math       120     12.0
   2 │ Grace   cs         130     13.0
   3 │ Edsger  cs         110     11.0
4×2 DataFrame
 Row │ name    salary
     │ String  Int64
─────┼────────────────
   1 │ Grace      130
   2 │ Ada        120
   3 │ Edsger     110
   4 │ Alan        95
4×2 DataFrame
 Row │ name    salary_usd
     │ String  Int64
─────┼────────────────────
   1 │ Ada         120000
   2 │ Alan         95000
   3 │ Grace       130000
   4 │ Edsger      110000

source => function => new_name is DataFrames' mini-language for transformations — used by select, transform, and combine.

Group, then combine

using DataFrames, Statistics

df = DataFrame(name = ["Ada", "Alan", "Grace", "Edsger"],
               dept = ["math", "cs", "cs", "cs"],
               salary = [120, 95, 130, 110])

summary = combine(groupby(df, :dept),
                  nrow => :people,
                  :salary => mean => :avg_salary,
                  :salary => maximum => :top)
println(summary)

2×4 DataFrame
 Row │ dept    people  avg_salary  top
     │ String  Int64   Float64     Int64
─────┼───────────────────────────────────
   1 │ math         1     120.0      120
   2 │ cs           3     111.667    130

This is the split–apply–combine pattern: groupby splits the rows, and combine computes one row of results per group.

Reading and writing CSV

using CSV, DataFrames

write("people.csv", """
name,age,city
Ada,36,London
Alan,,Wilmslow
Grace,85,Arlington
""")

people = CSV.read("people.csv", DataFrame; stringtype = String)
println(people)
println(eltype(people.age))
dropped = dropmissing(people, :age)
CSV.write("adults.csv", dropped)
print(read("adults.csv", String))

3×3 DataFrame
 Row │ name    age      city
     │ String  Int64?   String
─────┼────────────────────────────
   1 │ Ada          36  London
   2 │ Alan    missing  Wilmslow
   3 │ Grace        85  Arlington
Union{Missing, Int64}
name,age,city
Ada,36,London
Grace,85,Arlington

CSV.jl detects column types and turns empty cells into missing. (stringtype = String asks for ordinary String columns; by default CSV.jl 1.1 uses a compact string type from DataStrings.jl.) Other useful functions: leftjoin/innerjoin to join tables, stack and unstack to reshape, describe(df) for a quick summary. Generate test data with the random CSV generator.