Big Data with Apache Spark

Apache Spark processes data sets too big for one machine by splitting them across a cluster. Spark is written in Scala, and its Scala API feels like the collection operations you already know.

A first job

// illustrative
import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.functions.*

case class Trip(city: String, distanceKm: Double, fare: Double)

@main def trips(): Unit =
  val spark = SparkSession.builder.appName("trips").master("local[*]").getOrCreate()
  import spark.implicits.*

  val trips = spark.read
    .option("header", "true").option("inferSchema", "true")
    .csv("trips.csv")
    .as[Trip]

  trips
    .filter(_.distanceKm > 1)
    .groupBy($"city")
    .agg(count("*").as("trips"), round(avg($"fare"), 2).as("avgFare"))
    .orderBy(desc("trips"))
    .show()

  spark.stop()

+------+-----+-------+
|  city|trips|avgFare|
+------+-----+-------+
|London| 1824|  18.42|
| Paris| 1377|  21.07|
|Berlin|  903|  15.66|
+------+-----+-------+

• A Dataset[Trip] is a distributed collection of case classes; filter(_.distanceKm > 1) is type-checked like a normal Scala lambda.
• A DataFrame (Dataset[Row]) uses column expressions ($"city") that Spark's optimizer understands best.
• Transformations are lazy; nothing runs until an action like show, count, or write.
• master("local[*]") runs on all local cores — perfect for development. On a cluster, the same code runs across many machines.

Things to know

• Scala versions — check which Scala versions your Spark release supports. Spark 3 was built for Scala 2.12/2.13; Spark 4 for Scala 2.13. Scala 3 code can use Spark through Scala 2.13 artifacts, but most Spark projects still use Scala 2.13 syntax.
• Shuffles are expensive — operations like groupBy and join move data across the network; filter and select columns early.
• Write columnar formats — Parquet or Delta Lake instead of CSV.
• Structured Streaming uses the same API for real-time data from Kafka.