Advanced Projects
Prerequisites: Level 3 — Advanced
Build these using givens, type classes, futures or an effect system, JSON, web frameworks, and Spark from Level 3.
1. REST API Service
Build a JSON API for a small domain (for example, a bookshop) with Tapir or http4s, a type class for validation, opaque types for IDs, and a database layer. Add tests, structured logging, and a Docker image.
2. Concurrent Web Crawler
Crawl pages from a start URL to a set depth with a bounded number of concurrent requests using Cats Effect or ZIO, deduplicate visited links, handle timeouts and retries, and report the results.
3. Spark Data Pipeline
Load a public data set (for example, taxi trips) into a Spark
Dataset of case classes, clean and aggregate it, compare the Dataset and
DataFrame APIs, and write the results to Parquet. Add unit tests for the
transformations.