Hands-On Scala Programming
Hands-on Scala Programming
1Hands-on Scala1.1 Why Scala?151.2 Why This Book?161.3 How This Book Is Organized171.4 Code Snippet and Examples191.5 Online Materials21package app
object MinimalApplication extends cask.MainRoutes {
  @cask.get("/")
  def hello() = {
    "Hello World!"
  }

  initialize()
}
</> 1.1.scala
Snippet 1.1: a tiny Scala web app, one of many example programs we will encounter in this book
Hands-on Scala teaches you how to use the Scala programming language in a
practical, project-based fashion. Rather than trying to develop expertise in the
deep details of the Scala language itself, Hands-on Scala aims to develop
expertise using Scala in a broad range of practical applications. This book
takes you from "hello world" to building interactive websites, parallel web
crawlers, and distributed applications in Scala.
The book covers the concrete skills necessary for anyone using Scala
professionally: handling files, data serializing, querying databases,
concurrency, and so on. Hands-on Scala will guide you through completing
several non-trivial projects which reflect the applications you may end up
building as part of a software engineering job. This will let you quickly hit
the ground running using Scala professionally.
2Setting Up2.1 Windows Setup (Optional)252.2 Installing Java252.3 Installing Ammonite262.4 Installing Mill302.5 IDE Support32$ amm
Loading...
Welcome to the Ammonite Repl 2.2.0 (Scala 2.13.2 Java 11.0.7)
@ 1 + 1
res0: Int = 2

@ println("hello world" + "!" * 10)
hello world!!!!!!!!!!
</> 2.1.scala
Snippet 2.1: getting started with the Ammonite Scala REPL
In this chapter, we will set up a simple Scala programming environment, giving
you the ability to write, run, and test your Scala code. We will use this setup
throughout the rest of the book. It will be a simple setup, but enough so you
can get productive immediately with the Scala language.
Setting up your development environment is a crucial step in learning a new
programming language. Make sure you get the setup in this chapter working. If
you have issues, come to the online chat room https://www.handsonscala.com/chat
to get help resolving them so you can proceed with the rest of the book in peace
without tooling-related distractions.
3Basic Scala3.1 Values393.2 Loops, Conditionals, Comprehensions463.3 Methods and Functions503.4 Classes and Traits53for (i <- Range.inclusive(1, 100)) {
  println(
    if (i % 3 == 0 && i % 5 == 0) "FizzBuzz"
    else if (i % 3 == 0) "Fizz"
    else if (i % 5 == 0) "Buzz"
    else i
  )
}
</> 3.1.scala
Snippet 3.1: the popular "FizzBuzz" programming challenge, implemented in Scala
This chapter is a quick tour of the Scala language. For now we will focus on the
basics of Scala that are similar to what you might find in any mainstream
programming language.
The goal of this chapter is to get familiar you enough that you can take the
same sort of code you are used to writing in some other language and write it in
Scala without difficulty. This chapter will not cover more Scala-specific
programming styles or language features: those will be left for Chapter 5: Notable Scala Features.
4Scala Collections4.1 Operations594.2 Immutable Collections654.3 Mutable Collections714.4 Common Interfaces75@ def stdDev(a: Array[Double]): Double = {
    val mean = a.sum / a.length
    val squareErrors = a.map(x => x - mean).map(x => x * x)
    math.sqrt(squareErrors.sum / a.length)
  }
</> 4.1.scala
Snippet 4.1: calculating the standard deviation of an array using Scala Collection operations
The core of the Scala standard library is its collections: a common set of
containers and data structures that are shared by all Scala programs. Scala's
collections make it easy for you to manipulate arrays, linked lists, sets, maps
and other data structures in convenient ways, providing built-in many of the
data structures needed for implementing a typical application.
This chapter will walk through the common operations that apply to all
collection types, before discussing the individual data structures and when you
might use each of them in practice.
5Notable Scala Features5.1 Case Classes and Sealed Traits815.2 Pattern Matching835.3 By-Name Parameters895.4 Implicit Parameters915.5 Typeclass Inference94@ def getDayMonthYear(s: String) = s match {
    case s"$day-$month-$year" => println(s"found day: $day, month: $month, year: $year")
    case _ => println("not a date")
  }

@ getDayMonthYear("9-8-1965")
found day: 9, month: 8, year: 1965

@ getDayMonthYear("9-8")
not a date
</> 5.1.scala
Snippet 5.1: using Scala's pattern matching feature to parse simple string patterns
This chapter will cover some of the more interesting and unusual features of
Scala. For each such feature, we will cover both what the feature does as well
as some common use cases to give you an intuition for what it is useful for.
Not every feature in this chapter will be something you use day-to-day.
Nevertheless, even these less-commonly-used features are used often enough that
it is valuable to have a high-level understanding for when you eventually
encounter them in the wild.
6Implementing Algorithms in Scala6.1 Merge Sort1076.2 Prefix Tries1116.3 Breadth First Search1176.4 Shortest Paths120def breadthFirstSearch[T](start: T, graph: Map[T, Seq[T]]): Set[T] = {
  val seen = collection.mutable.Set(start)
  val queue = collection.mutable.ArrayDeque(start)
  while (queue.nonEmpty) {
    val current = queue.removeHead()
    for (next <- graph(current) if !seen.contains(next)) {
      seen.add(next)
      queue.append(next)
    }
  }
  seen.toSet
}
</> 6.1.scala
Snippet 6.1: a simple breadth-first-search algorithm we will implement using Scala in this chapter
In this chapter, we will walk you through the implementation of a number of
common algorithms using the Scala programming language. These algorithms are
commonly taught in schools and tested at professional job interviews, so you
have likely seen them before.
By implementing them in Scala, we aim to get you more familiar with using the
Scala programming language to solve small problems in isolation. We will also
see how some of the unique language features we saw in Chapter 5: Notable Scala Features can be applied to simplify the implementation of these
well-known algorithms. This will prepare us for subsequent chapters which will
expand in scope to include many different kinds of systems, APIs, tools and
techniques.
7Files and Subprocesses7.1 Paths1277.2 Filesystem Operations1297.3 Folder Syncing1337.4 Simple Subprocess Invocations1377.5 Interactive and Streaming Subprocesses141@ os.walk(os.pwd).filter(os.isFile).map(p => (os.size(p), p)).sortBy(-_._1).take(5)
res60: IndexedSeq[(Long, os.Path)] = ArrayBuffer(
  (6340270L, /Users/lihaoyi/test/post/Reimagining/GithubHistory.gif),
  (6008395L, /Users/lihaoyi/test/post/SmartNation/routes.json),
  (5499949L, /Users/lihaoyi/test/post/slides/Why-You-Might-Like-Scala.js.pdf),
  (5461595L, /Users/lihaoyi/test/post/slides/Cross-Platform-Development-in-Scala.js.pdf),
  (4576936L, /Users/lihaoyi/test/post/Reimagining/FluentSearch.gif)
)
</> 7.1.scala
Snippet 7.1: a short Scala code snippet to find the five largest files in a directory tree
Working with files and subprocesses is one of the most common things you do in
programming: from the Bash shell, to Python or Ruby scripts, to large
applications written in a compiled language. At some point everyone will have to
write to a file or talk to a subprocess. This chapter will walk you through how
to perform basic file and subprocess operations in Scala.
This chapter finishes with two small projects: building a simple file
synchronizer, and building a streaming subprocess pipeline. These projects will
form the basis for Chapter 17: Multi-Process Applications and Chapter 18: Building a Real-time File Synchronizer
8JSON and Binary Data Serialization8.1 Manipulating JSON1478.2 JSON Serialization of Scala Data Types1508.3 Writing your own Generic Serialization Methods1548.4 Binary Serialization157@ val output = ujson.Arr(
    ujson.Obj("hello" -> "world", "answer" -> 42),
    true
  )

@ output(0)("hello") = "goodbye"

@ output(0)("tags") = ujson.Arr("awesome", "yay", "wonderful")

@ println(output)
[{"hello":"goodbye","answer":42,"tags":["awesome","yay","wonderful"]},true]
</> 8.1.scala
Snippet 8.1: manipulating a JSON tree structure in the Scala REPL
Data serialization is an important tool in any programmer's toolbox. While
variables and classes are enough to store data within a process, most data tends
to outlive a single program process: whether saved to disk, exchanged between
processes, or sent over the network. This chapter will cover how to serialize
your Scala data structures to two common data formats - textual JSON and binary
MessagePack - and how you can interact with the structured data in a variety of
useful ways.
The JSON workflows we learn in this chapter will be used later in Chapter 12: Working with HTTP APIs and Chapter 14: Simple Web and API Servers, while the
binary serialization techniques we learn here will be used later in Chapter 17: Multi-Process Applications.
9Self-Contained Scala Scripts9.1 Reading Files Off Disk1639.2 Rendering HTML with Scalatags1649.3 Rendering Markdown with Commonmark-Java1669.4 Links and Bootstrap1709.5 Optionally Deploying the Static Site174os.write(
  os.pwd / "out" / "index.html",
  doctype("html")(
    html(
      body(
        h1("Blog"),
        for ((_, suffix, _) <- postInfo)
        yield h2(a(href := ("post/" + mdNameToHtml(suffix)))(suffix))
      )
    )
  )
)
</> 9.1.scala
Snippet 9.1: rendering a HTML page using the third-party Scalatags HTML library
Scala Scripts are a great way to write small programs. Each script is
self-contained and can download its own dependencies when necessary, and make
use of both Java and Scala libraries. This lets you write and distribute scripts
without spending time fiddling with build configuration or library installation.
In this chapter, we will write a static site generator script that uses
third-party libraries to process Markdown input files and generate a set of HTML
output files, ready for deployment on any static file hosting service. This will
form the foundation for Chapter 10: Static Build Pipelines, where we will
turn the static site generator into an efficient incremental build pipeline by
using the Mill build tool.
10Static Build Pipelines10.1 Mill Build Pipelines17910.2 Mill Modules18310.3 Revisiting our Static Site Script18710.4 Conversion to a Mill Build Pipeline18810.5 Extending our Static Site Pipeline192import mill._

def srcs = T.source(millSourcePath / "src")

def concat = T{
  os.write(T.dest / "concat.txt",  os.list(srcs().path).map(os.read(_)))
  PathRef(T.dest / "concat.txt")
}
</> 10.1.scala
Snippet 10.1: the definition of a simple Mill build pipeline
Build pipelines are a common pattern, where you have files and assets you want
to process but want to do so efficiently, incrementally, and in parallel. This
usually means only re-processing files when they change, and re-using the
already processed assets as much as possible. Whether you are compiling Scala,
minifying Javascript, or compressing tarballs, many of these file-processing
workflows can be slow. Parallelizing these workflows and avoiding unnecessary
work can greatly speed up your development cycle.
This chapter will walk through how to use the
Mill build tool to set up these build pipelines,
and demonstrate the advantages of a build pipeline over a naive build script. We
will take the the simple static site generator we wrote in Chapter 9: Self-Contained Scala Scripts and convert it into an efficient build pipeline
that can incrementally update the static site as you make changes to the
sources. We will be using the Mill build tool in several of the projects later
in the book, starting with Chapter 14: Simple Web and API Servers.
11Scraping Websites11.1 Scraping Wikipedia20311.2 MDN Web Documentation20711.3 Scraping MDN20911.4 Putting it Together213@ val doc = Jsoup.connect("http://en.wikipedia.org/").get()

@ doc.title()
res2: String = "Wikipedia, the free encyclopedia"

@ val headlines = doc.select("#mp-itn b a")
headlines: select.Elements =
<a href="/wiki/Bek_Air_Flight_2100" title="Bek Air Flight 2100">Bek Air Flight 2100</a>
<a href="/wiki/Assassination_of_..." title="Assassination of ...">2018 killing</a>
<a href="/wiki/State_of_the_..." title="State of the...">upholds a ruling</a>
...
</> 11.1.scala
Snippet 11.1: scraping Wikipedia's front-page links using the Jsoup third-party library in the Scala REPL
The user-facing interface of most networked systems is a website. In fact, often
that is the only interface! This chapter will walk you through using the Jsoup
library from Scala to scrape human-readable HTML pages, unlocking the ability to
extract data from websites that do not provide access via an API.
Apart from third-party scraping websites, Jsoup is also a useful tool for
testing the HTML user interfaces that we will encounter in Chapter 14: Simple Web and API Servers. This chapter is also a chance to get more familiar with
using Java libraries from Scala, a necessary skill to take advantage of the
broad and deep Java ecosystem. Lastly, it is an exercise in doing non-trivial
interactive development in the Scala REPL, which is a great place to prototype
and try out pieces of code that are not ready to be saved in a script or
project.
12Working with HTTP APIs12.1 The Task: Github Issue Migrator21912.2 Creating Issues and Comments22112.3 Fetching Issues and Comments22312.4 Migrating Issues and Comments228@ requests.post(
    "https://api.github.com/repos/lihaoyi/test/issues",
    data = ujson.Obj("title" -> "hello"),
    headers = Map("Authorization" -> s"token $token")
  )
res1: requests.Response = Response(
  "https://api.github.com/repos/lihaoyi/test/issues",
  201,
  "Created",
...
</> 12.1.scala
Snippet 12.1: interacting with Github's HTTP API from the Scala REPL
HTTP APIs have become the standard for any organization that wants to let
external developers integrate with their systems. This chapter will walk you
through how to access HTTP APIs in Scala, building up to a simple use case:
migrating Github issues from one repository to another using Github's public
API.
We will build upon techniques learned in this chapter in Chapter 13: Fork-Join Parallelism with Futures, where we will be writing a parallel web crawler using
the Wikipedia JSON API to walk the graph of articles and the links between them.
13Fork-Join Parallelism with Futures13.1 Parallel Computation using Futures23713.2 N-Ways Parallelism24013.3 Parallel Web Crawling24313.4 Asynchronous Futures24813.5 Asynchronous Web Crawling252def fetchAllLinksParallel(startTitle: String, depth: Int): Set[String] = {
  var seen = Set(startTitle)
  var current = Set(startTitle)
  for (i <- Range(0, depth)) {
    val futures = for (title <- current) yield Future{ fetchLinks(title) }
    val nextTitleLists = futures.map(Await.result(_, Inf))
    current = nextTitleLists.flatten.filter(!seen.contains(_))
    seen = seen ++ current
  }
  seen
}
</> 13.1.scala
Snippet 13.1: a simple parallel web-crawler implemented using Scala Futures
The Scala programming language comes with a Futures API. Futures make parallel
and asynchronous programming much easier to handle than working with traditional
techniques of threads, locks, and callbacks.
This chapter dives into Scala's Futures: how to use them, how they work, and how
you can use them to parallelize data processing workflows. It culminates in
using Futures together with the techniques we learned in Chapter 12: Working with HTTP APIs to write a high-performance concurrent web crawler in a
straightforward and intuitive way.
14Simple Web and API Servers14.1 A Minimal Webserver25914.2 Serving HTML26314.3 Forms and Dynamic Data26514.4 Dynamic Page Updates via API Requests27214.5 Real-time Updates with Websockets276object MinimalApplication extends cask.MainRoutes {
  @cask.get("/")
  def hello() = {
    "Hello World!"
  }

  @cask.post("/do-thing")
  def doThing(request: cask.Request) = {
    request.text().reverse
  }

  initialize()
}
</> 14.1.scala
Snippet 14.1: a minimal Scala web application, using the Cask web framework
Web and API servers are the backbone of internet systems. While in the last few
chapters we learned to access these systems from a client's perspective, this
chapter will teach you how to provide such APIs and Websites from the server's
perspective. We will walk through a complete example of building a simple
real-time chat website serving both HTML web pages and JSON API endpoints. We
will re-visit this website in Chapter 15: Querying SQL Databases, where we
will convert its simple in-memory datastore into a proper SQL database.
15Querying SQL Databases15.1 Setting up Quill and PostgreSQL28515.2 Mapping Tables to Case Classes28715.3 Querying and Updating Data29015.4 Transactions29515.5 A Database-Backed Chat Website297@ ctx.run(query[City].filter(_.population > 5000000).filter(_.countryCode == "CHN"))
res16: List[City] = List(
  City(1890, "Shanghai", "CHN", "Shanghai", 9696300),
  City(1891, "Peking", "CHN", "Peking", 7472000),
  City(1892, "Chongqing", "CHN", "Chongqing", 6351600),
  City(1893, "Tianjin", "CHN", "Tianjin", 5286800)
)
</> 15.1.scala
Snippet 15.1: using the Quill database query library from the Scala REPL
Most modern systems are backed by relational databases. This chapter will walk
you through the basics of using a relational database from Scala, using the
Quill query library. We will work through small self-contained examples of how
to store and query data within a Postgres database, and then convert the
interactive chat website we implemented in Chapter 14: Simple Web and API Servers to use a Postgres database for data storage.
16Message-based Parallelism with Actors16.1 Castor Actors30716.2 Actor-based Background Uploads30816.3 Concurrent Logging Pipelines31416.4 Debugging Actors321class SimpleUploadActor()(implicit cc: castor.Context) extends castor.SimpleActor[String]{
  def run(msg: String) = {
    val res = requests.post("https://httpbin.org/post", data = msg)
    println("response " + res.statusCode)
  }
}
</> 16.1.scala
Snippet 16.1: a simple actor implemented in Scala using the Castor library
Message-based parallelism is a technique that involves splitting your
application logic into multiple "actors", each of which can run concurrently,
and only interacts with other actors by exchanging asynchronous messages. This
style of programming was popularized by the Erlang
programming language and the Akka Scala actor library, but
the approach is broadly useful and not limited to any particular language or
library.
This chapter will introduce the fundamental concepts of message-based
parallelism with actors, and how to use them to achieve parallelism in scenarios
where the techniques we covered in Chapter 13: Fork-Join Parallelism with Futures cannot be applied. We will first discuss the basic actor APIs, see how
they can be used in a standalone use case, and then see how they can be used in
more involved multi-actor pipelines. The techniques in this chapter will come in
useful later in Chapter 18: Building a Real-time File Synchronizer.
17Multi-Process Applications17.1 Two-Process Build Setup32717.2 Remote Procedure Calls33017.3 The Agent Process33217.4 The Sync Process33417.5 Pipelined Syncing337def send[T: Writer](out: DataOutputStream, msg: T): Unit = {
  val bytes = upickle.default.writeBinary(msg)
  out.writeInt(bytes.length)
  out.write(bytes)
  out.flush()
}
def receive[T: Reader](in: DataInputStream) = {
  val buf = new Array[Byte](in.readInt())
  in.readFully(buf)
  upickle.default.readBinary[T](buf)
}
</> 17.1.scala
Snippet 17.1: RPC send and receive methods for sending data over an operating system pipe or network
While all our programs so far have run within a single process, in real world
scenarios you will be working as part of a larger system, and the application
itself may need to be split into multiple processes. This chapter will walk you
through how to do so: configuring your build tool to support multiple Scala
processes, sharing code and exchanging serialized messages. These are the
building blocks that form the foundation of any distributed system.
As this chapter's project, we will be building a simple multi-process file
synchronizer that can work over a network. This chapter builds upon the simple
single-process file synchronizer in Chapter 7: Files and Subprocesses, and
will form the basis for Chapter 18: Building a Real-time File Synchronizer.
18Building a Real-time File Synchronizer18.1 Watching for Changes34518.2 Real-time Syncing with Actors34618.3 Testing the Syncer35318.4 Pipelined Real-time Syncing35518.5 Testing the Pipelined Syncer358object SyncActor extends castor.SimpleActor[Msg]{
  def run(msg: Msg): Unit = msg match {
    case ChangedPath(value) => Shared.send(agent.stdin.data, Rpc.StatPath(value))
    case AgentResponse(Rpc.StatInfo(p, remoteHash)) =>
      val localHash = Shared.hashPath(src / p)
      if (localHash != remoteHash && localHash.isDefined) {
        Shared.send(agent.stdin.data, Rpc.WriteOver(os.read.bytes(src / p), p))
      }
  }
}
</> 18.1.scala
Snippet 18.1: an actor used as part of our real-time file synchronizer
In this chapter, we will write a file synchronizer that can keep the destination
folder up to date even as the source folder changes over time. This chapter
serves as a capstone project, tying together concepts from Chapter 17: Multi-Process Applications and Chapter 16: Message-based Parallelism with Actors.
The techniques in this chapter form the basis for "event driven" architectures,
which are common in many distributed systems. Real-time file synchronization is
a difficult problem, and we will see how we can use the Scala language and
libraries to approach it in an elegant and understandable way.
19Parsing Structured Text19.1 Simple Parsers36319.2 Parsing Structured Values36819.3 Implementing a Calculator37219.4 Parser Debugging and Error Reporting377@ def parser[_: P] =
    P( ("hello" | "goodbye").! ~ " ".rep(1) ~ ("world" | "seattle").! ~ End )

@ fastparse.parse("hello seattle", parser(_))
res41: Parsed[(String, String)] = Success(("hello", "seattle"), 13)

@ fastparse.parse("hello     world", parser(_))
res42: Parsed[(String, String)] = Success(("hello", "world"), 15)
</> 19.1.scala
Snippet 19.1: parsing simple text formats using the FastParse library
One common programming task is parsing structured text. This chapter will
introduce how to parse text in Scala using the FastParse library, before diving
into an example where we write a simple arithmetic parser in Scala. This will
allow you to work competently with unusual data formats, query languages, or
source code for which you do not already have an existing parser at hand.
We will build upon the parsing techniques learned in this chapter as part of
Chapter 20: Implementing a Programming Language.
20Implementing a Programming Language20.1 Interpreting Jsonnet38520.2 Jsonnet Language Features38520.3 Parsing Jsonnet38720.4 Evaluating the Syntax Tree39620.5 Serializing to JSON402def evaluate(expr: Expr, scope: Map[String, Value]): Value = expr match {
  case Expr.Str(s) => Value.Str(s)
  case Expr.Dict(kvs) => Value.Dict(kvs.map{case (k, v) => (k, evaluate(v, scope))})
  case Expr.Plus(left, right) =>
    val Value.Str(leftStr) = evaluate(left, scope)
    val Value.Str(rightStr) = evaluate(right, scope)
    Value.Str(leftStr + rightStr)
}
</> 20.1.scala
Snippet 20.1: evaluating a syntax tree using pattern matching
This chapter builds upon the simple parsers we learned in Chapter 19: Parsing Structured Text, and walks you through the process of implementing a simple
programming language in Scala.
Working with programming language source code is a strength of Scala: parsing,
analyzing, compiling, or interpreting it. This chapter should will you how easy
it is to write a simple interpreter to parse and evaluate program source code in
Scala. Even if your goal is not to implement an entirely new programming
language, these techniques are still useful: for writing linters, program
analyzers, query engines, and other such tools.
