anchor.diff:  We'll give a more thorough presentation of the _DataFrame_ -based API later in
   this chapter.
   
  -Cluster managers
  -~~~~~~~~~~~~~~~~
  +=== Cluster managers
   
   As mentioned earlier, Apache Spark is a distributed data processing framework.
   To scale to thousands of nodes, it relies on a cluster manager. A cluster
  -manager will try to satisfy Spark's requests for computing power based on which 
  +manager will try to satisfy Spark's requests for computing power based on which
   machines are available in the cluster.
   
   Three cluster managers are supported by Spark: Hadoop YARN, Apache Mesos and
  -Standalone (which comes bundled with Spark).
  -
  -
  -
  +Standalone (which comes bundled with Spark). +
  + +
  + +
   There are two main advantages to having those subprojects being part of Spark:
   you don't have to deal with version compatibility and new features and
   performance improvements propagate almost instantaneously to the other parts of
   Spark.
   
  -All those components can be used simultaneously. For example, you can query
  -in SQL or make predictions using a machine learning model in a streaming
  -application.
  -
  -Core Spark concepts
  --------------------
  -
  -In this section, we'll dig a bit deeper into the core concepts we'll use
  -throughout the book: _DataFrame_ and _Dataset_.
  -
  -Both _DataFrames_ and _Datasets_ can be built from a variety of sources such as
  -JSON, Parquet, tables in a database accessible through JDBC, Hive tables or
  -_RDDs_.
  -
  -DataFrame
  -~~~~~~~~~
  -
  -.An example DataFrame
  -[options="header"]
  -|==================
  -| Column 1 | Column 2 | Column 3
  -| 11 | "value 2 col2" | 13.36
  -| 12 | "value 1 col2" | 23.42
  -| 13 | "value 1 col2" | 33.76
  -|==================
  -
  -As discussed earlier, a _DataFrame_ can be seen as a weakly-typed, distributed
  -SQL table. If you come from R or pandas, the notion of dataframe is conceptually
  -equivalent but distributed.
  -
  -Inside a _DataFrame_, data is organized into rows and named columns. However,
  -the type information available to Spark for a _DataFrame_ doesn't translate into
  -the Scala type system, that is why they are a bit clunky to manipulate and type
  -errors are only seen at runtime.
  -
  -To demonstrate this, we'll go through a quick example where we'll create a
  -small _DataFrame_ and manipulate it. Don't worry if there are some parts you
  -don't totally understand, this is just to show off the _DataFrame_ API and its
  -pitfalls.
  -
  -Let us say that you have the following file containing JSONs at hand:
  -
  -.JSON file
  -[source,json]
  -----
  -{ "name": "Alice", "public_key": "MIIEogIB", "private_key": "656qoAVR" }
  -{ "name": "Bob", "public_key": "2800iBLw", "private_key": "jn1tCP/4" }
  -----
  -
  -
  -Let's load the dataset.
  -
  -.Loading a dataframe
  -[source,scala]
  -----
  -val dataframe = spark.read.json("users.json")
  -----
  -
  -Now, let's find out what was loaded.
  -
  -.Printing the content of a dataframe
  -[source,scala]
  -----
  -dataframe.show()
  -
  -// +-----+-----------+----------+
  -// | name|private_key|public_key|
  -// +-----+-----------+----------+
  -// |Alice|   656qoAVR|  MIIEogIB|
  -// |  Bob|   jn1tCP/4|  2800iBLw|
  -// +-----+-----------+----------+
  -----
  -
  -Next, let's project on the name column.
  -
  -.Projecting on the name column
  -[source,scala]
  -----
  -dataframe.select("name").show()
  -
  -// +-----+
  -// | name|
  -// +-----+
  -// |Alice|
  -// |  Bob|
  -// +-----+
  -----
  -
  -As mentioned earlier, you can also query your _DataFrame_ using vanilla SQL:
  -
  -.Projecting on the name column usin SQL
  -[source,scala]
  -----
  -// We need to register the DataFrame as a SQL table (here as a temporary view)
  -dataframe.createOrReplaceTempView("users")
  -
  -spark.sql("SELECT name FROM users").show()
  -----
  -
  -As you can see, we interact with columns just with strings with no knowledge
  -of the underlying type of the column we're dealing with. As a result, filtering
  -on a number for a string-typed column will work fine:
  -
  -.No type error
  -[source,scala]
  -----
  -dataframe.filter($"name" > 1).show()
  -
  -// +----+-----------+----------+
  -// |name|private_key|public_key|
  -// +----+-----------+----------+
  -// +----+-----------+----------+
  -----
  -
  -The result, an empty _DataFrame_, is kind of difficult to interpret, we would
  -have expected a type error saying that you cannot compare a string and an
  -integer.
  -
  -Dataset
  -~~~~~~~
  -
  -The _Dataset_ API is similar to the _DataFrame_ API in the sense that you still
  -get the optimizations Catalyst provides but you also get the strong typing and
  -API familiarity, if you have a Scala background, from the _RDD_ API. It's
  -basically the best of both worlds.
  -
  -Continuing with out previous example let's see the _Dataset_ API in action.
  -
  -We can turn our _DataFrame_ into a _Dataset_ by defining its model:
  -
  -.Turning a DataFrame into a Dataset
  -[source,scala]
  -----
  -// Our user model, notice that the field names in our model must match the
  -// field names in the data
  -case class User(name: String, public_key: String, private_key: String)
  -val dataset = dataframe.as[User]
  -----
  -
  -You can still project using the _DataFrame_ API as we've shown earlier. But,
  -now you can use the familiar and typesafe collection API from Scala that the
  -_RDD_ API benefits from:
  -
  -.Projecting on the name using the Dataset API
  -[source,scala]
  -----
  -dataset.map(user => user.name).show()
  -
  -// +-----+
  -// |value|
  -// +-----+
  -// |Alice|
  -// |  Bob|
  -// +-----+
  -----
  -
  -The added benefit is that you cannot do whatever you want with the different
  -types in your _Dataset_, this will not compile:
  -
  -.Cannot compare a string and an integer
  -[source,scala]
  -----
  -ds.filter(user => user.name > 1)
  -----
  -
  -What is Spark ML?
  ------------------
  -
  -As we hinted at earlier, the machine learning library in Apache Spark is split
  +Another benefit is that all those components can be used simultaneously. For
  +example, you can query in SQL or make predictions using a machine learning model
  +in a streaming application.
  +
  +== What is Spark ML?
  +
  +Suppose you're working for AirBnB and in charge of implementing the suggestions
  +for the daily rate you see when you want to list your apartment. How would you
  +go about it? One way to do it would be to build a regression model (a notion
  +we'll come back to) which can be summed to the construction of a function with
  +_x_ parameters corresponding to the characteristics of the apartment such as
  +its surface area, its location, the number of bedrooms, etc which outputs a
  +price:
  +
  +.The function we want to obtain taking all the apartment's characteristics as parameters and outputting the price we're going to suggest
  +asciimath:[`f(surface area, nb bedrooms, etc) = suggested price`]
  +
  +Of course, we're looking for the best function that outputs a suggested price
  +reflecting the reality of the market in the best possible way so we don't upset
  +our users by overpricing (the apartment will never be filled) or underpricing
  +(the people listing it will feel ripped off or the people looking for
  +accomodation will assume hidden defects).
  +
  +Keep in mind that we will build this function by looking through the history
  +of listings. There are more than 2 million active listings at any time on
  +AirBnB, we need to take into account the evolution of the price the listings
  +went through and all listings which are not active anymore. Moreover, we could
  +imagine that AirBnB compiles a thorough list of an apartment's characteristics
  +(also called features in a machine learning context). You can guess that this
  +amounts to a huge quantity of data. As a result, you won't be able to resort to
  +a non-distributed machine learning library like a regression library in R or
  +scikit-learn in Python because the data will not fit on your machine's RAM.
  +
  +Fortunately, Spark ML will solve all our issues since it's distributed so it
  +will be able to process all our data, as we mentioned earlier, there are quite a
  +few regression algorithms available out of the box and, at the same time, it
  +also promotes ease of use: the distributed nature of the algorithms is hidden
  +away from us.
  +
  +Spark ML is the primary API for machine learning in Spark, it supports a lot
  +of algorithms touching quite a few machine learning domains like classification,
  +clustering or recommender systems not just regression.
  +
  +The main programming abstraction in Spark ML is the _Pipeline_, an idea borrowed
  +from scikit-learn, a popular non-distributed Python machine learning library,
  +which lets you build and interact with your machine learning application as a
  +single unit consisting of several steps. We'll expand on this idea later in the
  +chapter.
  +
  +As a side note, the machine learning library in Apache Spark is split
   in two. On the one hand, we have the historical _RDD_ -based API which
   resides in the `org.apache.spark.mllib` package which we'll usually refer to
   as Spark MLlib and, on the other hand, the new _DataFrame_ / _Dataset_ -based
  
positive.code: to as Spark ML. As you might have guessed, this book will exclusively be about
  Spark ML.
  
  Spark ML is the primary API for machine learning in Spark, it supports a lot
  of algorithms touching quite a few machine learning domains like classification,
  regression, clustering or recommender systems.
  
  The main programming abstraction in Spark ML is the _Pipeline_, an idea borrowed
  from scikit-learn, which lets you build and interact with your machine learning
  application as a single unit consisting of several steps. We'll expand on this
  idea later in the chapter.
  
  Note that we'll use interchangeably _DataFrame_ and _Dataset_ throughout the
  book since, at the time of writing, the Spark ML API accepts _Datasets_ as
  input but usually returns _DataFrames_ plus there are ways to go from a
  _DataFrame_ to a _Dataset_ and vice versa.
  
  Benefits of Spark ML over Spark MLlib
  -------------------------------------
  
  If you've already used Spark MLlib you might wonder why you should switch to
  Spark ML or, if you've never used Spark or Spark for a machine learning task
  
negative.code: As with _Transformer_, _Estimators_ are also extendable if you want to try
  implementing an algorithm that is not already part of the library.
  
  === Pipeline
  
  A _Pipeline_ is the central component to Spark ML, it's an ordered sequence of
  _Estimator_ and _Transformer_ called stages. The idea is to chain algorithms and
  