anchor.diff:  
   We are all confronted with large-scale machine learning in our daily lives,
   whether it's through buying something on Amazon, watching a TV show or a movie
  -on Netflix, searching for an image on Google, finding people to follow on
  -Twitter, scrolling through a Facebook news feed, finding a place to stay on
  +on Netflix, finding people to follow on Twitter, scrolling through a Facebook
  +news feed, checking emails on GMail, finding out our bank account has been
  +frozen because of alleged fraudulent behavior, finding a place to stay on
   vacation on AirBnB or hailing an Uber.
   
   Behind those actions, which seem so simple and entrenched in our daily lives,
   lies different machine learning domains such as recommender systems (previewing
  -the next movie you might want to watch), classification (displaying all the
  -images where a cow appears), regression (estimating the fare for your next ride
  -or the daily rate for the place you're renting for your next holidays), and so
  -on.
  +the next movie you might want to watch), classification (displaying only the
  +non-spam emails), regression (estimating the fare for your next ride or the
  +daily rate for the place you're renting for your next holidays), and so on.
   
   Apache Spark and its machine learning library Spark ML, is used by some of
   these companies to implement them. For them, as we'll see later in this chapter
  
positive.code: single unit consisting of several steps. We'll expand on this idea later in the
  chapter.
  
  As a side note, the machine learning library in Apache Spark is split
  in two. On the one hand, we have the historical _RDD_ -based API which
  resides in the `org.apache.spark.mllib` package which we'll usually refer to
  as Spark MLlib and, on the other hand, the new _DataFrame_ / _Dataset_ -based
  API which resides in the `org.apache.spark.ml` package which we'll usually refer
  to as Spark ML. As you might have guessed, this book will exclusively be about
  Spark ML.
  
  Note that we'll use interchangeably _DataFrame_ and _Dataset_ throughout the
  book since, at the time of writing, the Spark ML API accepts _Datasets_ as
  input but usually returns _DataFrames_ plus there are ways to go from a
  _DataFrame_ to a _Dataset_ and vice versa.
  
  == Benefits of Spark ML over Spark MLlib
  
  If you've already used Spark MLlib you might wonder why you should switch to
  Spark ML or, if you've never used Spark or Spark for a machine learning task
  you might be asking yourself why choose Spark ML over Spark MLlib. There are
  quite a few reasons for this which we'll detail here.
  
  First and foremost, at the time of writing, the _RDD_ -based API has entered
  maintenance mode. As a result, no new features will be added to this API, only
  bug fixes. This effectively means that if you want to benefit from the wonderful
  improvements the contributors to Apache Spark are writing right now you'll have
  to switch to the _DataFrame_ / _Dataset_ -based API. Also, maintenance mode
  usually means impending deprecation which is usually followed by complete
  removal. Consequently, if you want your machine learning application to be
  resilient to Spark version upgrades you might want to go with Spark ML.
  
  Moreover, as we'll find out thoughout the book, Spark ML comes bundled with a
  lot of nice utilities which aren't part of Spark MLlib and will ease the
  construction of machine learning applications especially in the areas of model
  tuning and feature engineering, notions we'll cover later in the book.
  
  Another important feature which isn't available in MLlib is the ability to
  save and load the machine learning models that you built. This is especially
  important when you want to use your model across Spark applications. One
  example would be a data scientist who wrote a prediction model using Spark ML
  in Python and a data engineer in charge of building a streaming application
  who uses the previously built model to make predictions. The data scientist
  will simply save the model built and the data engineer will load it and use it
  to make the wanted near-realtime predictions.
  
  An added benefit, compared to MLlib, is that since we're relying on the
  _DataFrame_ / _Dataset_ API we'll have the same performance across languages
  which was not the case before. The examples in this book are written in Scala,
  however, since the API is the same in Java and Python, I believe you won't have
  any trouble translating them in the language of your choice at no performance
  cost.
  
  == Core Spark concepts
  
  In this section, we'll dig a bit deeper into the core concepts we'll use
  throughout the book: _DataFrame_ and _Dataset_.
  
  Both _DataFrames_ and _Datasets_ can be built from a variety of sources such as
  JSON, Parquet, tables in a database accessible through JDBC, Hive tables or
  _RDDs_.
  
  === DataFrame
  
  .An example DataFrame with 3 columns and 3 rows
  [options="header"]
  |==================
  | Column 1 | Column 2 | Column 3
  | 11 | "value 2 col2" | 13.36
  | 12 | "value 1 col2" | 23.42
  | 13 | "value 1 col2" | 33.76
  |==================
  
  As discussed earlier, a _DataFrame_ can be seen as a weakly-typed, distributed
  SQL table. If you come from R or pandas, the notion of dataframe is conceptually
  equivalent but distributed.
  
  Inside a _DataFrame_, data is organized into rows and named columns. However,
  the type information available to Spark for a _DataFrame_ doesn't translate into
  the Scala type system, that is why they are a bit clunky to manipulate and type
  errors are only seen at runtime.
  
  To demonstrate this, we'll go through a quick example where we'll create a
  small _DataFrame_ and manipulate it. Don't worry if there are some parts you
  don't totally understand, this is just to show off the _DataFrame_ API and its
  pitfalls.
  
  Let us say that you have the following file `users.json` containing JSONs at
  hand:
  
  .JSON file containing two users described with their name and couple of public and private keys
  [source,scala]
  
negative.code: ds.filter(user => user.name > 1)
  ----
  
  Note that we'll use interchangeably DataFrame and Dataset throughout the
  book since, at the time of writing, the Spark ML API accepts Datasets as
  input but usually returns DataFrames plus there are ways to go from a
  DataFrame to a Dataset and vice versa.
  
  == Core Spark ML concepts
  
  The following overview of the concepts specific to Spark ML provides insight on
  how the API is defined and how to use it.
  
  === Transformer
  
  A _Transformer_, in Spark ML parlance, is a component that takes a DataFrame as
  input and returns another DataFrame having applied some kind of transformation.
  Its interface can be summed up to a `transform()` method.
  
  Fortunately, Transformers for typical use cases come bundled with Spark ML. For
  example, `OneHotEncoder` takes a DataFrame with a column containing a
  categorical feature and returns the same DataFrame with an added column
  containing binary vectors with at most one 1 per vector.
  
  If we take the following DataFrame as an example (table 1.5):
  
  .An example DataFrame before the use of the OneHotEncoder Transformer
  [options="header"]
  |==================
  | Column 1 | Column 2 | Column 3
  | 11 | value 2 col2 | 0
  | 12 | value 1 col2 | 1
  | 13 | value 1 col2 | 0
  |==================
  
  And we decide to use OneHotEncoder on `Column 3`, we'll obtain table 1.6:
  
  .The DataFrame after application of OneHotEncoder on `Column 3`
  [options="header"]
  |==================
  | Column 1 | Column 2 | Column 3 | Result of OneHotEncoder
  | 11 | value 2 col2 | 0 | [1.0, 0.0]
  | 12 | value 1 col2 | 1 | [0.0, 1.0]
  | 13 | value 1 col2 | 0 | [1.0, 0.0]
  |==================
  
  Notice that a column "Result of OneHotEncoder" was added and contains binary
  vectors: we have two categories (0 and 1) and an input value of 1 for the
  first row, the encoder yields the [1.0, 0.0] binary vector.
  
  Another example of a Transformer is a prediction model. It takes a DataFrame
  without predictions and produce the same one with a new column containing the
  predictions.
  
  You can also create your own Transformers which can be extremely practical
  if you see yourself coding the same transformations over and over across your
  machine learning applications.
  
  ==== Estimator
  
  Another important concept is an _Estimator_. An Estimator is something that
  learns a machine learning model from a DataFrame. It effectively produces a
  Transformer that will, in turn, add a column containing the predictions
  made by the model. Its interface can be summed up as a `fit()` method.
  
  As you will learn throughout the book, Spark ML covers a large spectrum of
  machine learning algorithms as Estimators, such as decision trees, logistic
  regression, linear regression, K-means, alternating least squares, and so on.
  
  As with Transformers, Estimators are extendable if you want to try
  implementing an algorithm that is not already part of the library. However, this
  will be not covered in this book.
  
  === Pipeline
  
  A _Pipeline_ is the central component of Spark ML, it's defined as an Estimator
  (through inheritance) containing an ordered sequence of Estimators and
  Transformers (called stages in this context).
  
  Calling `fit()` on a Pipeline will result in a pass that goes through each stage
  and calls the `fit()` method on it if it's an Estimator.
  
  This step produces a _PipelineModel_, a Transformer since it's the result of
  calling `fit()` on an Estimator, which contains only Transformers since the
  Estimators in the original Pipeline have been converted to Transformers.
  
  Calling `transform()` on a PipelineModel is going to trigger a call to every
  Transformer's `transform()` method.
  
  The whole process is described in figure 1.3 and 1.4.
  
  .Calling `fit()` on the Pipeline
  image::../images/pipeline_fit.png[]
  
  The top row in figure 1.3 shows an example of a Pipeline composed of one
  Transformer (_VectorAssembler_) represented as a solid line and two Estimators
  (_StringIndexer_ and _DecisionTreeClassifier_) in the dashed lines.
  
  VectorAssembler takes several columns and produces a single column containing a
  vector composed of the input columns. StringIndexer converts a column containing
  categories to a column containing indices corresponding to the categories.
  DecisionTreeClassifier is classification we'll expand on later which could be
  used, for example, to tell spam apart from non-spam emails.
  
  The bottom row in figure 1.3 shows the result of calling `fit()` on the
  Pipeline of the top row. You can see that the Estimators have been converted to
  Transformers: StringIndexer became _StringIndexerModel_ and
  DecisionTreeClassifier became _DecisionTreeClassificationModel_.
  
  .Calling `transform()` on the PipelineModel produced by calling `fit()` on the Pipeline
  image::../images/pipeline_transform.png[]
  
  The top row in figure 1.4 represents the PipelineModel produced when calling the
  `fit()` method on the Pipeline.
  
  The bottom row in figure 1.4 shows the intermediate DataFrames produced after
  the application of each Transformer in sequence. The final DataFrame will
  contain our predictions.
  
  Throughout the book, we'll use schemas so you can get the big picture of each
  machine learning application we'll build.
  
  == About the book
  
  This book is not an exhaustive tour of the API proposed in Spark ML. Rather,
  it is a guide that will get you from an idea for a machine learning application
  to a full-fledged application running on a Spark cluster. In the process, we'll
  learn to deal with real data, raw and unsanitized; prototype a machine learning
  application quickly; understand the most common machine learning algorithms and
  how they work in a distributed fashion; make your machine learning application
  production-ready through unit testing; and finally deploy it to the cloud.
  