anchor.diff:  
   This chapter covers:
   
  +- What is machine learning?
   - What is Apache Spark?
   - What is Spark ML?
   - The benefits of Spark ML over MLlib
  
positive.code: ds.filter(user => user.name > 1)
  ----
  
  == Core Spark ML concepts
  
  Next up, we'll give an overview of the concepts specific to Spark ML in order
  to give you insights on how the API is defined and how to use it.
  
  === Transformer
  
  A _Transformer_, in Spark ML parlance, is a component that will take a
  _DataFrame_ as input and return another _DataFrame_ having applied some kind
  of transformations. Its interface can be summed up to a `transform()` method.
  
  Fortunately, _Transformers_ covering typical use cases come with Spark ML. For
  example, `StringIndexer` takes a _DataFrame_ with a column containing a
  categorical feature and returns the same _DataFrame_ with an added column
  containing an index corresponding to the category based on its frequency.
  
  If we go back to our example _DataFrame_ from earlier
  
  .An example _DataFrame_ with no columns indexed
  [options="header"]
  |==================
  | Column 1 | Column 2 | Column 3
  | 11 | "value 2 col2" | 13.36
  | 12 | "value 1 col2" | 23.42
  | 13 | "value 1 col2" | 33.76
  |==================
  
  And we decide to index `Column 2`, we'll obtain:
  
  .The same _DataFrame_ with `Column 2` indexed
  [options="header"]
  |==================
  | Column 1 | Column 2 | Column 3 | Column 2 indexed
  | 11 | "value 2 col2" | 13.36 | 1.0
  | 12 | "value 1 col2" | 23.42 | 0.0
  | 13 | "value 1 col2" | 33.76 | 0.0
  |==================
  
  Notice that a `Column 2 indexed` was added containing indices representing the
  initial values by decreasing frequency.
  
  Another example would be a prediction model, it takes a _DataFrame_ without
  predictions and produce the same one with a new column containing the
  predictions.
  
  You can also create your own _Transformers_ which can be extremely practical
  if you see yourself coding the same transformations over and over across your
  machine learning applications.
  
  ==== Estimator
  
  Another important concept is _Estimator_. An _Estimator_ is something that will
  learn a machine learning model from a _DataFrame_. It effectively produces a
  _Transformer_ which will, in turn, add a column containing the predictions
  made by the model. Its interface can be summed up as a `fit()` method.
  
  As we will see throughout the book, Spark ML covers quite a large spectrum of
  machine learning algorithms as _Estimators_ such as decision trees, logisitc
  regression, linear regression, K-means, alternating least squares, etc.
  
  As with _Transformer_, _Estimators_ are also extendable if you want to try
  implementing an algorithm that is not already part of the library.
  
  === Pipeline
  
  A _Pipeline_ is the central component to Spark ML, it's an ordered sequence of
  _Estimator_ and _Transformer_ called stages. The idea is to chain algorithms and
  obtain a model in the end. Because of that, a _Pipeline_ is also an _Estimator_.
  What will actually happen is that when calling the `fit()` method on the
  _Pipeline_ it will go through every one of its stages and call `transform()` or
  `fit()` on it depending on whether the stage is a _Transformer_ or an
  _Estimator_.
  
  Since a _Pipeline_ is an _Estimator_, once you call its `fit()` method you will
  obtain a _PipelineModel_ which is a _Transformer_ on which you can call
  `transform()` to obtain you predictions.
  
  .Calling `fit()` on the _Pipeline_
  image::../images/pipeline_fit.png[]
  
  The top row shows an example _Pipeline_ composed of two _Transformers_
  (`StringIndexer` and `VectorAssembler`) highlighted in blue and one _Estimator_
  (`DecisionTreeClassifier`) highlighted in red.
  
  As we've seen earlier, `StringIndexer` will convert a column containing
  categories to a column containing indices corresponding to the categories.
  `VectorAssembler` will take several columns and produce a single column
  containing a vector composed of the input columns. `DecisionTreeClassifier` is
  a machine learning model we'll expand on later.
  
  The bottom row represents the _DataFrames_ at every step and the produced
  _Transformer_ which is a `DecisionTreeClassificationModel`.
  
  .Calling `transform()` on the _PipelineModel_ produced by calling `fit()` on the _Pipeline_
  image::../images/pipeline_transform.png[]
  
  The top row represents the _PipelineModel_ produced when calling `fit()` on the
  _Pipeline_. Notice that every _Estimator_ has been replaced by a _Transformer_.
  Then, if we use _PipelineModel_ 's `transform()` method we'll get our
  predictions.
  
  Throughout the book, we'll make those schemas so you can get the big picture of
  each machine learning application we'll build.
  
  == About the book
  
  This book won't be an exhaustive tour of the API proposed in Spark ML. Rather,
  it is a guide that will get you from an idea for a machine learning application
  to a full-fledged application running on a Spark cluster. In the process, we'll
  learn to deal with real data: raw and unsanatized, prototype machine learning
  application quickly, understand the most common machine learning algorithms and
  how they work in a distributed fashion, make your machine learning application
  production-ready through unit testing and finally deploy it to the cloud.
  
negative.code: 
  We are all confronted with large-scale machine learning in our daily lives,
  whether it's through buying something on Amazon, watching a TV show or a movie
  on Netflix, finding people to follow on Twitter, scrolling through a Facebook
  news feed, checking emails on GMail, finding out our bank account has been
  frozen because of alleged fraudulent behavior, finding a place to stay on
  vacation on AirBnB or hailing an Uber.
  
  Behind those actions, which seem so simple and entrenched in our daily lives,
  lies different machine learning domains such as recommender systems (previewing
  the next movie you might want to watch), classification (displaying only the
  non-spam emails), regression (estimating the fare for your next ride or the
  daily rate for the place you're renting for your next holidays), and so on.
  
  Apache Spark and its machine learning library Spark ML, is used by some of
  these companies to implement them. For them, as we'll see later in this chapter
  