anchor.diff:  to as Spark ML. As you might have guessed, this book will exclusively be about
   Spark ML.
   
  -Spark ML is the primary API for machine learning in Spark, it supports a lot
  -of algorithms touching quite a few machine learning domains like classification,
  -regression, clustering or recommender systems.
  -
  -The main programming abstraction in Spark ML is the _Pipeline_, an idea borrowed
  -from scikit-learn, which lets you build and interact with your machine learning
  -application as a single unit consisting of several steps. We'll expand on this
  -idea later in the chapter.
  -
   Note that we'll use interchangeably _DataFrame_ and _Dataset_ throughout the
   book since, at the time of writing, the Spark ML API accepts _Datasets_ as
   input but usually returns _DataFrames_ plus there are ways to go from a
   _DataFrame_ to a _Dataset_ and vice versa.
   
  -Benefits of Spark ML over Spark MLlib
  --------------------------------------
  +== Benefits of Spark ML over Spark MLlib
   
   If you've already used Spark MLlib you might wonder why you should switch to
   Spark ML or, if you've never used Spark or Spark for a machine learning task
  
positive.code: You can find the paper presenting Catalyst at:
  http://people.csail.mit.edu/matei/papers/2015/sigmod_spark_sql.pdf
  
  Spark Streaming
  ~~~~~~~~~~~~~~~
  
  Spark Streaming handles the processing of data streams which can come from many
  message queue like systems like Apache Kafka, Kinesis or Flume.
  
negative.code: We'll give a more thorough presentation of the _DataFrame_ -based API later in
  this chapter.
  
  === Cluster managers
  
  As mentioned earlier, Apache Spark is a distributed data processing framework.
  To scale to thousands of nodes, it relies on a cluster manager. A cluster
  manager will try to satisfy Spark's requests for computing power based on which
  machines are available in the cluster.
  
  Three cluster managers are supported by Spark: Hadoop YARN, Apache Mesos and
  Standalone (which comes bundled with Spark). +
   +
   +
  There are two main advantages to having those subprojects being part of Spark:
  you don't have to deal with version compatibility and new features and
  performance improvements propagate almost instantaneously to the other parts of
  Spark.
  
  Another benefit is that all those components can be used simultaneously. For
  example, you can query in SQL or make predictions using a machine learning model
  in a streaming application.
  
  == What is Spark ML?
  
  Suppose you're working for AirBnB and in charge of implementing the suggestions
  for the daily rate you see when you want to list your apartment. How would you
  go about it? One way to do it would be to build a regression model (a notion
  we'll come back to) which can be summed to the construction of a function with
  _x_ parameters corresponding to the characteristics of the apartment such as
  its surface area, its location, the number of bedrooms, etc which outputs a
  price:
  
  .The function we want to obtain taking all the apartment's characteristics as parameters and outputting the price we're going to suggest
  asciimath:[`f(surface area, nb bedrooms, etc) = suggested price`]
  
  Of course, we're looking for the best function that outputs a suggested price
  reflecting the reality of the market in the best possible way so we don't upset
  our users by overpricing (the apartment will never be filled) or underpricing
  (the people listing it will feel ripped off or the people looking for
  accomodation will assume hidden defects).
  
  Keep in mind that we will build this function by looking through the history
  of listings. There are more than 2 million active listings at any time on
  AirBnB, we need to take into account the evolution of the price the listings
  went through and all listings which are not active anymore. Moreover, we could
  imagine that AirBnB compiles a thorough list of an apartment's characteristics
  (also called features in a machine learning context). You can guess that this
  amounts to a huge quantity of data. As a result, you won't be able to resort to
  a non-distributed machine learning library like a regression library in R or
  scikit-learn in Python because the data will not fit on your machine's RAM.
  
  Fortunately, Spark ML will solve all our issues since it's distributed so it
  will be able to process all our data, as we mentioned earlier, there are quite a
  few regression algorithms available out of the box and, at the same time, it
  also promotes ease of use: the distributed nature of the algorithms is hidden
  away from us.
  
  Spark ML is the primary API for machine learning in Spark, it supports a lot
  of algorithms touching quite a few machine learning domains like classification,
  clustering or recommender systems not just regression.
  
  The main programming abstraction in Spark ML is the _Pipeline_, an idea borrowed
  from scikit-learn, a popular non-distributed Python machine learning library,
  which lets you build and interact with your machine learning application as a
  single unit consisting of several steps. We'll expand on this idea later in the
  chapter.
  
  As a side note, the machine learning library in Apache Spark is split
  in two. On the one hand, we have the historical _RDD_ -based API which
  resides in the `org.apache.spark.mllib` package which we'll usually refer to
  as Spark MLlib and, on the other hand, the new _DataFrame_ / _Dataset_ -based
  