anchor.diff:  - Accessibility
   - Great out-of-the-box integration
   
  -First and foremost is speed. Spark relies on memory to store its intermediate
  +First and foremost is speed. Let us say you have to sort 100 TB of data, as of
  +2014, Apache Spark psent thrice less time than Hadoop (1406 seconds compared
  +to 4328) to sort a dataset of this size (you can check the sorting competition
  +at http://sortbenchmark.org). Spark relies on memory to store its intermediate
   results instead of disk which considerably decreases processing time so you can
  -spend more time making improving your machine learning applcation instead for
  -waiting for it to finish.
  +spend more time improving your machine learning applcation instead of waiting
  +for it to finish.
   
   Next, which I think, is as important as Spark's speed, is its accessibility.
   A lot of effort has been put into reducing the barrier of entry to be
  
positive.code: single unit consisting of several steps. We'll expand on this idea later in the
  chapter.
  
  As a side note, the machine learning library in Apache Spark is split
  in two. On the one hand, we have the historical _RDD_ -based API which
  resides in the `org.apache.spark.mllib` package which we'll usually refer to
  as Spark MLlib and, on the other hand, the new _DataFrame_ / _Dataset_ -based
  API which resides in the `org.apache.spark.ml` package which we'll usually refer
  to as Spark ML. As you might have guessed, this book will exclusively be about
  Spark ML.
  
  Note that we'll use interchangeably _DataFrame_ and _Dataset_ throughout the
  book since, at the time of writing, the Spark ML API accepts _Datasets_ as
  input but usually returns _DataFrames_ plus there are ways to go from a
  _DataFrame_ to a _Dataset_ and vice versa.
  
  == Benefits of Spark ML over Spark MLlib
  
  If you've already used Spark MLlib you might wonder why you should switch to
  Spark ML or, if you've never used Spark or Spark for a machine learning task
  you might be asking yourself why choose Spark ML over Spark MLlib. There are
  quite a few reasons for this which we'll detail here.
  
  First and foremost, at the time of writing, the _RDD_ -based API has entered
  maintenance mode. As a result, no new features will be added to this API, only
  bug fixes. This effectively means that if you want to benefit from the wonderful
  improvements the contributors to Apache Spark are writing right now you'll have
  to switch to the _DataFrame_ / _Dataset_ -based API. Also, maintenance mode
  usually means impending deprecation which is usually followed by complete
  removal. Consequently, if you want your machine learning application to be
  resilient to Spark version upgrades you might want to go with Spark ML.
  
  Moreover, as we'll find out thoughout the book, Spark ML comes bundled with a
  lot of nice utilities which aren't part of Spark MLlib and will ease the
  construction of machine learning applications especially in the areas of model
  tuning and feature engineering, notions we'll cover later in the book.
  
  Another important feature which isn't available in MLlib is the ability to
  save and load the machine learning models that you built. This is especially
  important when you want to use your model across Spark applications. One
  example would be a data scientist who wrote a prediction model using Spark ML
  in Python and a data engineer in charge of building a streaming application
  who uses the previously built model to make predictions. The data scientist
  will simply save the model built and the data engineer will load it and use it
  to make the wanted near-realtime predictions.
  
  An added benefit, compared to MLlib, is that since we're relying on the
  _DataFrame_ / _Dataset_ API we'll have the same performance across languages
  which was not the case before. The examples in this book are written in Scala,
  however, since the API is the same in Java and Python, I believe you won't have
  any trouble translating them in the language of your choice at no performance
  cost.
  
  == Core Spark concepts
  
  In this section, we'll dig a bit deeper into the core concepts we'll use
  throughout the book: _DataFrame_ and _Dataset_.
  
  Both _DataFrames_ and _Datasets_ can be built from a variety of sources such as
  JSON, Parquet, tables in a database accessible through JDBC, Hive tables or
  _RDDs_.
  
  === DataFrame
  
  .An example DataFrame with 3 columns and 3 rows
  [options="header"]
  |==================
  | Column 1 | Column 2 | Column 3
  | 11 | "value 2 col2" | 13.36
  | 12 | "value 1 col2" | 23.42
  | 13 | "value 1 col2" | 33.76
  |==================
  
  As discussed earlier, a _DataFrame_ can be seen as a weakly-typed, distributed
  SQL table. If you come from R or pandas, the notion of dataframe is conceptually
  equivalent but distributed.
  
  Inside a _DataFrame_, data is organized into rows and named columns. However,
  the type information available to Spark for a _DataFrame_ doesn't translate into
  the Scala type system, that is why they are a bit clunky to manipulate and type
  errors are only seen at runtime.
  
  To demonstrate this, we'll go through a quick example where we'll create a
  small _DataFrame_ and manipulate it. Don't worry if there are some parts you
  don't totally understand, this is just to show off the _DataFrame_ API and its
  pitfalls.
  
  Let us say that you have the following file `users.json` containing JSONs at
  hand:
  
  .JSON file containing two users described with their name and couple of public and private keys
  [source,scala]
  
negative.code: loop (REPL) which lets you experiment quickly and make quick and
  dirty prototypes. This is particularly important if you're in a "fail fast fail
  often" mindset.
  
  .REPL
  ****
  A read-eval-print loop can be seen as an interactive environment in which a user
  inputs expressions instead of whole classes. As its name implies, its use can be
  split into 3 steps:
  
  - It will read the user's input by parsing it
  - It will evaluate this input by, for example, calling a method on the
  appropriate object with the given parameters
  - It will print the result of the operation done during the eval step
  
  It usually facilitates debugging and trying things out quickly.
  
  You can start Spark's REPL by running `spark-shell` located in the `bin/` folder
  of yout Spark installation.
  
  Note also that Spark's REPL leverages Scala's.
  
  You can find out more at https://en.wikipedia.org/wiki/Read-eval-print_loop.
  ****
  
  Another aspect, which goes hand in hand with the REPL, is a simple and
  functional API that lets you express distributed data processing operations as
  you would write local collection transformations using the Scala API.
  Suppose you have to write a program counting words in a file, with Spark you
  can do so in 5 lines in a REPL which lets you get feedback about the code you
  write as you're writing it. With Hadoop MapReduce, you'd have to write a
  mapper, a reducer and a main in one (or more) Java classe, compile it into a JAR
  and finally run the application only to find out you get a runtime exception and
  you'll have to start the process all over again.
  
  Last but not least, Spark benefits from seamless integration with external data
  sources like Hive or Kafka and can be installed on a few different cluster
  managers like Hadoop YARN or Mesos which makes integrating Spark with your
  existing data infrastructure seamless. If you look at your own data
  infrastructure right now, there are certainly components that you'll be able to
  leverage when using Spark. For instance, let's say you're currently running
  your Hadoop MapReduce jobs over YARN which read data from Apache HBase. You'll
  be able to reuse YARN to run your Spark jobs and those Spark jobs will be able
  to read from HBase too.
  In addition to having great integration with other big data projects, a lot of
  functionalities have been baked directly into Spark such as stream processing or
  machine learning. This keeps you from having to combine multiple frameworks to
  solve your use case as was the case if you wanted to do machine learning using
  Apache Mahout on Hadoop. In this regard, Spark is self-sufficient for a lot of
  similar use cases.
  
  Now that we have a broad understanding of what Spark is and its strenghts, let's
  look under the hood. Figure 1.1 shows you an overview of a Spark cluster and its
  two fundamental components _Executor_ and _Driver_:
  
  .Overview of a Spark cluster with two nodes. The _SparkSession_ interacts with the cluster manager to obtain the required computation power. It also interacts with the executors to ditribute work.
  image::../images/cluster.png[]
  
  On the architecture side, each worker node runs its own Java Virtual Machine
  (JVM) process called an Executor in Spark parlance. Each Executor will run its
  own set of tasks,
  in a multithreaded manner, which have been scheduled by the _SparkSession_. The
  SparkSession is the entry point to everything Spark-related and is part of a
  driver program, another JVM, that can be run on a worker node or on your own
  computer. The SparkSession is also responsible for interacting with the cluster
  manager which is responsible for allocating resources to your Spark application.
  Note that each JVM in figure 1.1 is denoted by dashed lines.
  
  As opposed to other frameworks, when using Spark you can benefit from different
  subprojects which are integral components of Spark. As shown in Figure 1.2,
  there are 5 components to Spark:
  
  - Spark Core
  - Spark SQL
  - Spark Streaming
  - MLlib
  - GraphX
  
  .The different Spark subprojects: graph processing, stream processing, machine learning and the supported cluster managers
  image::../images/stack.png[]
  