Apache Spark is a cluster computing framework, currently one of the most actively developed in the open-source Big Data arena. It aims at being a general engine for large-scale data processing, supporting a number of platforms for cluster management (e.g. YARN or Mesos as well as Spark native) and a variety of distributed storage systems (e.g. HDFS or Amazon S3).
More interestingly, at least from a developer’s perspective, it supports a number of programming languages. Since the latest version 1.4 (June 2015), Spark supports R and Python 3 (to complement the previously available support for Java, Scala and Python 2).
This article is a brief introduction on how to use Spark on Python 3.
This folder contains several Spark commands (in $SPARK_HOME/bin) as well as examples of code (in $SPARK_HOME/examples/src/main/YOUR-LANGUAGE).
We can run Spark with Python in two ways: using the interactive shell, or submitting a standalone application.
SparkContext available as sc, HiveContext available as sqlContext.
The >>> prompt is the usual Python prompt, as effectively we are using a Python interactive shell.
SparkContext and HiveContext are Spark concepts that we’ll briefly explain below. The interactive shell is telling us that these two contexts have been initialised and are available as sc and sqlContext in this session. The shell is also telling us that we’re using Python 2.7!?
But I want to use Python 3!
Once you re-run the interactive shell, the Spark banner should be updated to reflect your version of Python 3.
Another core concept in Spark is the Resilient Distributed Dataset (RDD), an immutable distributed collection of objects. Each RDD is split into partitions, which might be processed on different nodes of a cluster.
RDDs can be loaded from external sources, e.g. from text files, or can be transformed into new RDDs.
Running the interactive shell can be useful for interactive analysis, but sometimes you need to launch a batch job, i.e. you need to submit a stand-alone application.
The application will take three parameters from the command line (via sys.argv), namely a file name and two search terms.
The sc variable contains the SparkContext, initialised as local context (we’re not using a cluster).
The data variable is the RDD, loaded from an external resource (the aforementioned file).
What follows the data import is a series of transformations and actions. The basic idea is to simply count how many lines contain the given search terms.
For this example, I’m using a data-set of tweets downloaded for a previous article, stored in data.json one tweet per line.
Notice the use of the backslash from the command-line, because we need to escape the # symbol: effectively the search terms are #ita and #eng.
Also notice that we don’t have information about repeated occurrences of the search terms, nor about partial matches (e.g. “#eng” will also match “#england”, etc.): this example just showcases the use of transformations and actions.
THANK YOU!!!!!!!!!!!!!!!!!! You saved me.
Can you link your data.json for download?