Spark RDD problems
Question: I am starting with spark and have never worked with Hadoop. I have 10 iMacs on
which I have installed Spark 1.6.1 with Hadoop 2.6. I downloaded the
precompiled version and just copied the extracted contents into
`/usr/local/spark/`. I did all the environment variables setup with
`SCALA_HOME`, changes to `PATH` and other spark conf. I am able to run both
`spark-shell` and `pyspark` (with anaconda's python).

I have setup the standalone cluster; all the nodes are showing up on my web
UI. Now, by using the python shell (ran on the cluster not locally) I followed
[this link's python interpreter word count
example](https://districtdatalabs.silvrback.com/getting-started-with-spark-in-
python).

This is the code I have used

    
    
    from operator import add
    
    def tokenize(text):
        return text.split()
    
    text = sc.textFile("Testing/shakespeare.txt")
    words = text.flatMap(tokenize)
    wc = words.map(lambda x: (x,1))
    counts = wc.reduceByKey(add)
    
    counts.saveAsTextFile("wc")
    

It is giving me error that the file `shakespeare.txt` was not found on a slave
nodes. Searching around I understood that if I am not using HDFS then the file
should be present on each slave node on the same path. Here is the stack trace
- [github
gist](https://gist.github.com/anonymous/6be913796d15493a913e8d99fc910d5b)

Now, I have a few questions-

  * Isn't RDD supposed to be distributed? That is, it should have distributed (when the action was run on RDD) the file on all the nodes instead of requiring me to distribute it.

  * I downloaded the spark with Hadoop 2.6, but any of the Hadoop commands are not available to make a HDFS. I extracted the Hadoop jar file found in the `spark/lib` hoping to find some executable but there was nothing. So, what Hadoop related files were provided in the spark download?

  * Lastly, how can I run a distributed application (spark-submit) or a distributed analysis (using pyspark) on the cluster? If I have to create a HDFS then what extra steps are required? Also, how can I create a HDFS here?


Answer: If you read the [Spark Programming
Guide](http://spark.apache.org/docs/latest/programming-guide.html#basics), you
will find the answer to your first question:

> To illustrate RDD basics, consider the simple program below:
>  
>  
>     val lines = sc.textFile("data.txt")
>     val lineLengths = lines.map(s => s.length)
>     val totalLength = lineLengths.reduce((a, b) => a + b)
>  
>
> The first line defines a base RDD from an external file. This dataset is not
> loaded in memory or otherwise acted on: lines is merely a pointer to the
> file. The second line defines lineLengths as the result of a map
> transformation. Again, lineLengths is not immediately computed, due to
> laziness. Finally, we run reduce, which is an action. At this point Spark
> breaks the computation into tasks to run on separate machines, and each
> machine runs both its part of the map and a local reduction, returning only
> its answer to the driver program.

Remember that transformations are executed on the Spark workers (see
[link](http://www.slideshare.net/databricks/strata-sj-everyday-im-shuffling-
tips-for-writing-better-spark-programs), slide n.21).

Regarding your second question, Spark contains only the libs, as you can see,
to use the Hadoop infrastructure. You need to setup the Hadoop cluster first
(Hdfs, etc etc), in order to use it (with the libs in Spark): have a look at
[Hadoop Cluster Setup](http://hadoop.apache.org/docs/current/hadoop-project-
dist/hadoop-common/ClusterSetup.html).

To answer your last question, I hope that the [official
documentation](http://spark.apache.org/docs/latest/cluster-overview.html)
helps, in particular [Spark
Standalone](http://spark.apache.org/docs/latest/spark-standalone.html).

