title: What is a task in Spark? How does the Spark worker execute the jar file?
body: <p>After reading some document on <a href="http://spark.apache.org/docs/0.8.0/cluster-overview.html">http://spark.apache.org/docs/0.8.0/cluster-overview.html</a>, I got some question that I want to clarify.</p>
  
  <p>Take this example from Spark:</p>
  
  <pre><code>JavaSparkContext spark = new JavaSparkContext(
    new SparkConf().setJars("...").setSparkHome....);
  JavaRDD&lt;String&gt; file = spark.textFile("hdfs://...");
  
  // step1
  JavaRDD&lt;String&gt; words =
    file.flatMap(new FlatMapFunction&lt;String, String&gt;() {
      public Iterable&lt;String&gt; call(String s) {
        return Arrays.asList(s.split(" "));
      }
    });
  
  // step2
  JavaPairRDD&lt;String, Integer&gt; pairs =
    words.map(new PairFunction&lt;String, String, Integer&gt;() {
      public Tuple2&lt;String, Integer&gt; call(String s) {
        return new Tuple2&lt;String, Integer&gt;(s, 1);
      }
    });
  
  // step3
  JavaPairRDD&lt;String, Integer&gt; counts =
    pairs.reduceByKey(new Function2&lt;Integer, Integer&gt;() {
      public Integer call(Integer a, Integer b) {
        return a + b;
      }
    });
  
  counts.saveAsTextFile("hdfs://...");
  </code></pre>
  
  <p>So let's say I have 3 nodes cluster, and node 1 running as master, and the above driver program has been properly jared (say application-test.jar). So now I'm running this code on the master node and I believe right after the <code>SparkContext</code> being created, the application-test.jar file will be copied to the worker nodes (and each worker will create a dir for that application).</p>
  
  <p>So now my question:
  Are step1, step2 and step3 in the example tasks that get sent over to the workers? If yes, then how does the worker execute that? Like <code>java -cp "application-test.jar" step1</code> and so on?</p>