title: Spark and YARN. How does the SparkPi example use the first argument as the master url?
body: <p>I'm just starting out learning Spark, and I'm trying to replicate the <code>SparkPi</code> example by copying the code into a new project and building a jar. The source for <code>SparkPi</code> is: <a href="https://github.com/apache/spark/blob/master/examples/src/main/scala/org/apache/spark/examples/SparkPi.scala" rel="nofollow">https://github.com/apache/spark/blob/master/examples/src/main/scala/org/apache/spark/examples/SparkPi.scala</a></p>
  
  <p>I have a working YARN cluster (running CDH 5.0.1), and I've uploaded the spark assembly jar and set it's hdfs location in <code>SPARK_JAR</code>.</p>
  
  <p>If I run this command, the example works:</p>
  
  <pre><code>$ SPARK_CLASSPATH=/usr/lib/spark/examples/lib/spark-examples_2.10-0.9.0-cdh5.0.1.jar /usr/lib/spark/bin/spark-class org.apache.spark.examples.SparkPi yarn-client 10
  </code></pre>
  
  <p>However, if I copy the source into a new project and build a jar and run the same command (with a different jar and classname), I get the following error:</p>
  
  <pre><code>$ SPARK_CLASSPATH=Spark.jar /usr/lib/spark/bin/spark-class spark.SparkPi yarn-client 10
  Exception in thread "main" org.apache.spark.SparkException: A master URL must be set in your configuration
      at org.apache.spark.SparkContext.&lt;init&gt;(SparkContext.scala:113)
      at spark.SparkPi$.main(SparkPi.scala:9)
      at spark.SparkPi.main(SparkPi.scala)
  </code></pre>
  
  <p>Somehow, the first argument isn't being passed as the master in the SparkContext in my version, but it works fine in the example. </p>
  
  <p>Looking at the SparkPi code, it seems to only expect a single numeric argument.</p>
  
  <p>So is there something about the Spark examples jar file which intercepts the first argument and somehow sets the <code>spark.master</code> property to be that?</p>