title: Running Spark jobs on a YARN cluster with additional files
body: <p>I'm writing a simple spark application that uses some input RDD, sends it to an external script via pipe, and writes an output of that script to a file. Driver code looks like this:</p>
  
  <pre><code>val input = args(0)
  val scriptPath = args(1)
  val output = args(2)
  val sc = getSparkContext
  if (args.length == 4) {
    //Here I pass an additional argument which contains an absolute path to a script on my local machine, only for local testing
    sc.addFile(args(3))
  }
  
  sc.textFile(input).pipe(Seq("python2", SparkFiles.get(scriptPath))).saveAsTextFile(output)
  </code></pre>
  
  <p>When I run it on my local machine it works fine. But when I submit it to a YARN cluster via</p>
  
  <pre><code>spark-submit --master yarn --deploy-mode cluster --files /absolute/path/to/local/test.py --class somepackage.PythonLauncher path/to/driver.jar path/to/input/part-* test.py path/to/output` 
  </code></pre>
  
  <p>it fails with an exception.</p>
  
  <pre><code>Lost task 1.0 in stage 0.0 (TID 1, rwds2.1dmp.ru): java.lang.Exception: Subprocess exited with status 2
  </code></pre>
  
  <p>I've tried different variations of the pipe command. For instance, <code>.pipe("cat")</code> works fine, and behaves as expected, but <code>.pipe(Seq("cat", scriptPath))</code> also fails with error code 1, so it seems that spark can't figure out a path to the script on a cluster node.</p>
  
  <p>Any suggestions?</p>