title: Writing files to local system with Spark in Cluster mode
body: <p>I know this is a weird way of using Spark but I'm trying to save a dataframe to the local file system (not hdfs) using Spark even though I'm in <code>cluster mode</code>.  I know I can use <code>client mode</code> but <strong>I do</strong> want to run in <code>cluster mode</code> and don't care which node (out of 3) the application is going to run on as driver.
  The code below is the pseudo code of what I'm trying to do.</p>
  
  <pre><code>// create dataframe
  val df = Seq(Foo("John", "Doe"), Foo("Jane", "Doe")).toDF()
  // save it to the local file system using 'file://' because it defaults to hdfs://
  df.coalesce(1).rdd.saveAsTextFile(s"file://path/to/file")
  </code></pre>
  
  <p>And this is how I'm submitting the spark application.</p>
  
  <p><code>spark-submit --class sample.HBaseSparkRSample --master yarn-cluster hbase-spark-r-sample-assembly-1.0.jar</code></p>
  
  <p>This works fine if I'm in <code>local mode</code> but doesn't in <code>yarn-cluster mode</code>.</p>
  
  <p>For example, <code>java.io.IOException: Mkdirs failed to create file</code> occurs with the above code.</p>
  
  <p>I've changed the <code>df.coalesce(1)</code> part to <code>df.collect</code> and attempted to save a file using plain Scala but it ended up with a <code>Permission denied</code>.</p>
  
  <p>I've also tried:</p>
  
  <ul>
  <li><code>spark-submit</code> with <code>root</code> user</li>
  <li><code>chown</code>ed <code>yarn:yarn</code>, <code>yarn:hadoop</code>, <code>spark:spark</code></li>
  <li>gave <code>chmod 777</code> to related directories</li>
  </ul>
  
  <p>but no luck.</p>
  
  <p>I'm assuming this has to do something with <code>clusters</code>, <code>drivers and executors</code>, and the <code>user</code> who's trying to write to the local file system but am pretty much stuck in solving this problem by myself.</p>
  
  <p>I'm using:</p>
  
  <ul>
  <li>Spark: 1.6.0-cdh5.8.2</li>
  <li>Scala: 2.10.5</li>
  <li>Hadoop: 2.6.0-cdh5.8.2</li>
  </ul>
  
  <p>Any support is welcome and thanks in advance.</p>
  
  <p>Some articles I've tried:</p>
  
  <ul>
  <li>"Spark saveAsTextFile() results in Mkdirs failed to create for half of the directory" -> Tried changing users but nothing changed</li>
  <li>"Failed to save RDD as text file to local file system" -> <code>chmod</code> didn't help me</li>
  </ul>
  
  <h1>Edited (2016/11/25)</h1>
  
  <p>This is the Exception I get.</p>
  
  <pre><code>java.io.IOException: Mkdirs failed to create file:/home/foo/work/rhbase/r/input/input.csv/_temporary/0/_temporary/attempt_201611242024_0000_m_000000_0 (exists=false, cwd=file:/yarn/nm/usercache/foo/appcache/application_1478068613528_0143/container_e87_1478068613528_0143_01_000001)
      at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:449)
      at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:435)
      at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:920)
      at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:813)
      at org.apache.hadoop.mapred.TextOutputFormat.getRecordWriter(TextOutputFormat.java:135)
      at org.apache.spark.SparkHadoopWriter.open(SparkHadoopWriter.scala:91)
      at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$13.apply(PairRDDFunctions.scala:1193)
      at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$13.apply(PairRDDFunctions.scala:1185)
      at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
      at org.apache.spark.scheduler.Task.run(Task.scala:89)
      at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      at java.lang.Thread.run(Thread.java:745)
  16/11/24 20:24:12 WARN scheduler.TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, localhost): java.io.IOException: Mkdirs failed to create file:/home/foo/work/rhbase/r/input/input.csv/_temporary/0/_temporary/attempt_201611242024_0000_m_000000_0 (exists=false, cwd=file:/yarn/nm/usercache/foo/appcache/application_1478068613528_0143/container_e87_1478068613528_0143_01_000001)
      at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:449)
      at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:435)
      at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:920)
      at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:813)
      at org.apache.hadoop.mapred.TextOutputFormat.getRecordWriter(TextOutputFormat.java:135)
      at org.apache.spark.SparkHadoopWriter.open(SparkHadoopWriter.scala:91)
      at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$13.apply(PairRDDFunctions.scala:1193)
      at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$13.apply(PairRDDFunctions.scala:1185)
      at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
      at org.apache.spark.scheduler.Task.run(Task.scala:89)
      at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      at java.lang.Thread.run(Thread.java:745)
  </code></pre>