title: Set spark.local.dir to different drive
body: <p>I'm trying to setup standalone Spark on Windows 10. I would like to set <code>spark.local.dir</code> to <code>D:\spark-tmp\tmp</code>, as currently it appears to be using <code>C:\Users\&lt;me&gt;\AppData\Local\Temp</code>, which in my case is on an SSD drive which might not have enough space given the size of some datasets.</p>
  
  <p>So I changed the file <code>%SPARK_HOME%\conf\spark-defaults.conf</code> to the following, without success</p>
  
  <pre><code>spark.eventLog.enabled           true
  spark.eventLog.dir               file:/D:/spark-tmp/log
  spark.local.dir                  file:/D:/spark-tmp/tmp
  </code></pre>
  
  <p>I also tried to run <code>%HADOOP_HOME\bin\winutils.exe chmod -R 777 D:/spark-tmp</code>, but it didn't change anything.</p>
  
  <p>The error that I get is the following:</p>
  
  <pre><code>java.io.IOException: Failed to create a temp directory (under file:/D:/spark-tmp/tmp) after 10 attempts!
  </code></pre>
  
  <p>If I start the path with <code>file://D:/...</code> (note the double slash) nothing changes. If I remove the scheme at all, a different exception says that the scheme <code>D:</code> is not recognized.</p>
  
  <p>I also noticed this warning:</p>
  
  <pre><code>WARN  SparkConf:66 - In Spark 1.0 and later spark.local.dir will be overridden by the value set by the cluster manager (via SPARK_LOCAL_DIRS in mesos/standalone and LOCAL_DIRS in YARN).
  </code></pre>
  
  <p>So I tried to put the following line in <code>%SPARK_HOME%\conf\spark-env.sh</code>:</p>
  
  <pre><code>SPARK_LOCAL_DIRS=file:/D:/spark-tmp/tmp
  </code></pre>
  
  <p>If I put this line and comment the <code>spark.local.dir</code> line in the <code>.conf</code> file, Spark works perfectly, but the temporary files are still saved in my <code>AppData\Local\Temp</code> folder. So the <code>SPARK_LOCAL_DIRS</code> line is not read.</p>
  
  <p>What's strange is that, if I let it run, it actually puts logs in <code>D:/spark-tmp/log</code>, which means that it's not a problem of syntax or permissions.</p>