title: Windows: Apache Spark History Server Config
body: <p>I wanted to use Spark's History Server to make use of the logging mechanisms of my Web UI, but I find some difficulty in running this code on my Windows machine. </p>
  
  <p>I have done the following:</p>
  
  <p>Set my spark-defaults.conf file to reflect </p>
  
  <pre><code>spark.eventLog.enabled=true
  spark.eventLog.dir=file://C:/spark-1.6.2-bin-hadoop2.6/logs
  spark.history.fs.logDirectory=file://C:/spark-1.6.2-bin-hadoop2.6/logs
  </code></pre>
  
  <p>My spark-env.sh to reflect:</p>
  
  <pre><code>SPARK_LOG_DIR    "file://C:/spark-1.6.2-bin-hadoop2.6/logs"
  SPARK_HISTORY_OPTS   "-Dspark.history.fs.logDirectory=file://C:/spark-1.6.2-bin-hadoop2.6/logs"
  </code></pre>
  
  <p>I am using Git-BASH to run the start-history-server.sh file, like this:</p>
  
  <pre><code>USERA@SYUHUH MINGW64 /c/spark-1.6.2-bin-hadoop2.6/sbin
  $ sh start-history-server.sh
  </code></pre>
  
  <p>And, I get this error:</p>
  
  <pre><code>USERA@SYUHUH MINGW64 /c/spark-1.6.2-bin-hadoop2.6/sbin
  $ sh start-history-server.sh
  C:\spark-1.6.2-bin-hadoop2.6/conf/spark-env.sh: line 69: SPARK_LOG_DIR: command not found
  C:\spark-1.6.2-bin-hadoop2.6/conf/spark-env.sh: line 70: SPARK_HISTORY_OPTS: command not found
  ps: unknown option -- o
  Try `ps --help' for more information.
  starting org.apache.spark.deploy.history.HistoryServer, logging to C:\spark-1.6.2-bin-hadoop2.6/logs/spark--org.apache.spark.deploy.history.HistoryServer-1-SGPF02M9ZB.out
  ps: unknown option -- o
  Try `ps --help' for more information.
  failed to launch org.apache.spark.deploy.history.HistoryServer:
    Spark Command: C:\Program Files (x86)\Java\jdk1.8.0_91\bin\java -cp C:\spark-1.6.2-bin-hadoop2.6/conf\;C:\spark-1.6.2-bin-hadoop2.6/lib/spark-assembly-1.6.2-hadoop2.6.0.jar;C:\spark-1.6.2-bin-hadoop2.6\lib\datanucleus-api-jdo-3.2.6.jar;C:\spark-1.6.2-bin-hadoop2.6\lib\datanucleus-core-3.2.10.jar;C:\spark-1.6.2-bin-hadoop2.6\lib\datanucleus-rdbms-3.2.9.jar -Xms1g -Xmx1g org.apache.spark.deploy.history.HistoryServer
    ========================================
  full log in C:\spark-1.6.2-bin-hadoop2.6/logs/spark--org.apache.spark.deploy.history.HistoryServer-1-SGPF02M9ZB.out
  </code></pre>
  
  <p>The full log from the output can be found below:</p>
  
  <pre><code>Spark Command: C:\Program Files (x86)\Java\jdk1.8.0_91\bin\java -cp C:\spark-1.6.2-bin-hadoop2.6/conf\;C:\spark-1.6.2-bin-hadoop2.6/lib/spark-assembly-1.6.2-hadoop2.6.0.jar;C:\spark-1.6.2-bin-hadoop2.6\lib\datanucleus-api-jdo-3.2.6.jar;C:\spark-1.6.2-bin-hadoop2.6\lib\datanucleus-core-3.2.10.jar;C:\spark-1.6.2-bin-hadoop2.6\lib\datanucleus-rdbms-3.2.9.jar -Xms1g -Xmx1g org.apache.spark.deploy.history.HistoryServer
  ========================================
  </code></pre>
  
  <p>I am running a sparkR script where I initialize my spark context and then call init(). </p>
  
  <p>Please advise whether I should be running the history server before I run my spark script?</p>
  
  <p>Pointers &amp; tips to proceed(with respect to logging) would be greatly appreciated.</p>