title: How to handle log in Spark Cluster Mode
body: <p>I am new to Spark. I am not able to find out how to handle logs in Spark Cluster Mode. I have added the below properties in Spark script.</p>
  
  <pre><code>spark.conf.set("yarn.log-aggregation-enable","true")
  spark.conf.set("yarn.nodemanager.log-dirs","HDFS_LOCATION")
  spark.conf.set("yarn.nodemanager.remote-app-log-dir","HDFS_LOCATION")
  spark.conf.set("spark.eventLog.enabled", "true")
  spark.conf.set("spark.eventLog.dir", "HDFS_LOCATION")
  spark.conf.set("spark.scheduler.mode", "FAIR")
  </code></pre>
  
  <p>And when runnig the spark-submit I am adding the below option:</p>
  
  <p><code>--driver-java-options "-Dlog4j.debug=true -Dlog4j.configuration=$LOCATION/log4j.properties"</code></p>
  
  <p>But I am getting the below exception :</p>
  
  <pre><code>Exception in thread "main" org.apache.spark.SparkException: Application
  </code></pre>
  
  <p>And I am unable to find any log in HDFS log location.</p>
  
  <p>Please help as I am stuck with the code.</p>