title: Configuring Executor and Driver memory in Spark-on-Yarn
body: <p>I am confused with configuring executor and driver memory in Spark-1.5.2.</p>
  
  <p>My environment settings are as below:</p>
  
  <pre><code>3 Node MAPR Cluster - Each Node: Memory 256G, 16 CPU 
  Hadoop 2.7.0 
  Spark 1.5.2 - Spark-on-Yarn
  </code></pre>
  
  <p>Input data information:</p>
  
  <p>460 GB Parquet format table from Hive
  I'm using spark-sql for querying the hive context with spark-on-yarn,but it's lot slower than the Hive, and am not sure with the right memory configurations for Spark,</p>
  
  <p>These are my config's,</p>
  
  <pre><code>    export SPARK_DAEMON_MEMORY=1g
      export SPARK_WORKER_MEMORY=88g
  
      spark.executor.memory              2g
      spark.logConf                      true
      spark.eventLog.dir                 maprfs:///apps/spark
      spark.eventLog.enabled             true
      spark.serializer                   org.apache.spark.serializer.KryoSerializer
      spark.driver.memory                5g
      spark.kryoserializer.buffer.max    1024m
  </code></pre>
  
  <p>How to avoid Spark java.lang.OutOfMemoryError: Java heap space exceptions and
  GC overhead limit exceeded exceptions!! ???</p>
  
  <p>Really appreciate your assistance in this!</p>