title: How to deal with executor memory and driver memory in Spark?
body: <p>I am confused about dealing with executor memory and driver memory in Spark.</p>
  
  <p>My environment settings are as below:</p>
  
  <ul>
  <li>Memory 128 G, 16 CPU for 9 VM</li>
  <li>Centos</li>
  <li>Hadoop 2.5.0-cdh5.2.0</li>
  <li>Spark 1.1.0</li>
  </ul>
  
  <p>Input data information:</p>
  
  <ul>
  <li>3.5 GB data file from HDFS</li>
  </ul>
  
  <p>For simple development, I executed my Python code in standalone cluster mode (8 workers, 20 cores, 45.3 G memory) with <code>spark-submit</code>. Now I would like to set executor memory or driver memory for performance tuning.</p>
  
  <p>From the <a href="https://spark.apache.org/docs/1.1.0/configuration.html#environment-variables" rel="noreferrer">Spark documentation</a>, the definition for executor memory is</p>
  
  <blockquote>
    <p>Amount of memory to use per executor process, in the same format as JVM memory strings (e.g. 512m, 2g).</p>
  </blockquote>
  
  <p>How about driver memory?</p>