title: which is the best startegy to configure the spark application parameters?
body: <p>Im so confusing on which approach i should used to configure the spark application parameters.</p>
  
  <p>Let consider the following cluster configuration : 10 Nodes, 16 cores per Node and 64GB RAM per Node (example from <a href="https://spoddutur.github.io/spark-notes/distribution_of_executors_cores_and_memory_for_spark_application.html0" rel="nofollow noreferrer">https://spoddutur.github.io/spark-notes/distribution_of_executors_cores_and_memory_for_spark_application.html0</a></p>
  
  <ol>
  <li>Based on the recommendations mentioned above, Let’s assign <code>5 core per executors =&gt; --executor-cores = 5</code> (for good HDFS throughput)</li>
  <li>Leave <code>1 core per node</code> for Hadoop/Yarn daemons => Num cores available per node = 16-1 = 15. So, Total available of cores in cluster = 15 x 10 = 150</li>
  <li>Number of available executors = (total cores/num-cores-per-executor) = 150/5 = 30</li>
  <li>Leaving 1 executor for ApplicationManager => <code>--num-executors = 29</code></li>
  <li>Number of executors per node = 30/10 = 3</li>
  <li>Memory per executor = 64GB/3 = 21GB</li>
  <li>Counting off heap overhead = 7% of 21GB = 3GB. So, actual <code>--executor-memory = 21 - 3 = 18GB</code></li>
  </ol>
  
  <p>As a result the recommended configuration will be: <strong>29 executors, 18GB memory each and 5 cores each</strong></p>
  
  <p>Consider another example where the cluster confirguration is: 6 nodes, 25 cores nodes and 125 GB memory per node (example from <a href="https://researchcomputing.princeton.edu/computational-hardware/hadoop/spark-memory" rel="nofollow noreferrer">https://researchcomputing.princeton.edu/computational-hardware/hadoop/spark-memory</a>)</p>
  
  <p>the recommended configuration is;  <strong>--num-executors 30 --executor-cores 4 --executor-memory 24G.</strong></p>
  
  <p>Question: if we use the rules applied in example 1 and compute the recommended configuration for example 2 the result will be very different. In fact, if we <code>--executor-cores = 4</code>, and then we substract 1 core from each node 6 *24=144 thus we get 144/4= 26 executors. after leaving out 1 core for AM ==> <code>--num-executors = 25</code>. Now if we want to compute the number of executors per node = 25/6 = WHATTTT? Where is my error?</p>