title: Apache Spark: The number of cores vs. the number of executors
body: <p>I'm trying to understand the relationship of the number of cores and the number of executors when running a Spark job on YARN.</p>
  
  <p>The test environment is as follows:</p>
  
  <ul>
  <li>Number of data nodes: 3</li>
  <li>Data node machine spec:
  
  <ul>
  <li>CPU: Core i7-4790 (# of cores: 4, # of threads: 8)</li>
  <li>RAM: 32GB (8GB x 4)</li>
  <li>HDD: 8TB (2TB x 4)</li>
  </ul></li>
  <li><p>Network: 1Gb</p></li>
  <li><p>Spark version: 1.0.0</p></li>
  <li><p>Hadoop version: 2.4.0 (Hortonworks HDP 2.1)</p></li>
  <li><p>Spark job flow: sc.textFile -> filter -> map -> filter -> mapToPair -> reduceByKey -> map -> saveAsTextFile</p></li>
  <li><p>Input data</p>
  
  <ul>
  <li>Type: single text file</li>
  <li>Size: 165GB</li>
  <li>Number of lines: 454,568,833</li>
  </ul></li>
  <li><p>Output</p>
  
  <ul>
  <li>Number of lines after second filter: 310,640,717</li>
  <li>Number of lines of the result file: 99,848,268</li>
  <li>Size of the result file: 41GB</li>
  </ul></li>
  </ul>
  
  <p>The job was run with following configurations:</p>
  
  <ol>
  <li><p><code>--master yarn-client --executor-memory 19G --executor-cores 7 --num-executors 3</code> (executors per data node, use as much as cores)</p></li>
  <li><p><code>--master yarn-client --executor-memory 19G --executor-cores 4 --num-executors 3</code> (# of cores reduced)</p></li>
  <li><p><code>--master yarn-client --executor-memory 4G --executor-cores 2 --num-executors 12</code> (less core, more executor)</p></li>
  </ol>
  
  <p>Elapsed times:</p>
  
  <ol>
  <li><p>50 min 15 sec</p></li>
  <li><p>55 min 48 sec</p></li>
  <li><p>31 min 23 sec</p></li>
  </ol>
  
  <p>To my surprise, (3) was much faster.<br>
  I thought that (1) would be faster, since there would be less inter-executor communication when shuffling.<br>
  Although # of cores of (1) is fewer than (3), #of cores is not the key factor since 2) did perform well.</p>
  
  <p>(Followings were added after pwilmot's answer.)</p>
  
  <p>For the information, the performance monitor screen capture is as follows:</p>
  
  <ul>
  <li>Ganglia data node summary for (1) - job started at 04:37.</li>
  </ul>
  
  <p><img src="https://i.stack.imgur.com/bVnyJ.png" alt="Ganglia data node summary for (1)"></p>
  
  <ul>
  <li>Ganglia data node summary for (3) - job started at 19:47. Please ignore the graph before that time.</li>
  </ul>
  
  <p><img src="https://i.stack.imgur.com/2K2Kp.png" alt="Ganglia data node summary for (3)"></p>
  
  <p>The graph roughly divides into 2 sections:</p>
  
  <ul>
  <li>First: from start to reduceByKey: CPU intensive, no network activity</li>
  <li>Second: after reduceByKey: CPU lowers, network I/O is done.</li>
  </ul>
  
  <p>As the graph shows, (1) can use as much CPU power as it was given. So, it might not be the problem of the number of the threads.</p>
  
  <p>How to explain this result?</p>