title: Any tips for scaling Spark horizontally
body: <p>Does anybody have any tips when moving Spark execution from a few large nodes to many, smaller nodes?</p>
  
  <p>I am running a system with 4 executors, each executor has 24Gb of ram and 12 cores. If I try to scale that out to 12 executors, 4 cores each and 8 Gb of ram (Same total RAM, same total cores, just distributed differently) I run into out of memory errors:
  <code>Container killed by YARN for exceeding memory limits. 8.8 GB of 8.8 GB physical memory used.</code></p>
  
  <p>I have increased the number partitions by a factor of 3 to create more (yet smaller) partitions, but this didn't help.</p>
  
  <p>Does anybody have any tips &amp; tricks when trying to scale spark horizontally? </p>