title: spark scalability: what am I doing wrong?
body: <p>I am processing data with spark and it works with a day worth of data (40G) but fails with <strong>OOM</strong> on a week worth of data:</p>
  
  <pre><code>import pyspark
  import datetime
  import operator
  sc = pyspark.SparkContext()
  sqc = pyspark.sql.SQLContext(sc)
  sc.union([sqc.parquetFile(hour.strftime('.....'))
            .map(lambda row:(row.id, row.foo))
            for hour in myrange(beg,end,datetime.timedelta(0,3600))]) \
    .reduceByKey(operator.add).saveAsTextFile("myoutput")
  </code></pre>
  
  <p>The number of different IDs is less than 10k.
  Each ID is a smallish <code>int</code>.
  The job fails because too many executors fail with OOM.
  When the job succeeds (on small inputs), <code>"myoutput"</code> is about 100k.</p>
  
  <ol>
  <li>what am I doing wrong?</li>
  <li>I tried replacing <code>saveAsTextFile</code> with <code>collect</code> (because I actually want to do some slicing and dicing in python before saving), there was no difference in behavior, same failure. is this to be expected?</li>
  <li>I used to have <code>reduce(lambda x,y: x.union(y), [sqc.parquetFile(...)...])</code> instead of <code>sc.union</code> - which is better? Does it make any difference?</li>
  </ol>
  
  <p>The cluster has <strong>25</strong> nodes with <strong>825GB</strong> RAM and <strong>224</strong> cores among them.</p>
  
  <p>Invocation is <code>spark-submit --master yarn --num-executors 50 --executor-memory 5G</code>.</p>
  
  <p>A single RDD has ~140 columns and covers one hour of data, so a week is a union of 168(=7*24) RDDs.</p>