title: avoid partitions unbalancing Spark
body: <p>I have a <strong>performance problem</strong> with a code I'm revisioning, everytime will give an <code>OOM</code> while performing a count.
  I think I found the problem, basically after <code>keyBy</code> tranformation, being executed <code>aggregateByKey.</code>
  The problem lies to the fact that almost 98% of the RDD elements has the same key, so aggregationByKey, generate shuffle, <strong>putting nearly all records into the same partition</strong>, bottom line: just few executors works, and has to much memory pressure.</p>
  
  <p>This is the code:</p>
  
  <pre><code>val rddAnomaliesByProcess : RDD[AnomalyPO] = rddAnomalies
      .keyBy(po =&gt; po.getProcessCreator.name)
      .aggregateByKey(List[AnomalyPO]())((list,value) =&gt; value +: list,_++_)
      .map {case(name,list) =&gt;
        val groupByKeys = list.groupBy(po =&gt; (po.getPodId, po.getAnomalyCode, po.getAnomalyReason, po.getAnomalyDate, po.getMeasureUUID))
        val lastOfGroupByKeys = groupByKeys.map{po =&gt; (po._1, List(po._2.sortBy { po =&gt; po.getProcessDate.getMillis }.last))}
        lastOfGroupByKeys.flatMap(f =&gt; f._2)
      }
      .flatMap(f =&gt; f)
  
  log.info("not duplicated Anomalies: " + rddAnomaliesByProcess.count)
  </code></pre>
  
  <p>I would a way to perform operation in a more parallel way, allowing all executors to work nearly equally. How can I do that?</p>
  
  <p><strong>Should I have to use a custom partitioner?</strong></p>