title: Spark OutOfMemory while repartition
body: <p>I struggle with an OutOfMemory Exception in Spark which is thrown while doing repartition. The program is processing the following steps:</p>
  
  <pre><code>JavaRDD&lt;A&gt; data = sc.objectFile(this.inSource);
  JavaPairRDD&lt;String, A&gt; dataWithKey = data.mapToPair(d -&gt; new Tuple2&lt;&gt;(d.getUjid(), d));
  JavaPairRDD&lt;ADesc, AStats&gt; dataInformation = dataWithKey.groupByKey()
              .flatMapToPair(v -&gt; getDataInformation(v._2()));
  dataInformation.groupByKey().repartition(PARTITIONS).map(v -&gt; merge(v._1(), v._2()));
  </code></pre>
  
  <p>getDataInformation maps a group of datapoints with the same id to several new datapoints:</p>
  
  <pre><code>Iterable&lt;Tuple2&lt;ADesc, AStats&gt;&gt; getDataInformation(Iterable&lt;A&gt; aIterator)
  </code></pre>
  
  <p>E.g.:
  (ID1, Data1), (ID1,Data2), (ID1,Data3) -> (Data_Description_1, Stats1), (Data_Description_2, Stats2)</p>
  
  <p>Information:</p>
  
  <ul>
  <li>A is a datastructure containing some information. It is a quite basic structure.</li>
  <li>Each datapoint A as an ID and several datapoints share a common ID. Therefore we map each datapoint to a tuple (ID, A)</li>
  <li>We group the datapoints by ID and extract several new datapoints with getDataInformation.</li>
  <li>Afterwards we want to group all statistics for the same data descriptions and merge them. </li>
  </ul>
  
  <p>While merging we get an OutOfMemory. Therefore we insert a repartition and run out of memory as well. All stages including flatMapToPair work correctly. We tried different values for PARTITIONS until we were up to 5000 tasks whereby the most tasks have very little work to do, while some have to progress a few MB and 3 tasks (independent from the number of partitions) always run out of memory. My question is why spark shuffels the data very unbalanced and is running out of memory while doing a repartition?</p>
  
  <p><a href="https://i.stack.imgur.com/pOMn4.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/pOMn4.png" alt="Bug"></a></p>