title: Spark executor out of memory on join
body: <p>Hi I am using spark Mllib and doing approxSimilarityJoin between a 1M dataset and a 1k dataset.<br />
  When i do it I bradcast the 1k one.<br />
  What I see is that thew job stops going forward at the second-last task.<br />
  All the executors are dead but one which keeps running for very long time until it reaches Out of memory.<br />
  I checked ganglia and it shows memory keeping rising until it reaches the limit
  <a href="https://i.stack.imgur.com/gfhGg.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/gfhGg.png" alt="enter image description here" /></a></p>
  <p>and the disk space keeps going down until it finishes:<br />
  <a href="https://i.stack.imgur.com/vbEmG.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/vbEmG.png" alt="enter image description here" /></a><br />
  The action I called is a write, but it does the same with count.<br />
  Now I wonder: is it possible that all the partitions in the cluster converge to only one node and creating this bottleneck?</p>
  <p>Here is my code snippet:</p>
  <pre class="lang-scala prettyprint-override"><code>var dfW = cookesWb.withColumn(&quot;n&quot;, monotonically_increasing_id())
      var bunchDf = dfW.filter(col(&quot;n&quot;).geq(0) &amp;&amp; col(&quot;n&quot;).lt(1000000) )
      bunchDf.repartition(3000)
      model.
      approxSimilarityJoin(bunchDf,broadcast(cookesNextLimited),80,&quot;EuclideanDistance&quot;).
      withColumn(&quot;min_distance&quot;, min(col(&quot;EuclideanDistance&quot;)).over(Window.partitionBy(col(&quot;datasetA.uid&quot;)))
                    ).
          filter(col(&quot;EuclideanDistance&quot;) === col(&quot;min_distance&quot;)).
        select(col(&quot;datasetA.uid&quot;).alias(&quot;weboId&quot;),
          col(&quot;datasetB.nextploraId&quot;).alias(&quot;nextId&quot;),
          col(&quot;EuclideanDistance&quot;)).write.format(&quot;parquet&quot;).mode(&quot;overwrite&quot;).save(&quot;approxJoin.parquet&quot;)
  
  </code></pre>