title: Is Spark's KMeans unable to handle bigdata?
body: <p>KMeans has several parameters for its <a href="http://spark.apache.org/docs/latest/api/python/pyspark.mllib.html?highlight=kmeans#pyspark.mllib.clustering.KMeans.train" rel="noreferrer">training</a>, with initialization mode defaulted to kmeans||. The problem is that it marches quickly (less than 10min) to the first 13 stages, but then <strong>hangs completely</strong>, without yielding an error!</p>
  
  <p><strong>Minimal Example</strong> which reproduces the issue (it will succeed if I use 1000 points or random initialization):</p>
  
  <pre><code>from pyspark.context import SparkContext
  
  from pyspark.mllib.clustering import KMeans
  from pyspark.mllib.random import RandomRDDs
  
  
  if __name__ == "__main__":
      sc = SparkContext(appName='kmeansMinimalExample')
  
      # same with 10000 points
      data = RandomRDDs.uniformVectorRDD(sc, 10000000, 64)
      C = KMeans.train(data, 8192,  maxIterations=10)    
  
      sc.stop()
  </code></pre>
  
  <p>The job does nothing (it doesn't succeed, fail or progress..), as shown below. There are no active/failed tasks in the Executors tab. Stdout and Stderr Logs don't have anything particularly interesting:</p>
  
  <p><a href="https://i.stack.imgur.com/aggpL.png" rel="noreferrer"><img src="https://i.stack.imgur.com/aggpL.png" alt="enter image description here"></a></p>
  
  <p>If I use <code>k=81</code>, instead of 8192, it will succeed:</p>
  
  <p><a href="https://i.stack.imgur.com/zBKqG.png" rel="noreferrer"><img src="https://i.stack.imgur.com/zBKqG.png" alt="enter image description here"></a></p>
  
  <p>Notice that the two calls of <code>takeSample()</code>, <a href="https://stackoverflow.com/questions/38986395/sparkkmeans-calls-takesample-twice">should not be an issue</a>, since there were called twice in the random initialization case.</p>
  
  <p>So, what is happening? Is Spark's Kmeans <strong>unable to scale</strong>? Does anybody know? Can you reproduce?</p>
  
  <hr>
  
  <p>If it was a memory issue, <a href="https://gsamaras.wordpress.com/code/memoryoverhead-issue-in-spark/" rel="noreferrer">I would get warnings and errors, as I had been before</a>. </p>
  
  <p>Note: placeybordeaux's comments are based on the execution of the job in <em>client mode</em>, where the driver's configurations are invalidated, causing the exit code 143 and such (see edit history), not in cluster mode, where there is <em>no error reported at all</em>, the application <em>just hangs</em>.</p>
  
  <hr>
  
  <p>From zero323: <a href="https://stackoverflow.com/questions/35512139/why-is-spark-mllib-kmeans-algorithm-extremely-slow">Why is Spark Mllib KMeans algorithm extremely slow?</a> is related, but I think he witnesses some progress, while mine hangs, I did leave a comment...</p>
  
  <p><a href="https://i.stack.imgur.com/osDrR.png" rel="noreferrer"><img src="https://i.stack.imgur.com/osDrR.png" alt="enter image description here"></a></p>