Is Spark's KMeans unable to handle bigdata?
Question: KMeans has several parameters for its
[training](http://spark.apache.org/docs/latest/api/python/pyspark.mllib.html?highlight=kmeans#pyspark.mllib.clustering.KMeans.train),
with initialization mode is defaulted to kmeans||. The problem is that it
marches quickly (less than 10min) to the first 13 stages, but then **hangs
completely** , without yielding an error!

**Minimal Example** which reproduces the issue (it will succeed if I use 1000
points or random initialization):

    
    
    from pyspark.context import SparkContext
    
    from pyspark.mllib.clustering import KMeans
    from pyspark.mllib.random import RandomRDDs
    
    
    if __name__ == "__main__":
        sc = SparkContext(appName='kmeansMinimalExample')
    
        # same with 10000 points
        data = RandomRDDs.uniformVectorRDD(sc, 10000000, 64)
        C = KMeans.train(data, 8192,  maxIterations=10)    
    
        sc.stop()
    

The job does nothing (it doesn't succeed, fail or progress..), as shown below.
There are no active/failed tasks in the Executors tab. Stdout and Stderr Logs
don't have anything particularly interesting:

[![enter image description
here](http://i.stack.imgur.com/aggpL.png)](http://i.stack.imgur.com/aggpL.png)

If I use `k=81`, instead of 8192, it will succeed:

[![enter image description
here](http://i.stack.imgur.com/zBKqG.png)](http://i.stack.imgur.com/zBKqG.png)

Notice that the two calls of `takeSample()`, [should not be an
issue](http://stackoverflow.com/questions/38986395/sparkkmeans-calls-
takesample-twice), since there were called twice in the random initialization
case.

So, what is happening? Is Spark's Kmeans **unable to scale**? Does anybody
know? Can you reproduce?

* * *

If it was a memory issue, [I would get warnings and errors, as I had been
before](https://gsamaras.wordpress.com/code/memoryoverhead-issue-in-spark/).

Note: placeybordeaux's comments are based on the execution of the job in
_client mode_ , where the driver's configurations are invalidated, causing the
exit code 143 and such (see edit history), not in cluster mode, where there is
_no error reported at all_ , the application _just hangs_.

* * *

From zero323: [Why is Spark Mllib KMeans algorithm extremely
slow?](http://stackoverflow.com/questions/35512139/why-is-spark-mllib-kmeans-
algorithm-extremely-slow) is related, but I think he witnesses some progress,
while mine hangs, I did leave a comment...

[![enter image description
here](http://i.stack.imgur.com/osDrR.png)](http://i.stack.imgur.com/osDrR.png)


Answer: I think the 'hanging' is because your executors keep dying. As I mentioned in
a side conversation, this code runs fine for me, locally and on a cluster, in
Pyspark and Scala. However, it takes a lot longer than it should. It is almost
all time spent in k-means|| initialization.

I opened <https://issues.apache.org/jira/browse/SPARK-17389> to track two main
improvements, one of which you can use now. Edit: really, see also
<https://issues.apache.org/jira/browse/SPARK-11560>

First, there are some code optimizations that would speed up the init by about
13%.

However most of the issue is that it default to 5 steps of k-means|| init,
when it seems that 2 is almost always just as good. You can set initialization
steps to 2 to see a speedup, especially in the stage that's hanging now.

In my (smaller) test on my laptop, init time went from 5:54 to 1:41 with both
changes, mostly due to setting init steps.

