DBSCAN clustering python - parallel run on multiple clustering tasks
Since your users are all independent, this clearly is an embarrassingly parallel problem. You want to run the same task (DBSCAN) millions of times. There are many ways to achieve this. You can probably use Spark (although I would consider using a Java based tool with it, such as ELKI - and you probably need to make sure you parallelize on the users, not within each user), MapReduce, or even Makefiles with locking, if you have a network file system with locking.
The key factor is how your data is organized. It makes a huge difference whether you can read in parallel for all workers, or route all your data through a master node (bad). You need to get the data efficiently to the workers, and need to store the clustering results.
I need to run DBSCAN clustering on about 14M users, each one has 1k data points. Each user is a different clustering case which is completely separate from other users. basically I have many small clustering tasks.
Running it on a single machine doesn't work for me, even when paralleling the tasks using python multiprocessing module, as IO and clustering take ages.
I thought about using Spark to manage a parallel run on a cluster, but decided it might not fit my case, since DBSCAN is not implemented in MLlib and the fact that I don't need to run each clustering task in parallel, but run each one separately. whenever I try to use anything outside of Spark native RDD or Dataframes it obviously has to collect all data to the driver node.
my question is weather there is a smarter solution to my problem than simply run many isolated processes on different nodes, when each one will select a subset of the users?
thanks