title: Implementing DBSCAN in distributed system
body: <p>I have a Big Data problem and I have very limited experience with parallel processing and Big data. I have 100s of millions of rows consisting of Latitude and Longitude data and several ID. For each ID I can have data ranging form 10000 -10 million.</p>
  
  <p>I am implementing the density based clustering algorithm (DBSCAN) to solve some business requirement. The clustering algorithm runns independently for each ID.</p>
  
  <p><strong>Current Implementation;</strong></p>
  
  <p>The current implementation is based out of Python code using sklearn Machine learning library, But it takes a day or more to perform (the clustering + other business logic) for appx 50 million datapoint.</p>
  
  <p>I can optimize the python code and reduce the time but I am looking for a sollution thats more feasible.</p>
  
  <p><strong>Availability</strong></p>
  
  <p>I have a spark cluster distributed accross appx 20 machines but pyspark has no implementation of DBSCAN. Upon some searching i could find some scala imlementation but they seem to be less reliable. The URL's from my search are. 
  <a href="https://github.com/irvingc/dbscan-on-spark" rel="noreferrer">https://github.com/irvingc/dbscan-on-spark</a> </p>
  
  <p><a href="https://stackoverflow.com/questions/36090906/dbscan-on-spark-which-implementation">DBSCAN on spark : which implementation</a> </p>
  
  <p>Since all my code is written in python I would like to stick with a solution thats more pythonic.</p>
  
  <p>Like I mentioned that the clustering algorithm runs independently for each devices, One way to reduce the time is to distribute the computation of each ID parallely to all 20 machines. So that I could atleast get 20x better performance. But I have no idea on how to achieve this. All I can think of is MapReduce.</p>
  
  <p>I am open to any sollution thats more robust. Any help would be greatly appreciated.</p>