title: Apache Spark - Two-sample Kolmogorov-Smirnov Test
body: <p>I have two sets of data (let's call them d1, d2) in Spark. I would like to perform a Two-sample Kolmogorov-Smirnov test, to test wether their underlying poplation distribution function is different. Can MLLib's <strong>Statistics.kolmogorovSmirnovTest</strong> do this?</p>
  
  <p>The documentation provides this example:</p>
  
  <pre><code>import org.apache.spark.mllib.stat.Statistics
  
  val data: RDD[Double] = ... // an RDD of sample data
  
  // perform a KS test using a cumulative distribution function of our making
  val myCDF: Double =&gt; Double = ...
  val testResult2 = Statistics.kolmogorovSmirnovTest(data, myCDF)
  </code></pre>
  
  <p>I tried computing the empirical cumulative distribution function of d2 (collecting it as Map) and comparing it with d1. </p>
  
  <pre><code>Statistics.kolmogorovSmirnovTest(d1, ecdf_map)
  </code></pre>
  
  <p>The test runs, but the results are wrong.</p>
  
  <p>Am I doing something wrong? Is it possible to do this? Any ideas?</p>
  
  <p>Thank you for the help!</p>