title: How to choose bucket length in Spark's Bucketed Random Projection?
body: <p>How should I optimally choose bucket length in Spark's LSH algorithm <a href="https://spark.apache.org/docs/2.1.0/ml-features.html#bucketed-random-projection-for-euclidean-distance" rel="noreferrer">Bucketed Random Projection</a>?</p>
  
  <p>It is this parameter set with .setBucketLength(), e.g. from the Spark documentation:</p>
  
  <pre><code>  val brp = new BucketedRandomProjectionLSH()
    .setBucketLength(2.0)
    .setNumHashTables(3)
    .setInputCol("keys")
    .setOutputCol("values")
  </code></pre>
  
  <p>Any theoretical references that support the choice of bucket length would be appreciated.</p>
  
  <p>The only suggestion I was able to find on the Web was <a href="https://vinta.ws/code/spark-ml-cookbook-scala.html" rel="noreferrer">this one</a>, which does not explain the choice of the chosen heuristics.</p>
  
  <blockquote>
    <p>To specify the value of bucketLength, if input vectors are normalized, 1-10 times of pow(numRecords, -1/inputDim) would be a reasonable value.</p>
  </blockquote>
  
  <p>Therefore I am not really sure why that heuristics is supposed to work in general.</p>