title: Mahout clustering - all text vectors in single cluster - why?
body: <p>I've run the following example:</p>
  
  <p><a href="https://github.com/technobium/mahout-clustering/blob/master/src/main/java/com/technobium/ClusteringDemo.java#L64" rel="nofollow noreferrer">https://github.com/technobium/mahout-clustering/blob/master/src/main/java/com/technobium/ClusteringDemo.java#L64</a></p>
  
  <pre><code>Document 1 -&gt; John saw a red car.
  Document 2 -&gt; Marta found a red bike.
  Document 3 -&gt; Don need a blue coat.
  Document 4 -&gt; Mike bought a blue boat.
  Document 5 -&gt; Albert wants a blue dish.
  Document 6 -&gt; Lara likes blue glasses.
  Document 7 -&gt; Donna, do you have red apples?
  Document 8 -&gt; Sonia needs blue books.
  Document 9 -&gt; I like blue eyes.
  Document 10 -&gt; Arleen has a red carpet.
  </code></pre>
  
  <p>and it works as expected with <code>EuclideanDistanceMeasure</code>. But I'm not sure why the text-intended distance measures (<code>TanimotoDistanceMeasure</code> and <code>CosineDistanceMeasure</code>) are giving me just a single cluster. </p>
  
  <p>Why is this? I'm not pretending I know anything about these 2 distance measures that are giving unsatisfactory results - but what might I need to change? There are a few too many numbers in there for me to understand the effect of each. I do have the book "Mahout in Action" though I have only read 2 chapters.</p>
  
  <h3>EuclideanDistanceMeasure (2 clusters - good)</h3>
  
  <pre><code> Clusters: 
           7 -&gt; wt: 1.0 distance: 4.4960791719810365  vec: Document 1 = [8:2.609, 21:2.609, 29:1.693, 30:2.609]
           7 -&gt; wt: 1.0 distance: 4.496079376645949  vec: Document 10 = [2:2.609, 9:2.609, 18:2.609, 29:1.693]
           7 -&gt; wt: 1.0 distance: 4.496079576525459  vec: Document 2 = [3:2.609, 16:2.609, 25:2.609, 29:1.693]
           9 -&gt; wt: 1.0 distance: 4.389955960700927  vec: Document 3 = [4:1.357, 10:2.609, 13:2.609, 27:2.609]
           9 -&gt; wt: 1.0 distance: 4.389956011306051  vec: Document 4 = [4:1.357, 5:2.609, 7:2.609, 26:2.609]
           9 -&gt; wt: 1.0 distance: 4.3899560687101395  vec: Document 5 = [0:2.609, 4:1.357, 11:2.609, 32:2.609]
           9 -&gt; wt: 1.0 distance: 4.389956137136399  vec: Document 6 = [4:1.357, 17:2.609, 22:2.609, 24:2.609]
           7 -&gt; wt: 1.0 distance: 5.577549042707083  vec: Document 7 = [1:2.609, 12:2.609, 14:2.609, 19:2.609, 29:1.693, 33:2.609]
           9 -&gt; wt: 1.0 distance: 4.389956708176695  vec: Document 8 = [4:1.357, 6:2.609, 28:2.609, 31:2.609]
           9 -&gt; wt: 1.0 distance: 4.389471924190491  vec: Document 9 = [4:1.357, 15:2.609, 20:2.609, 23:2.609]
  </code></pre>
  
  <p>produced by:</p>
  
  <pre><code>    CanopyDriver.run(new Path(vectorsFolder), new Path(canopyCentroids), new EuclideanDistanceMeasure(), 20, 5,
              true, 0, true);
  
      FuzzyKMeansDriver.run(new Path(vectorsFolder), new Path(canopyCentroids, "clusters-0-final"),
              new Path(clusterOutput), 0.01, 20, 2, true, true, 0, false);
  </code></pre>
  
  <h2>CosineDistanceMeasure (just 1 cluster - bad)</h2>
  
  <pre><code>Clusters: 
           0 -&gt; wt: 1.0 distance: 0.6362357041216559  vec: Document 1 = [8:2.609, 21:2.609, 29:1.693, 30:2.609]
           0 -&gt; wt: 1.0 distance: 0.6362357041216559  vec: Document 10 = [2:2.609, 9:2.609, 18:2.609, 29:1.693]
           0 -&gt; wt: 1.0 distance: 0.636235704121656  vec: Document 2 = [3:2.609, 16:2.609, 25:2.609, 29:1.693]
           0 -&gt; wt: 1.0 distance: 0.6328896123664868  vec: Document 3 = [4:1.357, 10:2.609, 13:2.609, 27:2.609]
           0 -&gt; wt: 1.0 distance: 0.6328896123664868  vec: Document 4 = [4:1.357, 5:2.609, 7:2.609, 26:2.609]
           0 -&gt; wt: 1.0 distance: 0.6328896123664868  vec: Document 5 = [0:2.609, 4:1.357, 11:2.609, 32:2.609]
           0 -&gt; wt: 1.0 distance: 0.6328896123664868  vec: Document 6 = [4:1.357, 17:2.609, 22:2.609, 24:2.609]
           0 -&gt; wt: 1.0 distance: 0.5876411474816594  vec: Document 7 = [1:2.609, 12:2.609, 14:2.609, 19:2.609, 29:1.693, 33:2.609]
           0 -&gt; wt: 1.0 distance: 0.6328896123664868  vec: Document 8 = [4:1.357, 6:2.609, 28:2.609, 31:2.609]
           0 -&gt; wt: 1.0 distance: 0.6328896123664868  vec: Document 9 = [4:1.357, 15:2.609, 20:2.609, 23:2.609]
  </code></pre>
  
  <p>produced by</p>
  
  <pre><code>    CanopyDriver.run(new Path(vectorsFolder), new Path(canopyCentroids), new CosineDistanceMeasure(), 20, 5,
              true, 0, true);
  
      FuzzyKMeansDriver.run(new Path(vectorsFolder), new Path(canopyCentroids, "clusters-0-final"),
              new Path(clusterOutput), 0.01, 20, 2, true, true, 0, false);
  </code></pre>
  
  <h2>TanimotoDistanceMeasure (just 1 cluster - bad)</h2>
  
  <pre><code> Clusters: 
           0 -&gt; wt: 1.0 distance: 0.8637279689324617  vec: Document 1 = [8:2.609, 21:2.609, 29:1.693, 30:2.609]
           0 -&gt; wt: 1.0 distance: 0.8637279689324617  vec: Document 10 = [2:2.609, 9:2.609, 18:2.609, 29:1.693]
           0 -&gt; wt: 1.0 distance: 0.8637279689324617  vec: Document 2 = [3:2.609, 16:2.609, 25:2.609, 29:1.693]
           0 -&gt; wt: 1.0 distance: 0.8596377086023765  vec: Document 3 = [4:1.357, 10:2.609, 13:2.609, 27:2.609]
           0 -&gt; wt: 1.0 distance: 0.8596377086023765  vec: Document 4 = [4:1.357, 5:2.609, 7:2.609, 26:2.609]
           0 -&gt; wt: 1.0 distance: 0.8596377086023765  vec: Document 5 = [0:2.609, 4:1.357, 11:2.609, 32:2.609]
           0 -&gt; wt: 1.0 distance: 0.8596377086023765  vec: Document 6 = [4:1.357, 17:2.609, 22:2.609, 24:2.609]
           0 -&gt; wt: 1.0 distance: 0.8723755210900389  vec: Document 7 = [1:2.609, 12:2.609, 14:2.609, 19:2.609, 29:1.693, 33:2.609]
           0 -&gt; wt: 1.0 distance: 0.8596377086023765  vec: Document 8 = [4:1.357, 6:2.609, 28:2.609, 31:2.609]
           0 -&gt; wt: 1.0 distance: 0.8596377086023765  vec: Document 9 = [4:1.357, 15:2.609, 20:2.609, 23:2.609]
  </code></pre>
  
  <p>produced via</p>
  
  <pre><code>    CanopyDriver.run(new Path(vectorsFolder), new Path(canopyCentroids), new TanimotoDistanceMeasure(), 20, 5,
              true, 0, true);
  
      FuzzyKMeansDriver.run(new Path(vectorsFolder), new Path(canopyCentroids, "clusters-0-final"),
              new Path(clusterOutput), 0.01, 20, 2, true, true, 0, false);
  </code></pre>