title: Bizarre HDBScan clustering result for cosine-similarity matrix
body: <p>I'm trying to cluster similar messages within machine log files (where e.g. I can't ignore numbers). Debugging my code with a subset of messages which all have the same "<em>degree of similarity</em>" I came across a very strange finding: below a certain number of these messages HDBScan produces the expected result (which is all message belong to one or no cluster) but above a certain number of messages HDBScan suddenly starts finding different clusters which intuitively doesn't make sense to me. </p>
  
  <p>And even more strange: that limit where I start seeing multiple clusters is 18 messages for the '<em>generic</em>' algorithm within HDBScan but 61 when using '<em>best</em>'. Well, maybe 'best' selects '<em>generic</em>' above 60 messages, not sure how to verify that...</p>
  
  <p>I've tried various settings for <code>min_cluster_size</code> and <code>min_samples</code> and various distance metrics but the issue remains the same. Below please find some self-contained code to see if you can reproduce the issue. Just change <code>n_msg</code> to any <code>number &gt;= 18</code> (or <code>&gt;=61</code> when using '<em>best</em>') and you should get more than one cluster ID. The code also prints the cosine-similarity matrix so you can see how symmetrical this example is.</p>
  
  <pre class="lang-py prettyprint-override"><code>from sklearn.feature_extraction.text import CountVectorizer
  from sklearn.metrics.pairwise import cosine_similarity
  import hdbscan
  import pandas as pd
  
  # set number of dummy messages to be created
  n_msg = 17
  
  # create dummy messages with three identical and one variable term
  msgs = pd.DataFrame()
  for i in range(0, n_msg):
      msg = ['bli bla blub ' + str(i)]
      msgs = msgs.append(msg)
  msgs.columns = ['msg']
  
  # tokenizer to split at space only so numbers will not be ignored
  def space_tokenizer(msg):
     return msg.split()
  
  # Vectorize dummy messages
  TFvectorizer = CountVectorizer(tokenizer = space_tokenizer)
  msgs_vect = TFvectorizer.fit_transform(msgs.msg)
  
  # Compute cosine similarity between message vectors
  msgs_CosSim = pd.DataFrame(cosine_similarity(msgs_vect, msgs_vect))
  print(msgs_CosSim,'\n')
  
  # Cluster the cosine similarity results
  hdbs = hdbscan.HDBSCAN(min_cluster_size=2, min_samples=1, metric='euclidean', \
                         algorithm='generic').fit(msgs_CosSim)
  CosSim_clstr_ID = pd.DataFrame(hdbs.labels_)
  CosSim_clstr_ID.columns = ['msg_ID']
  print(CosSim_clstr_ID,'\n')
  
  # Check number of cluster IDs generated
  print('Number of cluster IDs:', len(CosSim_clstr_ID.msg_ID.unique()))
  </code></pre>
  
  <p>So again, with these dummy messages above I would expect the same result (all message belong to one cluster ID or the outlier cluster -1) independent from the number of messages but I start getting different clusters above a certain number of messages (depending on the algorithm choice within HDBScan).</p>
  
  <p>Any idea what's going on?</p>
  
  <p>Update: did some more research on this using <code>allow_single_cluster=True</code> only (!) and looped through all <code>algorithm</code> values as well as different message types (between one and four fixed terms plus the numbers) and here is the result where <code>'generic'</code>, whilst being the fastest, seems to be most likely to produce strange results randomly:
  <a href="https://i.stack.imgur.com/mro1m.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/mro1m.png" alt="enter image description here"></a></p>