title: Why DBSCAN clustering returns single cluster on Movie lens data set?
body: <h2>The Scenario:</h2>
  
  <p>I'm performing Clustering over Movie Lens Dataset, where I have this Dataset in 2 formats:</p>
  
  <p><strong>OLD FORMAT:</strong></p>
  
  <pre><code>uid iid rat
  941 1   5
  941 7   4
  941 15  4
  941 117 5
  941 124 5
  941 147 4
  941 181 5
  941 222 2
  941 257 4
  941 258 4
  941 273 3
  941 294 4
  </code></pre>
  
  <p><strong>NEW FORMAT:</strong></p>
  
  <pre><code>uid 1               2               3               4
  1   5               3               4               3
  2   4               3.6185548023    3.646073985     3.9238342172
  3   2.8978348799    2.6692556753    2.7693015618    2.8973463681
  4   4.3320762062    4.3407749532    4.3111995162    4.3411425423
  940 3.7996234581    3.4979386925    3.5707888503    2
  941 5               NaN             NaN             NaN
  942 4.5762594612    4.2752554573    4.2522440019    4.3761477591
  943 3.8252406362    5               3.3748860659    3.8487417604
  </code></pre>
  
  <p>over which I need to perform Clustering using KMeans, DBSCAN and HDBSCAN.
  With KMeans I'm able to set and get clusters. </p>
  
  <h2>The Problem</h2>
  
  <p>The Problem persists only with DBSCAN &amp; HDBSCAN that I'm unable to get enough amount of clusters (I do know we cannot set Clusters manually)</p>
  
  <h2>Techniques Tried:</h2>
  
  <ul>
  <li>Tried this with <strong>IRIS</strong> <a href="https://en.wikipedia.org/wiki/Iris_flower_data_set#Data_set" rel="noreferrer">data-set</a>, where I found <em>Species</em> wasn't included. Clearly that is in String and besides is to be predicted, and everything just <a href="http://www.dummies.com/programming/big-data/data-science/how-to-create-an-unsupervised-learning-model-with-dbscan/" rel="noreferrer">works fine</a> with that Dataset (Snippet 1)</li>
  <li>Tried with Movie Lens 100K <a href="https://grouplens.org/datasets/movielens/" rel="noreferrer">dataset</a> in OLD FORMAT (with and without UID) since I tried an Analogy that, UID == SPECIES and hence tried without it. (Snippet 2)</li>
  <li>Tried same with NEW FORMAT (with and without UID) yet the results ended up in same style.</li>
  </ul>
  
  <p><strong>Snippet 1:</strong></p>
  
  <pre><code>print "\n\n FOR IRIS DATA-SET:"
  from sklearn.datasets import load_iris
  
  iris = load_iris()
  dbscan = DBSCAN()
  
  d = pd.DataFrame(iris.data)
  dbscan.fit(d)
  print "Clusters", set(dbscan.labels_)
  </code></pre>
  
  <p><strong>Snippet 1 (Output):</strong></p>
  
  <pre><code>FOR IRIS DATA-SET:
  Clusters set([0, 1, -1])
  Out[30]: 
  array([ 0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,
          0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,
          0,  0,  0,  0,  0,  0,  0, -1,  0,  0,  0,  0,  0,  0,  0,  0,  1,
          1,  1,  1,  1,  1,  1, -1,  1,  1, -1,  1,  1,  1,  1,  1,  1,  1,
         -1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,
          1,  1, -1,  1,  1,  1,  1,  1, -1,  1,  1,  1,  1, -1,  1,  1,  1,
          1,  1,  1, -1, -1,  1, -1, -1,  1,  1,  1,  1,  1,  1,  1, -1, -1,
          1,  1,  1, -1,  1,  1,  1,  1,  1,  1,  1,  1, -1,  1,  1, -1, -1,
          1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1])
  </code></pre>
  
  <p><strong>Snippet 2:</strong></p>
  
  <pre><code>import pandas as pd
  from sklearn.cluster import DBSCAN
  
  data_set = pd.DataFrame
  
  ch = int(input("Extended Cluster Methods for:\n1. Main Matrix IBCF \n2. Main Matrix UBCF\nCh:"))
  if ch is 1:
      data_set = pd.read_csv("MainMatrix_IBCF.csv")
      data_set = data_set.iloc[:, 1:]
      data_set = data_set.dropna()
  elif ch is 2:
      data_set = pd.read_csv("MainMatrix_UBCF.csv")
      data_set = data_set.iloc[:, 1:]
      data_set = data_set.dropna()
  else:
      print "Enter Proper choice!"
  
  print "Starting with DBSCAN for Clustering on\n", data_set.info()
  
  db_cluster = DBSCAN()
  db_cluster.fit(data_set)
  print "Clusters assigned are:", set(db_cluster.labels_)
  </code></pre>
  
  <p><strong>Snippet 2 (Output):</strong></p>
  
  <pre><code>Extended Cluster Methods for:
  1. Main Matrix IBCF 
  2. Main Matrix UBCF
  Ch:&gt;? 1
  Starting with DBSCAN for Clustering on
  &lt;class 'pandas.core.frame.DataFrame'&gt;
  Int64Index: 942 entries, 0 to 942
  Columns: 1682 entries, 1 to 1682
  dtypes: float64(1682)
  memory usage: 12.1 MB
  None
  Clusters assigned are: set([-1])
  </code></pre>
  
  <p>As seen, it returns only 1 Cluster. I'd like to hear what am I doing wrong.</p>