title: tf-idf document clustering with K-means in Apache Spark putting points into one cluster
body: <p>I am trying to do the classic job of clustering text documents by pre-processing, generating tf-idf matrix, and then applying K-means. However, testing this workflow on the classic 20NewsGroup dataset results in most documents being clustered into one cluster. (I have initially tried to cluster all documents from 6 of the 20 groups - so expecting to cluster into 6 clusters).</p>
  
  <p>I am implementing this in Apache Spark as my purpose is to utilise this technique on millions of documents. Here is the code written in Pyspark on Databricks:</p>
  
  <pre><code>#declare path to folder containing 6 of 20 news group categories
  path = "/mnt/%s/20news-bydate.tar/20new-bydate-train-lessFolders/*/*" % 
  MOUNT_NAME
  
  #read all the text files from the 6 folders. Each entity is an entire 
  document. 
  text_files = sc.wholeTextFiles(path).cache()
  
  #convert rdd to dataframe
  df = text_files.toDF(["filePath", "document"]).cache()
  
  from pyspark.ml.feature import HashingTF, IDF, Tokenizer, CountVectorizer 
  
  #tokenize the document text
  tokenizer = Tokenizer(inputCol="document", outputCol="tokens")
  tokenized = tokenizer.transform(df).cache()
  
  from pyspark.ml.feature import StopWordsRemover
  
  remover = StopWordsRemover(inputCol="tokens", 
  outputCol="stopWordsRemovedTokens")
  stopWordsRemoved_df = remover.transform(tokenized).cache()
  
  hashingTF = HashingTF (inputCol="stopWordsRemovedTokens", outputCol="rawFeatures", numFeatures=200000)
  tfVectors = hashingTF.transform(stopWordsRemoved_df).cache()    
  
  idf = IDF(inputCol="rawFeatures", outputCol="features", minDocFreq=5)
  idfModel = idf.fit(tfVectors)
  
  tfIdfVectors = idfModel.transform(tfVectors).cache()
  
  #note that I have also tried to use normalized data, but get the same result
  from pyspark.ml.feature import Normalizer
  from pyspark.ml.linalg import Vectors
  
  normalizer = Normalizer(inputCol="features", outputCol="normFeatures")
  l2NormData = normalizer.transform(tfIdfVectors)
  
  from pyspark.ml.clustering import KMeans
  
  # Trains a KMeans model.
  kmeans = KMeans().setK(6).setMaxIter(20)
  km_model = kmeans.fit(l2NormData)
  
  clustersTable = km_model.transform(l2NormData)
  </code></pre>
  
  <p><img src="https://i.stack.imgur.com/IEOTc.png" alt="output showing most documents get clustered into cluster 0"></p>
  
  <pre><code>ID number_of_documents_in_cluster
  0    3024
  3    5
  1    3
  5    2
  2    2
  4    1
  </code></pre>
  
  <p>As you can see most of my data points get clustered into cluster 0, and I cannot figure out what I am doing wrong as all the tutorials and code I have come across online point to using this method. </p>
  
  <p>In addition I have also tried normalizing the tf-idf matrix before K-means but that also produces the same result. I know cosine distance is a better measure to use, but I expected using standard K-means in Apache Spark would provide meaningful results.</p>
  
  <p>Can anyone help with regards to whether I have a bug in my code, or if something is missing in my data clustering pipeline?</p>
  
  <p>Thank you in advance!</p>
  
  <p>Here is the implementation in python which does not group all documents together even with a high number of max features:</p>
  
  <pre><code>#imports
  import pandas as pd
  import os
  import nltk
  from nltk.tokenize import RegexpTokenizer
  from nltk.corpus import stopwords
  
  import numpy as np
  
  from sklearn.feature_extraction.text import TfidfVectorizer
  from sklearn.cluster import KMeans, MiniBatchKMeans 
  
  vectorizer = TfidfVectorizer(max_features=200000, lowercase=True,
                               min_df=5, stop_words='english',
                               use_idf=True)
  
  X = vectorizer.fit_transform(df['document'])
  
  #Apply K-means to create cluster
  from time import time
  
  km = KMeans(n_clusters=20, init='k-means++', max_iter=20, n_init=1,
              verbose=False)
  
  km.fit(X)
  
  #result
  3     2634
  6     1720
  18    1307
  15     780
  0      745
  1      689
  16     504
  8      438
  7      421
  5      369
  11     347
  14     330
  4      243
  13     165
  10     136
  17     118
  9      113
  19     106
  12      87
  2       62
  </code></pre>
  
  <p>I would have thought that we could replicate something similar in pyspark using KMeans with Euclidean distance before trying cosine or Jaccard distances in KMeans. Any solutions or comments?</p>