title: Document Clustering in Java using Weka
body: <p>I'm trying to cluster a group of news articles in Java that are about a particular topic.  I crawled news sites about a particular topic using Crawler4J, rolled my own TF/IDF implementation comparing against a corpus (there were reasons that I didn't use the built in Weka or other implementations of TF/IDF, but they're probably out of scope for this question) and applied some other domain specific logic which leaves me with a bag of words + weights for each document (which I'm storing in a Map where the value is between 0 and 1).  I want to cluster the articles that are about a similar topic by looking at word weights, so I started playing with Weka's EM and SimpleKMeans clusterers.  The problem is that I'm getting pretty unimpressive results, and I'm trying to figure out whether I can do anything to better tune the clustering.</p>
  
  <p>For example in a toy data set of ~300 articles if I trim the words to the top 20 in each document and then use the full set of words as features I end up with ~2k unique features.  I'm using each article as a training instance, and trying some different parameters to tune the SimpleKMeans clusterer (for example ~20-30 clusters, 100 max iterations).  When I look at the data at a glance I see that the keywords for a lot of the articles seem very closely related, so I would expect high quality results and a bunch of clusters that contain ~5-10 articles.  Unfortunately the results are usually one cluster with > half of the articles, a bunch of clusters with 1 article each, and a few stragglers with 2-5 articles or so.</p>
  
  <p>Are these results expected, or is there a way to get higher quality results?  Note that I've also looked into LingPipe and other frameworks that provide clustering functionality, and played with rolling my own EM implementation (with modest but not great success).  Thanks in advance for any help!</p>