Using Clustering in text processing Hi this is my first question in the Data Science stack. I want to create an algorithm for text classification. Suppose i have a large set of text and articles. Lets say around 5000 plain texts. I first use a simple function to determine the frequency of all the four and above character words. I then use this as the feature of each training sample. Now i want my algorithm to be able to cluster the training sets to according to their features, which here is the frequency of each word in the article. (Note that in this example, each article would have its own unique feature since each article has a different feature, for example an article has 10 "water and 23 "pure" and another has 8 "politics" and 14 "leverage"). Can you suggest the best possible clustering algorithm for this example?
If you want to proceed on your existing path I suggest normalizing each term's frequency by its popularity in the entire corpus, so rare and hence predictive words are promoted. Then use random projections to reduce the dimensionality of these very long vectors down to size so your clustering algorithm will work better (you don't want to cluster in high dimensional spaces).

But there are other ways of topic modeling. Read this tutorial to learn more.