From text to K-Means Vectors input
Question: I've just started diving into Machine Learning, specifically into Clustering.
(I'm using Python but this is irrelevant) My goal is, starting from a
collection of tweets (100K) about fashion world, to perform KMeans over their
text.

Till now I've filtered texts, truncating stopwords, useless terms,
punctuation; done lemmatization (exploiting Part Of Speech tagging for better
results).

I show the user the most frequent terms, hashtags, bigrams, trigrams,..9grams
so that he can refine preprocessing adding words to useless terms.

My initial idea was to use the top n(1K) terms as features, creating foreach
tweet a vector of fixed size n(1K) having a cell set to a value if the top
term (of this cell) appear in this tweet (maybe calculating the cell's value
with TFIDF).

Am I missing something(the 0 values will be considered)? Can I exploit n-grams
in some way?

This [scikit article](http://scikit-
learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html#sklearn.feature_extraction.text.TfidfVectorizer)
is pretty general and I'm not understanding the whole thing.

(Is LSA dimensionality reduction useful or is it better reducing the number of
features (so vectors dimension) manually? )


Answer: This [other sklearn page](http://scikit-
learn.org/stable/auto_examples/text/document_clustering.html#example-text-
document-clustering-py) contains an example of k-means clustering of texts.

But to address some of your specific questions:

> My initial idea was to use the top n(1K) terms as features, creating foreach
> tweet a vector of fixed size n(1K) having a cell set to a value if the top
> term (of this cell) appear in this tweet (maybe calculating the cell's value
> with TFIDF).

A standard approach to achieve that is to use sklearn's
[CountVectorizer](http://scikit-
learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html)
and playing with the parameter `min_df`.

    
    
    from sklearn.feature_extraction.text import CountVectorizer
    cv = CountVectorizer(min_df=10)
    X = cv.fit_transform(texts)
    

The above piece of code converts an array of texts into features X. Setting
`min_df=10` will ignore all words with less than 10 occurrences (to my
understanding, there is no direct way to say "take the top 1000" but this is
equivalent).

> Can I exploit n-grams in some way?

Yes, CountVectorizer can deal with n-grams. The `ngram_range` parameter
specifies the range of ngrams to consider (which starting "n" and which ending
"n"). For instance,

    
    
    cv = CountVectorizer(min_df=10, ngram_range=(2,2))
    

will build features based on bigrams instead of individual words (unigrams).
For mixing unigrams and bigrams

    
    
    cv = CountVectorizer(min_df=10, ngram_range=(2,2))
    

Then you can replace a CountVectorizer by a TfIdfVectorizer, which transforms
the word counts to weight more informative words.

> Is LSA dimensionality reduction useful or is it better reducing the number
> of features (so vectors dimension) manually?

Short answer, it depends on your purpose. The example in the link I mentioned
above does apply LSA first. But also, in my experience, "topic model" methods
like LSA or NMF can be already considered a clustering into latent semantic
topics. For instance,

    
    
    from sklearn.decomposition import NMF
    nmf = NMF(n_components=20)
    mu = nmf.fit_transform(X)
    

This will convert the features X into projected feature vectors mu of 20
dimensions. Each dimension d can be interpreted as the score of the text in
topic d. By assigning each sample to the dimension with max score, this can
also be interpreted as a clustering.

