Why I have a different number of terms in word2vec and TFIDF? How I can fix it?
Your problem is that TFIDF is cutting out around 90 terms.

Since you aren't using the min_df or max_df parameters and as far as I can tell you aren't doing any stemming/lemmatization, the only difference I can see between the two methods is the tokenizer.

There are two things I'd try out if I were you:


Try explicitly converting the word2vec corpus to lowercase. TfidfVectorizer does this by default and I can't see where you're doing it in the word2vec pipeline. Ignore this if your corpus is already lowercased.
Try using the nltk.WordPunctTokenizer() with the TfidfVectorizer. You can do this like this:




wpt = nltk.WordPunctTokenizer()
tv = TfidfVectorizer(min_df=0., max_df=1., norm='l2', use_idf=True, smooth_idf=True,
                     tokenizer=wpt.tokenize)