TF-IDF vs TF for classification Let's suppose that I have a dataset of 1000 documents.

Each document is a restaurant review (so relatively short text) and it has labels {Negative, Indifferent, Positive}.

Let's suppose that the dataset has 600 positive reviews, 200 indifferent reviews and 200 negative reviews.

I want to train a classifier to classify a review as Negative or Indifferent or Positive based on the text of the review.

Without using any word embeddings for now the best way to go is to use a TF-IDF model.

However, as I am thinking this more carefully I am not entirely surely if this is the best way to go in comparison with a simply TF model.

Specifically, a TF-IDF model will take the inverse document frequency of terms irrespectively to any labels/categories.

So in the example above if many of the positive reviews have the word 'positive' in them then this word will automatically have modified (and in general lower) TF-IDF scores simply because the majority of the documents in the dataset are positive (600 documents).

On the other hand, with a simply TF then the word 'positive' would have a very high value and it would be evident that it is directly related to positive reviews.

Why the TF-IDF is necessarily the best way to go in cases like these?
So in the example above if many of the positive reviews have the word 'positive' in them then this word will automatically have modified (and in general lower) TF-IDF scores simply because the majority of the documents in the dataset are positive (600 documents).


You're thinking of the word positive as example because you are a human and you know what to select. The words which have the highest frequency in English are stop words: the, a, is... These are useless and would add a lot of noise in your model. You can remove them using a predefined list of stop words, but what about words which are not stop words but are frequent enough, for instance food, place, chef... This is where IDF helps.