TF-IDF Features vs Embedding Layer Have you guys tried to compare the performance of TF-IDF features* with a shallow neural network classifier vs a deep neural network models like an RNN that has an embedding layer with word embedding as weights next to the input layer? I tried this on a couple of tweet datasets and got surprising results: f1 score of~65% for the TF-IDF vs ~45% for the RNN. I tried the setup embedding layer + shallow fully connected layer vs TF-IDF + fully connected layer but got almost same result difference. Can you guys give some opinion on how TF-IDF features can outperform the embedding layer of a deep NN? Is this case common? Thanks!


I've used unigrams and bigrams to produce the TF-IDF features
It is common for TFIDF to be a strong model. People constantly get high places in Kaggle competitions with TFIDF models. Here is a link to the winning solution that used TFIDF as one of its features (1st place Otto product classification). You will most likely get a stronger model if you combine the TFIDF and RNN into one ensemble. Other results from Kaggle:


2nd place: https://www.kaggle.com/c/stumbleupon/discussion/6184
4th place: https://www.kaggle.com/c/avito-demand-prediction/discussion/59881
3rd place: https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge/discussion/52762


https://www.kaggle.com/c/avito-demand-prediction/discussion/56897:


  A good number of kernels are going the traditional route with
  CountVectorizer/TF-IDF, and some brave souls (I say brave because
  training is slower and the results don't seem as spectacular so far)
  have been experimenting with embeddings, as per the previous
  competitions.