"Rare words" on vocabulary I am trying to create a sentiment analysis model and I have a question.
After I preprocessed my tweets and created my vocabulary I've noticed that I have words that appear less than 5 times in my dataset (Also there are many of them that appear 1 time). Many of them are real words and not gibberish. My thinking is that if I keep those words then they will get wrong &quot;sentimental&quot; weights and gonna make my model worse.
Is my thinking right or am I missing something?
My vocab size is around 40000 words and those that are &quot;rare&quot; are around 10k.Should I &quot;sacrifice&quot; them?
Instead of dropping rare words or incorporating them risking their scarcity in the training data leads to poor predictions, you can opt for a third alternative: using a subword vocabulary.
You can use approaches like byte-pair encoding (BPE) to extract a subword vocabulary, that removes the out-of-vocabulary word problem and reduces data sparsity in general. There is the canonical python implementation as well as the popular implementation by Google called sentencepiece.