"Rare words" on vocabulary
Instead of dropping rare words or incorporating them risking their scarcity in the training data leads to poor predictions, you can opt for a third alternative: using a subword vocabulary.
You can use approaches like byte-pair encoding (BPE) to extract a subword vocabulary, that removes the out-of-vocabulary word problem and reduces data sparsity in general. There is the canonical python implementation as well as the popular implementation by Google called sentencepiece.