How to Avoid rarely used discrete feature values in a dataset
Working with Categorical Features can raise a few challenges. For instance, you might encounter features with high cardinality in certain values or the other case (your case), features with rare categories.

The first thing you can consider is, what you will lose if you drop those rows. What extra info do you get with keeping the rare categories. Here you probably need the domain knowledge of the project to identify the impact of the specific data. If your dataset is small or you don't feel safe to remove those rows, then you should look for alternatives.

The second approach is to group all the rare categories in one new called "Rare" or "Other". So, whenever you have a record with that value, you replace it with the new one. 

Finally, you can try either a Dimentionality Reduction such as PCA or a Feature Hashing where you will map multiple categories in one new value.