If a categorical feature only occurs a few times in a data set, should I drop it?
Indeed it's often a good idea to remove boolean features which are very rare, but the problem is that choosing a threshold by intuition is not necessarily optimal. Whenever possible the optimal value should be determined experimentally, and typically that should be possible for efficient methods such as log regression or SVM. The idea is simply to consider a range of values as threshold and run a grid search using a subset of the training set. The threshold is exactly like an hyper-parameter for the learning method.