Should I remove features that occur very rarely to build a model? I am trying ML techniques in language processing.
I have got 3000 short texts and I extract features(words and phrases) from all of them and build a vocabulary. I end up with 6000 od features and most of them occurs once or twice.
So for example from texts:

0: One text here
1: Another text there

I got
    One Another  Text  Here  There   Target
  0   1       0     1      1     0     True
  1   0       1     1      0     1    False

So if word &quot;one&quot; occured once I have got as a column and it is False for all other 2999 texts.
Should I drop these columns? Or should use different technique?
This amount of columns make me some problems becuase it takes a lot of time to build a classify.
Short answer: 

yes, if they occur so rarely, they can only lead to overfit, so it's better to ignore them as features.

Longer answer:

Usually one puts all those unique occurrence in a single feature/token, and that's the way I suggest you to proceed. So if you have two features that appears only once, you can join them to create a feature that appears twice.

But anyway, 3000 short text are too few to create a NLP model that can work well. 
To mitigate your lack of data, you can use pre-trained word embeddings like the ones available here. Doing so you can also keep most of the single occurring words/features, because in those word embeddings they already have a defined semantic meaning.