title: Algorithm to remove words in corpus with small occurrence
body: <p>I have a large (+/- 300,000 lines) dataset of text fragments that contain some noisy elements. With noisy I mean words of slang, type errors, etc… I wish to filter out these noisy elements to have a more clean dataset.</p>
  
  <p>I read some papers that propose to filter these out by keeping track of the occurrence of each word. By setting a treshold (eg. less than 20) we can assume these words are noise and thus can safely be removed from the corpus.</p>
  
  <p>Maybe there are some libraries or algorithms that do this in a fast and efficient way. Ofcourse I tried it myself first but this is EXTREMELY slow!</p>
  
  <p>So to summarize, I am looking for an algorithm that can filter out words, in a fast and efficient way, that occur less than a particular treshold. Maybe I add a small example:</p>
  
  <pre><code>This is just an example of whaat I wish to acccomplish.
  </code></pre>
  
  <p>The words 'whaat' and 'acccomplish' are misspelled and thus likely to occur less often (If we assume to live in a perfect world and typos are rare …). I wish to end up with</p>
  
  <pre><code>This is just an example of I wish to.
  </code></pre>
  
  <p>Thanks!</p>
  
  <p>PS: If possible, I'd like to have an algorithm in Java (or pseudo-code so I can write it myself)</p>