How to down-weight non-words in text classification?
Usually, those non-sensical words are not problematic because they appear in one or two documents, and people usually just filter words with such low frequencies.

Are you not able to do this? Or do your non-sensical words appear that much?

Anyhow, your suggestion is what people use:


  outright ignore tokens that aren't discernable by a dictionary. Is there?


Exactly. There is no magic way to know if a word is English or not. What word processors do is to use a dictionary, as you yourself suggested.

In python, before stemming, you could filter based on pyenchant.

import enchant
d = enchant.Dict("en_US")
d.check('hello')  # True
d.check('mmmmmmmm')  # False


I bet this would be good enough.

But, if there are a lot of false negatives, you could ask the dictionary for the most similar words and then apply a word distance measure. Here I will use Levenshtein distance  as implemented in python:

&gt;&gt;&gt; r = d.suggest('helo')
&gt;&gt;&gt; r
['hole', 'help', 'helot', 'hello', 'halo', 'hero', 'hell', 'held', 'helm', 'he lo', 'he-lo', 'heel', 'loathe', 'Helios', 'helicon']
&gt;&gt;&gt; min([LD('helo', w) for w in r])
1


The distance will be higher for the non-sensical words you suggested:

&gt;&gt;&gt; d.suggest('mmmmmmmm')
['mammogram', "Mammon's"]
&gt;&gt;&gt; min([LD('mmmmmmmm', w) for w in d.suggest('mmmmmmmm')])
5
&gt;&gt;&gt; min([LD('aaannnammmmrrr', w) for w in d.suggest('aaannnammmmrrr')])
12
&gt;&gt;&gt; min([LD('hdhhdhhhhhjjj', w) for w in d.suggest('hdhhdhhhhhjjj')])
10


To recap:


filter words that appear in less than 3 documents
if that's not enough or it is not a possibility for your use-case, use a dictionary


I think this last suggestion of mine of using Levenshtein distance to the closest English words is overkill.

If your documents involve a lot of urban words or hip terms, you could also filter based on the number of Google results, instead of using a common dictionary. Your word hdhhdhhhhhjjj, for instance, gives me only two Google results, one of which is this stackexchange. Data science is about being creative. :)

There probably are heuristics or statistics based on the number of consonants in a word, and letter combinations, which you could use to probabilistic shrink your documents' dictionary, but I wouldn't go there. It will be too much work and very brittle.