Learning token dictionary
Actually you can often just split on the white space, remove punctuation, and lowercase. This will provide your tokens. For example, if I had a string "jdf asdsa sdfr" (no English), then I could derive the tokens jdf, asdsa, and sdfr. The only thing that is known before hand would be the stopwords (and, be it the), which do come from the English dictionary. However, in this case, it does not sound like you would need stopwords. I would suggest looking at the following libraries available in python:

SpaCy

NLTK

Scikit-learn