1. A method, in a data processing system, for external word embedding neural network language models, the method comprising: configuring the data processing system with an external word embedding neural network language model that accepts as input a sequence of words and predicts a current word based on the sequence of words, wherein the external word embedding neural network language model combines an external embedding matrix to a history word embedding matrix and a prediction word embedding matrix of the external word embedding neural network language model; receiving a sequence of input words by the data processing system; applying a plurality of previous words in the sequence of input words as inputs to the external word embedding neural network language model in the data processing system, wherein the external word embedding neural network language model generates a predicted current word based on the plurality of previous words; and processing, by the data processing system, a current word in the sequence of input words based on the predicted current word generated by the external word embedding neural network language model.