1. A method, in a data processing system comprising a processor and a memory, the memory comprising instructions that are executed by the processor to specifically configure the processor to implement an automatic speech recognition system comprising an acoustic model and an external word embedding neural network language model, the method comprising: configuring the automatic speech recognition system with an external word embedding neural network language model that accepts as input a sequence of words and predicts a current word based on the sequence of words, wherein the external word embedding neural network language model combines an external embedding matrix to a history word embedding matrix and a prediction word embedding matrix of the external word embedding neural network language model; receiving as input an audio signal and generating a sequence of words based on the audio signal by the acoustic model; receiving the sequence of input words by the automatic speech recognition system from the acoustic model; applying a plurality of previous words in the sequence of input words as inputs to the external word embedding neural network language model in the automatic speech recognition system, wherein the external word embedding neural network language model generates a predicted current word based on the plurality of previous words; and processing, by the automatic speech recognition system, a current word in the sequence of input words based on the predicted current word generated by the external word embedding neural network language model, wherein processing the current word comprises recognizing a current spoken word in the audio signal based on the predicted current word.