LSTM language model not working I am trying to use a Keras LSTM neural network for character level language modelling. As the input, I give it the last 50 characters and it has to output the next one. It has 3 layers of 400 neurons each. For the training data, I am using 'War of The Worlds' by H.G. Wells which adds up to 269639 training samples and 67410 validation samples.

After 7 epochs the validation accuracy has reached 35.1% and the validation loss has reached 2.31. However, after being fed the first sentence of war of the worlds to start it outputs:


  the the the the the the the the the the the the the the the the...


I'm not sure where I'm going wrong; I don't want it to overfit and output passages straight from the training data but I also don't want it to just output 'the' repeatedly. I'm really at a loss as to what I should do to improve it.

Any help would be greatly appreciated. Thanks!
You can follow the below steps :


LSTMs are slower in terms of convergence. They take much time to train amd thereby give better results. Try training the network for a more number of epochs like 50 or 65.
Use smaller batch size. Try using a RMSProp optimizer. It takes a long time to converge but gives excellent results in case of recurrent nets.
Also, try word  level predictions. Means, train your LSTM in such a way that it can predict words and not characters.
Use dropout layers in between two LSTM layers. Use a rate like 0.2 or 0.35.


As a working example, you can refer this 
project which generates poem lines using LSTM in tensorflow.