Recurrent Neural Network (LSTM) not converging during optimization
I think the problem lies with your text processing to one-hot vectors.
Try using embeddings instead of one-hot vectors. 
An embedding is also a n-dimesional vector that allows words with similar meanings to have similar vectorial representation. 
One-hot vectors don't have such information. For them, there's a set of words of say cardinality c , then each vector is cx1 . The 1 in the vector just represents its location. It's like bit manipulation in this sense. 
So no semantic meaning is preserved. For e.g., in your corpus, there may be words like adore and love. Both have similar meaning. But one-hot vector for love and adore may be far away, depending upon where in set love and adore are mentioned. 

But if you use embeddings, words with similar meanings will have similar representations in predefined vector space. 

With this, your LSTM will learn dependencies better and will start converging.

Hope this helps :)