LSTM is not converging
I tried to play with your code and found changing loss function to the cross_entropy alternate of negative log-likelihood makes the difference between 2000th epoch's loss and  9000th epoch's loss is greater about 0.2 alternate of 0.09

I also tried to change optimizer and learning rate but no loss didn't improve.

you can explore the modified code may help you with another idea