Training LSTM Recurrent Network in TensorFlow
I removed the gradient noise (which did not seem to help, at least as you did it) and replaced your momentum optimizer with Adam using the default hyperparameters and it just worked. After 10,000 epochs I got a loss of ~8 with bcdefghijklabcdefghijklabcdefghijklabcde (actual) vs ccdeeffgghhiccdeeffgghhiccdeeffgghhiccde (predicted).

The moral is that optimizing neural networks is still an art.



p.s. It seems weird to use a regression loss with a classification problem.