LSTM network doesn't converge, what should be changed?
writing here my suggestion, because i haven't earned the right to comment yet.

Your main "problem" could be your loss function. It converges, this is why your loss value is decreasing. So I suggest to let it maybe train longer.

Alternatively you could change the loss function to fit your need. For example you could use:

loss  = tf.reduce_mean(tf.square(exps-outs))


You will get a smaller loss value which decreases clearly after every output.

I hope this helps :)