Training LSTM Recurrent Network in TensorFlow I have trained RNN's before "by hand" using basic tools like Numpy or BLAS, but I am having trouble getting a simple RNN to converge in TF. Full Code

I tried standard things like adjusting the learning rate, the momentum, and adding noise to the gradient, but I am concerned that I don't understand what TF is doing underneath.

In a couple tutorials, it appeared that in TF you train the networks like Echo State Networks where the hidden state is evolved on random interconnections to train up a reservoir, then you use a linear (or non-linear) transformation to map the hidden state onto your labels.

I examined the gradients that Tensorflow uses to train, and it appears that it is training the internal state transition matrices as one would expect if using "Back-Propagation Through Time", which I can't tell if it is trying to use.

Can you help me understand what I am doing/understanding incorrectly with regards to training RNN's in TF?

cell_layer = tf.contrib.rnn.BasicLSTMCell(state_size, state_is_tuple=True)
cell = tf.contrib.rnn.MultiRNNCell([cell_layer]*1)
x, y_ = tf.placeholder(tf.float32, shape=(batch_size,1)), tf.placeholder(tf.float32, shape=(batch_size,1))
outputs, states = tf.nn.dynamic_rnn(cell, tf.expand_dims(x, -1), dtype=tf.float32)# init_state)

W = tf.Variable(tf.random_uniform((state_size, 1), -1.0, 1.0))
b = tf.Variable(tf.random_uniform((1, 1), -1.0, 1.0))
y = tf.matmul(tf.reshape(outputs, (-1, state_size)), W) + b

loss = tf.reduce_mean(tf.square(y - y_))
opt = tf.train.MomentumOptimizer(1e-3, 0.9)
grad = [(g + tf.random_uniform(v.shape, -1.0, 1.0) * tf.reduce_mean(tf.abs(v))*noise, v)
        for g, v in opt.compute_gradients(loss)]
train = opt.apply_gradients(grad)


EDIT: Loss plot



I am less concerned about the loss than I am about the fact that there appear to not be any dynamics in between iterations. It appears to converge to the average. Below are some of the later losses and a part of the sequence to be predicted.

Loss:  11.9126
Actual:  bcdefghijklabcdefghijklabcdefghijklabcde
Pred:    ffffffffffffffffffffffffffffffffffffffff

Loss:  11.9092
Actual:  bcdefghijklabcdefghijklabcdefghijklabcde
Pred:    ffffffffffffffffffffffffffffffffffffffff

Loss:  11.918
Actual:  bcdefghijklabcdefghijklabcdefghijklabcde
Pred:    ffffffffffffffffffffffffffffffffffffffff

Loss:  11.9274
Actual:  bcdefghijklabcdefghijklabcdefghijklabcde
Pred:    ffffffffffffffffffffffffffffffffffffffff


Note

&gt;&gt;&gt; s = 'bcdefghijklabcdefghijklabcdefghijklabcde'
&gt;&gt;&gt; n = map(ord, s)
&gt;&gt;&gt; chr(sum(n)/len(n))
'f'
I removed the gradient noise (which did not seem to help, at least as you did it) and replaced your momentum optimizer with Adam using the default hyperparameters and it just worked. After 10,000 epochs I got a loss of ~8 with bcdefghijklabcdefghijklabcdefghijklabcde (actual) vs ccdeeffgghhiccdeeffgghhiccdeeffgghhiccde (predicted).

The moral is that optimizing neural networks is still an art.



p.s. It seems weird to use a regression loss with a classification problem.