title: Understanding why tensorflow RNN is not learning toy data
body: <p>I am trying to train a Recurrent Neural Network using Tensorflow (r0.10, python 3.5) on a toy classification problem, but I am getting confusing results.</p>
  
  <p>I want to feed in a sequence of zeros and ones into an RNN, and have the target class for a given element of the sequence to be the number represented by the current and previous values of the sequence, treated as a binary number. For example:</p>
  
  <pre><code>input sequence: [0,     0,     1,     0,     1,     1]
  binary digits : [-, [0,0], [0,1], [1,0], [0,1], [1,1]]
  target class  : [-,     0,     1,     2,     1,     3]
  </code></pre>
  
  <p>It seems like this is something an RNN should be able to learn quite easily, but instead my model is only able to distinguish classes [0,2] from [1,3]. In other words, it is able to distinguish the classes whose current digit is 0 from those whose current digit is 1. This is leading me to believe that the RNN model is not correctly learning to look at the previous value(s) of the sequence.</p>
  
  <p>There are several tutorials and examples ([<a href="https://www.tensorflow.org/versions/r0.9/tutorials/recurrent/index.html" rel="nofollow">1</a>], [<a href="https://github.com/sherjilozair/char-rnn-tensorflow" rel="nofollow">2</a>], [<a href="https://github.com/aymericdamien/TensorFlow-Examples/blob/master/notebooks/3_NeuralNetworks/recurrent_network.ipynb" rel="nofollow">3</a>]) that demonstrate how to build and use Recurrent Neural Networks (RNNs) in tensorflow, but after studying them I still do not see my problem (it does not help that all the examples use text as their source data).</p>
  
  <p>I am inputting my data to <code>tf.nn.rnn()</code> as a list of length <code>T</code>, whose elements are <code>[batch_size x input_size]</code> sequences. Since my sequence is one dimensional, <code>input_size</code> is equal to one, so essentially I believe I am inputting a list of sequences of length <code>batch_size</code> (the <a href="https://www.tensorflow.org/versions/r0.9/tutorials/recurrent/index.html" rel="nofollow">documentation</a> is unclear to me about which dimension is being treated as the time dimension). <strong>Is that understanding correct?</strong> If that is the case, then I don't understand why the RNN model is not learning correctly.</p>
  
  <p>It's hard to get a small set of code that can run through my full RNN, this is the best I could do (it is mostly adapted from <a href="https://github.com/tensorflow/tensorflow/tree/master/tensorflow/models/rnn/ptb" rel="nofollow">the PTB model here</a> and <a href="https://github.com/sherjilozair/char-rnn-tensorflow" rel="nofollow">the char-rnn model here</a>):</p>
  
  <pre><code>import tensorflow as tf
  import numpy as np
  
  input_size = 1
  batch_size = 50
  T = 2
  lstm_size = 5
  lstm_layers = 2
  num_classes = 4
  learning_rate = 0.1
  
  lstm = tf.nn.rnn_cell.BasicLSTMCell(lstm_size, state_is_tuple=True)
  lstm = tf.nn.rnn_cell.MultiRNNCell([lstm] * lstm_layers, state_is_tuple=True)
  
  x = tf.placeholder(tf.float32, [T, batch_size, input_size])
  y = tf.placeholder(tf.int32, [T * batch_size * input_size])
  
  init_state = lstm.zero_state(batch_size, tf.float32)
  
  inputs = [tf.squeeze(input_, [0]) for input_ in tf.split(0,T,x)]
  outputs, final_state = tf.nn.rnn(lstm, inputs, initial_state=init_state)
  
  w = tf.Variable(tf.truncated_normal([lstm_size, num_classes]), name='softmax_w')
  b = tf.Variable(tf.truncated_normal([num_classes]), name='softmax_b')
  
  output = tf.concat(0, outputs)
  
  logits = tf.matmul(output, w) + b
  
  probs = tf.nn.softmax(logits)
  
  cost = tf.reduce_mean(tf.nn.seq2seq.sequence_loss_by_example(
      [logits], [y], [tf.ones_like(y, dtype=tf.float32)]
  ))
  
  optimizer = tf.train.GradientDescentOptimizer(learning_rate)
  tvars = tf.trainable_variables()
  grads, _ = tf.clip_by_global_norm(tf.gradients(cost, tvars),
                                    10.0)
  train_op = optimizer.apply_gradients(zip(grads, tvars))
  
  init = tf.initialize_all_variables()
  
  with tf.Session() as sess:
      sess.run(init)
      curr_state = sess.run(init_state)
      for i in range(3000):
          # Create toy data where the true class is the value represented
          # by the current and previous value treated as binary, i.e.
          train_x = np.random.randint(0,2,(T * batch_size * input_size))
          train_y = train_x + np.concatenate(([0], (train_x[:-1] * 2)))
  
          # Reshape into T x batch_size x input_size
          train_x = np.reshape(train_x, (T, batch_size, input_size))
  
          feed_dict = {
              x: train_x, y: train_y
          }
          for j, (c, h) in enumerate(init_state):
              feed_dict[c] = curr_state[j].c
              feed_dict[h] = curr_state[j].h
  
          fetch_dict = {
              'cost': cost, 'final_state': final_state, 'train_op': train_op
          }
  
          # Evaluate the graph
          fetches = sess.run(fetch_dict, feed_dict=feed_dict)
  
          curr_state = fetches['final_state']
  
          if i % 300 == 0:
              print('step {}, train cost: {}'.format(i, fetches['cost']))
  
      # Test
      test_x = np.array([[0],[0],[1],[0],[1],[1]]*(T*batch_size*input_size))
      test_x = test_x[:(T*batch_size*input_size),:]
      probs_out = sess.run(probs, feed_dict={
              x: np.reshape(test_x, [T, batch_size, input_size]),
              init_state: curr_state
          })
      # Get the softmax outputs for the points in the sequence
      # that have [0, 0], [0, 1], [1, 0], [1, 1] as their
      # last two values.
      for i in [1, 2, 3, 5]:
          print('{}: [{:.4f} {:.4f} {:.4f} {:.4f}]'.format(
                  [1, 2, 3, 5].index(i), *list(probs_out[i,:]))
               )
  </code></pre>
  
  <p>The final output here is</p>
  
  <pre><code>0: [0.4899 0.0007 0.5080 0.0014]
  1: [0.0003 0.5155 0.0009 0.4833]
  2: [0.5078 0.0011 0.4889 0.0021]
  3: [0.0003 0.5052 0.0009 0.4936]
  </code></pre>
  
  <p>which indicates that it is only learning to distinguish [0,2] from [1,3]. <strong>Why isn't this model learning to use the previous value in the sequence?</strong></p>