title: How to use output of complete LSTM sequence ? tensorflow
body: <p>I might be wrong here but here goes.
  I'm using the code in this <a href="https://github.com/aymericdamien/TensorFlow-Examples/blob/master/examples/3%20-%20Neural%20Networks/recurrent_network.py" rel="nofollow noreferrer">post</a>.</p>
  
  <p>Specifically the code in </p>
  
  <pre><code>    outputs, states = rnn.rnn(lstm_cell, _X, initial_state=_istate)
  
      # Linear activation
      # Get inner loop last output
      return tf.matmul(outputs[-1], _weights['out']) + _biases['out']
  </code></pre>
  
  <p>The code above uses the many to one prediction scheme.</p>
  
  <p><a href="https://i.stack.imgur.com/Mpzfy.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/Mpzfy.jpg" alt="enter image description here"></a></p>
  
  <p>I would like to know if I could use the many to many scheme with this code.
  And the use the outputs of all the LSTM units to predict the class.
  I have tried replacing the last line with </p>
  
  <pre><code>return tf.matmul(outputs, _weights['out']) + _biases['out']
  </code></pre>
  
  <p>But then I get an error </p>
  
  <pre><code>  File "/media/anilil/Data/charm/Cnn/train_lstm_ucf.py", line 165, in &lt;module&gt;
      pred = RNN(x, istate, weights, biases,keep_prob)
    File "/media/anilil/Data/charm/Cnn/train_lstm_ucf.py", line 163, in RNN
      return tf.matmul(outputs, _weights['out']) + _biases['out']
    File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/math_ops.py", line 938, in matmul
      a = ops.convert_to_tensor(a, name="a")
    File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 529, in convert_to_tensor
      ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
    File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/constant_op.py", line 178, in _constant_tensor_conversion_function
      return constant(v, dtype=dtype, name=name)
    File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/constant_op.py", line 161, in constant
      tensor_util.make_tensor_proto(value, dtype=dtype, shape=shape))
    File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/tensor_util.py", line 319, in make_tensor_proto
      _AssertCompatible(values, dtype)
    File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/tensor_util.py", line 256, in _AssertCompatible
      raise TypeError("List of Tensors when single Tensor expected")
  TypeError: List of Tensors when single Tensor expected
  </code></pre>
  
  <h1>Background info (Non- vital)</h1>
  
  <p>Thanks for your input. I'm not sure myself if this approach would yield better results.</p>
  
  <p>I'm trying to replicate <a href="http://arxiv.org/pdf/1411.4389v3.pdf" rel="nofollow noreferrer">this paper</a> 
  Specifically the <a href="http://jeffdonahue.com/lrcn/" rel="nofollow noreferrer">activity recognition</a> from this paper.</p>
  
  <p>What they do is train a CNN with a single frame to predict which class the frame belongs to and then use the the dense layer features of this to train an LSTM and hence understanding the temporal relation between the single frames and improve the recognition accuracy.</p>
  
  <p>I have reproduced the results with the CNN and get an single frame accuracy (table 1:-  RGB singleframe)accuracy of 61%.</p>
  
  <p>I have extracted the (fc-6)features from this network and given it as an input to the LSTM but instead of an improvement in accuracy to ~71.2% I get the LSTM accuracy of 51% which is reduced. Not sure why this happens. (possibly the LSTM model they used was different is my guess)</p>
  
  <p>Any thoughts on that appreciated as well.</p>