title: Understanding CNN + LSTM RNN in Tensorflow
body: <p>Hey there I want to use a CNN + RNN in order to do a regression task on images and I am not sure how to properly handle sequence length and states.</p>
  
  <p>I thought about doing the following: Use the CNN to extract features for one frame. Put the flattened activation maps into the LSTM and save the state. Reduce the LSTM output to my regression value. For the next frame I would do the same with restoring the state of the LSTM with the previous iteration. 
  But that feels completly wrong since I am building a RNN around my LSTM cell which is not how its supposed to be right?</p>
  
  <p>But if I input a sequence of frames into the LSTM (after CNN application to all of them) I get multiple outputs and a state. If I reuse that state I dont see the point for the frame sequence at all. I am totally confused.</p>
  
  <p>Currently I am doing this but that is not working better than just a CNN for applied on every frame...</p>
  
  <pre><code>with tf.variable_scope('CNN'):
      for time_step in xrange(sequence_length):
          if time_step &gt; 0: tf.get_variable_scope().reuse_variables()
          cnn_res = CNN(images[time_step], normalizer_params=normalizer_params, regularizer=regularizer)
          cnn_outputs.append(cnn_res)
  
  cnn_outputs = tf.pack(cnn_outputs)
  with tf.variable_scope('RNN'):
      lstm_cell = LSTMBlockCell(128)
      lstm_cell = tf.nn.rnn_cell.DropoutWrapper(lstm_cell, output_keep_prob=keep_prob)
      cell = tf.nn.rnn_cell.MultiRNNCell([lstm_cell] *3)
      (rnn_outputs, state) = tf.nn.dynamic_rnn(cell, cnn_outputs,initial_state=initial_state, time_major=True,dtype=tf.float32)
      rnn_outputs = rnn_outputs[sequence_length-1] # Using only last output for sequence, also tried to take every output into account.
      rnn_outputs = layers.flatten(rnn_outputs)
  </code></pre>
  
  <p>Some fully connected layers reduce the <code>rnn_outputs</code> to my single value.</p>
  
  <p>Actually what I want to do something is this (only that I want to get a value for the currently received frame without having any future frames): <a href="https://stackoverflow.com/questions/36992340/how-do-you-pass-video-features-from-a-cnn-to-an-lstm">How do you pass video features from a CNN to an LSTM?</a> but I have a hard time realizing this in tensorflow</p>