Multi-dimensional tensors as input to rnn in tensorflow (tf.contrib.rnn.RNNCell)
As you have said RNN only accept as input a Tensor like [batch_size, sequence_lentgh, features].
In order to use RNN from tensorflow you will have to extract the features with a CNN for each frame and convert your CNN output data to a tensor that follows [batch_size, sequence_lentgh, features] shape in order to feed it to the RNN.
From tensorflow documentation about tf.contrib.rnn.RNNCell: "This definition of cell differs from the definition used in the literature. In the literature, 'cell' refers to an object with a single scalar output. This definition refers to a horizontal array of such units."
It seems, that rnn cell only accepts vectors as inputs. However I would like to feed images/videos to an rnn (e.g. [batch size, steps, height, width, channels]). Is there a way to do this using rnn cell and dynamic rnn, or do I have to manually construct an rnn?