title: Input shape of stacked RNNs
body: <p>I'm trying to stack a few RNNs in series using Keras with a TensorFlow back-end. I can create a model with a single <code>SimpleRNN</code> layer, but when I try to add second <code>SimpleRNN</code> layer, I can't figure out the appropriate input size.</p>
  
  <pre><code>from keras import models
  from keras.layers.recurrent import SimpleRNN
  from keras.layers import Activation
  
  
  model = models.Sequential()
  
  hidden_units = 256
  skeleton_dimensions = 3 * 16  # 3 dimensions for 16 joints
  input_temporal_length = 7
  
  in_shape = (input_temporal_length, skeleton_dimensions,)
  
  # three hidden layers of 256 each
  model.add(SimpleRNN(hidden_units, input_shape=in_shape,
                      activation='relu', use_bias=True,))
  # what input shape is this supposed to have?
  model.add(SimpleRNN(hidden_units, input_shape=(1, skeleton_dimensions,),
                      activation='relu', use_bias=True,))
  </code></pre>
  
  <p>What should my second <code>SimpleRNN</code> have as input shape?</p>
  
  <p>The documentation of the <a href="https://keras.io/layers/recurrent/" rel="nofollow noreferrer">Recurrent Layers</a> seems to imply:</p>
  
  <blockquote>
    <p><strong>Output shape</strong></p>
    
    <ul>
    <li>if return_sequences: 3D tensor with shape  (batch_size, timesteps,
    units). </li>
    <li>else, 2D tensor with shape (batch_size, units).</li>
    </ul>
  </blockquote>
  
  <p>Given <code>return_sequences</code> is automatically set to <code>False</code> I tried to set the <code>input_shape</code> of the next dimension appropriately, but I get the error:</p>
  
  <pre><code>Using TensorFlow backend.
  Traceback (most recent call last):
    File "rnn_agony.py", line 19, in &lt;module&gt;
      activation='relu', use_bias=True,))
    File "/usr/local/lib/python3.5/dist-packages/keras/models.py", line 455, in add
      output_tensor = layer(self.outputs[0])
    File "/usr/local/lib/python3.5/dist-packages/keras/layers/recurrent.py", line 252, in __call__
      return super(Recurrent, self).__call__(inputs, **kwargs)
    File "/usr/local/lib/python3.5/dist-packages/keras/engine/topology.py", line 511, in __call__
      self.assert_input_compatibility(inputs)
    File "/usr/local/lib/python3.5/dist-packages/keras/engine/topology.py", line 413, in assert_input_compatibility
      str(K.ndim(x)))
  ValueError: Input 0 is incompatible with layer simple_rnn_2: expected ndim=3, found ndim=2
  </code></pre>