title: keras SimpleRNN model data shape interpretation
body: <p>I have a very simple keras sequential model:</p>
  
  <pre class="lang-py prettyprint-override"><code>model = keras.models.Sequential([
      keras.layers.SimpleRNN(1, input_shape=[None, 1], return_sequences=True),
  ])
  model.compile(loss='mse')
  </code></pre>
  
  <p>My (toy) data looks like this:</p>
  
  <pre class="lang-py prettyprint-override"><code>X = np.array([[1,2],[3,4],[5,6], [7,8]])[...,np.newaxis]
  y = np.array([3,5,7,9])[..., np.newaxis]
  </code></pre>
  
  <p>X.shape -> (4,2,1)<br>
  y.shape -> (4,1)</p>
  
  <p>Since I <code>return_sequences</code>, I expect my output shape to be <code>(4,2,1)</code>.<br>
  This is also what is returned from <code>model.predict(X)</code>.<br>
  My question: How is the loss computed in the fit, when the shape of the model prediction output mismatches the shape of <code>y</code>?<br>
  <code>keras.layers.mse(model.predict(X), y)</code> raises a shape mismatch..</p>
  
  <p>The full code to experiment with:</p>
  
  <pre class="lang-py prettyprint-override"><code>from tensorflow import keras
  import numpy as np
  
  X = np.array([[1,2],[3,4],[5,6], [7,8]])[...,np.newaxis]
  y = np.array([3,5,7,9])[..., np.newaxis]
  print(X.shape)
  print(y.shape)
  
  
  model = keras.models.Sequential([
      keras.layers.SimpleRNN(1, input_shape=[None, 1], return_sequences=True),
  ])
  
  model.compile(loss='mse', optimizer='adam')
  model.summary()
  
  model.fit(X, y, epochs = 1)
  print(model.predict(X).shape)
  
  # this fails as expected
  # keras.losses.mse(model.predict(X), y)
  </code></pre>
  
  <p>versions: keras 2.2.4-tf with tensorflow 2.1.0</p>