ValueError in CNN+RNN model in keras I am trying to build a CNN+RNN model for a computer vision problem.
below is my code

def cnn_with_rnn(shape):
    model = Sequential()
    model.add(Conv2D(32, (3, 3), strides=(2, 2), activation="relu",kernel_initializer='truncated_normal',bias_initializer='truncated_normal', input_shape=shape))
    model.add(MaxPool2D(pool_size=(2, 2)))
    model.add(Conv2D(64, (3, 3), strides=(2, 2),kernel_initializer='truncated_normal',bias_initializer='truncated_normal',  activation="relu"))
    model.add(MaxPool2D(pool_size=(2, 2)))
    model.add(Conv2D(128, (3, 3), strides=(2, 2),kernel_initializer='truncated_normal',bias_initializer='truncated_normal',  activation="relu"))
    model.add(LSTM(50))
    model.add(Dense(1))
    model.compile(optimizer=Adam(lr=1e-5), loss="mse", metrics=[custom_metric])


My image is a RGB image with the following shape - [66,200,3], where 3 is the number of color channels.

When i am trying to run the above code , i am getting the following error

ValueError: Input 0 is incompatible with layer lstm_3: expected ndim=3, found ndim=4


How can i combine CNN+RNN for colored images and how to solve my above problem?
It might not be directly possible to shoehorn the output of your CNN directly into an LSTM (at least without being a lot more thorough with your dimensions).



Another approach is to have a look at the Keras wrapper layer: TimeDistributed. You essentially extract features using your Conv layers as usual, but do that over time-steps, not just a random mini-batch. The features the come out then are e.g. over 5 consecutive timeframes -> this would be one single sample for the LSTM. If you perform a minibatch of say 10 samples, this means 10 * 5 = 50 input frames.

Straight from the linked documentation:


  This wrapper applies a layer to every temporal slice of an input.
  
  The input should be at least 3D, and the dimension of index one will
  be considered to be the temporal dimension.
  
  Consider a batch of 32 samples, where each sample is a sequence of 10
  vectors of 16 dimensions. The batch input shape of the layer is then
  (32, 10, 16), and the input_shape, not including the samples
  dimension, is  (10, 16).
  
  You can then use TimeDistributed to apply a Dense layer to each of the
  10 timesteps, independently:


# as the first layer in a model
model = Sequential()
model.add(TimeDistributed(Dense(8), input_shape=(10, 16)))
# now model.output_shape == (None, 10, 8)




As a hint for dimensions, make use of model.summary() on a compiled model in order to inspect the dimensions of each layer through the model. You will likely have to get a pen and paper out to work through the dimensions, as things start to become a little complicated when using such a wrapper layer.

Unfortunately I don't have any links/tutorials to show how this is exactly done in Keras, but hopefully the description above gives you enough info for some useful Bing! searches. Just kiddin... Google searches  ;-)