Keras: Merged/Concatenated model perform worst than separate models memes recognition I have a dataset of memes, and I'm trying to predict if a certain meme is sexist or not, using image and text together. 

Right now I have two models, a VGG16 fine tuned CNN for images and a LSTM model for text, each of them with Keras.

Both models perform well alone (~0.8-0.9 accuracy), and I'm trying to merge them to see if I can get a better result.

I'm concatenating the output of each model like this:

CNN

input_tensor = layers.Input(shape=(image_size,image_size,3))

vgg_model = VGG16(input_tensor = input_tensor, weights = 'imagenet', include_top=False)

for layer in vgg_model.layers:
    layer.trainable = False

l2_strength = 1e-5
dropout_prob = 0.5

x = vgg_model.output
x = layers.Flatten(input_shape=vgg_model.output_shape[1:])(x)
x = layers.Dense(256, activation='relu', kernel_regularizer=regularizers.l2(l2_strength))(x)
x = layers.Dropout(dropout_prob)(x)
x = layers.Dense(64, activation='relu', kernel_regularizer=regularizers.l2(l2_strength))(x)

vgg_model = models.Model(vgg_model.input, x)



LSTM

input_text = layers.Input(shape=(1, 512))
x = layers.LSTM(32, dropout=0.5, name="LTSM")(input_text)
x = layers.Dense(10, kernel_initializer='normal', activation='relu')(x)

text_model = models.Model(input_text, x)


MERGED

x = layers.concatenate([vgg_model.output, text_model.output])
out = layers.Dense(1, activation='softmax', name='output_layer')(x)

merged_model = models.Model([vgg_model.input, text_model.input], out)

merged_model.compile(loss='binary_crossentropy', 
                     optimizer='adam', 
                     metrics=['accuracy'])


and I train the model like this:

merged_model.fit(
    [train_image, train_text], 
    y = train_y, 
    epochs=50,
    batch_size=64,
    validation_split=0.2)


where train_images are the images and train_text are the associated text, and the label are a 1 or 0 if a meme is sexist or not.

The problem is that both train and validation accuracy are always 0.5, with a loss of ~8.0-9.0, and never improve during training, so it's not able to learn, but it's weird seen that the model alone give good performance.

I was wondering if someone encountered the same problem and/or do you have any suggestion on why this happen.
I had the same problem and I solved it adding a BatchNormalization layer.
For example:

# create model ConvLSTM
input_convlstm = Input(name='convlstm', shape=(n_steps, 1, n_length, n_metadata))
branch_convlstm = BatchNormalization()(input_convlstm)
branch_convlstm = ConvLSTM2D(filters=64, kernel_size=(1,3), activation='tanh', input_shape=(n_steps, 1, n_length, n_metadata))(branch_convlstm)
branch_convlstm = Flatten()(branch_convlstm)
branch_convlstm = Dense(128, activation='tanh')(branch_convlstm)

# create model metadata
input_metadata = Input(name='metadata', shape=(21,))
branch_metadata = Dense(neurons[0], activation='tanh')(input_metadata)
branch_metadata = BatchNormalization()(branch_metadata)
branch_metadata = Dropout(dropout_rate)(branch_metadata)

# create final model
concat = Concatenate()([branch_convlstm, branch_metadata])
out = Dense(n_outputs, activation='softmax')(concat)
model = Model(inputs=[input_convlstm, input_metadata], outputs=out)