Keras: Merged/Concatenated model perform worst than separate models memes recognition
I had the same problem and I solved it adding a BatchNormalization layer.
For example:

# create model ConvLSTM
input_convlstm = Input(name='convlstm', shape=(n_steps, 1, n_length, n_metadata))
branch_convlstm = BatchNormalization()(input_convlstm)
branch_convlstm = ConvLSTM2D(filters=64, kernel_size=(1,3), activation='tanh', input_shape=(n_steps, 1, n_length, n_metadata))(branch_convlstm)
branch_convlstm = Flatten()(branch_convlstm)
branch_convlstm = Dense(128, activation='tanh')(branch_convlstm)

# create model metadata
input_metadata = Input(name='metadata', shape=(21,))
branch_metadata = Dense(neurons[0], activation='tanh')(input_metadata)
branch_metadata = BatchNormalization()(branch_metadata)
branch_metadata = Dropout(dropout_rate)(branch_metadata)

# create final model
concat = Concatenate()([branch_convlstm, branch_metadata])
out = Dense(n_outputs, activation='softmax')(concat)
model = Model(inputs=[input_convlstm, input_metadata], outputs=out)