My Keras bidirectional LSTM model is giving terrible predictions I am trying to predict velocity (dynamics) values for notes that make up a piece of music using a bi-directional LSTM, following this blog post pretty closely: http://imanmalik.com/cs/2017/06/05/neural-style.html

I think I have set up everything correctly, but when predicting my model seems to produce only gibberish. Here's a plot of a prediction done on a validation sample (y-axis is always time and x-axis is, for I., a note activation array; for II. the predicted velocities and for III. the true, expected velocities). As you can see, the predicted velocity values (in the middle graph) are identical for all time steps (save for a few in the very beginning and at the very end, but that's hard to see in the plot):



For this particular prediction I had trained my model with 44 batches of 4 samples each and for 1 epoch, but I have also tried training it for 20 epochs and with different batch sizes and it doesn't seem to give better results.

Using TensorFlow backend.
Loading numpy data ...
173 train sequences
10 test sequences
x_train shape: (173,)
y_train shape: (173,)
Setting up model ...
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
bidirectional_1 (Bidirection (4, None, 128)            123392
_________________________________________________________________
bidirectional_2 (Bidirection (4, None, 128)            98816
_________________________________________________________________
bidirectional_3 (Bidirection (4, None, 128)            98816
_________________________________________________________________
time_distributed_1 (TimeDist (4, None, 88)             11352
=================================================================
Total params: 332,376
Trainable params: 332,376
Non-trainable params: 0
_________________________________________________________________
None
Training model ...
Epoch 1/1
2018-03-10 14:47:27.650226: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.2 AVX
44/44 [==============================] - 2145s 49s/step - loss: 0.0783 - mean_squared_error: 0.0783
Saving model ...
Loss and metrics: [0.002714238129556179, 0.0027141105383634567]


So, in other words, I end up with a validation MSE of 0.0027 after 1 epoch. Here is the code for my Keras model:

number_of_notes = 88
input_size = number_of_notes * 2 # 88 notes * 2 states (pressed, sustained).
output_size = number_of_notes # 88 notes.

units = 64
dropout = 0.2 # Drop 20% of units for linear transformation of inputs.

model = Sequential()
model.add(Bidirectional(LSTM(units, return_sequences=True, dropout=dropout),
                        input_shape=(None,input_size),
                        batch_input_shape=(args.batch_size,None,input_size)))
model.add(Bidirectional(LSTM(units, return_sequences=True, dropout=dropout)))
model.add(Bidirectional(LSTM(units, return_sequences=True, dropout=dropout)))
model.add(TimeDistributed(Dense(output_size, activation='sigmoid')))
model.compile(loss='mse', optimizer=Adam(lr=0.001, clipnorm=10), metrics=['mse'])

print('Training model ...')

number_of_train_batches = np.ceil(len(x_train)/float(args.batch_size))
model.fit_generator(batch_generator(x_train, y_train),
                    steps_per_epoch=number_of_train_batches,
                    epochs=args.epochs)


Here is how I predict:

prediction_data = np.load(path)

# Copy prediction input N times to create a batch of the right size.
tiled = np.tile(prediction_data, [args.batch_size,1,1])

raw_prediction = model.predict(tiled, batch_size=args.batch_size)[0]
prediction = (raw_prediction * 127).astype(int) # Float -&gt; MIDI velocity.


So what I am asking is, where am I likely to be going wrong with my model? Which parameters should I try changing in order to get better results? Or how can I change my model or training set-up to get better results? Etc. Any and all help is appreciated!
I think you cannot use a bi-directional LSTM for prediction, because of the time dimension of the music. I mean the backwards layer has to predict the latest value first and only after predicting it sees the sequence which gives the context- This is like you watch a reversed movie and yo have to guess how the first frame looks like without knowing the rest of it. 

Also, check out stylenet of the imanmalik model. The interpretation lstm which does the prediction is unidirectional. Only the genre net units are bidirectional, but they get the predicted value already as input.