title: LSTM keras, music MIDI generation, output converges to a fixed configuration
body: <p>I have a project on Neural Networks on music using MIDI files as inputs.
  I'm thinking of a NN composed by 2 LSTM layers, followed by a dense one.</p>
  
  <p>My data is organized in arrays (matrix with shape = [length, 128] - representing the MIDI files).
  In the MIDI array, each row - a 128 elements array - represents an "instant", i.e. what notes are being played at each instant, with the corresponding velocity.</p>
  
  <p>Now, what I know about lstm is the theory behind it, but - in practice - I know almost nothing. </p>
  
  <p>My idea was to train an LSTM-NN using random "windows" of 256 rows input, to predict the following 1.
  In this way, the network should learn how to map a serie of 256 "insants" to the following 1.</p>
  
  <p><a href="https://i.stack.imgur.com/eEu97.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/eEu97.png" alt="enter image description here"></a></p>
  
  <p><a href="https://i.stack.imgur.com/EDEcm.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/EDEcm.png" alt="enter image description here"></a></p>
  
  <p>[GENERATION]</p>
  
  <p>Given that, the network should then try to start from a 256 instants window (taken from an existing song) and try to produce 1 following instants, then, taking the old 255 instants + the predicted 1, try to produce other 1 instants and so on... In this way I should have a "new" song, starting from 256 instants taken from a song.</p>
  
  <p>[PROBLEM]</p>
  
  <p>This brings the prediction to converge to a fixed result, so my result is a song that starts with the 256 "cool" instants, then - after few instants - is a constant vector representing some notes that are played forever. Like someone standing on a piano for 2 minutes.</p>
  
  <p>This problem makes me think that there is something wrong with the model.</p>
  
  <p>Given what I described before, is my model correct?</p>
  
  <p>[MODEL]</p>
  
  <pre><code>model = keras.models.Sequential()
  model.add(keras.layers.InputLayer((num_steps,ENCODING)))
  model.add(keras.layers.CuDNNLSTM(hidden_size, return_sequences=True))
  model.add(keras.layers.CuDNNLSTM(hidden_size2, return_sequences=False))
  model.add(keras.layers.Dense(hidden_size3,activation="relu"))
  if use_dropout:
      model.add(keras.layers.Dropout(0.5))
  model.add(keras.layers.Dense(ENCODING, activation=mapping_to_target_range))
  </code></pre>
  
  <p>This is my model, which uses these parameters:</p>
  
  <p>[PARAMETERS]</p>
  
  <pre><code>num_steps = 256
  prediction_steps = 12
  train_batch_size = 32
  val_batch_size = 1
  hidden_size = 512
  hidden_size2 = 256
  hidden_size3 = 128
  use_dropout = True
  num_epochs = 10
  data_path = "checkpoints"
  ENCODING = 128
  TIME_STEPS_PER_SONG = 500
  </code></pre>
  
  <p>This is my model.summary()
  [SUMMARY]</p>
  
  <pre><code>Layer (type)                 Output Shape              Param #   
  =================================================================
  cu_dnnlstm_1 (CuDNNLSTM)     (None, 256, 512)          1314816   
  _________________________________________________________________
  cu_dnnlstm_2 (CuDNNLSTM)     (None, 256)               788480    
  _________________________________________________________________
  dense_1 (Dense)              (None, 128)               32896     
  _________________________________________________________________
  dropout_1 (Dropout)          (None, 128)               0         
  _________________________________________________________________
  dense_2 (Dense)              (None, 128)               16512     
  =================================================================
  Total params: 2,152,704
  Trainable params: 2,152,704
  Non-trainable params: 0
  </code></pre>
  
  <p>And this is my generator
  [GENERATOR]</p>
  
  <pre><code>def generate(data, num_steps,prediction_steps, batch_size):
      size = len(data)
      while True:
          x = []
          y = []
  
          for i in range(batch_size): 
              j = np.random.randint(size)
              song = np.array(data[j])
              start = np.random.randint(0,len(song) - num_steps - prediction_steps )
              x.append(song[start:start + num_steps])
              y.append(song[(start + num_steps + 1)])
              j += 1
          yield np.array(x, dtype=np.float32), np.array(y ,dtype=np.float32)
  
  def mapping_to_target_range( x, target_min=0, target_max=127 ) :
      # x in range(0,1)
      return keras.backend.clip(x,0,127)
  </code></pre>
  
  <p>[TRAIN]</p>
  
  <pre><code>model.fit_generator(generate(data,num_steps,prediction_steps,train_batch_size), TIME_STEPS_PER_SONG, num_epochs,
                   validation_data=generate(data,num_steps,prediction_steps,val_batch_size),
                      validation_steps=1, callbacks=[checkpointer])
  </code></pre>
  
  <p>[TRAINING ERROR]</p>
  
  <p>The training error is something around a hundred, the strange thing is that it doesn't really seem to follow a trend (epoch by epoch), it bounces between 150 and 100, the val_loss instead seems to decrease, but it is around 50 and 20... It doesn't make much sense</p>
  
  <p>Is this correct for my purpose? 
  Do you know what could be wrong?</p>
  
  <p>Is there something wrong with the model structure, or the training samples?</p>
  
  <p>[IMPORTANT EDIT]
  I changed encoding for the input, I used a vector of 129 elements, the last column represents the "time this configuration will be the same".
  So for example, if I play "C2" and, after 50 "ticks", I play also "C4"; I'll have 2 vector, one representing C2, the following one representing both C2 and C4. 
  The first vector will have as last column the value 50, the second one will have the time to reach the following event.</p>
  
  <p>This configuration should avoid to train on something which prediction will be - highly probable - the last "instant" played. This happens maybe because it is highly improbable to find the instant in which a change in notes happens! </p>
  
  <p>Given this new configuration, the net generates a series of zeroes with the last column (representing the time that configuration keeps playing) that increases each generation (?)</p>
  
  <p>e.g.
  [[ 0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0
     0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0
     0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0
     0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0
     0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0
     0  0  0  0  0  0  0  0 36]]</p>
  
  <p>[GENERATION CODE]</p>
  
  <pre><code>model.load_weights("checkpoints/model-09.hdf5")
  data = np.load("datasets_raw/Hou01.npy")
  batt = np.array(data[:num_steps,:])
  num_ticks = len(data)
  generated_song = []
  for tick in np.array(batt):
      generated_song.append(tick)
  i = 0
  while len(generated_song) &lt; 2000:
  
      song = np.array(generated_song)
      batt = np.array([song[i:i+num_steps]])
      new_tick = model.predict(batt)
      tick = np.array(new_tick,dtype=np.uint8)
      generated_song.append(tick[0])
      print(tick)
      i += 1
  </code></pre>