title: Simple RNN(LSTM) model doesn't make progress
body: <p>I have <code>np_final_x(71520, 2, 50)</code> and <code>np_final_y(71520, 1, 50)</code> corpus</p>
  <p><a href="https://www.dropbox.com/s/k15dtcak78jaf34/np_final_x_len_2.npy?dl=0" rel="nofollow noreferrer">https://www.dropbox.com/s/k15dtcak78jaf34/np_final_x_len_2.npy?dl=0</a>
  <a href="https://www.dropbox.com/s/555lhbdnkl6gmrq/np_final_y_len_2.npy?dl=0" rel="nofollow noreferrer">https://www.dropbox.com/s/555lhbdnkl6gmrq/np_final_y_len_2.npy?dl=0</a></p>
  <p>This means predict like this <code>I use -&gt; this</code>,
  <code>you give -&gt; me</code></p>
  <p>predict next word from two words.</p>
  <p>And each words are encoded into 50 dimension vector.</p>
  <pre><code>from tensorflow.keras.models import Sequential
  from tensorflow.keras.layers import LSTM,Dropout
  from tensorflow.keras.layers import Activation, Dense
  from tensorflow.keras.optimizers import Adam
  import numpy as np
  
  final_x = np.load('np_final_x_len_2.npy')
  final_y = np.load('np_final_y_len_2.npy')
  
  in_out_neurons = 50
  n_hidden = 512 # not so much change
  #n_hidden = 1 # not so much change
  model = Sequential()
  model.add(LSTM(n_hidden, batch_input_shape=(None, 2, in_out_neurons), return_sequences=True))
  model.add(Dense(in_out_neurons,activation=&quot;tanh&quot;)) #not so much change
  #model.add(Dense(in_out_neurons,activation=&quot;sigmoid&quot;)) #not so much change
  #model.add(Dense(in_out_neurons, activation=&quot;relu&quot;))   #not so much change
  optimizer = Adam(learning_rate=0.0001)
  model.compile(loss=&quot;mean_squared_error&quot;, optimizer=optimizer)
  
  model.summary()
  model.fit(         
      final_x,final_y,
      batch_size=400,
      epochs=10,
      validation_split=0.1
  )
  </code></pre>
  <p>However what I got was around 0.017~0.019 not so much progress, even I changed any parameters.</p>
  <p>Moreover, even when <code>n_hidden = 1</code>,result doesn't change.</p>
  <p>So I guess something is wrong basically.</p>
  <p>appreciate any help and hints. thank you.</p>
  <pre><code>Epoch 1/10
  161/161 [==============================] - 9s 46ms/step - loss: 0.0195 - val_loss: 0.0192
  Epoch 2/10
  161/161 [==============================] - 8s 49ms/step - loss: 0.0191 - val_loss: 0.0188
  Epoch 3/10
  161/161 [==============================] - 8s 52ms/step - loss: 0.0187 - val_loss: 0.0186
  Epoch 4/10
  161/161 [==============================] - 11s 68ms/step - loss: 0.0185 - val_loss: 0.0184
  Epoch 5/10
  161/161 [==============================] - 12s 77ms/step - loss: 0.0184 - val_loss: 0.0183
  Epoch 6/10
  161/161 [==============================] - 13s 83ms/step - loss: 0.0183 - val_loss: 0.0183
  Epoch 7/10
  161/161 [==============================] - 14s 85ms/step - loss: 0.0183 - val_loss: 0.0182
  </code></pre>