title: Recurrent Neural Network predictions follow shape of test set but shifted down
body: <p>I was wondering if anyone has ever seen something like this. My predictions constantly seem to follow the actual form of the data but always end up shifted down or shifted up. Sometimes it'll seem to be right on as well. Thanks in advance for your help.</p>
  
  <p>EDIT: Data is from Amazon stock closing prices on finance.yahoo.com.</p>
  
  <p><a href="https://i.stack.imgur.com/sWeL8.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/sWeL8.png" alt="enter image description here"></a></p>
  
  <p>My code is shown below if you'd like to look at it. I have a callback that isn't outlined here since the results occur even when the callback isn't used though it seems to happen more frequently with the callback.</p>
  
  <p>FULL WORKING CODE:</p>
  
  <pre><code>import numpy as np
  import pandas as pd
  from sklearn import preprocessing
  from keras.preprocessing.sequence import TimeseriesGenerator
  from keras.models import Sequential
  from keras.layers import LSTM
  from keras.layers import Dense
  from keras.layers import Bidirectional
  from keras.layers import SimpleRNN
  from keras.layers import Flatten
  import matplotlib.pyplot as plt
  from keras.utils import plot_model
  from numpy.random import seed
  
  from keras.callbacks import ModelCheckpoint
  
  data = pd.read_csv("D:\\AMZN2.csv",header=0,usecols=['Date','Close'],parse_dates=True,index_col='Date')
  
  # Scaled data such that it is centered at 0 with unit variance.
  scaler = preprocessing.MinMaxScaler(feature_range=(0,1))
  scaledData = scaler.fit_transform(data)
  
  train = scaledData[:5000]
  validation = scaledData[5000:5250]
  test = scaledData[5250:]
  
  # Generator Creation #
  trainGen = TimeseriesGenerator(data=train,targets=train,length=1,
                                sampling_rate=1,stride=1,
                                shuffle=False,reverse=False,
                                batch_size=8)
  
  valGen = TimeseriesGenerator(data=validation,targets=validation,length=1,
                                sampling_rate=1,stride=1,
                                shuffle=False,reverse=False,
                                batch_size=8)
  testGen = TimeseriesGenerator(data=test,targets=test,length=1,
                                sampling_rate=1,stride=1,
                                shuffle=False,reverse=False,
                                batch_size=8) #length 1 batch_size 500
  
  # Designing the Model #
  AMZN = Sequential()
  AMZN.add(LSTM(32,return_sequences=False,input_shape=(1,1)))
  AMZN.add(Dense(1))
  
  # Compiling and training the model #
  steps_per_epoch = np.ceil(train.shape[0]/128)
  #RMSprop
  AMZN.compile(loss='mean_absolute_error',optimizer='adam',metrics=['mae','mse'])
  history = AMZN.fit_generator(trainGen,validation_data=valGen,#steps_per_epoch = steps_per_epoch,
                     epochs=20,verbose = 1)
  
  # Predicting and undoing the scaling #
  predictions = AMZN.predict_generator(testGen)
  predictions = scaler.inverse_transform(predictions)
  test = scaler.inverse_transform(test)
  
  # Plotting the Prediction and Test #
  plt.plot(predictions,'--',label='Predictions')
  plt.plot(test,label='Actual')
  plt.xlabel("Observation")
  plt.ylabel("Price")
  plt.title("AMZN Stock Prediction Vs. Actual")
  plt.legend()
  plt.show()
  </code></pre>
  
  <p>EDIT EDIT EDIT:</p>
  
  <hr>
  
  <pre><code>predict = np.zeros(len(test)+1)
  predict[0] = AMZN.predict(np.asarray([[train[-1]]]))
  
  for i in range(len(test)):
      predict[i+1] = AMZN.predict(test[i].reshape(1,1,1))
      predict[i] = scaler.inverse_transform(predict[i].reshape(1,1))
  </code></pre>
  
  <p>TRAINING RESULTS OF THIS MANUAL METHOD (The steep drop is because I used np.zeros with one extra zero):</p>
  
  <hr>
  
  <p><a href="https://i.stack.imgur.com/Ek2Ur.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/Ek2Ur.png" alt="enter image description here"></a></p>
  
  <p>EDIT:</p>
  
  <p>Based on how I ran my predictions, I would say length is the number of variables in the input space used to forecast. This is because when I designate a length of 3 I need to provide the model.predict function 3 values.</p>