title: LSTM. Calculating RMSE using inverse_transform
body: <p>I'm having some trouble calculating the RMSE (root-mean-squared-error) in my LSTM model. The model fits fine and I'm getting a good loss reduction, however when trying to inverse_transform my yhat results I get the following error: </p>
  
  <pre><code>non-broadcastable output operand with shape (399,1) doesn't match the broadcast shape (399,4)
  </code></pre>
  
  <p>Here's my code:</p>
  
  <p>Preprocessing:</p>
  
  <pre><code>btc = pd.read_csv('live_bitcoin.csv')
  twitter_sent = pd.read_csv('live_tweet.csv')
  reddit_sent = pd.read_csv('live_reddit.csv')
  
  btc.columns = ["price_usd","24h_volume_usd","market_cap_usd","available_supply","total_supply","percent_change_1h","percent_change_24h","percent_change_7d", "Sell", "Buy", "15m", "Stamp"]
  twitter_sent.columns = ["Sentiment", "Stamp"]
  reddit_sent.columns = ["Sentiment", "Stamp"]
  
  merged = pd.merge(twitter_sent, btc,  on='Stamp', how='inner').merge(reddit_sent, on='Stamp', how='inner')
  data = merged[["Sentiment_x", "Sentiment_y","24h_volume_usd", "market_cap_usd", "available_supply","price_usd"]].groupby(merged['Stamp']).mean()
  datag = data[["24h_volume_usd", "market_cap_usd", "available_supply","price_usd"]]
  tw_sentiment = data["Sentiment_x"]
  rdt_sentiment = data["Sentiment_y"]
  
  print "Dataset size: " + str(len(datag))
  print "Timespan: " + str(len(datag)/60) + " hours"
  
  from sklearn.preprocessing import MinMaxScaler
  
  scaler = MinMaxScaler(feature_range=(0, 1))
  values = datag.values.reshape(-1, datag.shape[1])
  tw_sentiment = tw_sentiment.values.reshape(-1, 1)
  rdt_sentiment = rdt_sentiment.values.reshape(-1, 1)
  tw_sentiment = tw_sentiment.astype('float32')
  rdt_sentiment = rdt_sentiment.astype('float32')
  values = values.astype('float32')
  scaled = scaler.fit_transform(values)
  </code></pre>
  
  <p>Training:</p>
  
  <pre><code>train_size = int(len(scaled) * 0.7)
  test_size = len(scaled) - train_size
  train, test = scaled[0:train_size,:], scaled[train_size:len(scaled),:]
  split = train_size
  
  def create_dataset(dataset, look_back, tw_sentiment, rdt_sentiment, sent=False):
      dataX, dataY = [], []
      for i in range(len(dataset) - look_back):
          if i &gt;= look_back:
              a = dataset[i-look_back:i+1, 0]
              a = a.tolist()
              if(sent==True):                
                  current_tw_sentiment = tw_sentiment[i].tolist()[0]
                  current_rdt_sentiment = rdt_sentiment[i].tolist()[0]
                  a.append(current_tw_sentiment)
                  a.append(current_rdt_sentiment)
              dataX.append(a)
              dataY.append(dataset[i + look_back, 0])
      print(len(dataY))
      return np.array(dataX), np.array(dataY)
  
  look_back = 2
  trainX, trainY = create_dataset(train, look_back, tw_sentiment[0:train_size], rdt_sentiment[0:train_size], sent=True)
  testX, testY = create_dataset(test, look_back, tw_sentiment[train_size:len(scaled)], rdt_sentiment[train_size:len(scaled)], sent=True)
  
  trainX = np.reshape(trainX, (trainX.shape[0], 1, trainX.shape[1]))
  testX = np.reshape(testX, (testX.shape[0], 1, testX.shape[1]))
  
  # Creating new model
  model = Sequential()
  model.add(LSTM(100, input_shape=(trainX.shape[1], trainX.shape[2]), return_sequences=True))
  model.add(LSTM(100))
  model.add(Dense(1))
  model.compile(loss='mae', optimizer='adam')
  model.save('LSTM_14-03-2018.h5')
  
  # Loading model
  # model = load_model('models/LSTM_12-03-2018_GOOD.h5')
  
  history = model.fit(trainX, trainY, epochs=300, batch_size=100, validation_data=(testX, testY), verbose=0, shuffle=False)
  yhat = model.predict(testX)
  yhat_inverse = scaler.inverse_transform(yhat.reshape(-1, 1))
  testY_inverse = scaler.inverse_transform(testY.reshape(-1, 1))
  rmse_sent = sqrt(mean_squared_error(testY_inverse, yhat_inverse))
  print "Done"
  print 'Test RMSE: %.3f' % rmse_sent
  </code></pre>
  
  <p>The main problem lies here: </p>
  
  <pre><code>  yhat_inverse = scaler.inverse_transform(yhat.reshape(-1,1))
    testY_inverse = scaler.inverse_transform(testY.reshape(-1,1))
  </code></pre>
  
  <p>For what I understand (still a beginner in ML), my yhat variable has a shape of (399, 1) as I'm trying to make a prediction based on several features. I'm only looking to revert my data to its previous transform so the RMSE error returns in an appropriate scale. I'm basically trying to reconvert prices to their normal scale. I'm also never re_transforming the data after the MinMaxScaler does it in the preprocessing stage.</p>
  
  <p>Any clues on what might be wrong?</p>