title: keras LSTM feeding input with the right shape
body: <p>I am getting some data from a pandas dataframe with the following shape</p>
  
  <pre><code>df.head()
  &gt;&gt;&gt;
  Value USD   Drop 7  Up 7    Mean Change 7   Change      Predict
  0.06480     2.0     4.0     -0.000429       -0.00420    4
  0.06900     1.0     5.0     0.000274        0.00403     2
  0.06497     1.0     5.0     0.000229        0.00007     2
  0.06490     1.0     5.0     0.000514        0.00200     2
  0.06290     2.0     4.0     0.000229        -0.00050    3
  </code></pre>
  
  <p>The first 5 columns are intended to be the <code>X</code> and predict the <code>y</code>. This is how I preprocess the data for the model</p>
  
  <pre><code>from keras.models import Sequential
  from keras.callbacks import EarlyStopping
  from keras.callbacks import ModelCheckpoint
  import pandas as pd
  from sklearn.model_selection import train_test_split
  import numpy as np
  from sklearn.metrics import accuracy_score
  from keras.layers import LSTM
  from sklearn import preprocessing
  
  # Convert a Pandas dataframe to the x,y inputs that TensorFlow needs
  def to_xy(df, target):
      result = []
      for x in df.columns:
          if x != target:
              result.append(x)
      # find out the type of the target column.  Is it really this hard? :(
      target_type = df[target].dtypes
      target_type = target_type[0] if hasattr(target_type, '__iter__') else target_type
      # Encode to int for classification, float otherwise. TensorFlow likes 32 bits.
      if target_type in (np.int64, np.int32):
          # Classification
          dummies = pd.get_dummies(df[target])
          return df.as_matrix(result).astype(np.float32), dummies.as_matrix().astype(np.float32)
      else:
          # Regression
          return df.as_matrix(result).astype(np.float32), df.as_matrix([target]).astype(np.float32)
  
  # Encode text values to indexes(i.e. [1],[2],[3] for red,green,blue).
  def encode_text_index(df, name):
      le = preprocessing.LabelEncoder()
      df[name] = le.fit_transform(df[name])
      return le.classes_
  
  df['Predict'].value_counts()
  &gt;&gt;&gt;
  4    1194
  3     664
  2     623
  0     405
  1      14
  Name: Predict, dtype: int64
  
  predictions = encode_text_index(df, "Predict")
  predictions
  &gt;&gt;&gt;
  array([0, 1, 2, 3, 4], dtype=int64)
  
  X,y = to_xy(df,"Predict")
  X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.20, shuffle=False)
  
  X_train
  &gt;&gt;&gt;
  array([[ 6.4800002e-02,  2.0000000e+00,  4.0000000e+00, -4.2857142e-04,
          -4.1999999e-03],
         [ 6.8999998e-02,  1.0000000e+00,  5.0000000e+00,  2.7414286e-04,
           4.0300000e-03],
         [ 6.4970002e-02,  1.0000000e+00,  5.0000000e+00,  2.2857143e-04,
           7.0000002e-05],
         ...,
         [ 9.5987000e+02,  5.0000000e+00,  2.0000000e+00, -1.5831429e+01,
          -3.7849998e+01],
         [ 9.9771997e+02,  5.0000000e+00,  2.0000000e+00, -1.6948572e+01,
          -1.8250000e+01],
         [ 1.0159700e+03,  5.0000000e+00,  2.0000000e+00, -1.3252857e+01,
          -7.1700001e+00]], dtype=float32)
  
  y_train
  &gt;&gt;&gt;
  array([[0., 0., 0., 0., 1.],
         [0., 0., 1., 0., 0.],
         [0., 0., 1., 0., 0.],
         ...,
         [0., 0., 0., 0., 1.],
         [0., 0., 0., 0., 1.],
         [0., 0., 0., 0., 1.]], dtype=float32)
  
  X_train[1]
  &gt;&gt;&gt;
  array([6.8999998e-02, 1.0000000e+00, 5.0000000e+00, 2.7414286e-04,
         4.0300000e-03], dtype=float32)
  
  X_train.shape
  &gt;&gt;&gt;
  (2320, 5)
  
  X_train[1].shape
  &gt;&gt;&gt;
  (5,)
  </code></pre>
  
  <p>and finally the LSTM model (also it might look like not the best way to write one so will appreciate a rewrite of the inner layers as well if that's the case)</p>
  
  <pre><code>model = Sequential()
  #model.add(LSTM(128, dropout=0.2, recurrent_dropout=0.2, input_shape=(None, 1)))
  model.add(LSTM(50, dropout=0.2, return_sequences=True, input_shape=X_train.shape))
  model.add(LSTM(50, dropout=0.2, return_sequences=True))
  model.add(LSTM(50, dropout=0.2, return_sequences=True))
  model.add(LSTM(50, dropout=0.2, return_sequences=True))
  #model.add(Dense(50, activation='relu'))
  model.add(Dense(y_train.shape[1], activation='softmax'))
  
  #model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])
  #model.fit(X_train, y_train, epochs=1000)
  
  model.compile(loss='categorical_crossentropy', optimizer='adam')
  monitor = EarlyStopping(monitor='val_loss', min_delta=1e-2, patience=15, verbose=1, mode='auto')
  checkpointer = ModelCheckpoint(filepath="best_weights.hdf5", verbose=0, save_best_only=True) # save best model
  
  model.fit(X_train, y_train, validation_data=(X_test, y_test), callbacks=[monitor,checkpointer], verbose=2, epochs=1000)
  model.load_weights('best_weights.hdf5') # load weights from best model
  </code></pre>
  
  <p>running this throws this error</p>
  
  <pre><code>---------------------------------------------------------------------------
  ValueError                                Traceback (most recent call last)
  &lt;ipython-input-67-a17835a382f6&gt; in &lt;module&gt;()
       15 checkpointer = ModelCheckpoint(filepath="best_weights.hdf5", verbose=0, save_best_only=True) # save best model
       16 
  ---&gt; 17 model.fit(X_train, y_train, validation_data=(X_test, y_test), callbacks=[monitor,checkpointer], verbose=2, epochs=1000)
       18 model.load_weights('best_weights.hdf5') # load weights from best model
  
  c:\users\samuel\appdata\local\programs\python\python35\lib\site-packages\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, **kwargs)
      948             sample_weight=sample_weight,
      949             class_weight=class_weight,
  --&gt; 950             batch_size=batch_size)
      951         # Prepare validation data.
      952         do_validation = False
  
  c:\users\samuel\appdata\local\programs\python\python35\lib\site-packages\keras\engine\training.py in _standardize_user_data(self, x, y, sample_weight, class_weight, check_array_lengths, batch_size)
      747             feed_input_shapes,
      748             check_batch_axis=False,  # Don't enforce the batch size.
  --&gt; 749             exception_prefix='input')
      750 
      751         if y is not None:
  
  c:\users\samuel\appdata\local\programs\python\python35\lib\site-packages\keras\engine\training_utils.py in standardize_input_data(data, names, shapes, check_batch_axis, exception_prefix)
      125                         ': expected ' + names[i] + ' to have ' +
      126                         str(len(shape)) + ' dimensions, but got array '
  --&gt; 127                         'with shape ' + str(data_shape))
      128                 if not check_batch_axis:
      129                     data_shape = data_shape[1:]
  
  ValueError: Error when checking input: expected lstm_48_input to have 3 dimensions, but got array with shape (2320, 5)
  </code></pre>
  
  <p>I've tried a lot of variations of the X_train input shape but every single one throws some error, I also checked the <a href="https://keras.io/layers/recurrent/#lstm" rel="noreferrer">Keras docs</a> but it wasn't clear on how the data should be fed to the model</p>
  
  <h2>Attempt No. 1 from Suggestions</h2>
  
  <p>First is reshaping X_train</p>
  
  <pre><code>data = np.resize(X_train,(X_train.shape[0],1,X_train.shape[1]))
  model.add(LSTM(50, dropout=0.2, return_sequences=True, input_shape=data.shape))
  </code></pre>
  
  <p>this fails with an error </p>
  
  <pre><code>ValueError: Input 0 is incompatible with layer lstm_52: expected ndim=3, found ndim=4 
  </code></pre>
  
  <p>suggested I feed it in as </p>
  
  <pre><code>model.add(LSTM(50, dropout=0.2, return_sequences=True, input_shape=X_train.shape[1:]))
  </code></pre>
  
  <p>that throws the same error</p>
  
  <pre><code>ValueError: Input 0 is incompatible with layer lstm_63: expected ndim=3, found ndim=2
  </code></pre>
  
  <h2>Sugestion 2</h2>
  
  <p>use the default X,y from pandas</p>
  
  <pre><code>y = df['Predict']
  X = df[['Value USD', 'Drop 7', 'Up 7', 'Mean Change 7', 'Change']]
  
  X = np.array(X)
  y = np.array(y)
  
  X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.20, shuffle=False)
  </code></pre>
  
  <p>also that LSTM expect input in the following way <code>(batch_size, timesteps, input_dim)</code></p>
  
  <p>so I tried this</p>
  
  <pre><code>model.add(LSTM(50, dropout=0.2, return_sequences=True, input_shape=(100, 100, X_train.shape)))
  </code></pre>
  
  <p>which throws this error</p>
  
  <pre><code>TypeError: Error converting shape to a TensorShape: int() argument must be a string, a bytes-like object or a number, not 'tuple'.
  </code></pre>
  
  <p>and a different way</p>
  
  <pre><code>model.add(LSTM(50, dropout=0.2, return_sequences=True, input_shape=(100, 100, X_train[1].shape)))
  </code></pre>
  
  <p>returns the same error</p>
  
  <pre><code>TypeError: Error converting shape to a TensorShape: int() argument must be a string, a bytes-like object or a number, not 'tuple'.
  </code></pre>