title: How to create input samples from pandas dataframe for a LSTM model?
body: <p>I'm trying to create a <code>LSTM</code> model that gives me binary output buy or not. I have data that is in the format of: <code>[date_time, close, volume]</code>, in millions of rows. I'm stuck at formatting the data as 3-D; Samples, Timesteps, features.</p>
  
  <p>I have used pandas to read the data. I want to format it so I can get 4000 samples with 400 timesteps each, and two features (close and volume). Can someone advise on how to do this?</p>
  
  <p>EDIT:
  I am using the TimeseriesGenerator as advised, but I am not sure how to check my sequences and replace the output Y with my own binary buy output.</p>
  
  <pre><code>df = normalize_data(df)
  
  print("Creating sequences for NN \n")
  targets = df.drop('date_time', 1)
  train = keras.preprocessing.sequence.TimeseriesGenerator(df, targets, 1, sampling_rate=1, stride=1,
                                                           start_index=0, end_index=int(len(df.index)*0.8),
                                                           shuffle=True, reverse=False, batch_size=time_steps)
  </code></pre>
  
  <p>This is running without error, but now the output is the first close value after input timeseries.</p>
  
  <p>EDIT 2:
  So thus far my code looks like this:</p>
  
  <pre><code>df = data.normalize_data(df)
  targets = df.iloc[:, 3]  # Buy signal target
  
  df.drop('y1', axis=1, inplace=True)
  df.drop('y2', axis=1, inplace=True)
  
  train = TimeseriesGenerator(df, targets, length=1, sampling_rate=1, stride=1,
                              start_index=0, end_index=int(len(df.index) * 0.8),
                              shuffle=True, reverse=False, batch_size=time_steps)
  
  # number of samples
  print("Samples: " + str(len(train)))
  x, y = train[0]
  print(str(x))
  </code></pre>
  
  <p>The output is as follows:</p>
  
  <pre><code>Samples: 8
  Traceback (most recent call last):
  File "/home/stian/.local/lib/python3.6/site- 
  packages/pandas/core/indexes/base.py", line 3078, in get_loc
  return self._engine.get_loc(key)
  File "pandas/_libs/index.pyx", line 140, in 
  pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/index.pyx", line 162, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/hashtable_class_helper.pxi", line 1492, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas/_libs/hashtable_class_helper.pxi", line 1500, in pandas._libs.hashtable.PyObjectHashTable.get_item
  KeyError: range(418, 419)
  
  During handling of the above exception, another exception occurred:
  
  Traceback (most recent call last):
  File "./main.py", line 94, in &lt;module&gt;
  data_menu()
  File "./main.py", line 42, in data_menu
  data_menu()
  File "./main.py", line 56, in data_menu
  nn_menu()
  File "./main.py", line 76, in nn_menu
  nn.nn_gen(pre_processed_data)
  File "/home/stian/git/stian9k/nn.py", line 33, in nn_gen
  x, y = train[0]
  File "/home/stian/.local/lib/python3.6/site-packages/keras_preprocessing/sequence.py", line 378, in __getitem__
  samples[j] = self.data[indices]
  File "/home/stian/.local/lib/python3.6/site-packages/pandas/core/frame.py", line 2688, in __getitem__
  return self._getitem_column(key)
  File "/home/stian/.local/lib/python3.6/site-packages/pandas/core/frame.py", line 2695, in _getitem_column
  return self._get_item_cache(key)
  File "/home/stian/.local/lib/python3.6/site-packages/pandas/core/generic.py", line 2489, in _get_item_cache
  values = self._data.get(item)
  File "/home/stian/.local/lib/python3.6/site-packages/pandas/core/internals.py", line 4115, in get
  loc = self.items.get_loc(item)
  File "/home/stian/.local/lib/python3.6/site-packages/pandas/core/indexes/base.py", line 3080, in get_loc
  return self._engine.get_loc(self._maybe_cast_indexer(key))
  File "pandas/_libs/index.pyx", line 140, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/index.pyx", line 162, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/hashtable_class_helper.pxi", line 1492, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas/_libs/hashtable_class_helper.pxi", line 1500, in pandas._libs.hashtable.PyObjectHashTable.get_item
  KeyError: range(418, 419)
  </code></pre>
  
  <p>So it seems that even tough I am getting 8 objects from the generator I am not able to look them up. If I test the type: print(str(type(train))) I get TimeseriesGenerator object. Any advise is much appreciated again.</p>
  
  <p>EDIT 3:
  it turns out timeseriesgenerator did not like pandas dataframes. The issue was resolved by converting to numpy array as well as converting pandas timestamp type to float.</p>