Most efficient way to prep data for pytorch lstm from pandas dataframe
I am trying to feed data into my lstm. I have the data across multiple csv's, so I created a generator to load it in. However, I am having some issues on prepping the data for my lstm.

I understand this code (I got it from the pytorch docs)

seq_len = 5
batch_size= 3
cols_num = 10
hidden_size=20
num_layers = 2

rnn = nn.LSTM(input_size=cols_num, hidden_size=hidden_size, num_layers=num_layers)

data = torch.randn(seq_len, batch_size, cols_num)
h0 = torch.randn(batch_size, seq_len, hidden_size)
c0 = torch.randn(batch_size, seq_len, hidden_size)
output, (hn, cn) = rnn(data)


However, I think my disconnect is with using actual data not the torch.randn().

This is my current generator:

def data_loader(batch_size, fp, dropcol, seq_len):
    while True:
        for f in fp:
            gc.collect()
            df=pd.read_csv(f)
            df=df.replace(np.nan, 0)
            df=df.drop(dropcol,1)
            df['minute'] = df['minute'].apply(lambda x: min_idx(x))
            row_count, col_count = df.shape
            encoder_input = []
            prev = 0
            for idx, b in enumerate(range(1, row_count)):
                end = prev + batch_size
                window = df.iloc[prev:end]
                prev = end - 1
                w = np.array(window, dtype='float64')
                if w.shape[0] != batch_size:  break
                encoder_input.append(w)
                if idx == seq_len:
                    w0 = encoder_input
                    encoder_input = []
                    yield w0


but I get errors when I run this:

loader = data_loader(batch_size=batch_size, fp=&lt;list of csvs&gt;, dropcol=idcol, seq_len=2)
lstm = nn.LSTM(input_size=input_size, hidden_size=hidden_size, num_layers=num_layers, batch_first=batch_first)


for batch in loader:
    b = torch.tensor(batch)
    output, hidden = lstm(b)


The error: 
RuntimeError: Expected object of scalar type Float but got scalar type Double for argument #4 'mat1'

What is the error in how I am thinking? Also, how should I format an h0 or c0 from the data?

The error is not in how you are thinking, the error is in how Pytorch models accept input. The default data type created in a Pytorch tensor is torch.float64
where the default (and possibly only) data type accepted by models is torch.float32. 

To fix this use:

b = torch.tensor(batch, dtype=torch.float32)


This will convert your inputs to torch.float32.
