How-To Generate a 3D Numpy Array On-Demand for an LSTM
I found the answer to this on the Keras slack from user rocketknight. Use the model.fit_generator function. Define a generator function somewhere within your main python script that "yields" a batch of data. Then call this function in the arguments of the model.fit_generator function.
I am currently trying to use a "simple" LSTM network implemented through Keras for a summer project. Looking at the example code given, it appears the LSTM code wants a pre-generated 3D numpy array. As the dataset and the associated time interval I want to use are both rather large, it would be very prohibitive for me to load a "complete array" all at once. Is it possible to load the raw dataset and apply the sequencing transform to it as needed by the network (in this case construct the 3D array from x time-interval windows that then increment by 1 each time)? If so, how would you go about doing this?
Thanks for any help you can provide!