Loading large numpy array (DAIC-WOZ) for LSTM model causes Out of memory errors I am working on using the DAIC-WOZ Dataset of audio and video extracts for depression detection. While training the audio files, which has three input of eGeMAPS, MFCC, and DENSENET arrays. They are input into a BLSTM model.  I am using an AzureML server GPU-N6 for training.
machine specs : STANDARD_NV6 (6 Cores, 56 GB RAM, 380 GB Disk) GPU - 1 x NVIDIA Tesla M60
It exits with code 137, which is an out of memory error. I see the data is downloaded into the local node and NumPy array is created. the OOM happens when the batch processing starts.
I have tried to run the model with only one input (eGeMaps) but run into memory issues.
My question is, is there a way to batch load the NumPy file, similar to an image dataset? I have tried to split the input into individual data files and use tensor dataset or slices, but it doesn't work when loading a NumPy file. Ideally, I would like an equivalent of ImageDataGenerator for NumPy files.
EGEMAPS={'FEATURES':100,'TIMESTEPS':14050}
MFCC={'FEATURES':100,'TIMESTEPS':14050}
DENSENET201={'FEATURES':1920,'TIMESTEPS':1415}
The training dataset consists of 136 participants each with an array of the sizes mentioned above. I created a single large NumPy array, each row representing a participant and array of the features.
This is the model I am training.
#Create Model :
inputs_eGeMAPS = Input(EGEMAPS_INPUT_SHAPE)
inputs_MFCC = Input(MFCC_INPUT_SHAPE)
inputs_Densenet = Input(DENSENET201_INPUT_SHAPE)

blstm1_eGeMAPS = Bidirectional(LSTM(200,return_sequences=True,recurrent_dropout =0.2,activation='relu'))(inputs_eGeMAPS)
print(f&quot;blstm1_eGeMAPS:{blstm1_eGeMAPS}&quot;)

blstm2_eGeMAPS= Bidirectional(LSTM(200,recurrent_dropout =0.2,activation='relu'))(blstm1_eGeMAPS)
print(f&quot;blstm1_eGeMAPS:{blstm1_eGeMAPS}&quot;)

blstm1_MFCC = Bidirectional(LSTM(200,return_sequences=True,recurrent_dropout =0.2,activation='relu'))(inputs_MFCC)
print(f&quot;blstm1_MFCC:{blstm1_MFCC}&quot;)

blstm2_MFCC= Bidirectional(LSTM(200,recurrent_dropout =0.2,activation='relu'))(blstm1_MFCC)
print(f&quot;blstm1_MFCC:{blstm1_MFCC}&quot;)

blstm1_Densenet = Bidirectional(LSTM(200,return_sequences=True,recurrent_dropout =0.2,activation='relu'))(inputs_Densenet)
print(f&quot;blstm1_Densenet:{blstm1_Densenet}&quot;)

blstm2_Densenet= Bidirectional(LSTM(200,recurrent_dropout =0.2,activation='relu'))(blstm1_Densenet)
print(f&quot;blstm1_Densenet:{blstm1_Densenet}&quot;)

audio_lstm_output = concatenate([blstm2_eGeMAPS,blstm2_MFCC,blstm2_Densenet])

dense1= Dense(500,activation='relu')(audio_lstm_output)

print(f&quot;dense:{dense1}&quot;)

dense2= Dense(100,activation='relu')(dense1)
print(f&quot;dense:{dense2}&quot;)

dense3= Dense(60,activation='relu')(dense2)
print(f&quot;dense:{dense3}&quot;)

output = Dense(1)(dense3)
print(f&quot;output:{output}&quot;)

model = Model(inputs=[inputs_eGeMAPS,inputs_MFCC,inputs_Densenet], outputs=output)
model.compile(optimizer='adam',
              loss='mean_squared_error',
              metrics=['accuracy'])

print(model.summary())

epochs = 1
hist = model.fit([BoAW_eGeMAPS_Dev_tf,BoAW_MFCC_Dev_tf,Densenet_Dev_tf],Y_train_tf,epochs=epochs, batch_size=batch_size,validation_split=0.2,shuffle=False)
Looking at your code, it looks like you're using Keras w/ a Tensorflow backend -- or Tensorflow functions using its Keras submodule. In that case, you can train your model using its fit_generator function.
As the name implies, if you convert your training data into a generator, it will substantially reduce the memory cost needed to train your model. Here's an example of training a Keras model using fit_generator and how you can preprocess your data to utilize it.