Training Keras model with multiple CSV files I'm currently trying to train a Keras model on several large CSV files. I can fit one in memory, but not all combined. From my point of view, there are several ways to deal with this problem.
I could merge all of the data into a single CSV file and then read the CSV by the chunks of the data. This would simplify the process, however, I would not be able to shuffle the data each epoch.
Other approach that I can think of is to create a custom fit_generator. However I'm not sure how to actually implement this. Should I create a special generator for each dataset, and then cycle them? This would allow me to shuffle each dataset every epoch and even their order.
I believe that it would be more elegant to implement just a single generator, and manage all of the files inside. However to make this work, I would need to know the number of batches prior to training. That would require me to first get number of samples in each file, which would take a while. Also, I would need to sort out the issue of some lines being incorrect after reading the file with pandas. That limits me from using some fast os-level functions to get the number of lines.
Or is it possible to actually create a fit_generator without setting the number of batches prior to training? What approach would you recommend?
I would use tensorflow 2.0 with tf.data
import tensorflow as tf

filenames = [&quot;filename1&quot;, &quot;filename2&quot;, ...]

dataset = tf.data.Dataset.list_files(filenames, seed=42, shuffle=True)

# this reads 5 text files at a time, skips the first row of each file
dataset.interleave(lambda filename: tf.data.TextLineDataset(filename).skip(1), cycle_length=5, num_parallel_calls=tf.data.experimental.AUTOTUNE)

for line in dataset.take(5):
    print(line)

You can then pass in the dataset objects directly into tensorflow.keras for training.
There are some good examples here but please beware that this is NOT for tensorflow 2.0, and somethings may have changed