How to store and load training data comprised 50 millions 25x25 numpy arrays while training a multi-class CNN model?
I have an image processing problem where there are five classes, each class has approximately 10 millions examples as training data where an image is a z-scored 25x25 numpy array. 

Obviously, I can’t load all the training data into memory, so I have to use fit_generator. 

I also the one who generates and augments these training data matrices, but I can’t do it in real time within fit_generator because it will be too slow to train the model. 

First, how to store 50 millions 25x25 .npy arrays on disk? What would be the best practice? 

Second, Should I use a database to store these matrices and to query from it during training? I don’t think SQLite supports multi threads, and SQL datasets support is still experimental in tensorflow. 

I would love to know if there is a neat way to store these 50 million matrices, so the retrieval during training will be optimal.

Third, what about using HDF5 format? Should I switch to pytorch instead?

Here is some code I found on medium (can't find the original post).
This will help to generate training data on-the-fly in a producer-consumer fashion:
import tensorflow as tf
import numpy as np

from time import sleep

class DataGen():
    counter = 0

    def __init__(self):
        self.gen_num = DataGen.counter
        DataGen.counter += 1

    def py_gen(self, gen_name):
        gen_name = gen_name.decode('utf8') + '_' + str(self.gen_num)
        for num in range(10):
            sleep(0.3)
            yield '{} yields {}'.format(gen_name, num)

Dataset = tf.data.Dataset
dummy_ds = Dataset.from_tensor_slices(['Gen1', 'Gen2', 'Gen3'])
dummy_ds = dummy_ds.interleave(lambda x: Dataset.from_generator(DataGen().py_gen, output_types=(tf.string), args=(x,)),
                               cycle_length=5,
                               block_length=2,
                               num_parallel_calls=5)
data_tf = dummy_ds.as_numpy_iterator()
for d in data_tf:
  print(d)

Output:
b'Gen1_0 yields 0'
b'Gen1_0 yields 1'
b'Gen2_0 yields 0'
b'Gen2_0 yields 1'
b'Gen3_0 yields 0'
b'Gen3_0 yields 1'
b'Gen1_0 yields 2'
b'Gen1_0 yields 3'
b'Gen2_0 yields 2'
b'Gen2_0 yields 3'
b'Gen3_0 yields 2'
b'Gen3_0 yields 3'
b'Gen1_0 yields 4'
b'Gen1_0 yields 5'
b'Gen2_0 yields 4'
b'Gen2_0 yields 5'
b'Gen3_0 yields 4'
b'Gen3_0 yields 5'
b'Gen1_0 yields 6'
b'Gen1_0 yields 7'
b'Gen2_0 yields 6'
b'Gen2_0 yields 7'
b'Gen3_0 yields 6'
b'Gen3_0 yields 7'
b'Gen1_0 yields 8'
b'Gen1_0 yields 9'
b'Gen2_0 yields 8'
b'Gen2_0 yields 9'
b'Gen3_0 yields 8'
b'Gen3_0 yields 9'

