title: Keras fit generator slow
body: <p>Keras <a href="https://keras.io/models/sequential/" rel="noreferrer"><code>fit_generator</code></a> is very slow. The GPU is not used constantly in training, sometimes it's usage drops to 0%. Even on 4 workers and <code>multiproceesing=True</code>. </p>
  
  <p>Also, the processes of the script are requesting too much virtual memory and are with a D status, <a href="https://www.linuxquestions.org/questions/linux-general-1/process-status-d-44989/%7D" rel="noreferrer"><em>uninterruptible sleep (usually IO)</em></a>.</p>
  
  <p>I already tried different combinations of <code>max_queue_size</code> but it didn't work.</p>
  
  <p><a href="https://i.stack.imgur.com/HArOL.png" rel="noreferrer">Screenshot</a> GPU Usage</p>
  
  <p><a href="https://i.stack.imgur.com/XRlvP.png" rel="noreferrer">Screenshot</a> of Processes Virtual Memory and Status</p>
  
  <p><a href="https://i.stack.imgur.com/OBtRA.png" rel="noreferrer">Hardware Info</a>
  GPU = Titan XP 12Gb</p>
  
  <p>Code of Data Generator Class</p>
  
  <pre><code>import numpy as np
  import keras
  import conf
  
  
  class DataGenerator(keras.utils.Sequence):
      'Generates data for Keras'
      def __init__(self, list_IDs, labels, batch_size=32, dim=(conf.max_file, 128),
                   n_classes=10, shuffle=True):
          'Initialization'
          self.dim = dim
          self.batch_size = batch_size
          self.labels = labels
          self.list_IDs = list_IDs
          self.n_classes = n_classes
          self.shuffle = shuffle
          self.on_epoch_end()
  
      def __len__(self):
          'Denotes the number of batches per epoch'
          return int(np.floor(len(self.list_IDs) / self.batch_size))
  
      def __getitem__(self, index):
          'Generate one batch of data'
          # Generate indexes of the batch
          indexes = self.indexes[index*self.batch_size:(index+1)*self.batch_size]
  
          # Find list of IDs
          list_IDs_temp = [self.list_IDs[k] for k in indexes]
  
          # Generate data
          X, y = self.__data_generation(list_IDs_temp)
  
          return X, y
  
      def on_epoch_end(self):
          'Updates indexes after each epoch'
          self.indexes = np.arange(len(self.list_IDs))
          if self.shuffle == True:
              np.random.shuffle(self.indexes)
  
      def __data_generation(self, list_IDs_temp):
          'Generates data containing batch_size samples' # X : (n_samples, *dim, n_channels)
          # Initialization
          X = np.empty((self.batch_size, *self.dim))
          y = np.empty((self.batch_size, conf.max_file, self.n_classes))
          # Generate data
          for i, ID in enumerate(list_IDs_temp):
              # Store sample
              X[i, ] = np.load(conf.dir_out_data+"data_by_file/" + ID)
  
              # Store class
              y[i, ] = np.load(conf.dir_out_data +
                              'data_by_file/' + self.labels[ID])
  
          return X, y
  </code></pre>
  
  <p>Code of python script</p>
  
  <pre><code>training_generator = DataGenerator(partition['train'], labels, **params)
  
  validation_generator = DataGenerator(partition['validation'], labels, **params)
  
  model.fit_generator(generator = training_generator,
                      validation_data = validation_generator,
                      epochs=steps,
                      callbacks=[tensorboard, checkpoint],
                      workers=4,
                      use_multiprocessing=True,
                      max_queue_size=50)
  </code></pre>