title: bad performance using tf dataset cache in an active learning framework
body: <p>I am trying to build an input pipeline for an active learning application.
  The script I wrote is loading in the data, augmenting and then caching it.</p>
  
  <p>I cache the whole dataset once and then split it up in an unlabeled and a labeled dataset, from where on I transfer samples from one dataset to the other in every iteration.</p>
  
  <p>After a few iteration the training gets extremely slow. The training time per sample is quadrupling after about 20 iterations, which is unacceptable for my model. I can see via tensorboard and the used computer ressources that lots of the time the training is idling and GPU is not even used but the CPU and Sysmem usage are really high.</p>
  
  <p>Has anybody had the same problem and can tell me how to fix it? Or any detailled information about how the tensorflow caching is working?</p>
  
  <p>Thanks!</p>
  
  <p>Edit: I split up the dataset using .take() and .skip() and transfer samples using .concatenate()</p>
  
  <p>Edit: these are the three methods I use, they are simplified though. <em>indexlist</em> is the list of samples that are to be transfered from unlabeled to labeled dataset</p>
  
  <pre><code>def input_pipeline()
          dataset  = tf.data.TFRecordDataset(filenames)
          dataset = dataset.map(__convert_tf_records, num_parallel_calls=1)
          dataset_bounding_boxes = dataset.map(get_lung_bb,num_parallel_calls=1)
  
          if not is_training:
              datasets['bounding_box'] = dataset_bounding_boxes
  
          if params.repeat &gt; 1 and is_training:
              dataset = dataset.repeat(count=params.repeat)
              dataset_bounding_boxes = dataset_bounding_boxes.repeat(count=params.repeat)
  
          if is_training:
                      dataset = dataset.map(data_aug_rot, num_parallel_calls=1)
                      dataset = dataset.map(data_aug_zoom, num_parallel_calls=1)
                      dataset = dataset.map(data_aug_elastic_deform, num_parallel_calls=1)
  
          dataset = tf.data.Dataset.zip((dataset, dataset_bounding_boxes))
  
          dataset = dataset.map(get_random_patches_fn, num_parallel_calls=1)
          dataset = dataset.unbatch()
  
          if params.b_shuffle and is_training:
              dataset = dataset.shuffle(buffer_size=params.batch_size*5)
          dataset = dataset.batch(params.batch_size)
  
          if params.b_cache and is_training:
              cache_path = os.path.join(experiment_dir, "cache")
              if not os.path.exists(cache_path):
                  os.makedirs(cache_path)
              dataset = dataset.cache(cache_path + '/')
          dataset = dataset.prefetch(buffer_size=params.batch_size)
  
          datasets['patches'] = dataset
          return datasets
  
  
  
  def get_trainingdatasets():
  
      initial_dataset = input_pipeline()
      dataset_all=initial_dataset['patches']
      dataset_all=dataset_all.unbatch()
  
      for element in dataset_all:
          dataset_size+=1
  
      dataset_unlabeled=dataset_all.skip( int(dataset_size*args.ds_labeled_perc) ).batch(params.batch_size)
      dataset_labeled=dataset_all.take( int(dataset_size* args.ds_labeled_perc) )
      dataset_labeled= dataset_labeled.shuffle(buffer_size=500, reshuffle_each_iteration=True).batch(params.batch_size)
  
      return dataset_labeled, dataset_unlabeled
  
  def transfer_samples():
  
      dataset_unlabeled = dataset_unlabeled.unbatch()
      dataset_labeled = dataset_labeled.unbatch()
  
      for idx in index_list:
          element_add=dataset_unlabeled.skip(idx).take(1)
          dataset_add_to_labeled =   dataset_add_to_labeled.concatenate(element_add)
  
      dataset_labeled = dataset_labeled.concatenate(dataset_add_to_labeled)
      dataset_labeled= dataset_labeled.shuffle(buffer_size=params.batch_size*5)
  
      dataset_labeled = dataset_labeled.batch(params.batch_size).prefetch(buffer_size=params.batch_size)
      dataset_unlabeled = dataset_unlabeled.batch(params.batch_size).prefetch(buffer_size=params.batch_size)
  
      return dataset_labeled, dataset_unlabeled
  </code></pre>