title: using tensorflow dataset for custom image oversampling
body: <p>OK.  I want to try to set up a custom datasets work flow in tensorflow for custom image oversampling.  I have an unbalanced data set with many more normal images than fibrosis images.</p>
  <p>I start with some variable...</p>
  <pre><code>num_fibrosis = len(glob.glob(WORKING_DIR_TS +'NIH_1stPA_Norm_Fib/Fibrosis/*.png'))
  num_normal = len(glob.glob(WORKING_DIR_TS +'NIH_1stPA_Norm_Fib/No Finding/*.png'))
  perc_for_val =.2
  oversample_multiplier = 5
  num_fibrosis_in_val = int(num_fibrosis*perc_for_val)
  oversample_count = num_fibrosis_in_val*oversample_multiplier
  </code></pre>
  <p>And create a dataset base on the folder structure.  This dataset contains images and labels.</p>
  <pre><code>full_ds = tf.keras.utils.image_dataset_from_directory(
      'folder_path',
      image_size=(SIZE,SIZE),
      batch_size= None,
      # shuffle=False
  )
  </code></pre>
  <p>Then i take 20% of the fibrosis images and put them into our validation dataset.  I also put an equal number of normal images in val_ds.</p>
  <pre><code>fibrosis_ds = full_ds.filter(lambda x, y:  tf.equal(y, 0) ) # y == 0 for fibrosis
  normal_ds = full_ds.filter(lambda x, y:  tf.equal(y, 1) ) # y == 1 for normal
  # Let's take 20% of fibrosis images, and an equal number of normals, for our validation dataset
  val_ds = fibrosis_ds.take( num_fibrosis_in_val )
  val_ds = val_ds.concatenate( normal_ds.take( num_fibrosis_in_val ) )
  val_ds = val_ds.batch(BATCH_SIZE)
  </code></pre>
  <p>And lastly I make the training dataset.  I use skip so I don't repeat any of images that I used earlier.  I use repeat to oversample the fibrosis images. I add an equal number of normal images to make sure the data classes are balanced.  And at the end I shuffle.</p>
  <pre><code># Make the traing set
  train_ds = fibrosis_ds.skip(num_fibrosis_in_val).take(num_fibrosis - num_fibrosis_in_val)
  train_ds = train_ds.repeat( oversample_multiplier )
  train_ds = train_ds.concatenate( normal_ds.skip(num_fibrosis_in_val).take(oversample_count) )
  train_ds = train_ds.shuffle(oversample_count*2)
  train_ds = train_ds.batch(BATCH_SIZE)
  </code></pre>
  <p>This seems to work but in google colab it almost fills RAM just went I loop over val_ds to confirm the count.  Is it holding the entire dataset in memory while applying the chained functions on top?  Is there a more reasonable way to do this?</p>