title: Tensorflow/Keras: volatile validation loss
body: <p>I've been training a U-Net for single class small lesion segmentation, and have been getting consistently volatile validation loss. I have about 20k images split 70/30 between training and validation sets-so I don't think the issue is too little data. I've tried shuffling and resplitting the sets a few times with no change in volatility-so I don't think the validation set is unrepresentative. I have tried lowering the learning rate with no effect on volatility. And I have tried a few loss functions (dice coefficient, focal tversky, weighted binary cross-entropy). I'm using a decent amount of augmentation so as to avoid overfitting. I've also run through all my data (512x512 float64s with corresponding 512x512 int64 masks--both stored as numpy arrays) do double check that the value range, dtypes, etc. aren't screwy...and I even removed any ROIs in the masks under 35 pixels in area which I thought might be artifact and messing with loss. </p>
  
  <p>I'm using keras ImageDataGen.flow_from_directory...I was initially using zca_whitening and brightness_range augmentation but I think this causes issues with flow_from_directory and the link between mask and image being lost.. so I skipped this.  </p>
  
  <p>I've tried validation generators with and without shuffle=True. Batch size is 8.</p>
  
  <p>Here's some of my code, happy to include more if it would help:</p>
  
  <pre><code># loss 
  
  from keras.losses import binary_crossentropy
  import keras.backend as K
  import tensorflow as tf 
  
  epsilon = 1e-5
  smooth = 1
  
  def dsc(y_true, y_pred):
      smooth = 1.
      y_true_f = K.flatten(y_true)
      y_pred_f = K.flatten(y_pred)
      intersection = K.sum(y_true_f * y_pred_f)
      score = (2. * intersection + smooth) / (K.sum(y_true_f) + K.sum(y_pred_f) + smooth)
      return score
  
  def dice_loss(y_true, y_pred):
      loss = 1 - dsc(y_true, y_pred)
      return loss
  
  def bce_dice_loss(y_true, y_pred):
      loss = binary_crossentropy(y_true, y_pred) + dice_loss(y_true, y_pred)
      return loss
  
  def confusion(y_true, y_pred):
      smooth=1
      y_pred_pos = K.clip(y_pred, 0, 1)
      y_pred_neg = 1 - y_pred_pos
      y_pos = K.clip(y_true, 0, 1)
      y_neg = 1 - y_pos
      tp = K.sum(y_pos * y_pred_pos)
      fp = K.sum(y_neg * y_pred_pos)
      fn = K.sum(y_pos * y_pred_neg) 
      prec = (tp + smooth)/(tp+fp+smooth)
      recall = (tp+smooth)/(tp+fn+smooth)
      return prec, recall
  
  def tp(y_true, y_pred):
      smooth = 1
      y_pred_pos = K.round(K.clip(y_pred, 0, 1))
      y_pos = K.round(K.clip(y_true, 0, 1))
      tp = (K.sum(y_pos * y_pred_pos) + smooth)/ (K.sum(y_pos) + smooth) 
      return tp 
  
  def tn(y_true, y_pred):
      smooth = 1
      y_pred_pos = K.round(K.clip(y_pred, 0, 1))
      y_pred_neg = 1 - y_pred_pos
      y_pos = K.round(K.clip(y_true, 0, 1))
      y_neg = 1 - y_pos 
      tn = (K.sum(y_neg * y_pred_neg) + smooth) / (K.sum(y_neg) + smooth )
      return tn 
  
  def tversky(y_true, y_pred):
      y_true_pos = K.flatten(y_true)
      y_pred_pos = K.flatten(y_pred)
      true_pos = K.sum(y_true_pos * y_pred_pos)
      false_neg = K.sum(y_true_pos * (1-y_pred_pos))
      false_pos = K.sum((1-y_true_pos)*y_pred_pos)
      alpha = 0.7
      return (true_pos + smooth)/(true_pos + alpha*false_neg + (1-alpha)*false_pos + smooth)
  
  def tversky_loss(y_true, y_pred):
      return 1 - tversky(y_true,y_pred)
  
  def focal_tversky(y_true,y_pred):
      pt_1 = tversky(y_true, y_pred)
      gamma = 0.75
      return K.pow((1-pt_1), gamma)
  
  
  
      model = BlockModel((len(os.listdir(os.path.join(imageroot,'train_ct','train'))), 512, 512, 1),filt_num=16,numBlocks=4)
      #model.compile(optimizer=Adam(learning_rate=0.001), loss=weighted_cross_entropy)
      #model.compile(optimizer=Adam(learning_rate=0.001), loss=dice_coef_loss)
      model.compile(optimizer=Adam(learning_rate=0.001), loss=focal_tversky)
      train_mask = os.path.join(imageroot,'train_masks')
      val_mask = os.path.join(imageroot,'val_masks')
  
      model.load_weights(model_weights_path) #I'm initializing with some pre-trained weights from a similar model
  
       data_gen_args_mask = dict(
          rotation_range=10,
          shear_range=20,
          width_shift_range=0.1, 
          height_shift_range=0.1,
          zoom_range=[0.8,1.2],
          horizontal_flip=True,
          #vertical_flip=True,
          fill_mode='nearest',
          data_format='channels_last'
          )
  
      data_gen_args = dict(
          **data_gen_args_mask
      )
  
      image_datagen_train = ImageDataGenerator(**data_gen_args)
      mask_datagen_train = ImageDataGenerator(**data_gen_args)#_mask)
  
      image_datagen_val = ImageDataGenerator()
      mask_datagen_val = ImageDataGenerator()
  
      seed = 1
      BS = 8
  
      steps = int(np.floor((len(os.listdir(os.path.join(train_ct,'train'))))/BS))
      print(steps)
      val_steps = int(np.floor((len(os.listdir(os.path.join(val_ct,'val'))))/BS))
      print(val_steps)
  
      train_image_generator = image_datagen_train.flow_from_directory(
           train_ct,
           target_size = (512, 512),
           color_mode = ("grayscale"),
           classes=None,
           class_mode=None,
           seed = seed,
           shuffle = True,
           batch_size = BS)
  
      train_mask_generator = mask_datagen_train.flow_from_directory(
          train_mask,
          target_size = (512, 512),
          color_mode = ("grayscale"),
          classes=None,
          class_mode=None,
          seed = seed,
          shuffle = True,
          batch_size = BS)
  
      val_image_generator = image_datagen_val.flow_from_directory(
          val_ct,
          target_size = (512, 512),
          color_mode = ("grayscale"),
          classes=None,
          class_mode=None,
          seed = seed,
          shuffle = True,
          batch_size = BS)
  
      val_mask_generator = mask_datagen_val.flow_from_directory(
          val_mask,
          target_size = (512, 512),
          color_mode = ("grayscale"),
          classes=None,
          class_mode=None,
          seed = seed,
          shuffle = True,
          batch_size = BS)
  
      train_generator = zip(train_image_generator, train_mask_generator)
      val_generator = zip(val_image_generator, val_mask_generator)
  
  # make callback for checkpointing
  
      plot_losses = PlotLossesCallback(skip_first=0,plot_extrema=False)
      %matplotlib inline
  
      filepath = os.path.join(versionPath, model_version + "_saved-model-{epoch:02d}-{val_loss:.2f}.hdf5")
  
      if reduce:
          cb_check = [ModelCheckpoint(filepath,monitor='val_loss',
                          verbose=1,save_best_only=False,
                          save_weights_only=True,mode='auto',period=1),
                          reduce_lr,
                          plot_losses]
      else:
          cb_check = [ModelCheckpoint(filepath,monitor='val_loss',
                          verbose=1,save_best_only=False,
                          save_weights_only=True,mode='auto',period=1),
                          plot_losses]
  
  # train model
      history = model.fit_generator(train_generator, epochs=numEp,
                      steps_per_epoch=steps,
                      validation_data=val_generator,
                      validation_steps=val_steps, 
                      verbose=1,
                      callbacks=cb_check,
                      use_multiprocessing = False
                               )
  </code></pre>
  
  <p>And here's how my loss looks:</p>
  
  <p><a href="https://i.stack.imgur.com/JRCmi.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/JRCmi.png" alt="enter image description here"></a></p>
  
  <p>Another potentially relevant thing: I tweaked the flow_from_directory code a bit (added npy to the white list). But training loss looks fine so assuming the issue isnt here</p>