title: How to select strategy to reduce overfitting?
body: <p>I am applying transfer-learning on a pre-trained network with keras. I have image patches with a binary class label and would like to use CNN to predict a class label in the range [0; 1] for unseen image patches.</p>
  
  <ul>
  <li><strong>network</strong>: ResNet50 pre-trained with imageNet to which I add 3 layers</li>
  <li><strong>data</strong>: 70305 training samples, 8000 validation samples, 66823 testing samples, all with a balanced number of both class labels</li>
  <li><strong>images</strong>: 3 bands (RGB) and 224x224 pixels</li>
  <li><p><strong>set-up</strong>: 32 batches, size of conv. layer: 16</p></li>
  <li><p><strong>result</strong>: after a few epochs, I already have an accuracy of almost 1 and a loss close to 0, while on the validation data the accuracy remains at 0.5 and loss varies per epoch. In the end, the CNN predicts only one class for all unseen patches. </p></li>
  <li><strong>problem</strong>: it seems like my network is overfitting. </li>
  </ul>
  
  <p><a href="https://i.stack.imgur.com/iIDip.png" rel="noreferrer"><img src="https://i.stack.imgur.com/iIDip.png" alt="results screenshot"></a>
  <a href="https://i.stack.imgur.com/JctTs.jpg" rel="noreferrer"><img src="https://i.stack.imgur.com/JctTs.jpg" alt="enter image description here"></a></p>
  
  <p>The following strategies could reduce overfitting:</p>
  
  <ul>
  <li>increase batch size</li>
  <li>decrease size of fully-connected layer</li>
  <li>add drop-out layer</li>
  <li>add data augmentation</li>
  <li>apply regularization by modifying the loss function</li>
  <li>unfreeze more pre-trained layers</li>
  <li>use different network architecture</li>
  </ul>
  
  <p>I have tried batch sizes up to 512 and changed the size of fully-connected layer without much success. Before just randomly testing the rest, <strong>I would like to ask how to investigate what goes wrong why in order to find out which of the above strategies has most potential</strong>.</p>
  
  <p>Below my code:</p>
  
  <pre><code>def generate_data(imagePathTraining, imagesize, nBatches):
      datagen = ImageDataGenerator(rescale=1./255)
      generator = datagen.flow_from_directory\
          (directory=imagePathTraining,                           # path to the target directory
           target_size=(imagesize,imagesize),                     # dimensions to which all images found will be resize
           color_mode='rgb',                                      # whether the images will be converted to have 1, 3, or 4 channels
           classes=None,                                          # optional list of class subdirectories
           class_mode='categorical',                              # type of label arrays that are returned
           batch_size=nBatches,                                   # size of the batches of data
           shuffle=True)                                          # whether to shuffle the data
      return generator
  
  def create_model(imagesize, nBands, nClasses):
      print("%s: Creating the model..." % datetime.now().strftime('%Y-%m-%d_%H-%M-%S'))
      # Create pre-trained base model
      basemodel = ResNet50(include_top=False,                     # exclude final pooling and fully connected layer in the original model
                           weights='imagenet',                    # pre-training on ImageNet
                           input_tensor=None,                     # optional tensor to use as image input for the model
                           input_shape=(imagesize,                # shape tuple
                                        imagesize,
                                        nBands),
                           pooling=None,                          # output of the model will be the 4D tensor output of the last convolutional layer
                           classes=nClasses)                      # number of classes to classify images into
      print("%s: Base model created with %i layers and %i parameters." %
            (datetime.now().strftime('%Y-%m-%d_%H-%M-%S'),
             len(basemodel.layers),
             basemodel.count_params()))
  
      # Create new untrained layers
      x = basemodel.output
      x = GlobalAveragePooling2D()(x)                             # global spatial average pooling layer
      x = Dense(16, activation='relu')(x)                         # fully-connected layer
      y = Dense(nClasses, activation='softmax')(x)                # logistic layer making sure that probabilities sum up to 1
  
      # Create model combining pre-trained base model and new untrained layers
      model = Model(inputs=basemodel.input,
                    outputs=y)
      print("%s: New model created with %i layers and %i parameters." %
            (datetime.now().strftime('%Y-%m-%d_%H-%M-%S'),
             len(model.layers),
             model.count_params()))
  
      # Freeze weights on pre-trained layers
      for layer in basemodel.layers:
          layer.trainable = False
  
      # Define learning optimizer
      optimizerSGD = optimizers.SGD(lr=0.01,                      # learning rate.
                                    momentum=0.0,                 # parameter that accelerates SGD in the relevant direction and dampens oscillations
                                    decay=0.0,                    # learning rate decay over each update
                                    nesterov=False)               # whether to apply Nesterov momentum
  
      # Compile model
      model.compile(optimizer=optimizerSGD,                       # stochastic gradient descent optimizer
                    loss='categorical_crossentropy',              # objective function
                    metrics=['accuracy'],                         # metrics to be evaluated by the model during training and testing
                    loss_weights=None,                            # scalar coefficients to weight the loss contributions of different model outputs
                    sample_weight_mode=None,                      # sample-wise weights
                    weighted_metrics=None,                        # metrics to be evaluated and weighted by sample_weight or class_weight during training and testing
                    target_tensors=None)                          # tensor model's target, which will be fed with the target data during training
      print("%s: Model compiled." % datetime.now().strftime('%Y-%m-%d_%H-%M-%S'))
      return model
  
  def train_model(model, nBatches, nEpochs, imagePathTraining, imagesize, nSamples, valX,valY, resultPath):
      history = model.fit_generator(generator=generate_data(imagePathTraining, imagesize, nBatches),
                                    steps_per_epoch=nSamples//nBatches,     # total number of steps (batches of samples)
                                    epochs=nEpochs,               # number of epochs to train the model
                                    verbose=2,                    # verbosity mode. 0 = silent, 1 = progress bar, 2 = one line per epoch
                                    callbacks=None,               # keras.callbacks.Callback instances to apply during training
                                    validation_data=(valX,valY),  # generator or tuple on which to evaluate the loss and any model metrics at the end of each epoch
                                    class_weight=None,            # optional dictionary mapping class indices (integers) to a weight (float) value, used for weighting the loss function
                                    max_queue_size=10,            # maximum size for the generator queue
                                    workers=32,                   # maximum number of processes to spin up when using process-based threading
                                    use_multiprocessing=True,     # whether to use process-based threading
                                    shuffle=True,                 # whether to shuffle the order of the batches at the beginning of each epoch
                                    initial_epoch=0)              # epoch at which to start training
      print("%s: Model trained." % datetime.now().strftime('%Y-%m-%d_%H-%M-%S')) 
      return history
  </code></pre>