title: Validation accuracy stagnates while training accuracy improves
body: <p>I'm pretty new to deep learning so I'm sorry if I'm missing something obvious.</p>
  
  <p>I am currently training a CNN with a dataset I put together.
  When training, the training accuracy behaves pretty normal and improves, reaching >99% accuracy. My validation accuracy starts off at about 75% and fluctuates around 81% ± 1%. After training, the model performs really well on completely new data.</p>
  
  <p><a href="https://i.stack.imgur.com/8rihZ.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/8rihZ.png" alt="model accuracy"></a>
  <a href="https://i.stack.imgur.com/3TEFz.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/3TEFz.png" alt="model loss"></a></p>
  
  <pre><code>Epoch 1/100
  187/187 [==============================] - 103s 550ms/step - loss: 1.1336 - acc: 0.5384 - val_loss: 0.8065 - val_acc: 0.7405
  Epoch 2/100
  187/187 [==============================] - 97s 519ms/step - loss: 0.8041 - acc: 0.7345 - val_loss: 0.7566 - val_acc: 0.7720
  Epoch 3/100
  187/187 [==============================] - 97s 519ms/step - loss: 0.7194 - acc: 0.7945 - val_loss: 0.7410 - val_acc: 0.7846
  Epoch 4/100
  187/187 [==============================] - 97s 517ms/step - loss: 0.6688 - acc: 0.8324 - val_loss: 0.7295 - val_acc: 0.7924
  Epoch 5/100
  187/187 [==============================] - 97s 518ms/step - loss: 0.6288 - acc: 0.8611 - val_loss: 0.7197 - val_acc: 0.7961
  Epoch 6/100
  187/187 [==============================] - 96s 515ms/step - loss: 0.5989 - acc: 0.8862 - val_loss: 0.7252 - val_acc: 0.7961
  Epoch 7/100
  187/187 [==============================] - 96s 514ms/step - loss: 0.5762 - acc: 0.8981 - val_loss: 0.7135 - val_acc: 0.8063
  Epoch 8/100
  187/187 [==============================] - 97s 518ms/step - loss: 0.5513 - acc: 0.9186 - val_loss: 0.7089 - val_acc: 0.8077
  Epoch 9/100
  187/187 [==============================] - 96s 513ms/step - loss: 0.5351 - acc: 0.9280 - val_loss: 0.7113 - val_acc: 0.8053
  Epoch 10/100
  187/187 [==============================] - 96s 514ms/step - loss: 0.5189 - acc: 0.9417 - val_loss: 0.7167 - val_acc: 0.8094
  Epoch 11/100
  187/187 [==============================] - 96s 515ms/step - loss: 0.5026 - acc: 0.9483 - val_loss: 0.7104 - val_acc: 0.8162
  Epoch 12/100
  187/187 [==============================] - 96s 516ms/step - loss: 0.4914 - acc: 0.9538 - val_loss: 0.7114 - val_acc: 0.8101
  Epoch 13/100
  187/187 [==============================] - 96s 515ms/step - loss: 0.4809 - acc: 0.9583 - val_loss: 0.7099 - val_acc: 0.8141
  Epoch 14/100
  187/187 [==============================] - 96s 512ms/step - loss: 0.4681 - acc: 0.9656 - val_loss: 0.7149 - val_acc: 0.8182
  Epoch 15/100
  187/187 [==============================] - 96s 515ms/step - loss: 0.4605 - acc: 0.9701 - val_loss: 0.7139 - val_acc: 0.8172
  Epoch 16/100
  187/187 [==============================] - 96s 514ms/step - loss: 0.4479 - acc: 0.9753 - val_loss: 0.7102 - val_acc: 0.8182
  Epoch 17/100
  187/187 [==============================] - 96s 513ms/step - loss: 0.4418 - acc: 0.9805 - val_loss: 0.7087 - val_acc: 0.8247
  Epoch 18/100
  187/187 [==============================] - 96s 512ms/step - loss: 0.4363 - acc: 0.9809 - val_loss: 0.7148 - val_acc: 0.8213
  Epoch 19/100
  187/187 [==============================] - 96s 516ms/step - loss: 0.4225 - acc: 0.9870 - val_loss: 0.7184 - val_acc: 0.8203
  Epoch 20/100
  187/187 [==============================] - 96s 513ms/step - loss: 0.4241 - acc: 0.9863 - val_loss: 0.7216 - val_acc: 0.8189
  Epoch 21/100
  187/187 [==============================] - 96s 513ms/step - loss: 0.4132 - acc: 0.9908 - val_loss: 0.7143 - val_acc: 0.8199
  Epoch 22/100
  187/187 [==============================] - 96s 515ms/step - loss: 0.4050 - acc: 0.9936 - val_loss: 0.7109 - val_acc: 0.8233
  Epoch 23/100
  187/187 [==============================] - 96s 515ms/step - loss: 0.4040 - acc: 0.9928 - val_loss: 0.7118 - val_acc: 0.8203
  Epoch 24/100
  187/187 [==============================] - 96s 511ms/step - loss: 0.3989 - acc: 0.9930 - val_loss: 0.7194 - val_acc: 0.8165
  Epoch 25/100
  187/187 [==============================] - 97s 517ms/step - loss: 0.3933 - acc: 0.9946 - val_loss: 0.7163 - val_acc: 0.8155
  Epoch 26/100
  187/187 [==============================] - 97s 516ms/step - loss: 0.3884 - acc: 0.9957 - val_loss: 0.7225 - val_acc: 0.8148
  Epoch 27/100
  187/187 [==============================] - 95s 510ms/step - loss: 0.3876 - acc: 0.9959 - val_loss: 0.7224 - val_acc: 0.8179
  </code></pre>
  
  <p>The plot in itself looks like overfitting, but I've taken plenty of measures to fix overfitting but none seem to work. Here is my model:</p>
  
  <pre><code># transfer learning with ResNet50
  base_model=ResNet50(weights='imagenet', include_top=False, input_shape=(224, 224, 3))
  
  # function to finetune model
  def build_finetune_model(base_model, dropout, fc_layers, num_classes):
      # make base model untrainable
      for layer in base_model.layers:
          layer.trainable = False
  
      x = base_model.output
      x = Flatten()(x)
  
      # add dense layers
      for fc in fc_layers:
          # use regularizer
          x = Dense(fc, use_bias=False, kernel_regularizer=l2(0.003))(x)
          # add batch normalization
          x = BatchNormalization()(x)
          x = Activation('relu')(x)
          # add dropout
          x = Dropout(dropout)(x)
  
      # New softmax layer
      x = Dense(num_classes, use_bias=False)(x) 
      x = BatchNormalization()(x)
      predictions = Activation('softmax')(x)
  
      finetune_model = Model(inputs=base_model.input, outputs=predictions)
      return finetune_model
  
  
  FC_LAYERS = [1024, 1024]
  dropout = 0.5
  
  model = build_finetune_model(base_model, dropout=dropout, fc_layers=FC_LAYERS,num_classes=len(categories))
  </code></pre>
  
  <p>I'm adjusting for class weights and have set a really low learning rate in hopes of slowing the learning down.</p>
  
  <pre><code>model.compile(optimizer=Adam(lr=0.000005),loss='categorical_crossentropy',metrics=['accuracy'], weighted_metrics=class_weight)
  </code></pre>
  
  <p>I'm really confused by the fact that the validation accuracy starts so high (significantly higher than training accuracy) and barely improves during the entire training process. As mentioned before it seems to be overfitting but I added dropouts, batch normalization and regularizers, it doesn't seem to work. Augmenting data with Horizontal flips, random cropping, random brightness and rotation does not change the accuracy significantly either. Turning shuffle off for my data inside <code>ImageDataGenerator().flow_from_directory()</code> for my training data makes the model train around 25% for training accuracy and &lt;50% for validation accuracy (Edit: accuracy seems to be so low because the learning rate was too low in that case).</p>
  
  <p>Again, the model works surprisingly well on new testing data. I'm looking to increase the validation accuracy and want to understand why the neural network is behaving that way.</p>