title: Transfer learning with Keras, validation accuracy does not improve from outset (beyond naive baseline) while train accuracy improves
body: <p>I am building a classifier for the Food-101 dataset (image dataset w/101 classes and 1k images per class). My approach has been to use Keras and transfer learning with the ResNet50 (weights from imagenet). </p>
  
  <p>When training the model, the train accuracy improves moderately well in a few epochs (30%-->45%), but the validation accuracy essentially stays at 0.9-1.0%. I have tried simplifying, swapping optimizers, reducing and increasing the units in the hidden layer, stripping out all image augmentation, and setting a consistent random seed on <code>flow_from_directory()</code>.</p>
  
  <p>When I look at the predictions made by the model on the validation set, it is always the same class.</p>
  
  <p>My sense it that the model is not overfitting so badly as to explain such a lack of movement in the validation accuracy. </p>
  
  <p><em>Any suggestions to get the validation accuracy to improve would be greatly appreciated.</em> </p>
  
  <p>For reference, below is the relevant code snippets:</p>
  
  <pre><code>datagen = ImageDataGenerator(rescale=1./255, validation_split=0.2)
  
  train_datagen = datagen.flow_from_directory('data/train/', seed=42, class_mode='categorical', subset='training', target_size=(256,256))
  # prints "60603 images belonging to 101 classes"
  val_datagen = datagen.flow_from_directory('data/train/', seed=42, class_mode='categorical', subset='validation', target_size=(256,256)) 
  # prints "15150 images belonging to 101 classes"
  
  train_steps = len(train_datagen) #1894
  val_steps = len(val_datagen) #474
  classes = len(list(train_datagen.class_indices.keys())) #101
  
  conv_base = ResNet50(weights='imagenet', include_top=False, pooling='avg', input_shape=(256, 256, 3))
  
  from keras.layers import GlobalAveragePooling2D
  from keras.layers import Dropout
  from keras.layers import Flatten
  from keras.layers import BatchNormalization
  
  model = Sequential()
  
  model.add(conv_base)
  model.add(BatchNormalization())
  model.add(Dropout(0.2))
  model.add(Dense(512, activation='relu'))
  model.add(Dropout(0.5))
  model.add(Dense(classes, activation='softmax'))
  
  conv_base.trainable = False
  
  from keras.optimizers import Adam
  
  model.compile(loss='categorical_crossentropy',
                optimizer=Adam(),
                metrics=['acc','top_k_categorical_accuracy'])
  
  history = model.fit_generator(
      train_datagen,
      steps_per_epoch=train_steps,
      epochs=5,
      verbose=2,
      validation_data=val_datagen,
      validation_steps=val_steps
  )
  </code></pre>
  
  <p>Here is the results of the <code>.fit_generator()</code>:</p>
  
  <ul>
  <li>Epoch 1/5
  
  <ul>
  <li>724s - loss: 3.1305 - acc: 0.3059 - top_k_categorical_accuracy: 0.5629 - val_loss: 6.5914  val_acc: 0.0099 - val_top_k_categorical_accuracy: 0.0494 </li>
  </ul></li>
  <li>Epoch 2/5
  
  <ul>
  <li>715s - loss: 2.4812 - acc: 0.4021 - top_k_categorical_accuracy: 0.6785 - val_loss: 7.4093 - val_acc: 0.0099 - val_top_k_categorical_accuracy: 0.0495 </li>
  </ul></li>
  <li>Epoch 3/5
  
  <ul>
  <li>714s - loss: 2.3559 - acc: 0.4248 - top_k_categorical_accuracy: 0.7026 - val_loss: 8.9146 - val_acc: 0.0094 - val_top_k_categorical_accuracy: 0.0495 </li>
  </ul></li>
  <li>Epoch 4/5
  
  <ul>
  <li>714s - loss: 2.2661 - acc: 0.4459 - top_k_categorical_accuracy: 0.7200 - val_loss: 8.0597 - val_acc: 0.0100 - val_top_k_categorical_accuracy: 0.0494 </li>
  </ul></li>
  <li>Epoch 5/5
  
  <ul>
  <li>715s - loss: 2.1870 - acc: 0.4583 - top_k_categorical_accuracy: 0.7348 - val_loss: 7.5171 - val_acc: 0.0100 - val_top_k_categorical_accuracy: 0.0483</li>
  </ul></li>
  </ul>
  
  <p>Here is the <code>model.summary()</code>:</p>
  
  <pre><code>Layer (type)                 Output Shape              Param #   
  =================================================================
  resnet50 (Model)             (None, 2048)              23587712  
  _________________________________________________________________
  batch_normalization_1 (Batch (None, 2048)              8192      
  _________________________________________________________________
  dropout_1 (Dropout)          (None, 2048)              0         
  _________________________________________________________________
  dense_1 (Dense)              (None, 512)               1049088   
  _________________________________________________________________
  dropout_2 (Dropout)          (None, 512)               0         
  _________________________________________________________________
  dense_2 (Dense)              (None, 101)               51813     
  =================================================================
  Total params: 24,696,805
  Trainable params: 1,104,997
  Non-trainable params: 23,591,808
  _________________________________________________________________
  </code></pre>