title: ResNet always predicting one class
body: <p>I am trying to do transfer learning in Keras + Tensorflow on a selected subset of Places-205 dataset, containing only 27 categories. I am using InceptionV3, DenseNet121 and ResNet50, pre-trained on ImageNet, and add a couple of extra layers to adapt to my classes. If the model is ResNet, I add Flatten + Dense for classfication, and if it is DenseNet or Inceptionv3, I add Global Avg Pool + Dense (relu) + Dense (classification). 
  This is the code snippet:</p>
  
  <pre><code>x = base_model.output
  if FLAGS.model in 'resnet50':
      x = Flatten(name="flatten")(x)
  else:
      x = GlobalAveragePooling2D()(x)
      # Let's add a fully-connected layer
      x = Dense(1024, activation = 'relu')(x)
  # And a logistic layer
  predictions = Dense(classes, activation = 'softmax')(x)
  </code></pre>
  
  <p>For DenseNet and Inceptionv3 the training is ok, and the validation accuracy hits 70%, but for ResNet the validation accuracy stays fixed at 0.0369/0.037 (which is 1/27, my number of classes). It seems like it always predicts one class, but it's weird because its training progresses ok and the unspecific model code is exactly the same as for DenseNet and InceptionV3, which do work as expected.</p>
  
  <p>Do you have any idea why it happens?</p>
  
  <p>Thanks a lot!</p>