Interpreting confusion matrix and validation results in convolutional networks I need some help in the assessment of the training results of a convolutional neural network. Here is my setup:


  
  Architecture: InceptionV3
  Pre-trained InceptionV3 with weights from image net
  replaced last layer, retrained the second half of the network
  Classification: Softmax with cross entropy ploss
  Adam optimizer with keras default parameters
  Hyper parameters: Learning rate 0.0001, batch size 64
  Goal: Multi-label classification with exclusive labels
  Data: 17 classes ranging from 0-16 with 48.600 images in total
  Train, Test, Validation: 80%, 10%, 10%
  Training: 2 epochs training only last layer + 2 epochs training second half of network
  


Results

Based on my validation set I created a confusion matrix with the following results.



Here are also some more per-class performance metrics:



As you can see, performance is not very good. The test-accuracy was at 62.2% so it was also not very good. Looking at the confusion matrix it is noticeable that the classes 0, 2, 3, 7, 10, 11, 13 and 14 are never predicted. These classes have also the smallest number of samples. I wonder why this happens? Those classes are more rare than the others but nver predicting them seems odd. What would you recommend going forward?
These are my recommendations: 


Try to train your model more, with different learning rates. Your current learning rate is so much small. At first steps put it to a big value then reduce it smoothly. 
Try to change the architecture. If your task does not have the same scale as image-net surely you would need a network with simpler network architecture, consequently the number of parameters diminishes. 
Changing the distribution of data is dangerous, but you can augment your entire data-set by some extent to let your network see different and more examples.
For those classes that have not been learnt you can set the class-weight to a bigger number to construct a cost function which cares more about those classes. 
Because your data-set is not balanced, use other metrics to evaluate your model, like F1 score.