Interpreting confusion matrix and validation results in convolutional networks
These are my recommendations: 


Try to train your model more, with different learning rates. Your current learning rate is so much small. At first steps put it to a big value then reduce it smoothly. 
Try to change the architecture. If your task does not have the same scale as image-net surely you would need a network with simpler network architecture, consequently the number of parameters diminishes. 
Changing the distribution of data is dangerous, but you can augment your entire data-set by some extent to let your network see different and more examples.
For those classes that have not been learnt you can set the class-weight to a bigger number to construct a cost function which cares more about those classes. 
Because your data-set is not balanced, use other metrics to evaluate your model, like F1 score.