Multi-class classification with discrete output: Which loss function and activation to choose?
You could try a Quadratic Kappa Metric loss function. This is a summary of what it actually does.


  A weighted Kappa is a metric which is used to calculate the amount of
  similarity between predictions and actuals. A perfect score of 1.0 is
  granted when both the predictions and actuals are the same. Whereas,
  the least possible score is -1 which is given when the predictions are
  furthest away from actuals. In our case, consider all actuals were 0's
  and all predictions were 4's. This would lead to a QWKP score of -1.
  The aim is to get as close to 1 as possible. Generally a score of 0.6+
  is considered to be a really good score.


The detailed explanation can be found here: https://www.kaggle.com/aroraaman/quadratic-kappa-metric-explained-in-5-simple-steps