Multi-class classification with discrete output: Which loss function and activation to choose? I'm working with a multi-class classification problem, using Keras Sequential models. In my dataset, the output class has one of the following values: (1, 2, 3, 4, 5).

Yes, I have already tried one-hot-encoding the output and using categorical_crossentropy as a loss function and softmax in the output layer.

However, I don't think that softmax and categorical cross entropy is the right choice in my case. In my dataset, the output classes have a certain "discreteness" (or scale). Class 1 is the "worst" and class 5 is the "best".

Let's say if on a specific input the model predicts 2 and the true class is 1, it's a much better prediction as when the model predicts class 5 and true class is 1.

I would like, that the loss function would take these "minor errors" into consideration and not treat all errors the same way.

To sum up, I'm wondering what my options are in terms of loss function and activation of the output layer for my given problem.

UPDATE: I have now tried to use mse as a loss function and replace my output layer with this:

model.add(Dense(1, activation='linear'))

The new method produces much better results, but I'm still wondering if there are any other options for my problem. Maybe a custom loss function?
You could try a Quadratic Kappa Metric loss function. This is a summary of what it actually does.


  A weighted Kappa is a metric which is used to calculate the amount of
  similarity between predictions and actuals. A perfect score of 1.0 is
  granted when both the predictions and actuals are the same. Whereas,
  the least possible score is -1 which is given when the predictions are
  furthest away from actuals. In our case, consider all actuals were 0's
  and all predictions were 4's. This would lead to a QWKP score of -1.
  The aim is to get as close to 1 as possible. Generally a score of 0.6+
  is considered to be a really good score.


The detailed explanation can be found here: https://www.kaggle.com/aroraaman/quadratic-kappa-metric-explained-in-5-simple-steps