Confusion matrix, when mistakes below diagonal are better than above the diagonal I have a classification problem and I am producing a confusion matrix. Ideally one wants to get all results in the diagonal. I get quite many points around diagonal for different algorithms. Still for my use-case I want to favor algorithms that underpredict the class (I have ordinal data) and not overpredict.
Is there a metric that can measure under and overprediction and rate those errors with a different weight? The typical accuracy, precision terms assume that all mistakes are the same.
Of course I can try to implement my own metric but I am quite sure that I am not the first that is having this issue.
Any metric available that you know already?
Thanks
Alex
You are incorrect when you say accuracy and precision assume the same mistakes. They are quite different in nature as to when they are applied for separate use-cases.
For more details on how they are different - https://towardsdatascience.com/accuracy-precision-recall-or-f1-331fb37c5cb9
As per your situation, you could opt for precision as you are looking only for the right answers, in case of a spam email detection.
There is no metric as such which calculates how much your model underfits or overfits but a genera idea can be made when you compare the validation curve and the training curve.