Title: Metrics - multi-class model comparisons
Tags: machine-learning-model;metric;jaccard-coefficient
Question: I am looking for a way to quantify the performance of multi-class model labelers, and thus compare them.
I want to account for the fact that some classes are ‘closer’ than others (for example a car is ‘closer’ to a ‘truck’ than a ‘flower’ is.   So, if a labeler classifies a car as a truck that is better than classifying the car as a flower.
I am considering using a Jaccard similarity score.  Will this do what I want?

Comment: How would you use Jaccard similarity exactly?

Comment: The Jaccard score computes the average of the Jaccard similarity coeffficients. So basically it's the average of the union and intersection of the two (or more) sets of labels.

Comment: yes but I mean what are the sets that you are going to compare? I don't see how Jaccard can find a higher similarity between classes "car" and "truck" than classes "car" and flowers". Or maybe you independently calculate the similarity based on the words context in a large corpus?

Comment: I was thinking also that RMSE might be valuable since that takes into consideration the distance from truth.

Comment: Agreed. I would have to manually indicate 'closeness'


Here is another answer: There is no commonly established metric do that. You'll have to write custom code based on manually indicating rank ordered preferences of misclassifications.


{'source': 'StackExchangeDataset', 'subset': 'datascience.stackexchange.com', 'metadata': {'AnswerCount': 1, 'CommentCount': 5, 'ContentLicense': 'CC BY-SA 4.0', 'CreationDate': '2020:00:02/05/20', 'FavoriteCount': None, 'PostTypeId': 1, 'Tags': '<machine-learning-model><metric><jaccard-coefficient>', 'Title': 'Metrics - multi-class model comparisons', 'ViewCount': 30, 'id': 67609, 'user_AboutMe': None, 'user_AccountId': 17697406.0, 'user_DisplayName': 'Tavi', 'user_Reputation': 21.0, 'user_Views': 0.0, 'user_id': 89506.0}}