Comparing accuracy of models in ordinal regression / classification I am looking into creating a model to predict whether an item is "Very Good", "Good", "Bad" or "Very Bad".

After I fit the training data to the models, comparing the accuracy of the models during test stump me: should it matter if a model misclassified a G to VG while the other G to VB? What about a model that has two misclassifications of one level away versus another model with only one misclassification but three levels away (eg VG to VB)?

Any guideline on what is the common approach? Also, my thinking at the moment is that this should be a regression problem, but I'm happy to be corrected if I should approach this labeling of datasets more as a classification problem.
Your classes express a certain order. You can classify apples to, say, "green", "red" or "yellow", and then every disagreement with a reference set is equal. After all, colours express no order. So as you already suggested, I would certainly use regression. Assume that the classes could be distributed as something like this:


Very bad = 0 - 0.25
Bad = 0.25 - 0.50
Good = 0.50 - 0.75
Very good = 0.75 - 1.00


Now, the mismatch of Very good vs. Bad is at least 0.25, where is must be at least 0.50 with Very good vs. Very bad, which gives a better and more honest impression of the performance of your model.