How to measure a almost (?) ordinal classification?
You could just calculate the mean absolute error.
However, this quality measure (and any other measure taking into account the distance) would not be consistent with the loss function you used while training your classifier I guess. So it would not be fair to your algorithm. It is like if your professor would change the grading system after you submitted your test.
Instead of changing the quality measure you might want to reconsider your approach. The suggestion to use the ordinal regression sounds good. In case if the cost of misestimation between neighbors is similar you could even use plain regression and then round.