After plotting my predicted values against the true label values, I didnt quite get the answer I was looking for
The true quality is 4,5,6... the predicted quality deviates from the true one. This deviation is expressed by the variation of the dots on the x axis. The line in the plot shows, that there is positive correlation between your prediction and true states (which is good and expected).

Think about the magnitude of mae 0.5 in your case. Means „half a class wrong“. You likely have some concentration of predicted values around the true value (not good to spot in such a figure) and some outliers, viz relatively bad predictions, which are far away from the true class. You can also see that your prediction is bad for low and high classes.

I guess this is a linear regression? Try multinominal logit or the like. In you linear regression the prediction can be any rational number. What you would like to have as a prediction is a class, so 4,5,6 or whatever the class is called.