Interpreting the Root Mean Squared Error (RMSE)!
How can I interpret RMSE?


RMSE is exactly what's defined. $24.5 is the square root of the average of squared differences between your prediction and your actual observation. Taking squared differences is more common than absolute difference in statistics, as you might have learnt from the classical linear regression.


  It confuses me a little. And I could not find any reliable reference
  to also clearly state that one can safely interpret RSME as one does
  MAE. Is RMSE is simply a only mathematically more convenient for
  optimization etc., and we are better off with MAE for the
  interpretation?


I think this post should help you. I'll answer your questions directly:


RMSE is easier mathematically and also practically. Have you heard of derivative? The derivative for MAE is undefined when your prediction is equal to observation but well defined everywhere for RMSE. In machine learning, a well defined gradient function is generally better.
Both RMSE and MAE are useful, but they are two very different metrics. In regression, it's generally about choosing between linear regression and quantile regression. They are two very different models!
As stated in the link, if you don't want your residuals affect your model too much, MAE could be better. Otherwise, if your data set is well defined (not many residuals), RMSE could be better. 


There is no right or wrong on which one is better. Think like two different algorithms for doing modelling.