How close is close enough, with regression? When exploring different techniques in machine learning (neural networks), I like to use binary classification problems as a test-bed, because it's very easy to understand how well the technique is working: 50% training/test accuracy is no better than chance, 70% is okay, 98% is very good, etc.
However, sometimes I need to use regression problems instead, and here I struggle to interpret my results. If I draw a scatter plot of training and test mean squared error before and after training, and before training I get around 0.1 loss, and afterwards I get 0.01 loss... is that... good? How do I recognize the difference between success, and barely doing any better than chance?
Typically I'm working with synthetic problems, by the way: fitting polynomials or other kinds of mathematical functions on random vector inputs.
I think it is a far question, since we should always try to translate any metric result to a useful interpretation.
When talking about regression, we can judge the success of our model by checking whether that MAE or RMSE exceed what we are willing to accept. 
For instance, if we want to predict the temperature for the next days, and our result (in this case in a time series problem type) is a MAE of about 2ºC (after re-scaling our data to the natural units like celsius degrees), is it enough for our goal? Is a mean (absolute) error of 2ºC above or below the real temperature enough for the case being studied? Maybe it is for an approximate model, but maybe it is not a good result if we want to use it in an environment where tempreature could have more impact on whatever being studied...
Another additional chance you have is estimating confidence intervals (see this answer) for your model coefficents and, more precisely, prediction intervals (see this other validated answer) to have an estimate for your prediction together with an interval.