How to compare two regression models? Which measurement(s) should one choose to compare two regression models?

After modifying a learning algorithm(specifically, a regression algorithm, let's call it M1) to generate another learning algorithm M2, how to validate if the above modification is efficient?

here is what I did(with 10-fold cross-validation)

I choose MSE as the only measurement, at each run, for M1 and M2, calculate the MSE of both the training and testing set.

And the result shows that:


average MSE of the training set of 10 runs: M2 &lt; M1
average MSE of the testing set of 10 runs: M2 &lt; M1 


Question: 

according to the above list, can we draw a conclusion that M2 is better than M1? thus, the modification of algorithm M1 is efficient(at least on this dataset)? 

Or:

Did I miss some other important measurements? Is there a rule of thumb of comparing two regression models?
There are two things to consider:


Sampling bias
Metric


The sampling bias problem is that your test set is likely not the complete set of things you're interested in. So, no, you can't simply check MSE_1 &lt; MSE_2 and conclude it is always the case when it's "just" for your dataset the case. This is what significance tests are for. (Although this kind of reasoning is super common in machine learning and I did it myself already )

Then the question if the metric is the correct one for your application. Typical choices are: MSE, mean absolute error, custom cost functions