How to compare two regression models?
There are two things to consider:


Sampling bias
Metric


The sampling bias problem is that your test set is likely not the complete set of things you're interested in. So, no, you can't simply check MSE_1 &lt; MSE_2 and conclude it is always the case when it's "just" for your dataset the case. This is what significance tests are for. (Although this kind of reasoning is super common in machine learning and I did it myself already )

Then the question if the metric is the correct one for your application. Typical choices are: MSE, mean absolute error, custom cost functions