Reason for generally using RMSE instead of MSE in Linear Regression
However RMSE seems similar to MSE and is the root of it, gradient of RMSE with respect to $i^{th}$ prediction differs from that of MSE.
$$\frac{\sigma{RMSE}}{\sigma{y_i}} =\frac{1}{2}\frac{1}{\sqrt{MSE}}\frac{\sigma MSE}{\sigma y_i}$$
Gradient of RMSE is equal to the gradient of MSE multiplied by this $\frac{1}{2}\frac{1}{\sqrt{MSE}}$ value which is constant and is called learning rate. And it shows that RMSE and MSE cannot be interchangeably used when using gradient based methods like when it comes to use linear regression with gradient descent optimization algorithm.
Further, in cases when it is better to give more weight (higher error) to large errors, RMSE can be a better error measure. I believe it depends on your data distribution which one of them to pick to use. But in linear regression, maybe not caring about the points which are considerably off the actual prediction line leads to being biased to considering these points and having a bad model. So, it might be better to give a higher error value to these points.
This and this might also help.