When to use RMSE as opposed to MSE and vice versa?
Answers in the comments are decent, particularly DuttA.
DuttA gives these, approximately

ease of derivative
Don't have to worry about ~0 in denominator causing huge gradient
But to me the most important is mathematical convenience, someone
might easily make the mistake of RMSE is just equal the difference
y−y′ instead of root of mean square of y−y′. The answer this might
start depending on conventions.
In maths (Don't know the reason and might be inaccurate) we mainly
work with variances instead of standard deviation.

Here are my reasons for using the MSE instead of RMSE:

Doesn't have the sqrt operations, so it computes faster
the square root isn't easy, its Newtons method, so it could be a
dozens steps per iteration
MSE has all the information of RMSE, there is 1-to-1 mapping, so no loss
the storage of the square root typically doesn't save any memory in
IEEE-784 and compute vs. memory is big thing in complexity
tools like gradient boosted machine can &quot;recycle&quot; the squared error
computation for speedup and working on O(n) complexity
there is hidden scaling and regularization because many gpu hardware
elements are fundamentally 8-bit, so if you can make your code more
8-bit in its guts then you don't have as much in the back-conversion
and it runs a lot faster