understanding image error metrics, mean squared error
First of all, the mean squared error should not have a square root. As the name implies, it is the mean of the squared difference. What you are referring to is the root mean squared error. Second, the this is a pixel-wise comparison between to images. Therefore the number of pixels is the dimension, and the sum is performed on it. There is even a comment on that page about color images.