`SSIM(YUV)` vs. `SSIM(Y)`: Values Are Very Different
I figured it out: the ssim() function in MATLAB, does not handle RGB images, but instead threats it as a 3D-image (with the third dimension time for example, to measure the quality of a video). So it interpreted my 3-color-component image as a video with 3 frames.