Cycles always renders RGB channels to the same values under sun lamp?
I figured out why. The reason is that my lamp size is too big, which leads to almost no shadow on the shadow catcher.

So my region of interest is actually illuminated by the white light source (instead of being illuminated by the diffuse reflections). With a grayscale diffuse map, the whole thing renders to grayscale.