SRCNN - how to get a color image output training only with luminance
The authors use the Y-channel as luminance channel since their images are in the YCbCr color space. The other two channels are upscaled by bicubic interpolation. Later in the paper, they write:


  Specifically, we first transform the color images into the YCbCr space. The SR algorithms are only applied on the Y channel, while the Cb, Cr channels are upscaled by bicubic interpolation.


In their paper, the authors also compare this method to other learning strategies, e.g. training on all three channels or first training the Y-channel and then fine-tune the network with all three channels. The results are on page 11. The section ends with:


  It is also worth noting that the improvement compared with the single-channel network is not that significant (i.e., 0.07 dB). This indicates that the Cb, Cr channels barely help in improving the performance.