GANs and grayscale imagery colorization
Colorfulness, specifically managing overly gray output, was touched on in the pix2pix paper, down in Section 4.2. It might be worthwhile to try out their formulation of a conditional GAN, as their results indicate that their objective function was useful in generating a more diverse set of colors.