Convert filters pre-trained with ImageNet to grayscale? I am wondering how to convert caffe reference model trained with ImageNet (color pics) for grayscale image to save memory and to speed up.

The filters in the caffe convolution layers are different for RGB but this can be, for example, averaged to form one channel filter.

Since my data is grayscale and speed at test (prediction) is important for me, this may help.

I can of course convert my grayscale images in RGB representation to reuse pretrained convolution layers as they are. But I want to avoid it if possible.

My questions are:

Would this be possible?

How exactly one should combine filters? (Average?)

After filters converted, weights between conv layers need to be re-trained?
I doubt either of your proposed approaches will work, these convolutional layers learn all kind of relationships between the colors which will be non-existent in grayscale, upsampling it to RGB makes your input space very different from the input space that ImageNet was trained on and averaging these filters of the trained ImageNet model makes a whole lot of assumptions that are unlikely to hold. The second approach does seem more promising than the first, so if you want to test an approach I would go for the second one. It's possible that it is at least a better initialization than randomly. I think you will end up needing a lot of data and computation anyway, so it might be better to start off with your own network shape and start from scratch.