Should we keep all channels when doing image classification?
Well, it depends on your images and how you want to classify them.

If we take a really basic example, if you want to classify images by their color, you obviously should keep all 3 channels.

In the other hand, if you want to classify images by a shape. For example squares and circle, you don't necessarily need to use all 3 channels.

But changing the number of channel isn't the only preprocessing you can do on your images before classifying them. 

You can also change the color space for example (the colorspace LAB can be usefull in some case) , or even use Morphological Image Processing methods to improve your image by, for exemple, bringing out the contrasts of your images or cleaning impurities.

All of this depends of your images, and what you want to do with them.