Transferring colors from my Direct/Indirect passes into Color
Why there is color in other passes than Color
The direct and indirect stands for direct or indirect lighting so if the light has color, it will show in direct passes and if objects in the scene have color and light bounces off them, the light color will change accordingly in the indirect passes. These passes are supposed to have color. You can change the colors of your objects without changing lighting by changing the Color passes. Unfortunately this means indirect light that bounces off those objects has to be recalculated during rendering for it to change.
Separating color can be done
If for whatever artistic purposes you would like to separate all the color, you can do that in Blender or in any other software that supports 32bit image editing just by manipulating the images.
In this case you could make a copy of the image you wish to separate with a RGB to BW node and simply divide the colored image by it. You would have to multiply them together to get the colors back, so now you can multiply only the color 'passes' together and have the greyscale image for the separated monochromatic part of lighting:

Should you do it?
Obviously, you can do this separation at any moment in the compositing, even with the final image, but then passes are not needed. This can be done inside most image editing software. In many cases the purposes of rendering passes includes separation of the lighting from the colors and textures of the objects.