Is there a way to measure or infer the color of a material from a distance (~2m), without using visible light?
What is color?

Color is the light that bounces back after you shine white light (all the visible spectrum) at a surface. Then the surface will absorb, reflect and refract the light and bounce back into your eye ball. What you perceive is the reflected part. 

If you only use red light then you can't tell anything about how much the material absorbs or reflects or refracts the color blue, or any other wavelength for that matter. You only know about the color red. Same thing holds true if you use IR, you only know about IR. You can't find out about things that aren't there.



Since you're probably going to only care about RGB values and not the wavelength (I don't know, you haven't mentioned this). Then you only care about red, green and blue. So send those three colors, and measure them, and/or have three sensors that are sensitive to red, green and blue. 

If you want to be done with it and call it a day, then get a RGB sensor and focus the light coming in to it. Or get a 640x480 camera and only look at one pixel. 

If you want to do it better then you should poke the surface and measure the reflected waves. You could use three one-way-mirrors angled at 45° so the three light waves comes out at the same axis, that will ensure that all the three beams hit the same point. See ascii drawing below.

R  G  B      &lt;-- 3 lasers pointing down
|  |  |  
\--\--\---   &lt;--- R + G + B going right

^  ^  ^
45° mirror


And then you modulate the three lasers with some carrier frequency and measure with your RGB sensor and demodulate it. With modulation you don't have to be super focused on the point. With this you'll get a pretty robust system.



If it isn't obvious by now, you can't acquire the color if you are not interacting with it. You must use visible light somewhere. Either when you receive it or when you poke the material and then measure the reflection.

If you are outdoors then there's always some lamps somewhere and maybe even a moon, with enough measuring and filtering you can make it look like day. But that's because the moon is poking everything in the visible spectrum.

There is a reason why night vision cameras capture in black and white when they only use an IR lamp.