RGB & Proximity Sensor at about 6 feet
Light sensors aren't limited to any distance.  It makes no difference whether the photons they sense came from someone's clothes 1 foot away or the sun 93 Mmiles away.

Your problem with distance is probably that the light is not focused, so the sensor is seeing the average of a large angle.  When a person is close, that angle of view will be dominated by light reflecting off their clothes.  When more distant, all kinds of other ambient light is  mixed in.

The solution is therefore to either focus the light so that only light arriving from a narrow angle hits the sensor, or to just block the light coming from outside the view of interest.  The latter can be easily done by putting the sensor at the back of a tube that is diffuse black on the inside.