IR sensor readings at different voltages
It sounds to me like your system works, but you have trouble if your voltage drops. 

First of all here's 2 tips that might solve your problem.


instead of just making 1 reading do 10 and take an average - this reduces noise.
the voltage reference is allready based on the voltage that powers the system so with your adjusted reading you try to accomplish something which has already happened an therefore has the opposite effect and actually increase the error.


see http://tronixstuff.com/2013/12/12/arduino-tutorials-chapter-22-aref-pin/ for a very good explanation.


Add a RC filter to removes random fluctuations in the signal from you photoresistor 


see https://electronics.stackexchange.com/questions/17754/where-to-place-a-capacitor-to-smoothen-ir-sensor-reading for a short description and the calculated example.

If however it is correct, and your readings of colors don't follow a linear scale, you would have to deduce that actual curve and apply it in the code - but that might not be easy. 

So it would be much more easy to remove that variable by ensuring that the voltage is constant. Most Arduino boards have an 5V onboard voltage regulator that will keep the voltage exactly at 5 Volt when connected to battery via the power Jack!

(USB bypasses this regulator and have a little voltage drop so it is sligthly lower than 5V, and more very prone to fluctations if coming from a AC source affecting your analog read considerably)

The battery connected onboard voltage regulator however requires a higher input to do the regulation so the battery needs to supply atleast 6.2V or higher, in your case you would have to use a voltage source of less than 6.2V if this is the source of your problems.

(the voltage regulator on original arduino boards needs voltage of 1.2V higher than the needed output - clones might differ and require upto 2v more than the needed output)

hope this can help solve the issue

Mikael