Inconsistency of ADC output
ADCs are not perfect and a 12 bit device with a 3.3V reference will have a theoretical resolution of about 0.8mV. You are seeing variations of a few millivolts and this doesn't surprise me at all - check the INL (integral non-linearity) of the device - it might be +/-3 bits. Check the DC offset spec of the device - this might be anywhere between 2mV and 10mV. Check the DNL (differential non-linearity) of the device - this might be 2 bits.

All these errors add up to make the picture worse. 

Also, your 3V3 supply - how accurate is it and how much does it drift? This drift will directly affect accuracy so why not consider using a really tight voltage reference (if it can be applied).