Should I always put a low-pass filter on an ADC input?
It's usually a good idea to put an anti-alias filter in front of the ADC's input and if this also removes unwanted noise from the signal that is also useful. However, I don't think the filter is the full answer to why your circuit suffers "variations" when disconnected from the AC supply. You should try using a resistor potential divider to halve the dc rail to your ADC and feed this into the input to see if it is (somehow) and ADC problem or an external problem. It could be a problem with wires on the input and there ability to pick-up AC magnetically or it could be something else.

It's also worth saying that allowing a little high frequency noise thru with your signal can lead to high resolution results - it's called dithering.