Fluctuating ADC readings upon changing power source
simulate this circuit &ndash; Schematic created using CircuitLab

What kind of switch are you using that switches both charger and battery?  Beware of arcs. You only need to switch the load.