Arduino: Measure external voltage
Most, if not all, input pins on AVR controllers are internally protected by clamping diodes. These diodes prevent that an input voltage can be higher than the supply voltage. 

Why these internal diodes are there:

It is important to realize that when an input pin does rise above the supply voltage and these diodes weren't there, the chip might act as an SCR, latch, short the battery, and the chip itself would probably get damaged by overcurrent/overheating.

Why your Arduino runs from the battery
Take a look at the circuit diagram below. When Vcc is detached from the normal power source, there is a path from the battery, via D1, to Vcc. The battery voltage is just high enough for the controller to operate.







simulate this circuit &ndash; Schematic created using CircuitLab

How to solve it

The solution is simple, add a high value resistor in series between the battery and the controller's input. Most AVR's have a maximum input leakage current specified (IIL) of 1μA, so you want to make sure the voltage drop is acceptable with that current, say 100kΩ. 

But beware!

You do have to realize that the input may draw this current from your button cell and shorten its life time.