Arduino power mysteriously stays on
It looks like the Arduino must still be receiving power from the 5 volts via a non-disconnected GPIO line. Those GPIO pins that are still connected to 5 volts will route the 5 volts to its own local rail inside the chip with maybe a volt drop of about 0.5 volts. This is quite a common occurrence in CMOS chips and works equally well when GPIO pins are inputs and outputs.

Here's what Dave says.