Arduino "Phantom Powering"
When you disconnect the arduino from 5V you must also remove either


disconnect any other device that is connected to the arduino and has its own power supply
or disconnect every device that is connected to the arduino from its power supply


When you disconnect 5V from the arduino while devices connected to a gpio are still powered, this gpio might see a voltage larger than zero. But the voltage at any pin must not exceed the arduino's actual supply voltage by more than 0.3V. Otherwise the atmega's supply rail will be pulled high through esd protection diodes and the atmega will try to operate. It will then probably draw more current than what the esd diodes are rated for and they will eventually fail (destroying that gpio).