7805 backfeeds 4v to circuit
You should never backfeed power to a 78xx; it can damage the regulator. The usual method of avoiding this is putting a diode from output to input; this avoids damaging the regulator, but not backfeeding power.



simulate this circuit &ndash; Schematic created using CircuitLab
A second diode on the input of the regulator would help with that, preventing both damage to the regulator and to your source, though it would increase the effective dropout voltage of your regulator by one diode drop, so be aware of that if you're close to the limit.



simulate this circuit
It goes without saying, but be sure to use a diode that can handle the amount of current you expect your system to use. A 1N4148 might not cut it.