Linear regulator drawing power when powered from output only
A few options:


find a regulator that tolerates 'back feeding' and consumes less current in that situation
isolate the regulator with a (schottky) diode. this will cause some voltage drop that might need compensation. A primitive way to do that is to put the same type of diode in the ground lead of the regulator.
automatically switch between the regulator and the battery, for instance with a swiitch (which might double as a on-off switch).