How to power gate device without voltage drop
Perhaps you could use a dedicated power switch ic like these from TI or maybe just a logic-level mosfet. I recommend the power switch IC -- they're designed for it.