Power issues with SD to USB IC?
When "no SD access", it doesn't mean that USB is in SUSPEND state. Even if no access, the SD is still mounted and USB bus is active, clocks are running, current is consumed in all communication layers. 

To get power saving in "idle" mode, the port (where USB2244 device is attached) must be forced into SUSPEND state. It means that the any traffic must be terminated by host, and then the USB2244 conteroller will enter into low power state (350 uA typical from 3.3V) after 3-ms bus idle time.

More, there is a caveat: even if the 2244 controller goes to SUSPEND, the attached SD card might still consume idle current, unless its power supply is completely turned OFF. For this purpose the 2244 has a built-in FET power switch, which turns SD power off when entering SUSPEND state. If the card is designed that the SD power is not supplied from this switch, power savings might be seriously reduced.

Without SUSPEND state the 2244 controller will consume up to 80-100 mA, and obviously drain your battery.