Direct USB Charging Battery
I did something similar for my kids' rocking swing, only I bypassed the battery. Find one of the 64 old phone chargers you have around that matches the voltage and current requirements of the mobile, and connect that to the battery box. (You did save all those proprietary Motorola and LG flip-phone chargers for hobby projects, right?)

You can either solder directly to the terminals, or solder to a couple metal plates that you hold in place with a battery-sized dowel. This way you can use batteries again later if you like. Consider installing a fuse inline with the positive wire for added safety.