How to ensure my device never powers down when using two batteries (Hot swapping batteries)?
This depends on the kind of battery you are using.

If you are using lead acid batteries; this is all fine and will work the way you describe.

If you are using practically any other type of battery; you should not do this, the batteries could charge at unsafe rates. It would be safer to wire a "schottky power diode" in series with each battery so that they can both power the system but nether can charge the other. You would swap the batteries and charge them away from this system.