No-break (IPS) circuit analysis and improvement
It's not elegant, but you could put a diode right across your battery.  It would be reverse biased if the battery is inserted correctly, otherwise you would have a fuse or resettable polyfuse in series with the battery that would open when the battery is inserted backwards.  

Other possibilities get more complicated or less efficient in normal operation.