Will it hurt a non-powered power supply to see 12/24V on the output?
Some PSU's have physically disconnectable outputs (i.e. a relay on the output) and/or some other kind of over voltage detection/protection when on. OVP is usually a marketed feature, but doesn't necessarily mean it will work when the PSU has no power at all. Physically disconnectable outputs is not a feature I've typically found advertised.

Regardless of whether it's safe or not, you can add 2 diodes to protect both the PSU and the battery:







simulate this circuit &ndash; Schematic created using CircuitLab

This will select the output with the higher voltage and use that to power the charger while protecting the other source from harmful reverse currents/voltages.

You should pick diodes with relatively low forward voltage drops (i.e. Schottky diodes) and capable of handling the currents the charger will draw.