A "Voltage Backdrive" on DC-DC Converters?
You've got reverse current flowing in the regulator, probably through protection diodes in one or more of its chips.
It's a module, and it doesn't specify that situation, so you don't know if it'll be damaged or not.
If it were me, and I were determined to use a module there, I'd start by shopping around for one that does specify that sort of a back-drive situation.
If my back were up against the wall and I just had to ship product with that module, and I couldn't make changes, I'd measure the current flowing from the +5V line into the regulator -- if it's less than a mA, I'd reluctantly say the product is OK to ship -- then I'd go home and make sure my resume is current.
Ideally, if you use that module, you'd put a switch between that regulator and the rest of the circuit that cuts off the output of the regulator when the 24V supply drops below 8V, or when USB power is applied.