Why do LEDs stay lit when a circuit is broken?
You are quite right that a LED remains lit while current flows through it. Where you see equipment remain alive after a power-down switch has been flipped "off", stored energy in its power supply is depleted until it is consumed. The stored energy likely comes from a charged capacitor. In this case, the power switch disconnects the power supply from its source (at its input, rather than its output). Since all the circuits (including the LED) are always connected to the power supply output, any stored energy in the supply is available to that LED after the switch disconnects the supply from its source.
However, it is becoming more common that power supplies remain alive while plugged into the wall. The power switch operates differently than the scenario described above...it tells the supply to stop accepting power, rather than crudely disconnecting the power source. A computer or display can use a tiny inexpensive switch to perform this function. In this scenario, it is possible that shut-down might be delayed for a very short time while house-keeping functions are run. It is also possible that any delay is due to the same energy-storage in the power supply capacitor(s).
Incandescent lamps and motors do remain running after power is denied (for a very short while). A lamp has stored energy in the thermal mass of its filament, which gets quickly radiated away. A motor has stored energy in its rotating mass, which spins down quickly when power is denied. Energy is conserved...at power-up, both lamp and motor require a surge of power equal to the "free energy" dissipated at power-down.