LED still glowing with a capacitor in series
The electrolytic cap in your circuit is reverse biased. The reverse biasing of the capacitor removes the isolating oxide layer, so it allows current to pass. If you connect the capacitor the right way after mistreating it this way, the electrochemical process that dissolved the oxide layer is reversed and the capacitor recovers. You have to be careful to limit the current through the capacitor, because high currents caused by leaks in the oxide layer can permanently damage the capacitor. The dropper resistor in your LED circuit is likely enough to limit the current to safe values, but do not connect an electrolytic capacitor that got reversed directly to a battery or lab supply without setting a current limit.

The isolating oxide layer can not only be damaged by reverse polarity, but also dissolve over time if no voltage is applied to the cap. As soon as forward voltage is applied, the leakage current cures the oxide layer by electrolysis, so the cap heals itself. This is why it is sometimes recommended to slowly ramp up the voltage on vintage tube radios that were unpowered for dozens of years, although in some configuration under-heating a tube is bad for the cathode. The process of creating or restoring the oxide layer is called forming.

So as you likely understood, you can abuse an electrolytic cap as rectifier, because it conducts current when reverse biased, but blocks current when polarized the correct way. People actually did use a similar configuration of electrodes and electrolyte as you find in an electrolytic cap as rectifiers before semiconductor rectifiers were invented. They were called electrolytic rectifiers.