Does a capacitor regulate voltage when power is off?
No, capacitors do not regulate voltage when power is off. Your terminology is incorrect.

What capacitors do is serve as a short-term charge reservoir which provides charge when the voltage droops which helps keep it up, and absorbs charge when the voltage spikes which helps keep it down.

They are, however, finite reserves of charge and need to be replenished from a power supply.