What happens when the voltage applied to fully charged capacitor becomes lower than initial voltage?
If the supply voltage is changed quickly enough, the the capacitor starts sourcing voltage, the current flows backwards into the supply.

Bypass capacitors are used to regulate voltage, but mostly for short term voltage drops from cables or trace inductance. The capacitor can supply voltages to the load in the event the voltage drops.