Super capacitor directly on the power supply
You can use them like that. The only drawback is the self-discharge of them internally. For a battery application this can be disastrous. For a mains powered application it's probably neglectable.

If you are using them in series to get 5+ V rating, you might need to consider the difference in leakage current though them and balance them off with external resistors.