Capacitors on a voltage regulator? Or just in general
A capacitor is (in simplistic terms) a small reservoir of electricity.

A voltage regulator is quite a slow device. When more current is needed the regulator has to respond to that and make more current available. That takes it (on the scale of things) quite a long time to do. It pretty much has to go down to the corner store to get more current for you.

The capacitor, though, has a reserve of extra electricity it can use to meet your demands. It gives you the extra current you need from its reserves, then the voltage regulator can go off and get more for you, which it then uses to replenish the reserves in the capacitor.

This is called decoupling because you have decoupled the demand from the source.

If the capacitor is too small it can't provide enough current fast enough (or if the capacitor isn't there at all) and this is seen as ripple on the voltage. As more current is requested the voltage drops to compensate until the regulator can make the required adjustments. When less current is required the voltage rises, again until the regulator can adjust itself accordingly. 

If the capacitor is too big it can take too long to respond to high frequency demands for current.

It is normal to have multiple capacitors at different locations in a power system - large ones at the power supply, and places like where power enters a board. Medium sized ones to help distribute the power around the place more smoothly, and small ones right by where the electricity is used to respond to the small bug high frequency switching demands of things like microcontrollers and logic chips.

There are other more technical uses for capacitors as well - signal filtering, resonant circuits, etc. and I am sure someone else will cover those uses.