When to use Capacitors?
When I first started out in electronics I struggled with the same question. The problem is that capacitors are used in a vast number of different ways.

However, as you're just starting out in electronics you probably only need to know about a few of these to start with. The most widely used and basic of these are:

Power Supply Smoothing

This is the easiest and very widely used application of a capacitor. If you stick a big beefy electrolytic capacitor (the bigger the better), it will fill in all the gaps created by rectifying an AC waveform, to create a relatively smooth DC. It works by repeatedly charging during the peaks, and discharging during the gaps. However, the more load you put on it, the quicker it will drain the capacitor and the more ripple you'll get.

Timing

If you supply power to a capacitor through a resistor, it will take time to charge. If you connect a resistive load to a capacitor, it will take time to discharge. The key thing to understand here about timing circuits is that capacitors appear as though they are short circuit while they are charging, but as soon as they are charged, they appear to be open circuit.

Filtering

If you pass DC through a capacitor, it will charge and then block any further current from flowing. However, if you pass AC through a capacitor, it will flow. How much current flows depends on the frequency of the AC, and the value of the capacitor.