Capacitor Differences
Capacitance per volume
Electrolytic capacitors have good capacitance per volume (and cost). Ceramics do not have such good capacitance per volume. Tantalums lie somewhere in the middle.

Frequency response/Effective Series Inductance (ESL)
If the series parasitic inductance is high, the capacitor will have poor high frequency response. Electrolytic capacitors have poor high frequency response. Ceramics have good high frequency response. Important if you are decoupling high currents at low frequencies (i.e. motors) or low currents at high frequencies (i.e. high speed digital circuits). 

Polarization
Ceramic capacitors are not polarized. Electrolytics and tantalums are polarized so you just can't use them in circuits where the voltages across them are going to be bipolar.

Temperature Stability
Capacitors will have capacitance that varies with temperature. This is important if you need accurate capacitances for things like filters. C0G capacitors are the most common temperature-stable capacitors.

DC Bias Stability
Most ceramic capacitors (i.e. not C0G/NP0) have reduced capacitance as the DC bias voltage across them increases. This can be particularly important in filters.

I believe electrolytics are fairly stable in this regard. Apparently, tantalums do not suffer from this at all.

Microphonic Effects
Most ceramic capacitors (that aren't C0G/NP0) also have piezoelectric microphonic effects where vibrations will produces voltages in the capacitor which isn't good if you're using it as signal filter. Electrolytics and tantalums (I think) do not have this.

Effective Series Resistance (ESR)
Too high ESR can cause a capacitor to heat up too much when passing currents which may be important for high current applications (motor decoupling). But too low ESR can cause certain circuits to become unstable (regulators) or produce ringing  since there is insufficient resistant to provide effective damping (do not use C0G/NP0 to decoupling circuits for this reason, other than the fact it will cost $$$)