What causes a breaker to trip?
Short-circuit Protection
A breaker that offers short-circuit protection, will trip if there is a short somewhere in the circuit.  A short-circuit is characterized by a very high current increase in a very short period of time.  This type of protection is usually provided by an electromagnetic device; commonly an electromechanical solenoid, which physically opens the circuit when a short-circuit is detected. Most residential short-circuit protection devices trip at 10 kA (10,000 A).
Overcurrent Protection
Breakers that offer overcurrent protection, trip when a current higher than the rated current is drawn through the device over a specific amount of time. An overcurrent situation is usually detected using heat, and so is slower acting than short-circuit protection.  Overcurrent protection is usually provided by a bimetallic strip, which trips the breaker if it is heated too much for too long.
Most typical residential circuit breakers are combination devices, and offer both short-circuit and overcurrent protection. Wikipedia has a great image of the inside of a circuit breaker, which shows both short-circuit and overcurrent protection mechanisms.


5 - Bimetallic strip providing overcurrent protection.
7 - Electromechanical solenoid offering short-circuit protection.

Ground-fault Protection
Ground-fault protection is a recent addition to circuit breakers, which offers protection to people and equipment.  Breakers that offer ground-fault protection, trip when the current flowing out to the circuit is higher than the current returning from the circuit.  A current transformer (CT) inside the breaker, is a common way used to measure current imbalances.  The CT will be wrapped around both conductors, and if a large enough current is induced on the CT the breaker will open the circuit.  Residential ground-fault circuit breakers are almost exclusively combination devices offering short-circuit, overcurrent, and ground-fault protection.
Arc-fault Protection
Arc-fault protection is the most recent addition to circuit breakers, and offers protection against dangerous electrical arcs. They use complex electronics to analyze the circuit looking for tell tale signs of dangerous electric arcs, all the while ignoring non-hazardous arcs.  As with ground-fault circuit breakers, arc-fault breakers are almost always combination devices as well.