Servo does not stop in a stable position
The reason that adding a capacitor helps is because when the DC motor which actually drives the servo is starts or stops, it injects some noise into your power supply (+5V). This causes the sensor part of your servo to read a slightly different value for the position of the actuator, since the sensor is just a potentiometer wired between +5 and GND with the wiper in the middle which sets a voltage due to voltage divison. 

Consider for example that you want the servo to be oriented "half way". If the voltage exactly corresponds to to position of the servo, the servo electronics compare the actual voltage to the halfway between +5 and GND (2.5V), turning the motor to reduce the voltage and therefore positional difference between the actual and target positions/voltages. However, suppose that in turning the motor on, the voltage of the +5V power droops slightly. Thus, the voltage being read is slightly lower than it should be given the position of the actuator. This causes (for example) the motor to turn off prematurely. However, when it does so, the voltage rises back up to +5V, and therefore the servo surmises that it is not close enough, and turns the motor back on, but this reduces the voltage of the power, so the cycle continues. This creates an internal feedback path that causes the servo jittering or buzzing.

Adding a capacitor close to the servo creates a low pass filter which reduces this noise on the power rail, which allows it to remain at a constant voltage irrespective of the motor turning on and off.