My Servo Won't Sweep Smoothly
In electronics, as usual in situations where a cricuit is misbehaving and we don't understand why, a solution that generally works is to add a decoupling cap between +5V and GND. 

In your case, I would put at least 220uF; that would ensure two things:


the voltage used to control the servo never drops
the voltage brought to Arduino by the USB does not drop either (which would have very bad impact on the Arduino, probably resetting it)


Electrolytic caps are a must-have for every electronics hobbyist (and professional as well, of course).

You can read more about decoupling capacitors on Wikipedia.

Edit:

According to @gwideman comment, it seems that using a decoupling capacitance above 10uF in a USB 2.0 device would violate USB specs and may damage the USB host controller on the computer.

I was not aware of this limitation but I have already used decoupling capacitance up to 100uF in my circuits and never got any issue with my USB computer port.

It might be due to extra electronics on the Arduino board that would "isolate" USB power from Arduino +5V anf d GND pins, I don't know (that should probably be the topic of a specific question); or maybe I have just been lucky all this time...