What is the main source of yaw stability?
Wing sweep does affect yaw stability, but all planes with a vertical tail will "weathervane" into the wind.  It seems you are on the right track considering angular drag as a mechanism of stability, angular lift works even better.  A vertical stabilizer can act as a vertical wing.  Symmetrically, it can generate lift against either deflection.

Secondly, the "overshoot" is a real concern in design, but will be a function of weight as well as "angular drag". Much harder to stop a piano on ice compared to a light cardboard box.  A tail will typically be light, pivoting around the center of gravity.  Ample tail area will make it easier to control.

Finally, as far as "damping" is concerned, one must realize the "viscosity" of air will resist motion as well.  Once a force is removed, drag will help it slow to a stop, so the opposite "correcting force" need not be as large.