Pole Magnitude and Damping Ratio relationship
The magnitude of the poles alone cannot determine the damping ratio, which in turn is related to the rate at which the oscillations about the steady state value will decay.
In fact it is the ratio of the absolute value of the real part of the poles and the magnitude of the poles that gives the value of $\zeta$.
If you think about a second order system its characteristic polynomial would be:
$$
s^2+2\,\zeta\,\omega_n\cdot s \,+\, \omega_n^2 = \\\left(s\,+\omega_n\cdot \zeta-j\,\omega_n\,\sqrt{1-\zeta^2}\right)\cdot \left(s\,+\omega_n\cdot \zeta+j\,\omega_n\,\sqrt{1-\zeta^2}\right)
$$
with poles: $p_{1,2}\,=\,-\omega_n\cdot \zeta \pm j\,\omega_n\,\sqrt{1-\zeta^2}$
However, the above pole forms are valid for $0 \le \zeta \lt 1$. These systems are called underdamped.
Whereas the damping ratio is:
$$
\zeta = \dfrac{|\Re(p)|}{\omega_n}
$$
As a result of the above the magnitude and the damping ratio relate to each other through:
$$
\zeta = \dfrac{|\Re(p)|}{||p||}
$$
You are correct in that the angle of the pole, with respect to the left hand side real axis relates to $\zeta$, and this is the formula:
$$
\cos{\theta} = \dfrac{|\Re{p}|}{\omega_n} = \dfrac{\zeta\cdot\omega_n}{\omega_n} = \zeta
$$
The conclusion that the magnitude alone, whether small or big, cannot determine the decay rate of the oscillations is also apparent from the fact that if we normalize the poles with the natural frequency $\omega_n$ their magnitude will be always equal to 1, this means that if the angle of the pole somehow varies, then they will always be on the unit circle. In such case as the angle varies the damping ratio will also vary but the magnitude will always be the same no matter what the angle.
So its the real part of the normalized pole that determines the damping ratio
rather than its magnitude, alone.
Note: A slow, from a steady state point of view, pole is in fact one that has a small real part, while a fast is one that has larger real part, in absolute value.
Systems with $\zeta = 1$ are critically damped, having conjugate imaginary poles with magnitudes at the system's natural frequency:
$$
p_{1,2} = \pm\, j\, \omega_n
$$
So this means that the hypothetical system has a pair of conjugate poles with equal magnitude:
$$
||p_{1,2}|| = \omega_n
$$
Those systems may exhibit oscillatory behavior sustained longer than a underdamped system, for example, a sinusoidal response. The poles angle is $90^o$ degrees because their real part is zero. Hence $\zeta = \cos(90^o) = 0$ implying an undamped system but the magnitude of the poles is $\omega_n$, thus it depends on the natural frequency of the system which could be huge in value.
In many engineering applications, the measure that is used extensively for pole quality characterization is the $Q$-factor.
In fact, the quality factor and the damping ratio relate to each other through the following formula
$$
Q = \dfrac{1}{2\zeta}
$$
In addition, you may come across transfer functions with characteristic polynomials parameterized with $Q$ instead of $\zeta$.
$$
s^2+2\,\zeta\,\omega_n\cdot s \,+\, \omega_n^2 =
s^2+\dfrac{\omega_n}{Q}\cdot s \,+\, \omega_n^2
$$