Unique solution for the damping factor in second order system
If \$\alpha\$ is very large (a highly peaky 2nd order low pass filter) the \$x\$ formula tends towards this: -
$$x = \dfrac{1}{2} \pm\sqrt{\dfrac{1}{4} - \dfrac{1}{4\cdot\infty}} = \dfrac{1}{2} \pm\sqrt{\dfrac{1}{4} }$$
And, if \$\alpha\$ is very large, we know that \$x\$ (and \$\zeta\$) have to be very small hence, the sign cannot be positive because it would make \$x\$ be very nearly unity. And that would be incorrect.
$$\boxed{\text{So the sign has to be negative}}$$