Why do we take gain and phase margin from these points in stability?
In a unity, negative feedback loop, the transfer function is of the form \$s \mapsto {g(s) \over 1+g(s)}\$. The poles of the system will be the values of \$s\$ for which \$1+g(s) = 0\$, or \$g(s) = -1\$.

In the complex frequency plane, the point \$-1\$ corresponds to a magnitude of 1 and a phase of 180°, so one measure of how 'close' the system is to being unstable is to see how many degrees the open loop phase is at unity gain, and similarly for the gain margin.

Generally, for such things, I find it more satisfactory to look at a Nyquist diagram rather than Bode plots, albeit you lose the nice frequency parameterization.

Note that despite the nice characterisation by two precise numbers, the measure of stability is qualitative rather than quantitative.