How do I obtain the frequencies of each value in a 2D FFT and what do they mean?
It's not over your Nyquist limit, because you're doing sampling at your nyquist rate in two independent dimensions and look at the combined result.

Your square-root formula is easily explained: It's pythagoras, matching your combined frequencies in x and y to a single frequency.