Inverting radial distortion
I'm not sure inverting these equations will help you gain any understanding. Consider for example the simple case in which $a = b$. In that case,
$$
x'^2 + y'^2 = (x^2 + y^2) (1- a\|P\|)^2 = \|P\|^2(1-a\|P\|)^2,
$$
and taking square roots,
$$
\sqrt{x'^2+y'^2} = \|P\|\,\big|1-a\|P\|\big|.
$$
You know the left-hand side. The right-hand side is either $\|P\|(1-a\|P\|)$ or $\|P\|(a\|P\|-1)$ (though I'm not sure the latter is physically possible in your case), and in each case you get a quadratic for $\|P\|$. You get at most four solutions for $\|P\|$, and for each positive one you can substitute back, calculate $x,y$, and check that the equations are satisfied.

The case $a \neq b$ looks more complicated, but perhaps can be solved explicitly using a bit of creative algebra. Failing that, there are numerical method that can invert these equations.