Making a model of a transformer
The core losses Rc are quite strongly frequency dependent and level dependent, so varying the frequency will lead to mis-modelling of all the lossy R terms. I would expect the system of 6 equation to be quite ill-conditioned, so very sensitive to small reading errors.

Normally, we estimate Rp and Rs from DC measurements. We estimate Lp from an open circuit measurement and Lm from a short circuit measurement, as described in the rest of the paper you've linked.