Quadrature adjustment circuit
'I'm unable to model the phase shifter ... '

When you wrote 'I know its phase change constant in rad/V  ...', you have modelled the phase shifter. I would expect any phase shifter to be fast enough that you can represent it as a simple gain, at the speed of any control loop you want to put round it.

All you need to do is to detect the average DC coming out of the mixer. This is the parameter that you need to drive to approximately zero to make the measurement. To do this, you need a high(ish) gain op-amp back from the mixer to the phase shifter. 

If you put an integrator pole into the op-amp feedback, then it will control the mixer output to exactly zero (within the amplifier's input offset voltage). 

As a control loop to stabilise, it's actually easier than a PLL (90 degrees easier to be exact) as it doesn't have that nasty VCO frequency to phase integrator 90 phase shift that PLLs have.