How to scale measured QPSK points to determine magnitude error?
Even though have AGC etc in your system, it is not ensured that the received constellation points will lie exactly at the points that you would expect from the TX constellation. 

There can be residual offset, gain and phase rotation (at least). In the following , there's a simple method to estimate and remove offset and gain. Essentially, you calculate the mean of the RX constellation, which is your estimated offset. Then, after offset removal, you measure the energy of the constellation and scale the constellation accordingly, such that the constellation has same energy as tx constellation energy:

N = 10000 # number of TX symbols
QAM = np.array([1+1j, 1-1j, -1+1j, -1-1j])
txEnergy = np.var(QAM)

tx = QAM[np.random.randint(4, size=(N,))]

chanGain = 2 # channel gain (combined of AGC, channel and all other processing)
offset = 0.3-0.5j # channel offset
sigma2 = 0.3  # noise variance


noise = np.sqrt(sigma2/2) * (np.random.randn(N) + 1j*np.random.randn(N))

rx = chanGain * tx + offset + noise  # emulate effects of channel + AGC + ...
# RX is the received signal after demodulation

# estimate mean and remove it from the data
rx_mean = np.mean(rx) 
rx_mean_correct = rx - rx_mean

# estimate the energy of the constellation
rx_energy = np.var(rx_mean_correct)

print ("Estimated vs real offset: %s &lt;-&gt; %s" % (rx_mean, offset))
print ("Estimated vs real energy: %s &lt;-&gt; %s" % (rx_energy, chanGain**2*txEnergy))

# scale rx constellation to have same energy as tx constellation
corrected = rx_mean_correct / np.sqrt(rx_energy) * np.sqrt(txEnergy)

# calculate constellation MSE
MSE = np.var(rx_mean_correct - tx)
print ("RX MSE: %f" % MSE)

# plot the stuff
plt.plot(rx.real, rx.imag, 'x', label='Received')
plt.plot(corrected.real, corrected.imag, 'x', label='corrected')
plt.plot(tx.real, tx.imag, 'x', label='tx')
plt.legend();


Program output

Estimated vs real offset: (0.300844347234-0.50994604822j) &lt;-&gt; (0.3-0.5j)
Estimated vs real energy: 8.29630318416 &lt;-&gt; 8.0
RX MSE: 2.297161




In order to remove the phase rotation, you need to have some information about the transmitted symbols. I.e. after the received constellation has been processed as above, you need to compare the RX constellation with the known TX points to get the rotation between them. Then, you rotate the RX constellation back according to the estimated phase rotation.