How to test I/Q modulation with only one branch (I or Q)? Given is a direct-conversion I/Q up- and downconverter system. Receiver and transmitter share the same (10MHz) reference and hence the LO frequency is identical and there is an (unknown) phase difference between them. The transmitted RF signal is given as:

$$
x_{\rm rf} = x_i \cos(\omega_c t) + x_q \sin(\omega_c t)
$$

Now I look at only one channel (e.g. the I channel) at the receiver (that means that any I/Q imbalance of the receiver is irrelevant). At the I branch, this signal is downconverted with an arbitrary phase shift phi and filtered with H:

$$
y_i = H( x_{\rm rf} \cos(\omega_c t + \phi) ) = \cdots \\
\approx H( \cos(\phi) x_i/2 - \sin(\phi) x_q/2 ) \\
= \frac{\cos(\phi)}{2}H(x_i + x_q \tan \phi) \\
= G(x_i + \alpha x_q) \\
= G(x_i) + \alpha G( x_q) \\
= G_1(x_i) + G_2(x_q)
$$

The filter H is contains all filtering effects (mainly anti-aliasing filter).

I want to find a linear relationship between the time-domain samples $y_i[n]$, $x_i[n]$ and $x_q[n]$. A straight forward way is to bring this equation in a vector-/matrix form, model $G_1$ and $G_2$ as FIR filters with $Q$ taps and solve it via Least Squares:

$$
\mathbf{y}_i = \mathbf{X}_i \mathbf{g}_1 + \mathbf{X}_q \mathbf{g}_2
$$

This works nicely and I get the result as expected (in my case, the error between $\mathbf{y}_i$ and the received samples is -45dB). The caveat is I get a set of $2Q$ (real-valued) coefficients but according my derivation, $G_1$ should identical to $G_2$ up to a constant factor. If I would do the same for the $Q$ channel I get again $2Q$ coefficients and a total of $4Q$ real-valued coefficients if I were to model the I/Q relationship that way.

Clearly I can model everything directly in complex domain (with $Q$ complex-valued or $2Q$ real-valued coefficients) but then I cannot look at only the I channel without effect from the Q channel.

Hence a more proper way is to look at this relationship:

$$
\mathbf{y}_i = \mathbf{X}_i \mathbf{g}_1 + \mathbf{X}_q \alpha \mathbf{g}_1
$$

However, this is not a least squares problem any more. For that reason, I sweep $\alpha$ within a reasonable range, compute the LS solution and take the $\alpha$ with the minimum error (this seems to be convex). But if I do this, the error -35 dB (as opposed to -45dB above).

I really do not understand why this approach would not work.
Why? And what is the proper way to model this?

Background info: I am testing my own direct conversion receiver. If I transmit data in only one channel (I or Q) and look at only one output channel (e.g. I) I get the performance that I expect based on linearity-, noise- and phase noise measurements of the receiver. If I look at I/Q at the same time (and model the the effect of the receiver as complex-valued FIR filter) then the error is more than 5 dB worse. For debugging I would like to avoid any interaction (such as I/Q imbalance) between I and Q and only look at the I channel output. But as soon as I transmit a full complex-valued input and look only at the I output my result is again at least 5 dB worse than predicted by noise, linearity, phase noise.

As another step, I use an Arbitrary Waveform Generator (Rohde &amp; Schwarz SMW200A) used as transmitter and a VSA (Rohde &amp; Schwarz FSW) as a receiver where I see the same issues. Both instruments are calibrated and the transmitter has superior image rejection. For these reasons I assume I made a conceptional or algebraic mistake in my derivations/assumptions.
The FSW is superheterodyne.

This means that it will not have a DC offset or I/Q imbalance, but the IF path will appear as an asymmetrical lowpass (because it is really a shifted bandpass), which is then compensated for in the calibration.

The first question: have you run the calibration properly?


leave both machines running for half an hour in a stable environment.
disable the generator output (OUTP OFF)
calibrate both (*CAL)


Lots of equipment lenders don't understand that the calibration sequence is used to compensate temperature, run the calibration once (usually on a cold machine) to make the error message go away and tell customers to never touch that button. This gives a fairly bad filter to compensate the IF, and since that filter is applied before downconverting the IF to zero, this will also appear as a complex FIR filter to you.

Second, test your setup with a simple test:


generate a sine wave (*RST, FREQ 2 GHZ, OUTP ON)
look at the vector display in IQ Analyzer (*RST, INST IQ, LAY:REPL '1',VECT)


You should see a single stationary dot there, with a position that corresponds to an arbitrary phase angle that is the sum of the phase between the LOs (they are certainly not the same frequency!) and the digital downconversion inside the FSW, which together give a constant, but only as long as all the oscillators keep running at their fixed frequency.

Once you have reached that state, you should be able to repeat your measurement with more complex signals.