How to test I/Q modulation with only one branch (I or Q)?
The FSW is superheterodyne.

This means that it will not have a DC offset or I/Q imbalance, but the IF path will appear as an asymmetrical lowpass (because it is really a shifted bandpass), which is then compensated for in the calibration.

The first question: have you run the calibration properly?


leave both machines running for half an hour in a stable environment.
disable the generator output (OUTP OFF)
calibrate both (*CAL)


Lots of equipment lenders don't understand that the calibration sequence is used to compensate temperature, run the calibration once (usually on a cold machine) to make the error message go away and tell customers to never touch that button. This gives a fairly bad filter to compensate the IF, and since that filter is applied before downconverting the IF to zero, this will also appear as a complex FIR filter to you.

Second, test your setup with a simple test:


generate a sine wave (*RST, FREQ 2 GHZ, OUTP ON)
look at the vector display in IQ Analyzer (*RST, INST IQ, LAY:REPL '1',VECT)


You should see a single stationary dot there, with a position that corresponds to an arbitrary phase angle that is the sum of the phase between the LOs (they are certainly not the same frequency!) and the digital downconversion inside the FSW, which together give a constant, but only as long as all the oscillators keep running at their fixed frequency.

Once you have reached that state, you should be able to repeat your measurement with more complex signals.