How to determine SNR required for adding AWGN noise to clean IQ samples?
Note that the MATLAB result is floating point (or may be fixed point but with much higher precision) while the VSA hardware is fixed point with lower precision that also has dynamic range limitations in the analog portion of the hardware as well. To see this directly and more clearly, try these two experiments:

Create a single carrier tone and capture results at different signal levels of that carrier. Ensure the sampling rate is much higher than the tone frequency (such as > 12x higher to capture the 3rd and 5th harmonic). 

Observe the output with a properly windowed FFT (I recommend the Kaiser window) and increase the capture duration until the higher harmonics are clearly visible. (increasing the duration will lower the FFT noise floor). 

From such testing you can determine the SNR (based on the wideband noise floor) as well as spurious free dynamic range (SFDR) of your test set-up. The SNR is estimated as the difference between the tone and the integrated power that is not at the tone frequency (and outside the main lobe of the window used).  The SFDR is the difference in power between the tone and the highest spur which may or may not be at a harmonic of the signal (In some definitions of SFDR the harmonic components are not considered and that distortion is captured in Harmonic Distortion). In any event find the maximum tone power that maximizes SFDR, and this sets a mark for "Full Scale" from which the other metrics can be computed. 

If you proceed with this, please post your results. What would be interesting too is a plot of Pin vs Pout for your tone, from which you could also determine where it starts to compress. The point where Pout is 1 dB lower than the otherwise linear relationship is known as the 1 dB compression point. 

To be complete you could also do two tone testing, where you generate two sinusoids closely spaced (but wider than the width of the main lobe of your window in the frequency domain) and measure the two-tone third order intercept point. See Wikipedia for more details on that test:

https://en.wikipedia.org/wiki/Third-order_intercept_point

I would want to understand all these parameters in conjunction with the use of any test equipment such that I understand what is being introduced by the test equipment rather than in the waveform itself. 

Finally when adding AWGN to the signal to simulate the SNR for a discrete system, the AWGN is a power spectral density that is flat across the entire sampling bandwidth. The SNR of interest would be the portion of the spectrum that is in-band---thus if you are sampling at a higher rate, the AWGN with the same variance would be spread across a wider bandwidth resulting in lower SNR given the portion of the noise that would occupy the same bandwidth of the signal. Any noise outside of the signal bandwidth can be filtered so is not part of the SNR computation.