VNAs and different characteristic impedances
So, here is the solution, after several hours of work. To help with the context, I am using a cold-war era HP 8753B with an 85046B S-parameter test set- and yes, the B-model is 75Ω. And please don't get confused below. This answer is about using a VNA with Z0 that is not 50Ω, but I have to make a few comments about hand-made standards because you aren't likely to find standards in anything other than 50Ω. And I haven't found a way in my VNA to have it interpolate a 50Ω load to some other impedance.

The mathematics in my VNA (and I assume most VNAs) are based on scaled values. It doesn't really know what 50Ω real, or 250Ω real is. And this applies to complex loads as well. All it knows in the world is what it measures from the calibration standards- and please comment where I am wrong below. 

Though not actually part of the answer to my question, as I refer to calibration loads below, picture in your mind all the different ways those loads are made. Think about their geometry, and the fact that any adjacent conductors can be capacitors, and any conductor can be an inductor, whether the cal standards you use are:


supplied by the manufacturer
3 coax pigtails that start life as identical in every way possible especially including the length and how the ends are stripped, before you make them into an open, load and short
opened, loaded and short copies of a PCB you need to measure


At any rate, here is what I learned about my question. When you calibrate a VNA, what you are actually teaching it is what it should call ∞, 1 and 0. Just like the normalized Smith chart. There is no magic in 50Ω. Nowhere in all the complex math on SWR, the Smith Chart, the reflection coefficient, etc. is there a special case for 50Ω, BUT ∞, 1 and 0 are all very special numbers.

Back to the point. When calibrating the VNA, it is reading the response of what you call an infinite impedance, a characteristic impedance, and 0 impedance. 

So, if you are working with a transmitter or receiver that works best at some impedance different than 50Ω, what do you do? There are only 3 steps. 

First, calibrate the VNA with your standards, an open, a load at the relavent impedance you need (be it 10Ω, 50Ω or 300Ω), and a short. And again, make your own open, load and short standards carefully- and NEATNESS COUNTS!! You are going to want to trust your readings. Conversely if you trust readings that are actually wrong- in the accented words of my Korean professor "you waste your life". The load should be resistors- preferably something like 0603 surface mount resistors. And make the resistance from at least 2 resistors in parallel, if not 4, to minimize inductance. 

Second, looking at the smith chart view, adjust the electrical delay on your VNA so that the ∞ and 0 loads actually read very close to ∞ and 0.

Third, adjust the characteristic impedance (on my VNA this is in the calibration menu) to your actual load resistance. But remember, all you are actually doing is setting the scale value for readings that need it, like the actual impedance readings in the smith chart. BUT, scale values are NOT used on the actual smith graph, nor in the return loss and SWR readings.

Those charts are nothing more than comparisons (actually mathematical difference I think) to the open, load and short standards it read during the calibration sequence.

But now that your VNA is calibrated to your target impedance, all the charts start making sense again. 

Below are 3 pictures of readings of a transmitter circuit that I needed a good match for at 319MHz. The MAX7044 is most efficient with a 125Ω impedance. Incidentally (and this is for free), the MAX7044 has an output capacitance of about 5.6pF at 319MHz, which seems to me to be a CRUCIALLY overlooked tidbit of knowledge on the data sheet.


the actual swept response of a transmitter and matching network on a spectrum analyzer. Notice the peak on the left at 319Mz, and the peak on the right at about 400MHz. I injected a swept frequency in place of the crystal in the MAX7044 transmitter, and used a coax stub stripped back 1/4" or so as the test antenna so it should hopefully have a flat response at 300MHz to 450MHz.

the log mag on the VNA calibrated with a 50Ω load. Notice it says the 400MHz peak is better (which it clearly isn't in the actual circuit), owing to the fact that the circuit is at about 50Ω at 400MHz


the log mag on the VNA calibrated with a 125Ω load. Notice the response is a much better prediction of the actual circuit, because the circuit is at about 125Ω at 319MHz- the ideal impedance for the transmitter




Hope this helps.