Characterizing a capacitor
You'll have to work with the assumption that your capacitor is in parallel with the impedance of the analyser input.
The analyser 50 ohm input is calibrated to show the correct voltage when connected to a 50 ohm source, and the output is calibrated to be the correct voltage when connected to a 50 ohm load.
You'll have a voltage divider composed of \$Z_{out}\$ and \$Z_C \parallel Z_{in}\$.  Calculating \$Z_C\$ will mean measuring the voltage at a particular frequency then calculating \$Z_C \parallel Z_{in}\$, then calculating \$Z_C\$ from that given that you know \$Z_{in}\$.
You might consider working with the capacitor in series.  I think the math will be slightly less ugly, and you will get better results for the higher impedance areas.