How to Filter Harmonics in Wideband VCOs
The HP3326 synthesizer was a delight.
I included the schematic of that VCO, in this answer
Crystal oscillator application
Tuned DC -- 13MHz, in 0.1Hertz steps. I recall tuning it to 3.578 MHz Chroma Carrier, to cause a slow phase rotation, and thus color rotation, in a composite_video experiment.
But the frequency_scheme (signal flow) was this:

oscillator tuning 20Mhz to 33MHz. The narrow tuning range allowed high_Q tank and thus lots of harmonic_suppression. The VCO has very defined input DC power, and well defined Power LOSS, thus the internal oscillation AMPLITUDE is very controlled, and there is no clipping so no overt distortion.

20MHz fixed_tune oscillator

mixer with lots of double_balanced harmonic_cancellation

filter on the Mixer output was just DC__to 13MHz LC lowpass FIXED, not tracking.


Key? the low harmonic_distortion VCO (is a emitter_coupled oscillator, with emitter current set by a resistor, to pin down the maximum power available; also a fixed LOSS element in the extract_signal_for_PLL;

the VCO's  fixed emitter current and fixed loss element are key for setting the VCO oscillation amplitude, without any clipping

===============================
Most oscillators (just about every VCO and every XTAL oscillator) I have seen do NOT control the amplitude of the oscillation; the circuits depend on HARD or on SOFT limiting; whenever limiting is used, harmonics will be generated.
The HP3326 circuit provides a FIXED input power thru the shared emitter resistor.
And the circuit has a well defined LOSS of energy in the circulating current, with currents into that common_base transistor. But this LOSS is not linear; the LOSS is P = V^2 /R.
So far as references, I do not find any. But the old &quot;HP Journal&quot; was the source for my initial appreciation of the circuit, many decades ago.
=================================
Notice this is a big change to your topology.
You need a narrow_range VCO (phase_locked to some crystal reference). Say the range is 500 to 800MHz.
You need a clean fixed frequency oscillator at 500MHz.
You need a good double_balanced mixer, so the RF (500 to 800) is mixed with the LO (500MHz) and the input energy (RF and LO) are nearly perfectly cancelled.
And you need a LOW PASS FILTER DC--300MHz.
========================================
How does that oscillator work?

There is a positive_feedback path. The circuit uses a differential pair, one base grounded, the other base fed from the opposite collector, which is positive feedback.

There is a closed path for the circulating resonant current. The arrows show that path, both the highcurrent path straight to Ground and the smaller current into the &quot;load resistor&quot; which serves as energy_absorber and thus defines the steady_state amplitude.

Most of the resonant current flows in L1/C4/C1 loop. There are NO LOSSES in this loop. The inductor/C4 node is in parallel with the output impedance of the 2N5179 (a 1,500MHz Ftau bipolar with 1 pF Cob and, depending on which manufacturer you select, quite a good Early Voltage).You want to choose a high quality VDD bypass capacitor, because almost ALL the circulating current flows thru that capacitor; and you want that capacitor to not be microphonic, or mechanical tapping will upset the VCO operation and show up as timing jitter (residual Frequency Modulation).

The 2N5179 has 4.5 dB Noise figure in a 50 ohm system. If was 3dB, then the internal Rnoise of the 2N5179 would be 50 ohms. If was 200 ohms, then the noise voltage would be 2x higher and the Noise Figure would be 6dB worse at 9dB.

In the plot for reverse transfer admittance, let us find the reactive and the real parts. The real part is 0.1 milliMhos, or 10Kohms. Thus our R_early at 1.5mA is 10Kohms. We can use that ---- 10Kohms ---- as the parallel LOSSY element.