How do I reuse the LO when designing a loopback receiver?
If you have access to unmodulated LO, use that to drive LO port of a mixer, while feeding the TX output (leakage of carrier, USB, LSB) into RF port of mixer. The output DC will be proportional to TX carrier leakage. The two sidebands will fold one atop the other, hiding details you want to observe.

"However, I understand however that mixers aren't linear... if the LO+100 MHz sideband is 25 dB lower than LO-100 MHz, will it contribute at all?" YES it will contribute, proportional to amplitude.

I'd consider, as you already are, the sampling detector. What dynamic range of desired to undesired energy must you view?

I've assisted in implementing -40dBc LO and Sideband suppression circuits in the 500MHz region. We ended up with -50dBc and were pleased. You want -70dBc and 4 octaves higher LO. That requires much careful balancing of parasitics in the 4,000 to 8,000MHz range; on a PCB this will not happen.

Consider 400Mhz and 444MHz sampling clocks, so you can place the fold-over regions (200, 400, 600, 800, 1000, 1200 etc for the 400MHz sampling clock) in clear regions by using the other sampling clock.



Or use a synthesizer IC and Crystal reference, to generate a separate LO for a separate Mixer IC. Get some IC company's ReferenceDesign PCB.