If I have a signal that switches between 5MHz and 4MHz faster than the sample rate of the ADC on a spectrum analyzer
If the signal is changing faster than the bandwidth of the bandpass filter, then the following circuitry won't see the effect as the signal changing -- they'll see the effect in the frequency domain.
For instance, if the signal is changing between 5MHz and 4MHz, randomly every 500ns, but staying phase coherent (meaning that there's no time-domain jump discontinuities in the signal), then the signal is MSK modulated, and when you look at it with a spectrum analyzer that's set up correctly, you'll see a classic MSK spectrum like this one from Wikimedia Commons.

In this case &quot;properly set up&quot; would mean that you want an IF bandwidth that is much less than the modulation rate (i.e., less than 200kHz), and a sweep rate that's low enough that the filter bandwidth doesn't distort the signal (although you don't have to worry about that anymore -- a spectrum analyzer with a computer in it won't let you make that mistake).
In the case of your signal at 5MHz that's being &quot;switched between 10dB and 50dB every 20ns&quot;, you reformulate that statement to say that the signal is being chopped at 25MHz.  In that case what a properly set up spectrum analyzer would show you peaks at 5MHz, 20MHz, 30MHz, then 70MHz &amp; 80MHz, then 120MHz &amp; 130MHz, repeating every 50MHz. -- basically you'd see peaks at \$(25\mathrm{MHz})(2n - 1) \pm 5\mathrm{MHz}\$ for all \$n \in \mathbb{I}\$.
Google &quot;frequency mixing&quot;, &quot;mixer&quot; or &quot;heterodyning&quot; for more details.  The hand-wavy answer is that when you multiply two sine waves together in the time domain, in the frequency domain you get signals at the sum and difference of their frequencies -- so a 5MHz sine wave multiplied by 25MHz gives you tones at 20MHz and 30MHz.  A 25MHz square wave can be decomposed into sine waves at 25MHz, 75MHz, 125MHz, etc. -- so you get this sine wave multiplication effect at all of those frequencies.
A more comprehensive answer has to do with the properties of the Fourier transform (a spectrum analyzer is basically an analog computer dedicated to finding the Fourier transform of its input -- sorta kinda).  I can't really go into that here, but if you're interested in this stuff you probably want to pick up a book on signal processing and start studying.