Oscilloscope waveform shows overlapping levels, second harmonic?
If you are sampling at 80GHz you by definition do not have 40GHz of bandwidth! See the sampling theorem for why.

The single shot is suggestive as to what is going on, you have the 80GHz sample clock and the 40GHz signal, but probably they are not locked together so you have two mixing products very close together, the sum of which is giving you your amplitude modulation. 
If you were in fact sampling at EXACTLY twice the input frequency, you would see something, but it would have an arbitrary level depending on the phase relationship. 

For 40GHz of bandwidth you need to be sampling FASTER then 80GHz, but 36GHz works with a 80GHz sampling rate.