Is it bad to let an oscilloscope trace go off screen?
Is this bad practice?


I don't think so, unless you push the voltage far above the rated limits and stress the input clamping diodes for a long time.


  What does the analog front end do when the waveform is off screen?


Probably the input amp will be saturated against one of its power rails. But I won't worry too much about it, because the input amp is down the signal chain, after the clamping diodes. In any decent oscilloscope design this should be an operating condition that has been accounted for.

Take a look at this, it may interest you: schematics, full post.