Proof that spectrum analyzer measures mathematical Power Spectral Density
I am assuming $X(t)$ is a passband signal having finite support (non zero value for only a finite range of frequencies within spectrum analyzers range).

In your equation (1) what you have missed is $f_{LO}$ to $f_{IF}$ conversion. So $W_{\Delta t}$ should be $W_{IF}(t) = e^{j2\pi (f_{LO}-f_{IF})t}W_{\Delta t}(t)$. When you convolve $X(t)$ and $W_{IF}(t)$, in frequency domain you will get the magnitude as $|(X(f - f_{\Delta}) * W_{IF}(f - f_{\Delta})|^2$. Here $f_{\Delta}$ is the frequency shift needed to bring $X$ to IF and $W$ also includes the windowing parameter necessary to remove discontinuities around $\pm\Delta t/2$. 

In (2), you are first applying a mixer-filter but then comparing (1) - a frequency domain quantity with (2) a time domain quantity. Let us say, the quantity in (2), you have applied the windowing after mixing to IF. ie, $X_{IF}(t) = (X(t)*F_{IF}(t)) \times W_{\Delta t}(t)$.

If you take fourier transform of above, you will get

$|(X(f - f_{\Delta }) F_{IF}(f - f_{\Delta})) * W_{\Delta t}(f - f_{\Delta })|^2$. But for the frequencies of interest (remember the spectrum analyzer has only finite bandwidth, so it will filter out frequencies outside it) the value of $F_{IF}(f) = 1$. So finally it will be $|X(f - f_{\Delta}) * W_{\Delta t}(f - f_{\Delta})|^2$ which is the same as we got above. Your intuition is right about (2). It is indeed convolution in time domain. But if transform it back to frequency domain, you can see it is same as (1). There are lot of things happening in the background. We have sampling and before sampling we have the Low Pass filter. All this will eliminate the unwanted frequencies outside the band of interest. Hence we can 'forget' the effect of filter $F_{IF}$