Proof that spectrum analyzer measures mathematical Power Spectral Density It seems to me the power spectral density (PSD) is defined as follows. Consider a signal $X(t)$ The windowed version of this signal is

Consider a signal $X(t)$. We defined a time-windowed version of $X(t)$ as 

\begin{align}
X_{\Delta t}(t) = X(t)W_{\Delta t}(t)
\end{align}

Where $W_{\Delta t}(t)$ is a time windowing function. We will choose

\begin{align}
W_{\Delta t}(t) = \frac{1}{\sqrt{\Delta t}}\theta\left(t-\frac{\Delta t}{2}\right)\theta\left(\frac{\Delta t}{2} - t\right)
\end{align}

Where $\theta(t)$ is the Heaviside function. $W_{\Delta t}(t)$ thus windows $X(t)$ to the range $-\frac{\Delta t}{2} &lt; t &lt; \frac{\Delta t}{2}$ and scales by $\frac{1}{\sqrt{\Delta t}}$.

The Fourier transform of a signal is defined as

\begin{align}
\tilde{X}(f) = \int e^{-i2\pi f t}X(t) dt
\end{align}

I understand the PSD to be defined as

\begin{align} \tag{1}
S_{XX}(f) = \lim_{\Delta t \rightarrow \infty} \langle |\tilde{X}_{\Delta t}(f)|^2\rangle
\end{align}

Where $\tilde{X}_{\Delta t}(f)$ is the Fourier transform of $X_{\Delta t}(t)$.

My question is how to prove than the output of a heterodyne style spectrum analyzer approximates the power spectral density.

My understanding of such a spectrum analyzer is that it takes in a signal $X(t)$ which is mixed with a local oscillator signal $f_{LO}$. This mixing brings the component of the signal at frequency $f_{LO}+f_{IF}$ down to $f_{IF}$ where the mixed signal is passed through a narrow bandpass IF filter $\tilde{F}_{IF}(f-f_{IF})$. Here $\tilde{F}_{IF}(f)$ is a filter centered about zero frequency so that $\tilde{F}_{IF}(f-f_{IF})$ is centered about $f_{IF}$. The two steps of mixing + filtering can be combined into one step of passing the signal through an effective filter $\tilde{F}_{IF}(f - f_{IF} - f_{LO}) = \tilde{F}_{IF}(f - f_0)$ where $f_0$ can be tuned by tuning the $f_{LO}$

Finally after passing through the IF filter the total power in the signal is detected.

it then mixes it to an intermediate frequency $f_{IF}$ and then passes it through a narrow bandpass filter, $\tilde{F}_{IF}(f)$ and then detects the resultant power. 

Let us define

\begin{align}
\tilde{F}_{IF,f_0}(f) = \tilde{F}_{IF}(f-f_0)
\end{align}

The output of the spectrum analyzer is then

\begin{align} \tag{2}
X(t) \rightarrow (X \ast F_{IF,f_0})(t) \rightarrow |(X \ast F_{IF,f_0})(t)|^2
\end{align}

Here, because the spectrum analyzer technically measures in the time-domain, I have shown the action of the IF filter as convolution with the filter impulse response $F_{IF,f_0}(t)$.

My question is how to show $(1)$ and $(2)$ end up being the same for measured signals. It is clear that they are intuitively very similar but I'm having a hard time showing their equality mathematically. 

Here are my half-baked ideas towards a solution
I know that

\begin{align}
\tilde{X}_{\Delta t}(f) = (\tilde{W}_{\Delta t} \ast \tilde{X})(f)
\end{align}

This convolution in Fourier space due to the windowing in time is very suggestively similar to the "swept filtering" which comes from demodulating the signal with the tuned LO signal. It is clear that $W_{\Delta t}(t)$ is related to $F_{IF}(f)$. However, it seems tricky because in $(1)$ the convolution happens in frequency space while in $(2)$ the convolution happens in time space.

In a spectrum analyzer the averaging happens by taking the time average of the measured power, that is 

\begin{align}
X_{sig}(t) \rightarrow \frac{1}{t_{aq}}\int_{t=0}^{t_{aq}} |X_{sig}(t)|^2 dt \rightarrow \langle |X_{sig}(t)|^2\rangle
\end{align}

where in our case $X_{sig}(t)$ is the output of the IF filter. Perhaps at this point I could assume $t_{aq} \rightarrow \infty$ to apply Parsevals theorem to get something like

\begin{align}
\int |\tilde{X}_{sig}(f)|^2 df = \int |\tilde{F}_{IF,f_0}(f)\tilde{X}(f)|^2 df
\end{align}

This is starting to seem close but it's still not quite the convolution that appears in $(1)$

Can anyone give me some direction on these mathematical manipulations?
I am assuming $X(t)$ is a passband signal having finite support (non zero value for only a finite range of frequencies within spectrum analyzers range).

In your equation (1) what you have missed is $f_{LO}$ to $f_{IF}$ conversion. So $W_{\Delta t}$ should be $W_{IF}(t) = e^{j2\pi (f_{LO}-f_{IF})t}W_{\Delta t}(t)$. When you convolve $X(t)$ and $W_{IF}(t)$, in frequency domain you will get the magnitude as $|(X(f - f_{\Delta}) * W_{IF}(f - f_{\Delta})|^2$. Here $f_{\Delta}$ is the frequency shift needed to bring $X$ to IF and $W$ also includes the windowing parameter necessary to remove discontinuities around $\pm\Delta t/2$. 

In (2), you are first applying a mixer-filter but then comparing (1) - a frequency domain quantity with (2) a time domain quantity. Let us say, the quantity in (2), you have applied the windowing after mixing to IF. ie, $X_{IF}(t) = (X(t)*F_{IF}(t)) \times W_{\Delta t}(t)$.

If you take fourier transform of above, you will get

$|(X(f - f_{\Delta }) F_{IF}(f - f_{\Delta})) * W_{\Delta t}(f - f_{\Delta })|^2$. But for the frequencies of interest (remember the spectrum analyzer has only finite bandwidth, so it will filter out frequencies outside it) the value of $F_{IF}(f) = 1$. So finally it will be $|X(f - f_{\Delta}) * W_{\Delta t}(f - f_{\Delta})|^2$ which is the same as we got above. Your intuition is right about (2). It is indeed convolution in time domain. But if transform it back to frequency domain, you can see it is same as (1). There are lot of things happening in the background. We have sampling and before sampling we have the Low Pass filter. All this will eliminate the unwanted frequencies outside the band of interest. Hence we can 'forget' the effect of filter $F_{IF}$