What mathematical tools exist for understanding modulated noise?
I'm not sure specifically what you're looking for here. Noise is typically described via its power spectral density, or equivalently its autocorrelation function; the autocorrelation function of a random process and its PSD are a Fourier transform pair. White noise, for example, has an impulsive autocorrelation; this transforms to a flat power spectrum in the Fourier domain.

Your example (while somewhat impractical) is analogous to a communication receiver that observes carrier-modulated white noise at a carrier frequency of $ 2 \omega $. The example receiver is quite fortunate, as it has its an oscillator that is coherent with that of the transmitter; there is no phase offset between the sinusoids generated at the modulator and demodulator, allowing for the possibility of "perfect" downconversion to baseband. This isn't impractical on its own; there are numerous structures for coherent communications receivers. However, noise is typically modeled as an additive element of the communication channel that is uncorrelated with the modulated signal that the receiver seeks to recover; it would be rare for a transmitter to actually transmit noise as part of its modulated output signal.

With that out of the way, though, a look at the mathematics behind your example can explain your observation. In order to get the results that you describe (at least in the original question), the modulator and demodulator have oscillators that operate at an identical reference frequency and phase. The modulator outputs the following:

$$
\begin{align}
n(t) &amp;\sim \mathcal{N}(0, \sigma^2) \\
x(t) &amp; = n(t) \sin(2\omega t)
\end{align}
$$

The receiver generates the downconverted I and Q signals as follows:

$$
\begin{align}
I(t) &amp;= x(t) \sin(2 \omega t) = n(t) \sin^2(2 \omega t)\\
Q(t) &amp;= x(t) \cos(2 \omega t) = n(t) \sin(2 \omega t) \cos(2 \omega t)
\end{align}
$$

Some trigonometric identities can help flesh out $ I(t) $ and $ Q(t) $ some more:

$$
\begin{align}
\sin^2(2 \omega t) &amp;= \frac{1 - \cos(4 \omega t)}{2}\\
\sin(2 \omega t) \cos(2 \omega t) &amp;= \frac{\sin(4 \omega t) + \sin(0)}{2} = \frac{1}{2} \sin(4 \omega t)
\end{align}
$$

Now we can rewrite the downconverted signal pair as:

$$
\begin{align}
I(t) &amp;= n(t) \frac{1 - \cos(4 \omega t)}{2}\\
Q(t) &amp;= \frac{1}{2} n(t) \sin(4 \omega t)
\end{align}
$$

The input noise is zero-mean, so $ I(t) $ and $ Q(t) $ are also zero-mean. This means that their variances are:

$$
\begin{align}
\sigma^{2}_{I(t)} &amp;= \mathbb{E}(I^2(t)) = \mathbb{E}\left(n^2(t) \left[\frac{1 - \cos(4 \omega t)}{2}\right]^2\right) = \mathbb{E}\left(n^2(t)\right) \mathbb{E}\left(\left[\frac{1 - \cos(4 \omega t)}{2}\right]^2\right) \\
\sigma^{2}_{Q(t)} &amp;= \mathbb{E}(Q^2(t)) = \mathbb{E}\left(n^2(t) \sin^2(4 \omega t)\right) = \mathbb{E}\left(n^2(t)\right) \mathbb{E}\left(\sin^2(4 \omega t)\right)
\end{align}
$$

You noted the ratio between the variances of $ I(t) $ and $ Q(t) $ in your question. It can be simplified to:

$$
\frac{\sigma^{2}_{I(t)}}{\sigma^{2}_{Q(t)}} = \frac{\mathbb{E}\left(\left[\frac{1 - \cos(4 \omega t)}{2}\right]^2\right)}{\mathbb{E}\left(\sin^2(4 \omega t)\right)}
$$

The expectations are taken over the random process $ n(t) $ 's time variable $ t $. Since the functions are deterministic and periodic, this is really just equivalent to the mean-squared value of each sinusoidal function over one period; for the values shown here, you get a ratio of $ \sqrt 3 $, as you noted. The fact that you get more noise power in the I channel is an artifact of noise being modulated coherently (i.e. in phase) with the demodulator's own sinusoidal reference. Based on the underlying mathematics, this result is to be expected. As I stated before, however, this type of situation is not typical.

Although you didn't directly ask about it, I wanted to note that this type of operation (modulation by a sinusoidal carrier followed by demodulation of an identical or nearly-identical reproduction of the carrier) is a fundamental building block in communication systems. A real communication receiver, however, would include an additional step after the carrier demodulation: a lowpass filter to remove the I and Q signal components at frequency $ 4 \omega $. If we eliminate the double-carrier-frequency components, the ratio of I energy to Q energy looks like:

$$
\frac{\sigma^{2}_{I(t)}}{\sigma^{2}_{Q(t)}} = \frac{\mathbb{E}\left((\frac{1}{2})^2\right)}{\mathbb{E}(0)} = \infty
$$

This is the goal of a coherent quadrature modulation receiver: signal that is placed in the in-phase (I) channel is carried into the receiver's I signal with no leakage into the quadrature (Q) signal.

Edit: I wanted to address your comments below. For a quadrature receiver, the carrier frequency would in most cases be at the center of the transmitted signal bandwidth, so instead of being bandlimited to the carrier frequency $ \omega\ $, a typical communications signal would be bandpass over the interval $ [\omega - \frac{B}{2}, \omega + \frac{B}{2}] $, where $ B $ is its modulated bandwidth. A quadrature receiver aims to downconvert the signal to baseband as an initial step; this can be done by treating the I and Q channels as the real and imaginary components of a complex-valued signal for subsequent analysis steps.

With regard to your comment on the second-order statistics of the cyclostationary $ x(t) $, you have an error. The cyclostationary nature of the signal is captured in its autocorrelation function. Let the function be $ R(t, \tau) $:

$$
R(t, \tau) = \mathbb{E}(x(t)x(t - \tau))
$$

$$
R(t, \tau) = \mathbb{E}(n(t)n(t - \tau) \sin(2 \omega t) \sin(2 \omega(t - \tau)))
$$

$$
R(t, \tau) = \mathbb{E}(n(t)n(t - \tau)) \sin(2 \omega t) \sin(2 \omega(t - \tau))
$$

Because of the whiteness of the original noise process $ n(t) $, the expectation (and therefore the entire right-hand side of the equation) is zero for all nonzero values of $ \tau $.

$$
R(t, \tau) = \sigma^2 \delta(\tau) \sin^2(2 \omega t)
$$

The autocorrelation is no longer just a simple impulse at zero lag; instead, it is time-variant and periodic because of the sinusoidal scaling factor. This causes the phenomenon that you originally observed, in that there are periods of "high variance" in $ x(t) $ and other periods where the variance is lower. The "high variance" periods are selected by demodulating by a sinusoid that is coherent with the one used to modulate it, which stands to reason.