How do I calculate the noise variance in this problem?
Since the signal $r(t)$ is expressed as the low-pass equivalent of a bandpass signal, in general it will be complex. That means that the noise is also complex.

White Gaussian noise has a power spectral density commonly denoted by $N_0/2$. This means that if you filter the noise with an ideal filter of bandwidth $B$, the noise at the filter's output is $\sigma^2=BN_0$.

Note that this is the power "per dimension". That is, the in-phase and the quadrature components of the low-pass signal $r(t)$ each suffer from noise of power $\sigma^2$.

You define the SNR as $E|\alpha|/\sigma^2$ and you need it to be equal to 10. One way to do this is to configure your simulation so that $E|\alpha|=1$, and then choose $\sigma=\sqrt{0.1}$. If you later need a different SNR, you can just change one value ($\sigma$) in your simulation.

You can generate the complex noise as follows:

N = length(s);
noise = sqrt(0.1)*(randn(1,N)+1i*randn(1,N));
r = alpha*s + noise;


Note that in this setup, the total noise power is $2\sigma^2$. The in-phase noise and the quadrature noise each have power $\sigma^2$.