Why is the variance going down so much in this weight initialization problem(using pytorch)? first look at this example

&gt;&gt;&gt; x = t.randn(512)
&gt;&gt;&gt; w = t.randn(512, 500000)
&gt;&gt;&gt; (x @ w).var()
tensor(513.9548)


it makes sense that the variance is close to 512 because each one of 500000, is a dot product of a 512 vector and a 512 vector, that is sampled from a distribution with a standard deviation of 1 and mean of 0

However, I wanted the variance to go down to 1, and consequently the std to be 1 since standard deviation is square root of variance, where 1 is the variance.

To do this I tried the below 

&gt;&gt;&gt; x = t.randn(512)
&gt;&gt;&gt; w = t.randn(512, 500000) * (1/512)
&gt;&gt;&gt; (x @ w).var()
tensor(0.0021)


However the variance is actually now 512 / 512 / 512 instead of 512/ 512

In order to do this correctly, I needed to try 

&gt;&gt;&gt; x = t.randn(512)
&gt;&gt;&gt; w = t.randn(512, 500000) * (1 / (512 ** .5))
&gt;&gt;&gt; (x @ w).var()
tensor(1.0216)


Why is that the case?
Basically, because the variance is quadratic in w.

You can consider your problem as computing the variance over scalar products $z = \vec{x}\vec{w}$ by drawing 500000 samples of $\vec{w}$ from $p(\vec{w})$. Let's compute the mean first

$$ E_{p(\vec{w})} [ \vec{w}^T\vec{x} ] =  \vec{\mu}_w^T \vec{x} = 0$$

because you draw the $\vec{w}$ from a zero-centered Gaussian. With the expectation value as zero, the variance is given by

$$ E_{p(\vec{w})} [ (\vec{w}^T\vec{x})^2 ] \\
= E_{p(\vec{w})} [ \vec{x}^T\vec{w} \vec{w}^T \vec{x} ] \\
= \vec{x}^T E_{p(\vec{w})} [ \vec{w} \vec{w}^T ] \vec{x}
$$

$E_{p(\vec{w})} [ \vec{w} \vec{w}^T ]$ is the covariance matrix of the $\vec{w}$ and since they stem from independent standard normal distributions is diagonal with $\sigma_w^2 = 1$ on the diagonal. So that gives

$$
E_{p(\vec{w})} [ (\vec{w}^T\vec{x})^2 ] = \vec{x}^T \vec{x} \approx 512
$$

as you are approximating the sum of variances of 512 independent samples of standard Gaussian. If you now scale the $\vec{w}$ with a constant $k$, you can do this calculation again and it will give you

$$
E_{p(\vec{w})} [ (\frac{1}{k}\vec{w}^T\vec{x})^2 ] \\
= \frac{1}{k^2} E_{p(\vec{w})} [ (\vec{w}^T\vec{x})^2 ] \\
\approx \frac{1}{k^2} 512
$$

So in order to end up with 1 in the very end, you need to scale with $k=\sqrt{512}$.

I hope this answers your question.