Why is it so common to initialize weights with a Guassian distribution divided by the square root of number of neurons in a layer?
I think they use the Xavier/Glorot's initialization method. You can read from the original paper:

We initialized the biases to be 0 and the weights $W_{ij}$ at each layer with the following commonly used heuristic:
$W_{ij} \sim U [ -\frac{1}{\sqrt{n}}, \frac{1}{\sqrt{n}}] $
where $U[−a, a]$ is the uniform distribution in the interval $(−a, a)$ and $n$ is the size of the previous layer (the number of columns of $W$)

Some people use this as some reports said this initialization method lead to better result