Why is orthogonal weights initialization so important for PPO? I have implemented PPO to solve Atari environments. For the longest time I couldn't figure out why my model would not converge as fast as other open source solutions. Eventually it boiled down to this single weights initialization (in PyTorch) for the CNN:

init_ = lambda m: init(m, nn.init.orthogonal_, lambda x: nn.init.constant_(x, 0), nn.init.calculate_gain("relu")) 

which is then applied to each of the layers. In other words, the neural network HAS to be initialized orthogonally. Once I added this, my model converges as expected.

Why is it that orthogonal initialization is so much superior to xavier initialization for PPO?
See this paper's (Exact solutions to the nonlinear dynamics of learning in
deep linear neural networks) result:

Moreover, we introduce a mathematical condition for faithful backpropagation of error signals, namely dynamical isometry, and show, surprisingly that random scaled Gaussian
initializations cannot achieve this condition despite their norm-preserving nature, while greedy pre-training
and random orthogonal initialization can, thereby achieving depth independent learning times. Finally, we
show that the property of dynamical isometry survives to good approximation even in extremely deep nonlinear random orthogonal networks operating just beyond the edge of chaos.

I think this is an answer to your question.