Why is scale parameter on batch normalization not needed on ReLU?
Scaling in batch normalization is said not to be required because of the nature of ReLU. Although it is a non-linear operation, a multiplication with a positive scalar $\gamma$ over the values of a vector $x$ which are already normalized and centered to the trained bias $\beta$ makes no influence on its non-linear outcome: for $\gamma &gt; 0$, $max(0, \gamma \times x) = \gamma \times max(0, x)$. In fact, the extra parameter adds no representative power in the model, for the same reason that we must include non-linearities between other neuron layers. 

And so, the weights of the following layer would scale the values to fit accordingly. We could have $z = Wgx$, but that would be the same thing as $z = Wx$ if $W$ is scaled by $g$. The same reasoning can't be made for other non-linearities, such as the sigmoid or tanh, which have a saturating effect that is clearly influenced by a scalar.

Curiously, the official documentation on TensorFlow's tf.contrib.layers.batch_norm makes the same recommendation:


  
  scale: If True, multiply by gamma. If False, gamma is not used. When the next layer is linear (also e.g. nn.relu), this can be disabled since the scaling can be done by the next layer.