Should the parameters of the BatchNorm layer be regularized
When I use pytorch to train my CNN, the L2 regularization will be used to panalize the parameters in the model. But the pytorch code &quot;weight decay&quot; will use L2 to all the parameters which can be updated. Should the parameters in BatchNorm Layers be panalized by L2, too?

If you are using BatchNorm right after nn.Conv2d or nn.Linear, you can &quot;fold&quot; the learned weight and bias into the conv/linear layer.
hence, the learned weigh and bias has a direct effect on the actual L2 norm of the &quot;effective&quot; weights of your network.
Therefore, if you are using L2 regularization to push your weights towards zero - you must also regularize the batch norm parameters.
