Can someone explain R1 regularization function in simple terms?
Here is how I understand this regularization.
$R_1$ is simply the norm of the gradients, which indicates how fast the weights will be updated. Gradient regularization penalizes large changes in the output of some neural network layer.
$$
R_{1}\left(\psi\right) = \frac{\gamma}{2}E_{p_{D}\left(x\right)}\left[||\nabla{D_{\psi}\left(x\right)}||^{2}\right]\text{,}
$$
where $\psi$ is discriminator weights, $E_{p_{D}\left(x\right)}$ means that we sample data only form the real distribution (i.e. only real images) and $\gamma$ is a hyperparameter.
Since we don't know if $G$ can already generate data from real distribution, we apply this regularization to $D$ only on real data, because we don't want the discriminator to create a non-zero gradient without suffering a loss if we are already in a Nash Equilibrium. I guess this also prevents $G$ from updating if it generates data from the real distribution.
The authors also investigate which value is best for $\gamma$ by analyzing the eigenvalues of the Jacobian of the the associated gradient vector field, but in my opinion, this value is highly dependent on dataset and architecture.
&quot;Gradient orthogonal to the data manifold&quot; simply means zero gradients. From a GAN perspective, the data manifold is a lower-dimensional latent features manifold embedded in a higher-dimensional space and our goal is to approximate it. Since the gradient vector shows the direction in which we need to update our function, if it is orthogonal to this manifold, we do not need to update the function.