In the MINE paper, why is $\hat{G}_B$ biased, and how does the exponential moving average reduce the bias?
The lower bound in MINE is as follows:
$$\widehat{I(X;Z)}_n = \sup_{\theta\in\Theta} \mathbb{E}_{\mathbb{P}_{XZ}^{(n)}}[T_\theta] - \log{\mathbb{E}_{\mathbb{P}_X^{(n)} \otimes \hat{\mathbb{P}}_Z^{(n)}}[e^{T_\theta}]}$$
Here $\mathbb{\hat{P}^{(n)}}$ denotes the empirical distribution that we get from n i.i.d samples of $\mathbb{P}.$
Note that in the above equation, the first term is calculated from the joint distribution while the second term from the marginals of $X$ and $Z$. In the implementation of MINE, these statistics are calculated over the data from a minibatch. The marginal distribution is obtained by shuffling the values of Z (or X) along the batch dimension. Hence, in this case, the gradient is as follows.
$$\hat{G}_B = \mathbb{E}_B[\nabla_{\theta}T_{\theta}] - \frac{\mathbb{E_B}[\nabla_{\theta}T_{\theta}e^{T_{\theta}}]}{\mathbb{E}_B[e^{T_{\theta}}]},$$

As mentioned, the expectation over the marginals is not calculated
over the true marginal distribution (i.e. over the entire dataset)
but from the shuffled samples in the minibatch. Hence, the above gradient $G_B$ is biased.
When we maintain an exponential moving average of
$\mathbb{E}_B[e^{T_{\theta}}]$, we kind of incorporate statistics from outside the current minibatch also (i.e. over the entire dataset). This is an attempt to get an approximation of the true marginal estimate. The denominator term in the gradient allows for such a computationally inexpensive bias reduction trick.