During neural network training, can gradients leak sensitive information in case training data fed is encrypted (homomorphic)?
It will recover the encrypted inputs.
The algorithm starts with dummy data and dummy labels, and then iteratively optimizes the dummy gradients to be close as to the original. This makes the dummy data close to the real training data:
$$\mathbf{x}^{\prime *}, \mathbf{y}^{\prime *}=\underset{\mathbf{x}^{\prime}, \mathbf{y}^{\prime}}{\arg \min }\left\|\nabla W^{\prime}-\nabla W\right\|^{2}=\underset{\mathbf{x}^{\prime}, \mathbf{y}^{\prime}}{\arg \min }\left\|\frac{\partial \ell\left(F\left(\mathbf{x}^{\prime}, W\right), \mathbf{y}^{\prime}\right)}{\partial W}-\nabla W\right\|^{2}$$
As the distance is minimized, the algorithm restores the original training data; in the case of encrypted training data - you should get an encrypted input (up to failures when the resulted 'input' isn't close to the original input).