title: Find PyTorch model parameters that don't contribute to loss
body: <p>In PyTorch (v1.10) Distibuted DataParallel, unused parameters in a model that don't contribute to the final loss can raise a RuntimeError (as mentioned in <a href="https://stackoverflow.com/questions/67783991/pytorch-runtimeerror-expected-to-have-finished-reduction-in-the-prior-iteratio">this other question</a>, <a href="https://discuss.pytorch.org/t/need-help-runtimeerror-expected-to-have-finished-reduction-in-the-prior-iteration-before-starting-a-new-one/119247" rel="nofollow noreferrer">this PyTorch forums thread</a>).</p>
  <blockquote>
  <p><em>&quot;RuntimeError: Expected to have finished reduction in the prior iteration before starting a new one. This error indicates that your module has parameters that were not used in producing loss. You can enable unused parameter detection by passing the keyword argument <code>find_unused_parameters=True</code> to <code>torch.nn.parallel.DistributedDataParallel</code>, and by making sure all <code>forward</code> function outputs participate in calculating loss.&quot;</em></p>
  </blockquote>
  <p>Although it's possible to inspect which parameters are affected at error-time (as mentioned above, or setting env var <code>TORCH_DISTRIBUTED_DEBUG=&quot;INFO&quot;</code>), it seems like there should be a way to statically inspect a model to locate (and presumably prune or disable gradient on) parameters that aren't contributing to the current loss objective?</p>
  <p>So given a <code>torch.nn.Module</code>-based <code>model</code> whose <code>forward()</code> function returns some <code>loss</code> tensor (maybe alongside others) - How can we programmatically, before starting to train, find all parameters (including nested modules) that aren't contributing to <code>loss</code>?</p>