Gradient of NN output with respect to inputs
Yes there is. This is called "back-propagation to the input". I would like to invite you to read this awesome blog which relies on lucid. You will see some code on their notebooks.