Is there any model agnostic way to calculate the weight importance for neural networks given a set of inputs?
You should look into fisher matrices. Elastic weight consolidation finds weights which are important for dataset 1 when training dataset 2 by finding the fisher matrix and slows the tuning of them down by adding a regularization term to the loss function. It's non trivial though and I got a lot of trouble implementing it in PyTorch.
The paper which proposed elastic weight consolidation is here: https://arxiv.org/pdf/1612.00796.pdf