How to compute the parameter importance in pytorch?
I want to develop a lifelong learning system,so i need to prevent important parameter from changing.I read related paper 'Memory Aware Synapses: Learning what (not) to forget',a method was mentioned,I need to calculate the gradient of each parameter conresponding to each input image,so how should i write my code in pytorch?
'Memory Aware Synapses: Learning what (not) to forget'

You can do it using standard optimization procedure and .backward() method on your loss function.
First, scaling as defined in your link:
class Scaler:
    def __init__(self, parameters, delta):
        self.parameters = parameters
        self.delta = delta

    def step(self):
        &quot;&quot;&quot;Multiplies gradients in place.&quot;&quot;&quot;
        for param in self.parameters:
            if param.grad is None:
                raise ValueError(&quot;backward() has to be called before running scaler&quot;)

            param.grad *= self.delta

One can use it just like optimizer.step(), see below (see comments):
model = torch.nn.Sequential(
    torch.nn.Linear(10, 100), torch.nn.ReLU(), torch.nn.Linear(100, 1)
)

scaler = Scaler(model.parameters(), delta=0.001)
optimizer = torch.optim.Adam(model.parameters())
criterion = torch.nn.MSELoss()

X, y = torch.randn(64, 10), torch.randn(64)

# Optimization loop
EPOCHS = 10
for _ in range(EPOCHS):
    output = model(X)
    loss = criterion(output, y)

    loss.backward()  # Now model has the gradients

    optimizer.step()  # Optimize model's parameters

    print(next(model.parameters()).grad)

    scaler.step()  # Scaler gradients

    optimizer.zero_grad()  # Zero gradient before next step 

After scaler.step() you will have gradient scaled available inside param.grad for each parameter (just like those are accessed within Scaler's step method) so you can do whatever you want with them.
