Optimize Custom Parameters in Pytorch
I'm trying to create some custom parameters to optimize and came across this helpful link here. However, I'm a bit confused as to why this code works.

Here is the code with some slight modifications from the original post to clearly see the behavior of the optimization.

import torch
import torch.nn as nn
import torch.nn.functional as F
import numpy as np

class Mask(nn.Module):
    def __init__(self):
        super(Mask, self).__init__()
        self.weight = torch.nn.Parameter(data=torch.Tensor(5, 5), requires_grad=True)

        self.weight.data.uniform_(-1, 1)

    def forward(self, x):
        masked_wt = self.weight.mul(1)
        return masked_wt


class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.mask = Mask()

    def forward(self,x):
        x = self.mask(x)
        return x

model = Model()
indata = torch.ones(5,5)
optimizer = torch.optim.Adam(model.parameters(),lr=0.001)

while True:
    x = model(indata)
    optimizer.zero_grad()
    loss = F.l1_loss(indata,x)
    loss.backward()
    optimizer.step()

    print(model.mask.weight)


So two questions:


In the forward function of the Mask class, why do I need to do self.weight.mul(1) and why doesn't x get used?
x = model(indata) clearly does per element multiplication between the 5x5 matrix indata and the weight, but how does this happen if we didn't use x in the forward function of Mask?


You are right, Mask.forward discard x completely. However, the output of your model "sees" indata when computing the loss.
What you are actually teaching your model is to have mask.weight == indata.
