title: Why PyTorch optimizer might fail to update its parameters?
body: <p>I am trying to do a simple loss-minimization for a specific variable <code>coeff</code> using PyTorch optimizers. This variable is supposed to be used as an interpolation coefficient for two vectors <code>w_foo</code> and <code>w_bar</code> to find a third vector, <code>w_target</code>.</p>
  <pre><code>w_target = `w_foo + coeff * (w_bar - w_foo)
  </code></pre>
  <p>With <code>w_foo</code> and <code>w_bar</code> set as constant, at each optimization step I calculate <code>w_target</code> for the given <code>coeff</code>. Loss is determined from <code>w_target</code> using a fairly complex process beyond the scope of this question.</p>
  <pre><code># w_foo.shape = [1, 16, 512]
  # w_bar.shape = [1, 16, 512]
  # num_layers = 16
  # num_steps = 10000
  
  vgg_loss = VGGLoss()
  
  coeff = torch.randn([num_layers, ])
  optimizer = torch.optim.Adam([coeff], lr=initial_learning_rate)
  
  for step in range(num_steps):
      w_target = w_foo + torch.matmul(coeff, (w_bar - w_foo))
      optimizer.zero_grad()
      target_image = generator.synthesis(w_target)
      processed_target_image = process(target_image)
      loss = vgg_loss(processed_target_image, source_image)
      loss.backward()
      optimizer.step()
  </code></pre>
  <p>However, when running this optimizer, I am met with <code>query_opt</code> not changing from one step to another, optimizer being essentially useless. I would like to ask for some advice on what I am doing wrong here.</p>
  <p>Edit:
  As suggested, I will try to elaborate on the loss function. Essentially, <code>w_target</code> is used to generate an image, and VGGLoss uses VGG feature extractor to compare this synthetic image with a certain exemplar source image.</p>
  <pre><code>class VGGLoss(torch.nn.Module):
      def __init__(self, device, vgg):
          super().__init__()
          
          for param in self.parameters():
              param.requires_grad = True
  
          self.vgg = vgg # VGG16 in eval mode
          
      def forward(self, source, target):
          loss = 0 
          source_features = self.vgg(source, resize_images=False, return_lpips=True)
          target_features = self.vgg(target, resize_images=False, return_lpips=True)
          loss += (source_features - target_features).square().sum()
  
          return loss 
  </code></pre>