Weight decay loss
I need to write a code to gradually decay the weight of my loss function by computes lambda with given steps, But I don't have any idea. Any help will be appreciated.

This is my Loss function:

loss_A = criterion(recov_A, real_A)
loss_Final = lambda_A * loss_A + #lambda_A is a fixed number: 10


I don't want the lambda_A to be fixed. I need to gradually decay the lambda after passing the specified number of steps

# write function that computes lambda given the steps
cur_lambda  = compute_lambda(step, decay_params, initial_lamdba) 
Loss_Final = cur_lambda * loss_A


To decay the fixed number depends on the number of steps or even the number of epochs you can use the following code or you can write the code as a function and call it whenever you want.

final_value = 1e-3 # Small number because dont end up with 0 
initial_value = 20
starting_step = 25 
total_step = 100
for i in range(total_step):
  if i &lt;= starting_step:
     print(i, initial_value)
  else:
    print (i, initial_value + i * (final_value-initial_value)/total_step) 

