How to view the weight_decay loss in PyTorch during the training?
I train a model with Adam optimizer in PyTorch and set the weight_decay parameter to 1.0.
optimizer = optim.Adam(model.parameters(), lr=args.lr, weight_decay=1.0)
optimizer.zero_grad()
loss.backward()
optimizer.step()

If I want to compare the number of the weight_decay loss and the model loss, how do I view the value of the loss caused by the weight_decay?

Are you familiar with L2 regularization? If not, you can study it. I find this tutorial very helpful.

There is a subtle difference between L2 regularization and weight decay and that is: 


  Weight decay is usually defined as a term that’s added directly to the update rule. On the other hand, the L2 regularization term is added to the loss function.


You may find this tutorial helpful to study the differences between weight decay and L2 regularization.

[Update] I find the lecture by Prof. Andrew Ng very helpful.
