Restarting with adam
I am training my network with early stopping strategy. I start with a higher learning rate, and based on validation loss, I need to restart training from an earlier snapshot.

I am able to save/load snapshot with model and optimizer state_dicts. No problem with that.

My question is, once I restart training, how do I set the learning rate of adam again? Should I restart adam fresh instead of using a state_dict or should I use
optimizer.param_groups[0][‘lr’] = lr to adjust learning rate with loaded optimizer state_dict?

For example,
I train my network with lr = 1e-6 for 5 epochs, saved model and optimizer state_dict.
I am now restarting from epoch 6, but I need lr = 1e-7 instead. What is the best approach for this?

Thanks!

Looking further into the scheduler code, I found the correct way to do it as:

def get_lr(gamma, optimizer):
    return [group['lr'] * gamma
            for group in optimizer.param_groups]

for param_group, lr in zip(optimizer.param_groups, get_lr(gamma, optimizer)):
    param_group['lr'] = lr

