Why does by torch.optim.SGD method learning rate change?
With SGD learning rate should not be changed during epochs but it is. Help me understand why it happens please and how to prevent this LR changing?

import torch
params = [torch.nn.Parameter(torch.randn(1, 1))]
optimizer = torch.optim.SGD(params, lr=0.9)
scheduler = torch.optim.lr_scheduler.StepLR(optimizer, 1, gamma=0.9)
for epoch in range(5):
    print(scheduler.get_lr())
    scheduler.step()


Output is:

[0.9]
[0.7290000000000001]
[0.6561000000000001]
[0.5904900000000002]
[0.5314410000000002]


My torch version is 1.4.0

Since you are using the command  torch.optim.lr_scheduler.StepLR(optimizer, 1, gamma=0.9) (meaning actually torch.optim.lr_scheduler.StepLR(optimizer, step_size=1, gamma=0.9)) thus you are multiplying the learning rate by gamma=0.9 every step_size=1 step:


0.9     = 0.9
0.729   = 0.9*0.9*0.9
0.6561  = 0.9*0.9*0.9*0.9
0.59049 = 0.9*0.9*0.9*0.9*0.9


The only "strange" point is that it missing 0.81=0.9*0.9 at the second step (UPDATE: see Szymon Maszke answer for an explanation)

To prevent early decreasing, if you have N samples in your dataset, and the batch size is D, then set torch.optim.lr_scheduler.StepLR(optimizer, step_size=N/D, gamma=0.9) to decrease at each epoch. To decrease each E epoch set torch.optim.lr_scheduler.StepLR(optimizer, step_size=E*N/D, gamma=0.9)
