<p dir="auto">I think there may be a bug with some of the LR schedulers eg StepLR.  essentially if the epoch is a multiple of the step size, the LR will keep multiplying by gamma, even if the epoch never increases:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="sched = torch.optim.lr_scheduler.StepLR(opt, 2, 2.)  # step size = 2, gamma = 2
for i in range(5):
    opt.step()
    opt.step()
    sched.step(epoch=2)  # fix the epoch
    print(opt.state_dict()['param_groups'][0]['lr'])

2.0
4.0
8.0
16.0
32.0"><pre class="notranslate"><span class="pl-s1">sched</span> <span class="pl-c1">=</span> <span class="pl-s1">torch</span>.<span class="pl-s1">optim</span>.<span class="pl-s1">lr_scheduler</span>.<span class="pl-v">StepLR</span>(<span class="pl-s1">opt</span>, <span class="pl-c1">2</span>, <span class="pl-c1">2.</span>)  <span class="pl-c"># step size = 2, gamma = 2</span>
<span class="pl-k">for</span> <span class="pl-s1">i</span> <span class="pl-c1">in</span> <span class="pl-en">range</span>(<span class="pl-c1">5</span>):
    <span class="pl-s1">opt</span>.<span class="pl-en">step</span>()
    <span class="pl-s1">opt</span>.<span class="pl-en">step</span>()
    <span class="pl-s1">sched</span>.<span class="pl-en">step</span>(<span class="pl-s1">epoch</span><span class="pl-c1">=</span><span class="pl-c1">2</span>)  <span class="pl-c"># fix the epoch</span>
    <span class="pl-en">print</span>(<span class="pl-s1">opt</span>.<span class="pl-en">state_dict</span>()[<span class="pl-s">'param_groups'</span>][<span class="pl-c1">0</span>][<span class="pl-s">'lr'</span>])

<span class="pl-c1">2.0</span>
<span class="pl-c1">4.0</span>
<span class="pl-c1">8.0</span>
<span class="pl-c1">16.0</span>
<span class="pl-c1">32.0</span></pre></div>
<p dir="auto">versus if epoch % step_size != 0:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="sched = torch.optim.lr_scheduler.StepLR(opt, 2, 2.)  # step size = 2, gamma = 2
for i in range(5):
    opt.step()
    opt.step()
    sched.step(epoch=3)  # fix the epoch
    print(opt.state_dict()['param_groups'][0]['lr'])
1.0
1.0
1.0
1.0
1.0"><pre class="notranslate"><span class="pl-s1">sched</span> <span class="pl-c1">=</span> <span class="pl-s1">torch</span>.<span class="pl-s1">optim</span>.<span class="pl-s1">lr_scheduler</span>.<span class="pl-v">StepLR</span>(<span class="pl-s1">opt</span>, <span class="pl-c1">2</span>, <span class="pl-c1">2.</span>)  <span class="pl-c"># step size = 2, gamma = 2</span>
<span class="pl-k">for</span> <span class="pl-s1">i</span> <span class="pl-c1">in</span> <span class="pl-en">range</span>(<span class="pl-c1">5</span>):
    <span class="pl-s1">opt</span>.<span class="pl-en">step</span>()
    <span class="pl-s1">opt</span>.<span class="pl-en">step</span>()
    <span class="pl-s1">sched</span>.<span class="pl-en">step</span>(<span class="pl-s1">epoch</span><span class="pl-c1">=</span><span class="pl-c1">3</span>)  <span class="pl-c"># fix the epoch</span>
    <span class="pl-en">print</span>(<span class="pl-s1">opt</span>.<span class="pl-en">state_dict</span>()[<span class="pl-s">'param_groups'</span>][<span class="pl-c1">0</span>][<span class="pl-s">'lr'</span>])
<span class="pl-c1">1.0</span>
<span class="pl-c1">1.0</span>
<span class="pl-c1">1.0</span>
<span class="pl-c1">1.0</span>
<span class="pl-c1">1.0</span></pre></div>
<p dir="auto">IIUC, calling <code class="notranslate">sched.step(epoch=i)</code> multiple times should be idempotent.  Note that this seems to have been introduced in 1.1; 1.0 works as intended.</p>
<p dir="auto">Pytorch 1.1.0<br>
Python 3.6.5</p>
<p dir="auto">When the StepLR, MultiStepLR, ExponentialLR or CosineAnnealingLR scheduler is called with the same epoch parameter the optimizer value is further reduced even though it's the same epoch</p>
<p dir="auto">a sample code</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="import torch.optim as optim
from torch import nn

conv = nn.Conv2d(3,3,3)
optimizer = optim.Adam(conv.parameters()) 
lr_scheduler = optim.lr_scheduler.StepLR(optimizer, 2)"><pre class="notranslate"><code class="notranslate">import torch.optim as optim
from torch import nn

conv = nn.Conv2d(3,3,3)
optimizer = optim.Adam(conv.parameters()) 
lr_scheduler = optim.lr_scheduler.StepLR(optimizer, 2)
</code></pre></div>
<p dir="auto">now if we call <code class="notranslate">lr_scheduler.step(epoch=2)</code> multiple times</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="for _ in range(10):
    lr_scheduler.step(2)
    print(optimizer.param_groups[0]['lr'])"><pre class="notranslate"><code class="notranslate">for _ in range(10):
    lr_scheduler.step(2)
    print(optimizer.param_groups[0]['lr'])
</code></pre></div>
<p dir="auto">output:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="&gt;&gt;&gt; 0.0001
&gt;&gt;&gt; 1e-05
&gt;&gt;&gt; 1.0000000000000002e-06
&gt;&gt;&gt; 1.0000000000000002e-07
&gt;&gt;&gt; 1.0000000000000004e-08
&gt;&gt;&gt; 1.0000000000000005e-09
&gt;&gt;&gt; 1.0000000000000006e-10
&gt;&gt;&gt; 1.0000000000000006e-11
&gt;&gt;&gt; 1.0000000000000006e-12
&gt;&gt;&gt; 1.0000000000000007e-13"><pre class="notranslate"><code class="notranslate">&gt;&gt;&gt; 0.0001
&gt;&gt;&gt; 1e-05
&gt;&gt;&gt; 1.0000000000000002e-06
&gt;&gt;&gt; 1.0000000000000002e-07
&gt;&gt;&gt; 1.0000000000000004e-08
&gt;&gt;&gt; 1.0000000000000005e-09
&gt;&gt;&gt; 1.0000000000000006e-10
&gt;&gt;&gt; 1.0000000000000006e-11
&gt;&gt;&gt; 1.0000000000000006e-12
&gt;&gt;&gt; 1.0000000000000007e-13
</code></pre></div>
<p dir="auto">Even if such use-case is bizarre, this is extremely unexpected.<br>
This is happens using PyTorch version 1.1.0 but not 1.0.1. Because pull(<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="380976590" data-permission-text="Title is private" data-url="https://github.com/pytorch/pytorch/issues/14010" data-hovercard-type="pull_request" data-hovercard-url="/pytorch/pytorch/pull/14010/hovercard" href="https://github.com/pytorch/pytorch/pull/14010">#14010</a>) redefined StepLR, MultiStepLR, ExponentialLR and CosineAnnealingLR to directly use the learning rate variable of the optimizer rather than using <code class="notranslate">base_lr</code> defined by <code class="notranslate">_LRScheduler</code>. This was in order to support multiple simultaneous schedulers (<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="373244157" data-permission-text="Title is private" data-url="https://github.com/pytorch/pytorch/issues/13022" data-hovercard-type="issue" data-hovercard-url="/pytorch/pytorch/issues/13022/hovercard" href="https://github.com/pytorch/pytorch/issues/13022">#13022</a>)!</p>