<h2 dir="auto"><g-emoji class="g-emoji" alias="bug" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/1f41b.png">🐛</g-emoji> Bug</h2>
<p dir="auto">Adding <code class="notranslate">epoch</code> argument to <code class="notranslate">step()</code> function of MultiStepLR lead to false learning rate.</p>
<h2 dir="auto">To Reproduce</h2>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="from torch import nn
import torch
net = nn.Linear(30, 10)
optimizer = torch.optim.Adam(net.parameters(), lr=0.001)
s = torch.optim.lr_scheduler.MultiStepLR(optimizer, [10, 20, 30], gamma=0.1)
print(s.get_lr())
s.step(1)
print(s.get_lr())"><pre class="notranslate"><span class="pl-k">from</span> <span class="pl-s1">torch</span> <span class="pl-k">import</span> <span class="pl-s1">nn</span>
<span class="pl-k">import</span> <span class="pl-s1">torch</span>
<span class="pl-s1">net</span> <span class="pl-c1">=</span> <span class="pl-s1">nn</span>.<span class="pl-v">Linear</span>(<span class="pl-c1">30</span>, <span class="pl-c1">10</span>)
<span class="pl-s1">optimizer</span> <span class="pl-c1">=</span> <span class="pl-s1">torch</span>.<span class="pl-s1">optim</span>.<span class="pl-v">Adam</span>(<span class="pl-s1">net</span>.<span class="pl-en">parameters</span>(), <span class="pl-s1">lr</span><span class="pl-c1">=</span><span class="pl-c1">0.001</span>)
<span class="pl-s1">s</span> <span class="pl-c1">=</span> <span class="pl-s1">torch</span>.<span class="pl-s1">optim</span>.<span class="pl-s1">lr_scheduler</span>.<span class="pl-v">MultiStepLR</span>(<span class="pl-s1">optimizer</span>, [<span class="pl-c1">10</span>, <span class="pl-c1">20</span>, <span class="pl-c1">30</span>], <span class="pl-s1">gamma</span><span class="pl-c1">=</span><span class="pl-c1">0.1</span>)
<span class="pl-en">print</span>(<span class="pl-s1">s</span>.<span class="pl-en">get_lr</span>())
<span class="pl-s1">s</span>.<span class="pl-en">step</span>(<span class="pl-c1">1</span>)
<span class="pl-en">print</span>(<span class="pl-s1">s</span>.<span class="pl-en">get_lr</span>())</pre></div>
<p dir="auto">Output</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="[0.001]
[1.0000000000000002e-06]"><pre class="notranslate"><code class="notranslate">[0.001]
[1.0000000000000002e-06]
</code></pre></div>
<h2 dir="auto">Expected behavior</h2>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="[0.001]
[0.001]"><pre class="notranslate"><code class="notranslate">[0.001]
[0.001]
</code></pre></div>
<h2 dir="auto">Environment</h2>
<p dir="auto">PyTorch version: 1.4.0a0+d5bf51b<br>
Is debug build: No<br>
CUDA used to build PyTorch: 9.0</p>
<p dir="auto">OS: Ubuntu 16.04.6 LTS<br>
GCC version: (Ubuntu 5.4.0-6ubuntu1~16.04.11) 5.4.0 20160609<br>
CMake version: version 3.14.0</p>
<p dir="auto">Python version: 3.6<br>
Is CUDA available: Yes<br>
CUDA runtime version: 9.0.176<br>
GPU models and configuration:<br>
GPU 0: TITAN Xp<br>
GPU 1: TITAN Xp<br>
GPU 2: TITAN Xp<br>
GPU 3: TITAN Xp</p>
<p dir="auto">Nvidia driver version: 430.26<br>
cuDNN version: Could not collect</p>
<p dir="auto">Versions of relevant libraries:</p>
<p dir="auto">[pip] numpy==1.17.3<br>
[pip] torch==1.4.0a0+d5bf51b<br>
[conda] blas                      1.0                         mkl<br>
[conda] magma-cuda90              2.5.0                         1    pytorch<br>
[conda] mkl                       2019.4                      243<br>
[conda] mkl-include               2019.4                      243<br>
[conda] mkl-service               2.3.0            py36he904b0f_0<br>
[conda] mkl_fft                   1.0.15           py36ha843d7b_0<br>
[conda] mkl_random                1.1.0            py36hd6b4f25_0<br>
[conda] torch                     1.4.0a0+d5bf51b          pypi_0    pypi</p>
<h2 dir="auto">Additional context</h2>
<p dir="auto">Possible cause might be that the milestones of MultiStepLR is a <code class="notranslate">counter</code> rather then a list, which leads to false action of <code class="notranslate">bisect</code> in <code class="notranslate">get_lr</code> function.</p>
<p dir="auto">cc <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/vincentqb/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/vincentqb">@vincentqb</a></p>
<h2 dir="auto"><g-emoji class="g-emoji" alias="bug" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/1f41b.png">🐛</g-emoji> Bug</h2>
<p dir="auto">MultiStepLR is broken with <code class="notranslate">.step(epoch=xyz)</code>:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="import torch
t = torch.tensor(1.0, requires_grad=True)
opt = torch.optim.SGD([t], lr=0.01)
s = torch.optim.lr_scheduler.MultiStepLR(opt, milestones=[19])
print('lr', opt.param_groups[0]['lr'])

opt.step()
s.step(0)
print('lr', opt.param_groups[0]['lr'])"><pre class="notranslate"><code class="notranslate">import torch
t = torch.tensor(1.0, requires_grad=True)
opt = torch.optim.SGD([t], lr=0.01)
s = torch.optim.lr_scheduler.MultiStepLR(opt, milestones=[19])
print('lr', opt.param_groups[0]['lr'])

opt.step()
s.step(0)
print('lr', opt.param_groups[0]['lr'])
</code></pre></div>
<p dir="auto">Outputs:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="lr 0.01
lr 0.001"><pre class="notranslate"><code class="notranslate">lr 0.01
lr 0.001
</code></pre></div>
<p dir="auto">I'm aware that <code class="notranslate">.step(epoch=xyz)</code> is deprecated, but it's used by third-party libraries, like <a href="https://github.com/PyTorchLightning/pytorch-lightning/blob/master/pytorch_lightning/trainer/training_loop.py#L341">https://github.com/PyTorchLightning/pytorch-lightning/blob/master/pytorch_lightning/trainer/training_loop.py#L341</a>.</p>
<p dir="auto">cc <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/vincentqb/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/vincentqb">@vincentqb</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/williamFalcon/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/williamFalcon">@williamFalcon</a> ^</p>
<p dir="auto">I've debugged the issue and it seems to be caused by <code class="notranslate">bisect_right()</code> not playing well with <code class="notranslate">Counter</code>: <a href="https://github.com/pytorch/pytorch/blob/master/torch/optim/lr_scheduler.py#L399">https://github.com/pytorch/pytorch/blob/master/torch/optim/lr_scheduler.py#L399</a></p>
<p dir="auto">This can be fixed by casting self.milestones to list before bisection.  If that's an acceptable fix, I'm happy to send a PR.</p>
<h2 dir="auto">Environment</h2>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Collecting environment information...
PyTorch version: 1.4.0
Is debug build: No
CUDA used to build PyTorch: 10.1

OS: Ubuntu 18.04.3 LTS
GCC version: (Ubuntu 4.8.5-4ubuntu8) 4.8.5
CMake version: version 3.12.1

Python version: 3.6
Is CUDA available: Yes
CUDA runtime version: 10.1.243
GPU models and configuration:
GPU 0: GeForce RTX 2080 Ti
GPU 1: GeForce RTX 2080 Ti

Nvidia driver version: 440.33.01
cuDNN version: /usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.3

Versions of relevant libraries:
[pip3] numpy==1.17.2
[pip3] pytorch-lightning==0.5.3.2
[pip3] torch==1.4.0
[pip3] torch2trt==0.0.3
[pip3] torchvision==0.6.0a0
[conda] Could not collect"><pre class="notranslate"><code class="notranslate">Collecting environment information...
PyTorch version: 1.4.0
Is debug build: No
CUDA used to build PyTorch: 10.1

OS: Ubuntu 18.04.3 LTS
GCC version: (Ubuntu 4.8.5-4ubuntu8) 4.8.5
CMake version: version 3.12.1

Python version: 3.6
Is CUDA available: Yes
CUDA runtime version: 10.1.243
GPU models and configuration:
GPU 0: GeForce RTX 2080 Ti
GPU 1: GeForce RTX 2080 Ti

Nvidia driver version: 440.33.01
cuDNN version: /usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.3

Versions of relevant libraries:
[pip3] numpy==1.17.2
[pip3] pytorch-lightning==0.5.3.2
[pip3] torch==1.4.0
[pip3] torch2trt==0.0.3
[pip3] torchvision==0.6.0a0
[conda] Could not collect
</code></pre></div>