<h2 dir="auto"><g-emoji class="g-emoji" alias="bug" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/1f41b.png">🐛</g-emoji> Bug</h2>

<p dir="auto">The CyclicLR scheduler tries to access the optimizer's momentum parameter even if <code class="notranslate">cycle_momentum=False</code>.</p>
<h2 dir="auto">To Reproduce</h2>
<p dir="auto">Steps to reproduce the behavior:</p>
<ol dir="auto">
<li>Setup a model named <code class="notranslate">net</code></li>
<li>Run the following:</li>
</ol>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="import torch.optim as optim
optimizer = optim.Adam(net.parameters(), lr=0.001)
scheduler = optim.lr_scheduler.CyclicLR(optimizer, base_lr=0.0001, max_lr=0.01, cycle_momentum=False)"><pre class="notranslate"><code class="notranslate">import torch.optim as optim
optimizer = optim.Adam(net.parameters(), lr=0.001)
scheduler = optim.lr_scheduler.CyclicLR(optimizer, base_lr=0.0001, max_lr=0.01, cycle_momentum=False)
</code></pre></div>

<p dir="auto">Resulting error:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="KeyError                                  Traceback (most recent call last)
&lt;ipython-input-113-a9df59301cc9&gt; in &lt;module&gt;
      1 optimizer = optim.Adam(net.parameters(), lr=0.001)
----&gt; 2 scheduler = optim.lr_scheduler.CyclicLR(optimizer, base_lr=0.0001, max_lr=0.01, cycle_momentum=False)

~/.pyenv/versions/3.6.6/lib/python3.6/site-packages/torch/optim/lr_scheduler.py in __init__(self, optimizer, base_lr, max_lr, step_size_up, step_size_down, mode, gamma, scale_fn, scale_mode, cycle_momentum, base_momentum, max_momentum, last_epoch)
    591                 for momentum, group in zip(base_momentums, optimizer.param_groups):
    592                     group['momentum'] = momentum
--&gt; 593         self.base_momentums = list(map(lambda group: group['momentum'], optimizer.param_groups))
    594         self.max_momentums = self._format_param('max_momentum', optimizer, max_momentum)
    595 

~/.pyenv/versions/3.6.6/lib/python3.6/site-packages/torch/optim/lr_scheduler.py in &lt;lambda&gt;(group)
    591                 for momentum, group in zip(base_momentums, optimizer.param_groups):
    592                     group['momentum'] = momentum
--&gt; 593         self.base_momentums = list(map(lambda group: group['momentum'], optimizer.param_groups))
    594         self.max_momentums = self._format_param('max_momentum', optimizer, max_momentum)
    595 

KeyError: 'momentum'"><pre class="notranslate"><code class="notranslate">KeyError                                  Traceback (most recent call last)
&lt;ipython-input-113-a9df59301cc9&gt; in &lt;module&gt;
      1 optimizer = optim.Adam(net.parameters(), lr=0.001)
----&gt; 2 scheduler = optim.lr_scheduler.CyclicLR(optimizer, base_lr=0.0001, max_lr=0.01, cycle_momentum=False)

~/.pyenv/versions/3.6.6/lib/python3.6/site-packages/torch/optim/lr_scheduler.py in __init__(self, optimizer, base_lr, max_lr, step_size_up, step_size_down, mode, gamma, scale_fn, scale_mode, cycle_momentum, base_momentum, max_momentum, last_epoch)
    591                 for momentum, group in zip(base_momentums, optimizer.param_groups):
    592                     group['momentum'] = momentum
--&gt; 593         self.base_momentums = list(map(lambda group: group['momentum'], optimizer.param_groups))
    594         self.max_momentums = self._format_param('max_momentum', optimizer, max_momentum)
    595 

~/.pyenv/versions/3.6.6/lib/python3.6/site-packages/torch/optim/lr_scheduler.py in &lt;lambda&gt;(group)
    591                 for momentum, group in zip(base_momentums, optimizer.param_groups):
    592                     group['momentum'] = momentum
--&gt; 593         self.base_momentums = list(map(lambda group: group['momentum'], optimizer.param_groups))
    594         self.max_momentums = self._format_param('max_momentum', optimizer, max_momentum)
    595 

KeyError: 'momentum'
</code></pre></div>
<h2 dir="auto">Expected behavior</h2>

<p dir="auto">I expect the scheduler not try to access the <code class="notranslate">momentum</code> key if I've told it not to cycle the momentum parameter. The error seems to be due to a minor indentation mistake here:</p>
<p dir="auto"><a href="https://github.com/pytorch/pytorch/blob/master/torch/optim/lr_scheduler.py#L594-L595">https://github.com/pytorch/pytorch/blob/master/torch/optim/lr_scheduler.py#L594-L595</a></p>
<p dir="auto">The instance variables <code class="notranslate">base_momentums</code> and <code class="notranslate">max_momentums</code> are only ever accessed if <code class="notranslate">cycle_momentum</code> is true. Thus the two lines where they're being should be indented.</p>
<h2 dir="auto">Environment</h2>
<p dir="auto">N/A</p>
<h2 dir="auto">Additional comments</h2>
<p dir="auto">I'll push a PR to fix this in a minute, though may need a little help with PyTorch's specific contribution process.</p>
<h2 dir="auto">Issue description</h2>
<p dir="auto">If I use an optimizer like adam with no momentum and follow the message of line 578 below here, passing <code class="notranslate">cycle_momentum=False</code>, line 584 throws a KeyError because key 'momentum' is not set.</p>
<p dir="auto"></p><div class="Box Box--condensed my-2">
  <div class="Box-header f6">
    <p class="mb-0 text-bold">
      <a href="https://github.com/pytorch/pytorch/blob/173f224570017b4b1a3a1a13d0bff280a54d9cd9/torch/optim/lr_scheduler.py#L576-L585">pytorch/torch/optim/lr_scheduler.py</a>
    </p>
    <p class="mb-0 color-fg-muted">
        Lines 576 to 585
      in
      <a data-pjax="true" class="commit-tease-sha" href="/pytorch/pytorch/commit/173f224570017b4b1a3a1a13d0bff280a54d9cd9">173f224</a>
    </p>
  </div>
  <div itemprop="text" class="Box-body p-0 blob-wrapper blob-wrapper-embedded data">
    <table class="highlight tab-size mb-0 js-file-line-container" data-tab-size="8" data-paste-markdown-skip="">

        <tbody><tr class="border-0">
          <td id="L576" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="576"></td>
          <td id="LC576" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line"> <span class="pl-k">if</span> <span class="pl-s1">cycle_momentum</span>: </td>
        </tr>

        <tr class="border-0">
          <td id="L577" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="577"></td>
          <td id="LC577" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">     <span class="pl-k">if</span> <span class="pl-s">'momentum'</span> <span class="pl-c1">not</span> <span class="pl-c1">in</span> <span class="pl-s1">optimizer</span>.<span class="pl-s1">defaults</span>: </td>
        </tr>

        <tr class="border-0">
          <td id="L578" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="578"></td>
          <td id="LC578" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">         <span class="pl-k">raise</span> <span class="pl-v">ValueError</span>(<span class="pl-s">'optimizer must support momentum with `cycle_momentum` option enabled'</span>) </td>
        </tr>

        <tr class="border-0">
          <td id="L579" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="579"></td>
          <td id="LC579" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">  </td>
        </tr>

        <tr class="border-0">
          <td id="L580" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="580"></td>
          <td id="LC580" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">     <span class="pl-s1">base_momentums</span> <span class="pl-c1">=</span> <span class="pl-s1">self</span>.<span class="pl-en">_format_param</span>(<span class="pl-s">'base_momentum'</span>, <span class="pl-s1">optimizer</span>, <span class="pl-s1">base_momentum</span>) </td>
        </tr>

        <tr class="border-0">
          <td id="L581" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="581"></td>
          <td id="LC581" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">     <span class="pl-k">if</span> <span class="pl-s1">last_epoch</span> <span class="pl-c1">==</span> <span class="pl-c1">-</span><span class="pl-c1">1</span>: </td>
        </tr>

        <tr class="border-0">
          <td id="L582" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="582"></td>
          <td id="LC582" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">         <span class="pl-k">for</span> <span class="pl-s1">momentum</span>, <span class="pl-s1">group</span> <span class="pl-c1">in</span> <span class="pl-en">zip</span>(<span class="pl-s1">base_momentums</span>, <span class="pl-s1">optimizer</span>.<span class="pl-s1">param_groups</span>): </td>
        </tr>

        <tr class="border-0">
          <td id="L583" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="583"></td>
          <td id="LC583" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">             <span class="pl-s1">group</span>[<span class="pl-s">'momentum'</span>] <span class="pl-c1">=</span> <span class="pl-s1">momentum</span> </td>
        </tr>

        <tr class="border-0">
          <td id="L584" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="584"></td>
          <td id="LC584" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line"> <span class="pl-s1">self</span>.<span class="pl-s1">base_momentums</span> <span class="pl-c1">=</span> <span class="pl-en">list</span>(<span class="pl-en">map</span>(<span class="pl-k">lambda</span> <span class="pl-s1">group</span>: <span class="pl-s1">group</span>[<span class="pl-s">'momentum'</span>], <span class="pl-s1">optimizer</span>.<span class="pl-s1">param_groups</span>)) </td>
        </tr>

        <tr class="border-0">
          <td id="L585" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="585"></td>
          <td id="LC585" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line"> <span class="pl-s1">self</span>.<span class="pl-s1">max_momentums</span> <span class="pl-c1">=</span> <span class="pl-s1">self</span>.<span class="pl-en">_format_param</span>(<span class="pl-s">'max_momentum'</span>, <span class="pl-s1">optimizer</span>, <span class="pl-s1">max_momentum</span>) </td>
        </tr>
    </tbody></table>
  </div>
</div>
<p></p>
<ol dir="auto">
<li>I believe the last two lines should be indented under the if-condition as well.</li>
<li>I also think the expected behavior is that cycle_momentum is set to False by default when the passed in optimizer doesn't support momentum, instead of throwing an error.</li>
</ol>
<h2 dir="auto">Code example</h2>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="optimizer = optim.Adam(params_to_update, lr=5e-4)
scheduler = CyclicLR(optimizer, base_lr=5e-6, max_lr=5e-2, cycle_momentum=False, step_size_up=2500)"><pre class="notranslate"><code class="notranslate">optimizer = optim.Adam(params_to_update, lr=5e-4)
scheduler = CyclicLR(optimizer, base_lr=5e-6, max_lr=5e-2, cycle_momentum=False, step_size_up=2500)
</code></pre></div>
<h2 dir="auto">System Info</h2>
<p dir="auto">In my setup, I am using the stable version of pytorch, but I copied over CyclicLR and relative imports from master and I am using it in my project.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Collecting environment information...
PyTorch version: 1.0.1.post2
Is debug build: No
CUDA used to build PyTorch: 10.0.130

OS: Debian GNU/Linux 9.8 (stretch)
GCC version: (Debian 6.3.0-18+deb9u1) 6.3.0 20170516
CMake version: Could not collect

Python version: 3.7
Is CUDA available: Yes
CUDA runtime version: 10.0.130
GPU models and configuration: GPU 0: Tesla P4
Nvidia driver version: 410.72
cuDNN version: Could not collect

Versions of relevant libraries:
[pip3] intel-numpy==1.15.1
[pip3] numpy==1.15.1
[pip3] torch==1.0.1.post2
[pip3] torchvision==0.2.2.post3
[conda] blas                      1.0                         mkl
[conda] mkl                       2019.1                      144
[conda] mkl-service               1.1.2            py37he904b0f_5
[conda] mkl_fft                   1.0.6            py37hd81dba3_0
[conda] mkl_random                1.0.2            py37hd81dba3_0
[conda] pytorch                   1.0.1           py3.7_cuda10.0.130_cudnn7.4.2_2    pytorch
[conda] torchvision               0.2.2                      py_3    pytorch"><pre class="notranslate"><code class="notranslate">Collecting environment information...
PyTorch version: 1.0.1.post2
Is debug build: No
CUDA used to build PyTorch: 10.0.130

OS: Debian GNU/Linux 9.8 (stretch)
GCC version: (Debian 6.3.0-18+deb9u1) 6.3.0 20170516
CMake version: Could not collect

Python version: 3.7
Is CUDA available: Yes
CUDA runtime version: 10.0.130
GPU models and configuration: GPU 0: Tesla P4
Nvidia driver version: 410.72
cuDNN version: Could not collect

Versions of relevant libraries:
[pip3] intel-numpy==1.15.1
[pip3] numpy==1.15.1
[pip3] torch==1.0.1.post2
[pip3] torchvision==0.2.2.post3
[conda] blas                      1.0                         mkl
[conda] mkl                       2019.1                      144
[conda] mkl-service               1.1.2            py37he904b0f_5
[conda] mkl_fft                   1.0.6            py37hd81dba3_0
[conda] mkl_random                1.0.2            py37hd81dba3_0
[conda] pytorch                   1.0.1           py3.7_cuda10.0.130_cudnn7.4.2_2    pytorch
[conda] torchvision               0.2.2                      py_3    pytorch
</code></pre></div>