title: How to get the actual learning rate in PyTorch?
body: <p>I'm trying to find the appropriate learning rate for my Neural Network using PyTorch. I've implemented the <code>torch.optim.lr_scheduler.CyclicLR</code> to get the learning rate. But I'm unable to figure out what is the actual learning rate that should be selected. The dataset is MNIST_TINY.</p>
  <p><strong>Code:</strong></p>
  <pre><code>optimizer = torch.optim.SGD(model.parameters(), lr=0.1)
  scheduler = torch.optim.lr_scheduler.CyclicLR(optimizer, base_lr=1e-7, max_lr=0.1, step_size_up=5., mode='triangular')
  
  lrs= []
  for epoch in range(5):
      model.train()
      for data, label in train_dls:
          optimizer.zero_grad()
          target = model(data)
          train_step_loss = loss_fn(target, label) #CrossEntropyLoss
          train_step_loss.backward()
          optimizer.step()
          print(f'Epoch:{epoch+1} | Optim:{optimizer.param_groups[0][&quot;lr&quot;]:.4f} | Loss: {train_step_loss:.2f}')
          lrs.append(optimizer.param_groups[0][&quot;lr&quot;])
      scheduler.step()
  </code></pre>
  <p><strong>Output</strong></p>
  <pre><code>Epoch:1 | Optim:0.0000 | Loss: 0.70
  Epoch:1 | Optim:0.0000 | Loss: 0.70
  Epoch:1 | Optim:0.0000 | Loss: 0.70
  Epoch:1 | Optim:0.0000 | Loss: 0.70
  Epoch:1 | Optim:0.0000 | Loss: 0.70
  Epoch:1 | Optim:0.0000 | Loss: 0.69
  Epoch:1 | Optim:0.0000 | Loss: 0.69
  Epoch:1 | Optim:0.0000 | Loss: 0.68
  Epoch:1 | Optim:0.0000 | Loss: 0.69
  Epoch:1 | Optim:0.0000 | Loss: 0.69
  Epoch:1 | Optim:0.0000 | Loss: 0.69
  Epoch:1 | Optim:0.0000 | Loss: 0.72
  Epoch:2 | Optim:0.0200 | Loss: 0.70
  Epoch:2 | Optim:0.0200 | Loss: 0.70
  Epoch:2 | Optim:0.0200 | Loss: 0.70
  Epoch:2 | Optim:0.0200 | Loss: 0.70
  Epoch:2 | Optim:0.0200 | Loss: 0.69
  Epoch:2 | Optim:0.0200 | Loss: 0.69
  Epoch:2 | Optim:0.0200 | Loss: 0.69
  Epoch:2 | Optim:0.0200 | Loss: 0.69
  Epoch:2 | Optim:0.0200 | Loss: 0.69
  Epoch:2 | Optim:0.0200 | Loss: 0.69
  Epoch:2 | Optim:0.0200 | Loss: 0.70
  Epoch:2 | Optim:0.0200 | Loss: 0.68
  Epoch:3 | Optim:0.0400 | Loss: 0.70
  Epoch:3 | Optim:0.0400 | Loss: 0.70
  Epoch:3 | Optim:0.0400 | Loss: 0.70
  Epoch:3 | Optim:0.0400 | Loss: 0.68
  Epoch:3 | Optim:0.0400 | Loss: 0.68
  Epoch:3 | Optim:0.0400 | Loss: 0.69
  Epoch:3 | Optim:0.0400 | Loss: 0.70
  Epoch:3 | Optim:0.0400 | Loss: 0.68
  Epoch:3 | Optim:0.0400 | Loss: 0.70
  Epoch:3 | Optim:0.0400 | Loss: 0.69
  Epoch:3 | Optim:0.0400 | Loss: 0.70
  Epoch:3 | Optim:0.0400 | Loss: 0.65
  Epoch:4 | Optim:0.0600 | Loss: 0.69
  Epoch:4 | Optim:0.0600 | Loss: 0.68
  Epoch:4 | Optim:0.0600 | Loss: 0.68
  Epoch:4 | Optim:0.0600 | Loss: 0.73
  Epoch:4 | Optim:0.0600 | Loss: 0.70
  Epoch:4 | Optim:0.0600 | Loss: 0.71
  Epoch:4 | Optim:0.0600 | Loss: 0.71
  Epoch:4 | Optim:0.0600 | Loss: 0.68
  Epoch:4 | Optim:0.0600 | Loss: 0.71
  Epoch:4 | Optim:0.0600 | Loss: 0.69
  Epoch:4 | Optim:0.0600 | Loss: 0.69
  Epoch:4 | Optim:0.0600 | Loss: 0.72
  Epoch:5 | Optim:0.0800 | Loss: 0.69
  Epoch:5 | Optim:0.0800 | Loss: 0.69
  Epoch:5 | Optim:0.0800 | Loss: 0.70
  Epoch:5 | Optim:0.0800 | Loss: 0.69
  Epoch:5 | Optim:0.0800 | Loss: 0.69
  Epoch:5 | Optim:0.0800 | Loss: 0.68
  Epoch:5 | Optim:0.0800 | Loss: 0.71
  Epoch:5 | Optim:0.0800 | Loss: 0.68
  Epoch:5 | Optim:0.0800 | Loss: 0.71
  Epoch:5 | Optim:0.0800 | Loss: 0.69
  Epoch:5 | Optim:0.0800 | Loss: 0.71
  Epoch:5 | Optim:0.0800 | Loss: 0.70
  </code></pre>
  <p>In a nutshell, I want to ask how do I find the <strong>correct</strong> learning rate?
  Would appreciate if anybody could show how to plot the <code>learning rate</code> by <code>loss</code> plot.</p>