We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
Trainer(precision=16) fails with optim.lr_scheduler.ReduceLROnPlateau
<!-- 
### Common bugs:
1. Tensorboard not showing in Jupyter-notebook see [issue 79](https://github.com/PyTorchLightning/pytorch-lightning/issues/79).    
2. PyTorch 1.1.0 vs 1.2.0 support [see FAQ](https://github.com/PyTorchLightning/pytorch-lightning#faq)    
-->

## 🐛 Bug

<!-- A clear and concise description of what the bug is. -->

### To Reproduce

Steps to reproduce the behavior:

1. Create a `pl.LightningModule` that returns your optimizer along with a `optim.lr_scheduler.ReduceLROnPlateau` scheduler from `configure_optimizers`
2. Create a `pl.Trainer` wit `precision=16`
3. Run your training (i.e., `trainer.fit(model)`)
4. See error

```console
Traceback (most recent call last):                                                                                                  
  File "main.py", line 65, in <module>                                                                                              
    main()                                                                                                                          
  File "main.py", line 61, in main                                                                                                  
    trainer.fit(model)                                                                                                              
  File "/workspace/pytorch-lightning/pytorch_lightning/trainer/trainer.py", line 889, in fit                             
    self.dp_train(model)                                                                                                            
  File "/workspace/pytorch-lightning/pytorch_lightning/trainer/distrib_parts.py", line 223, in dp_train                  
    self.reinit_scheduler_properties(optimizers, self.lr_schedulers)                                                                
  File "/workspace/pytorch-lightning/pytorch_lightning/trainer/optimizers.py", line 122, in reinit_scheduler_properties  
    scheduler.__class__.__mro__[idx].__init__(scheduler, optimizer)                                                                 
UnboundLocalError: local variable 'idx' referenced before assignment                                                                
```
<!-- If you have a code sample, error messages, stack traces, please provide it here as well -->


<!-- #### Code sample -->
<!-- Ideally attach a minimal code sample to reproduce the decried issue. 
Minimal means having the shortest code but still preserving the bug. -->

<!-- ### Expected behavior -->

<!-- A clear and concise description of what you expected to happen. -->

<!-- ### Environment

Please copy and paste the output from our
[environment collection script](https://raw.githubusercontent.com/PyTorchLightning/pytorch-lightning/master/tests/collect_env_details.py)
(or fill out the checklist below manually).

You can get the script and run it with:
```
wget https://raw.githubusercontent.com/PyTorchLightning/pytorch-lightning/master/tests/collect_env_details.py
# For security purposes, please check the contents of collect_env_details.py before running it.
python collect_env_details.py
```
 - PyTorch Version (1.5):
 - OS (Linux):

### Additional context
-->

<!-- Add any other context about the problem here. -->

The error occurs in `pytorch-lightning/pytorch_lightning/trainer/optimizers.py", line 122`.

```python
def reinit_scheduler_properties(self, optimizers: list, schedulers: list):
    # Reinitialize optimizer.step properties added by schedulers
    for scheduler in schedulers:
        for optimizer in optimizers:
            scheduler = scheduler['scheduler']
            # check that we dont mix users optimizers and schedulers
            if scheduler.optimizer == optimizer:
                # Find the mro belonging to the base lr scheduler class
                for i, mro in enumerate(scheduler.__class__.__mro__):
                    if mro == optim.lr_scheduler._LRScheduler:
                        idx = i
                scheduler.__class__.__mro__[idx].__init__(scheduler, optimizer)
```
The `idx` local variable is unassigned because `optim.lr_scheduler.ReduceLROnPlateau` is not a subclass of `optim.lr_scheduler._LRScheduler`.

I could work around the error by adding a specific check for `optim.lr_scheduler.ReduceLROnPlateau` but I'm not sure if this is a good solution.

```python
def reinit_scheduler_properties(self, optimizers: list, schedulers: list):
    # Reinitialize optimizer.step properties added by schedulers
    for scheduler in schedulers:
        for optimizer in optimizers:
            scheduler = scheduler['scheduler']
            # check that we dont mix users optimizers and schedulers
            if scheduler.optimizer == optimizer:
                # Find the mro belonging to the base lr scheduler class
                for i, mro in enumerate(scheduler.__class__.__mro__):
                    if mro == optim.lr_scheduler._LRScheduler:
                        idx = i
                    elif mro == optim.lr_scheduler.ReduceLROnPlateau:
                        idx = i
                scheduler.__class__.__mro__[idx].__init__(scheduler, optimizer)
```

### Related issue in PyTorch:
ReduceLROnPlateau parent class is not _LRScheduler #21981
https://github.com/pytorch/pytorch/issues/21981

--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
--- BEGIN FILES ---
Path: `pytorch_lightning/trainer/optimizers.py`
Content:
```
1 from abc import ABC
2 from typing import List, Tuple
3 
4 import torch
5 from torch import optim
6 from torch.optim.optimizer import Optimizer
7 
8 from pytorch_lightning.core.lightning import LightningModule
9 from pytorch_lightning.utilities import rank_zero_warn
10 
11 
12 class TrainerOptimizersMixin(ABC):
13 
14     def init_optimizers(
15             self,
16             model: LightningModule
17     ) -> Tuple[List, List, List]:
18         optim_conf = model.configure_optimizers()
19 
20         if optim_conf is None:
21             rank_zero_warn('`LightningModule.configure_optimizers` returned `None`, '
22                            'this fit will run with no optimizer', UserWarning)
23             optim_conf = _MockOptimizer()
24 
25         # single output, single optimizer
26         if isinstance(optim_conf, Optimizer):
27             return [optim_conf], [], []
28 
29         # two lists, optimizer + lr schedulers
30         elif isinstance(optim_conf, (list, tuple)) and len(optim_conf) == 2 \
31                 and isinstance(optim_conf[0], list):
32             optimizers, lr_schedulers = optim_conf
33             lr_schedulers = self.configure_schedulers(lr_schedulers)
34             return optimizers, lr_schedulers, []
35 
36         # single dictionary
37         elif isinstance(optim_conf, dict):
38             optimizer = optim_conf["optimizer"]
39             lr_scheduler = optim_conf.get("lr_scheduler", [])
40             if lr_scheduler:
41                 lr_schedulers = self.configure_schedulers([lr_scheduler])
42             else:
43                 lr_schedulers = []
44             return [optimizer], lr_schedulers, []
45 
46         # multiple dictionaries
47         elif isinstance(optim_conf, (list, tuple)) and isinstance(optim_conf[0], dict):
48             optimizers = [opt_dict["optimizer"] for opt_dict in optim_conf]
49             # take only lr wif exists and ot they are defined - not None
50             lr_schedulers = [
51                 opt_dict["lr_scheduler"] for opt_dict in optim_conf if opt_dict.get("lr_scheduler")
52             ]
53             # take only freq wif exists and ot they are defined - not None
54             optimizer_frequencies = [
55                 opt_dict["frequency"] for opt_dict in optim_conf if opt_dict.get("frequency") is not None
56             ]
57 
58             # clean scheduler list
59             if lr_schedulers:
60                 lr_schedulers = self.configure_schedulers(lr_schedulers)
61             # assert that if frequencies are present, they are given for all optimizers
62             if optimizer_frequencies and len(optimizer_frequencies) != len(optimizers):
63                 raise ValueError("A frequency must be given to each optimizer.")
64             return optimizers, lr_schedulers, optimizer_frequencies
65 
66         # single list or tuple, multiple optimizer
67         elif isinstance(optim_conf, (list, tuple)):
68             return list(optim_conf), [], []
69 
70         # unknown configuration
71         else:
72             raise ValueError(
73                 'Unknown configuration for model optimizers.'
74                 ' Output from `model.configure_optimizers()` should either be:'
75                 ' * single output, single `torch.optim.Optimizer`'
76                 ' * single output, list of `torch.optim.Optimizer`'
77                 ' * single output, a dictionary with `optimizer` key (`torch.optim.Optimizer`)'
78                 '    and an optional `lr_scheduler` key (`torch.optim.lr_scheduler`)'
79                 ' * two outputs, first being a list of `torch.optim.Optimizer` second being'
80                 '    a list of `torch.optim.lr_scheduler`'
81                 ' * multiple outputs, dictionaries as described with an optional `frequency` key (int)')
82 
83     def configure_schedulers(self, schedulers: list):
84         # Convert each scheduler into dict structure with relevant information
85         lr_schedulers = []
86         default_config = {'interval': 'epoch',  # default every epoch
87                           'frequency': 1,  # default every epoch/batch
88                           'reduce_on_plateau': False,  # most often not ReduceLROnPlateau scheduler
89                           'monitor': 'val_loss'}  # default value to monitor for ReduceLROnPlateau
90         for scheduler in schedulers:
91             if isinstance(scheduler, dict):
92                 if 'scheduler' not in scheduler:
93                     raise ValueError('Lr scheduler should have key `scheduler`',
94                                      ' with item being a lr scheduler')
95                 scheduler['reduce_on_plateau'] = isinstance(
96                     scheduler['scheduler'], optim.lr_scheduler.ReduceLROnPlateau)
97 
98                 lr_schedulers.append({**default_config, **scheduler})
99 
100             elif isinstance(scheduler, optim.lr_scheduler.ReduceLROnPlateau):
101                 lr_schedulers.append({**default_config, 'scheduler': scheduler,
102                                       'reduce_on_plateau': True})
103 
104             elif isinstance(scheduler, optim.lr_scheduler._LRScheduler):
105                 lr_schedulers.append({**default_config, 'scheduler': scheduler})
106             else:
107                 raise ValueError(f'Input {scheduler} to lr schedulers '
108                                  'is a invalid input.')
109         return lr_schedulers
110 
111     def reinit_scheduler_properties(self, optimizers: list, schedulers: list):
112         # Reinitialize optimizer.step properties added by schedulers
113         for scheduler in schedulers:
114             for optimizer in optimizers:
115                 scheduler = scheduler['scheduler']
116                 # check that we dont mix users optimizers and schedulers
117                 if scheduler.optimizer == optimizer:
118                     # Find the mro belonging to the base lr scheduler class
119                     for i, mro in enumerate(scheduler.__class__.__mro__):
120                         if mro == optim.lr_scheduler._LRScheduler:
121                             idx = i
122                     scheduler.__class__.__mro__[idx].__init__(scheduler, optimizer)
123 
124 
125 class _MockOptimizer(Optimizer):
126     """The `_MockOptimizer` will be used inplace of an optimizer in the event that `None`
127     is returned from `configure_optimizers`.
128     """
129 
130     def __init__(self):
131         super().__init__([torch.zeros(1)], {})
132 
133     def add_param_group(self, param_group):
134         pass  # Do Nothing
135 
136     def load_state_dict(self, state_dict):
137         pass  # Do Nothing
138 
139     def state_dict(self):
140         return {}  # Return Empty
141 
142     def step(self, closure=None):
143         if closure is not None:
144             closure()
145 
146     def zero_grad(self):
147         pass  # Do Nothing
148 
149     def __repr__(self):
150         return 'No Optimizer'
151
```
--- END FILES ---

Please first localize the bug based on the issue statement, and then generate a patch according to the `git diff` format fenced by three backticks.

Here is an example:

```diff
diff --git a/examples/server_async.py b/examples/server_async.py
--- a/examples/server_async.py
+++ b/examples/server_async.py
@@ -313,4 +313,4 @@
 
 
 if __name__ == "__main__":
-    asyncio.run(run_async_server("."), debug=True)
+    asyncio.run(run_async_server(), debug=True)
diff --git a/examples/server_sync.py b/examples/server_sync.py
--- a/examples/server_sync.py
+++ b/examples/server_sync.py
@@ -313,5 +313,5 @@
 
 
 if __name__ == "__main__":
-    server = run_sync_server(".")
+    server = run_sync_server()
     server.shutdown()

```

diff --git a/pytorch_lightning/trainer/optimizers.py b/pytorch_lightning/trainer/optimizers.py
--- a/pytorch_lightning/trainer/optimizers.py
+++ b/pytorch_lightning/trainer/optimizers.py
@@ -111,15 +111,25 @@
     def reinit_scheduler_properties(self, optimizers: list, schedulers: list):
         # Reinitialize optimizer.step properties added by schedulers
         for scheduler in schedulers:
+            scheduler = scheduler['scheduler']
+
             for optimizer in optimizers:
-                scheduler = scheduler['scheduler']
                 # check that we dont mix users optimizers and schedulers
                 if scheduler.optimizer == optimizer:
                     # Find the mro belonging to the base lr scheduler class
                     for i, mro in enumerate(scheduler.__class__.__mro__):
-                        if mro == optim.lr_scheduler._LRScheduler:
+                        if (
+                            mro == optim.lr_scheduler._LRScheduler
+                            or mro == optim.lr_scheduler.ReduceLROnPlateau
+                        ):
                             idx = i
-                    scheduler.__class__.__mro__[idx].__init__(scheduler, optimizer)
+                            state = scheduler.state_dict()
+                        else:
+                            state = None
+
+                scheduler.__class__.__mro__[idx].__init__(scheduler, optimizer)
+                if state is not None:
+                    scheduler.load_state_dict(state)
 
 
 class _MockOptimizer(Optimizer):

{"golden_diff": "diff --git a/pytorch_lightning/trainer/optimizers.py b/pytorch_lightning/trainer/optimizers.py\n--- a/pytorch_lightning/trainer/optimizers.py\n+++ b/pytorch_lightning/trainer/optimizers.py\n@@ -111,15 +111,25 @@\n     def reinit_scheduler_properties(self, optimizers: list, schedulers: list):\n         # Reinitialize optimizer.step properties added by schedulers\n         for scheduler in schedulers:\n+            scheduler = scheduler['scheduler']\n+\n             for optimizer in optimizers:\n-                scheduler = scheduler['scheduler']\n                 # check that we dont mix users optimizers and schedulers\n                 if scheduler.optimizer == optimizer:\n                     # Find the mro belonging to the base lr scheduler class\n                     for i, mro in enumerate(scheduler.__class__.__mro__):\n-                        if mro == optim.lr_scheduler._LRScheduler:\n+                        if (\n+                            mro == optim.lr_scheduler._LRScheduler\n+                            or mro == optim.lr_scheduler.ReduceLROnPlateau\n+                        ):\n                             idx = i\n-                    scheduler.__class__.__mro__[idx].__init__(scheduler, optimizer)\n+                            state = scheduler.state_dict()\n+                        else:\n+                            state = None\n+\n+                scheduler.__class__.__mro__[idx].__init__(scheduler, optimizer)\n+                if state is not None:\n+                    scheduler.load_state_dict(state)\n \n \n class _MockOptimizer(Optimizer):\n", "issue": "Trainer(precision=16) fails with optim.lr_scheduler.ReduceLROnPlateau\n<!-- \r\n### Common bugs:\r\n1. Tensorboard not showing in Jupyter-notebook see [issue 79](https://github.com/PyTorchLightning/pytorch-lightning/issues/79).    \r\n2. PyTorch 1.1.0 vs 1.2.0 support [see FAQ](https://github.com/PyTorchLightning/pytorch-lightning#faq)    \r\n-->\r\n\r\n## \ud83d\udc1b Bug\r\n\r\n<!-- A clear and concise description of what the bug is. -->\r\n\r\n### To Reproduce\r\n\r\nSteps to reproduce the behavior:\r\n\r\n1. Create a `pl.LightningModule` that returns your optimizer along with a `optim.lr_scheduler.ReduceLROnPlateau` scheduler from `configure_optimizers`\r\n2. Create a `pl.Trainer` wit `precision=16`\r\n3. Run your training (i.e., `trainer.fit(model)`)\r\n4. See error\r\n\r\n```console\r\nTraceback (most recent call last):                                                                                                  \r\n  File \"main.py\", line 65, in <module>                                                                                              \r\n    main()                                                                                                                          \r\n  File \"main.py\", line 61, in main                                                                                                  \r\n    trainer.fit(model)                                                                                                              \r\n  File \"/workspace/pytorch-lightning/pytorch_lightning/trainer/trainer.py\", line 889, in fit                             \r\n    self.dp_train(model)                                                                                                            \r\n  File \"/workspace/pytorch-lightning/pytorch_lightning/trainer/distrib_parts.py\", line 223, in dp_train                  \r\n    self.reinit_scheduler_properties(optimizers, self.lr_schedulers)                                                                \r\n  File \"/workspace/pytorch-lightning/pytorch_lightning/trainer/optimizers.py\", line 122, in reinit_scheduler_properties  \r\n    scheduler.__class__.__mro__[idx].__init__(scheduler, optimizer)                                                                 \r\nUnboundLocalError: local variable 'idx' referenced before assignment                                                                \r\n```\r\n<!-- If you have a code sample, error messages, stack traces, please provide it here as well -->\r\n\r\n\r\n<!-- #### Code sample -->\r\n<!-- Ideally attach a minimal code sample to reproduce the decried issue. \r\nMinimal means having the shortest code but still preserving the bug. -->\r\n\r\n<!-- ### Expected behavior -->\r\n\r\n<!-- A clear and concise description of what you expected to happen. -->\r\n\r\n<!-- ### Environment\r\n\r\nPlease copy and paste the output from our\r\n[environment collection script](https://raw.githubusercontent.com/PyTorchLightning/pytorch-lightning/master/tests/collect_env_details.py)\r\n(or fill out the checklist below manually).\r\n\r\nYou can get the script and run it with:\r\n```\r\nwget https://raw.githubusercontent.com/PyTorchLightning/pytorch-lightning/master/tests/collect_env_details.py\r\n# For security purposes, please check the contents of collect_env_details.py before running it.\r\npython collect_env_details.py\r\n```\r\n - PyTorch Version (1.5):\r\n - OS (Linux):\r\n\r\n### Additional context\r\n-->\r\n\r\n<!-- Add any other context about the problem here. -->\r\n\r\nThe error occurs in `pytorch-lightning/pytorch_lightning/trainer/optimizers.py\", line 122`.\r\n\r\n```python\r\ndef reinit_scheduler_properties(self, optimizers: list, schedulers: list):\r\n    # Reinitialize optimizer.step properties added by schedulers\r\n    for scheduler in schedulers:\r\n        for optimizer in optimizers:\r\n            scheduler = scheduler['scheduler']\r\n            # check that we dont mix users optimizers and schedulers\r\n            if scheduler.optimizer == optimizer:\r\n                # Find the mro belonging to the base lr scheduler class\r\n                for i, mro in enumerate(scheduler.__class__.__mro__):\r\n                    if mro == optim.lr_scheduler._LRScheduler:\r\n                        idx = i\r\n                scheduler.__class__.__mro__[idx].__init__(scheduler, optimizer)\r\n```\r\nThe `idx` local variable is unassigned because `optim.lr_scheduler.ReduceLROnPlateau` is not a subclass of `optim.lr_scheduler._LRScheduler`.\r\n\r\nI could work around the error by adding a specific check for `optim.lr_scheduler.ReduceLROnPlateau` but I'm not sure if this is a good solution.\r\n\r\n```python\r\ndef reinit_scheduler_properties(self, optimizers: list, schedulers: list):\r\n    # Reinitialize optimizer.step properties added by schedulers\r\n    for scheduler in schedulers:\r\n        for optimizer in optimizers:\r\n            scheduler = scheduler['scheduler']\r\n            # check that we dont mix users optimizers and schedulers\r\n            if scheduler.optimizer == optimizer:\r\n                # Find the mro belonging to the base lr scheduler class\r\n                for i, mro in enumerate(scheduler.__class__.__mro__):\r\n                    if mro == optim.lr_scheduler._LRScheduler:\r\n                        idx = i\r\n                    elif mro == optim.lr_scheduler.ReduceLROnPlateau:\r\n                        idx = i\r\n                scheduler.__class__.__mro__[idx].__init__(scheduler, optimizer)\r\n```\r\n\r\n### Related issue in PyTorch:\r\nReduceLROnPlateau parent class is not _LRScheduler #21981\r\nhttps://github.com/pytorch/pytorch/issues/21981\n", "before_files": [{"content": "from abc import ABC\nfrom typing import List, Tuple\n\nimport torch\nfrom torch import optim\nfrom torch.optim.optimizer import Optimizer\n\nfrom pytorch_lightning.core.lightning import LightningModule\nfrom pytorch_lightning.utilities import rank_zero_warn\n\n\nclass TrainerOptimizersMixin(ABC):\n\n    def init_optimizers(\n            self,\n            model: LightningModule\n    ) -> Tuple[List, List, List]:\n        optim_conf = model.configure_optimizers()\n\n        if optim_conf is None:\n            rank_zero_warn('`LightningModule.configure_optimizers` returned `None`, '\n                           'this fit will run with no optimizer', UserWarning)\n            optim_conf = _MockOptimizer()\n\n        # single output, single optimizer\n        if isinstance(optim_conf, Optimizer):\n            return [optim_conf], [], []\n\n        # two lists, optimizer + lr schedulers\n        elif isinstance(optim_conf, (list, tuple)) and len(optim_conf) == 2 \\\n                and isinstance(optim_conf[0], list):\n            optimizers, lr_schedulers = optim_conf\n            lr_schedulers = self.configure_schedulers(lr_schedulers)\n            return optimizers, lr_schedulers, []\n\n        # single dictionary\n        elif isinstance(optim_conf, dict):\n            optimizer = optim_conf[\"optimizer\"]\n            lr_scheduler = optim_conf.get(\"lr_scheduler\", [])\n            if lr_scheduler:\n                lr_schedulers = self.configure_schedulers([lr_scheduler])\n            else:\n                lr_schedulers = []\n            return [optimizer], lr_schedulers, []\n\n        # multiple dictionaries\n        elif isinstance(optim_conf, (list, tuple)) and isinstance(optim_conf[0], dict):\n            optimizers = [opt_dict[\"optimizer\"] for opt_dict in optim_conf]\n            # take only lr wif exists and ot they are defined - not None\n            lr_schedulers = [\n                opt_dict[\"lr_scheduler\"] for opt_dict in optim_conf if opt_dict.get(\"lr_scheduler\")\n            ]\n            # take only freq wif exists and ot they are defined - not None\n            optimizer_frequencies = [\n                opt_dict[\"frequency\"] for opt_dict in optim_conf if opt_dict.get(\"frequency\") is not None\n            ]\n\n            # clean scheduler list\n            if lr_schedulers:\n                lr_schedulers = self.configure_schedulers(lr_schedulers)\n            # assert that if frequencies are present, they are given for all optimizers\n            if optimizer_frequencies and len(optimizer_frequencies) != len(optimizers):\n                raise ValueError(\"A frequency must be given to each optimizer.\")\n            return optimizers, lr_schedulers, optimizer_frequencies\n\n        # single list or tuple, multiple optimizer\n        elif isinstance(optim_conf, (list, tuple)):\n            return list(optim_conf), [], []\n\n        # unknown configuration\n        else:\n            raise ValueError(\n                'Unknown configuration for model optimizers.'\n                ' Output from `model.configure_optimizers()` should either be:'\n                ' * single output, single `torch.optim.Optimizer`'\n                ' * single output, list of `torch.optim.Optimizer`'\n                ' * single output, a dictionary with `optimizer` key (`torch.optim.Optimizer`)'\n                '    and an optional `lr_scheduler` key (`torch.optim.lr_scheduler`)'\n                ' * two outputs, first being a list of `torch.optim.Optimizer` second being'\n                '    a list of `torch.optim.lr_scheduler`'\n                ' * multiple outputs, dictionaries as described with an optional `frequency` key (int)')\n\n    def configure_schedulers(self, schedulers: list):\n        # Convert each scheduler into dict structure with relevant information\n        lr_schedulers = []\n        default_config = {'interval': 'epoch',  # default every epoch\n                          'frequency': 1,  # default every epoch/batch\n                          'reduce_on_plateau': False,  # most often not ReduceLROnPlateau scheduler\n                          'monitor': 'val_loss'}  # default value to monitor for ReduceLROnPlateau\n        for scheduler in schedulers:\n            if isinstance(scheduler, dict):\n                if 'scheduler' not in scheduler:\n                    raise ValueError('Lr scheduler should have key `scheduler`',\n                                     ' with item being a lr scheduler')\n                scheduler['reduce_on_plateau'] = isinstance(\n                    scheduler['scheduler'], optim.lr_scheduler.ReduceLROnPlateau)\n\n                lr_schedulers.append({**default_config, **scheduler})\n\n            elif isinstance(scheduler, optim.lr_scheduler.ReduceLROnPlateau):\n                lr_schedulers.append({**default_config, 'scheduler': scheduler,\n                                      'reduce_on_plateau': True})\n\n            elif isinstance(scheduler, optim.lr_scheduler._LRScheduler):\n                lr_schedulers.append({**default_config, 'scheduler': scheduler})\n            else:\n                raise ValueError(f'Input {scheduler} to lr schedulers '\n                                 'is a invalid input.')\n        return lr_schedulers\n\n    def reinit_scheduler_properties(self, optimizers: list, schedulers: list):\n        # Reinitialize optimizer.step properties added by schedulers\n        for scheduler in schedulers:\n            for optimizer in optimizers:\n                scheduler = scheduler['scheduler']\n                # check that we dont mix users optimizers and schedulers\n                if scheduler.optimizer == optimizer:\n                    # Find the mro belonging to the base lr scheduler class\n                    for i, mro in enumerate(scheduler.__class__.__mro__):\n                        if mro == optim.lr_scheduler._LRScheduler:\n                            idx = i\n                    scheduler.__class__.__mro__[idx].__init__(scheduler, optimizer)\n\n\nclass _MockOptimizer(Optimizer):\n    \"\"\"The `_MockOptimizer` will be used inplace of an optimizer in the event that `None`\n    is returned from `configure_optimizers`.\n    \"\"\"\n\n    def __init__(self):\n        super().__init__([torch.zeros(1)], {})\n\n    def add_param_group(self, param_group):\n        pass  # Do Nothing\n\n    def load_state_dict(self, state_dict):\n        pass  # Do Nothing\n\n    def state_dict(self):\n        return {}  # Return Empty\n\n    def step(self, closure=None):\n        if closure is not None:\n            closure()\n\n    def zero_grad(self):\n        pass  # Do Nothing\n\n    def __repr__(self):\n        return 'No Optimizer'\n", "path": "pytorch_lightning/trainer/optimizers.py"}], "after_files": [{"content": "from abc import ABC\nfrom typing import List, Tuple\n\nimport torch\nfrom torch import optim\nfrom torch.optim.optimizer import Optimizer\n\nfrom pytorch_lightning.core.lightning import LightningModule\nfrom pytorch_lightning.utilities import rank_zero_warn\n\n\nclass TrainerOptimizersMixin(ABC):\n\n    def init_optimizers(\n            self,\n            model: LightningModule\n    ) -> Tuple[List, List, List]:\n        optim_conf = model.configure_optimizers()\n\n        if optim_conf is None:\n            rank_zero_warn('`LightningModule.configure_optimizers` returned `None`, '\n                           'this fit will run with no optimizer', UserWarning)\n            optim_conf = _MockOptimizer()\n\n        # single output, single optimizer\n        if isinstance(optim_conf, Optimizer):\n            return [optim_conf], [], []\n\n        # two lists, optimizer + lr schedulers\n        elif isinstance(optim_conf, (list, tuple)) and len(optim_conf) == 2 \\\n                and isinstance(optim_conf[0], list):\n            optimizers, lr_schedulers = optim_conf\n            lr_schedulers = self.configure_schedulers(lr_schedulers)\n            return optimizers, lr_schedulers, []\n\n        # single dictionary\n        elif isinstance(optim_conf, dict):\n            optimizer = optim_conf[\"optimizer\"]\n            lr_scheduler = optim_conf.get(\"lr_scheduler\", [])\n            if lr_scheduler:\n                lr_schedulers = self.configure_schedulers([lr_scheduler])\n            else:\n                lr_schedulers = []\n            return [optimizer], lr_schedulers, []\n\n        # multiple dictionaries\n        elif isinstance(optim_conf, (list, tuple)) and isinstance(optim_conf[0], dict):\n            optimizers = [opt_dict[\"optimizer\"] for opt_dict in optim_conf]\n            # take only lr wif exists and ot they are defined - not None\n            lr_schedulers = [\n                opt_dict[\"lr_scheduler\"] for opt_dict in optim_conf if opt_dict.get(\"lr_scheduler\")\n            ]\n            # take only freq wif exists and ot they are defined - not None\n            optimizer_frequencies = [\n                opt_dict[\"frequency\"] for opt_dict in optim_conf if opt_dict.get(\"frequency\") is not None\n            ]\n\n            # clean scheduler list\n            if lr_schedulers:\n                lr_schedulers = self.configure_schedulers(lr_schedulers)\n            # assert that if frequencies are present, they are given for all optimizers\n            if optimizer_frequencies and len(optimizer_frequencies) != len(optimizers):\n                raise ValueError(\"A frequency must be given to each optimizer.\")\n            return optimizers, lr_schedulers, optimizer_frequencies\n\n        # single list or tuple, multiple optimizer\n        elif isinstance(optim_conf, (list, tuple)):\n            return list(optim_conf), [], []\n\n        # unknown configuration\n        else:\n            raise ValueError(\n                'Unknown configuration for model optimizers.'\n                ' Output from `model.configure_optimizers()` should either be:'\n                ' * single output, single `torch.optim.Optimizer`'\n                ' * single output, list of `torch.optim.Optimizer`'\n                ' * single output, a dictionary with `optimizer` key (`torch.optim.Optimizer`)'\n                '    and an optional `lr_scheduler` key (`torch.optim.lr_scheduler`)'\n                ' * two outputs, first being a list of `torch.optim.Optimizer` second being'\n                '    a list of `torch.optim.lr_scheduler`'\n                ' * multiple outputs, dictionaries as described with an optional `frequency` key (int)')\n\n    def configure_schedulers(self, schedulers: list):\n        # Convert each scheduler into dict structure with relevant information\n        lr_schedulers = []\n        default_config = {'interval': 'epoch',  # default every epoch\n                          'frequency': 1,  # default every epoch/batch\n                          'reduce_on_plateau': False,  # most often not ReduceLROnPlateau scheduler\n                          'monitor': 'val_loss'}  # default value to monitor for ReduceLROnPlateau\n        for scheduler in schedulers:\n            if isinstance(scheduler, dict):\n                if 'scheduler' not in scheduler:\n                    raise ValueError('Lr scheduler should have key `scheduler`',\n                                     ' with item being a lr scheduler')\n                scheduler['reduce_on_plateau'] = isinstance(\n                    scheduler['scheduler'], optim.lr_scheduler.ReduceLROnPlateau)\n\n                lr_schedulers.append({**default_config, **scheduler})\n\n            elif isinstance(scheduler, optim.lr_scheduler.ReduceLROnPlateau):\n                lr_schedulers.append({**default_config, 'scheduler': scheduler,\n                                      'reduce_on_plateau': True})\n\n            elif isinstance(scheduler, optim.lr_scheduler._LRScheduler):\n                lr_schedulers.append({**default_config, 'scheduler': scheduler})\n            else:\n                raise ValueError(f'Input {scheduler} to lr schedulers '\n                                 'is a invalid input.')\n        return lr_schedulers\n\n    def reinit_scheduler_properties(self, optimizers: list, schedulers: list):\n        # Reinitialize optimizer.step properties added by schedulers\n        for scheduler in schedulers:\n            scheduler = scheduler['scheduler']\n\n            for optimizer in optimizers:\n                # check that we dont mix users optimizers and schedulers\n                if scheduler.optimizer == optimizer:\n                    # Find the mro belonging to the base lr scheduler class\n                    for i, mro in enumerate(scheduler.__class__.__mro__):\n                        if (\n                            mro == optim.lr_scheduler._LRScheduler\n                            or mro == optim.lr_scheduler.ReduceLROnPlateau\n                        ):\n                            idx = i\n                            state = scheduler.state_dict()\n                        else:\n                            state = None\n\n                scheduler.__class__.__mro__[idx].__init__(scheduler, optimizer)\n                if state is not None:\n                    scheduler.load_state_dict(state)\n\n\nclass _MockOptimizer(Optimizer):\n    \"\"\"The `_MockOptimizer` will be used inplace of an optimizer in the event that `None`\n    is returned from `configure_optimizers`.\n    \"\"\"\n\n    def __init__(self):\n        super().__init__([torch.zeros(1)], {})\n\n    def add_param_group(self, param_group):\n        pass  # Do Nothing\n\n    def load_state_dict(self, state_dict):\n        pass  # Do Nothing\n\n    def state_dict(self):\n        return {}  # Return Empty\n\n    def step(self, closure=None):\n        if closure is not None:\n            closure()\n\n    def zero_grad(self):\n        pass  # Do Nothing\n\n    def __repr__(self):\n        return 'No Optimizer'\n", "path": "pytorch_lightning/trainer/optimizers.py"}]}