We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
Multi-controlnet formatting issue
### Describe the bug

Hi.
There is an inconsistency between `from_pretrained` and `save_pretrained` within the Multicontrolnet class.
The from_pretrained function returns a directory structure like this: controlnet, controlnet_1, controlnet_2, 
whereas save_pretrained is like this: controlnet, controlnet_1, controlnet_1_2.
When loading a saved model, if there are 3 controlnets, the last controlnet will not be loaded. (more than 2 always same issue)


### Reproduction

I don't think there is no need to reproduce the code as it's pretty clear issue.
For compatibility, how about changing the `save_pretrained` function in Multi-ControlNet to look like the code below? 
```
def save_pretrained(
        self,
        save_directory: Union[str, os.PathLike],
        is_main_process: bool = True,
        save_function: Callable = None,
        safe_serialization: bool = True,
        variant: Optional[str] = None,
    ):
        """
        Save a model and its configuration file to a directory, so that it can be re-loaded using the
        `[`~pipelines.controlnet.MultiControlNetModel.from_pretrained`]` class method.

        Arguments:
            save_directory (`str` or `os.PathLike`):
                Directory to which to save. Will be created if it doesn't exist.
            is_main_process (`bool`, *optional*, defaults to `True`):
                Whether the process calling this is the main process or not. Useful when in distributed training like
                TPUs and need to call this function on all processes. In this case, set `is_main_process=True` only on
                the main process to avoid race conditions.
            save_function (`Callable`):
                The function to use to save the state dictionary. Useful on distributed training like TPUs when one
                need to replace `torch.save` by another method. Can be configured with the environment variable
                `DIFFUSERS_SAVE_MODE`.
            safe_serialization (`bool`, *optional*, defaults to `True`):
                Whether to save the model using `safetensors` or the traditional PyTorch way (that uses `pickle`).
            variant (`str`, *optional*):
                If specified, weights are saved in the format pytorch_model.<variant>.bin.
        """
        model_path_to_save = save_directory
        for idx, controlnet in enumerate(self.nets):
            suffix = "" if idx == 0 else f"_{idx}"
            controlnet.save_pretrained(
                model_path_to_save + suffix,
                is_main_process=is_main_process,
                save_function=save_function,
                safe_serialization=safe_serialization,
                variant=variant,
            )
```

### Logs

_No response_

### System Info

Diffusers 0.27.2

### Who can help?

@sayakpaul 

--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
--- BEGIN FILES ---
Path: `src/diffusers/pipelines/controlnet/multicontrolnet.py`
Content:
```
1 import os
2 from typing import Any, Callable, Dict, List, Optional, Tuple, Union
3 
4 import torch
5 from torch import nn
6 
7 from ...models.controlnet import ControlNetModel, ControlNetOutput
8 from ...models.modeling_utils import ModelMixin
9 from ...utils import logging
10 
11 
12 logger = logging.get_logger(__name__)
13 
14 
15 class MultiControlNetModel(ModelMixin):
16     r"""
17     Multiple `ControlNetModel` wrapper class for Multi-ControlNet
18 
19     This module is a wrapper for multiple instances of the `ControlNetModel`. The `forward()` API is designed to be
20     compatible with `ControlNetModel`.
21 
22     Args:
23         controlnets (`List[ControlNetModel]`):
24             Provides additional conditioning to the unet during the denoising process. You must set multiple
25             `ControlNetModel` as a list.
26     """
27 
28     def __init__(self, controlnets: Union[List[ControlNetModel], Tuple[ControlNetModel]]):
29         super().__init__()
30         self.nets = nn.ModuleList(controlnets)
31 
32     def forward(
33         self,
34         sample: torch.Tensor,
35         timestep: Union[torch.Tensor, float, int],
36         encoder_hidden_states: torch.Tensor,
37         controlnet_cond: List[torch.tensor],
38         conditioning_scale: List[float],
39         class_labels: Optional[torch.Tensor] = None,
40         timestep_cond: Optional[torch.Tensor] = None,
41         attention_mask: Optional[torch.Tensor] = None,
42         added_cond_kwargs: Optional[Dict[str, torch.Tensor]] = None,
43         cross_attention_kwargs: Optional[Dict[str, Any]] = None,
44         guess_mode: bool = False,
45         return_dict: bool = True,
46     ) -> Union[ControlNetOutput, Tuple]:
47         for i, (image, scale, controlnet) in enumerate(zip(controlnet_cond, conditioning_scale, self.nets)):
48             down_samples, mid_sample = controlnet(
49                 sample=sample,
50                 timestep=timestep,
51                 encoder_hidden_states=encoder_hidden_states,
52                 controlnet_cond=image,
53                 conditioning_scale=scale,
54                 class_labels=class_labels,
55                 timestep_cond=timestep_cond,
56                 attention_mask=attention_mask,
57                 added_cond_kwargs=added_cond_kwargs,
58                 cross_attention_kwargs=cross_attention_kwargs,
59                 guess_mode=guess_mode,
60                 return_dict=return_dict,
61             )
62 
63             # merge samples
64             if i == 0:
65                 down_block_res_samples, mid_block_res_sample = down_samples, mid_sample
66             else:
67                 down_block_res_samples = [
68                     samples_prev + samples_curr
69                     for samples_prev, samples_curr in zip(down_block_res_samples, down_samples)
70                 ]
71                 mid_block_res_sample += mid_sample
72 
73         return down_block_res_samples, mid_block_res_sample
74 
75     def save_pretrained(
76         self,
77         save_directory: Union[str, os.PathLike],
78         is_main_process: bool = True,
79         save_function: Callable = None,
80         safe_serialization: bool = True,
81         variant: Optional[str] = None,
82     ):
83         """
84         Save a model and its configuration file to a directory, so that it can be re-loaded using the
85         `[`~pipelines.controlnet.MultiControlNetModel.from_pretrained`]` class method.
86 
87         Arguments:
88             save_directory (`str` or `os.PathLike`):
89                 Directory to which to save. Will be created if it doesn't exist.
90             is_main_process (`bool`, *optional*, defaults to `True`):
91                 Whether the process calling this is the main process or not. Useful when in distributed training like
92                 TPUs and need to call this function on all processes. In this case, set `is_main_process=True` only on
93                 the main process to avoid race conditions.
94             save_function (`Callable`):
95                 The function to use to save the state dictionary. Useful on distributed training like TPUs when one
96                 need to replace `torch.save` by another method. Can be configured with the environment variable
97                 `DIFFUSERS_SAVE_MODE`.
98             safe_serialization (`bool`, *optional*, defaults to `True`):
99                 Whether to save the model using `safetensors` or the traditional PyTorch way (that uses `pickle`).
100             variant (`str`, *optional*):
101                 If specified, weights are saved in the format pytorch_model.<variant>.bin.
102         """
103         idx = 0
104         model_path_to_save = save_directory
105         for controlnet in self.nets:
106             controlnet.save_pretrained(
107                 model_path_to_save,
108                 is_main_process=is_main_process,
109                 save_function=save_function,
110                 safe_serialization=safe_serialization,
111                 variant=variant,
112             )
113 
114             idx += 1
115             model_path_to_save = model_path_to_save + f"_{idx}"
116 
117     @classmethod
118     def from_pretrained(cls, pretrained_model_path: Optional[Union[str, os.PathLike]], **kwargs):
119         r"""
120         Instantiate a pretrained MultiControlNet model from multiple pre-trained controlnet models.
121 
122         The model is set in evaluation mode by default using `model.eval()` (Dropout modules are deactivated). To train
123         the model, you should first set it back in training mode with `model.train()`.
124 
125         The warning *Weights from XXX not initialized from pretrained model* means that the weights of XXX do not come
126         pretrained with the rest of the model. It is up to you to train those weights with a downstream fine-tuning
127         task.
128 
129         The warning *Weights from XXX not used in YYY* means that the layer XXX is not used by YYY, therefore those
130         weights are discarded.
131 
132         Parameters:
133             pretrained_model_path (`os.PathLike`):
134                 A path to a *directory* containing model weights saved using
135                 [`~diffusers.pipelines.controlnet.MultiControlNetModel.save_pretrained`], e.g.,
136                 `./my_model_directory/controlnet`.
137             torch_dtype (`str` or `torch.dtype`, *optional*):
138                 Override the default `torch.dtype` and load the model under this dtype. If `"auto"` is passed the dtype
139                 will be automatically derived from the model's weights.
140             output_loading_info(`bool`, *optional*, defaults to `False`):
141                 Whether or not to also return a dictionary containing missing keys, unexpected keys and error messages.
142             device_map (`str` or `Dict[str, Union[int, str, torch.device]]`, *optional*):
143                 A map that specifies where each submodule should go. It doesn't need to be refined to each
144                 parameter/buffer name, once a given module name is inside, every submodule of it will be sent to the
145                 same device.
146 
147                 To have Accelerate compute the most optimized `device_map` automatically, set `device_map="auto"`. For
148                 more information about each option see [designing a device
149                 map](https://hf.co/docs/accelerate/main/en/usage_guides/big_modeling#designing-a-device-map).
150             max_memory (`Dict`, *optional*):
151                 A dictionary device identifier to maximum memory. Will default to the maximum memory available for each
152                 GPU and the available CPU RAM if unset.
153             low_cpu_mem_usage (`bool`, *optional*, defaults to `True` if torch version >= 1.9.0 else `False`):
154                 Speed up model loading by not initializing the weights and only loading the pre-trained weights. This
155                 also tries to not use more than 1x model size in CPU memory (including peak memory) while loading the
156                 model. This is only supported when torch version >= 1.9.0. If you are using an older version of torch,
157                 setting this argument to `True` will raise an error.
158             variant (`str`, *optional*):
159                 If specified load weights from `variant` filename, *e.g.* pytorch_model.<variant>.bin. `variant` is
160                 ignored when using `from_flax`.
161             use_safetensors (`bool`, *optional*, defaults to `None`):
162                 If set to `None`, the `safetensors` weights will be downloaded if they're available **and** if the
163                 `safetensors` library is installed. If set to `True`, the model will be forcibly loaded from
164                 `safetensors` weights. If set to `False`, loading will *not* use `safetensors`.
165         """
166         idx = 0
167         controlnets = []
168 
169         # load controlnet and append to list until no controlnet directory exists anymore
170         # first controlnet has to be saved under `./mydirectory/controlnet` to be compliant with `DiffusionPipeline.from_prertained`
171         # second, third, ... controlnets have to be saved under `./mydirectory/controlnet_1`, `./mydirectory/controlnet_2`, ...
172         model_path_to_load = pretrained_model_path
173         while os.path.isdir(model_path_to_load):
174             controlnet = ControlNetModel.from_pretrained(model_path_to_load, **kwargs)
175             controlnets.append(controlnet)
176 
177             idx += 1
178             model_path_to_load = pretrained_model_path + f"_{idx}"
179 
180         logger.info(f"{len(controlnets)} controlnets loaded from {pretrained_model_path}.")
181 
182         if len(controlnets) == 0:
183             raise ValueError(
184                 f"No ControlNets found under {os.path.dirname(pretrained_model_path)}. Expected at least {pretrained_model_path + '_0'}."
185             )
186 
187         return cls(controlnets)
188
```
--- END FILES ---

Please first localize the bug based on the issue statement, and then generate a patch according to the `git diff` format fenced by three backticks.

Here is an example:

```diff
diff --git a/examples/server_async.py b/examples/server_async.py
--- a/examples/server_async.py
+++ b/examples/server_async.py
@@ -313,4 +313,4 @@
 
 
 if __name__ == "__main__":
-    asyncio.run(run_async_server("."), debug=True)
+    asyncio.run(run_async_server(), debug=True)
diff --git a/examples/server_sync.py b/examples/server_sync.py
--- a/examples/server_sync.py
+++ b/examples/server_sync.py
@@ -313,5 +313,5 @@
 
 
 if __name__ == "__main__":
-    server = run_sync_server(".")
+    server = run_sync_server()
     server.shutdown()

```

diff --git a/src/diffusers/pipelines/controlnet/multicontrolnet.py b/src/diffusers/pipelines/controlnet/multicontrolnet.py
--- a/src/diffusers/pipelines/controlnet/multicontrolnet.py
+++ b/src/diffusers/pipelines/controlnet/multicontrolnet.py
@@ -100,20 +100,16 @@
             variant (`str`, *optional*):
                 If specified, weights are saved in the format pytorch_model.<variant>.bin.
         """
-        idx = 0
-        model_path_to_save = save_directory
-        for controlnet in self.nets:
+        for idx, controlnet in enumerate(self.nets):
+            suffix = "" if idx == 0 else f"_{idx}"
             controlnet.save_pretrained(
-                model_path_to_save,
+                save_directory + suffix,
                 is_main_process=is_main_process,
                 save_function=save_function,
                 safe_serialization=safe_serialization,
                 variant=variant,
             )
 
-            idx += 1
-            model_path_to_save = model_path_to_save + f"_{idx}"
-
     @classmethod
     def from_pretrained(cls, pretrained_model_path: Optional[Union[str, os.PathLike]], **kwargs):
         r"""

{"golden_diff": "diff --git a/src/diffusers/pipelines/controlnet/multicontrolnet.py b/src/diffusers/pipelines/controlnet/multicontrolnet.py\n--- a/src/diffusers/pipelines/controlnet/multicontrolnet.py\n+++ b/src/diffusers/pipelines/controlnet/multicontrolnet.py\n@@ -100,20 +100,16 @@\n             variant (`str`, *optional*):\n                 If specified, weights are saved in the format pytorch_model.<variant>.bin.\n         \"\"\"\n-        idx = 0\n-        model_path_to_save = save_directory\n-        for controlnet in self.nets:\n+        for idx, controlnet in enumerate(self.nets):\n+            suffix = \"\" if idx == 0 else f\"_{idx}\"\n             controlnet.save_pretrained(\n-                model_path_to_save,\n+                save_directory + suffix,\n                 is_main_process=is_main_process,\n                 save_function=save_function,\n                 safe_serialization=safe_serialization,\n                 variant=variant,\n             )\n \n-            idx += 1\n-            model_path_to_save = model_path_to_save + f\"_{idx}\"\n-\n     @classmethod\n     def from_pretrained(cls, pretrained_model_path: Optional[Union[str, os.PathLike]], **kwargs):\n         r\"\"\"\n", "issue": "Multi-controlnet formatting issue\n### Describe the bug\r\n\r\nHi.\r\nThere is an inconsistency between `from_pretrained` and `save_pretrained` within the Multicontrolnet class.\r\nThe from_pretrained function returns a directory structure like this: controlnet, controlnet_1, controlnet_2, \r\nwhereas save_pretrained is like this: controlnet, controlnet_1, controlnet_1_2.\r\nWhen loading a saved model, if there are 3 controlnets, the last controlnet will not be loaded. (more than 2 always same issue)\r\n\r\n\r\n### Reproduction\r\n\r\nI don't think there is no need to reproduce the code as it's pretty clear issue.\r\nFor compatibility, how about changing the `save_pretrained` function in Multi-ControlNet to look like the code below? \r\n```\r\ndef save_pretrained(\r\n        self,\r\n        save_directory: Union[str, os.PathLike],\r\n        is_main_process: bool = True,\r\n        save_function: Callable = None,\r\n        safe_serialization: bool = True,\r\n        variant: Optional[str] = None,\r\n    ):\r\n        \"\"\"\r\n        Save a model and its configuration file to a directory, so that it can be re-loaded using the\r\n        `[`~pipelines.controlnet.MultiControlNetModel.from_pretrained`]` class method.\r\n\r\n        Arguments:\r\n            save_directory (`str` or `os.PathLike`):\r\n                Directory to which to save. Will be created if it doesn't exist.\r\n            is_main_process (`bool`, *optional*, defaults to `True`):\r\n                Whether the process calling this is the main process or not. Useful when in distributed training like\r\n                TPUs and need to call this function on all processes. In this case, set `is_main_process=True` only on\r\n                the main process to avoid race conditions.\r\n            save_function (`Callable`):\r\n                The function to use to save the state dictionary. Useful on distributed training like TPUs when one\r\n                need to replace `torch.save` by another method. Can be configured with the environment variable\r\n                `DIFFUSERS_SAVE_MODE`.\r\n            safe_serialization (`bool`, *optional*, defaults to `True`):\r\n                Whether to save the model using `safetensors` or the traditional PyTorch way (that uses `pickle`).\r\n            variant (`str`, *optional*):\r\n                If specified, weights are saved in the format pytorch_model.<variant>.bin.\r\n        \"\"\"\r\n        model_path_to_save = save_directory\r\n        for idx, controlnet in enumerate(self.nets):\r\n            suffix = \"\" if idx == 0 else f\"_{idx}\"\r\n            controlnet.save_pretrained(\r\n                model_path_to_save + suffix,\r\n                is_main_process=is_main_process,\r\n                save_function=save_function,\r\n                safe_serialization=safe_serialization,\r\n                variant=variant,\r\n            )\r\n```\r\n\r\n### Logs\r\n\r\n_No response_\r\n\r\n### System Info\r\n\r\nDiffusers 0.27.2\r\n\r\n### Who can help?\r\n\r\n@sayakpaul \n", "before_files": [{"content": "import os\nfrom typing import Any, Callable, Dict, List, Optional, Tuple, Union\n\nimport torch\nfrom torch import nn\n\nfrom ...models.controlnet import ControlNetModel, ControlNetOutput\nfrom ...models.modeling_utils import ModelMixin\nfrom ...utils import logging\n\n\nlogger = logging.get_logger(__name__)\n\n\nclass MultiControlNetModel(ModelMixin):\n    r\"\"\"\n    Multiple `ControlNetModel` wrapper class for Multi-ControlNet\n\n    This module is a wrapper for multiple instances of the `ControlNetModel`. The `forward()` API is designed to be\n    compatible with `ControlNetModel`.\n\n    Args:\n        controlnets (`List[ControlNetModel]`):\n            Provides additional conditioning to the unet during the denoising process. You must set multiple\n            `ControlNetModel` as a list.\n    \"\"\"\n\n    def __init__(self, controlnets: Union[List[ControlNetModel], Tuple[ControlNetModel]]):\n        super().__init__()\n        self.nets = nn.ModuleList(controlnets)\n\n    def forward(\n        self,\n        sample: torch.Tensor,\n        timestep: Union[torch.Tensor, float, int],\n        encoder_hidden_states: torch.Tensor,\n        controlnet_cond: List[torch.tensor],\n        conditioning_scale: List[float],\n        class_labels: Optional[torch.Tensor] = None,\n        timestep_cond: Optional[torch.Tensor] = None,\n        attention_mask: Optional[torch.Tensor] = None,\n        added_cond_kwargs: Optional[Dict[str, torch.Tensor]] = None,\n        cross_attention_kwargs: Optional[Dict[str, Any]] = None,\n        guess_mode: bool = False,\n        return_dict: bool = True,\n    ) -> Union[ControlNetOutput, Tuple]:\n        for i, (image, scale, controlnet) in enumerate(zip(controlnet_cond, conditioning_scale, self.nets)):\n            down_samples, mid_sample = controlnet(\n                sample=sample,\n                timestep=timestep,\n                encoder_hidden_states=encoder_hidden_states,\n                controlnet_cond=image,\n                conditioning_scale=scale,\n                class_labels=class_labels,\n                timestep_cond=timestep_cond,\n                attention_mask=attention_mask,\n                added_cond_kwargs=added_cond_kwargs,\n                cross_attention_kwargs=cross_attention_kwargs,\n                guess_mode=guess_mode,\n                return_dict=return_dict,\n            )\n\n            # merge samples\n            if i == 0:\n                down_block_res_samples, mid_block_res_sample = down_samples, mid_sample\n            else:\n                down_block_res_samples = [\n                    samples_prev + samples_curr\n                    for samples_prev, samples_curr in zip(down_block_res_samples, down_samples)\n                ]\n                mid_block_res_sample += mid_sample\n\n        return down_block_res_samples, mid_block_res_sample\n\n    def save_pretrained(\n        self,\n        save_directory: Union[str, os.PathLike],\n        is_main_process: bool = True,\n        save_function: Callable = None,\n        safe_serialization: bool = True,\n        variant: Optional[str] = None,\n    ):\n        \"\"\"\n        Save a model and its configuration file to a directory, so that it can be re-loaded using the\n        `[`~pipelines.controlnet.MultiControlNetModel.from_pretrained`]` class method.\n\n        Arguments:\n            save_directory (`str` or `os.PathLike`):\n                Directory to which to save. Will be created if it doesn't exist.\n            is_main_process (`bool`, *optional*, defaults to `True`):\n                Whether the process calling this is the main process or not. Useful when in distributed training like\n                TPUs and need to call this function on all processes. In this case, set `is_main_process=True` only on\n                the main process to avoid race conditions.\n            save_function (`Callable`):\n                The function to use to save the state dictionary. Useful on distributed training like TPUs when one\n                need to replace `torch.save` by another method. Can be configured with the environment variable\n                `DIFFUSERS_SAVE_MODE`.\n            safe_serialization (`bool`, *optional*, defaults to `True`):\n                Whether to save the model using `safetensors` or the traditional PyTorch way (that uses `pickle`).\n            variant (`str`, *optional*):\n                If specified, weights are saved in the format pytorch_model.<variant>.bin.\n        \"\"\"\n        idx = 0\n        model_path_to_save = save_directory\n        for controlnet in self.nets:\n            controlnet.save_pretrained(\n                model_path_to_save,\n                is_main_process=is_main_process,\n                save_function=save_function,\n                safe_serialization=safe_serialization,\n                variant=variant,\n            )\n\n            idx += 1\n            model_path_to_save = model_path_to_save + f\"_{idx}\"\n\n    @classmethod\n    def from_pretrained(cls, pretrained_model_path: Optional[Union[str, os.PathLike]], **kwargs):\n        r\"\"\"\n        Instantiate a pretrained MultiControlNet model from multiple pre-trained controlnet models.\n\n        The model is set in evaluation mode by default using `model.eval()` (Dropout modules are deactivated). To train\n        the model, you should first set it back in training mode with `model.train()`.\n\n        The warning *Weights from XXX not initialized from pretrained model* means that the weights of XXX do not come\n        pretrained with the rest of the model. It is up to you to train those weights with a downstream fine-tuning\n        task.\n\n        The warning *Weights from XXX not used in YYY* means that the layer XXX is not used by YYY, therefore those\n        weights are discarded.\n\n        Parameters:\n            pretrained_model_path (`os.PathLike`):\n                A path to a *directory* containing model weights saved using\n                [`~diffusers.pipelines.controlnet.MultiControlNetModel.save_pretrained`], e.g.,\n                `./my_model_directory/controlnet`.\n            torch_dtype (`str` or `torch.dtype`, *optional*):\n                Override the default `torch.dtype` and load the model under this dtype. If `\"auto\"` is passed the dtype\n                will be automatically derived from the model's weights.\n            output_loading_info(`bool`, *optional*, defaults to `False`):\n                Whether or not to also return a dictionary containing missing keys, unexpected keys and error messages.\n            device_map (`str` or `Dict[str, Union[int, str, torch.device]]`, *optional*):\n                A map that specifies where each submodule should go. It doesn't need to be refined to each\n                parameter/buffer name, once a given module name is inside, every submodule of it will be sent to the\n                same device.\n\n                To have Accelerate compute the most optimized `device_map` automatically, set `device_map=\"auto\"`. For\n                more information about each option see [designing a device\n                map](https://hf.co/docs/accelerate/main/en/usage_guides/big_modeling#designing-a-device-map).\n            max_memory (`Dict`, *optional*):\n                A dictionary device identifier to maximum memory. Will default to the maximum memory available for each\n                GPU and the available CPU RAM if unset.\n            low_cpu_mem_usage (`bool`, *optional*, defaults to `True` if torch version >= 1.9.0 else `False`):\n                Speed up model loading by not initializing the weights and only loading the pre-trained weights. This\n                also tries to not use more than 1x model size in CPU memory (including peak memory) while loading the\n                model. This is only supported when torch version >= 1.9.0. If you are using an older version of torch,\n                setting this argument to `True` will raise an error.\n            variant (`str`, *optional*):\n                If specified load weights from `variant` filename, *e.g.* pytorch_model.<variant>.bin. `variant` is\n                ignored when using `from_flax`.\n            use_safetensors (`bool`, *optional*, defaults to `None`):\n                If set to `None`, the `safetensors` weights will be downloaded if they're available **and** if the\n                `safetensors` library is installed. If set to `True`, the model will be forcibly loaded from\n                `safetensors` weights. If set to `False`, loading will *not* use `safetensors`.\n        \"\"\"\n        idx = 0\n        controlnets = []\n\n        # load controlnet and append to list until no controlnet directory exists anymore\n        # first controlnet has to be saved under `./mydirectory/controlnet` to be compliant with `DiffusionPipeline.from_prertained`\n        # second, third, ... controlnets have to be saved under `./mydirectory/controlnet_1`, `./mydirectory/controlnet_2`, ...\n        model_path_to_load = pretrained_model_path\n        while os.path.isdir(model_path_to_load):\n            controlnet = ControlNetModel.from_pretrained(model_path_to_load, **kwargs)\n            controlnets.append(controlnet)\n\n            idx += 1\n            model_path_to_load = pretrained_model_path + f\"_{idx}\"\n\n        logger.info(f\"{len(controlnets)} controlnets loaded from {pretrained_model_path}.\")\n\n        if len(controlnets) == 0:\n            raise ValueError(\n                f\"No ControlNets found under {os.path.dirname(pretrained_model_path)}. Expected at least {pretrained_model_path + '_0'}.\"\n            )\n\n        return cls(controlnets)\n", "path": "src/diffusers/pipelines/controlnet/multicontrolnet.py"}], "after_files": [{"content": "import os\nfrom typing import Any, Callable, Dict, List, Optional, Tuple, Union\n\nimport torch\nfrom torch import nn\n\nfrom ...models.controlnet import ControlNetModel, ControlNetOutput\nfrom ...models.modeling_utils import ModelMixin\nfrom ...utils import logging\n\n\nlogger = logging.get_logger(__name__)\n\n\nclass MultiControlNetModel(ModelMixin):\n    r\"\"\"\n    Multiple `ControlNetModel` wrapper class for Multi-ControlNet\n\n    This module is a wrapper for multiple instances of the `ControlNetModel`. The `forward()` API is designed to be\n    compatible with `ControlNetModel`.\n\n    Args:\n        controlnets (`List[ControlNetModel]`):\n            Provides additional conditioning to the unet during the denoising process. You must set multiple\n            `ControlNetModel` as a list.\n    \"\"\"\n\n    def __init__(self, controlnets: Union[List[ControlNetModel], Tuple[ControlNetModel]]):\n        super().__init__()\n        self.nets = nn.ModuleList(controlnets)\n\n    def forward(\n        self,\n        sample: torch.Tensor,\n        timestep: Union[torch.Tensor, float, int],\n        encoder_hidden_states: torch.Tensor,\n        controlnet_cond: List[torch.tensor],\n        conditioning_scale: List[float],\n        class_labels: Optional[torch.Tensor] = None,\n        timestep_cond: Optional[torch.Tensor] = None,\n        attention_mask: Optional[torch.Tensor] = None,\n        added_cond_kwargs: Optional[Dict[str, torch.Tensor]] = None,\n        cross_attention_kwargs: Optional[Dict[str, Any]] = None,\n        guess_mode: bool = False,\n        return_dict: bool = True,\n    ) -> Union[ControlNetOutput, Tuple]:\n        for i, (image, scale, controlnet) in enumerate(zip(controlnet_cond, conditioning_scale, self.nets)):\n            down_samples, mid_sample = controlnet(\n                sample=sample,\n                timestep=timestep,\n                encoder_hidden_states=encoder_hidden_states,\n                controlnet_cond=image,\n                conditioning_scale=scale,\n                class_labels=class_labels,\n                timestep_cond=timestep_cond,\n                attention_mask=attention_mask,\n                added_cond_kwargs=added_cond_kwargs,\n                cross_attention_kwargs=cross_attention_kwargs,\n                guess_mode=guess_mode,\n                return_dict=return_dict,\n            )\n\n            # merge samples\n            if i == 0:\n                down_block_res_samples, mid_block_res_sample = down_samples, mid_sample\n            else:\n                down_block_res_samples = [\n                    samples_prev + samples_curr\n                    for samples_prev, samples_curr in zip(down_block_res_samples, down_samples)\n                ]\n                mid_block_res_sample += mid_sample\n\n        return down_block_res_samples, mid_block_res_sample\n\n    def save_pretrained(\n        self,\n        save_directory: Union[str, os.PathLike],\n        is_main_process: bool = True,\n        save_function: Callable = None,\n        safe_serialization: bool = True,\n        variant: Optional[str] = None,\n    ):\n        \"\"\"\n        Save a model and its configuration file to a directory, so that it can be re-loaded using the\n        `[`~pipelines.controlnet.MultiControlNetModel.from_pretrained`]` class method.\n\n        Arguments:\n            save_directory (`str` or `os.PathLike`):\n                Directory to which to save. Will be created if it doesn't exist.\n            is_main_process (`bool`, *optional*, defaults to `True`):\n                Whether the process calling this is the main process or not. Useful when in distributed training like\n                TPUs and need to call this function on all processes. In this case, set `is_main_process=True` only on\n                the main process to avoid race conditions.\n            save_function (`Callable`):\n                The function to use to save the state dictionary. Useful on distributed training like TPUs when one\n                need to replace `torch.save` by another method. Can be configured with the environment variable\n                `DIFFUSERS_SAVE_MODE`.\n            safe_serialization (`bool`, *optional*, defaults to `True`):\n                Whether to save the model using `safetensors` or the traditional PyTorch way (that uses `pickle`).\n            variant (`str`, *optional*):\n                If specified, weights are saved in the format pytorch_model.<variant>.bin.\n        \"\"\"\n        for idx, controlnet in enumerate(self.nets):\n            suffix = \"\" if idx == 0 else f\"_{idx}\"\n            controlnet.save_pretrained(\n                save_directory + suffix,\n                is_main_process=is_main_process,\n                save_function=save_function,\n                safe_serialization=safe_serialization,\n                variant=variant,\n            )\n\n    @classmethod\n    def from_pretrained(cls, pretrained_model_path: Optional[Union[str, os.PathLike]], **kwargs):\n        r\"\"\"\n        Instantiate a pretrained MultiControlNet model from multiple pre-trained controlnet models.\n\n        The model is set in evaluation mode by default using `model.eval()` (Dropout modules are deactivated). To train\n        the model, you should first set it back in training mode with `model.train()`.\n\n        The warning *Weights from XXX not initialized from pretrained model* means that the weights of XXX do not come\n        pretrained with the rest of the model. It is up to you to train those weights with a downstream fine-tuning\n        task.\n\n        The warning *Weights from XXX not used in YYY* means that the layer XXX is not used by YYY, therefore those\n        weights are discarded.\n\n        Parameters:\n            pretrained_model_path (`os.PathLike`):\n                A path to a *directory* containing model weights saved using\n                [`~diffusers.pipelines.controlnet.MultiControlNetModel.save_pretrained`], e.g.,\n                `./my_model_directory/controlnet`.\n            torch_dtype (`str` or `torch.dtype`, *optional*):\n                Override the default `torch.dtype` and load the model under this dtype. If `\"auto\"` is passed the dtype\n                will be automatically derived from the model's weights.\n            output_loading_info(`bool`, *optional*, defaults to `False`):\n                Whether or not to also return a dictionary containing missing keys, unexpected keys and error messages.\n            device_map (`str` or `Dict[str, Union[int, str, torch.device]]`, *optional*):\n                A map that specifies where each submodule should go. It doesn't need to be refined to each\n                parameter/buffer name, once a given module name is inside, every submodule of it will be sent to the\n                same device.\n\n                To have Accelerate compute the most optimized `device_map` automatically, set `device_map=\"auto\"`. For\n                more information about each option see [designing a device\n                map](https://hf.co/docs/accelerate/main/en/usage_guides/big_modeling#designing-a-device-map).\n            max_memory (`Dict`, *optional*):\n                A dictionary device identifier to maximum memory. Will default to the maximum memory available for each\n                GPU and the available CPU RAM if unset.\n            low_cpu_mem_usage (`bool`, *optional*, defaults to `True` if torch version >= 1.9.0 else `False`):\n                Speed up model loading by not initializing the weights and only loading the pre-trained weights. This\n                also tries to not use more than 1x model size in CPU memory (including peak memory) while loading the\n                model. This is only supported when torch version >= 1.9.0. If you are using an older version of torch,\n                setting this argument to `True` will raise an error.\n            variant (`str`, *optional*):\n                If specified load weights from `variant` filename, *e.g.* pytorch_model.<variant>.bin. `variant` is\n                ignored when using `from_flax`.\n            use_safetensors (`bool`, *optional*, defaults to `None`):\n                If set to `None`, the `safetensors` weights will be downloaded if they're available **and** if the\n                `safetensors` library is installed. If set to `True`, the model will be forcibly loaded from\n                `safetensors` weights. If set to `False`, loading will *not* use `safetensors`.\n        \"\"\"\n        idx = 0\n        controlnets = []\n\n        # load controlnet and append to list until no controlnet directory exists anymore\n        # first controlnet has to be saved under `./mydirectory/controlnet` to be compliant with `DiffusionPipeline.from_prertained`\n        # second, third, ... controlnets have to be saved under `./mydirectory/controlnet_1`, `./mydirectory/controlnet_2`, ...\n        model_path_to_load = pretrained_model_path\n        while os.path.isdir(model_path_to_load):\n            controlnet = ControlNetModel.from_pretrained(model_path_to_load, **kwargs)\n            controlnets.append(controlnet)\n\n            idx += 1\n            model_path_to_load = pretrained_model_path + f\"_{idx}\"\n\n        logger.info(f\"{len(controlnets)} controlnets loaded from {pretrained_model_path}.\")\n\n        if len(controlnets) == 0:\n            raise ValueError(\n                f\"No ControlNets found under {os.path.dirname(pretrained_model_path)}. Expected at least {pretrained_model_path + '_0'}.\"\n            )\n\n        return cls(controlnets)\n", "path": "src/diffusers/pipelines/controlnet/multicontrolnet.py"}]}