We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
Keras documentation: multi_gpu_model examples do not show properly on the home page keras.io
Some examples of usage of `multi_gpu_model` appear on the documentation of the function in the [source code](https://github.com/keras-team/keras/blob/master/keras/utils/multi_gpu_utils.py).  However they do not display correctly on the [Keras home page](https://keras.io/utils/):

```Example 1 - Training models with weights merge on CPU

$Example_2_-_Training_models_with_weights_merge_on_CPU_using_cpu_relocation$0

Example 2 - Training models with weights merge on CPU using cpu_relocation

$Example_2_-_Training_models_with_weights_merge_on_CPU_using_cpu_relocation$1

Example 3 - Training models with weights merge on GPU (recommended for NV-link)

$Example_2_-_Training_models_with_weights_merge_on_CPU_using_cpu_relocation$2```
Keras documentation of multi_gpu_model: example 2 can be misleading 
In the Keras documentation for `multi_gpu_model`, it is stated:

> To save the multi-gpu model, use .save(fname) or .save_weights(fname) with the template model (the argument you passed to multi_gpu_model), rather than the model returned by multi_gpu_model.

However in example 2 the template model is overwritten by the multi-gpu model:

```python
         ..
         # Not needed to change the device scope for model definition:
         model = Xception(weights=None, ..)
         try:
             model = multi_gpu_model(model, cpu_relocation=True)
             print("Training using multiple GPUs..")
         except:
             print("Training using single GPU or CPU..")
         model.compile(..)
         ..
```

This means that in this example it would not be possible to save the weights of the template model. I suggest rewritting to something like:


```python
         ..
         # Not needed to change the device scope for model definition:
         model = Xception(weights=None, ..)
         try:
             parallel_model = multi_gpu_model(model, cpu_relocation=True)
             print("Training using multiple GPUs..")
         except ValueError:
             parallel_model = model
             print("Training using single GPU or CPU..")
         parallel_model.compile(..)
         ..
```

(I take this opportunity to except only a specific error)


--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
--- BEGIN FILES ---
Path: `keras/utils/multi_gpu_utils.py`
Content:
```
1 """Multi-GPU training utilities.
2 """
3 from __future__ import absolute_import
4 from __future__ import division
5 from __future__ import print_function
6 
7 from ..layers.merge import concatenate
8 from .. import backend as K
9 from ..layers.core import Lambda
10 from ..engine.training import Model
11 from ..models import clone_model
12 from ..utils.generic_utils import to_list
13 
14 
15 def _get_available_devices():
16     return [x.name for x in K.get_session().list_devices()]
17 
18 
19 def _normalize_device_name(name):
20     name = '/' + ':'.join(name.lower().replace('/', '').split(':')[-2:])
21     return name
22 
23 
24 def multi_gpu_model(model, gpus=None, cpu_merge=True, cpu_relocation=False):
25     """Replicates a model on different GPUs.
26 
27     Specifically, this function implements single-machine
28     multi-GPU data parallelism. It works in the following way:
29 
30     - Divide the model's input(s) into multiple sub-batches.
31     - Apply a model copy on each sub-batch. Every model copy
32         is executed on a dedicated GPU.
33     - Concatenate the results (on CPU) into one big batch.
34 
35     E.g. if your `batch_size` is 64 and you use `gpus=2`,
36     then we will divide the input into 2 sub-batches of 32 samples,
37     process each sub-batch on one GPU, then return the full
38     batch of 64 processed samples.
39 
40     This induces quasi-linear speedup on up to 8 GPUs.
41 
42     This function is only available with the TensorFlow backend
43     for the time being.
44 
45     # Arguments
46         model: A Keras model instance. To avoid OOM errors,
47             this model could have been built on CPU, for instance
48             (see usage example below).
49         gpus: Integer >= 2 or list of integers, number of GPUs or
50             list of GPU IDs on which to create model replicas.
51         cpu_merge: A boolean value to identify whether to force
52             merging model weights under the scope of the CPU or not.
53         cpu_relocation: A boolean value to identify whether to
54             create the model's weights under the scope of the CPU.
55             If the model is not defined under any preceding device
56             scope, you can still rescue it by activating this option.
57 
58     # Returns
59         A Keras `Model` instance which can be used just like the initial
60         `model` argument, but which distributes its workload on multiple GPUs.
61 
62     # Example 1 - Training models with weights merge on CPU
63 
64     ```python
65         import tensorflow as tf
66         from keras.applications import Xception
67         from keras.utils import multi_gpu_model
68         import numpy as np
69 
70         num_samples = 1000
71         height = 224
72         width = 224
73         num_classes = 1000
74 
75         # Instantiate the base model (or "template" model).
76         # We recommend doing this with under a CPU device scope,
77         # so that the model's weights are hosted on CPU memory.
78         # Otherwise they may end up hosted on a GPU, which would
79         # complicate weight sharing.
80         with tf.device('/cpu:0'):
81             model = Xception(weights=None,
82                              input_shape=(height, width, 3),
83                              classes=num_classes)
84 
85         # Replicates the model on 8 GPUs.
86         # This assumes that your machine has 8 available GPUs.
87         parallel_model = multi_gpu_model(model, gpus=8)
88         parallel_model.compile(loss='categorical_crossentropy',
89                                optimizer='rmsprop')
90 
91         # Generate dummy data.
92         x = np.random.random((num_samples, height, width, 3))
93         y = np.random.random((num_samples, num_classes))
94 
95         # This `fit` call will be distributed on 8 GPUs.
96         # Since the batch size is 256, each GPU will process 32 samples.
97         parallel_model.fit(x, y, epochs=20, batch_size=256)
98 
99         # Save model via the template model (which shares the same weights):
100         model.save('my_model.h5')
101     ```
102 
103     # Example 2 - Training models with weights merge on CPU using cpu_relocation
104 
105     ```python
106          ..
107          # Not needed to change the device scope for model definition:
108          model = Xception(weights=None, ..)
109 
110          try:
111              model = multi_gpu_model(model, cpu_relocation=True)
112              print("Training using multiple GPUs..")
113          except:
114              print("Training using single GPU or CPU..")
115 
116          model.compile(..)
117          ..
118     ```
119 
120     # Example 3 - Training models with weights merge on GPU (recommended for NV-link)
121 
122     ```python
123          ..
124          # Not needed to change the device scope for model definition:
125          model = Xception(weights=None, ..)
126 
127          try:
128              model = multi_gpu_model(model, cpu_merge=False)
129              print("Training using multiple GPUs..")
130          except:
131              print("Training using single GPU or CPU..")
132 
133          model.compile(..)
134          ..
135     ```
136 
137     # On model saving
138 
139     To save the multi-gpu model, use `.save(fname)` or `.save_weights(fname)`
140     with the template model (the argument you passed to `multi_gpu_model`),
141     rather than the model returned by `multi_gpu_model`.
142     """
143     if K.backend() != 'tensorflow':
144         raise ValueError('`multi_gpu_model` is only available '
145                          'with the TensorFlow backend.')
146 
147     available_devices = _get_available_devices()
148     available_devices = [_normalize_device_name(name)
149                          for name in available_devices]
150     if not gpus:
151         # Using all visible GPUs when not specifying `gpus`
152         # e.g. CUDA_VISIBLE_DEVICES=0,2 python keras_mgpu.py
153         gpus = len([x for x in available_devices if 'gpu' in x])
154 
155     if isinstance(gpus, (list, tuple)):
156         if len(gpus) <= 1:
157             raise ValueError('For multi-gpu usage to be effective, '
158                              'call `multi_gpu_model` with `len(gpus) >= 2`. '
159                              'Received: `gpus=%s`' % gpus)
160         num_gpus = len(gpus)
161         target_gpu_ids = gpus
162     else:
163         if gpus <= 1:
164             raise ValueError('For multi-gpu usage to be effective, '
165                              'call `multi_gpu_model` with `gpus >= 2`. '
166                              'Received: `gpus=%d`' % gpus)
167         num_gpus = gpus
168         target_gpu_ids = range(num_gpus)
169 
170     import tensorflow as tf
171 
172     target_devices = ['/cpu:0'] + ['/gpu:%d' % i for i in target_gpu_ids]
173     for device in target_devices:
174         if device not in available_devices:
175             raise ValueError(
176                 'To call `multi_gpu_model` with `gpus=%s`, '
177                 'we expect the following devices to be available: %s. '
178                 'However this machine only has: %s. '
179                 'Try reducing `gpus`.' % (gpus,
180                                           target_devices,
181                                           available_devices))
182 
183     def get_slice(data, i, parts):
184         shape = K.shape(data)
185         batch_size = shape[:1]
186         input_shape = shape[1:]
187         step = batch_size // parts
188         if i == parts - 1:
189             size = batch_size - step * i
190         else:
191             size = step
192         size = K.concatenate([size, input_shape], axis=0)
193         stride = K.concatenate([step, input_shape * 0], axis=0)
194         start = stride * i
195         return K.slice(data, start, size)
196 
197     # Relocate the model definition under CPU device scope if needed
198     if cpu_relocation:
199         with tf.device('/cpu:0'):
200             model = clone_model(model)
201 
202     all_outputs = []
203     for i in range(len(model.outputs)):
204         all_outputs.append([])
205 
206     # Place a copy of the model on each GPU,
207     # each getting a slice of the inputs.
208     for i, gpu_id in enumerate(target_gpu_ids):
209         with tf.device('/gpu:%d' % gpu_id):
210             with tf.name_scope('replica_%d' % gpu_id):
211                 inputs = []
212                 # Retrieve a slice of the input.
213                 for x in model.inputs:
214                     # In-place input splitting which is not only
215                     # 5% ~ 12% faster but also less GPU memory
216                     # duplication.
217                     with tf.device(x.device):
218                         input_shape = K.int_shape(x)[1:]
219                         slice_i = Lambda(get_slice,
220                                          output_shape=input_shape,
221                                          arguments={'i': i,
222                                                     'parts': num_gpus})(x)
223                         inputs.append(slice_i)
224 
225                 # Apply model on slice
226                 # (creating a model replica on the target device).
227                 outputs = model(inputs)
228                 outputs = to_list(outputs)
229 
230                 # Save the outputs for merging back together later.
231                 for o in range(len(outputs)):
232                     all_outputs[o].append(outputs[o])
233 
234     # Deduplicate output names to handle Siamese networks.
235     occurrences = {}
236     for n in model.output_names:
237         if n not in occurrences:
238             occurrences[n] = 1
239         else:
240             occurrences[n] += 1
241     conflict_counter = {n: 0 for n, count in occurrences.items() if count > 1}
242     output_names = []
243     for n in model.output_names:
244         if n in conflict_counter:
245             conflict_counter[n] += 1
246             n += '_%d' % conflict_counter[n]
247         output_names.append(n)
248 
249     # Merge outputs under expected scope.
250     with tf.device('/cpu:0' if cpu_merge else '/gpu:%d' % target_gpu_ids[0]):
251         merged = []
252         for name, outputs in zip(output_names, all_outputs):
253             merged.append(concatenate(outputs,
254                                       axis=0, name=name))
255         return Model(model.inputs, merged)
256
```
--- END FILES ---

Please first localize the bug based on the issue statement, and then generate a patch according to the `git diff` format fenced by three backticks.

Here is an example:

```diff
diff --git a/examples/server_async.py b/examples/server_async.py
--- a/examples/server_async.py
+++ b/examples/server_async.py
@@ -313,4 +313,4 @@
 
 
 if __name__ == "__main__":
-    asyncio.run(run_async_server("."), debug=True)
+    asyncio.run(run_async_server(), debug=True)
diff --git a/examples/server_sync.py b/examples/server_sync.py
--- a/examples/server_sync.py
+++ b/examples/server_sync.py
@@ -313,5 +313,5 @@
 
 
 if __name__ == "__main__":
-    server = run_sync_server(".")
+    server = run_sync_server()
     server.shutdown()

```

diff --git a/keras/utils/multi_gpu_utils.py b/keras/utils/multi_gpu_utils.py
--- a/keras/utils/multi_gpu_utils.py
+++ b/keras/utils/multi_gpu_utils.py
@@ -59,7 +59,9 @@
         A Keras `Model` instance which can be used just like the initial
         `model` argument, but which distributes its workload on multiple GPUs.
 
-    # Example 1 - Training models with weights merge on CPU
+    # Examples
+
+    Example 1 - Training models with weights merge on CPU
 
     ```python
         import tensorflow as tf
@@ -100,7 +102,7 @@
         model.save('my_model.h5')
     ```
 
-    # Example 2 - Training models with weights merge on CPU using cpu_relocation
+    Example 2 - Training models with weights merge on CPU using cpu_relocation
 
     ```python
          ..
@@ -108,16 +110,16 @@
          model = Xception(weights=None, ..)
 
          try:
-             model = multi_gpu_model(model, cpu_relocation=True)
+             parallel_model = multi_gpu_model(model, cpu_relocation=True)
              print("Training using multiple GPUs..")
-         except:
+         except ValueError:
+             parallel_model = model
              print("Training using single GPU or CPU..")
-
-         model.compile(..)
+         parallel_model.compile(..)
          ..
     ```
 
-    # Example 3 - Training models with weights merge on GPU (recommended for NV-link)
+    Example 3 - Training models with weights merge on GPU (recommended for NV-link)
 
     ```python
          ..
@@ -125,12 +127,13 @@
          model = Xception(weights=None, ..)
 
          try:
-             model = multi_gpu_model(model, cpu_merge=False)
+             parallel_model = multi_gpu_model(model, cpu_merge=False)
              print("Training using multiple GPUs..")
          except:
+             parallel_model = model
              print("Training using single GPU or CPU..")
 
-         model.compile(..)
+         parallel_model.compile(..)
          ..
     ```

{"golden_diff": "diff --git a/keras/utils/multi_gpu_utils.py b/keras/utils/multi_gpu_utils.py\n--- a/keras/utils/multi_gpu_utils.py\n+++ b/keras/utils/multi_gpu_utils.py\n@@ -59,7 +59,9 @@\n         A Keras `Model` instance which can be used just like the initial\n         `model` argument, but which distributes its workload on multiple GPUs.\n \n-    # Example 1 - Training models with weights merge on CPU\n+    # Examples\n+\n+    Example 1 - Training models with weights merge on CPU\n \n     ```python\n         import tensorflow as tf\n@@ -100,7 +102,7 @@\n         model.save('my_model.h5')\n     ```\n \n-    # Example 2 - Training models with weights merge on CPU using cpu_relocation\n+    Example 2 - Training models with weights merge on CPU using cpu_relocation\n \n     ```python\n          ..\n@@ -108,16 +110,16 @@\n          model = Xception(weights=None, ..)\n \n          try:\n-             model = multi_gpu_model(model, cpu_relocation=True)\n+             parallel_model = multi_gpu_model(model, cpu_relocation=True)\n              print(\"Training using multiple GPUs..\")\n-         except:\n+         except ValueError:\n+             parallel_model = model\n              print(\"Training using single GPU or CPU..\")\n-\n-         model.compile(..)\n+         parallel_model.compile(..)\n          ..\n     ```\n \n-    # Example 3 - Training models with weights merge on GPU (recommended for NV-link)\n+    Example 3 - Training models with weights merge on GPU (recommended for NV-link)\n \n     ```python\n          ..\n@@ -125,12 +127,13 @@\n          model = Xception(weights=None, ..)\n \n          try:\n-             model = multi_gpu_model(model, cpu_merge=False)\n+             parallel_model = multi_gpu_model(model, cpu_merge=False)\n              print(\"Training using multiple GPUs..\")\n          except:\n+             parallel_model = model\n              print(\"Training using single GPU or CPU..\")\n \n-         model.compile(..)\n+         parallel_model.compile(..)\n          ..\n     ```\n", "issue": "Keras documentation: multi_gpu_model examples do not show properly on the home page keras.io\nSome examples of usage of `multi_gpu_model` appear on the documentation of the function in the [source code](https://github.com/keras-team/keras/blob/master/keras/utils/multi_gpu_utils.py).  However they do not display correctly on the [Keras home page](https://keras.io/utils/):\r\n\r\n```Example 1 - Training models with weights merge on CPU\r\n\r\n$Example_2_-_Training_models_with_weights_merge_on_CPU_using_cpu_relocation$0\r\n\r\nExample 2 - Training models with weights merge on CPU using cpu_relocation\r\n\r\n$Example_2_-_Training_models_with_weights_merge_on_CPU_using_cpu_relocation$1\r\n\r\nExample 3 - Training models with weights merge on GPU (recommended for NV-link)\r\n\r\n$Example_2_-_Training_models_with_weights_merge_on_CPU_using_cpu_relocation$2```\nKeras documentation of multi_gpu_model: example 2 can be misleading \nIn the Keras documentation for `multi_gpu_model`, it is stated:\r\n\r\n> To save the multi-gpu model, use .save(fname) or .save_weights(fname) with the template model (the argument you passed to multi_gpu_model), rather than the model returned by multi_gpu_model.\r\n\r\nHowever in example 2 the template model is overwritten by the multi-gpu model:\r\n\r\n```python\r\n         ..\r\n         # Not needed to change the device scope for model definition:\r\n         model = Xception(weights=None, ..)\r\n         try:\r\n             model = multi_gpu_model(model, cpu_relocation=True)\r\n             print(\"Training using multiple GPUs..\")\r\n         except:\r\n             print(\"Training using single GPU or CPU..\")\r\n         model.compile(..)\r\n         ..\r\n```\r\n\r\nThis means that in this example it would not be possible to save the weights of the template model. I suggest rewritting to something like:\r\n\r\n\r\n```python\r\n         ..\r\n         # Not needed to change the device scope for model definition:\r\n         model = Xception(weights=None, ..)\r\n         try:\r\n             parallel_model = multi_gpu_model(model, cpu_relocation=True)\r\n             print(\"Training using multiple GPUs..\")\r\n         except ValueError:\r\n             parallel_model = model\r\n             print(\"Training using single GPU or CPU..\")\r\n         parallel_model.compile(..)\r\n         ..\r\n```\r\n\r\n(I take this opportunity to except only a specific error)\r\n\n", "before_files": [{"content": "\"\"\"Multi-GPU training utilities.\n\"\"\"\nfrom __future__ import absolute_import\nfrom __future__ import division\nfrom __future__ import print_function\n\nfrom ..layers.merge import concatenate\nfrom .. import backend as K\nfrom ..layers.core import Lambda\nfrom ..engine.training import Model\nfrom ..models import clone_model\nfrom ..utils.generic_utils import to_list\n\n\ndef _get_available_devices():\n    return [x.name for x in K.get_session().list_devices()]\n\n\ndef _normalize_device_name(name):\n    name = '/' + ':'.join(name.lower().replace('/', '').split(':')[-2:])\n    return name\n\n\ndef multi_gpu_model(model, gpus=None, cpu_merge=True, cpu_relocation=False):\n    \"\"\"Replicates a model on different GPUs.\n\n    Specifically, this function implements single-machine\n    multi-GPU data parallelism. It works in the following way:\n\n    - Divide the model's input(s) into multiple sub-batches.\n    - Apply a model copy on each sub-batch. Every model copy\n        is executed on a dedicated GPU.\n    - Concatenate the results (on CPU) into one big batch.\n\n    E.g. if your `batch_size` is 64 and you use `gpus=2`,\n    then we will divide the input into 2 sub-batches of 32 samples,\n    process each sub-batch on one GPU, then return the full\n    batch of 64 processed samples.\n\n    This induces quasi-linear speedup on up to 8 GPUs.\n\n    This function is only available with the TensorFlow backend\n    for the time being.\n\n    # Arguments\n        model: A Keras model instance. To avoid OOM errors,\n            this model could have been built on CPU, for instance\n            (see usage example below).\n        gpus: Integer >= 2 or list of integers, number of GPUs or\n            list of GPU IDs on which to create model replicas.\n        cpu_merge: A boolean value to identify whether to force\n            merging model weights under the scope of the CPU or not.\n        cpu_relocation: A boolean value to identify whether to\n            create the model's weights under the scope of the CPU.\n            If the model is not defined under any preceding device\n            scope, you can still rescue it by activating this option.\n\n    # Returns\n        A Keras `Model` instance which can be used just like the initial\n        `model` argument, but which distributes its workload on multiple GPUs.\n\n    # Example 1 - Training models with weights merge on CPU\n\n    ```python\n        import tensorflow as tf\n        from keras.applications import Xception\n        from keras.utils import multi_gpu_model\n        import numpy as np\n\n        num_samples = 1000\n        height = 224\n        width = 224\n        num_classes = 1000\n\n        # Instantiate the base model (or \"template\" model).\n        # We recommend doing this with under a CPU device scope,\n        # so that the model's weights are hosted on CPU memory.\n        # Otherwise they may end up hosted on a GPU, which would\n        # complicate weight sharing.\n        with tf.device('/cpu:0'):\n            model = Xception(weights=None,\n                             input_shape=(height, width, 3),\n                             classes=num_classes)\n\n        # Replicates the model on 8 GPUs.\n        # This assumes that your machine has 8 available GPUs.\n        parallel_model = multi_gpu_model(model, gpus=8)\n        parallel_model.compile(loss='categorical_crossentropy',\n                               optimizer='rmsprop')\n\n        # Generate dummy data.\n        x = np.random.random((num_samples, height, width, 3))\n        y = np.random.random((num_samples, num_classes))\n\n        # This `fit` call will be distributed on 8 GPUs.\n        # Since the batch size is 256, each GPU will process 32 samples.\n        parallel_model.fit(x, y, epochs=20, batch_size=256)\n\n        # Save model via the template model (which shares the same weights):\n        model.save('my_model.h5')\n    ```\n\n    # Example 2 - Training models with weights merge on CPU using cpu_relocation\n\n    ```python\n         ..\n         # Not needed to change the device scope for model definition:\n         model = Xception(weights=None, ..)\n\n         try:\n             model = multi_gpu_model(model, cpu_relocation=True)\n             print(\"Training using multiple GPUs..\")\n         except:\n             print(\"Training using single GPU or CPU..\")\n\n         model.compile(..)\n         ..\n    ```\n\n    # Example 3 - Training models with weights merge on GPU (recommended for NV-link)\n\n    ```python\n         ..\n         # Not needed to change the device scope for model definition:\n         model = Xception(weights=None, ..)\n\n         try:\n             model = multi_gpu_model(model, cpu_merge=False)\n             print(\"Training using multiple GPUs..\")\n         except:\n             print(\"Training using single GPU or CPU..\")\n\n         model.compile(..)\n         ..\n    ```\n\n    # On model saving\n\n    To save the multi-gpu model, use `.save(fname)` or `.save_weights(fname)`\n    with the template model (the argument you passed to `multi_gpu_model`),\n    rather than the model returned by `multi_gpu_model`.\n    \"\"\"\n    if K.backend() != 'tensorflow':\n        raise ValueError('`multi_gpu_model` is only available '\n                         'with the TensorFlow backend.')\n\n    available_devices = _get_available_devices()\n    available_devices = [_normalize_device_name(name)\n                         for name in available_devices]\n    if not gpus:\n        # Using all visible GPUs when not specifying `gpus`\n        # e.g. CUDA_VISIBLE_DEVICES=0,2 python keras_mgpu.py\n        gpus = len([x for x in available_devices if 'gpu' in x])\n\n    if isinstance(gpus, (list, tuple)):\n        if len(gpus) <= 1:\n            raise ValueError('For multi-gpu usage to be effective, '\n                             'call `multi_gpu_model` with `len(gpus) >= 2`. '\n                             'Received: `gpus=%s`' % gpus)\n        num_gpus = len(gpus)\n        target_gpu_ids = gpus\n    else:\n        if gpus <= 1:\n            raise ValueError('For multi-gpu usage to be effective, '\n                             'call `multi_gpu_model` with `gpus >= 2`. '\n                             'Received: `gpus=%d`' % gpus)\n        num_gpus = gpus\n        target_gpu_ids = range(num_gpus)\n\n    import tensorflow as tf\n\n    target_devices = ['/cpu:0'] + ['/gpu:%d' % i for i in target_gpu_ids]\n    for device in target_devices:\n        if device not in available_devices:\n            raise ValueError(\n                'To call `multi_gpu_model` with `gpus=%s`, '\n                'we expect the following devices to be available: %s. '\n                'However this machine only has: %s. '\n                'Try reducing `gpus`.' % (gpus,\n                                          target_devices,\n                                          available_devices))\n\n    def get_slice(data, i, parts):\n        shape = K.shape(data)\n        batch_size = shape[:1]\n        input_shape = shape[1:]\n        step = batch_size // parts\n        if i == parts - 1:\n            size = batch_size - step * i\n        else:\n            size = step\n        size = K.concatenate([size, input_shape], axis=0)\n        stride = K.concatenate([step, input_shape * 0], axis=0)\n        start = stride * i\n        return K.slice(data, start, size)\n\n    # Relocate the model definition under CPU device scope if needed\n    if cpu_relocation:\n        with tf.device('/cpu:0'):\n            model = clone_model(model)\n\n    all_outputs = []\n    for i in range(len(model.outputs)):\n        all_outputs.append([])\n\n    # Place a copy of the model on each GPU,\n    # each getting a slice of the inputs.\n    for i, gpu_id in enumerate(target_gpu_ids):\n        with tf.device('/gpu:%d' % gpu_id):\n            with tf.name_scope('replica_%d' % gpu_id):\n                inputs = []\n                # Retrieve a slice of the input.\n                for x in model.inputs:\n                    # In-place input splitting which is not only\n                    # 5% ~ 12% faster but also less GPU memory\n                    # duplication.\n                    with tf.device(x.device):\n                        input_shape = K.int_shape(x)[1:]\n                        slice_i = Lambda(get_slice,\n                                         output_shape=input_shape,\n                                         arguments={'i': i,\n                                                    'parts': num_gpus})(x)\n                        inputs.append(slice_i)\n\n                # Apply model on slice\n                # (creating a model replica on the target device).\n                outputs = model(inputs)\n                outputs = to_list(outputs)\n\n                # Save the outputs for merging back together later.\n                for o in range(len(outputs)):\n                    all_outputs[o].append(outputs[o])\n\n    # Deduplicate output names to handle Siamese networks.\n    occurrences = {}\n    for n in model.output_names:\n        if n not in occurrences:\n            occurrences[n] = 1\n        else:\n            occurrences[n] += 1\n    conflict_counter = {n: 0 for n, count in occurrences.items() if count > 1}\n    output_names = []\n    for n in model.output_names:\n        if n in conflict_counter:\n            conflict_counter[n] += 1\n            n += '_%d' % conflict_counter[n]\n        output_names.append(n)\n\n    # Merge outputs under expected scope.\n    with tf.device('/cpu:0' if cpu_merge else '/gpu:%d' % target_gpu_ids[0]):\n        merged = []\n        for name, outputs in zip(output_names, all_outputs):\n            merged.append(concatenate(outputs,\n                                      axis=0, name=name))\n        return Model(model.inputs, merged)\n", "path": "keras/utils/multi_gpu_utils.py"}], "after_files": [{"content": "\"\"\"Multi-GPU training utilities.\n\"\"\"\nfrom __future__ import absolute_import\nfrom __future__ import division\nfrom __future__ import print_function\n\nfrom ..layers.merge import concatenate\nfrom .. import backend as K\nfrom ..layers.core import Lambda\nfrom ..engine.training import Model\nfrom ..models import clone_model\nfrom ..utils.generic_utils import to_list\n\n\ndef _get_available_devices():\n    return [x.name for x in K.get_session().list_devices()]\n\n\ndef _normalize_device_name(name):\n    name = '/' + ':'.join(name.lower().replace('/', '').split(':')[-2:])\n    return name\n\n\ndef multi_gpu_model(model, gpus=None, cpu_merge=True, cpu_relocation=False):\n    \"\"\"Replicates a model on different GPUs.\n\n    Specifically, this function implements single-machine\n    multi-GPU data parallelism. It works in the following way:\n\n    - Divide the model's input(s) into multiple sub-batches.\n    - Apply a model copy on each sub-batch. Every model copy\n        is executed on a dedicated GPU.\n    - Concatenate the results (on CPU) into one big batch.\n\n    E.g. if your `batch_size` is 64 and you use `gpus=2`,\n    then we will divide the input into 2 sub-batches of 32 samples,\n    process each sub-batch on one GPU, then return the full\n    batch of 64 processed samples.\n\n    This induces quasi-linear speedup on up to 8 GPUs.\n\n    This function is only available with the TensorFlow backend\n    for the time being.\n\n    # Arguments\n        model: A Keras model instance. To avoid OOM errors,\n            this model could have been built on CPU, for instance\n            (see usage example below).\n        gpus: Integer >= 2 or list of integers, number of GPUs or\n            list of GPU IDs on which to create model replicas.\n        cpu_merge: A boolean value to identify whether to force\n            merging model weights under the scope of the CPU or not.\n        cpu_relocation: A boolean value to identify whether to\n            create the model's weights under the scope of the CPU.\n            If the model is not defined under any preceding device\n            scope, you can still rescue it by activating this option.\n\n    # Returns\n        A Keras `Model` instance which can be used just like the initial\n        `model` argument, but which distributes its workload on multiple GPUs.\n\n    # Examples\n\n    Example 1 - Training models with weights merge on CPU\n\n    ```python\n        import tensorflow as tf\n        from keras.applications import Xception\n        from keras.utils import multi_gpu_model\n        import numpy as np\n\n        num_samples = 1000\n        height = 224\n        width = 224\n        num_classes = 1000\n\n        # Instantiate the base model (or \"template\" model).\n        # We recommend doing this with under a CPU device scope,\n        # so that the model's weights are hosted on CPU memory.\n        # Otherwise they may end up hosted on a GPU, which would\n        # complicate weight sharing.\n        with tf.device('/cpu:0'):\n            model = Xception(weights=None,\n                             input_shape=(height, width, 3),\n                             classes=num_classes)\n\n        # Replicates the model on 8 GPUs.\n        # This assumes that your machine has 8 available GPUs.\n        parallel_model = multi_gpu_model(model, gpus=8)\n        parallel_model.compile(loss='categorical_crossentropy',\n                               optimizer='rmsprop')\n\n        # Generate dummy data.\n        x = np.random.random((num_samples, height, width, 3))\n        y = np.random.random((num_samples, num_classes))\n\n        # This `fit` call will be distributed on 8 GPUs.\n        # Since the batch size is 256, each GPU will process 32 samples.\n        parallel_model.fit(x, y, epochs=20, batch_size=256)\n\n        # Save model via the template model (which shares the same weights):\n        model.save('my_model.h5')\n    ```\n\n    Example 2 - Training models with weights merge on CPU using cpu_relocation\n\n    ```python\n         ..\n         # Not needed to change the device scope for model definition:\n         model = Xception(weights=None, ..)\n\n         try:\n             parallel_model = multi_gpu_model(model, cpu_relocation=True)\n             print(\"Training using multiple GPUs..\")\n         except ValueError:\n             parallel_model = model\n             print(\"Training using single GPU or CPU..\")\n         parallel_model.compile(..)\n         ..\n    ```\n\n    Example 3 - Training models with weights merge on GPU (recommended for NV-link)\n\n    ```python\n         ..\n         # Not needed to change the device scope for model definition:\n         model = Xception(weights=None, ..)\n\n         try:\n             parallel_model = multi_gpu_model(model, cpu_merge=False)\n             print(\"Training using multiple GPUs..\")\n         except:\n             parallel_model = model\n             print(\"Training using single GPU or CPU..\")\n\n         parallel_model.compile(..)\n         ..\n    ```\n\n    # On model saving\n\n    To save the multi-gpu model, use `.save(fname)` or `.save_weights(fname)`\n    with the template model (the argument you passed to `multi_gpu_model`),\n    rather than the model returned by `multi_gpu_model`.\n    \"\"\"\n    if K.backend() != 'tensorflow':\n        raise ValueError('`multi_gpu_model` is only available '\n                         'with the TensorFlow backend.')\n\n    available_devices = _get_available_devices()\n    available_devices = [_normalize_device_name(name)\n                         for name in available_devices]\n    if not gpus:\n        # Using all visible GPUs when not specifying `gpus`\n        # e.g. CUDA_VISIBLE_DEVICES=0,2 python keras_mgpu.py\n        gpus = len([x for x in available_devices if 'gpu' in x])\n\n    if isinstance(gpus, (list, tuple)):\n        if len(gpus) <= 1:\n            raise ValueError('For multi-gpu usage to be effective, '\n                             'call `multi_gpu_model` with `len(gpus) >= 2`. '\n                             'Received: `gpus=%s`' % gpus)\n        num_gpus = len(gpus)\n        target_gpu_ids = gpus\n    else:\n        if gpus <= 1:\n            raise ValueError('For multi-gpu usage to be effective, '\n                             'call `multi_gpu_model` with `gpus >= 2`. '\n                             'Received: `gpus=%d`' % gpus)\n        num_gpus = gpus\n        target_gpu_ids = range(num_gpus)\n\n    import tensorflow as tf\n\n    target_devices = ['/cpu:0'] + ['/gpu:%d' % i for i in target_gpu_ids]\n    for device in target_devices:\n        if device not in available_devices:\n            raise ValueError(\n                'To call `multi_gpu_model` with `gpus=%s`, '\n                'we expect the following devices to be available: %s. '\n                'However this machine only has: %s. '\n                'Try reducing `gpus`.' % (gpus,\n                                          target_devices,\n                                          available_devices))\n\n    def get_slice(data, i, parts):\n        shape = K.shape(data)\n        batch_size = shape[:1]\n        input_shape = shape[1:]\n        step = batch_size // parts\n        if i == parts - 1:\n            size = batch_size - step * i\n        else:\n            size = step\n        size = K.concatenate([size, input_shape], axis=0)\n        stride = K.concatenate([step, input_shape * 0], axis=0)\n        start = stride * i\n        return K.slice(data, start, size)\n\n    # Relocate the model definition under CPU device scope if needed\n    if cpu_relocation:\n        with tf.device('/cpu:0'):\n            model = clone_model(model)\n\n    all_outputs = []\n    for i in range(len(model.outputs)):\n        all_outputs.append([])\n\n    # Place a copy of the model on each GPU,\n    # each getting a slice of the inputs.\n    for i, gpu_id in enumerate(target_gpu_ids):\n        with tf.device('/gpu:%d' % gpu_id):\n            with tf.name_scope('replica_%d' % gpu_id):\n                inputs = []\n                # Retrieve a slice of the input.\n                for x in model.inputs:\n                    # In-place input splitting which is not only\n                    # 5% ~ 12% faster but also less GPU memory\n                    # duplication.\n                    with tf.device(x.device):\n                        input_shape = K.int_shape(x)[1:]\n                        slice_i = Lambda(get_slice,\n                                         output_shape=input_shape,\n                                         arguments={'i': i,\n                                                    'parts': num_gpus})(x)\n                        inputs.append(slice_i)\n\n                # Apply model on slice\n                # (creating a model replica on the target device).\n                outputs = model(inputs)\n                outputs = to_list(outputs)\n\n                # Save the outputs for merging back together later.\n                for o in range(len(outputs)):\n                    all_outputs[o].append(outputs[o])\n\n    # Deduplicate output names to handle Siamese networks.\n    occurrences = {}\n    for n in model.output_names:\n        if n not in occurrences:\n            occurrences[n] = 1\n        else:\n            occurrences[n] += 1\n    conflict_counter = {n: 0 for n, count in occurrences.items() if count > 1}\n    output_names = []\n    for n in model.output_names:\n        if n in conflict_counter:\n            conflict_counter[n] += 1\n            n += '_%d' % conflict_counter[n]\n        output_names.append(n)\n\n    # Merge outputs under expected scope.\n    with tf.device('/cpu:0' if cpu_merge else '/gpu:%d' % target_gpu_ids[0]):\n        merged = []\n        for name, outputs in zip(output_names, all_outputs):\n            merged.append(concatenate(outputs,\n                                      axis=0, name=name))\n        return Model(model.inputs, merged)\n", "path": "keras/utils/multi_gpu_utils.py"}]}