title: any doable approach to use multiple GPUs, multiple process with tensorflow?
body: <p>I am using docker container to run my experiment. I have multiple GPUs available and I want to use all of them for my experiment. I mean I want to utilize all GPUs for one program. To do so, I used <code>tf.distribute.MirroredStrategy</code> that suggested on tensorflow site, but it is not working. here is the <a href="https://gist.github.com/adamFlyn/5c2e2797f78acfc0136eeb283bb2ec77" rel="nofollow noreferrer">full error messages on gist</a>.</p>
  <p>here is available GPUs info:</p>
  <pre><code>+-----------------------------------------------------------------------------+
  | NVIDIA-SMI 450.51.06    Driver Version: 450.51.06    CUDA Version: 11.1     |
  |-------------------------------+----------------------+----------------------+
  | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
  | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
  |                               |                      |               MIG M. |
  |===============================+======================+======================|
  |   0  Tesla T4            Off  | 00000000:6A:00.0 Off |                    0 |
  | N/A   31C    P8    15W /  70W |      0MiB / 15109MiB |      0%      Default |
  |                               |                      |                  N/A |
  +-------------------------------+----------------------+----------------------+
  |   1  Tesla T4            Off  | 00000000:6B:00.0 Off |                    0 |
  | N/A   31C    P8    15W /  70W |      0MiB / 15109MiB |      0%      Default |
  |                               |                      |                  N/A |
  +-------------------------------+----------------------+----------------------+
  |   2  Tesla T4            Off  | 00000000:6C:00.0 Off |                    0 |
  | N/A   34C    P8    15W /  70W |      0MiB / 15109MiB |      0%      Default |
  |                               |                      |                  N/A |
  +-------------------------------+----------------------+----------------------+
  |   3  Tesla T4            Off  | 00000000:6D:00.0 Off |                    0 |
  | N/A   34C    P8    15W /  70W |      0MiB / 15109MiB |      0%      Default |
  |                               |                      |                  N/A |
  +-------------------------------+----------------------+----------------------+
                                                                                 
  +-----------------------------------------------------------------------------+
  | Processes:                                                                  |
  |  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
  |        ID   ID                                                   Usage      |
  |=============================================================================|
  |  No running processes found                                                 |
  +-----------------------------------------------------------------------------+
  </code></pre>
  <p><strong>my current attempt</strong></p>
  <p>here is my attempt using <code>tf.distribute.MirroredStrategy</code>:</p>
  <pre><code>device_type = &quot;GPU&quot;
  devices = tf.config.experimental.list_physical_devices(device_type)
  devices_names = [d.name.split(&quot;e:&quot;)[1] for d in devices]
  strategy = tf.distribute.MirroredStrategy(devices=devices_names[:3])
  
  with strategy.scope():
      model.compile(optimizer=opt, loss=&quot;categorical_crossentropy&quot;, metrics=[&quot;accuracy&quot;])
  </code></pre>
  <p>The above attempt is not working and gave the error that listed on above gist. I don't find another way of using multiple GPUs for a single experiment.</p>
  <p>does anyone any workable approach to make this happens? any thoughts?</p>