Cannot place 2 pre-trained models on separate GPUs
* i have NOT reproduced with tf nightly Hi, @boocheck 

Apologize for the delay and if you're using single machine with multiple GPU's (in your case 2 GPU's) then you'll have to go with [tf.distribute.MirroredStrategy](https://www.tensorflow.org/api_docs/python/tf/distribute/MirroredStrategy) and you can create `MirroredStrategy` as below which will create a `MirroredStrategy` instance, which will use all the GPUs that are visible to TensorFlow :

```
mirrored_strategy = tf.distribute.MirroredStrategy()

## If you wish to use only some of the GPUs on your machine, you can do so like this:

mirrored_strategy = tf.distribute.MirroredStrategy(devices=["/gpu:0", "/gpu:1"])
```

You'll have to compile and build your model within `MirroredStrategy` scope so do something like below :

```
with mirrored_strategy.scope():
        model = tf.keras.Model(inputs=inputs, outputs=x)
        model.compile(...)
        model.fit(...)
```

Please refer our official documentation of [Distributed training with TensorFlow](https://www.tensorflow.org/guide/distributed_training) and please refer these references [Ref-1](https://towardsdatascience.com/multi-gpus-and-custom-training-loops-in-tensorflow-2-15b4b86b53bd), [Ref-2](https://www.youtube.com/watch?v=S1tN9a4Proc), I hope it will help you to resolve your issue

If issue still persists please let us know ? Thank you! I am sorry, but this is not solving my problem. I am aware of existance of MirroredStrategy, but too my knowledge, it tries to place whole model on specified GPUs. I cannot use it, because after CNN part (here: VGG16) I need to use heavy memory operation, which cannot fit any GPU memory. 

My problem is solvable by architecture like this:
![image](https://user-images.githubusercontent.com/1492258/222903864-74a84fd8-a08b-4534-ab7c-458ffcf89b99.png)

So what would preferably do is:
![image](https://user-images.githubusercontent.com/1492258/222904415-515ea792-658d-4708-b128-d4472c6d7c8e.png)

What is puzzling, and makes me believe this may be not intended behavior, when put some plain matmul in place of VGG16 (look: model2 in my original post) it is all working like intended
 Hi, @SuryanarayanaY 

Could you please look into this issue? Thank you! Hi @boocheck ,

Right now Tensorflow supports Data Parallelism i.e Data will be splitted across the accelerators but the same model will be copied across all these devices(each device have same copy of model but different slices of data).

Your ask is for Model parallelism which is not yet supported by Tensorflow. i.e you can't train different models/layers on different machines.This is still under development stage only.

Please refer this official tensorflow video [tutorial](https://www.youtube.com/watch?v=cdNLtMszdLs) for more detail on this concept.

Right now you can use data parallelism using Mirrored strategy.

Thankyou! Hello @SuryanarayanaY 

this seems not to be true. Have a look at this https://www.tensorflow.org/guide/gpu#manual_device_placement Hi @boocheck ,

You can choose the device manually using `tf.device` for Ops.AFAIK It does execution of code line by line but not parallel. 

But in your case on same model you want to use different devices for different blocks of model which is nothing but Model parallelism which is not yet supported in Tensorflow.For model training on GPUs right now Tensorflow supports data parallelism concept using Distribution training.Since you are trying to use same model but on different devices which is not supported yet on tensorflow.Distribution training is not straight forward, it needs copying the data/model across replicas and it uses all reduce algorithms for cross device communication which is implemented using distribution strategies and only for Data Parallelism as of now.

Please refer the attached [source](https://www.tensorflow.org/api_docs/python/tf/distribute#:~:text=Glossary,async%20between%20groups.) that states limitations and status regarding model parallelism.

Also Please refer to distribution training [documentation](https://www.tensorflow.org/guide/distributed_training) for more details. This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you. This issue was closed because it has been inactive for 7 days since being marked as stale. Please reopen if you'd like to work on this further. Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/59865">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/59865">No</a>

<details><summary>Click to expand!</summary> 
 
 ### Issue Type

Bug

### Have you reproduced the bug with TF nightly?

Yes

### Source

source

### Tensorflow Version

2.4.1

### Custom Code

Yes

### OS Platform and Distribution

centos 7

### Mobile device

_No response_

### Python version

3.8

### Bazel version

_No response_

### GCC/Compiler version

_No response_

### CUDA/cuDNN version

_No response_

### GPU model and memory

2x tesla P40

### Current Behaviour?

```shell
I expected both vgg16 models to run in parallel on 2 GPUs at near 100% capacity, but nvidia-smi shows  one being completely idle. When I used some matmul in place of pretrained vgg16 (function 'model2') I observe both GPUs utilisation.
```


### Standalone code to reproduce the issue

```shell
import numpy as np
import tensorflow as tf
from tensorflow.python.keras import Input, Model
from tensorflow.python.keras.applications.vgg16 import VGG16
from tensorflow.python.keras.layers import Dense, GlobalAveragePooling2D

tf.debugging.set_log_device_placement(True)


def model(images):
    with tf.device("/cpu:0"):
        first, second = tf.split(images, 2)
    with tf.device("/gpu:0"):
        vgg1 = VGG16(False)
        vgg1._name = "vgg1"  # without this i get error about layers not having unique names         
        vgg1.trainable = False
        vgged1 = vgg1(first)
    with tf.device("/gpu:1"):
        vgg2 = VGG16(False)  # it feels like this does not create second model but somehow reuses vgg1 on GPU:0
        vgg2._name = "vgg2"  
        vgg2.trainable = False
        vgged2 = vgg2(second)
    with tf.device("/cpu:0"):
        gap = GlobalAveragePooling2D()
        concated = tf.concat([gap(vgged1), gap(vgged2)], 1)
        d10 = Dense(10)(concated)
        return d10

def model2(images):
    flat_size=224*224*3

    with tf.device("/cpu:0"):
        first, second = tf.split(images, 2)
    with tf.device("/gpu:0"):
        flat1=tf.reshape(first, [-1, flat_size])
        w1 = tf.Variable(tf.random.uniform([flat_size, 10000]))
        res1= tf.matmul(flat1, w1)
    with tf.device("/gpu:1"):
        flat2 = tf.reshape(second, [-1, flat_size])
        w2 = tf.Variable(tf.random.uniform([flat_size, 10000]))
        res2 = tf.matmul(flat2, w2)
    with tf.device("/cpu:0"):
        concated = tf.concat([res1, res2], 1)
        w3 = tf.Variable(tf.random.uniform([20000, 10]))
        return tf.matmul(concated,w3)


inp = Input((224, 224, 3))
out = model(inp)  # if I change this to "out = model2(inp)" both GPUs work

m = Model(inputs=inp, outputs=out)

x = np.random.random([32, 224, 224, 3])

m.summary()

for
m(x)
```


### Relevant log output

```shell
full output available at: https://snippet.host/hyrgbj

2023-02-28 12:41:01.741524: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2023-02-28 12:41:03.001461: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2023-02-28 12:41:03.002735: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2023-02-28 12:41:03.010097: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties:
pciBusID: 0000:08:00.0 name: Tesla P40 computeCapability: 6.1
coreClock: 1.531GHz coreCount: 30 deviceMemorySize: 22.38GiB deviceMemoryBandwidth: 323.21GiB/s
2023-02-28 12:41:03.010449: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 1 with properties:
pciBusID: 0000:84:00.0 name: Tesla P40 computeCapability: 6.1
coreClock: 1.531GHz coreCount: 30 deviceMemorySize: 22.38GiB deviceMemoryBandwidth: 323.21GiB/s
2023-02-28 12:41:03.010499: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2023-02-28 12:41:03.017621: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2023-02-28 12:41:03.017668: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2023-02-28 12:41:03.052369: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2023-02-28 12:41:03.053152: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2023-02-28 12:41:03.055905: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2023-02-28 12:41:03.057101: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2023-02-28 12:41:03.057265: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2023-02-28 12:41:03.063035: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0, 1
2023-02-28 12:41:03.063516: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-02-28 12:41:03.069400: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2023-02-28 12:41:03.329039: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties:
pciBusID: 0000:08:00.0 name: Tesla P40 computeCapability: 6.1
coreClock: 1.531GHz coreCount: 30 deviceMemorySize: 22.38GiB deviceMemoryBandwidth: 323.21GiB/s
2023-02-28 12:41:03.329290: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 1 with properties:
pciBusID: 0000:84:00.0 name: Tesla P40 computeCapability: 6.1
coreClock: 1.531GHz coreCount: 30 deviceMemorySize: 22.38GiB deviceMemoryBandwidth: 323.21GiB/s
2023-02-28 12:41:03.329353: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2023-02-28 12:41:03.329389: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2023-02-28 12:41:03.329408: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2023-02-28 12:41:03.329443: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2023-02-28 12:41:03.329473: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2023-02-28 12:41:03.329491: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2023-02-28 12:41:03.329517: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2023-02-28 12:41:03.329537: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2023-02-28 12:41:03.330198: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0, 1
2023-02-28 12:41:03.330243: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2023-02-28 12:41:04.083881: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2023-02-28 12:41:04.083958: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 1
2023-02-28 12:41:04.083972: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N N
2023-02-28 12:41:04.083981: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 1:   N N
2023-02-28 12:41:04.085081: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 21298 MB memory) -> physical GPU (device: 0, name: Tesla P40, pci bus id: 0000:08:00.0, compute capability: 6.1)
2023-02-28 12:41:04.085905: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 21298 MB memory) -> physical GPU (device: 1, name: Tesla P40, pci bus id: 0000:84:00.0, compute capability: 6.1)
2023-02-28 12:41:04.106894: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op RandomUniform in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.311401: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op Sub in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.311641: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op Mul in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.311778: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op Add in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.312103: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.312365: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.313784: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op Fill in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.313966: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.314080: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.321015: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op RandomUniform in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.321222: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op Sub in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.321295: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op Mul in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.321356: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op Add in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.321509: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.321620: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.321957: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op Fill in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.322085: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.322183: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.329542: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op RandomUniform in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.329740: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op Sub in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.329821: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op Mul in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.329882: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op Add in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.330035: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.330153: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.330489: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op Fill in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.330650: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.330757: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.336373: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op RandomUniform in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.336577: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op Sub in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.336651: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op Mul in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.336713: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op Add in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.336886: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.336996: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.337341: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op Fill in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.337460: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.337556: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.344730: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op RandomUniform in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.344917: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op Sub in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.344990: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op Mul in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.345050: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op Add in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.345212: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.345367: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.345719: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op Fill in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.345882: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.345985: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.351300: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op RandomUniform in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.351501: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op Sub in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.351576: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op Mul in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.351637: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op Add in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.351790: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.351907: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.352251: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op Fill in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.352372: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.352472: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.357894: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op RandomUniform in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.357971: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op Sub in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.358047: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op Mul in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.358113: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op Add in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.358229: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.358333: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.358666: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op Fill in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.358787: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.358885: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.366153: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op RandomUniform in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.366357: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op Sub in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.366447: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op Mul in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.366507: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op Add in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.366668: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.366779: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.367122: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op Fill in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.367288: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.367393: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.372946: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op RandomUniform in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.373388: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op Sub in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.373463: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op Mul in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.373542: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op Add in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.373696: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.373806: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.374161: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op Fill in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.374283: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.374399: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.379794: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op RandomUniform in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.379872: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op Sub in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.379950: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op Mul in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.380008: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op Add in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.380129: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.380234: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.380564: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op Fill in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.380684: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.380780: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.387982: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op RandomUniform in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.388062: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op Sub in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.388136: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op Mul in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.388195: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op Add in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.388309: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.388414: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.388756: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op Fill in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.388879: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.388981: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.394514: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op RandomUniform in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.394591: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op Sub in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.394657: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op Mul in device /job:localhost/replica:0/task:0/device:GPU:0
2023-02-28 12:41:04.394724: I tensorflow/core/common_runtime/eager/execute.cc:760] Executing op Add in device /job:localhost/replica:0/task:0/device:GPU:0
```
</details>