Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10175 MB memory) -> physical GPU (device: 0, name: Tesla T4, pci bus id: 0000:00:07.0, compute capability: 7.5)
Hello, @9527-csroad!
To use the Tesla T4 GPU in TensorFlow, please specify the `device:GPU device` type when creating a model or running a session with latest TF version as you are using an older version which is not actively supported. 
Thank you! Thanks for your reply, @sushreebarsa . Which version should I install? You can see that no matter which version I installed, tensorflow always take so much GPU memory.
```python
(noya-mat) root@iZbp1b192y9va5podaabh6Z:/noya/matting# python3 optimize.py
2023-11-06 15:31:45,263 - modelscope - INFO - PyTorch version 1.11.0+cu113 Found.
2023-11-06 15:31:45,265 - modelscope - INFO - TensorFlow version 2.6.0 Found.
2023-11-06 15:31:45,266 - modelscope - INFO - Loading ast index from /root/.cache/modelscope/ast_indexer
2023-11-06 15:31:45,303 - modelscope - INFO - Loading done! Current index file version is 1.9.4, with md5 6fd956e57d818f09de7f4c4f31828009 and a total number of 945 components indexed
2023-11-06 15:31:46,762 - modelscope - WARNING - Model revision not specified, use revision: v1.1.0
2023-11-06 15:31:47,362 - modelscope - INFO - initiate model from /root/.cache/modelscope/hub/damo/cv_tinynas_human-detection_damoyolo
2023-11-06 15:31:47,362 - modelscope - INFO - initiate model from location /root/.cache/modelscope/hub/damo/cv_tinynas_human-detection_damoyolo.
2023-11-06 15:31:47,364 - modelscope - INFO - initialize model from /root/.cache/modelscope/hub/damo/cv_tinynas_human-detection_damoyolo
2023-11-06 15:31:48,284 - modelscope - WARNING - No val key and type key found in preprocessor domain of configuration.json file.
2023-11-06 15:31:48,284 - modelscope - WARNING - Cannot find available config to build preprocessor at mode inference, current config: {'train': {'mosaic_mixup': {'mosaic_prob': 1.0, 'mosaic_scale': [0.1, 2.0], 'mosaic_size': [640, 640], 'mixup_prob': 0.15, 'mixup_scale': [0.5, 1.5], 'degrees': 10.0, 'translate': 0.2, 'shear': 2.0}, 'transform': {'image_mean': [0.0, 0.0, 0.0], 'image_std': [1.0, 1.0, 1.0], 'image_max_range': [640, 640], 'flip_prob': 0.5, 'autoaug_dict': {'box_prob': 0.3, 'num_subpolicies': 5, 'scale_splits': [2048, 10240, 51200], 'autoaug_params': [6, 9, 5, 3, 3, 4, 2, 4, 4, 4, 5, 2, 4, 1, 4, 2, 6, 4, 2, 2, 2, 6, 2, 2, 2, 0, 5, 1, 3, 0, 8, 5, 2, 8, 7, 5, 1, 3, 3, 3]}}}, 'evaluation': {'transform': {'image_mean': [0.0, 0.0, 0.0], 'image_std': [1.0, 1.0, 1.0], 'image_max_range': [640, 640], 'flip_prob': 0.0}}, 'model_dir': '/root/.cache/modelscope/hub/damo/cv_tinynas_human-detection_damoyolo'}. trying to build by task and model information.
2023-11-06 15:31:49,116 - modelscope - WARNING - Model revision not specified, use revision: v1.0.0
2023-11-06 15:31:50,585 - modelscope - INFO - initiate model from /root/.cache/modelscope/hub/damo/cv_unet_image-matting
2023-11-06 15:31:50,585 - modelscope - INFO - initiate model from location /root/.cache/modelscope/hub/damo/cv_unet_image-matting.
2023-11-06 15:31:50,587 - modelscope - WARNING - No preprocessor field found in cfg.
2023-11-06 15:31:50,587 - modelscope - WARNING - No val key and type key found in preprocessor domain of configuration.json file.
2023-11-06 15:31:50,587 - modelscope - WARNING - Cannot find available config to build preprocessor at mode inference, current config: {'model_dir': '/root/.cache/modelscope/hub/damo/cv_unet_image-matting'}. trying to build by task and model information.
2023-11-06 15:31:50,587 - modelscope - WARNING - Find task: portrait-matting, model type: None. Insufficient information to build preprocessor, skip building preprocessor
WARNING:tensorflow:From /root/anaconda3/envs/noya-mat/lib/python3.8/site-packages/modelscope/utils/device.py:60: is_gpu_available (from tensorflow.python.framework.test_util) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.config.list_physical_devices('GPU')` instead.
2023-11-06 15:31:50.589019: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-11-06 15:31:50.590884: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-11-06 15:31:50.598626: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-11-06 15:31:50.599297: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-11-06 15:31:55.306670: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-11-06 15:31:55.307388: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-11-06 15:31:55.307976: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-11-06 15:31:55.308559: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1510] Created device /device:GPU:0 with 11456 MB memory:  -> device: 0, name: Tesla T4, pci bus id: 0000:00:07.0, compute capability: 7.5
```

```bash
python3 optimize.py
2023-11-06 15:40:24,458 - modelscope - INFO - PyTorch version 1.11.0+cu113 Found.
2023-11-06 15:40:24,460 - modelscope - INFO - TensorFlow version 2.13.1 Found.
2023-11-06 15:40:24,461 - modelscope - INFO - Loading ast index from /root/.cache/modelscope/ast_indexer
2023-11-06 15:40:24,498 - modelscope - INFO - Loading done! Current index file version is 1.9.4, with md5 6fd956e57d818f09de7f4c4f31828009 and a total number of 945 components indexed
2023-11-06 15:40:25,950 - modelscope - WARNING - Model revision not specified, use revision: v1.1.0
2023-11-06 15:40:26,489 - modelscope - INFO - initiate model from /root/.cache/modelscope/hub/damo/cv_tinynas_human-detection_damoyolo
2023-11-06 15:40:26,489 - modelscope - INFO - initiate model from location /root/.cache/modelscope/hub/damo/cv_tinynas_human-detection_damoyolo.
2023-11-06 15:40:26,490 - modelscope - INFO - initialize model from /root/.cache/modelscope/hub/damo/cv_tinynas_human-detection_damoyolo
2023-11-06 15:40:27,396 - modelscope - WARNING - No val key and type key found in preprocessor domain of configuration.json file.
2023-11-06 15:40:27,397 - modelscope - WARNING - Cannot find available config to build preprocessor at mode inference, current config: {'train': {'mosaic_mixup': {'mosaic_prob': 1.0, 'mosaic_scale': [0.1, 2.0], 'mosaic_size': [640, 640], 'mixup_prob': 0.15, 'mixup_scale': [0.5, 1.5], 'degrees': 10.0, 'translate': 0.2, 'shear': 2.0}, 'transform': {'image_mean': [0.0, 0.0, 0.0], 'image_std': [1.0, 1.0, 1.0], 'image_max_range': [640, 640], 'flip_prob': 0.5, 'autoaug_dict': {'box_prob': 0.3, 'num_subpolicies': 5, 'scale_splits': [2048, 10240, 51200], 'autoaug_params': [6, 9, 5, 3, 3, 4, 2, 4, 4, 4, 5, 2, 4, 1, 4, 2, 6, 4, 2, 2, 2, 6, 2, 2, 2, 0, 5, 1, 3, 0, 8, 5, 2, 8, 7, 5, 1, 3, 3, 3]}}}, 'evaluation': {'transform': {'image_mean': [0.0, 0.0, 0.0], 'image_std': [1.0, 1.0, 1.0], 'image_max_range': [640, 640], 'flip_prob': 0.0}}, 'model_dir': '/root/.cache/modelscope/hub/damo/cv_tinynas_human-detection_damoyolo'}. trying to build by task and model information.
2023-11-06 15:40:28,238 - modelscope - WARNING - Model revision not specified, use revision: v1.0.0
2023-11-06 15:40:28.780384: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-11-06 15:40:29.588528: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2023-11-06 15:40:30,385 - modelscope - INFO - initiate model from /root/.cache/modelscope/hub/damo/cv_unet_image-matting
2023-11-06 15:40:30,385 - modelscope - INFO - initiate model from location /root/.cache/modelscope/hub/damo/cv_unet_image-matting.
2023-11-06 15:40:30,387 - modelscope - WARNING - No preprocessor field found in cfg.
2023-11-06 15:40:30,387 - modelscope - WARNING - No val key and type key found in preprocessor domain of configuration.json file.
2023-11-06 15:40:30,387 - modelscope - WARNING - Cannot find available config to build preprocessor at mode inference, current config: {'model_dir': '/root/.cache/modelscope/hub/damo/cv_unet_image-matting'}. trying to build by task and model information.
2023-11-06 15:40:30,387 - modelscope - WARNING - Find task: portrait-matting, model type: None. Insufficient information to build preprocessor, skip building preprocessor
WARNING:tensorflow:From /root/anaconda3/envs/noya-mat/lib/python3.8/site-packages/modelscope/utils/device.py:60: is_gpu_available (from tensorflow.python.framework.test_util) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.config.list_physical_devices('GPU')` instead.
2023-11-06 15:40:30.390254: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-11-06 15:40:30.397435: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-11-06 15:40:30.398099: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-11-06 15:40:35.308779: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-11-06 15:40:35.309475: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-11-06 15:40:35.310045: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-11-06 15:40:35.310599: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Created device /device:GPU:0 with 11426 MB memory:  -> device: 0, name: Tesla T4, pci bus id: 0000:00:07.0, compute capability: 7.5
2023-11-06 15:40:36.160844: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:375] MLIR V1 optimization pass is not enabled
2023-11-06 15:40:38.756156: E tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:425] Loaded runtime CuDNN library: 8.1.1 but source was compiled with: 8.6.0.  CuDNN library needs to have matching major version and equal or higher minor version. If using a binary install, upgrade your CuDNN library.  If building from sources, make sure the library loaded at runtime is compatible with the version specified during compile configuration.
2023-11-06 15:40:38.756592: W tensorflow/core/framework/op_kernel.cc:1828] OP_REQUIRES failed at conv_ops_impl.h:770 : UNIMPLEMENTED: DNN library is not found.
2023-11-06 15:40:38.756662: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 17908648775275799695
Traceback (most recent call last):
  File "/root/anaconda3/envs/noya-mat/lib/python3.8/site-packages/tensorflow/python/client/session.py", line 1379, in _do_call
    return fn(*args)
  File "/root/anaconda3/envs/noya-mat/lib/python3.8/site-packages/tensorflow/python/client/session.py", line 1362, in _run_fn
    return self._call_tf_sessionrun(options, feed_dict, fetch_list,
  File "/root/anaconda3/envs/noya-mat/lib/python3.8/site-packages/tensorflow/python/client/session.py", line 1455, in _call_tf_sessionrun
    return tf_session.TF_SessionRun_wrapper(self._session, options, feed_dict,
tensorflow.python.framework.errors_impl.UnimplementedError: 2 root error(s) found.
  (0) UNIMPLEMENTED: DNN library is not found.
         [[{{node coarse/Conv/Conv2D}}]]
         [[output_png/_3]]
  (1) UNIMPLEMENTED: DNN library is not found.
         [[{{node coarse/Conv/Conv2D}}]]
0 successful operations.
0 derived errors ignored.
```

```bash
(noya-mat) root@iZbp1b192y9va5podaabh6Z:/noya/matting# python3 optimize.py
2023-11-06 15:44:22,216 - modelscope - INFO - PyTorch version 1.11.0+cu113 Found.
2023-11-06 15:44:22,219 - modelscope - INFO - TensorFlow version 2.11.0 Found.
2023-11-06 15:44:22,219 - modelscope - INFO - Loading ast index from /root/.cache/modelscope/ast_indexer
2023-11-06 15:44:22,260 - modelscope - INFO - Loading done! Current index file version is 1.9.4, with md5 6fd956e57d818f09de7f4c4f31828009 and a total number of 945 components indexed
2023-11-06 15:44:23,739 - modelscope - WARNING - Model revision not specified, use revision: v1.1.0
2023-11-06 15:44:24,284 - modelscope - INFO - initiate model from /root/.cache/modelscope/hub/damo/cv_tinynas_human-detection_damoyolo
2023-11-06 15:44:24,284 - modelscope - INFO - initiate model from location /root/.cache/modelscope/hub/damo/cv_tinynas_human-detection_damoyolo.
2023-11-06 15:44:24,285 - modelscope - INFO - initialize model from /root/.cache/modelscope/hub/damo/cv_tinynas_human-detection_damoyolo
2023-11-06 15:44:25,196 - modelscope - WARNING - No val key and type key found in preprocessor domain of configuration.json file.
2023-11-06 15:44:25,196 - modelscope - WARNING - Cannot find available config to build preprocessor at mode inference, current config: {'train': {'mosaic_mixup': {'mosaic_prob': 1.0, 'mosaic_scale': [0.1, 2.0], 'mosaic_size': [640, 640], 'mixup_prob': 0.15, 'mixup_scale': [0.5, 1.5], 'degrees': 10.0, 'translate': 0.2, 'shear': 2.0}, 'transform': {'image_mean': [0.0, 0.0, 0.0], 'image_std': [1.0, 1.0, 1.0], 'image_max_range': [640, 640], 'flip_prob': 0.5, 'autoaug_dict': {'box_prob': 0.3, 'num_subpolicies': 5, 'scale_splits': [2048, 10240, 51200], 'autoaug_params': [6, 9, 5, 3, 3, 4, 2, 4, 4, 4, 5, 2, 4, 1, 4, 2, 6, 4, 2, 2, 2, 6, 2, 2, 2, 0, 5, 1, 3, 0, 8, 5, 2, 8, 7, 5, 1, 3, 3, 3]}}}, 'evaluation': {'transform': {'image_mean': [0.0, 0.0, 0.0], 'image_std': [1.0, 1.0, 1.0], 'image_max_range': [640, 640], 'flip_prob': 0.0}}, 'model_dir': '/root/.cache/modelscope/hub/damo/cv_tinynas_human-detection_damoyolo'}. trying to build by task and model information.
2023-11-06 15:44:25,910 - modelscope - WARNING - Model revision not specified, use revision: v1.0.0
2023-11-06 15:44:26.202622: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-11-06 15:44:27.097726: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /root/anaconda3/envs/noya-mat/lib/python3.8/site-packages/cv2/../../lib64:/usr/local/cuda-11.2/lib64:
2023-11-06 15:44:27.097846: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /root/anaconda3/envs/noya-mat/lib/python3.8/site-packages/cv2/../../lib64:/usr/local/cuda-11.2/lib64:
2023-11-06 15:44:27.097860: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
2023-11-06 15:44:27,877 - modelscope - INFO - initiate model from /root/.cache/modelscope/hub/damo/cv_unet_image-matting
2023-11-06 15:44:27,877 - modelscope - INFO - initiate model from location /root/.cache/modelscope/hub/damo/cv_unet_image-matting.
2023-11-06 15:44:27,881 - modelscope - WARNING - No preprocessor field found in cfg.
2023-11-06 15:44:27,881 - modelscope - WARNING - No val key and type key found in preprocessor domain of configuration.json file.
2023-11-06 15:44:27,881 - modelscope - WARNING - Cannot find available config to build preprocessor at mode inference, current config: {'model_dir': '/root/.cache/modelscope/hub/damo/cv_unet_image-matting'}. trying to build by task and model information.
2023-11-06 15:44:27,881 - modelscope - WARNING - Find task: portrait-matting, model type: None. Insufficient information to build preprocessor, skip building preprocessor
WARNING:tensorflow:From /root/anaconda3/envs/noya-mat/lib/python3.8/site-packages/modelscope/utils/device.py:60: is_gpu_available (from tensorflow.python.framework.test_util) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.config.list_physical_devices('GPU')` instead.
2023-11-06 15:44:27.882293: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-11-06 15:44:27.883792: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-11-06 15:44:27.890937: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-11-06 15:44:27.891624: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-11-06 15:44:32.758004: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-11-06 15:44:32.758740: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-11-06 15:44:32.759339: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-11-06 15:44:32.759909: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1613] Created device /device:GPU:0 with 11432 MB memory:  -> device: 0, name: Tesla T4, pci bus id: 0000:00:07.0, compute capability: 7.5
```

 Hi @9527-csroad ,

Could you please confirm the exact problem? As per my understanding here you have a 15 GB GPU but while code execution you find that only 10.175GB is visible. Is that what you want report?

I can also see Pytorch version also indeed installed here.It seems your environment has too many dependencies that is taking more memory resources ?
 Hello @SuryanarayanaY ,
My question is why it allocated so much GPU memory?  You can find it take around 10GB while it not need so much memory（I'm not train a model but deploy a model).  I deployed some models and they usually use 3GB - 6GB cuda memory,  but now, the procedure take all of the rest of the cuda memory. > ```python
> ```python
> 2023-11-06 15:31:55.308559: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1510] Created device /device:GPU:0 with 11456 MB memory:  -> device: 0, name: Tesla T4, pci bus id: 0000:00:07.0, compute capability: 7.5
> ```
> 
> 
>     
>       
>     
> 
>       
>     
> 
>     
>   
> ```

Hi @9527-csroad , If you are referring to this log then its just printing the GPU device and its available memory for process. It's not locking the memory though. If so, I'm sorry to waste your time. I have end this project long time, and I will pay attention to this next time. Thanks for your patience reply.
Best regard. Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/62170">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/62170">No</a>

### Issue type

Bug

### Have you reproduced the bug with TensorFlow Nightly?

Yes

### Source

source

### TensorFlow version

tf-gpu 2.4.0

### Custom code

Yes

### OS platform and distribution

Ubuntu 18.04

### Mobile device

Ubuntu 18.04

### Python version

3.8

### Bazel version

_No response_

### GCC/compiler version

_No response_

### CUDA/cuDNN version

11.0 8.0

### GPU model and memory

T4 15G

### Current behavior?

I deploment an u-net like model and use it to matting, why it take so much memory

### Standalone code to reproduce the issue

```shell
As the above description
```


### Relevant log output

```shell
python'''
2023-10-19 17:12:05,653 - modelscope - INFO - PyTorch version 2.0.1 Found.
2023-10-19 17:12:05,654 - modelscope - INFO - TensorFlow version 2.4.0 Found.
2023-10-19 17:12:05,654 - modelscope - INFO - Loading ast index from /root/.cache/modelscope/ast_indexer
2023-10-19 17:12:05,764 - modelscope - INFO - Updating the files for the changes of local files, first time updating will take longer time! Please wait till updating done!
2023-10-19 17:12:05,765 - modelscope - INFO - AST-Scanning the path "/root/miniconda/envs/modelscope/lib/python3.8/site-packages/modelscope" with the following sub folders ['models', 'metrics', 'pipelines', 'preprocessors', 'trainers', 'msdatasets', 'exporters']
2023-10-19 17:12:05,769 - modelscope - INFO - Scanning done! A number of 2 components indexed or updated! Time consumed 0.0038497447967529297s
2023-10-19 17:12:05,809 - modelscope - INFO - Loading done! Current index file version is 1.9.2, with md5 e5b1c859add8b0dd7419f7819a63c9a0 and a total number of 941 components indexed
2023-10-19 17:12:06,946 - modelscope - INFO - Model revision not specified, use revision: v1.0.0
2023-10-19 17:12:07.433075: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2023-10-19 17:12:08,529 - modelscope - INFO - initiate model from /root/.cache/modelscope/hub/damo/cv_unet_universal-matting
2023-10-19 17:12:08,529 - modelscope - INFO - initiate model from location /root/.cache/modelscope/hub/damo/cv_unet_universal-matting.
2023-10-19 17:12:08,532 - modelscope - WARNING - No preprocessor field found in cfg.
2023-10-19 17:12:08,532 - modelscope - WARNING - No val key and type key found in preprocessor domain of configuration.json file.
2023-10-19 17:12:08,532 - modelscope - WARNING - Cannot find available config to build preprocessor at mode inference, current config: {'model_dir': '/root/.cache/modelscope/hub/damo/cv_unet_universal-matting'}. trying to build by task and model information.
2023-10-19 17:12:08,532 - modelscope - WARNING - Find task: universal-matting, model type: None. Insufficient information to build preprocessor, skip building preprocessor
WARNING:tensorflow:From /root/miniconda/envs/modelscope/lib/python3.8/site-packages/modelscope/utils/device.py:60: is_gpu_available (from tensorflow.python.framework.test_util) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.config.list_physical_devices('GPU')` instead.
2023-10-19 17:12:08.533865: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX512F
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-10-19 17:12:08.534257: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2023-10-19 17:12:08.534330: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2023-10-19 17:12:08.545027: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-10-19 17:12:08.545636: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:00:07.0 name: Tesla T4 computeCapability: 7.5
coreClock: 1.59GHz coreCount: 40 deviceMemorySize: 14.75GiB deviceMemoryBandwidth: 298.08GiB/s
2023-10-19 17:12:08.545678: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2023-10-19 17:12:08.545736: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2023-10-19 17:12:08.545821: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2023-10-19 17:12:08.545859: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2023-10-19 17:12:08.545897: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2023-10-19 17:12:08.549506: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2023-10-19 17:12:08.549600: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2023-10-19 17:12:08.549652: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2023-10-19 17:12:08.549761: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-10-19 17:12:08.550454: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-10-19 17:12:08.551001: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2023-10-19 17:12:10.620467: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2023-10-19 17:12:10.620510: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2023-10-19 17:12:10.620518: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2023-10-19 17:12:10.620749: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-10-19 17:12:10.621411: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-10-19 17:12:10.622007: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-10-19 17:12:10.622561: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/device:GPU:0 with 10175 MB memory) -> physical GPU (device: 0, name: Tesla T4, pci bus id: 0000:00:07.0, compute capability: 7.5)
2023-10-19 17:12:10.623148: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2023-10-19 17:12:10.623261: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-10-19 17:12:10.623834: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:00:07.0 name: Tesla T4 computeCapability: 7.5
coreClock: 1.59GHz coreCount: 40 deviceMemorySize: 14.75GiB deviceMemoryBandwidth: 298.08GiB/s
2023-10-19 17:12:10.623921: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2023-10-19 17:12:10.623944: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2023-10-19 17:12:10.623960: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2023-10-19 17:12:10.624018: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2023-10-19 17:12:10.624068: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2023-10-19 17:12:10.624169: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2023-10-19 17:12:10.624189: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2023-10-19 17:12:10.624208: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2023-10-19 17:12:10.624284: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-10-19 17:12:10.624888: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-10-19 17:12:10.625400: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2023-10-19 17:12:10.625769: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2023-10-19 17:12:10.625880: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-10-19 17:12:10.626610: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:00:07.0 name: Tesla T4 computeCapability: 7.5
coreClock: 1.59GHz coreCount: 40 deviceMemorySize: 14.75GiB deviceMemoryBandwidth: 298.08GiB/s
2023-10-19 17:12:10.626686: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2023-10-19 17:12:10.626737: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2023-10-19 17:12:10.626769: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2023-10-19 17:12:10.626816: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2023-10-19 17:12:10.626863: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2023-10-19 17:12:10.626962: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2023-10-19 17:12:10.626986: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2023-10-19 17:12:10.627003: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2023-10-19 17:12:10.627074: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-10-19 17:12:10.627931: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-10-19 17:12:10.628514: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2023-10-19 17:12:10.628550: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2023-10-19 17:12:10.628561: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2023-10-19 17:12:10.628569: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2023-10-19 17:12:10.628681: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-10-19 17:12:10.629268: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-10-19 17:12:10.629781: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10175 MB memory) -> physical GPU (device: 0, name: Tesla T4, pci bus id: 0000:00:07.0, compute capability: 7.5)
2023-10-19 17:12:10.630834: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2023-10-19 17:12:10.630927: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-10-19 17:12:10.631480: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:00:07.0 name: Tesla T4 computeCapability: 7.5
coreClock: 1.59GHz coreCount: 40 deviceMemorySize: 14.75GiB deviceMemoryBandwidth: 298.08GiB/s
2023-10-19 17:12:10.631536: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2023-10-19 17:12:10.631553: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2023-10-19 17:12:10.631570: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2023-10-19 17:12:10.631586: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2023-10-19 17:12:10.631604: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2023-10-19 17:12:10.631645: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2023-10-19 17:12:10.631709: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2023-10-19 17:12:10.631755: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2023-10-19 17:12:10.631851: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-10-19 17:12:10.632539: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-10-19 17:12:10.633141: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2023-10-19 17:12:10.633167: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2023-10-19 17:12:10.633175: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2023-10-19 17:12:10.633183: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2023-10-19 17:12:10.633288: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-10-19 17:12:10.633865: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-10-19 17:12:10.634366: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10175 MB memory) -> physical GPU (device: 0, name: Tesla T4, pci bus id: 0000:00:07.0, compute capability: 7.5)
2023-10-19 17:12:10,634 - modelscope - INFO - loading model from /root/.cache/modelscope/hub/damo/cv_unet_universal-matting/tf_graph.pb
WARNING:tensorflow:From /root/miniconda/envs/modelscope/lib/python3.8/site-packages/modelscope/pipelines/cv/image_matting_pipeline.py:47: FastGFile.__init__ (from tensorflow.python.platform.gfile) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.gfile.GFile.
2023-10-19 17:12:14,106 - modelscope - INFO - load model done
INFO:     Started server process [26718]
'''
```
