TF 2.9 on Ubuntu 22 utilizes less memory than TF 2.3 on Ubuntu 16
@JackTemaki 
Could you please provide a reproducible code or complete code to replicate the issue reported here ?

Thank you !
 The two lines of python code I posted are already sufficient to show the mismatch in allocated memory in the "created device" log statement. If you want something that shows also the mismatch in `nvidia-smi` usage, I can try to produce a minimal example. Opening an `ipython` session with just:
```python3
In [1]: import tensorflow as tf
In [2]: x = tf.constant(1)
```
Is sufficient. This will block the GPU and show the difference in memory allocation in both the log message and in `nvidia-smi`.

I also tried `export TF_FORCE_GPU_ALLOW_GROWTH=true` but this did not change anything. It seems that TF changed its default memory allocation at some point. I was able to increase the allocated memory correctly using the old `v1` API with e.g.:
```python
gpu_options = tf.compat.v1.GPUOptions(per_process_gpu_memory_fraction=1.0)
config = tf.compat.v1.ConfigProto(gpu_options=gpu_options)

loaded_graph = tf.Graph()
with tf.compat.v1.Session(graph=loaded_graph, config=config) as sess:
    pass
```

~~Nevertheless, with the "new" API this did not work, I tried it with the following:~~
For the new API it does work using:
```python
gpus = tf.config.list_physical_devices('GPU')
if gpus:
  try:
    tf.config.set_logical_device_configuration(
        gpus[0],
        [tf.config.LogicalDeviceConfiguration(memory_limit=10500)])  # force 10.5GB
    logical_gpus = tf.config.list_logical_devices('GPU')
    print(len(gpus), "Physical GPUs,", len(logical_gpus), "Logical GPUs")
  except RuntimeError as e:
    # Virtual devices must be set before GPUs have been initialized
    print(e)
```
 Additional note: Manually setting the memory limit to a factor of 0.92 for the old API or to the limit of 10100 for the new API gave us the old behavior again.

Maybe someone can comment if this is a dangerous solution and newer TF versions need more free space to function properly, but at least for us it seems to work this way. Hi @JackTemaki ,
As I don't have environment for 2.3v now, I tried to replicate the reported behaviour with Tf2.10,TF2.11 and Tf 2.12v with Ubuntu VM having 2XA100 GPU(40GB) and I observed there is no much difference among Tf2.10,Tf2.11 and Tf2.12 versions.

**With TF2.9.2 :** Got 38251 MB out of 40000 MB
```
2.9.2
2023-02-22 09:17:40.823117: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F AVX512_VNNI FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-02-22 09:17:42.034974: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 38251 MB memory:  -> device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:00:04.0, compute capability: 8.0
2023-02-22 09:17:42.036673: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:1 with 38251 MB memory:  -> device: 1, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:00:05.0, compute capability: 8.0
(tf2.9) suryanarayanay@surya-ubuntu-22-04:~$ 
```

**With TF2.11v:** Got 38235 MB/ 40000 MB
```
2.11.0
2023-02-22 09:03:54.865804: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F AVX512_VNNI FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-02-22 09:03:56.192602: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1613] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 38235 MB memory:  -> device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:00:04.0, compute capability: 8.0
2023-02-22 09:03:56.194130: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1613] Created device /job:localhost/replica:0/task:0/device:GPU:1 with 38235 MB memory:  -> device: 1, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:00:05.0, compute capability: 8.0
(tf2.11) suryanarayanay@surya-ubuntu-22-04:~$ 
```
**With TF2.12v:** 38201MB/ 40000MB
```
(tf) suryanarayanay@surya-ubuntu-22-04:~$ python 59732.py
2023-02-22 09:05:10.877697: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2023-02-22 09:05:11.704764: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-02-22 09:05:13.640413: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2.12.0-rc0
2023-02-22 09:05:22.580744: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1635] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 38201 MB memory:  -> device: 0, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:00:04.0, compute capability: 8.0
2023-02-22 09:05:22.583282: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1635] Created device /job:localhost/replica:0/task:0/device:GPU:1 with 38201 MB memory:  -> device: 1, name: NVIDIA A100-SXM4-40GB, pci bus id: 0000:00:05.0, compute capability: 8.0
(tf) suryanarayanay@surya-ubuntu-22-04:~$ 
```
From your case it's around 400 MB difference wrt 2.3v and 2.9v which is not much higher and this amount of memory shortage should not lead OOM except very few cases which needs around neck to neck 10 GB RAM. 

The proposed workaround using logical devices with memory setting to full available memory may works for TF but its not recommended always.Keeping some buffer GPU memory for other processes might be needed depending on individual requirement.Also if you are configuring logical gpus the memory limit configuration will be applied for current process only and for model trainings you may try it.Hope I answered your query. Thanks !
 > From your case it's around 400 MB difference wrt 2.3v and 2.9v which is not much higher and this amount of memory shortage should not lead OOM except very few cases which needs around neck to neck 10 GB RAM.

Our training runs are extremely tuned towards the maximum, so 400mb make the difference between running OOM or not.

> The proposed workaround using logical devices with memory setting to full available memory may works for TF but its not recommended always.Keeping some buffer GPU memory for other processes might be needed depending on individual requirement.Also if you are configuring logical gpus the memory limit configuration will be applied for current process only and for model trainings you may try it.Hope I answered your query. Thanks !

Yes, for us everything is solved. Taking 100% is certainly not a good idea and lead to crashes. This is why we sticked to 92-93% now, which seems stable and gives us even slightly more memory than before.

 Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/59732">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/59732">No</a>

<details><summary>Click to expand!</summary> 
 
 ### Issue Type

Performance

### Have you reproduced the bug with TF nightly?

No

### Source

binary

### Tensorflow Version

tf 2.9

### Custom Code

No

### OS Platform and Distribution

Ubuntu 22

### Mobile device

_No response_

### Python version

3.10

### Bazel version

_No response_

### GCC/Compiler version

_No response_

### CUDA/cuDNN version

11.2

### GPU model and memory

_No response_

### Current Behaviour?


We recently switched our cluster system from Ubuntu 16.04, TF2.3, CUDA 10.1 to Ubuntu 22.04, TF2.9 and CUDA 11.2. After some trainings crashed with an out-of-memory error we found that Tensorflow is now utilizing less memory than before (ca. 600mb on an RTX 2080 Ti).

The nvidia-driver is 510.85.02 for Ubuntu 22 and 510.54 for Ubuntu 16.

I understand many factors can play a role here (OS version, TF version, Cuda version). I will update the information once I am able to also run TF2.3/Cuda10.1 on Ubuntu 22.



### Standalone code to reproduce the issue

```shell
import tensorflow as tf
tf.device("gpu")
```


### Relevant log output


Before (Ubuntu 16.04, TF 2.3, CUDA 10.1):

```
Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10078 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:e1:00.0, compute capability: 7.5)
```

relevant `nvidia-smi` output:
```
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:01:00.0 Off |                  N/A |
| 45%   73C    P2   215W / 250W |  10838MiB / 11264MiB |     90%      Default |
|                               |                      |                  N/A |
```

After (Ubuntu 22.04, TF2.9, CUDA 11.2)

```
Created device /device:GPU:0 with 9650 MB memory:  -> device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:20:00.0, compute capability: 7.5
```

relevant `nvidia-smi` output:
```
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:1D:00.0 Off |                  N/A |
| 78%   72C    P2   191W / 250W |  10266MiB / 11264MiB |     79%      Default |
|                               |                      |                  N/A |
```
```
</details>