Setting memory_limit inside docker does not have an effect
@michaeldiener,
The **memory_limit** in **tf.config.LogicalDeviceConfiguration** API mainly funtions for  Maximum memory (in MB) to allocate on the virtual device. Currently it is only supported for GPUs. 
Could you please confirm whether it is having the problem whenever you are running inside a docker container and also provide if there is any error log? Thank you! @tilakrayal 
Thanks for responding!
Yes, this is for a GPU and Tensorflow is running inside a docker container. 

This is how I call docker:
`docker run --gpus all -it --rm -v ~/tmp2:/exttmp -v ~/jupyter:/extjupyter --shm-size=1g --ulimit memlock=-1 -p 8889:8889 nvcr.io/nvidia/tensorflow:23.05-tf2-py3`

This is the output from setting the memory_limit. No log after setting it:
```
Python 3.10.6 (main, Mar 10 2023, 10:55:28) [GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow as tf
2023-06-12 07:33:41.210045: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2023-06-12 07:33:41.232097: I tensorflow/core/platform/cpu_feature_guard.cc:183] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: SSE3 SSE4.1 SSE4.2 AVX, in other operations, rebuild TensorFlow with the appropriate compiler flags.
>>> gpus = tf.config.experimental.list_physical_devices('GPU')
2023-06-12 07:34:12.541732: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:1013] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-06-12 07:34:12.545833: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:1013] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-06-12 07:34:12.545905: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:1013] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
>>> tf.config.set_logical_device_configuration(
...     gpus[0],
...     [tf.config.LogicalDeviceConfiguration(memory_limit=4096)]
... )
>>> 
``` @michaeldiener,
Currently we have --per_process_gpu_memory_fraction to limit the memory usage of model server and there is no such method available to limit GPU usage on the model level.


Example code to run model server image with memory limit enabled:

`docker run --runtime=nvidia -p 8501:8501 \ --mount type=bind,\ source=/tmp/tfserving/serving/tensorflow_serving/servables/tensorflow/testdata/saved_model_half_plus_two_gpu,\ target=/models/half_plus_two \ -e MODEL_NAME=half_plus_two -t tensorflow/serving:latest-gpu --per_process_gpu_memory_fraction=0.5
`

https://github.com/keras-team/tf-keras/issues/125
Thank you!
 Thanks, didn't know about the model server. This might come handy eventually, but I need to be able to limit GPU memory while training in Jupyter inside docker. There is no way to do that? Again, setting set_logical_device_configuration does not have any effect. I cannot reproduce on a Titan V. I started the docker container with `docker run --privileged --gpus all -it --rm --shm-size=1g --ulimit memlock=-1 -p 8889:8889 nvcr.io/nvidia/tensorflow:23.05-tf2-py3`.

Then I ran the following program in the container to set the memory limit, initialize the GPU and print the memory usage with `nvidia-smi`

```python
import tensorflow as tf
import subprocess

gpus = tf.config.experimental.list_physical_devices('GPU')
tf.config.set_logical_device_configuration(
    gpus[0],
    [tf.config.LogicalDeviceConfiguration(memory_limit=4096)]
)

tf.constant(0.)  # Run GPU initialization
subprocess.run(['nvidia-smi'])
```

nvidia-smi showed only a bit more than 4096 MiB of memory being used.

@michaeldiener are there any extra steps needed to reproduce the issue for you? I doubt the fact I have a different GPU is causing `set_logical_device_configuration` to work correctly only for me. @reedwm Strangely enough, this works now! 
Not sure if there was an update in the meantime for any of the involved components, but either way, problem solved. Thanks for looking into this, I appreciate it. Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/60811">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/60811">No</a>

<details><summary>Click to expand!</summary> 
 
 ### Issue Type

Bug

### Have you reproduced the bug with TF nightly?

No

### Source

source

### Tensorflow Version

2.12.0

### Custom Code

No

### OS Platform and Distribution

Ubuntu 22.04 (inside docker)

### Mobile device

_No response_

### Python version

3.10.6

### Bazel version

_No response_

### GCC/Compiler version

_No response_

### CUDA/cuDNN version

12.1

### GPU model and memory

Nvidia GeForce RTX 4090

### Current Behaviour?

Setting the memory_limit does not have any effect whatsoever when running inside a docker container.
I use the following NGC https://docs.nvidia.com/deeplearning/frameworks/tensorflow-release-notes/rel-23-05.html

### Standalone code to reproduce the issue

```shell
import tensorflow as tf

gpus = tf.config.experimental.list_physical_devices('GPU')
tf.config.set_logical_device_configuration(
	gpus[0],
	[tf.config.LogicalDeviceConfiguration(memory_limit=4096)]
)
```


### Relevant log output

_No response_</details>