Python tests fails in GPU VM fails with tensorflow and CUDA execeptions of capable devices are busy or unavailable
@arielman,
Could you please try [limiting gpu memory growth](https://www.tensorflow.org/guide/gpu#limiting_gpu_memory_growth) and see if it helps. Before that please make sure you close all python sessions and exit python interpreter, we want to make sure your gpu is not utilized any simulations.

```
# on the top of your script
import tensorflow as tf
gpus = tf.config.list_physical_devices('GPU')
tf.config.experimental.set_memory_growth(gpus[0], True)
# Paste rest of your code here
```

Also please have a look at  this thread from the Nvidia forum for the similar issue.
https://forums.developer.nvidia.com/t/nvidia-persistenced-failed-to-initialize-check-syslog-for-more-details/74052/4
https://forums.developer.nvidia.com/t/nvidia-persistenced-failed-to-initialize-check-syslog-for-more-details/74052/7

Thank you! This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you. This issue was closed because it has been inactive for 7 days since being marked as stale. Please reopen if you'd like to work on this further. Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/62351">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/62351">No</a>

In the last days, some python tests started to fail on our GPU VM:

`E         tensorflow.python.framework.errors_impl.InternalError: CUDA runtime implicit initialization on GPU:0 failed. Status: all CUDA-capable devices are busy or unavailable`

We check the VM and couldn't find anything suspicious: driver is ok, docker engine is ok, gpu is ok.

this is the output of NVIDIA-SMI from the machine

`NVIDIA-SMI 460.73.01    Driver Version: 460.73.01    CUDA Version: 11.2`
Can someone direct me what else I need to check?