GPU error Tensorflow with 2.1
@junaidjawaid1,
Have you installed CUDA from the [official site](https://developer.nvidia.com/cuda-downloads)? And have you installed the right version? The [current one is 11.8](https://www.tensorflow.org/install/source#gpu).

Also you are trying with tensorflow v2.1 which is a pretty older version.  It's unlikely for TF 2.1 version to receive any bug fixes except when we have security patches. There is a high possibility that this was fixed with later TF versions. Perhaps you can use the latest tf versions for your case. 
https://www.tensorflow.org/install
Thank you! > 

Thank you so much for the response. I am using v2.1 cause LMS support is only available with this version. If you can guide me regarding memory swapping in the latest versions that would really helpful.
Thank you. @junaidjawaid1,
Tensorflow v2.1 is compatible with python 2.7, 3.5-3.7, compile GCC 7.3.1, Bazel 0.27.1, CUDA  7.6, cuDNN 10.1. 
https://www.tensorflow.org/install/source#gpu
Also as the error(ModuleNotFoundError: No module named 'nvidia') stated, this error occurred due to Nvidia and not from the tensorflow. Thank you!


 This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you. This issue was closed because it has been inactive for 7 days since being marked as stale. Please reopen if you'd like to work on this further. Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/61649">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/61649">No</a>
 > Are you satisfied with the resolution of your issue? [Yes](https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/61649) [No](https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/61649)

yes, I am satisfied. Thank you all for your response.
### Issue type

Others

### Have you reproduced the bug with TensorFlow Nightly?

No

### Source

source

### TensorFlow version

2.1

### Custom code

Yes

### OS platform and distribution

CentOS Linux release 7.4.1708 

### Mobile device

_No response_

### Python version

3.6

### Bazel version

_No response_

### GCC/compiler version

_No response_

### CUDA/cuDNN version

10/7

### GPU model and memory

Nvidia A100 partitioned virtually in two 40GB GPUs, I am using one of them

### Current behavior?

I am working on 3D_ U-net. I am getting the ptax error with Tensorflow 2.1 when I run the 3D U-net , I am using tensorflow-large-model-support to scale the algorithm



### Standalone code to reproduce the issue

```shell
https://github.com/junaidjawaid1/3d_U-Net-TFLMS/tree/main```


### Relevant log output

```shell
Traceback (most recent call last):

File "<string>", line 1, in <module>

ModuleNotFoundError: No module named 'nvidia'

dirname: missing operand

Try 'dirname --help' for more information.

/opt/gridengine/default/spool/compute-0-3/job_scripts/108258: line 8: $'\342\200\213': command not found

/opt/gridengine/default/spool/compute-0-3/job_scripts/108258: line 10: $'\342\200\213': command not found

Traceback (most recent call last):

File "<string>", line 1, in <module>

ModuleNotFoundError: No module named 'nvidia'

dirname: missing operand

Try 'dirname --help' for more information.

2023-08-18 18:25:55.025052: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.2

2023-08-18 18:25:56.869820: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libnvinfer.so.7

2023-08-18 18:25:56.881355: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libnvinfer_plugin.so.7

2023-08-18 18:26:00.093020: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1

2023-08-18 18:26:00.188610: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero

2023-08-18 18:26:00.192419: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1558] Found device 0 with properties:

pciBusID: 2c8f7:00:00.0 name: NVIDIA A100 80GB PCIe MIG 1c.4g.40gb computeCapability: 8.0

coreClock: 1.41GHz coreCount: 14 deviceMemorySize: 39.25GiB deviceMemoryBandwidth: 901.22GiB/s

2023-08-18 18:26:00.192453: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.2

2023-08-18 18:26:00.192490: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10

2023-08-18 18:26:00.218166: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10

2023-08-18 18:26:00.304953: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10

2023-08-18 18:26:00.356784: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10

2023-08-18 18:26:00.396554: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10

2023-08-18 18:26:00.396599: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7

2023-08-18 18:26:00.396731: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero

2023-08-18 18:26:00.398058: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero

2023-08-18 18:26:00.399160: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1700] Adding visible gpu devices: 0

2023-08-18 18:26:00.411267: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 AVX512F FMA

2023-08-18 18:26:00.434767: I tensorflow/core/platform/profile_utils/cpu_utils.cc:101] CPU Frequency: 2249595000 Hz

2023-08-18 18:26:00.439151: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55fc06e5d220 initialized for platform Host (this does not guarantee that XLA will be used). Devices:

2023-08-18 18:26:00.439189: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version

2023-08-18 18:26:00.648000: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero

2023-08-18 18:26:00.648956: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55fc06ec3b80 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:

2023-08-18 18:26:00.648984: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA A100 80GB PCIe MIG 1c.4g.40gb, Compute Capability 8.0

2023-08-18 18:26:00.649332: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero

2023-08-18 18:26:00.650184: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1558] Found device 0 with properties:

pciBusID: 2c8f7:00:00.0 name: NVIDIA A100 80GB PCIe MIG 1c.4g.40gb computeCapability: 8.0

coreClock: 1.41GHz coreCount: 14 deviceMemorySize: 39.25GiB deviceMemoryBandwidth: 901.22GiB/s

2023-08-18 18:26:00.650216: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.2

2023-08-18 18:26:00.650232: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10

2023-08-18 18:26:00.650249: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10

2023-08-18 18:26:00.650258: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10

2023-08-18 18:26:00.650267: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10

2023-08-18 18:26:00.650276: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10

2023-08-18 18:26:00.650294: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7

2023-08-18 18:26:00.650351: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero

2023-08-18 18:26:00.651127: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero

2023-08-18 18:26:00.651846: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1700] Adding visible gpu devices: 0

2023-08-18 18:26:00.651877: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.2

2023-08-18 18:32:04.446608: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1099] Device interconnect StreamExecutor with strength 1 edge matrix:

2023-08-18 18:32:04.446938: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1105] 0

2023-08-18 18:32:04.446953: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1118] 0: N

2023-08-18 18:32:04.447365: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero

2023-08-18 18:32:04.448403: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero

2023-08-18 18:32:04.449678: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1244] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 120000 MB memory) -> physical GPU (device: 0, name: NVIDIA A100 80GB PCIe MIG 1c.4g.40gb, pci bus id: 2c8f7:00:00.0, compute capability: 8.0)

2023-08-18 18:32:04.460014: I tensorflow/stream_executor/cuda/cuda_driver.cc:801] failed to allocate 117.19G (125829120000 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory

2023-08-18 18:32:04.464233: I tensorflow/stream_executor/cuda/cuda_driver.cc:801] failed to allocate 105.47G (113246208000 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory

2023-08-18 18:32:04.468360: I tensorflow/stream_executor/cuda/cuda_driver.cc:801] failed to allocate 94.92G (101921587200 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory

2023-08-18 18:32:04.472455: I tensorflow/stream_executor/cuda/cuda_driver.cc:801] failed to allocate 85.43G (91729428480 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory

2023-08-18 18:32:04.476553: I tensorflow/stream_executor/cuda/cuda_driver.cc:801] failed to allocate 76.89G (82556485632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory

2023-08-18 18:32:04.480809: I tensorflow/stream_executor/cuda/cuda_driver.cc:801] failed to allocate 69.20G (74300833792 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory

2023-08-18 18:32:04.484893: I tensorflow/stream_executor/cuda/cuda_driver.cc:801] failed to allocate 62.28G (66870747136 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory

2023-08-18 18:32:04.488957: I tensorflow/stream_executor/cuda/cuda_driver.cc:801] failed to allocate 56.05G (60183670784 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory

2023-08-18 18:32:04.493121: I tensorflow/stream_executor/cuda/cuda_driver.cc:801] failed to allocate 50.45G (54165303296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory

2023-08-18 18:32:04.497205: I tensorflow/stream_executor/cuda/cuda_driver.cc:801] failed to allocate 45.40G (48748773376 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory

2023-08-18 18:32:04.501259: I tensorflow/stream_executor/cuda/cuda_driver.cc:801] failed to allocate 40.86G (43873894400 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory

WARNING:tensorflow:sample_weight modes were coerced from

...

to

['...']

WARNING:tensorflow:sample_weight modes were coerced from

...

to

['...']

2023-08-18 18:33:25.857187: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10

2023-08-18 18:35:29.287470: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7

2023-08-18 18:46:02.247907: W tensorflow/stream_executor/gpu/redzone_allocator.cc:312] Internal: ptxas exited with non-zero error code 65280, output: ptxas fatal : Value 'sm_80' is not defined for option 'gpu-name'

Relying on driver to perform ptx compilation. This message will be only logged once.

2023-08-18 18:46:36.474700: F tensorflow/stream_executor/cuda/cuda_dnn.cc:516] Check failed: cudnnSetTensorNdDescriptor(handle_.get(), elem_type, nd, [dims.data](https://dims.data/)(), [strides.data](https://strides.data/)()) == CUDNN_STATUS_SUCCESS (3 vs. 0)batch_descriptor: {count: 7 feature_map_count: 146 spatial: 64 0 64 value_min: 0.000000 value_max: 0.000000 layout: BatchDepthYX}

/opt/gridengine/default/spool/compute-0-3/job_scripts/108258: line 13: 237129 Aborted python $PYTHON_SCRIPT
```
