tensorflow/core/common_runtime/gpu/gpu_util.cc:293] GPU->CPU Memcpy failed
@HAMZA12337,
We see that you are using tf version 1.15, 1.x is not actively supported, please update to 2.x and let us know if you are facing the same issue.

And also you have provided the code which is suitable for 1.x version. Could you please try to migrate your TensorFlow code from TensorFlow 1.x to TensorFlow 2 with the help of this official document.
https://www.tensorflow.org/guide/migrate

Also please try to install the tensorflow latest stable version by referring to the official build documentation.
https://www.tensorflow.org/install

Thank you! But i don't now why i am using tensorflow 1.15 with gpu == geforce gtx 1070 it's work but with gpu rtx 4070 no @HAMZA12337,
 Its unlikely for TF 1.x version to receive any bug fixes. There is a high possibility that this was fixed with later TF versions. Perhaps you can use latest tf versions for your case. 
https://www.tensorflow.org/install

Could you please try to migrate your TensorFlow code from TensorFlow 1.x to TensorFlow 2 with the help of this official document. Also **tf.compat.v1.Session** API was designed for TensorFlow v1. See the [TensorFlow v1 to TensorFlow v2 migration guide](https://www.tensorflow.org/guide/migrate) for instructions on how to migrate the rest of your code.

Thank you! This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you. This issue was closed because it has been inactive for 7 days since being marked as stale. Please reopen if you'd like to work on this further. Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/61327">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/61327">No</a>

### Issue type

Bug

### Have you reproduced the bug with TensorFlow Nightly?

Yes

### Source

source

### TensorFlow version

1.15

### Custom code

Yes

### OS platform and distribution

Windows

### Mobile device

_No response_

### Python version

3.7

### Bazel version

_No response_

### GCC/compiler version

_No response_

### CUDA/cuDNN version

cuda version = 10.0
cudnn=7.6.4
_No response_

### GPU model and memory
Geforce RTX 4070 TI 12 GB
_No response_

### Current behavior?

I am using 

gpu geforce rtx 4070 ti 12 gb 

i add in my training file 

    config1 = tf.compat.v1.ConfigProto()
    config1.gpu_options.allow_growth = True
    session = tf.compat.v1.Session(config=config1)

But Nothing happen I get this issue 

### Standalone code to reproduce the issue

```shell
2023-07-19 13:40:37.744821: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_100.dll
2023-07-19 13:40:37.744955: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cublas64_100.dll
2023-07-19 13:40:37.745047: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cufft64_100.dll
2023-07-19 13:40:37.745166: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library curand64_100.dll
2023-07-19 13:40:37.745290: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusolver64_100.dll
2023-07-19 13:40:37.745374: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusparse64_100.dll
2023-07-19 13:40:37.745493: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudnn64_7.dll
2023-07-19 13:40:37.745596: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2023-07-19 13:40:37.745706: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2023-07-19 13:40:37.745789: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165]      0
2023-07-19 13:40:37.745866: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0:   N
2023-07-19 13:40:37.746009: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10400 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 4070 Ti, pci bus id: 0000:01:00.0, compute capability: 8.9)
WARNING:tensorflow:From C:\Users\user\AppData\Roaming\Python\Python37\site-packages\keras\backend\tensorflow_backend.py:300: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.

WARNING:tensorflow:From C:\Users\user\AppData\Roaming\Python\Python37\site-packages\keras\backend\tensorflow_backend.py:308: The name tf.variables_initializer is deprecated. Please use tf.compat.v1.variables_initializer instead.

2023-07-19 13:40:40.254879: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudnn64_7.dll
2023-07-19 13:43:12.728939: W tensorflow/stream_executor/cuda/redzone_allocator.cc:312] Internal: Invoking ptxas not supported on Windows
Relying on driver to perform ptx compilation. This message will be only logged once.
2023-07-19 13:43:12.932855: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cublas64_100.dll
2023-07-19 13:43:21.066511: E tensorflow/stream_executor/cuda/cuda_blas.cc:428] failed to run cuBLAS routine: CUBLAS_STATUS_EXECUTION_FAILED
Exception: Blas GEMM launch failed : a.shape=(2, 2048), b.shape=(2, 36), m=2048, n=36, k=2
         [[node gradients_1/dense_regress_10/MatMul_grad/MatMul_1 (defined at C:\Users\user\AppData\Roaming\Python\Python37\site-packages\tensorflow_core\python\framework\ops.py:1748) ]]

Original stack trace for 'gradients_1/dense_regress_10/MatMul_grad/MatMul_1':
  File "c:/Users/user/Desktop/Binarios/keras_frcnn-master-atelier-B/keras_frcnn-master/train_frcnn_kitti.py", line 262, in <module>
    train_kitti()
  File "c:/Users/user/Desktop/Binarios/keras_frcnn-master-atelier-B/keras_frcnn-master/train_frcnn_kitti.py", line 205, in train_kitti
    [Y1[:, sel_samples, :], Y2[:, sel_samples, :]])
  File "C:\Users\user\AppData\Roaming\Python\Python37\site-packages\keras\engine\training.py", line 1620, in train_on_batch
    self._make_train_function()
  File "C:\Users\user\AppData\Roaming\Python\Python37\site-packages\keras\engine\training.py", line 1002, in _make_train_function
    self.total_loss)
  File "C:\Users\user\AppData\Roaming\Python\Python37\site-packages\keras\optimizers.py", line 381, in get_updates
    grads = self.get_gradients(loss, params)
  File "C:\Users\user\AppData\Roaming\Python\Python37\site-packages\keras\optimizers.py", line 47, in get_gradients
    grads = K.gradients(loss, params)
  File "C:\Users\user\AppData\Roaming\Python\Python37\site-packages\keras\backend\tensorflow_backend.py", line 2138, in gradients
    return tf.gradients(loss, variables, colocate_gradients_with_ops=True)
  File "C:\Users\user\AppData\Roaming\Python\Python37\site-packages\tensorflow_core\python\ops\gradients_impl.py", line 158, in gradients
    unconnected_gradients)
  File "C:\Users\user\AppData\Roaming\Python\Python37\site-packages\tensorflow_core\python\ops\gradients_util.py", line 679, in _GradientsHelper
    lambda: grad_fn(op, *out_grads))
  File "C:\Users\user\AppData\Roaming\Python\Python37\site-packages\tensorflow_core\python\ops\gradients_util.py", line 350, in _MaybeCompile
    return grad_fn()  # Exit early
  File "C:\Users\user\AppData\Roaming\Python\Python37\site-packages\tensorflow_core\python\ops\gradients_util.py", line 679, in <lambda>
    lambda: grad_fn(op, *out_grads))
  File "C:\Users\user\AppData\Roaming\Python\Python37\site-packages\tensorflow_core\python\ops\math_grad.py", line 1586, in _MatMulGrad
    grad_b = gen_math_ops.mat_mul(a, grad, transpose_a=True)
  File "C:\Users\user\AppData\Roaming\Python\Python37\site-packages\tensorflow_core\python\ops\gen_math_ops.py", line 6136, in mat_mul
    name=name)
  File "C:\Users\user\AppData\Roaming\Python\Python37\site-packages\tensorflow_core\python\framework\op_def_library.py", line 794, in _apply_op_helper
    op_def=op_def)
  File "C:\Users\user\AppData\Roaming\Python\Python37\site-packages\tensorflow_core\python\util\deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "C:\Users\user\AppData\Roaming\Python\Python37\site-packages\tensorflow_core\python\framework\ops.py", line 3357, in create_op
    attrs, op_def, compute_device)
  File "C:\Users\user\AppData\Roaming\Python\Python37\site-packages\tensorflow_core\python\framework\ops.py", line 3426, in _create_op_internal
    op_def=op_def)
  File "C:\Users\user\AppData\Roaming\Python\Python37\site-packages\tensorflow_core\python\framework\ops.py", line 1748, in __init__
    self._traceback = tf_stack.extract_stack()

...which was originally created as op 'dense_regress_10/MatMul', defined at:
  File "c:/Users/user/Desktop/Binarios/keras_frcnn-master-atelier-B/keras_frcnn-master/train_frcnn_kitti.py", line 262, in <module>
    train_kitti()
  File "c:/Users/user/Desktop/Binarios/keras_frcnn-master-atelier-B/keras_frcnn-master/train_frcnn_kitti.py", line 88, in train_kitti
    classifier = nn.classifier(shared_layers, roi_input, cfg.num_rois, nb_classes=len(classes_count), trainable=True)
  File "c:\Users\user\Desktop\Binarios\keras_frcnn-master-atelier-B\keras_frcnn-master\keras_frcnn\resnet.py", line 270, in classifier
    name='dense_regress_{}'.format(nb_classes))(out)
  File "C:\Users\user\AppData\Roaming\Python\Python37\site-packages\keras\engine\topology.py", line 578, in __call__
    output = self.call(inputs, **kwargs)
  File "C:\Users\user\AppData\Roaming\Python\Python37\site-packages\keras\layers\wrappers.py", line 177, in call
    y = self.layer.call(inputs)  # (num_samples * timesteps, ...)
  File "C:\Users\user\AppData\Roaming\Python\Python37\site-packages\keras\layers\core.py", line 840, in call
    output = K.dot(inputs, self.kernel)
  File "C:\Users\user\AppData\Roaming\Python\Python37\site-packages\keras\backend\tensorflow_backend.py", line 848, in dot
    out = tf.matmul(x, y)
  File "C:\Users\user\AppData\Roaming\Python\Python37\site-packages\tensorflow_core\python\util\dispatch.py", line 180, in wrapper
    return target(*args, **kwargs)
  File "C:\Users\user\AppData\Roaming\Python\Python37\site-packages\tensorflow_core\python\ops\math_ops.py", line 2754, in matmul
    a, b, transpose_a=transpose_a, transpose_b=transpose_b, name=name)
  File "C:\Users\user\AppData\Roaming\Python\Python37\site-packages\tensorflow_core\python\ops\gen_math_ops.py", line 6136, in mat_mul
    name=name)
  File "C:\Users\user\AppData\Roaming\Python\Python37\site-packages\tensorflow_core\python\framework\op_def_library.py", line 794, in _apply_op_helper
    op_def=op_def)
  File "C:\Users\user\AppData\Roaming\Python\Python37\site-packages\tensorflow_core\python\util\deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "C:\Users\user\AppData\Roaming\Python\Python37\site-packages\tensorflow_core\python\framework\ops.py", line 3357, in create_op
    attrs, op_def, compute_device)
  File "C:\Users\user\AppData\Roaming\Python\Python37\site-packages\tensorflow_core\python\framework\ops.py", line 3426, in _create_op_internal
    op_def=op_def)
  File "C:\Users\user\AppData\Roaming\Python\Python37\site-packages\tensorflow_core\python\framework\ops.py", line 1748, in __init__
    self._traceback = tf_stack.extract_stack()

2023-07-19 13:43:21.297408: I tensorflow/stream_executor/stream.cc:1990] [stream=0000029EFE927EC0,impl=0000029EB937CFB0] did not wait for [stream=0000029EFE926CC0,impl=0000029EB937CF20]
2023-07-19 13:43:21.297815: I tensorflow/stream_executor/stream.cc:4925] [stream=0000029EFE927EC0,impl=0000029EB937CFB0] did not memcpy device-to-host; source: 00000007129B6500
2023-07-19 13:43:21.298246: F tensorflow/core/common_runtime/gpu/gpu_util.cc:293] GPU->CPU Memcpy failed
2023-07-19 13:43:21.298255: I tensorflow/stream_executor/stream.cc:1990] [stream=0000029EFE927EC0,impl=0000029EB937CFB0] did not wait for [stream=0000029EFE926CC0,impl=0000029EB937CF20]
```


### Relevant log output

_No response_