Tensorflow Profiler with 2.13. cuptiGetTimestamp: error 999, not reporting events, no data to show...
Try with updating CUDA and cuDNN versions to the latest compatible versions for TensorFlow 2.13. Make sure the path to the CUPTI library is correctly set up. If the issue persists, consider downgrading TensorFlow to version 2.12 or earlier, as the error started occurring after the upgrade. Additionally, check if there are any compatibility issues between TensorFlow and the RTX A6000 and Quatro RTX 8000 GPUs. thanks @msf-caesar I forgot about having this issue open. I have somehow resolved the issue after I remade my environment. I also run into some issue with profiler logs being in wrong directory and unavailable in Tensorboard UI, but I've also found solution in github issues Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/61765">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/61765">No</a>

### Issue type

Bug

### Have you reproduced the bug with TensorFlow Nightly?

No

### Source

binary

### TensorFlow version

2.13

### Custom code

Yes

### OS platform and distribution

Ubuntu 22

### Mobile device

_No response_

### Python version

3.10

### Bazel version

_No response_

### GCC/compiler version

_No response_

### CUDA/cuDNN version

11.8, 8.6

### GPU model and memory

RTX A6000, quatro RTX 8000

### Current behavior?

I'm trying to setup profiler for my trainer, using TF.data from TFRecords, but otherwise training with standard keras code and MirroredStrategy.

I've noticed some strange errors after adding `profile_batch=5` to my tensorboard callback. I've attached logs below.

It's worth noting that I started getting `Local randezvous recv item cancelled` as well after upgrade to 2.13 from 2.12.

I have conda environment with cudatoolkit==11.8.0, cuda-nvcc, and pip installed nvidia-cudnn-cu11==8.6.0.163, tensorflow==2.13.*, tensorboard_plugin_profile... Path to CUPTI should be setup well, as it's in the same path as cudatoolkit installed libcuda

### Standalone code to reproduce the issue

```shell
log_dir = "logs/fit/" + datetime.datetime.now().strftime("%Y%m%d-%H%M%S")
tensorboard_callback = tf.keras.callbacks.TensorBoard(log_dir=log_dir, histogram_freq=1, profile_batch=5)
```


### Relevant log output

```shell
2023-08-31 19:57:55.092538: I tensorflow/tsl/profiler/lib/profiler_session.cc:104] Profiler session initializing.
2023-08-31 19:57:55.092572: I tensorflow/tsl/profiler/lib/profiler_session.cc:119] Profiler session started.
2023-08-31 19:57:55.092639: I tensorflow/compiler/xla/backends/profiler/gpu/cupti_tracer.cc:1671] Profiler found 8 GPUs
2023-08-31 19:57:55.093149: E tensorflow/compiler/xla/backends/profiler/gpu/cupti_error_manager.cc:137] cuptiGetTimestamp: error 999: 
2023-08-31 19:57:55.093178: E tensorflow/compiler/xla/backends/profiler/gpu/cupti_error_manager.cc:186] cuptiSubscribe: ignored due to a previous error.
2023-08-31 19:57:55.093189: E tensorflow/compiler/xla/backends/profiler/gpu/cupti_error_manager.cc:459] cuptiGetResultString: ignored due to a previous error.
2023-08-31 19:57:55.093200: E tensorflow/compiler/xla/backends/profiler/gpu/cupti_tracer.cc:1723] function cupti_interface_->Subscribe( &subscriber_, (CUpti_CallbackFunc)ApiCallback, this)failed with error 
2023-08-31 19:57:55.093257: I tensorflow/tsl/profiler/lib/profiler_session.cc:131] Profiler session tear down.
2023-08-31 19:57:55.093422: E tensorflow/compiler/xla/backends/profiler/gpu/cupti_error_manager.cc:142] cuptiFinalize: ignored due to a previous error.
2023-08-31 19:57:55.093435: E tensorflow/compiler/xla/backends/profiler/gpu/cupti_error_manager.cc:459] cuptiGetResultString: ignored due to a previous error.
2023-08-31 19:57:55.093445: E tensorflow/compiler/xla/backends/profiler/gpu/cupti_tracer.cc:1814] function cupti_interface_->Finalize()failed with error 
Epoch 1/100
...
2023-08-31 19:59:33.523444: I ./tensorflow/compiler/jit/device_compiler.h:186] Compiled cluster using XLA!  This line is logged at most once for the lifetime of the process.
      4/Unknown - 109s 3s/step - loss: 0.7206 - binary_accuracy: 0.5451 - f1_score: 0.1138 - auc: 0.49592023-08-31 19:59:44.532963: I tensorflow/tsl/profiler/lib/profiler_session.cc:10
4] Profiler session initializing.             
2023-08-31 19:59:44.533038: I tensorflow/tsl/profiler/lib/profiler_session.cc:119] Profiler session started.                                                                            
2023-08-31 19:59:44.533099: E tensorflow/compiler/xla/backends/profiler/gpu/cupti_error_manager.cc:135] cuptiGetTimestamp: ignored due to a previous error.
2023-08-31 19:59:44.533216: E tensorflow/compiler/xla/backends/profiler/gpu/cupti_error_manager.cc:186] cuptiSubscribe: ignored due to a previous error.
2023-08-31 19:59:44.533247: E tensorflow/compiler/xla/backends/profiler/gpu/cupti_error_manager.cc:459] cuptiGetResultString: ignored due to a previous error.
2023-08-31 19:59:44.533273: E tensorflow/compiler/xla/backends/profiler/gpu/cupti_tracer.cc:1723] function cupti_interface_->Subscribe( &subscriber_, (CUpti_CallbackFunc)ApiCallback, t
his)failed with error                         
2023-08-31 19:59:48.299203: I tensorflow/tsl/profiler/lib/profiler_session.cc:70] Profiler session collecting data.                                                                     
2023-08-31 19:59:48.352778: E tensorflow/compiler/xla/backends/profiler/gpu/cupti_error_manager.cc:142] cuptiFinalize: ignored due to a previous error.
2023-08-31 19:59:48.352844: E tensorflow/compiler/xla/backends/profiler/gpu/cupti_error_manager.cc:459] cuptiGetResultString: ignored due to a previous error.
2023-08-31 19:59:48.352869: E tensorflow/compiler/xla/backends/profiler/gpu/cupti_tracer.cc:1814] function cupti_interface_->Finalize()failed with error 
2023-08-31 19:59:49.390774: E tensorflow/compiler/xla/backends/profiler/gpu/cupti_error_manager.cc:135] cuptiGetTimestamp: ignored due to a previous error.
2023-08-31 19:59:49.390830: E tensorflow/compiler/xla/backends/profiler/gpu/cupti_error_manager.cc:135] cuptiGetTimestamp: ignored due to a previous error.
2023-08-31 19:59:49.390843: I tensorflow/compiler/xla/backends/profiler/gpu/cupti_collector.cc:541]  GpuTracer has collected 0 callback api events and 0 activity events. 
2023-08-31 19:59:49.413800: I tensorflow/tsl/profiler/lib/profiler_session.cc:131] Profiler session tear down.                                                                          
2023-08-31 19:59:49.415262: I tensorflow/tsl/profiler/rpc/client/save_profile.cc:144] Collecting XSpace to repository: logs/fit/20230831-195755/plugins/profile/2023_08_31_19_59_49/****.net.xplane.pb
```
