My customized OP gives incorrect outputs on GPUs since `tf-nightly 2.13.0.dev20230413`
Hi, 

It is difficult to point to the exact commit and mention what caused this change without any simple reproducible code.

Also, did you test with the latest Nightly version and observed the same behavior in that as well?
 Thanks. I can confirm that the latest nightly version has the same behavior.

I will try my best to provide a reproducible code. I find the reason: in https://github.com/tensorflow/tensorflow/commit/9d1262082e761cd85d6726bcbdfdef331d6d72c6, the stream becomes non-blocking (in `CU_STREAM_NON_BLOCKING` mode). Our codes use the default CUDA stream, so this change breaks our program. The workaround is to call `cudaDeviceSynchronize` before executing our CUDA codes. Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/61209">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/61209">No</a>

### Issue type

Bug

### Have you reproduced the bug with TensorFlow Nightly?

Yes

### Source

binary

### TensorFlow version

2.13

### Custom code

Yes

### OS platform and distribution

fedora 36

### Mobile device

_No response_

### Python version

3.11.4

### Bazel version

_No response_

### GCC/compiler version

_No response_

### CUDA/cuDNN version

_No response_

### GPU model and memory

_No response_

### Current behavior?

I have a complex program based on TensorFlow with several customized OPs. These OPs were created following https://www.tensorflow.org/guide/create_op. Yesterday TF 2.13.0 was released, but after I upgraded to 2.13.0, I found that one of my customized OP gives incorrect results on GPUs and still has the correct outputs on CPUs.

Then I tested many `tf-nightly` versions and found that `tf-nightly 2.13.0.dev20230412` works but `tf-nightly 2.13.0.dev20230413` fails. So the situation is shown in the following table:
| version | CPU       | GPU          |
| -------- | --------- | ----------- |
| tensorflow 2.12.0 | Correct  | Correct      |
| tensorflow 2.13.0 | Correct  | Incorrect   |
| tf-nightly  2.13.0.dev20230412 | Correct  | Correct      |
| tf-nightly  2.13.0.dev20230413 | Correct      | Incorrect   |

I'd like to know what changed between April 12th and 13th related to the customized OPs. This can be a breaking change to downstream applications or an internal bug. Thanks!

Here is a quick link for commits between April 12th and 13th:
https://github.com/tensorflow/tensorflow/commits/master?before=525da8a93eca846e32e5c41eddc0496b25a2ef5b+770


### Standalone code to reproduce the issue

```shell
Indeed, the reason is still unclear to me, so it is hard to create a minimal example.

The code of our customized OPs is https://github.com/deepmodeling/deepmd-kit/blob/37fd8d193362f91c925cf7c2f3a58b97dc921b27/source/op/prod_force_multi_device.cc#L49-L166
```


### Relevant log output

_No response_