GPU:0 is joining a group with incompatible device typeCPU
@dengpanyin Thanks for reporting the issue.

In order to expedite the troubleshooting process, please share a stand alone code/snippet to reproduce the issue.

Thanks. @pjpratik  stand alone code is unavailable since this issue is not able to reproduce every time. We still want to understand the root cause since multi-models have the same issue. @dengpanyin Could you please provide more details to debug the issue. It will help us to investigate the issue further.

 Thanks.
 @dengpanyin can you explicitly specify the device type ,so that all the devices in the collective group have the same device type This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you. This issue was closed because it has been inactive for 7 days since being marked as stale. Please reopen if you'd like to work on this further. Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/59888">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/59888">No</a>

<details><summary>Click to expand!</summary> 
 
 ### Issue Type

Bug

### Have you reproduced the bug with TF nightly?

Yes

### Source

source

### Tensorflow Version

tf2.9

### Custom Code

Yes

### OS Platform and Distribution

_No response_

### Mobile device

_No response_

### Python version

_No response_

### Bazel version

_No response_

### GCC/Compiler version

_No response_

### CUDA/cuDNN version

_No response_

### GPU model and memory

_No response_

### Current Behaviour?

```shell
tensorflow.python.framework.errors_impl.InternalError: Collective ops is aborted by: Device /job:worker/replica:0/task:0/device:GPU:0 is joining a group with incompatible device typeCPU (group_key=3)
```


### Standalone code to reproduce the issue

```shell
Observed the above errors twice a week. Can you explain what's the root cause and how to fix it?
```


### Relevant log output

_No response_</details>