How to implement CollectiveAllReduceStrategy?
running multiworkerstrategy using two node getting following error.
```
ica:0/task:1/device:GPU:0 is joining a group with incompatible device typeCPU (group_key=3)
2023-06-13 13:30:32.188736: E tensorflow/core/common_runtime/base_collective_executor.cc:249] BaseCollectiveExecutor::StartAbort INTERNAL: Device /job:worker/replica:0/task:1/device:GPU:0 is joining a group with incompatible device typeCPU (group_key=3)
Traceback (most recent call last):
  File "multinode_training/test1.py", line 62, in <module>
    multi_worker_model = build_and_compile_cnn_model()
  File "multinode_training/test1.py", line 22, in build_and_compile_cnn_model
    model = tf.keras.Sequential(
  File "/home/idps/.local/lib/python3.8/site-packages/tensorflow/python/trackable/base.py", line 205, in _method_wrapper
    result = method(self, *args, **kwargs)
  File "/home/idps/.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 70, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "/home/idps/.local/lib/python3.8/site-packages/tensorflow/python/eager/execute.py", line 52, in quick_execute
    tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.InternalError: {{function_node __wrapped__CollectiveBcastSend_device_/job:worker/replica:0/task:0/device:CPU:0}} Collective ops is aborted by: Device /job:worker/replica:0/task:1/device:GPU:0 is joining a group with incompatible device typeCPU (group_key=3)
The error could be from a previous operation. Restart your program to reset. [Op:CollectiveBcastSend]
```
**Sample code**
```
import json
import os
import sys
import time
import numpy as np
import tensorflow as tf

#os.environ["CUDA_VISIBLE_DEVICES"] = "-1"
if "." not in sys.path:
    sys.path.insert(0, ".")

def mnist_dataset(batch_size):
    (x_train, y_train), _ = tf.keras.datasets.mnist.load_data()
    # The `x` arrays are in uint8 and have values in the range [0, 255].
    # You need to convert them to float32 with values in the range [0, 1]
    x_train = x_train / np.float32(255)
    y_train = y_train.astype(np.int64)
    train_dataset = tf.data.Dataset.from_tensor_slices((x_train, y_train)).shuffle(60000).repeat().batch(batch_size)
    return train_dataset

def build_and_compile_cnn_model():
    model = tf.keras.Sequential(
        [
            tf.keras.Input(shape=(28, 28)),
            tf.keras.layers.Reshape(target_shape=(28, 28, 1)),
            tf.keras.layers.Conv2D(32, 3, activation="relu"),
            tf.keras.layers.Flatten(),
            tf.keras.layers.Dense(128, activation="relu"),
            tf.keras.layers.Dense(10),
        ]
    )
    model.compile(
        loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
        optimizer=tf.keras.optimizers.SGD(learning_rate=0.001),
        metrics=["accuracy"],
    )
    return model

start_time = time.time()

tf_config = {
    'cluster': {
        'worker': [server1ip:8087', server2ip:8087']
    },
    'task': {'type': 'worker', 'index': 1}
}
os.environ['TF_CONFIG'] = json.dumps(tf_config)
print(os.environ['TF_CONFIG'])
per_worker_batch_size = 64
tf_config = json.loads(os.environ["TF_CONFIG"])
num_workers = len(tf_config["cluster"]["worker"])

strategy = tf.distribute.MultiWorkerMirroredStrategy()
global_batch_size = 64
options = tf.data.Options()
options.experimental_distribute.auto_shard_policy = tf.data.experimental.AutoShardPolicy.DATA
multi_worker_dataset = mnist_dataset(global_batch_size)
multi_worker_dataset_with_shrd = multi_worker_dataset.with_options(options)

with strategy.scope():
    # Model building/compiling need to be within `strategy.scope()`.
    multi_worker_model = build_and_compile_cnn_model()
multi_worker_model.fit(multi_worker_dataset_with_shrd, epochs=50, steps_per_epoch=70)
elapsed_time = time.time() - start_time
str_elapsed_time = time.strftime("%H : %M : %S", time.gmtime(elapsed_time))
print(">> Finished. Time elapsed: {}.".format(str_elapsed_time))
```
**worker tf config**
```
{'cluster': {'worker': [server1ip:8087', server2ip:8087']},'task': {'type': 'worker', 'index': 0}}
{'cluster': {'worker': [server1ip:8087', server2ip:8087']},'task': {'type': 'worker', 'index': 1}}
``` @purvang3,
Distribution strategy that uses collective ops for all-reduce. It is similar to MirroredStrategy but it uses collective ops for reduction. By default it uses all local GPUs or CPU for single-worker training.

When 'TF_CONFIG' environment variable is given, it parses cluster_spec, task_type and task_id from 'TF_CONFIG' and turns into a multi-worker strategy which mirrores models on GPUs of all machines in a cluster. In the current implementation, it uses all GPUs in a cluster and it assumes all workers have the same number of GPUs.

Could you please take a look at the below file where the details are mentioned for the CollectiveAllReduceStrategy.
https://github.com/tensorflow/tensorflow/blob/6f650f54e4c16be4fe94bc9e2172077481b224ac/tensorflow/python/distribute/collective_all_reduce_strategy.py
 @tilakrayal Thanks for reply.
1) Based on your answer, CollectiveAllReduceStrategy **is not** MultiWorkerStrategy. Right? If not, then how to implement CollectiveAllReduceStrategy.

2)  I followed above example as well below for MultiWorkerStrategy, but training hangs and doesn't proceed. how can I make it work?
https://www.tensorflow.org/tutorials/distribute/multi_worker_with_ctl using NCCL makes training hang. AUTO works fine.

options = tf.distribute.experimental.CommunicationOptions(                                                                                                      implementation=tf.distribute.experimental.CommunicationImplementation.NCCL                                                                              
)                                                                                                                                                           
strategy = tf.distribute.MultiWorkerMirroredStrategy(communication_options=options)   CollectiveAllReduceStrategy is MultiWorkerMirrorStrategy. CollectiveAllReduceStrategy is a name we used in the implementation. Please refer to documentations of MultiWorkerMirrorStrategy. Thanks @w-xinyi. That was I expected after reading code.

Also as mentioned earlier, I am not able to run NCCL based communication. AUTO and RING works fine.

NCCL version : 2.12.12-1+cuda11.6
CUDA version : 11.6
tensorflow version : 2.11 According to the error message it's saying that you have both CPU and GPU in your cluster. In this case NCCL cannot be used. @w-xinyi . I am trying to run 2 nodes, each with 4 A100 gpus and same version of NCCL, cuda and tensorflow. I am running only on GPU. Also mentioned error is solved and there is no error when I use
tf.distribute.experimental.CommunicationImplementation.RING or tf.distribute.experimental.CommunicationImplementation.AUTO.

when I use tf.distribute.experimental.CommunicationImplementation.NCCL, **training hangs**, no error. This is expected because there are some edge cases that tf.distribute.experimental.CommunicationImplementation.NCCL could not support. From experiences, AUTO and RING could cover more cases. And when using AUTO, it will fall back to the supported CommunicationImplementation type when running into these cases.
 Thanks for reply @w-xinyi.

getting following error for my code. how can I resolve this?

```
2023-06-22 12:03:00.324764: I tensorflow/compiler/xla/stream_executor/cuda/cuda_blas.cc:630] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
2023-06-22 12:03:02.385092: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at xla_ops.cc:417 : INVALID_ARGUMENT: Trying to access resource Resource-5-at-0x3584b140 located in device /job:worker/r
eplica:0/task:0/device:GPU:0 from device /job:worker/replica:0/task:0/device:GPU:1
 Cf. https://www.tensorflow.org/xla/known_issues#tfvariable_on_a_different_device
2023-06-22 12:03:02.385305: E tensorflow/core/common_runtime/ring_alg.cc:291] Aborting RingReduce with CANCELLED: Operation was cancelled for BufRendezvous key 1:159:0:0:0:0:1
2023-06-22 12:03:02.385359: E tensorflow/core/common_runtime/ring_alg.cc:291] Aborting RingReduce with CANCELLED: RPC Request was cancelled
2023-06-22 12:03:02.385971: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at xla_ops.cc:417 : INVALID_ARGUMENT: Trying to access resource Resource-5-at-0x3584b140 located in device /job:worker/r
eplica:0/task:0/device:GPU:0 from device /job:worker/replica:0/task:0/device:GPU:1
 Cf. https://www.tensorflow.org/xla/known_issues#tfvariable_on_a_different_device
2023-06-22 12:03:02.386056: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at xla_ops.cc:417 : INVALID_ARGUMENT: Trying to access resource Resource-5-at-0x3584b140 located in device /job:worker/r
eplica:0/task:0/device:GPU:0 from device /job:worker/replica:0/task:0/device:GPU:1
 Cf. https://www.tensorflow.org/xla/known_issues#tfvariable_on_a_different_device
2023-06-22 12:03:02.386133: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at xla_ops.cc:417 : INVALID_ARGUMENT: Trying to access resource Resource-5-at-0x3584b140 located in device /job:worker/r
eplica:0/task:0/device:GPU:0 from device /job:worker/replica:0/task:0/device:GPU:1
 Cf. https://www.tensorflow.org/xla/known_issues#tfvariable_on_a_different_device
2023-06-22 12:03:02.386209: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at xla_ops.cc:417 : INVALID_ARGUMENT: Trying to access resource Resource-5-at-0x3584b140 located in device /job:worker/r
eplica:0/task:0/device:GPU:0 from device /job:worker/replica:0/task:0/device:GPU:1
 Cf. https://www.tensorflow.org/xla/known_issues#tfvariable_on_a_different_device
2023-06-22 12:03:02.386278: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at xla_ops.cc:417 : INVALID_ARGUMENT: Trying to access resource Resource-5-at-0x3584b140 located in device /job:worker/r
eplica:0/task:0/device:GPU:0 from device /job:worker/replica:0/task:0/device:GPU:1
 Cf. https://www.tensorflow.org/xla/known_issues#tfvariable_on_a_different_device
2023-06-22 12:03:02.386336: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at xla_ops.cc:417 : INVALID_ARGUMENT: Trying to access resource Resource-5-at-0x3584b140 located in device /job:worker/r
eplica:0/task:0/device:GPU:0 from device /job:worker/replica:0/task:0/device:GPU:1
 Cf. https://www.tensorflow.org/xla/known_issues#tfvariable_on_a_different_device
2023-06-22 12:03:02.386400: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at xla_ops.cc:417 : INVALID_ARGUMENT: Trying to access resource Resource-5-at-0x3584b140 located in device /job:worker/r
eplica:0/task:0/device:GPU:0 from device /job:worker/replica:0/task:0/device:GPU:1
 Cf. https://www.tensorflow.org/xla/known_issues#tfvariable_on_a_different_device
2023-06-22 12:03:02.386461: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at xla_ops.cc:417 : INVALID_ARGUMENT: Trying to access resource Resource-5-at-0x3584b140 located in device /job:worker/r
eplica:0/task:0/device:GPU:0 from device /job:worker/replica:0/task:0/device:GPU:1
 Cf. https://www.tensorflow.org/xla/known_issues#tfvariable_on_a_different_device
2023-06-22 12:03:02.386524: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at xla_ops.cc:417 : INVALID_ARGUMENT: Trying to access resource Resource-5-at-0x3584b140 located in device /job:worker/r
eplica:0/task:0/device:GPU:0 from device /job:worker/replica:0/task:0/device:GPU:1
 Cf. https://www.tensorflow.org/xla/known_issues#tfvariable_on_a_different_device
```
 Could you please try adding tf.config.set_soft_device_placement(True)? Also, could you tell if this error is encountered when defining the model / compiling / model.fit /etc? This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you. it was packages issue. solved. Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/60847">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/60847">No</a>

<details><summary>Click to expand!</summary> 
 
 ### Issue Type

Others

### Have you reproduced the bug with TF nightly?

Yes

### Source

binary

### Tensorflow Version

2,9

### Custom Code

Yes

### OS Platform and Distribution

Ubuntu 18.04

### Mobile device

_No response_

### Python version

3.9

### Bazel version

_No response_

### GCC/Compiler version

_No response_

### CUDA/cuDNN version

_No response_

### GPU model and memory

_No response_

### Current Behaviour?

I am trying use CollectiveAllReduceStrategy using tf 2.9, but couldn't find much information.

Based on following image, is **CollectiveAllReduceStrategy** is **MultiWorkerMirrorStrategy**?

<img width="1095" alt="image" src="https://github.com/tensorflow/tensorflow/assets/39809304/fa23ea39-b713-4642-a7cd-6de8bb118c56">

Thanks


### Standalone code to reproduce the issue

```shell
NA
```


### Relevant log output

```shell
NA
```
</details>