PR #11762: [GPU] Enable nccl comm split by default
PR #11762: [GPU] Enable nccl comm split by default

Imported from GitHub PR https://github.com/openxla/xla/pull/11762

With https://github.com/openxla/xla/pull/11761 NCCL comm split will be available for multiprocess applications. Using that change, I tested a variety of PAXML and T5X workloads. I found no crashes, hangs, or performance regressions and a good amount of memory savings depending on the model. I tested many more workloads aside from these four, but there was no interesting results since they only used a single communicator.

| Workload                                                                 | Mem diff (MiB) | Speedup |
| ------------------------------------------------------------------------ | ------------- | --------- |
| paxml llama70b evaluate 16xh100 bfloat16 bs-4 ici--1-8-1 dcn--1-2-1      | 0 | 0.9915 |
| paxml GPT3 train 8xh100 bfl16 bs-4 ici--4-1-2                            | \-320 | 1.0029 |
| paxml GPT3 train 8xh100 bfl16 bs-8 ici--1-4-2                            | \-1252 |  1.0026 |
| paxml GPT3pp train 16xh100 bf16 bs-8 ici--2-2-1-2                        | \-2990 | 0.9947 |


Copybara import of the project:

--
a336c41e11ed95bf067ae6c8bebeb167698ddfbb by Trevor Morris <tmorris@nvidia.com>:

Enable nccl comm split by default

Merging this change closes #11762

FUTURE_COPYBARA_INTEGRATE_REVIEW=https://github.com/openxla/xla/pull/11762 from trevor-m:tmorris-split-default a336c41e11ed95bf067ae6c8bebeb167698ddfbb
