PR #10965: [GPU] Make xla_gpu_enable_nccl_per_stream_comms false by default
PR #10965: [GPU] Make xla_gpu_enable_nccl_per_stream_comms false by default

Imported from GitHub PR https://github.com/openxla/xla/pull/10965

In #9845 I added a flag to toggle between per-stream communicators. The default option was true to preserve the current behavior. While per-stream comms can provide a speedup at the cost of additional memory, it doesn't seem to help for many common workloads which I have tested. I am proposing that we set the default value to false to reduce the amount of GPU memory used by XLA.

The table below shows how the total GPU memory usage changes (absolute value) and how the steps/sec changes (relative value) by changing --xla_gpu_enable_nccl_per_stream_comms **from true to false**. In all of these cases, we use less memory and the performance stays in general within 1%.

| Workload                                                                 | Mem diff (MiB) | Speedup |
| ------------------------------------------------------------------------ | ------------- | --------- |
| t5x xxl pretrain 8xh100 fp8 bs-36 insize-512 outsize-128 tp-8 te_fmha-1   | \-2596 | 1.0045 |
| t5x xxl pretrain 8xh100 fp8 bs-36 insize-512 outsize-128 tp-8 te_fmha-0   | \-2594 | 0.9963 |
| t5x xxl pretrain 8xh100 bf16 bs-36 insize-512 outsize-128 tp-8 te_fmha-1 | \-2596 | 0.9996 |
| t5x xxl pretrain 8xh100 bf16 bs-36 insize-512 outsize-128 tp-8 te_fmha-0 | \-2596 | 1.0012 |
| t5x xxl pretrain 8xh100 fp8 bs-256 insize-512 outsize-128 tp-8 te_fmha-1 | \-2596 | 1.0040 |
| t5x xxl pretrain 8xh100 fp8 bs-256 insize-512 outsize-128 tp-8 te_fmha-0 | \-2596 | 1.0000 |
| paxml llama70b evaluate 16xh100 bfloat16 bs-4 ici--1-8-1 dcn--1-2-1      | \-2644 | 1.0192 |
| paxml GPT3 train 8xh100 bfl16 bs-4 ici--4-1-2                            | \-2822 | 0.9929 |
| paxml GPT3 train 8xh100 bfl16 bs-8 ici--1-4-2                            | \-5800 |  0.9990 |
| paxml GPT3pp train 16xh100 bf16 bs-8 ici--2-2-1-2                        | \-6582 | 0.9968 |


Test settings
* T5X container: [ghcr.io/nvidia/jax:t5x-2024-03-15](http://ghcr.io/nvidia/jax:t5x-2024-03-15)
* PAX container: [ghcr.io/nvidia/jax:pax-2024-03-15](http://ghcr.io/nvidia/jax:pax-2024-03-15)
* Performed using DGXH100
* NCCL_NVLS_ENABLE=1
* XLA_PYTHON_CLIENT_MEM_FRACTION=0.8 (except paxml GPT3pp which uses 0.7 because it goes OOM otherwise when enable_nccl_per_stream_comms is true)
Copybara import of the project:

--
fed944b06a9fb830a79b874f68732bdd41a9cd75 by Trevor Morris <tmorris@nvidia.com>:

Make xla_gpu_enable_nccl_per_stream_comms false by default

Merging this change closes #10965

FUTURE_COPYBARA_INTEGRATE_REVIEW=https://github.com/openxla/xla/pull/10965 from trevor-m:tmorris-perstreamcomms-false fed944b06a9fb830a79b874f68732bdd41a9cd75
