tensorflow2.11 with MultiWorkerMirroredStrategy  cannot work
The program is stuck here and cannot be continued。 And it will output log  like below:
2023-05-25 11:10:57.934361: E tensorflow/core/common_runtime/base_collective_executor.cc:249] BaseCollectiveExecutor::StartAbort DEADLINE_EXCEEDED: Deadline Exceeded
Additional GRPC error information from remote target /job:worker/replica:0/task:0:
:{"created":"@1685013057.934031384","description":"Deadline Exceeded","file":"external/com_github_grpc_grpc/src/core/ext/filters/deadline/deadline_filter.cc","file_line":69,"grpc_status":4}
2023-05-25 11:10:57.934497: E tensorflow/core/common_runtime/eager/context_distributed_manager.cc:737] Deadline Exceeded
Additional GRPC error information from remote target /job:worker/replica:0/task:0:
:{"created":"@1685013057.934031384","description":"Deadline Exceeded","file":"external/com_github_grpc_grpc/src/core/ext/filters/deadline/deadline_filter.cc","file_line":69,"grpc_status":4}
2023-05-25 11:10:57.934547: F tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:103] Non-OK-status: Stop() status: UNIMPLEMENTED: Clean shutdown is not currently implemented
Fatal Python error: Aborted

Current thread 0x00007f2947732740 (most recent call first):
  File "/public/home/qianyj/virtualenv/dtk23.04/tf2.9/venv/lib/python3.8/site-packages/tensorflow/python/eager/context.py", line 623 in ensure_initialized
  File "/public/home/qianyj/virtualenv/dtk23.04/tf2.9/venv/lib/python3.8/site-packages/tensorflow/python/distribute/collective_all_reduce_strategy.py", line 508 in _initialize_multi_worker
  File "/public/home/qianyj/virtualenv/dtk23.04/tf2.9/venv/lib/python3.8/site-packages/tensorflow/python/distribute/collective_all_reduce_strategy.py", line 341 in _initialize_strategy
  File "/public/home/qianyj/virtualenv/dtk23.04/tf2.9/venv/lib/python3.8/site-packages/tensorflow/python/distribute/collective_all_reduce_strategy.py", line 329 in __init__
  File "/public/home/qianyj/virtualenv/dtk23.04/tf2.9/venv/lib/python3.8/site-packages/tensorflow/python/distribute/collective_all_reduce_strategy.py", line 185 in __init__
  File "/public/home/qianyj/virtualenv/dtk23.04/tf2.9/venv/lib/python3.8/site-packages/tensorflow/python/distribute/collective_all_reduce_strategy.py", line 252 in __init__
  File "/public/home/qianyj/virtualenv/dtk23.04/tf2.9/venv/lib/python3.8/site-packages/tensorflow/python/util/deprecation.py", line 357 in new_func
  File "/public/home/qianyj/TF_test/dtk23.04/tf2.9/models-2.9.2_fp32_4m/official/common/distribute_utils.py", line 155 in get_distribution_strategy
  File "/public/home/qianyj/TF_test/dtk23.04/tf2.9/models-2.9.2_fp32_4m/official/benchmark/models/resnet_imagenet_main.py", line 125 in run
  File "/public/home/qianyj/TF_test/dtk23.04/tf2.9/models-2.9.2_fp32_4m/official/benchmark/models/resnet_imagenet_main.py", line 362 in main
  File "/public/home/qianyj/.local/lib/python3.8/site-packages/absl/app.py", line 258 in _run_main
  File "/public/home/qianyj/.local/lib/python3.8/site-packages/absl/app.py", line 312 in run
  File "/public/home/qianyj/TF_test/dtk23.04/tf2.9/models-2.9.2_fp32_4m/official/benchmark/models/resnet_imagenet_main.py", line 369 in <module>
2023-05-25 11:10:57.977399: E tensorflow/core/common_runtime/base_collective_executor.cc:249] BaseCollectiveExecutor::StartAbort DEADLINE_EXCEEDED: Deadline Exceeded
Additional GRPC error information from remote target /job:worker/replica:0/task:0:
:{"created":"@1685013057.977083688","description":"Deadline Exceeded","file":"external/com_github_grpc_grpc/src/core/ext/filters/deadline/deadline_filter.cc","file_line":69,"grpc_status":4}
2023-05-25 11:10:57.977561: E tensorflow/core/common_runtime/eager/context_distributed_manager.cc:737] Deadline Exceeded
Additional GRPC error information from remote target /job:worker/replica:0/task:0:
:{"created":"@1685013057.977083688","description":"Deadline Exceeded","file":"external/com_github_grpc_grpc/src/core/ext/filters/deadline/deadline_filter.cc","file_line":69,"grpc_status":4}
2023-05-25 11:10:57.977640: F tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:103] Non-OK-status: Stop() status: UNIMPLEMENTED: Clean shutdown is not currently implemented
Fatal Python error: Aborted

Current thread 0x00007fa0241df740 (most recent call first):
  File "/public/home/qianyj/virtualenv/dtk23.04/tf2.9/venv/lib/python3.8/site-packages/tensorflow/python/eager/context.py", line 623 in ensure_initialized
  File "/public/home/qianyj/virtualenv/dtk23.04/tf2.9/venv/lib/python3.8/site-packages/tensorflow/python/distribute/collective_all_reduce_strategy.py", line 508 in _initialize_multi_worker
  File "/public/home/qianyj/virtualenv/dtk23.04/tf2.9/venv/lib/python3.8/site-packages/tensorflow/python/distribute/collective_all_reduce_strategy.py", line 341 in _initialize_strategy
  File "/public/home/qianyj/virtualenv/dtk23.04/tf2.9/venv/lib/python3.8/site-packages/tensorflow/python/distribute/collective_all_reduce_strategy.py", line 329 in __init__
  File "/public/home/qianyj/virtualenv/dtk23.04/tf2.9/venv/lib/python3.8/site-packages/tensorflow/python/distribute/collective_all_reduce_strategy.py", line 185 in __init__
  File "/public/home/qianyj/virtualenv/dtk23.04/tf2.9/venv/lib/python3.8/site-packages/tensorflow/python/distribute/collective_all_reduce_strategy.py", line 252 in __init__
  File "/public/home/qianyj/virtualenv/dtk23.04/tf2.9/venv/lib/python3.8/site-packages/tensorflow/python/util/deprecation.py", line 357 in new_func
  File "/public/home/qianyj/TF_test/dtk23.04/tf2.9/models-2.9.2_fp32_4m/official/common/distribute_utils.py", line 155 in get_distribution_strategy
  File "/public/home/qianyj/TF_test/dtk23.04/tf2.9/models-2.9.2_fp32_4m/official/benchmark/models/resnet_imagenet_main.py", line 125 in run
  File "/public/home/qianyj/TF_test/dtk23.04/tf2.9/models-2.9.2_fp32_4m/official/benchmark/models/resnet_imagenet_main.py", line 362 in main
  File "/public/home/qianyj/.local/lib/python3.8/site-packages/absl/app.py", line 258 in _run_main
  File "/public/home/qianyj/.local/lib/python3.8/site-packages/absl/app.py", line 312 in run
  File "/public/home/qianyj/TF_test/dtk23.04/tf2.9/models-2.9.2_fp32_4m/official/benchmark/models/resnet_imagenet_main.py", line 369 in <module>
2023-05-25 11:10:58.273265: E tensorflow/core/common_runtime/base_collective_executor.cc:249] BaseCollectiveExecutor::StartAbort DEADLINE_EXCEEDED: Deadline Exceeded
Additional GRPC error information from remote target /job:worker/replica:0/task:0:
:{"created":"@1685013058.272930461","description":"Deadels-2.9.2_fp32_4m/official/common/distribute_utils.py", line 155 in get_distribution_strategy
  File "/public/home/qianyj/TF_test/dtk23.04/tf2.9/models-2.9.2_fp32_4m/official/benchmark/models/resnet_imagenet_main.py", line 125 in run
  File "/public/home/qianyj/TF_test/dtk23.04/tf2.9/models-2.9.2_fp32_4m/official/benchmark/models/resnet_imagenet_main.py", line 362 in main
  File "/public/home/qianyj/.local/lib/python3.8/site-packages/absl/app.py", line 258 in _run_main
  File "/public/home/qianyj/.local/lib/python3.8/site-packages/absl/app.py", line 312 in run
  File "/public/home/qianyj/TF_test/dtk23.04/tf2.9/models-2.9.2_fp32_4m/official/benchmark/models/resnet_imagenet_main.py", line 369 in <module>
2023-05-25 11:10:58.273265: E tensorflow/core/common_runtime/base_collective_executor.cc:249] BaseCollectiveExecutor::StartAbort DEADLINE_EXCEEDED: Deadline Exceeded
Additional GRPC error information from remote target /job:worker/replica:0/task:0:
:{"created":"@1685013058.272930461","description":"Deadline Exceeded","file":"external/com_github_grpc_grpc/src/core/ext/filters/deadline/deadline_filter.cc","file_line":69,"grpc_status":4}
2023-05-25 11:10:58.273404: E tensorflow/core/common_runtime/eager/context_distributed_manager.cc:737] Deadline Exceeded
Additional GRPC error information from remote target /job:worker/replica:0/task:0:
:{"created":"@1685013058.272930461","description":"Deadline Exceeded","file":"external/com_github_grpc_grpc/src/core/ext/filters/deadline/deadline_filter.cc","file_line":69,"grpc_status":4}
2023-05-25 11:10:58.273458: F tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:103] Non-OK-status: Stop() status: UNIMPLEMENTED: Clean shutdown is not currently implemented
Fatal Python error: Aborted

Current thread 0x00007fb79fed7740 (most recent call first):
  File "/public/home/qianyj/virtualenv/dtk23.04/tf2.9/venv/lib/python3.8/site-packages/tensorflow/python/eager/context.py", line 623 in ensure_initialized
  File "/public/home/qianyj/virtualenv/dtk23.04/tf2.9/venv/lib/python3.8/site-packages/tensorflow/python/distribute/collective_all_reduce_strategy.py", line 508 in _initialize_multi_worker
  File "/public/home/qianyj/virtualenv/dtk23.04/tf2.9/venv/lib/python3.8/site-packages/tensorflow/python/distribute/collective_all_reduce_strategy.py", line 341 in _initialize_strategy
  File "/public/home/qianyj/virtualenv/dtk23.04/tf2.9/venv/lib/python3.8/site-packages/tensorflow/python/distribute/collective_all_reduce_strategy.py", line 329 in __init__
  File "/public/home/qianyj/virtualenv/dtk23.04/tf2.9/venv/lib/python3.8/site-packages/tensorflow/python/distribute/collective_all_reduce_strategy.py", line 185 in __init__
  File "/public/home/qianyj/virtualenv/dtk23.04/tf2.9/venv/lib/python3.8/site-packages/tensorflow/python/distribute/collective_all_reduce_strategy.py", line 252 in __init__
  File "/public/home/qianyj/virtualenv/dtk23.04/tf2.9/venv/lib/python3.8/site-packages/tensorflow/python/util/deprecation.py", line 357 in new_func
  File "/public/home/qianyj/TF_test/dtk23.04/tf2.9/models-2.9.2_fp32_4m/official/common/distribute_utils.py", line 155 in get_distribution_strategy
  File "/public/home/qianyj/TF_test/dtk23.04/tf2.9/models-2.9.2_fp32_4m/official/benchmark/models/resnet_imagenet_main.py", line 125 in run
  File "/public/home/qianyj/TF_test/dtk23.04/tf2.9/models-2.9.2_fp32_4m/official/benchmark/models/resnet_imagenet_main.py", line 362 in main
  File "/public/home/qianyj/.local/lib/python3.8/site-packages/absl/app.py", line 258 in _run_main
  File "/public/home/qianyj/.local/lib/python3.8/site-packages/absl/app.py", line 312 in run
  File "/public/home/qianyj/TF_test/dtk23.04/tf2.9/models-2.9.2_fp32_4m/official/benchmark/models/resnet_imagenet_main.py", line 369 in <module>
2023-05-25 11:10:58.361393: E tensorflow/core/common_runtime/base_collective_executor.cc:249] BaseCollectiveExecutor::StartAbort DEADLINE_EXCEEDED: Deadline Exceeded
Additional GRPC error information from remote target /job:worker/replica:0/task:0:
:{"created":"@1685013058.361101419","description":"Deadline Exceeded","file":"external/com_github_grpc_grpc/src/core/ext/filters/deadline/deadline_filter.cc","file_line":69,"grpc_status":4}
2023-05-25 11:10:58.361543: E tensorflow/core/common_runtime/eager/context_distributed_manager.cc:737] Deadline Exceeded
Additional GRPC error information from remote target /job:worker/replica:0/task:0:
:{"created":"@1685013058.361101419","description":"Deadline Exceeded","file":"external/com_github_grpc_grpc/src/core/ext/filters/deadline/deadline_filter.cc","file_line":69,"grpc_status":4}
2023-05-25 11:10:58.361600: F tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:103] Non-OK-status: Stop() status: UNIMPLEMENTED: Clean shutdown is not currently implemented
Fatal Python error: Aborted

Current thread 0x00007fd5013c4740 (most recent call first):
  File "/public/home/qianyj/virtualenv/dtk23.04/tf2.9/venv/lib/python3.8/site-packages/tensorflow/python/eager/context.py", line 623 in ensure_initialized
  File "/public/home/qianyj/virtualenv/dtk23.04/tf2.9/venv/lib/python3.8/site-packages/tensorflow/python/distribute/collective_all_reduce_strategy.py", line 508 in _initialize_multi_worker
  File "/public/home/qianyj/virtualenv/dtk23.04/tf2.9/venv/lib/python3.8/site-packages/tensorflow/python/distribute/collective_all_reduce_strategy.py", line 341 in _initialize_strategy
  File "/public/home/qianyj/virtualenv/dtk23.04/tf2.9/venv/lib/python3.8/site-packages/tensorflow/python/distribute/collective_all_reduce_strategy.py", line 329 in __init__
  File "/public/home/qianyj/virtualenv/dtk23.04/tf2.9/venv/lib/python3.8/site-packages/tensorflow/python/distribute/collective_all_reduce_strategy.py", line 185 in __init__
  File "/public/home/qianyj/virtualenv/dtk23.04/tf2.9/venv/lib/python3.8/site-packages/tensorflow/python/distribute/collective_all_reduce_strategy.py", line 252 in __init__
  File "/public/home/qianyj/virtualenv/dtk23.04/tf2.9/venv/lib/python3.8/site-packages/tensorflow/python/util/deprecation.py", line 357 in new_func
  File "/public/home/qianyj/TF_test/dtk23.04/tf2.9/models-2.9.2_fp32_4m/official/common/distribute_utils.py", line 155 in get_distribution_strategy
  File "/public/home/qianyj/TF_test/dtk23.04/tf2.9/models-2.9.2_fp32_4m/official/benchmark/models/resnet_imagenet_main.py", line 125 in run
  File "/public/home/qianyj/TF_test/dtk23.04/tf2.9/models-2.9.2_fp32_4m/official/benchmark/models/resnet_imagenet_main.py", line 362 in main
  File "/public/home/qianyj/.local/lib/python3.8/site-packages/absl/app.py", line 258 in _run_main
  File "/public/home/qianyj/.local/lib/python3.8/site-packages/absl/app.py", line 312 in run Hi @south-ocean ,

Could you please confirm whether you are trying to use multiple systems/servers with each having single or more GPUs ? 
Are you using `MirroredStrategy` or `MultiWorkerMirroredStrategy` and I hope you know which to use when.

Please submit the steps followed and code you have used for enabling the strategy.

The problem seems to be related to communication between different workers. One probable reason might be mentioned below.

> 'TF_CONFIG' is parsed and TensorFlow's GRPC servers are started at the time you call tf.distribute.MultiWorkerMirroredStrategy. Therefore, you must set the 'TF_CONFIG' environment variable before you instantiate a tf.distribute.Strategy. To save time in this illustrative example, this is not demonstrated in this tutorial, so that servers do not need to start. You can find a full example in the last section of this tutorial.


Please refer some of the documentation tutorials for `MultiWorkerMirroredStrategy` which might be useful. [link1](https://www.tensorflow.org/tutorials/distribute/multi_worker_with_ctl) and [link2](https://www.tensorflow.org/tutorials/distribute/multi_worker_with_keras)

Thanks!
 I have also met the simarly issue.
When I use 3 workers and each with 8GPUs, it works fine. When 8 workers and each with 8 GPUs, it will also show this kind of error:  "BaseCollectiveExecutor::StartAbort DEADLINE_EXCEEDED: Collective has timed out during execution."

It seems to be related to the network bandwidth?
 it related to network settings， but I also solved it by accidental setting，I don't know how to debug this kind of problem about MirroredStrategy or MultiWorkerMirroredStrategy. Hi @south-ocean ,

Could you please confirm whether the issue resolved now. Also it would be helpful for community if you can able to share which settings helped you. 

Thanks! This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you. This issue was closed because it has been inactive for 7 days since being marked as stale. Please reopen if you'd like to work on this further. Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/60706">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/60706">No</a>

<details><summary>Click to expand!</summary> 
 
 ### Issue Type

Bug

### Have you reproduced the bug with TF nightly?

Yes

### Source

source

### Tensorflow Version

tf2.9，tf2.11

### Custom Code

Yes

### OS Platform and Distribution

cnetos7.9

### Mobile device

_No response_

### Python version

3.8.12

### Bazel version

5.3.0,5.0.0

### GCC/Compiler version

9.3

### CUDA/cuDNN version

_No response_

### GPU model and memory

_No response_

### Current Behaviour?

A bug happened!
The four-process four-card simulation based on the stand-alone four-card simulation is running, but I found that my program is stuck in the communication link of GRPC, and the program cannot continue to execute downward.

### Standalone code to reproduce the issue

```shell
resnet50 based on the https://github.com/tensorflow/models/tree/v2.11.0
```


### Relevant log output

```shell
Instructions for updating:
use distribute.MultiWorkerMirroredStrategy instead
W0525 11:06:39.106403 139892523235136 deprecation.py:350] From /public/home/qianyj/TF_test/dtk23.04/tf2.11/models-2.11.0_fp16_4m/official/common/distribute_utils.py:155: _CollectiveAllReduceStrategyExperimental.__init__ (from tensorflow.python.distribute.collective_all_reduce_strategy) is deprecated and will be removed in a future version.
Instructions for updating:
use distribute.MultiWorkerMirroredStrategy instead
2023-05-25 11:06:39.107776: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  SSE3 SSE4.1 SSE4.2 AVX AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-05-25 11:06:39.107986: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  SSE3 SSE4.1 SSE4.2 AVX AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-05-25 11:06:39.109601: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1613] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 15601 MB memory:  -> device: 0, name: Vega 20, pci bus id: 0000:43:00.0
2023-05-25 11:06:39.109709: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1613] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 15601 MB memory:  -> device: 0, name: Vega 20, pci bus id: 0000:04:00.0
2023-05-25 11:06:40.104680: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1613] Created device /job:worker/replica:0/task:1/device:GPU:0 with 15601 MB memory:  -> device: 0, name: Vega 20, pci bus id: 0000:26:00.0
2023-05-25 11:06:40.104734: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1613] Created device /job:worker/replica:0/task:2/device:GPU:0 with 15601 MB memory:  -> device: 0, name: Vega 20, pci bus id: 0000:43:00.0
2023-05-25 11:06:40.105277: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1613] Created device /job:worker/replica:0/task:0/device:GPU:0 with 15601 MB memory:  -> device: 0, name: Vega 20, pci bus id: 0000:04:00.0
2023-05-25 11:06:40.105313: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1613] Created device /job:worker/replica:0/task:3/device:GPU:0 with 15601 MB memory:  -> device: 0, name: Vega 20, pci bus id: 0000:63:00.0
2023-05-25 11:06:40.114474: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:447] Started server with target: grpc://localhost:40013
2023-05-25 11:06:40.115146: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:447] Started server with target: grpc://localhost:40014
2023-05-25 11:06:40.115326: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:447] Started server with target: grpc://localhost:40015
2023-05-25 11:06:40.115823: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:447] Started server with target: grpc://localhost:40016
```
</details>