tf.distribute.MultiWorkerMirroredStrategy: Program hangs when connecting with gRPC.
I think it is related to the hostlist expansion, fix here #58033, because your nodelist is:

```
node[01-04]
```
But Tensorflow got:

```
node1, node2, node3 and node4.
```

Instead of:

```
node01, node02, node03 and node04.
```
 > I think it is related to the hostlist expansion, fix here #58033, because your nodelist is:
> 
> ```
> node[01-04]
> ```
> 
> But Tensorflow got:
> 
> ```
> node1, node2, node3 and node4.
> ```
> 
> Instead of:
> 
> ```
> node01, node02, node03 and node04.
> ```

Good find! That was indeed an issue for me, so updated the file on each device for tensorflow/python/distribute/cluster_resolver/slurm_cluster_resolver.py, but the process still hangs. 

new output:

```shell
2023-02-06 03:16:26.130945: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:447] Started server with target: grpc://node01:15000
2023-02-06 03:16:26.183769: I tensorflow/core/distributed_runtime/coordination/coordination_service.cc:502] /job:worker/replica:0/task:0 has connected to coordination service. Incarnation: 4221482841861972805
2023-02-06 03:16:26.188286: I tensorflow/core/distributed_runtime/coordination/coordination_service_agent.cc:277] Coordination agent has successfully connected.
2023-02-06 03:16:39.871488: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:447] Started server with target: grpc://node02:15000
2023-02-06 03:16:40.414216: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:447] Started server with target: grpc://node04:15000
2023-02-06 03:16:45.344939: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:447] Started server with target: grpc://node03:15000
``` Fixed the issue. All host config files except the master on my machines were managed by a service, so on reboot the configurations for ip-hostname associations were removed. Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/59573">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/59573">No</a>

<details><summary>Click to expand!</summary> 
 
 ### Issue Type

Bug

### Have you reproduced the bug with TF nightly?

No

### Source

binary

### Tensorflow Version

tensorflow-aarch64 2.11.0

### Custom Code

No

### OS Platform and Distribution

Ubuntu 22.10

### Mobile device

_No response_

### Python version

Python 3.10.8

### Bazel version

_No response_

### GCC/Compiler version

_No response_

### CUDA/cuDNN version

_No response_

### GPU model and memory

_No response_

### Current Behaviour?

```shell
The script hangs when defining strategy, and not just for MultiWorkerMirrored, but any of the strategies from tf.distribute.
```


### Standalone code to reproduce the issue

```shell
def train_dense_model(batch_size):
    # limit imports oustide the call to the function, in order to launch quickly
    # when using dask
    import tensorflow as tf
    from tensorflow import keras
    from tensorflow.keras import layers
    import json
    # model building
    tf.keras.backend.clear_session()  # For easy reset of notebook state.

    slurm_resolver = tf.distribute.cluster_resolver.SlurmClusterResolver(port_base=15000)
    
    tf_config = json.dumps({'cluster' : slurm_resolver.cluster_spec().as_dict()})
    os.environ['TF_CONFIG'] = tf_config
    
    communication_options = tf.distribute.experimental.CommunicationOptions(bytes_per_pack=50 * 1024 * 1024,
                                                              timeout_seconds=120.0,
                                                              implementation=tf.distribute.experimental.CommunicationImplementation.RING
    )
    
    mirrored_strategy = tf.distribute.MultiWorkerMirroredStrategy(cluster_resolver=slurm_resolver, 
                                                                  communication_options=communication_options)
```


### Relevant log output

```shell
2023-02-05 19:36:27.162088: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:447] Started server with target: grpc://node1:15000
2023-02-05 19:36:38.280982: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:447] Started server with target: grpc://node2:15000
2023-02-05 19:36:38.691138: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:447] Started server with target: grpc://node4:15000
2023-02-05 19:36:40.504698: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:447] Started server with target: grpc://node3:15000
```
### Accompanying script for "sbatch script.sh":

```shell
#!/bin/bash
#SBATCH --job-name=mnist_tf_distributed     # job name
#SBATCH --nodelist=node[01-04]              # number of nodes
#SBATCH --ntasks-per-node=1                 # number of MPI task per node
#SBATCH --cpus-per-task=4                   # since nodes have 4 cpus
#SBATCH --distribution=block:block          # distribution, might be better to have contiguous blocks
#SBATCH --time=00:10:00                     # job length
#SBATCH --exclusive                         # we reserve the entire node for our job
#SBATCH --output=mnist_tf_distr_log_%j.out  # std out
#SBATCH --error=mnist_tf_distr_log_%j.out   # std err

unset http_proxy https_proxy HTTP_PROXY HTTPS_PROXY

set -x
cd ${SLURM_SUBMIT_DIR}

srun --nodelist=node[01-04] python ./mnist_example.py
```
</details>
