Why training speed per step decrease when number of workers increase in distributed training on CPUs?
Hi, @tiruk007, do you have any comment on this? Or do you need more information? When you increase the number of workers in distributed training on CPUs in TensorFlow, the training speed per step may decrease due to  something called communication overhead.

Basically in distributed training, the workers need to communicate frequently with each other to exchange stuff like weights and other info. This communication involves sending data over the network, which can be slow and expensive compared to local computations on a single machine. As you add more workers, the amount of communication increases, leading to higher overhead.

Furthermore, the communication overhead can become a bottleneck that limits the amount of work each worker can perform per step. This can lead to a situation where the workers spend more time waiting for each other to finish their tasks, rather than actually performing computations. As a result, the training speed per step may decrease as you add more workers.

To get rid of this issue, you can try to optimize the communication between workers, for example, by reducing the amount of data that needs to be exchanged or by using more efficient communication protocols. The other way would be to use GPUs or TPUs that basically can compute tasks faster than a typical CPU. Hi @pogrushan,  thanks a lot for your reply. I wonder if there is any benchmark efficiency performance test results on CPUs for us to refer to? Hi @lalalapotter ,

Thanks for reaching us. Yes, with increase in no of workers there is some performance affect due to cross communications. AFAIK I am not aware of any benchmarks especially for CPU. I will update if find any such results.

Since you mentioned that you are using same physical node can you try [tf.distribute.MirroredStrategy](https://www.tensorflow.org/api_docs/python/tf/distribute/MirroredStrategy) and confirm the outcome. Thanks!
 Hi @SuryanarayanaY,

Thanks for answering the question : ). Actually I use the [tf.distribute.MultiWorkerMirroredStrategy](https://www.tensorflow.org/api_docs/python/tf/distribute/MultiWorkerMirroredStrategy) same as the [keras example](https://www.tensorflow.org/tutorials/distribute/multi_worker_with_keras) and trained on multiple nodes(2 workers per node). Following is the test results on my cluster, could you please check if the scalability is normal for TensoFlow CPU training. 

|# of worker | time per step(ms)|
-- | --
2 | 62
3 | 76
4 | 87
5 | 94
6 | 98
7 | 110
8 | 116
9 | 121
10 | 124

Besides, do you know any environment variable or other methods that can improve the scalability(training speed per step)? Update:
I found that if I use `MirroredStrategy` rather than `MultiWorkerMirroredStrategy`, the training speed per step do not decrease that much when worker number increases. So what is the difference between `MirroredStrategy` and `MultiWorkerMirroredStrategy`? Hi @lalalapotter ,

`MirroredStrategy` to be used when all the devices are in same physical machines. Whereas `MultiWorkerMirroredStrategy` to be used when we want to use Devices/accelerators that spread across different physical machines/servers. Hence you observed the difference in performance as you are using single physical server.

If the issue resolved can we mark it as closed.Please feel free to close the issue if resolved for you. Thanks! Hi @SuryanarayanaY,
Thanks for your quick reply! Actually I have tried run distributed training on different physical nodes with `MirroredStrategy` and get better performance than `MultiWorkerStrategy`. I wonder if there is any problem to run distributed training on different physical nodes with `MirroredStrategy`? FYI, my test code are as follows:

```python
import os
import json

import tensorflow as tf
from tensorflow.python.distribute import collective_util

import mnist_setup


per_worker_batch_size = 64
tf_config = {
    'cluster': {
        'worker': ['node-001:12345', 'node-002:23456']
    },
    'task': {'type': 'worker', 'index': 0}
}
os.environ['TF_CONFIG'] = json.dumps(tf_config)
tf_config = json.loads(os.environ['TF_CONFIG'])
num_workers = len(tf_config['cluster']['worker'])

strategy = tf.distribute.MirroredStrategy()

global_batch_size = per_worker_batch_size * num_workers
multi_worker_dataset = mnist_setup.mnist_dataset(global_batch_size)

with strategy.scope():
  # Model building/compiling need to be within `strategy.scope()`.
  multi_worker_model = mnist_setup.build_and_compile_cnn_model()

print(multi_worker_model.summary())
multi_worker_model.fit(multi_worker_dataset, epochs=30)

```
 Hi @lalalapotter ,

For training on different nodes there will be some additional communication overload compared to training within same machine. Hence there are two different strategies for both. The way MirroredStrategy designed not suitable for Multiple physical devices. Both the implementations are different and specific to their use case. Thanks! Hi @SuryanarayanaY,
Thanks for your reply. I wonder if there is any benchmark test result on efficiency for me to compare with? And if there is any method to optimize the efficiency while using MultiWorkerMirroredStrategy and ring all reduce(cannot use NCCL) algorithm. @lalalapotter ,

AFAIK, There is no such benchmarks for comparing. As per our documentation NCCL is the best performant for GPU but wont work for CPUs. 

Please refer to the argument `communication_options` from `[MultiWorkerMirroredStrategy](https://www.tensorflow.org/api_docs/python/tf/distribute/MultiWorkerMirroredStrategy)` which have options to choose like AUTO,RING, NCCL and refer this doc [link](https://www.tensorflow.org/api_docs/python/tf/distribute/experimental/CommunicationOptions) for more details.

Thanks! This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you. This issue was closed because it has been inactive for 7 days since being marked as stale. Please reopen if you'd like to work on this further. Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/60245">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/60245">No</a>

<details><summary>Click to expand!</summary> 
 
 ### Issue Type

Performance

### Have you reproduced the bug with TF nightly?

Yes

### Source

binary

### Tensorflow Version

2.9.1

### Custom Code

Yes

### OS Platform and Distribution

Linux Ubuntu 16.04

### Mobile device

None

### Python version

3.7

### Bazel version

None

### GCC/Compiler version

None

### CUDA/cuDNN version

None

### GPU model and memory

None

### Current Behaviour?

When I follow the tutorial, [Multi-worker training with Keras](https://www.tensorflow.org/tutorials/distribute/multi_worker_with_keras), using CPUs to do training. I found that the speed of each step(given the batch size of each worker is constant) decrease when I increase the number of workers. I know there will be communication cost when the number of workers is increasing, but I run it in the same physical node and the speed drop sharply. I expect that the speed of each step should be similar or just decrease within a small proportion when increasing the workers. Or could you please give an explanation?


### Standalone code to reproduce the issue

Just follow the tutorial in colab: [link](https://colab.research.google.com/github/tensorflow/docs/blob/master/site/en/tutorials/distribute/multi_worker_with_keras.ipynb)
and increase the number of workers.


### Relevant log output

_No response_</details>