Why training speed per step decrease when number of workers increase in distributed training on multiple cpus?
Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/60244">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/60244">No</a>

<details><summary>Click to expand!</summary> 
 
 ### Issue Type

Performance

### Have you reproduced the bug with TF nightly?

Yes

### Source

binary

### Tensorflow Version

2.9.1

### Custom Code

Yes

### OS Platform and Distribution

Linux Ubuntu 16.04

### Mobile device

None

### Python version

3.7

### Bazel version

None

### GCC/Compiler version

None

### CUDA/cuDNN version

None

### GPU model and memory

None

### Current Behaviour?

```shell
When I follow the tutorial, [Distrited training with Keras](https://www.tensorflow.org/tutorials/distribute/keras), using CPU to do training. I found that the speed of each step(given the batch size of each worker is constant) decrease when I increase the number of workers. I know there will be communication cost when the number of workers is increasing, but I run it in the same physical node and the speed drop sharply. I expect that the speed of each step should be similar or just decrease within a small proportion when increasing the workers.
```


### Standalone code to reproduce the issue

```shell
Just follow the tutorial in colab: https://colab.research.google.com/github/tensorflow/docs/blob/master/site/en/tutorials/distribute/keras.ipynb
and increase the number of workers.
```


### Relevant log output

_No response_</details>