Training speed and GPU utilization 50% lower after upgrading from 2.9 to 2.12
@ShuaiShao93 
Could you please provide complete code or colab gist to replicate the issue reported here ?

Thank You! Check if the code changes are causing the issue: If you have made changes to your custom code while upgrading TensorFlow, it might be causing the issue. Try to isolate the TensorFlow upgrade by running the old code with the new TensorFlow version and check if the issue persists.

Check TensorFlow configurations: Verify that you have the correct TensorFlow configurations for your system. Make sure that you have installed the correct CUDA and cuDNN versions that are compatible with TensorFlow 2.12. Also, ensure that the TensorFlow version you are using is built with the right configurations.

Check multi-worker setup: Ensure that the multi-worker setup is properly configured. Make sure that the communication between the workers is working correctly and that the resources are distributed evenly.

Check for bottlenecks: Check if there are any bottlenecks in the code. Use profiling tools to identify any slow operations or data transfer between the workers that might be causing the issue.

Upgrade to the latest version: Try upgrading to the latest version of TensorFlow, as newer versions might have addressed the issue.

In summary, the issue might be related to the custom code changes, TensorFlow configurations, multi-worker setup, bottlenecks, or outdated TensorFlow version. You can try the suggestions above to debug the issue. Sorry, the description was wrong. This is also reproducible on single worker.

The only change on the code is replacing tf.keras.optimizers with tf.keras.optimizers.legacy.

By profiling it, I found there is a huge gap on GPU between the steps, and below is what it does on host in the gap. Basically it's `EagerLocalExecute:WriteSummary`?
<img width="526" alt="gap" src="https://user-images.githubusercontent.com/7556010/233800388-89e919db-b9b8-452c-a6d8-153bf2ba1a0d.png">

I added more logging in WriteSummaryOp, and it showed that it took 900ms to write a single float scalar.
```
step 6
tag batch_loss
duration 882ms
tensor Tensor<type: float shape: [] values: 0.0100328028>
```

By raising max_queue in `tf.summary.create_file_writer`, the training speed is fixed, but I think this is still probably a bug that `SummaryFileWriter::InternalFlush` is too slow. Hi @ShuaiShao93 ,

Thanks for reporting this. By looking at template I am assuming you have used Build from Source package and  bazel version mentioned is 0.27.1. Please confirm whether the info is correct. If correct right now tested bazel version is 5.3.0 for Tf 2.12 versions. You can find tested configurations [here](https://www.tensorflow.org/install/source#gpu).

Please find and do necessary upgrades applicable and test the code and please confirm the results. Also please submit minimal code snippet to replicate the issue. Thanks! I just verified this is even reproducible on pip installed TensorFlow, so I believe it has nothing to do with the bazel version. @ShuaiShao93 ,

Could you please share the reproducible code for this behaviour. Also is the issue is only reproducible with legacy optimizers? Please confirm. > @ShuaiShao93 ,
> 
> Could you please share the reproducible code for this behaviour. Also is the issue is only reproducible with legacy optimizers? Please confirm.

I'm happy to make a repro on colab, but for some reason [this profiler](https://colab.research.google.com/github/tensorflow/tensorboard/blob/master/docs/tensorboard_profiling_keras.ipynb) demo on github doesn't work. 
<img width="1356" alt="image" src="https://user-images.githubusercontent.com/7556010/234989348-907ce95f-c720-4967-a555-2aa0740a9b17.png">

If you can help fix it, I'm happy to provide a repro with it. > I added more logging in WriteSummaryOp, and it showed that it took 900ms to write a single float scalar.

> By raising max_queue in `tf.summary.create_file_writer`, the training speed is fixed, but I think this is still probably a bug that `SummaryFileWriter::InternalFlush` is too slow.

It looks like this is the culprit, and so I doubt this is GPU related specically. @poulsbo, do you know if there have been any changes to the TF summary ops that would cause such a big regression?
<details><summary>Click to expand!</summary> 
 
 ### Issue Type

Performance

### Have you reproduced the bug with TF nightly?

No

### Source

source

### Tensorflow Version

2.12.0

### Custom Code

Yes

### OS Platform and Distribution

Ubuntu 20.04

### Mobile device

_No response_

### Python version

3.8

### Bazel version

0.27.1

### GCC/Compiler version

_No response_

### CUDA/cuDNN version

11.8/8.6

### GPU model and memory

A100

### Current Behaviour?

After upgrading TF from 2.9 to 2.12, when training our point segmentation model, the GPU utilization dropped from 70% to 30%, and the training speed is 50% slower. Is there any suggestion why this may happen? And what's the best way to debug it?

### Standalone code to reproduce the issue

```shell
N/A
```


### Relevant log output

_No response_</details>