tf performance deteriorated very sharply when qps arrived a level
Hi @xiedeacc ,

Thanks for reporting. I am unable to quiet follow through your issue. Actually AFAIK there is no such benchmark to compare the qps and what the average qps. I am not quite sure whether the issue is due to network overload or something. Is that possible that can we replicate this behaviour with TF code snippet ? With the provided information I can't infer much.

Also could you explain what is the value 99% represents here. Is that CPU usage? And the second one is average time for a query right? 

Please submit more context along with some code snippet to replicate the behaviour.

Thanks!

 99% means  99% request return at that latency. under stress test, cpu olny achive 50%, cannot achive highier. after a lot perf, it seems too many memory copy, I use jemalloc relief this problem a lot. maybe I need split ads into many batches to avoid big memory allocation and copy Hi, 

Could you please test your findings using latest Tensorflow stable version 2.12,

 `2.13rc0` using `pip install tensorflow==2.13.0rc0` ,

Tensorflow  Nightly version using `pip install tf-nightly`

 to see if there is any improvement in qps. Thank you. This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you. > Hi,
> 
> Could you please test your findings using latest Tensorflow stable version 2.12,
> 
> `2.13rc0` using `pip install tensorflow==2.13.0rc0` ,
> 
> Tensorflow Nightly version using `pip install tf-nightly`
> 
> to see if there is any improvement in qps. Thank you.

ok  I will try, thanks This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you. This issue was closed because it has been inactive for 7 days since being marked as stale. Please reopen if you'd like to work on this further. Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/60601">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/60601">No</a>
 I run into this same issue on tf2.9.0 rc2 built from source, would you share the test result? @xiedeacc 
### Source

source

### Tensorflow Version

tf 2.8.4


### OS Platform and Distribution

ubuntu18.04


### Python version

3.10

### Bazel version

5.2.0

### GCC/Compiler version

gcc-11.3.0



### Current Behaviour?

our project use tf2.8.4. we meet two problem, 
1. performance deteriorated very sharply, does this normally?
50q/s    35ms 99%
100q/s   62ms 99%
120q/s   75ms  99%
130q/s   100ms 99%
140q/s   150ms 99%
150q/s   370ms 99%

2. qps cannot archive to expected level, our cpu model is Intel(R) Xeon(R) Gold 5118 CPU @ 2.30GHz, two cpu, so we have 24 physical cores. assume each request cost 100ms,  qps at least can archive 1000 / 100 * 24 = 240qps,  but permance is so bad when qps archive 150.

ps. our model was fm with 4000000 * 6 matrix