Allow merging compute-copy streams
Thanks for the review, Changhui! Improved the coding style according to your advice. Hi @changhuilin Can you please review this PR ? Thank you! Hi @changhuilin , what's our next step on this PR? Thank you! Hi @changhuilin Can you please review this PR ? Thank you! Hi @changhuilin Can you please review this PR ? Thank you! Hi @buptzyb Can you please resolve conflicts? Thank you! This PR is stale because it has been open for 14 days with no activity. It will be closed if no further activity occurs. Thank you. This change looks good to me.
- Can you try to merge this to the head? - sorry that it takes long to get this done  
- Can you also expand the PR description a bit, e.g., add some experiment number to show how much improvement you observed?  @changhuilin  Thank you for taking care of this PR! Updated the description and merged to the head. @changhuilin So, what's our next move on this? @gbaned Can you help to merge? I noticed there are some check failures, and some status waiting. Which ones are real and should be addressed? Thanks! > @gbaned Can you help to merge? I noticed there are some check failures, and some status waiting. Which ones are real and should be addressed? Thanks!

Hi @changhuilin Sure thing, please click on the import/copybara details link, you can see the corresponding CL. It require runtime-team approval and presubmit checks are failing. Please take a look on it. Thank you! > Hi @changhuilin Sure thing, please click on the import/copybara details link, you can see the corresponding CL. It require runtime-team approval and presubmit checks are failing. Please take a look on it. Thank you!

Thanks, @gbaned. I got the CL - will fix the presubmit failures. I can approve on behalf of runtime-team.
This PR works as a part of the whole Multi-Stream feature in TF, which is proposed in #61185.

Allow merging the host_to_device/device_to_host/device_to_device data copy streams into the compute stream in one stream group. This is useful to reduce the overhead caused by GPU stream synchronization, especially when data transfers are frequent. Another benefit is, for host_to_device copy, merging streams allows early scheduling of subsequent ops, doesn't have to wait until the data copy is really finished.

As a part of the multi-stream feature, it can help multi-stream reach a much higher throughput. Taking our proto models as an example, the original model inference throughput is **1524** samples/second, and **2229** samples/ second with multi-stream, and **2471** samples/second further with stream-merging.

However, stream-merging can also be used separately. We got inference throughput gain from **1028** samples/second to **1187** samples/second by enabling stream-merging.

Please refer to the 'Performance' part in our [document](https://docs.google.com/document/d/1yL3lWk_iFKqLTyekkuaiKXZ78I0lPmD5kM1fghHRs4Y/edit?usp=sharing) for detailed and more experiment results.