PR #61632: Allow merging compute-copy streams
PR #61632: Allow merging compute-copy streams

Imported from GitHub PR https://github.com/tensorflow/tensorflow/pull/61632

This PR works as a part of the whole Multi-Stream feature in TF, which is proposed in #61185.

Allow merging the host_to_device/device_to_host/device_to_device data copy streams into the compute stream in one stream group. This is useful to reduce the overhead caused by GPU stream synchronization, especially when data transfers are frequent. Another benefit is, for host_to_device copy, merging streams allows early scheduling of subsequent ops, doesn't have to wait until the data copy is really finished.

As a part of the multi-stream feature, it can help multi-stream reach a much higher throughput. Taking our proto models as an example, the original model inference throughput is **1524** samples/second, and **2229** samples/ second with multi-stream, and **2471** samples/second further with stream-merging.

However, stream-merging can also be used separately. We got inference throughput gain from **1028** samples/second to **1187** samples/second by enabling stream-merging.

Please refer to the 'Performance' part in our [document](https://docs.google.com/document/d/1yL3lWk_iFKqLTyekkuaiKXZ78I0lPmD5kM1fghHRs4Y/edit?usp=sharing) for detailed and more experiment results.
Copybara import of the project:

--
9e51f38c3a4e31de89be520c4bc5315abf30cd4b by Robin Zhang <robinz@nvidia.com>:

Allow merging compute-copy streams

--
a45967f67b026fc9f6b314e4e6581578da430629 by Robin Zhang <robinz@nvidia.com>:

Improve coding style

--
ccae79bb3156219c1bf7010d146194ae38887806 by Robin Zhang <robinz@nvidia.com>:

Rename stream_merge_options_

--
332e1fef23cd09b0972b7b6d7520bf5507afbda6 by Robin Zhang <robinz@nvidia.com>:

Put stream checking out of callback

--
4a0c7892f9ed9ac110af63ac6832d813743b7cc7 by Robin Zhang <robinz@nvidia.com>:

Move StreamMergeOptions to Experimental

--
efe56d7143c2b11f87e478cabeab894e264b4314 by Robin Zhang <robinz@nvidia.com>:

add some comments


Merging this change closes #61632

Reverts changelist 525613555

FUTURE_COPYBARA_INTEGRATE_REVIEW=https://github.com/tensorflow/tensorflow/pull/61632 from buptzyb:multistream-streammerge 5aabb58431e56868c1f462ec3851785132d4cd47
