Reland PR #9757: GpuTimer: improve kernel execution time measurement accuracy
Reland PR #9757: GpuTimer: improve kernel execution time measurement accuracy

Imported from GitHub PR https://github.com/openxla/xla/pull/9757

This PR changes how `GpuTimer` measures execution time, which should improve the measurement accuracy.
Quoting my comment in the code:
```c++
// When a timer is created it launches a delay kernel into the given stream and
// queues a start event immediately afterwards. This delay kernel blocks
// execution on the stream until GetElapsedDuration() is called, at which point
// an end event is queued and the delay kernel exits. This allows the device
// execution time of the tasks queued to the stream while the timer is active
// to be measured more accurately.
```
this should improve the accuracy of the measurements that are used to make auto-tuning decisions, especially for small kernels.

There are a couple of edge cases that have special treatment:
- if `CUDA_LAUNCH_BLOCKING=1` then the delay kernel will not achieve anything, so we don't launch it
- if any code in the timed region synchronises the device then the host will wait for the delay kernel to time out before actually launching the kernels to be measured. This will lead to a poor quality measurement, and an error message is printed. This condition can be met if one of the kernels being measured is lazily loaded, as lazy loading can trigger synchronisation. Best practice is to execute a warmup run (without timing enabled) before the timed execution.

Two parts of the autotuning code are updated to follow this best practice.

The `GpuTimer::Create*` signatures that were deprecated in https://github.com/openxla/xla/pull/9841 do not benefit from these accuracy improvements.

cc: @sergachev @nouiz
Copybara import of the project:

--
2b834f7154158ddc7294d45bb5481b8c38efdc61 by Olli Lupton <olupton@nvidia.com>:

GpuTimer: use delay kernel to improve accuracy

This ensures that all the device operations to be timed are queued to
the relevant stream before any of them are executed, resulting in a more
accurate measurement. This is skipped if CUDA_LAUNCH_BLOCKING=1 or
unified addressing is not available.

In addition, make sure that auto-tuning code paths have warm-up
executions of the kernels being measured. The warm-up executions should
not be inside GpuTimer regions.

With the example HLO:
  parameter_0 = bf16[128,12,128]{2,1,0} parameter(0)
  parameter_1 = bf16[12,128,128]{2,1,0} parameter(1)
  ROOT dot.26 = bf16[12,128,128]{2,1,0} dot(parameter_0, parameter_1),
    lhs_batch_dims={1}, lhs_contracting_dims={0}, rhs_batch_dims={0},
    rhs_contracting_dims={2}
a cuBLAS kernel was selected with a CUPTI/NSys-measured runtime of
3.9µs and before this change, GpuTimer measured a runtime of 17.5µs.
With this change, GpuTimer measures a runtime of 8.1µs.

With the example HLO:
  p0 = bf16[128,1536]{0,1} parameter(0)
  p1 = s8[1536,12288]{0,1} parameter(1)
  c = bf16[1536,12288]{0,1} convert(p1)
  ROOT d = bf16[128,12288]{1,0} dot(p0, c), lhs_contracting_dims={1},
    rhs_contracting_dims={0}
five cuDNN plans were auto-tuned, with CUPTI/Nsys-measured runtimes of
{34.9, 18.5, 30.9, 32.4, 31.5}µs. Before this change, GpuTimer measured
runtimes of {76.6, 40.0, 52.5, 55.7, 51.8}µs. With this change, GpuTimer
measures runtimes of {39.6, 23.0, 35.4, 35.1, 35.1}µs.

In summary, with this change, GpuTimer gives results that are much
closer to the CUPTI/Nsys measurements, with a ~uniform offset of ~4µs. A
constant offset doesn't matter for auto-tuning.

--
8f5ce0e7e309a46bf69c369bc4b1da8c26fd1579 by Olli Lupton <olupton@nvidia.com>:

Address CR

Merging this change closes #9757

FUTURE_COPYBARA_INTEGRATE_REVIEW=https://github.com/openxla/xla/pull/9757 from olupton:gpu-timer-improvement 8f5ce0e7e309a46bf69c369bc4b1da8c26fd1579
