PR #10316: [NVIDIA GPU] Add a mechanism in GpuAsyncTracker to force delaying scheduling of an instruction.
PR #10316: [NVIDIA GPU] Add a mechanism in GpuAsyncTracker to force delaying scheduling of an instruction.

Imported from GitHub PR https://github.com/openxla/xla/pull/10316

latency hiding scheduler has a property for each HloScheduleNode called SetForceDelay, setting this to true for a node will result in forcing LHS to ignore all cost info and schedule the instruction to the very beginning of the instruction sequence.
This is useful in manually enforcing an instruction to run before anything else in the scheduled graph.
Passes can set `should_force_delay` attribute in the GpuBackendConfig to instruct how this instruction will be scheduled. It's set to false by default.
An example usage:
```
original sequence:

allreduce-start
allreduce-done
add0, backend_config={should_force_delay=true}
add1

schedule after LHS:
add0, backend_config={should_force_delay=true}
allreduce-start
add1
allreduce-done
```
Copybara import of the project:

--
74267e60142c5161310580db5b760a3fe6d4f625 by TJ <tjx@nvidia.com>:

Add a mechanism in GpuAsyncTracker to force delaying scheduling
of an instruction.

--
0aac4cf8c42f2ca210450581f05239378d8a33c2 by TJ <tjx@nvidia.com>:

Fix failing backend config test

--
4deab8393ecb01ca010e0cf4ea30930ac448b815 by TJ <tjx@nvidia.com>:

Use force delay for windowed einsum loops

--
a1321e88fd336f83a4fcfcc89037b8d4a3d1903f by TJ <tjx@nvidia.com>:

Revert changes in gpu_windowed_einsum_handler, moving them to a separate
pr

--
86ffb97db8ce167438562f997316033b61316003 by TJ Xu <tjx@nvidia.com>:

Fix internal linter error

Merging this change closes #10316

FUTURE_COPYBARA_INTEGRATE_REVIEW=https://github.com/openxla/xla/pull/10316 from Tixxx:tixxx/force_delay_lhs 86ffb97db8ce167438562f997316033b61316003
