//tensorflow/compiler/xla/service/gpu:fusion_merger_test fails on AARCH64
@nSircombe @cfRod FYI Digging into this a bit.
It fails on AARCH64 because  of "Fused would run slower than unfused!" which seems to be because the exec time gets calculated as 'inf'. Tracing back this comes as a result of the utilization by the consumer being 0.
The difference with x86 is that although it also has 0 utilization by the consumer, the exec time comes out as '-inf'.
So I think that both x86 and AARCH64 should fail and x86 only works by accident.
But the question I have yet to answer is why the utilization by the consumer is 0.
### Issue type

Bug

### Have you reproduced the bug with TensorFlow Nightly?

Yes

### Source

source

### TensorFlow version

git HEAD

### Custom code

No

### OS platform and distribution

Ubuntu 20.04

### Mobile device

n/a

### Python version

3.9.17

### Bazel version

6.1.0

### GCC/compiler version

10.2.1

### CUDA/cuDNN version

n/a

### GPU model and memory

n/a

### Current behavior?

//tensorflow/compiler/xla/service/gpu:fusion_merger_test unit test fails when run on AARCH64 machine.

### Standalone code to reproduce the issue

```shell
bazel test --cache_test_results=no --config=mkl_aarch64_threadpool --jobs=75 --copt=-flax-vector-conversions --test_env=TF_ENABLE_ONEDNN_OPTS=1 --test_env=TF2_BEHAVIOR=1 --define=tf_api_version=2 --test_output=errors --verbose_failures=true --test_keep_going --notest_verbose_timeout_warnings --build_tests_only -- //tensorflow/compiler/xla/service/gpu:fusion_merger_test
```


### Relevant log output

```shell
FAIL: //tensorflow/compiler/xla/service/gpu:fusion_merger_test (see /home/andrew/src/tf_test/tensorflow-git/bazel-ci_build-cache/.cache/bazel/_bazel_andrew/eab0d61a99b6696edb3d2aff87b585e8/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/service/gpu/fusion_merger_test/test.log)
INFO: From Testing //tensorflow/compiler/xla/service/gpu:fusion_merger_test:
==================== Test output for //tensorflow/compiler/xla/service/gpu:fusion_merger_test:
[==========] Running 21 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 21 tests from FusionMergerTest
[ RUN      ] FusionMergerTest.MergeSharedFusionInstruction
[       OK ] FusionMergerTest.MergeSharedFusionInstruction (9 ms)
[ RUN      ] FusionMergerTest.MoreMemoryAccessIfFused
[       OK ] FusionMergerTest.MoreMemoryAccessIfFused (1 ms)
[ RUN      ] FusionMergerTest.LessMemoryAccessIfFused
[       OK ] FusionMergerTest.LessMemoryAccessIfFused (1 ms)
[ RUN      ] FusionMergerTest.WillMergeIntoInputFusion
[       OK ] FusionMergerTest.WillMergeIntoInputFusion (1 ms)
[ RUN      ] FusionMergerTest.WillMergeIntoUnfusedConsumer
[       OK ] FusionMergerTest.WillMergeIntoUnfusedConsumer (1 ms)
[ RUN      ] FusionMergerTest.WillNotMergeReduceUnfriendlyLayouts
[       OK ] FusionMergerTest.WillNotMergeReduceUnfriendlyLayouts (0 ms)
[ RUN      ] FusionMergerTest.WillMergeReduceNotTooUnfriendlyLayouts
[       OK ] FusionMergerTest.WillMergeReduceNotTooUnfriendlyLayouts (1 ms)
[ RUN      ] FusionMergerTest.AvoidsLargeFusion
[       OK ] FusionMergerTest.AvoidsLargeFusion (1 ms)
[ RUN      ] FusionMergerTest.WillNotMergeIfFusionEmitterIsInefficient
[       OK ] FusionMergerTest.WillNotMergeIfFusionEmitterIsInefficient (1 ms)
[ RUN      ] FusionMergerTest.WillMergeSliceIntoReusingConsumer
[       OK ] FusionMergerTest.WillMergeSliceIntoReusingConsumer (0 ms)
[ RUN      ] FusionMergerTest.WillMergeExpensiveFusionsIfSavesMemory
[       OK ] FusionMergerTest.WillMergeExpensiveFusionsIfSavesMemory (1 ms)
[ RUN      ] FusionMergerTest.WillMergeExpensiveFusionsWithSingleConsumer
[       OK ] FusionMergerTest.WillMergeExpensiveFusionsWithSingleConsumer (0 ms)
[ RUN      ] FusionMergerTest.WillNotMergeExpensiveFusionsWithReusingConsumer
[       OK ] FusionMergerTest.WillNotMergeExpensiveFusionsWithReusingConsumer (0 ms)
[ RUN      ] FusionMergerTest.NoMergeWithBitcast
[       OK ] FusionMergerTest.NoMergeWithBitcast (1 ms)
[ RUN      ] FusionMergerTest.CostBasedMerge
[       OK ] FusionMergerTest.CostBasedMerge (1 ms)
[ RUN      ] FusionMergerTest.CostBasedNoMerge
[       OK ] FusionMergerTest.CostBasedNoMerge (4 ms)
[ RUN      ] FusionMergerTest.NoMergeBecauseTooManyBasicBlockSplits
[       OK ] FusionMergerTest.NoMergeBecauseTooManyBasicBlockSplits (4 ms)
[ RUN      ] FusionMergerTest.CommonElementwiseUsedParameter
[       OK ] FusionMergerTest.CommonElementwiseUsedParameter (1 ms)
[ RUN      ] FusionMergerTest.IncompatibleNonTrivialHeroes
[       OK ] FusionMergerTest.IncompatibleNonTrivialHeroes (0 ms)
[ RUN      ] FusionMergerTest.DoNotMergeDUSFusions
[       OK ] FusionMergerTest.DoNotMergeDUSFusions (1 ms)
[ RUN      ] FusionMergerTest.MergeDUSFusionWithElementwiseFusion
tensorflow/compiler/xla/service/gpu/fusion_merger_test.cc:1127: Failure
Value of: fusion_merger_.Run(module.get()).value()
  Actual: false
Expected: true
[  FAILED  ] FusionMergerTest.MergeDUSFusionWithElementwiseFusion (0 ms)
[----------] 21 tests from FusionMergerTest (41 ms total)

[----------] Global test environment tear-down
[==========] 21 tests from 1 test suite ran. (42 ms total)
[  PASSED  ] 20 tests.
[  FAILED  ] 1 test, listed below:
[  FAILED  ] FusionMergerTest.MergeDUSFusionWithElementwiseFusion

 1 FAILED TEST
================================================================================
Target //tensorflow/compiler/xla/service/gpu:fusion_merger_test up-to-date:
  bazel-bin/tensorflow/compiler/xla/service/gpu/fusion_merger_test
INFO: Elapsed time: 392.381s, Critical Path: 276.82s
INFO: 1497 processes: 613 internal, 884 local.
INFO: Build completed, 1 test FAILED, 1497 total actions
//tensorflow/compiler/xla/service/gpu:fusion_merger_test                 FAILED in 1.2s
  /home/andrew/src/tf_test/tensorflow-git/bazel-ci_build-cache/.cache/bazel/_bazel_andrew/eab0d61a99b6696edb3d2aff87b585e8/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/xla/service/gpu/fusion_merger_test/test.log

Executed 1 out of 1 test: 1 fails locally.
```
