[NVIDIA XLA] Support Conv-Bias-Relu6/LeakyRelu fusion in XLA using cuDNN runtime fusion
@reedwm could you please help run Google internal tests? Hi @reedwm does this PR pass the internal tests? Hi @reedwm Can you please assist on above comments from @Young768. Thank you! @Young768, sorry for the delay. This causes compile time regressions, and in some cases compile time increases 300x. I'm worried that if I accept this, it will be rolled back, and rolling back large PRs is messy.

The ideal solution would be to improve compile time performance of cuDNN's relu fusions. Can you ask the cuDNN team to see if that is possible? But even if possible, this will take a while. In the meantime, can you guard this feature behind a [debug option flag](https://github.com/tensorflow/tensorflow/blob/9804d0b5eba7a17fce316573be32d2593c8acdd9/tensorflow/compiler/xla/debug_options_flags.cc#L176)? Then after this PR is submitted, we can then try turning the flag on by default, and rolling back that one-line change is easier than rolling back this entire PR. Perhaps the flag can be called `xla_gpu_extra_conv_relu_fusions`, but maybe you can think of a better name. Hi @reedwm, what do you mean by `compile time`? Do you mean the cudnn runtime compilation time during the model training/inference? If yes, maybe we can simply use `xla_gpu_use_runtime_fusion`, which was introduced in our first cudnn runtime XLA PR. At that time, it was turned on by default but it seems we now want to turn it off to be more conservative.

Also, can you show us in which cases you saw the 300x slowdown? We can use them as the motivation examples to ask our cudnn team to improve the compilation overhead. They are looking for such cases quite enthusiastically. For reference, all these 3 extra fusions are handled here: https://github.com/tensorflow/tensorflow/pull/60377/files#diff-0476808ba47a3d580457c1189ffc5677cea8f4fc0fcce600e1f26e342937cc26R82. > Hi @reedwm, what do you mean by `compile time`? Do you mean the cudnn runtime compilation time during the model training/inference? If yes, maybe we can simply use `xla_gpu_use_runtime_fusion`, which was introduced in our first cudnn runtime XLA PR. At that time, it was turned on by default but it seems we now want to turn it off to be more conservative.
> 
> Also, can you show us in which cases you saw the 300x slowdown? We can use them as the motivation examples to ask our cudnn team to improve the compilation overhead. They are looking for such cases quite enthusiastically.

I have no details about the compilation regression that @reedwm mentioned, but to me it sounds like using xla_gpu_runtime_fusion flag for this (and disabling it by default) could also be a solution until the problem with compile time is fixed. I'm going to try to submit this internally at Google.  It will require some minor modifications (that I will take care of). This is closed by https://github.com/tensorflow/tensorflow/commit/24f323096dd48615ca59c208708ca2cb8d88685f.

Hope it sticks. Unfortunately I have had to disable leaky-relu fusion.  On Ampere Waymo hit a convolution that got 0 algorithms during autotuning.  Therefore the model fails to run.

The failing testcase will be documented in the code so you all can have a look at your leisure.

relu6 has not gotten rolled back.  Yet.  :) https://github.com/tensorflow/tensorflow/commit/cb73612a496950fa5ace9ee9967757edb2e5a493 disables leaky-relu and has a testcase that was breaking for us with cudnn 8.9.0. And I have now found a case where we have the same behavior with kRelu6: cudnn provides 0 algorithms for running a kRelu6 fusion.  I will have to disable that as well.  :( I entirely disabled cudnn runtime fusion in https://github.com/tensorflow/tensorflow/commit/83bbee775f381a481cbfad4e910ce7fbafa93df1.

Please see that commit for testcases that illustrate the problem we're hitting. Thanks for the finding. Let me double check it and may need to file a bug to our cudnn team to fix. I think the problem may be that we were missing the input/output-channels divisible by 2 check for relu6 and leaky-relu.  This check was present for kElu but it looks like it should have been general.
This is the re-submission of PR#5[9614](https://github.com/tensorflow/tensorflow/pull/59614).

@pjannaty @nluehr
