DepthwiseConv2D is 24 times slower then Conv2D with CUDA 11.8 and jit_compile=True
Downgrading to cuda 11.8 did not help.
```
RTX4090 + Cuda 11.8 + Cudnn 8.8 + TF nightly with mixed_fp16
dw=True,  jit_compile=True,  ETA after 200 steps: 1:17:38
```

But when i downgraded to TF 2.11 and cuda 11.2, depthwise conv become just 2 times slower then normal one.
```
RTX4090 + Cuda 11.2 + Cudnn 8.1 + TF 2.11 with mixed_fp16
dw=True,  jit_compile=True,  ETA after 200 steps:    9:39
dw=False, jit_compile=True,  ETA after 200 steps:    4:58
dw=True,  jit_compile=False, ETA after 200 steps:   21:10
```

Compilation from source with TF 2.11 and Cuda 11.2 required to limit compute compatibility with 8.6
And now, when running model i get warning:
```
2023-02-22 16:45:51.222824: W tensorflow/compiler/xla/stream_executor/gpu/asm_compiler.cc:234] Falling back to the CUDA driver for PTX compilation; ptxas does not support CC 8.9
2023-02-22 16:45:51.222852: W tensorflow/compiler/xla/stream_executor/gpu/asm_compiler.cc:237] Used ptxas at ptxas
2023-02-22 16:45:51.222934: W tensorflow/compiler/xla/stream_executor/gpu/redzone_allocator.cc:318] UNIMPLEMENTED: ptxas ptxas too old. Falling back to the driver to compile.
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
2023-02-22 16:45:51.225390: W tensorflow/compiler/xla/service/gpu/buffer_comparator.cc:641] UNIMPLEMENTED: ptxas ptxas too old. Falling back to the driver to compile.
Relying on driver to perform ptx compilation. 
Setting XLA_FLAGS=--xla_gpu_cuda_data_dir=/path/to/cuda  or modifying $PATH can be used to set the location of ptxas
``` @shkarupa-alex,
Thank you for reporting the issue. This issue is more related to Keras. Development of keras moved to another [repository](https://github.com/keras-team/keras/issues). 

Could you please post this issue on keras-team/keras [repo](https://github.com/keras-team/keras/issues).
To know more please refer:
https://discuss.tensorflow.org/t/keras-project-moved-to-new-repository-in-https-github-com-keras-team-keras/1999
Thank you!
 @tilakrayal could you please explain why do you think this issue is related to keras?
Under the hood it uses tf.nn.depthwise_conv2d which is a part of tensorflow core. There is no something unusual in this layer except that.

So i think this issue is fully related to TF. @sachinprasadhs,
I was able to reproduce the issue on tensorflow [v2.11](https://colab.research.google.com/gist/tilakrayal/379181981726a53a61e16b1912e13672/untitled24.ipynb) and tf-nightly. Kindly find the gist of it [here](https://colab.research.google.com/gist/tilakrayal/1b02979b66e15bf3b68c21bb0097c038/untitled1004.ipynb). Hi, TF 2.12 or TF 2.13 is still a development branch, we are planning to migrate the cuDNN, CUDA versions in the upcoming Tensorflow 2.12 release. 
Till then could you please test it against the published tested configurations as below.


Version | Python version | Compiler | Build tools | cuDNN | CUDA
-- | -- | -- | -- | -- | --
tensorflow_gpu-2.11.0 | 3.7-3.10 | MSVC 2019 | Bazel 5.3.0 | 8.1 | 11.2

As per your findings, `DepthwiseConv2D` is 2x slower than Conv2D, it might be because of the different implementation methods followed in [DepthwiseConv2D](https://www.tensorflow.org/api_docs/python/tf/keras/layers/DepthwiseConv2D#used-in-the-notebooks). > Till then could you please test it against the published tested configurations as below.

@sachinprasadhs , my test with stable release and cuda 11.2 is here https://github.com/tensorflow/tensorflow/issues/59772#issuecomment-1440047780
// ptxas warnings and DWConv is 2x slower More confusing is that DWConv with jit_compile=True is slower then itself without jit_compile

> RTX4090 + Cuda 12.0 + Cudnn 8.8 + TF nightly with fp32
> dw=True,  jit_compile=True,  ETA after 200 steps: 3:43:26
> dw=False, jit_compile=True,  ETA after 200 steps:    9:13
> dw=True,  jit_compile=False, ETA after 200 steps:   26:18
 Tested with fresh 2.12 release + cuda 11.8 and saw same numbers.

Conv2D = 4m 31s  vs DepthwiseConv2d = 1h 17m 22s
<details><summary>Click to expand!</summary> 
 
 ### Issue Type

Performance

### Have you reproduced the bug with TF nightly?

Yes

### Source

source

### Tensorflow Version

'v1.12.1-89750-g9f16e373ca8', '2.13.0'

### Custom Code

No

### OS Platform and Distribution

Ubuntu 22.04

### Mobile device

_No response_

### Python version

3.10.6

### Bazel version

5.3.0

### GCC/Compiler version

11.3.0

### CUDA/cuDNN version

12.0 / 8.8

### GPU model and memory

RTX 4090 Aorus Master

### Current Behaviour?

```shell
After upgrading to RTX4090 i installed cuda 12.0 + cudnn 8.8 and found that some models become slower (e.g. EfficientNets) and some not (SwinTransformer).

I localized the issue to combination of `DepthwiseConv2D + model.compile(jit_compile=True)`. 
In my case this combination is 24 times slower then `Conv2D + model.compile(jit_compile=True)` and 8 times slower then `DepthwiseConv2D + model.compile(jit_compile=False)`.
Here are some numbers:

Colab + TF 2.11.0 with fp32
dw=True,  jit_compile=True,  ETA after 200 steps: 1:18:06
dw=False, jit_compile=True,  ETA after 200 steps: 1:18:20
dw=True,  jit_compile=False, ETA after 200 steps: 2:29:57


Colab + TF nightly with fp32
dw=True,  jit_compile=True,  ETA after 200 steps: 1:21:25
dw=False, jit_compile=True,  ETA after 200 steps: 1:18:23
dw=True,  jit_compile=False, ETA after 200 steps: 2:29:40


RTX4090 + Cuda 12.0 + Cudnn 8.8 + TF nightly with fp32
dw=True,  jit_compile=True,  ETA after 200 steps: 3:43:26
dw=False, jit_compile=True,  ETA after 200 steps:    9:13
dw=True,  jit_compile=False, ETA after 200 steps:   26:18


RTX4090 + Cuda 12.0 + Cudnn 8.8 + TF nightly with mixed_fp16
dw=True,  jit_compile=True,  ETA after 200 steps: 1:17:36
dw=False, jit_compile=True,  ETA after 200 steps:    4:33
dw=True,  jit_compile=False, ETA after 200 steps:   21:57
```


### Standalone code to reproduce the issue

```shell
https://colab.research.google.com/drive/1KTMAVhpQLQhRphRfRjHNo8faR9NBOlK6#scrollTo=IEGDQp7G6aUq
```


### Relevant log output

_No response_</details>