No registered '_MklLayerNorm' OpKernel for 'GPU' devices compatible with node {{node custom_model/layer_normalization/add}}
@YmdTb 
I was able to run the code successfully on colab, please find the gist [here](https://colab.research.google.com/gist/sushreebarsa/16e688d7193bb0c3596bbdd6622be702/untitled917.ipynb#scrollTo=MmcQoxDOfcLj).
If you are using an older version of MKL, update it to the latest version. This may resolve the compatibility issues.

Thank you! This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you. Encountered same issue, updating MKL to the latest version does not work.
Found out that using the default epsilon=1e-3 in `LayerNormalization` will trigger this error, whereas setting it to a smaller float, say 1e-7 will not trigger the error. Try modifying epsilon to see if it works for you. Had the same experience as @Yuanjimengmengda. Changing the epsilon value solved the problem. This issue was closed because it has been inactive for 7 days since being marked as stale. Please reopen if you'd like to work on this further. Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/62607">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/62607">No</a>
 This issue is still present. I installed Ubuntu 23.10, installed nvidia drivers 545:

+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 545.29.06              Driver Version: 545.29.06    CUDA Version: 12.3     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce RTX 4090        Off | 00000000:01:00.0  On |                  Off |
|  0%   33C    P8              28W / 450W |    465MiB / 24564MiB |      4%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
                                                                                         
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A      5120      G   /usr/lib/xorg/Xorg                          233MiB |
|    0   N/A  N/A      5261      G   /usr/bin/gnome-shell                         55MiB |
|    0   N/A  N/A      9628      G   ...51,262144 --variations-seed-version      160MiB |
+---------------------------------------------------------------------------------------+

Created a new venv, installed tensorflow according to standard directions and got the following when running the given code above:

2024-02-04 10:44:52.974186: W tensorflow/core/grappler/utils/graph_view.cc:849] No registered '_MklLayerNorm' OpKernel for GPU devices compatible with node {{node custom_model/layer_normalization/add}}
	.  Registered:  device='CPU'; T in [DT_BFLOAT16]
  device='CPU'; T in [DT_FLOAT]

Traceback (most recent call last):
  File "/home/nicholaus/git/glpwd.com/git/main/python/pepper/src/broken.py", line 40, in <module>
    predictions = model.predict(X_test)
                  ^^^^^^^^^^^^^^^^^^^^^
  File "/home/nicholaus/venvs/pepper/lib/python3.11/site-packages/keras/src/utils/traceback_utils.py", line 70, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "/home/nicholaus/venvs/pepper/lib/python3.11/site-packages/tensorflow/python/eager/execute.py", line 53, in quick_execute
    tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
tensorflow.python.framework.errors_impl.NotFoundError: Graph execution error:

Detected at node custom_model/layer_normalization/add defined at (most recent call last):
  File "/home/nicholaus/git/glpwd.com/git/main/python/pepper/src/broken.py", line 40, in <module>

  File "/home/nicholaus/venvs/pepper/lib/python3.11/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler

  File "/home/nicholaus/venvs/pepper/lib/python3.11/site-packages/keras/src/engine/training.py", line 2655, in predict

  File "/home/nicholaus/venvs/pepper/lib/python3.11/site-packages/keras/src/engine/training.py", line 2440, in predict_function

  File "/home/nicholaus/venvs/pepper/lib/python3.11/site-packages/keras/src/engine/training.py", line 2425, in step_function

  File "/home/nicholaus/venvs/pepper/lib/python3.11/site-packages/keras/src/engine/training.py", line 2413, in run_step

  File "/home/nicholaus/venvs/pepper/lib/python3.11/site-packages/keras/src/engine/training.py", line 2381, in predict_step

  File "/home/nicholaus/venvs/pepper/lib/python3.11/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler

  File "/home/nicholaus/venvs/pepper/lib/python3.11/site-packages/keras/src/engine/training.py", line 590, in __call__

  File "/home/nicholaus/venvs/pepper/lib/python3.11/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler

  File "/home/nicholaus/venvs/pepper/lib/python3.11/site-packages/keras/src/engine/base_layer.py", line 1149, in __call__

  File "/home/nicholaus/venvs/pepper/lib/python3.11/site-packages/keras/src/utils/traceback_utils.py", line 96, in error_handler

  File "/home/nicholaus/git/glpwd.com/git/main/python/pepper/src/broken.py", line 23, in call

  File "/home/nicholaus/venvs/pepper/lib/python3.11/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler

  File "/home/nicholaus/venvs/pepper/lib/python3.11/site-packages/keras/src/engine/base_layer.py", line 1149, in __call__

  File "/home/nicholaus/venvs/pepper/lib/python3.11/site-packages/keras/src/utils/traceback_utils.py", line 96, in error_handler

  File "/home/nicholaus/venvs/pepper/lib/python3.11/site-packages/keras/src/layers/normalization/layer_normalization.py", line 343, in call

No registered '_MklLayerNorm' OpKernel for 'GPU' devices compatible with node {{node custom_model/layer_normalization/add}}
	.  Registered:  device='CPU'; T in [DT_BFLOAT16]
  device='CPU'; T in [DT_FLOAT]

	 [[custom_model/layer_normalization/add]] [Op:__inference_predict_function_1537]

Note that I'm here after trying numerous other things on Ubuntu 22.04, including running 5.15 kernel, multiple nvidia driver versions, using conda instead of pip, etc.
 I am having the same issue when I use the latest docker colab local runner. Using CPU works. Anyone has a solution for that? I solved this error.
As the other people said, this error is occurred by the value of epsilon, which is one of the arguments of **LayerNormalization** layer in keras API.
https://keras.io/api/layers/normalization_layers/layer_normalization/
You can see the specification of LayerNormalization layer in the above Web page.

```
keras.layers.LayerNormalization(
    axis=-1,
    epsilon=0.001,
    center=True,
    scale=True,
    rms_scaling=False,
    beta_initializer="zeros",
    gamma_initializer="ones",
    beta_regularizer=None,
    gamma_regularizer=None,
    beta_constraint=None,
    gamma_constraint=None,
    \*\*kwargs
)
```

The default value of epsilon is 0.001(1e-3). I fixed it into 1e-7 and now I'm free from this horrible error.

My graphic card is RTX3060 and your graphic card is RTX4070 Ti, these two graphic cards do not support float16 data type operation in keras(I don't know why). And you can see **[DT_BFLOAT16]** in your error log. In the LayerNormalization layer, there's such an operation which adds **epsilon** to some value. Its default value is 1e-3(float16 type) so the error is occurred. If you fix the value of epsilon into 1e-7, its data type will be changed into float32 and the error will not be occurred.

Like this
```
...
        self.layernorm_1 = layers.LayerNormalization(epsilon=1e-7)
        self.layernorm_2 = layers.LayerNormalization(epsilon=1e-7)
...
``` > I am having the same issue when I use the latest docker colab local runner. Using CPU works. Anyone has a solution for that?

I replied below. Hope it works.
### Issue type

Bug

### Have you reproduced the bug with TensorFlow Nightly?

No

### Source

source

### TensorFlow version

2.15.0

### Custom code

Yes

### OS platform and distribution

Docker image: tensorflow/tensorflow:2.15.0-gpu

### Mobile device

Ubuntu 22.04.3 LTS

### Python version

3.11.0rc1

### Bazel version

_No response_

### GCC/compiler version

11.4.0

### CUDA/cuDNN version

12.3

### GPU model and memory

_No response_

### Current behavior?

I used Add and LayerNormalization layers at the same time. The fit works well. But when it comes to predict or anything else, it will occur errors. Detail is below.
Hint: If I use with tf.device('/cpu:0'): before predict, it is good. Idon't know why the prediction can't work in gpu.
What I want is to run all the train and predict on the gpu.

### Standalone code to reproduce the issue

```shell
import tensorflow as tf
from tensorflow.keras.layers import Dense, Add, LayerNormalization
from tensorflow.keras.models import Model

class CustomModel(tf.keras.Model):
    def __init__(self):
        super(CustomModel, self).__init__()

        self.dense1 = Dense(64, activation='relu')
        self.dense2 = Dense(32, activation='relu')
        self.dense3 = Dense(10, activation='relu')
        self.output_layer = Dense(10)
        self.add_layer = Add()
        self.norm_layer = LayerNormalization()

    def call(self, inputs):
        x = self.dense1(inputs)
        x = self.dense2(x)
        x = self.dense3(x)

        x = self.add_layer([x, inputs])

        x = self.norm_layer(x)

        return self.output_layer(x)
print("Num GPUs Available: ", len(tf.config.experimental.list_physical_devices('GPU')))
    
model = CustomModel()

model.compile(optimizer='adam', loss='mse')

import numpy as np
X_train = np.random.rand(100, 10)
y_train = np.random.rand(100, 10)

model.fit(X_train, y_train, epochs=5)

X_test = np.random.rand(10, 10)

predictions = model.predict(X_test)

print(predictions)
```


### Relevant log output

```shell
2023-12-10 09:21:40.012550: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2023-12-10 09:21:40.038118: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2023-12-10 09:21:40.038186: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2023-12-10 09:21:40.039635: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2023-12-10 09:21:40.048947: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-12-10 09:21:41.218693: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:887] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.
2023-12-10 09:21:41.221912: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:887] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.
2023-12-10 09:21:41.221964: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:887] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.
Num GPUs Available:  1
2023-12-10 09:21:41.226791: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:887] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.
2023-12-10 09:21:41.226836: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:887] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.
2023-12-10 09:21:41.226858: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:887] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.
2023-12-10 09:21:41.328324: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:887] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.
2023-12-10 09:21:41.328371: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:887] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.
2023-12-10 09:21:41.328385: I tensorflow/core/common_runtime/gpu/gpu_device.cc:2022] Could not identify NUMA node of platform GPU id 0, defaulting to 0.  Your kernel may not have been built with NUMA support.
2023-12-10 09:21:41.328410: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:887] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.
2023-12-10 09:21:41.328431: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1929] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 9516 MB memory:  -> device: 0, name: NVIDIA GeForce RTX 4070 Ti, pci bus id: 0000:01:00.0, compute capability: 8.9
Epoch 1/5
2023-12-10 09:21:44.298982: I external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:454] Loaded cuDNN version 8906
2023-12-10 09:21:44.471868: I external/local_xla/xla/service/service.cc:168] XLA service 0x7fdcd0a348f0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2023-12-10 09:21:44.471895: I external/local_xla/xla/service/service.cc:176]   StreamExecutor device (0): NVIDIA GeForce RTX 4070 Ti, Compute Capability 8.9
2023-12-10 09:21:44.475045: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:269] disabling MLIR crash reproducer, set env var `MLIR_CRASH_REPRODUCER_DIRECTORY` to enable.
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1702200104.523638   25028 device_compiler.h:186] Compiled cluster using XLA!  This line is logged at most once for the lifetime of the process.
4/4 [==============================] - 3s 4ms/step - loss: 1.4712
Epoch 2/5
4/4 [==============================] - 0s 3ms/step - loss: 1.3851
Epoch 3/5
4/4 [==============================] - 0s 3ms/step - loss: 1.3103
Epoch 4/5
4/4 [==============================] - 0s 3ms/step - loss: 1.2321
Epoch 5/5
4/4 [==============================] - 0s 3ms/step - loss: 1.1533
2023-12-10 09:21:45.056658: W tensorflow/core/grappler/utils/graph_view.cc:849] No registered '_MklLayerNorm' OpKernel for GPU devices compatible with node {{node custom_model/layer_normalization/add}}
        .  Registered:  device='CPU'; T in [DT_BFLOAT16]
  device='CPU'; T in [DT_FLOAT]

Traceback (most recent call last):
  File "/pycharm_project/tttt.py", line 48, in <module>
    predictions = model.predict(X_test)
                  ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/keras/src/utils/traceback_utils.py", line 70, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "/usr/local/lib/python3.11/dist-packages/tensorflow/python/eager/execute.py", line 53, in quick_execute
    tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
tensorflow.python.framework.errors_impl.NotFoundError: Graph execution error:

Detected at node custom_model/layer_normalization/add defined at (most recent call last):
  File "/pycharm_project/tttt.py", line 48, in <module>

  File "/usr/local/lib/python3.11/dist-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler

  File "/usr/local/lib/python3.11/dist-packages/keras/src/engine/training.py", line 2655, in predict

  File "/usr/local/lib/python3.11/dist-packages/keras/src/engine/training.py", line 2440, in predict_function

  File "/usr/local/lib/python3.11/dist-packages/keras/src/engine/training.py", line 2425, in step_function

  File "/usr/local/lib/python3.11/dist-packages/keras/src/engine/training.py", line 2413, in run_step

  File "/usr/local/lib/python3.11/dist-packages/keras/src/engine/training.py", line 2381, in predict_step

  File "/usr/local/lib/python3.11/dist-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler

  File "/usr/local/lib/python3.11/dist-packages/keras/src/engine/training.py", line 590, in __call__

  File "/usr/local/lib/python3.11/dist-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler

  File "/usr/local/lib/python3.11/dist-packages/keras/src/engine/base_layer.py", line 1149, in __call__

  File "/usr/local/lib/python3.11/dist-packages/keras/src/utils/traceback_utils.py", line 96, in error_handler

  File "/pycharm_project/tttt.py", line 25, in call

  File "/usr/local/lib/python3.11/dist-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler

  File "/usr/local/lib/python3.11/dist-packages/keras/src/engine/base_layer.py", line 1149, in __call__

  File "/usr/local/lib/python3.11/dist-packages/keras/src/utils/traceback_utils.py", line 96, in error_handler

  File "/usr/local/lib/python3.11/dist-packages/keras/src/layers/normalization/layer_normalization.py", line 343, in call

No registered '_MklLayerNorm' OpKernel for 'GPU' devices compatible with node {{node custom_model/layer_normalization/add}}
        .  Registered:  device='CPU'; T in [DT_BFLOAT16]
  device='CPU'; T in [DT_FLOAT]

         [[custom_model/layer_normalization/add]] [Op:__inference_predict_function_1537]
```
