tf.matmul gives wrong result on CPUs with avx512_vnni
+@yehudaorel, @agramesh1  @zhezherun thank you for raising this issue, I was able to reproduce the issue using tensorflow==2.13.0 binary release:

**TF_ENABLE_ONEDNN_OPTS=1**
```
onednn_verbose,info,oneDNN v2.7.3 (commit N/A)
onednn_verbose,info,cpu,runtime:threadpool,nthr:224
onednn_verbose,info,cpu,isa:Intel AVX-512 with float16, Intel DL Boost and bfloat16 support and Intel AMX with bfloat16 and 8-bit integer support
onednn_verbose,info,gpu,runtime:none
onednn_verbose,info,prim_template:operation,engine,primitive,implementation,prop_kind,memory_descriptors,attributes,auxiliary,problem_desc,exec_time
onednn_verbose,exec,cpu,matmul,brg:avx512_core,undef,src_f32::blocked:acb:f0 wei_f32::blocked:abc:f0 dst_f32::blocked:abc:f0,attr-scratchpad:user ,,3x3x5000:3x5000x3:3x3x3,2.125
[[[2.3496000e+04 4.5627100e+02 9.7381091e+02]
  [4.5627100e+02 2.3422373e+04 4.2656726e+02]
  [9.7381091e+02 4.2656726e+02 2.3594883e+04]]

 [[4.5893926e-41 0.0000000e+00 0.0000000e+00]
  [1.4012985e-45 1.7743547e+28 0.0000000e+00]
  [0.0000000e+00 2.2420775e-43 0.0000000e+00]]

 [[1.1210388e-43 0.0000000e+00 1.1527105e-35]
  [0.0000000e+00 7.0064923e-45 0.0000000e+00]
  [6.7421017e+22 1.6815582e-43 0.0000000e+00]]]
```

**TF_ENABLE_ONEDNN_OPTS=0**  
```
[[[5000.        128.78204   131.61632 ]
  [ 128.78204  4929.8867    -54.454735]
  [ 131.61632   -54.454735 4999.6367  ]]

 [[5000.        128.78204   131.61632 ]
  [ 128.78204  4929.8867    -54.454735]
  [ 131.61632   -54.454735 4999.6367  ]]

 [[5000.        128.78204   131.61632 ]
  [ 128.78204  4929.8867    -54.454735]
  [ 131.61632   -54.454735 4999.6367  ]]]
```

@agramesh1 @vpirogov I also found that this issue only happens when **length is greater than 2681**

Currently building tf from source to see if issue still there in latest oneDNN release.



 I can reproduce this with oneDNN tests. Looks like oneDNN v2.7.3 is affected, but not the current version, oneDNN v3.2.

Output with `ONEDNN_VERBOSE=1`:
```
$ ONEDNN_VERBOSE=1 python3 ./test.py
2023-08-09 10:54:14.079700: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2023-08-09 10:54:14.081993: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.
2023-08-09 10:54:14.131952: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.
2023-08-09 10:54:14.132449: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI AVX512_BF16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-08-09 10:54:16.185646: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
onednn_verbose,info,oneDNN v2.7.3 (commit N/A)
onednn_verbose,info,cpu,runtime:threadpool,nthr:72
onednn_verbose,info,cpu,isa:Intel AVX-512 with Intel DL Boost and bfloat16 support
onednn_verbose,info,gpu,runtime:none
onednn_verbose,info,prim_template:operation,engine,primitive,implementation,prop_kind,memory_descriptors,attributes,auxiliary,problem_desc,exec_time
onednn_verbose,exec,cpu,matmul,brg:avx512_core,undef,src_f32::blocked:acb:f0 wei_f32::blocked:abc:f0 dst_f32::blocked:abc:f0,attr-scratchpad:user ,,3x3x5000:3x5000x3:3x3x3,0.442139
[[[ 1.1928000e+04  4.0859866e+02  3.7975815e+02]
  [ 4.0859866e+02  1.1693020e+04 -9.1308746e+00]
  [ 3.7975815e+02 -9.1308746e+00  1.1971855e+04]]

 [[ 0.0000000e+00  0.0000000e+00  0.0000000e+00]
  [ 0.0000000e+00  1.8600073e+02  0.0000000e+00]
  [ 0.0000000e+00  1.1210388e-44  0.0000000e+00]]

 [[ 4.5191123e-06  2.0196074e-40  1.1210388e-44]
  [ 0.0000000e+00  3.0517854e-36  0.0000000e+00]
  [ 0.0000000e+00  0.0000000e+00  3.0522912e-36]]]
```

Benchdnn results:
```
$ DNNL_VERBOSE=1 ./benchdnn --matmul --stag=acb 3x3x5000:3x5000x3:3x3x3
onednn_verbose,info,oneDNN v2.7.3 (commit 7710bdc92064a08b985c5cbdb09de773b19cba1f)
onednn_verbose,info,cpu,runtime:OpenMP,nthr:72
onednn_verbose,info,cpu,isa:Intel AVX-512 with Intel DL Boost and bfloat16 support
onednn_verbose,info,gpu,runtime:none
onednn_verbose,info,prim_template:operation,engine,primitive,implementation,prop_kind,memory_descriptors,attributes,auxiliary,problem_desc,exec_time
onednn_verbose,exec,cpu,reorder,jit:uni,undef,src_f32::blocked:abc:f0 dst_f32::blocked:acb:f0,,,3x3x5000,3.79419
onednn_verbose,exec,cpu,reorder,simple:any,undef,src_f32::blocked:abc:f0 dst_f32::blocked:abc:f0,,,3x5000x3,0.0388184
onednn_verbose,exec,cpu,matmul,brg:avx512_core,undef,src_f32::blocked:acb:f0 wei_f32::blocked:abc:f0 dst_f32::blocked:abc:f0,,,3x3x5000:3x5000x3:3x3x3,0.166016
onednn_verbose,exec,cpu,reorder,simple:any,undef,src_f32::blocked:abc:f0 dst_f32::blocked:abc:f0,,,3x3x3,0.0319824
onednn_verbose,exec,cpu,reorder,simple:any,undef,src_f32::blocked:abc:f0 dst_f32::blocked:abc:f0,,,3x3x3,0.0361328
[   0][DST][0:0:0] exp_f32:  -35516 exp:  -35516 got: -539159 diff:  503643 rdiff: 14.1807
[   1][DST][0:0:1] exp_f32:  114098 exp:  114098 got:  -19606 diff:  133704 rdiff: 1.17183
[   2][DST][0:0:2] exp_f32:     929 exp:     929 got:   22304 diff:   21375 rdiff: 23.0086
[   3][DST][0:1:0] exp_f32:   68428 exp:   68428 got: -518338 diff:  586766 rdiff: 8.57494
[   4][DST][0:1:1] exp_f32:   51054 exp:   51054 got: -275533 diff:  326587 rdiff: 6.39689
[   5][DST][0:1:2] exp_f32:   58651 exp:   58651 got:  154408 diff:   95757 rdiff: 1.63266
[   6][DST][0:2:0] exp_f32: -137905 exp: -137905 got:  221970 diff:  359875 rdiff: 2.60959
[   7][DST][0:2:1] exp_f32:  -73033 exp:  -73033 got:  346390 diff:  419423 rdiff: 5.74292
[   8][DST][0:2:2] exp_f32:  287192 exp:  287192 got:   45300 diff:  241892 rdiff:0.842266
0:FAILED (errors:27 total:27) __REPRO: --matmul --stag=acb 3x3x5000:3x5000x3:3x3x3
tests:1 passed:0 skipped:0 mistrusted:0 unimplemented:0 invalid_arguments:0 failed:1 listed:0
total compute_ref: sum(s):0.00
``` The fix is now available in [oneDNN v2.7.5](https://github.com/oneapi-src/oneDNN/releases/tag/v2.7.5). Current version, [oneDNN v3.2.1](https://github.com/oneapi-src/oneDNN/releases/tag/v3.2.1) is not affected. > The fix is now available in [oneDNN v2.7.5](https://github.com/oneapi-src/oneDNN/releases/tag/v2.7.5). Current version, [oneDNN v3.2.1](https://github.com/oneapi-src/oneDNN/releases/tag/v3.2.1) is not affected.

Thanks a lot for a quick fix! Is it possible to include it in 2.13.1? hi @zhezherun there is no plan for updating oneDNN to 2.7.5 in TF 2.13.1.   We are expecting to update oneDNN with this fix in the upcoming release of TF 2.14. > hi @zhezherun there is no plan for updating oneDNN to 2.7.5 in TF 2.13.1. We are expecting to update oneDNN with this fix in the upcoming release of TF 2.14.

I see, unfortunately this means that TF 2.13 is unusable for us :( @zhezherun  are you able to build TF by yourself?  You can patch TF 2.13 to use oneDNN 2.7.5. I also observed similar computing bugs in the following APIs in some older versions of tensorflow.
Users should be cautious when using them on CPU from tensorflow 2.9.0 (v2.9.0-rc2-42-g8a20d54a3c1) to 2.13.0 (v2.13.0-rc2-7-g1cb1a030a62).

- `tf.linalg.matmul`, `(tf.matmul)`, `tf.compat.v1.linalg.matmul`, `tf.compat.v1.matmul`

<details>
  <summary>Code to reproduce the issue in <code>tf.linalg.matmul</code> APIs in older versions</summary>

```python
import tensorflow as tf
print(tf.version.GIT_VERSION, tf.version.VERSION, flush=True)
print(tf.config.list_physical_devices(), flush=True)


tf.keras.utils.set_random_seed(1)
length = 5000
x = tf.concat([tf.ones([length, 1]), tf.random.normal([length, 2])], axis=1)
x = tf.tile(x[None, ...], [3, 1, 1])
xx = tf.linalg.matmul(x, x, transpose_a=True)
# xx = tf.matmul(x, x, transpose_a=True)
# xx = tf.compat.v1.linalg.matmul(x, x, transpose_a=True)
# xx = tf.compat.v1.matmul(x, x, transpose_a=True)
print(f"{xx.numpy()}", flush=True)
```

The GPU provides the following outputs:

```text
v2.13.0-rc2-7-g1cb1a030a62 2.13.0
[PhysicalDevice(name='/physical_device:CPU:0', device_type='CPU'), PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]
[[[5000.        128.77832   131.63278 ]
  [ 128.77832  4929.636     -54.427303]
  [ 131.63278   -54.427303 4999.2427  ]]

 [[5000.        128.77832   131.63278 ]
  [ 128.77832  4929.636     -54.427303]
  [ 131.63278   -54.427303 4999.2427  ]]

 [[5000.        128.77832   131.63278 ]
  [ 128.77832  4929.636     -54.427303]
  [ 131.63278   -54.427303 4999.2427  ]]]
```

While the CPU provides the following outputs, which are inconsistent with the GPU outputs:

```text
v2.13.0-rc2-7-g1cb1a030a62 2.13.0
[PhysicalDevice(name='/physical_device:CPU:0', device_type='CPU')]
[[[ 2.2984000e+04  5.9729730e+02  9.9359949e+02]
  [ 5.9729730e+02  2.2604811e+04  5.0240503e+02]
  [ 1.1394688e+03  4.4723795e+02  2.4646848e+04]]

 [[ 6.7804801e-10  1.7862294e+31  1.6027861e-10]
  [ 1.8617160e+25  1.8639604e+02  6.7787642e-10]
  [-5.4521859e-04  0.0000000e+00  0.0000000e+00]]

 [[ 5.6192068e-43  0.0000000e+00 -1.4373869e+22]
  [ 4.5728573e-41 -1.4373869e+22  4.5728573e-41]
  [ 0.0000000e+00  0.0000000e+00  0.0000000e+00]]]
```
</details>

It seems to be fixed by pull request #61595, and tensorflow 2.14.0-rc1 (v2.14.0-rc0-34-gdd01672d9a9) and nightly (2.15.0-dev20230906) are not affected.

Besides, I also found a slight numerical precision difference between CPU and GPU, which is acceptable in many cases but should be noted.

<details>
  <summary>Show outputs of the slight numerical precision difference between CPU and GPU</summary>

The GPU outputs:

```text
v2.14.0-rc0-34-gdd01672d9a9 2.14.0-rc1
[PhysicalDevice(name='/physical_device:CPU:0', device_type='CPU'), PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]
[[[5000.        128.77832   131.63278 ]
  [ 128.77832  4929.636     -54.427303]
  [ 131.63278   -54.427303 4999.2427  ]]

 [[5000.        128.77832   131.63278 ]
  [ 128.77832  4929.636     -54.427303]
  [ 131.63278   -54.427303 4999.2427  ]]

 [[5000.        128.77832   131.63278 ]
  [ 128.77832  4929.636     -54.427303]
  [ 131.63278   -54.427303 4999.2427  ]]]
```

The CPU outputs:

```text
v2.14.0-rc0-34-gdd01672d9a9 2.14.0-rc1
[PhysicalDevice(name='/physical_device:CPU:0', device_type='CPU')]
[[[5000.       128.78206  131.61627]
  [ 128.78206 4929.8867   -54.45468]
  [ 131.61627  -54.45468 4999.636  ]]

 [[5000.       128.78206  131.61627]
  [ 128.78206 4929.8867   -54.45468]
  [ 131.61627  -54.45468 4999.636  ]]

 [[5000.       128.78206  131.61627]
  [ 128.78206 4929.8867   -54.45468]
  [ 131.61627  -54.45468 4999.636  ]]]
```
</details> @zhezherun,
The related PR #61595 has been merged and also tried to execute the above mentioned code on tensorflow v2.14 & it was executed with the result expected. Kindly find the gist of it here [CPU](https://colab.research.google.com/gist/tilakrayal/a03a8c725734519e5f76d62adc75bde3/untitled1540.ipynb) and [GPU](https://colab.research.google.com/gist/tilakrayal/7aff4ed034c0770efe11611a3a9d9402/untitled1541.ipynb).

```
[[[5000.        128.78201   131.6163  ]
  [ 128.78201  4929.8857    -54.454735]
  [ 131.6163    -54.454735 4999.6367  ]]

 [[5000.        128.78201   131.6163  ]
  [ 128.78201  4929.8857    -54.454735]
  [ 131.6163    -54.454735 4999.6367  ]]

 [[5000.        128.78201   131.6163  ]
  [ 128.78201  4929.8857    -54.454735]
  [ 131.6163    -54.454735 4999.6367  ]]]
```

Thank you! This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you. Thanks @tilakrayal, I confirm that 2.14 is working Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/61510">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/61510">No</a>

### Issue type

Bug

### Have you reproduced the bug with TensorFlow Nightly?

No

### Source

binary

### TensorFlow version

2.13.0

### Custom code

No

### OS platform and distribution

Linux

### Mobile device

_No response_

### Python version

_No response_

### Bazel version

_No response_

### GCC/compiler version

_No response_

### CUDA/cuDNN version

_No response_

### GPU model and memory

_No response_

### Current behavior?

When running on a CPU with avx512_vnni instructions (e.g. Xeon Platinum 8260), tf.matmul in 2.13 gives a completely wrong result that also changes from run to run. Other functions, e.g. tf.einsum, are affected too. A reproducer is included below. 2.12 is working correctly.

I believe this is due to a bug in oneDNN, since running with TF_ENABLE_ONEDNN_OPTS=0 restores the correct behaviour.

Could you please try building TF against the latest oneDNN to see if this bug is already fixed there, and either upgrade oneDNN or revert back to the version used in 2.12? If this bug is still present in latest oneDNN, could you also forward this issue to them so that they can fix it? tf.matmul is such a fundamental part of TensorFlow, so it would be great to have a fix for this as soon as possible (and perhaps add further tests to prevent this kind of bug from reoccurring?)

### Standalone code to reproduce the issue

```shell
import tensorflow as tf
tf.keras.utils.set_random_seed(1)
length = 5000
x = tf.concat([tf.ones([length, 1]), tf.random.normal([length, 2])], axis=1)
x = tf.tile(x[None, ...], [3, 1, 1])
xx = tf.matmul(x, x, transpose_a=True)
# xx = tf.einsum("ijk,ijm->ikm", x, x)  # Also doesn't work
print(f"{xx.numpy()}")
```


### Relevant log output

```shell
[[[ 2.0936000e+04  4.1334631e+02  8.8164221e+02]
  [ 4.1334631e+02  2.0951623e+04  5.5098944e+02]
  [ 8.1284619e+02  4.7815466e+02  1.9981070e+04]]

 [[ 0.0000000e+00  6.8663625e-44  0.0000000e+00]
  [ 2.7628342e-35  0.0000000e+00  2.6752920e-35]
  [ 0.0000000e+00  2.3822074e-44  0.0000000e+00]]

 [[-7.9164143e+31  8.9978802e+02  1.7936620e-43]
  [ 0.0000000e+00  1.1210388e-43  0.0000000e+00]
  [ 2.7767702e-35  0.0000000e+00  7.0064923e-45]]]
```
