tf.mul after tf.split + tf.sigmoid produces wrong numerical results with MKL enabled
Hi @curufinwe ,

You have used TF1.x version styling which is not supported currently.Though i tried to replicate the issue but for me results are same with both tensorflow-cpu and tensorflow-intel also. I tried to convert the code to V2 style and testing done and it also produces same results. Please refer to attached [gist](https://colab.research.google.com/gist/SuryanarayanaY/8062edb9c446b9027030b5720be5f451/60941.ipynb). I think we were using the package `intel-tensorflow`, not the other way around. I'm not sure it uses the correct TF package in your Gist. Hi @SuryanarayanaY ,
you can reproduce the issue in a docker container build from the following Dockerfile:
```
FROM ubuntu:22.04

RUN apt update && apt upgrade -y && apt install -y python3-pip
RUN pip3 install intel-tensorflow
ADD test.py .
CMD python3 test.py
```
This assumes that you put the python script from the bug description next to the Dockerfile in a file called test.py. I tested the following TF2 version from your colab and it indeed produces the correct result. Seems to be an issue only in the v1 API.

```
import math

import tensorflow as tf
import numpy as np

def sigmoid(x):
  return 1 / (1 + math.exp(-x))

data = [[[2.0, 3.0]]]
a, b = tf.split(data, 2, axis=2)
print(a,b)

tf_sig = tf.sigmoid(a) * b
print(tf_sig.numpy()[0][0][0])

math_sig = sigmoid(data[0][0][0]) * data[0][0][1]

print(math_sig)
``` Hi @curufinwe ,

Thanks for confirmation.It might be possible the bug was fixed in later versions. Since 1.x versions are not supported anymore and the issue fixed in latest versions already shall we mark it as closed?

Thanks! This bug occurs with recent TF 2 versions. @curufinwe refers to using `disable_eager_execution` in TF 2.
 If there is no intention to fix the v1 API anymore then yes, one can consider this bug closed. HI @albertz , @curufinwe ,

The API `tf.compat.v1.disable_eager_execution` was designed for TensorFlow v1 . For TF V2 coding we need to use `tf.function` for enabling graph mode.Please refer the attached doc [source](https://www.tensorflow.org/api_docs/python/tf/compat/v1/disable_eager_execution#:~:text=Caution%3A%20This,tf.function.) for same.

If the reported behaviour can able to reproducible with `tf.function()` then definitely it needs to be looked into. I modified the code using `tf.function` and can see same results as per attached [gist](https://colab.research.google.com/gist/SuryanarayanaY/a88b2e7c8e63d659506ada9993683112/60941.ipynb) which indicates this might have fixed already in TF 2.X versions. WDYT ?

Thanks! From your blog post [Building the Future of TensorFlow, 2022](https://blog.tensorflow.org/2022/10/building-the-future-of-tensorflow.html), you state:

> The future of TensorFlow will be 100% backwards-compatible

By that, I understand that this V1-style API using `tf.compat.v1.disable_eager_execution()` is still sth you want to support. If so, then in any case I think the bug is still relevant.

---

Despite that, as we still do not fully understand the bug itself, it might be that the bug also exists when using `tf.function`, just that it will not be reproduced this way, but needs some other code to be reproduced (but we don't know).

I think it still would in any case be good if this can be verified.
 I have eliminated the `disable_eager_execution` function call and am still able to reproduce the error by importing the tf function into a graph and running it within a session. The problem is that this graph based execution is the only form of execution for programs that use the C++ API for tensorflow afaik. Thus I think the bug is still relevant.

```
#!/usr/bin/env python3

import math

import tensorflow as tf
import numpy as np

def sigmoid(x):
  return 1 / (1 + math.exp(-x))

data = [[[2.0, 3.0]]]

@tf.function
def my_func(data):
    a, b = tf.split(data, 2, axis=2)
    sig = tf.sigmoid(a)
    m = sig * b
    return m

concrete_func = my_func.get_concrete_function(tf.constant(data))
g = concrete_func.graph.as_graph_def()

s = tf.compat.v1.Session()
tf.import_graph_def(g)
out = s.run(['mul:0'], feed_dict={'data:0': data})

print('computed: ', out[0][0][0])
print('expected: ', sigmoid(data[0][0][0]) * data[0][0][1])
``` Hi @curufinwe ,

I tried to replicate the behaviour with the latest provided code snippet but failed to replicate the behaviour.Please refer attached [gist](https://colab.research.google.com/gist/SuryanarayanaY/363da2885015d4dcda3c3152e12d922f/60941-r1.ipynb#scrollTo=CLcYoLBalLND).

`tensorflow-intel` package is built,maintained and tested by `intel` for Windows CPU.Same mentioned in TF [documentation](https://www.tensorflow.org/install/pip#windows-native_1:~:text=Note%3A%20Starting,about%20this%20collaboration.) as well as [Pypi](https://pypi.org/project/tensorflow-intel/). As per my understanding this package has to be installed on windows only. I am not sure whether it is cross compiled for other OS also.

CC' ing @TensorFlow-MKL for more details. Hi @SuryanarayanaY ,

the tensorflow-intel package on pypi is indeed a Windows only package. But intel-tensorflow is also available for Linux and you can use the Dockerfile I provided above to reproduce the issue. Just replace the test.py with the latest code-snippet I posted.
I have now also created a [gist](https://colab.research.google.com/gist/curufinwe/f40fb6fc8749d80631cf00a7f23743df/60941-r1.ipynb) to reproduce the issue. Hi @curufinwe ,

I am acknowledging that now the issue can be replicable from my side as per attached [gist](https://colab.research.google.com/gist/SuryanarayanaY/412bf387a967778be75993c1cef82768/60941-r2-tensorflow-intel.ipynb). The issue persists with `intel-tensorflow` package only but not with TF package itself which was tested in attached gist.

Though the code involves V1 style session there is difference in outputs wr.t `intel-tensorflow` and `tensorflow`. Hence assigning the issue to respective team for their comments.

CC : @TensorFlow-MKL for commenting o this issue.
 CC: @learning-to-play 
 I can reprodue this on tensorflow 2.10, 2.11, 2,12 on a ice-lake CPU, Intel(R) Xeon(R) Gold 6346.
```
$  ONEDNN_VERBOSE=1 python test_repoduce.py
2023-06-28 17:29:34.639477: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-06-28 17:29:35.705623: I tensorflow/core/common_runtime/process_util.cc:146] Creating new thread pool with default inter op setting: 2. Tune using inter_op_parallelism_threads for best performance.
Tensor("split:0", dtype=float32)
Tensor("split:1", dtype=float32)
2023-06-28 17:29:35.708384: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:353] MLIR V1 optimization pass is not enabled
onednn_verbose,info,oneDNN v2.7.3 (commit N/A)
onednn_verbose,info,cpu,runtime:OpenMP,nthr:64
onednn_verbose,info,cpu,isa:Intel AVX-512 with Intel DL Boost
onednn_verbose,info,gpu,runtime:none
onednn_verbose,info,prim_template:operation,engine,primitive,implementation,prop_kind,memory_descriptors,attributes,auxiliary,problem_desc,exec_time
onednn_verbose,exec,cpu,eltwise,jit:avx512_core,forward_training,data_f32::blocked:abc:f0 diff_undef::undef::f0,,alg:eltwise_swish alpha:1 beta:0,1x1x1,40.9431
[array([[[2., 3.]]], dtype=float32), array([[[1.7615942]]], dtype=float32)]
computed:  1.7615942
expected:  2.642391233933647
```

>> If `m = sig * b` is changed to `m = sig * (b + 0.0)` one can get the correct result.
In this case, oneDNN log is not output.

Looks like related to oneDNN. 

Please  run `ONEDNN_VERBOSE=1 python test.py` to output oneDNN verbose log, to double confirm if oneDNN library and the incorrect result appear at the same time for all cases.
Not sure it's happed on all CPU or only those support AVX512 instrucions only.

We'll look into this.

Thanks! change the input data = [[[2.0, 3.0]]] to  data = [[[2.0, 1.0]]] 
computed:  1.7615942
expected:  0.8807970779778823

or data = [[[2.0, 4.0]]]
computed:  1.7615942
expected:  3.5231883119115293

It seemed the calculate process or input is not passed correctly.
I am not familiar to the process. 
Could you help to check this? Hi @xiguiw ,

I have an AMD system (3970X) that does not have AVX-512, so it's definetly not related to that.
I think what the tensorflow code computes is `sigmoid(data[0][0][0]) * data[0][0][0]` (wrong index on second multiplier.

Here the code to test it:
```
#!/usr/bin/env python3

import math
import os

import tensorflow as tf
import numpy as np

os.environ['ONEDNN_VERBOSE'] = '1'

def sigmoid(x):
  return 1 / (1 + math.exp(-x))

data = [[[2.0, 3.0]]]

@tf.function
def my_func(data):
    a, b = tf.split(data, 2, axis=2)
    sig = tf.sigmoid(a)
    m = sig * b
    return m

concrete_func = my_func.get_concrete_function(tf.constant(data))
g = concrete_func.graph.as_graph_def()

s = tf.compat.v1.Session()
tf.import_graph_def(g)
out = s.run(['mul:0'], feed_dict={'data:0': data})

print('computed: ', out[0][0][0])
print('expected: ', sigmoid(data[0][0][0]) * data[0][0][1])
print('expected error computation: ', sigmoid(data[0][0][0]) * data[0][0][0])
```

and here the output on my system:
```
2023-06-28 15:24:14.769259: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-06-28 15:24:16.354506: I tensorflow/core/common_runtime/process_util.cc:146] Creating new thread pool with default inter op setting: 2. Tune using inter_op_parallelism_threads for best performance.
2023-06-28 15:24:16.393142: I tensorflow/core/common_runtime/process_util.cc:146] Creating new thread pool with default inter op setting: 2. Tune using inter_op_parallelism_threads for best performance.
2023-06-28 15:24:16.394160: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:353] MLIR V1 optimization pass is not enabled
onednn_verbose,info,oneDNN v2.7.3 (commit N/A)
onednn_verbose,info,cpu,runtime:OpenMP,nthr:64
onednn_verbose,info,cpu,isa:Intel AVX2
onednn_verbose,info,gpu,runtime:none
onednn_verbose,info,prim_template:operation,engine,primitive,implementation,prop_kind,memory_descriptors,attributes,auxiliary,problem_desc,exec_time
onednn_verbose,exec,cpu,eltwise,jit:avx2,forward_training,data_f32::blocked:abc:f0 diff_undef::undef::f0,,alg:eltwise_swish alpha:1 beta:0,1x1x1,39.157
computed:  [1.7615942]
expected:  2.642391233933647
expected error computation:  1.7615941559557646
``` Hi @curufinwe,

Thank you for your feedback!

I agree with you that it's not related to oneDNN libary itself.
I changed data = [[[2.0, 3.0]]] to data = [[[2.0, 1.0]]] or data = [[[2.0, 4.0]]]
The output keep the same.

However, with or without oneDNN, the result are different.
My understanding is it's a graph calcuation in tensorflow. 
It creates a 'graph' and setup the calculation process internal.

It looks with or withou oneDNN, the calculation of graph is different (maybe fused ops are used in oneDNN).

I guess the input data are not passed in correctly.

Anyway, we are investigating this issue.
Thank you!

Xigui @curufinwe 
Intel TF team has a fix for it, and we are working on upstreaming the fix into TF.
The fix will be in future TF release soon.

thanks Glad to hear it. Thank you very much! @curufinwe ,

The proposed PR by intel team has been merged and changes might be reflected in next release. Could you please check and confirm whether the issue can be considered as resolved or wait till next release to check the results ?
 Hi @SuryanarayanaY,

I have applied the changes to a locally compiled version of tensorflow 1.13 and I do get the correct result now. This issue can be closed now.

Thank you for your efforts! Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/60941">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/60941">No</a>

<details><summary>Click to expand!</summary> 
 
 ### Issue Type

Bug

### Have you reproduced the bug with TF nightly?

No

### Source

binary

### Tensorflow Version

intel-tensorflow 2.8 - 2.12

### Custom Code

No

### OS Platform and Distribution

Ubuntu 22.04

### Mobile device

_No response_

### Python version

3.10

### Bazel version

_No response_

### GCC/Compiler version

_No response_

### CUDA/cuDNN version

_No response_

### GPU model and memory

_No response_

### Current Behaviour?

When running with an MKL enabled tensorflow (e.g. intel-tensorflow from pypi) (or self-compiled with `--config=mkl`). Starting with tensorflow 2.8.0 up until 2.12.0 The attached code produces the wrong numerical result. (1.7615 vs expected 2.6439).

If line 25 is changed to `m = sig * (b + 0.0)` one can get the correct result.
This issue does not occur if installing "vanilla" tesorflow from pip with `pip install tensorflow`.

This issue also does not occur if one uses `tf.exp` or `tf.log` instead of `tf.sigmoid`.

### Standalone code to reproduce the issue

```shell
#!/usr/bin/env python3

import math

import tensorflow as tf
import numpy as np

def sigmoid(x):
  return 1 / (1 + math.exp(-x))

data = [[[2.0, 3.0]]]

tf.compat.v1.disable_eager_execution()

s = tf.compat.v1.Session()
p = tf.compat.v1.placeholder(dtype=tf.float32)
a, b = tf.split(p, 2, axis=2)
sig = tf.sigmoid(a)
m = sig * b

out = s.run([p, m], feed_dict={p: data})
print(out)

print('computed: ', out[-1][0,0,0])
print('expected: ', sigmoid(data[0][0][0]) * data[0][0][1])
```


### Relevant log output

```shell
2023-06-21 18:37:35.713027: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-06-21 18:37:35.715264: I tensorflow/core/common_runtime/process_util.cc:146] Creating new thread pool with default inter op setting: 
[array([[[2., 3.]]], dtype=float32), array([[[1.7615942]]], dtype=float32)]
computed:  1.7615942
expected:  2.642391233933647
```
</details>