Uncompliant tflite model when converting "MultiHeadAttention" layer
Hi @kobygold, Can you help me out my providing the code you use to 

> when trying to run this model on [tflite-micro](https://github.com/tensorflow/tflite-micro) engine (C/C++ engine) it fail with assert!

I've tried reproducing on my end, with your model and the hello-world micro example but I am running into a different issue, b/c I want to actually resolve your issue and not my own, if you can help me short-cut the process that'll be great. A minimally reproducible example would be best. 

The reason I want to reproduce is I am unsure if that assert is the same as expecting the weights to have a 0 zero-point. Though assymetric_quantization == false and a non-zero zero-point seems inconsistent at the very least for the FC layer. If we can reproduce we can be 100% sure this is an issue. Thanks for your help. Dear @pkgoogle,
thanks for your help!

tflite-micro gave us a bit of hard time as well at the beginning, but now we have it working flawlessly.
We are running many models on that engine without issues, so I can tell for sure that this issue is due to the asymmetric quantization of the weights. We will try to create a minimally reproducible example code and send you, but it may take a day or two to prepare... In the meanwhile I can show you a screenshot from my visual studio environment with this assert, and you can see that the failure is exactly on the zero-point = -18 (and can also see the output zero-point = 28, matching the model, which is OK):

![image](https://github.com/tensorflow/tensorflow/assets/6744658/0f756762-859d-4559-8b0b-a02747f83f7e)

![image](https://github.com/tensorflow/tensorflow/assets/6744658/0c8dc559-d8ca-4193-bcf4-5f897e536d61)

Until our reproducible code is ready it would be great if you could take a look at the tflite converter code for this "MultiHeadAttention" layer, to see how difficult it would be to fix the quantization of those tensors.

Thanks,
Koby Hi @pkgoogle ,
We've created a fork of the tflite-micro repo on github with this model inside. Here is the code:
https://github.com/kslavka/tflite-micro-asymmetric

To build use:
`make -f tensorflow/lite/micro/tools/make/Makefile network_tester_test
`

To run use:
`./gen/linux_x86_64_default/bin/network_tester_test
`

I hope this helps.

Thanks,
Koby
 Hi @kobygold, thanks for the information it definitely helps.

@abattery can you please take a look? Thanks. @pkgoogle @abattery 
Hi guys,
Any update...?
 Hi guys,
Can you give some life signs...?
Is someone watching this issue...?
I'm checking for updates every few days, and waiting patiently, but please let me know that someone is looking at it...
Is there something I can help with?

Thanks,
Koby
 Hi @kobygold, apologies, there are many simultaneous issues. There are ways you can help yourself, it is an open source framework after all. Please review https://github.com/tensorflow/tensorflow/blob/master/CONTRIBUTING.md, if you are able to fix the issue please feel free to submit a PR.

Everyone debugs differently, but one option is to use a debugger, generally I recommend gdb for C++ code. Thanks @pkgoogle, 
debugging C++ code is my bread & butter :-)
But in this case the python converter is the problem...
I need to dive deep into the tflite converter (which is a pretty complicated code) and change those Fully Connected weights to use symmetric quantization, or alternatively replace those special Fully Connected layers with some kind of MUL operator that would implement matrix multiplication (y = A*x) if possible. The MUL operator supports the two inputs as asymmetric quantized, so maybe that could be easier... Do you know if matrix multiplication can be implemented by tflite MUL operator (with some "broadcast" mode perhaps? Simple element-wise is not the right mathematical operation here)... I'm not an expert at that... Where can I see the various MUL options, mathematical equations? I would love to contribute, but need a bit of guidance at the right direction on the tflite converter... 

Thanks,
Koby  @kobygold, haha yes, I am familiar with the complexity as well. So one of the reasons the converter is hard is because a general TF (graph) program can be thought in many ways as a general program, converting to TFLite is essentially kind of, sort of, going through multiple passes of converting one program to another. This is very similar to how compiler's go through passes of an AST.

This is probably a good readme to start
https://github.com/tensorflow/tensorflow/blob/master/tensorflow/compiler/mlir/lite/README.md

Then I recommend just putting a breakpoint in the C++ code and start exploring. Also let me update my gdb recommendation, It's the most reliable one but if you can use lldb, it will have better integration with mlir/llvm.

Kernels are where most of the math happens: https://github.com/tensorflow/tensorflow/tree/master/tensorflow/lite/kernels
### 1. System information

- OS Platform and Distribution: WIndows 10
- TensorFlow installation: pip package
- TensorFlow library (version): 2.13.0

### 2. Code

```
import tensorflow as tf
from tensorflow.keras import layers
import numpy as np

def model(q, v):
    x = layers.MultiHeadAttention(num_heads=2, key_dim=2)(q, v)
    return x

def representative_dataset():
    for _ in range(100):
        q = np.log(np.random.random((8, 16)))
        v = np.log(np.random.random((4, 16)))

        yield [q.astype(np.float32), v.astype(np.float32)]

target = tf.keras.Input(shape=[8, 16])
source = tf.keras.Input(shape=[4, 16])
out = model(target, source)
model = tf.keras.Model(inputs=(target, source), outputs=out)
model.summary()

model.save('MultiHeadAttention.h5')

run_model = tf.function(model)
# let's fix the input size.
concrete_func = run_model.get_concrete_function((
    tf.TensorSpec([1, 8, 16], model.inputs[0].dtype), tf.TensorSpec([1, 4, 16], model.inputs[0].dtype)))

# model directory.
MODEL_DIR = "MultiHeadAttention"
model.save(MODEL_DIR, save_format="tf", signatures=concrete_func)

converter = tf.lite.TFLiteConverter.from_saved_model(MODEL_DIR)
converter.optimizations = [tf.lite.Optimize.DEFAULT]
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
converter.representative_dataset = representative_dataset
converter.inference_input_type = tf.int8  # or tf.uint8
converter.inference_output_type = tf.int8  # or tf.uint8
tflite_model = converter.convert()
# Save the model.
with open('MultiHeadAttention.tflite', 'wb') as f:
    f.write(tflite_model)

print(f'Created model: MultiHeadAttention.tflite')
```

generated tf (h5) model: [link](https://drive.google.com/file/d/1QuSu77qI6o5y6bnoGzFJanlO6NP-lQXz/view?usp=sharing)
generated tflite model: [link](https://drive.google.com/file/d/1_DA5bJleAhLeBxGqmj3dfnK8u0ZhZiu5/view?usp=sharing)


### 3. Failure after conversion
The conversion to quantized 8bit is successful, but the generated model has problems:
The "MultiHeadAttention" layer is converted to many simpler layers on the tflite model (which is fine),
but among those layers there are 4 fully connected (FC) layers that are not tflite compliant and when trying to run this model on  [tflite-micro](https://github.com/tensorflow/tflite-micro) engine (C/C++ engine) it fail with assert! The problem with those 4 FC layers is that their weights quantization is asymmetric where one of the fundamental assumptions of tflite quantization is that the weights (on FC or CONV layers) should be symmetric, i.e. the "zero-point" should be zero. And as can be seen on my model screenshot the zero_point is 18. On the layer attributes it looks like "assymetric_quantization = false", but that is not correct, as we see the zero-point is not 0.

![image](https://github.com/tensorflow/tensorflow/assets/6744658/3319d61a-0230-46cc-b985-fafe7b541bbd)

See more about the tflite weights symmetric quantization requirement [here](https://www.tensorflow.org/lite/performance/quantization_spec#:~:text=Weights%20are%20symmetric). 

See also the assert check in the tflite-micro code [here](https://github.com/tensorflow/tflite-micro/blob/main/tensorflow/lite/micro/kernels/fully_connected_common.cc).
Lines 67-71:
```
    // Filter weights will always be symmetric quantized since we only support
    // int8 quantization. See
    // https://github.com/tensorflow/tensorflow/issues/44912 for additional
    // context.
    TFLITE_DCHECK(filter->params.zero_point == 0);

```

**Note**: those FC layers take their weights from previous layers outputs (unlike simple FC layers where the weights are known in advance). Those layer outputs should have symmetric quantization to be qualified as FC weights. 
