Internal quantize ops don't match external quantization
 Hi @pkgoogle,

I have verified that the weights and biases are the same for both the model but zero point and scale are not the same for these two models. Please find the [gist.](https://colab.research.google.com/gist/LakshmiKalaKadali/4b031619787edc6b8952d653cb3f3009/external_internal_quantize-tf2-15.ipynb) Could you please look into the issue. 


Thank You I tried looking at the MLIR representations:

Internal model:
```sh
$ ./flatbuffer_translate --tflite-flatbuffer-to-mlir MNIST_PTQ_internal.tflite
...
%0 = "tfl.quantize"(%arg0) {qtype = tensor<?x28x28x!quant.uniform<i8:f32, 0.0039215688593685627:-128>>} : (tensor<?x28x28xf32>) -> tensor<?x28x28x!quant.uniform<i8:f32, 0.0039215688593685627:-128>>
...
%10 = "tfl.dequantize"(%9) : (tensor<?x10x!quant.uniform<i8:f32, 3.906250e-03:-128>>) -> tensor<?x10xf32>
```

External model:
```sh
$ ./flatbuffer_translate --tflite-flatbuffer-to-mlir MNIST_PTQ_external.tflite
...
func.func @main(%arg0: tensor<?x28x28x!quant.uniform<i8:f32, 0.0039215688593685627:-128>> {tf_saved_model.index_path = ["flatten_input"]}) -> (tensor<?x10x!quant.uniform<i8:f32, 3.906250e-03:-128>> {tf_saved_model.index_path = ["softmax"]}) attributes {tf.entry_function = {inputs = "serving_default_flatten_input:0", outputs = "StatefulPartitionedCall:0"}, tf_saved_model.exported_names = ["serving_default"]}
...
return %8 : tensor<?x10x!quant.uniform<i8:f32, 3.906250e-03:-128>>
```

As you said the scale factors are equivalent. I tried explicitly casting the external computations to np.float32 (just in case we got the rare python floats are actually 64bit issues) but that didn't change anything. I also cloned the colab and did each model in their separate notebooks and still got this discrepancy (just in case there's some randomization from quantizing/evaluating in a different order or one after another).

@abattery, can you please take a look? Thanks.

 Hi @EClemMarq , if you are able to access a linux system you may be able to resolve your issue by using [AI-Edge-Torch](https://github.com/google-ai-edge/ai-edge-torch), you can find more information here: [googleblog](https://developers.googleblog.com/en/ai-edge-torch-high-performance-inference-of-pytorch-models-on-mobile-devices/).

I have actually created a simple script for converting your model here , you can quantize your model after this step:

```
import torch
import torch.nn as nn
import torch.nn.functional as F
import ai_edge_torch

class SimpleNN(nn.Module):
    def __init__(self):
        super(SimpleNN, self).__init__()
        self.flatten = nn.Flatten()
        self.fc1 = nn.Linear(28 * 28, 128)
        self.dropout = nn.Dropout(0.2)
        self.fc2 = nn.Linear(128, 10)
        self.softmax = nn.Softmax(dim=1)

    def forward(self, x):
        x = self.flatten(x)
        x = F.relu(self.fc1(x))
        x = self.dropout(x)
        x = self.fc2(x)
        x = self.softmax(x)
        return x

model = SimpleNN()

example_input = torch.randn(1, 1, 28, 28)

edge_model = ai_edge_torch.convert(model.eval(), (example_input,))

edge_model.export('simple_nn.tflite')
```


If you want to, you can actually try visualizing the result in [model-explorer](https://github.com/google-ai-edge/model-explorer) as well.

Please try them out and let us know if this resolves your issue. If you still need further help, feel free to open a new issue at the respective repo.


### 1. System information

- Occurs in Google Colab w/ TF 2.14
- Have also verified w. TF 2.7 (Anaconda) on Windows 10

### 2. Code

[Colab to reproduce issue](https://colab.research.google.com/drive/17aDzDSG-1DcE3sJn16B4Q4zd9QewTXkT?usp=sharing)

The Colab trains a basic MNIST model (deterministically, so your model should be the same as mine) and performs Post-Training Quantization.  One quantized model (left side of image) uses INT8 input/output, requiring the provided external quant/de-quant code.  The other quantized model (right side of image) takes FP32 input, embedding the quantize/dequantize layers into the model graph.

![image](https://github.com/tensorflow/tensorflow/assets/79734663/a91ab1cc-912e-455b-a711-1b433b736088)

### 3. Failure after conversion
Conversion executes successfully, but model outputs differ.  Why? Shouldn't the two methods be interchangeable?

I have verified that the weights, biases, quantization zero points and quantization scales match between the two models.

I have also run into this issue using a CPU-only Colab runtime, meaning it doesn't happen because one approach quantizes on the CPU while the other quantizes on the GPU/TPU.




Obviously, the difference between 0.5938 and 0.5468 is not significant in this MNIST model.  However, in applications where softmax outputs are averaged to produce a final output, such small discrepancies can significantly alter model accuracy.  Thank you!