TFLITE mobilenetv2 conversion from tf leads to poor accuracy on imagenet evaluation tool
@suyash-narain 
Could you please provide detailed steps of the tflite imagenet evaluation tool to replicate the issue reported here ?

Thank you! > @suyash-narain Could you please provide detailed steps of the tflite imagenet evaluation tool to replicate the issue reported here ?
> 
> Thank you!

@tiruk007 Please find the detailed steps I exercised towards building the provided tflite imagenet evaluation tool:

- clone tensorflow from github:

> $ git clone "https://github.com/tensorflow/tensorflow.git"
> $ cd tensorflow

- Download ILSVRC validation dataset consisting of 50k images from  http://image-net.org/request: ILSVRC2012_img_val https://image-net.org/data/ILSVRC/2012/ILSVRC2012_img_val.tar
- Download ILSVRC 2012 dev kit: ILSVRC2012_devkit_t12 (tasks1&2) https://image-net.org/data/ILSVRC/2012/ILSVRC2012_devkit_t12.tar.gz
- Generate Ground Truth Labels:

> python /tensorflow/lite/tools/evaluation/tasks/imagenet_image_classification/generate_validation_labels.py \
> --ilsvrc_devkit_dir=path/to/ILSVRC2012_devkit_t12 \
> --validation_labels_output=output_labels.txt

- Download imagenet labels from https://storage.googleapis.com/download.tensorflow.org/data/ImageNetLabels.txt

- Build and run on Ubuntu desktop:

> bazel run -c opt \
>   -- \
>   //tensorflow/lite/tools/evaluation/tasks/imagenet_image_classification:run_eval \
>   --model_file=mbnv2_model_test1.tflite \
>   --ground_truth_images_path=path/to/ILSVRC2012_img_val \
>   --ground_truth_labels=path/to/output_labels.txt \
>   --model_output_labels=path/to/ImageNetLabels.txt \
>   --output_file_path=accuracy_output.txt \
>   --num_images=0 # Run on all images.
> 


When i run this on a tflite model sourced from tfhub: https://storage.googleapis.com/download.tensorflow.org/models/tflite_11_05_08/mobilenet_v2_1.0_224.tgz
i get the below results:

> INFO: Num evaluation runs: 50000
> INFO: Preprocessing latency: avg=3790.96(us), std_dev=0(us)
> INFO: Inference latency: avg=6675.06(us), std_dev=206(us)
> INFO: Top-1 Accuracy: 0.71922
> INFO: Top-2 Accuracy: 0.82604
> INFO: Top-3 Accuracy: 0.86718
> INFO: Top-4 Accuracy: 0.8892
> INFO: Top-5 Accuracy: 0.90452
> INFO: Top-6 Accuracy: 0.9152
> INFO: Top-7 Accuracy: 0.9234
> INFO: Top-8 Accuracy: 0.9298
> INFO: Top-9 Accuracy: 0.93518
> INFO: Top-10 Accuracy: 0.9394

Top1 ~72% which is very close to actual mobilenetv2 accuracy on imagenet.

But when i run the same tool on the converted tflite model which I had mentioned in issue, i get accuracy of 43% 
 Is it because of the way image is being processed before bring provided as input to the model?
I convert the model in the similar way as the mnist example provided here:
https://colab.research.google.com/github/tensorflow/tensorflow/blob/master/tensorflow/lite/g3doc/performance/post_training_integer_quant.ipynb

do i need to further preprocess the images? Hi @tiruk007 @pjpratik 
I changed my representative dataset, and used images from imagenet dataset for my representative dataset to convert model to int8 and uint8 format. 
On checking the accuracy against the imagenet evaluation tool, Igot the following accuracies:

**accuracy for INT8 tflite:**

> INFO: Num evaluation runs: 50000
> INFO: Preprocessing latency: avg=3807.62(us), std_dev=0(us)
> INFO: Inference latency: avg=5723.94(us), std_dev=96(us)
> INFO: Top-1 Accuracy: 0.63438
> INFO: Top-2 Accuracy: 0.7515
> INFO: Top-3 Accuracy: 0.80124
> INFO: Top-4 Accuracy: 0.83126
> INFO: Top-5 Accuracy: 0.8516
> INFO: Top-6 Accuracy: 0.86576
> INFO: Top-7 Accuracy: 0.87676
> INFO: Top-8 Accuracy: 0.88508
> INFO: Top-9 Accuracy: 0.89138
> INFO: Top-10 Accuracy: 0.89808
> 

But interesting thing to note is if i don't convert this model to int8 tflite and let it remain as default float32 tflite model, my accuracies are way lower:

**accuracy for fp32 tflite:** 
> 
> INFO: Num evaluation runs: 50000
> INFO: Preprocessing latency: avg=3837.7(us), std_dev=0(us)
> INFO: Inference latency: avg=16046.3(us), std_dev=486(us)
> INFO: Top-1 Accuracy: 0.43192
> INFO: Top-2 Accuracy: 0.5436
> INFO: Top-3 Accuracy: 0.60002
> INFO: Top-4 Accuracy: 0.6364
> INFO: Top-5 Accuracy: 0.66124
> INFO: Top-6 Accuracy: 0.68194
> INFO: Top-7 Accuracy: 0.69918
> INFO: Top-8 Accuracy: 0.71268
> INFO: Top-9 Accuracy: 0.72398
> INFO: Top-10 Accuracy: 0.73382

this is odd. shouldn't top1 accuracies of float32 be higher than INT8? mobilenetv2 fp32 should give me ~71% whereas int8 should give me about ~63% according to data mentioned here: https://www.tensorflow.org/lite/performance/model_optimization#quantization

 Hi @suyash-narain 

The input images for the tensorflow hub model are expected to have color values in the range [0,1], following the [common image input](https://www.tensorflow.org/hub/common_signatures/images#input) conventions as per the [documentation](https://tfhub.dev/google/imagenet/mobilenet_v2_100_224/classification/5).

Also, we might have to apply `softmax` activation as the model returns raw logits as per the this [colab](https://colab.research.google.com/github/tensorflow/hub/blob/master/examples/colab/image_classification.ipynb). Please check this [example](https://colab.research.google.com/github/tensorflow/hub/blob/master/examples/colab/image_classification.ipynb) on image classification using tensorflow hub.

Thanks. Hi @pjpratik 

thank you for your reply. So you mean to say I cannot use the imagenet classification evaluation tool for tflite with models sourced from tfhub since they require input image color values in range [0,1] but the imagenet tool fails with that?
the output from the imagenet tool seems like softmax activation is already applied as i get top classifications between [0-1]

how can i use the tflite imagenet classification tool against this mobilenetv2 modle sourced from tfhub to get correct accuracies? 

if i source the model from keras application layers as below:

> model = tf.keras.applications.mobilenet_v2.MobileNetV2(
>     input_shape=None,
>     alpha=1.0,
>     include_top=True,
>     weights='imagenet',
>     input_tensor=None,
>     pooling='avg',
>     classes=1000,
>     classifier_activation='softmax'
> )
> converter = tf.lite.TFLiteConverter.from_keras_model(model)
> tflite_file = "mobilenet_v2_keras_model.tflite"
> with open(tflite_file, 'wb') as f:
>        f.write(converter.convert())

in this case i get perfect accuracies (~71.9%) as mentioned in official documentation)  while running against imagenet evaluation tool provided by tflite. 
So what different thing i need to do against models sourced from tfhub to get similar accuracies as i am getting with keras application layer model?

thanks Hi @suyash-narain 

I have observed that the default normalization the tool does is for the output range `[-1,1]`. 

https://github.com/tensorflow/tensorflow/blob/0c7968df89e4e3edb7c48d6b0f50e3a9a477f81a/tensorflow/lite/tools/evaluation/stages/image_preprocessing_stage.h#L168

The Mobilenetv2 from  keras application applies the `tf` normalization which normalizes to the output range `[-1,1]`. Please check the source code below

https://github.com/keras-team/keras-applications/blob/06fbeb0f16e1304f239b2296578d1c50b15a983a/keras_applications/mobilenet.py#L84

https://github.com/keras-team/keras-applications/blob/06fbeb0f16e1304f239b2296578d1c50b15a983a/keras_applications/imagenet_utils.py#L42

For the tool to normalize the data in `[0,1]`, replace 

 `AddNormalizationStep(127.5, 1.0 / 127.5);` with

`AddNormalizationStep(0.0, 1.0 / 255.0);` and run the tool. I have tested with fp32 model converted from tf hub with code below
```
m = tf.keras.Sequential([hub.KerasLayer("https://tfhub.dev/google/imagenet/mobilenet_v2_100_224/classification/5")])

m.build([None, 224, 224, 3])# Batch input shape.


converter = tf.lite.TFLiteConverter.from_keras_model(m)
converter.optimizations = [tf.lite.Optimize.DEFAULT]

tflite_file = "mbnv2_model_fp32_test1.tflite"
with open(tflite_file, 'wb') as f:
  f.write(converter.convert())
``` 
and obtained the following results.

<img width="416" alt="Screenshot 2023-03-30 at 4 57 41 PM" src="https://user-images.githubusercontent.com/118897289/228825703-aab2094d-1e47-4995-8712-c8fde0d29fb5.png">

Thanks. thanks @pjpratik 
this worked for me. 

Is there a similar evaluation tool for pretrained tf models as well?
I am trying to use the same pretrained tf model on the same dataset https://image-net.org/data/ILSVRC/2012/ILSVRC2012_img_val.tar 
and ground truth labels generated using 

> python /tensorflow/lite/tools/evaluation/tasks/imagenet_image_classification/generate_validation_labels.py
> --ilsvrc_devkit_dir=path/to/ILSVRC2012_devkit_t12
> --validation_labels_output=output_labels.txt

How do i preprocess such a dataset wherein i have 50000 images in a test directory and a labels.txt file containing ground truth labels for these 50000 images and use the same against the pretrained mobilenet_v2 saved_model downloaded from: https://tfhub.dev/google/imagenet/mobilenet_v2_100_224/classification/5

thanks Hi @suyash-narain, glad it worked.

I see that a ticket has already been opened #60181 for that issue and it is being tracked there. 

Feel free to close this issue if it is resolved.

Thanks. Hi @pjpratik 

thanks for your reply. I had one more question. What are the official accuracies for mobilenet_v2 and quantized mobilenet_v2 on imagenet?

for fp32, I know it should be 72.9%

for int8/uint8 it is 63.7% (post training quantization) or 70.9% (quantization aware training) as mentioned here https://www.tensorflow.org/lite/performance/model_optimization#quantization

but when I evaluate the accuracy on quantized mobilenet_v2 model sourced from https://tfhub.dev/tensorflow/lite-model/mobilenet_v2_1.0_224_quantized/1/default/, I get accuracy of 71.5% which is higher than both the mentioned methods.

When I quantize model sourced from https://tfhub.dev/google/imagenet/mobilenet_v2_100_224/classification/5 and perform post training quantization on the same using a representative dataset consisting of 100 images from imagenet dataset, I still get a 71.30% accuracy for int8 and 71.25% accuracy for uint8 quantized tflite model when in reality i should be getting 63% right?

But when I perform post training quantization on model sourced from keras application layers,

> model = tf.keras.applications.mobilenet_v2.MobileNetV2(
> input_shape=None,
> alpha=1.0,
> include_top=True,
> weights='imagenet',
> input_tensor=None,
> pooling='avg',
> classes=1000,
> classifier_activation='softmax'
> )

in this case, I get an accuracy of 63.43% on int8 and 63.69% on uint8 quantized models while running the imagenet evaluation tool. 

which models were the original accuracies as mentioned in documentation obtained upon? 

thanks Hi @suyash-narain 

I think the reported accuracies are using the test set of 100,000 images whereas here it is being tested on val data. Incase of quantized tflite models, the models might have been trained using QAT and when tested on val data might report higher accuracies as. Inorder to reproduce the reported accuracies, we might need to replicate the experiments with the same test data the models are tested on. 

Thanks. This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you. > 

thanks @pjpratik 
I will close this issue Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/60072">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/60072">No</a>

### System information

-   **Have I written custom code (as opposed to using a stock example script
    provided in TensorFlow)**: No
-   **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: Ubuntu 20.04
-   **TensorFlow installed from (source or binary)**: Binary
-   **TensorFlow version (use command below)**: 2.11
-   **Python version**: 3.9

### Describe the problem
Mobilenetv2 converted from TF to TFLITE using default MLIR converter leads to poor accuracy on tflite imagenet evaluation tool. On imagenet evaluation tool I get the below output:

> INFO: Num evaluation runs: 50000
> INFO: Preprocessing latency: avg=3837.7(us), std_dev=0(us)
> INFO: Inference latency: avg=16046.3(us), std_dev=486(us)
> INFO: Top-1 Accuracy: 0.43192
> INFO: Top-2 Accuracy: 0.5436
> INFO: Top-3 Accuracy: 0.60002
> INFO: Top-4 Accuracy: 0.6364
> INFO: Top-5 Accuracy: 0.66124
> INFO: Top-6 Accuracy: 0.68194
> INFO: Top-7 Accuracy: 0.69918
> INFO: Top-8 Accuracy: 0.71268
> INFO: Top-9 Accuracy: 0.72398
> INFO: Top-10 Accuracy: 0.73382

I used imagenet ILSVRC2012_img_val dataset with mobilenet labels downloaded from https://storage.googleapis.com/download.tensorflow.org/data/ImageNetLabels.txt. I ran the evaluation on all 50000 images. Top1 accuracy came to 43% whereas official mobilenetv2 top1 accuracy is ~72%. I did not quantize the model and used the default float32. 
what can be the cause of this delta in accuracy? If i remove `converter.optimizations = [tf.lite.Optimize.DEFAULT]` from script, 
 i still get 43% Top1 accuracy on imagenet. Shouldn't tflite models have similar accuracy to tf models?

When i quantize the model to use uint8/int8, i get even worse accuracy to the tune of 0.4%.

below is the output log for a uint8 model converted from mobilenetv2:

> INFO: Num evaluation runs: 50000
> INFO: Preprocessing latency: avg=3801.64(us), std_dev=0(us)
> INFO: Inference latency: avg=5958.53(us), std_dev=166(us)
> INFO: Top-1 Accuracy: 0.0044
> INFO: Top-2 Accuracy: 0.00758
> INFO: Top-3 Accuracy: 0.01062
> INFO: Top-4 Accuracy: 0.01338
> INFO: Top-5 Accuracy: 0.01634
> INFO: Top-6 Accuracy: 0.0191
> INFO: Top-7 Accuracy: 0.02216
> INFO: Top-8 Accuracy: 0.02482
> INFO: Top-9 Accuracy: 0.02758
> INFO: Top-10 Accuracy: 0.03016


### Source code / logs
Below code is used for conversion:
1. fp32: 

> import tensorflow as tf
> import tensorflow_hub as hub
> 
> m = tf.keras.Sequential([hub.KerasLayer("https://tfhub.dev/google/imagenet/mobilenet_v2_100_224/classification/5")])
> m.build([None, 224, 224, 3])# Batch input shape.
> 
> m.save('model')
> 
> converter = tf.lite.TFLiteConverter.from_saved_model("/content/model")
> converter.target_spec.supported_ops = [
>     tf.lite.OpsSet.TFLITE_BUILTINS,  # enable TensorFlow Lite ops.
>     tf.lite.OpsSet.SELECT_TF_OPS,  # enable TensorFlow ops.
> ]
> converter.optimizations = [tf.lite.Optimize.DEFAULT]
> 
> tflite_file = "mbnv2_model_test1.tflite"
> with open(tflite_file, 'wb') as f:
>   f.write(converter.convert())

generated model is attached in zip file. 
[mbnv2_model_test1.zip](https://github.com/tensorflow/tensorflow/files/11044725/mbnv2_model_test1.zip)


2. uint8:

> import tensorflow as tf
> import tensorflow_hub as hub
> 
> m = tf.keras.Sequential([hub.KerasLayer("https://tfhub.dev/google/imagenet/mobilenet_v2_100_224/classification/5")])
> m.build([None, 224, 224, 3])# Batch input shape.
> 
> m.save('model')
> 
> import numpy as np
> def representative_dataset():
>     for _ in range(100):
>         data = tf.random.normal([1,224,224,3])
>         yield [(tf.cast(data, tf.float32) / 127.5) - 127.5]
> 
> converter = tf.lite.TFLiteConverter.from_saved_model("/content/model")
> converter.target_spec.supported_ops = [ tf.lite.OpsSet.TFLITE_BUILTINS_INT8] # enable TensorFlow int8ops.
> converter.optimizations = [tf.lite.Optimize.DEFAULT]
> converter.representative_dataset = representative_dataset
> converter.inference_input_type = tf.uint8  # or tf.int8
> converter.inference_output_type = tf.uint8  # or tf.int8
> 
> tflite_file = "mbnv2_model_uint8_test1.tflite"
> with open(tflite_file, 'wb') as f:
>   f.write(converter.convert())

generated model is attached [mbnv2_model_uint8_test1.zip](https://github.com/tensorflow/tensorflow/files/11044895/mbnv2_model_uint8_test1.zip)