How to evaluate a pretrained TF mobilenet_v2 saved_model for accuracy on test dataset
it seems that the labels in the validation set are not in the correct format for the model's output. MobileNetV2 outputs a 1001-dimensional vector, where each element represents the probability of the input image belonging to a specific class. The validation set labels are just integers representing the class index.

To preprocess the data, you need to convert the labels to one-hot vectors. You can use the tf.one_hot function for this purpose > it seems that the labels in the validation set are not in the correct format for the model's output. MobileNetV2 outputs a 1001-dimensional vector, where each element represents the probability of the input image belonging to a specific class. The validation set labels are just integers representing the class index.
> 
> To preprocess the data, you need to convert the labels to one-hot vectors. You can use the tf.one_hot function for this purpose

Hi @sunilgiri7 @sushreebarsa 
I was missing the one-hot vectors and once I converted the labels to one-hot, the error disappeared. But on evaluation, I was getting really bad accuracy on test data, to the tune of 0.1% accuracy on pretrained mobilenetv2 sourced from tfhub, which is totally incorrect.

- for test data, I was using imagenet_val dataset consisting of 50000 images. 
- val.txt consists of 50000 ground truth labels for the imagenet_val dataset.
- ImageNetLabels.txt was downloaded from  https://storage.googleapis.com/download.tensorflow.org/data/ImageNetLabels.txt 

on using the below script:

```
import tensorflow as tf
import tensorflow_hub as hub
import numpy as np
import os

m = tf.keras.Sequential([hub.KerasLayer("https://tfhub.dev/google/imagenet/mobilenet_v2_100_224/classification/4", output_shape=(1001,))])
m.build([None, 224, 224, 3]) # Batch input shape.

images = '/home/Downloads/ILSVRC2012_img_val/'
classes = '/home/Documents/tflite_models/imagenet_classes.txt'
labels ='/home/Documents/val.txt'
#labels ={}

with open(labels, 'r') as f:
	label_name = [line.strip() for line in f.readlines()]

class_map = {}
with open(classes, 'r') as f:
	classes = [line.strip() for line in f]
	for i, class_name in enumerate(classes):
		class_map[class_name] = i

test_labels=[]
for label in label_name:
	if label in class_map:
		test_labels.append(class_map[label])
	else:
		print(f"label '{label} not found in class_map")

#print(type(test_labels))
test_labels = tf.one_hot(test_labels, 1001)

image_paths = [os.path.join(images, filename) for filename in os.listdir(images)]

dataset = tf.data.Dataset.from_tensor_slices((image_paths, test_labels))

def preprocess_image(image_path):
	image = tf.io.read_file(image_path)
	image = tf.image.decode_jpeg(image, channels=3)
	image = tf.image.resize(image, [224,224])
	#image = tf.image.convert_image_dtype(image, tf.float32)
	image = tf.cast(image, tf.float32) / 255.0
	return image

dataset = dataset.map(lambda image_path, label: (preprocess_image(image_path), label))
dataset = dataset.batch(batch_size=32)

model.compile(loss='categorical_crossentropy', optimizer='rmsprop', metrics=['accuracy'])
loss, accuracy = model.evaluate(dataset)

print('loss: ', loss)
print('accuracy: ', accuracy)
```

i get the results as:

> loss:  10.239829063415527
> accuracy:  0.0009599999757483602

am i missing something here?
my val.txt and imagenetlabels.txt file are attached

thanks
[val.txt](https://github.com/tensorflow/tensorflow/files/11127384/val.txt)
[ImageNetLabels.txt](https://github.com/tensorflow/tensorflow/files/11127385/ImageNetLabels.txt)


 issue might be with the evaluation process or the dataset suyash here is
some possibilities>
- make sure image is in correct format(JPEG)
- check preprocess function working correctly
- check if class_map dict map each class name to correct index you could
print out dict and compare to ImageNetLabel.text if there any mismatch
- Try evaluating the model on a smaller subset of the dataset (e.g. 100
images) to see if the accuracy improves. If it does, then the issue might
be with the size of the dataset or the distribution of the classes
- Finally, you could try retraining the model on the ImageNet dataset
hope this helps you suyash

On Fri, 31 Mar 2023 at 15:24, Suyash ***@***.***> wrote:

> it seems that the labels in the validation set are not in the correct
> format for the model's output. MobileNetV2 outputs a 1001-dimensional
> vector, where each element represents the probability of the input image
> belonging to a specific class. The validation set labels are just integers
> representing the class index.
>
> To preprocess the data, you need to convert the labels to one-hot vectors.
> You can use the tf.one_hot function for this purpose
>
> Hi @sunilgiri7 <https://github.com/sunilgiri7>
> I was missing the one-hot vectors and once I converted the labels to
> one-hot, the error disappeared. But on evaluation, I was getting really bad
> accuracy on test data, to the tune of 0.1% accuracy on pretrained
> mobilenetv2 sourced from tfhub, which is totally incorrect.
>
>    - for test data, I was using imagenet_val dataset consisting of 50000
>    images.
>    - val.txt consists of 50000 ground truth labels for the imagenet_val
>    dataset.
>    - ImageNetLabels.txt was downloaded from
>    https://storage.googleapis.com/download.tensorflow.org/data/ImageNetLabels.txt
>
> on using the below script:
>
> import tensorflow as tf
> import tensorflow_hub as hub
> import numpy as np
> import os
>
> m = tf.keras.Sequential([hub.KerasLayer("https://tfhub.dev/google/imagenet/mobilenet_v2_100_224/classification/4", output_shape=(1001,))])
> m.build([None, 224, 224, 3]) # Batch input shape.
>
> images = '/home/mtk/Downloads/ILSVRC2012_img_val/'
> classes = '/home/mtk/Documents/tflite_models/imagenet_classes.txt'
> labels ='/home/mtk/Documents/val.txt'
> #labels ={}
>
> with open(labels, 'r') as f:
> 	label_name = [line.strip() for line in f.readlines()]
>
> class_map = {}
> with open(classes, 'r') as f:
> 	classes = [line.strip() for line in f]
> 	for i, class_name in enumerate(classes):
> 		class_map[class_name] = i
>
> test_labels=[]
> for label in label_name:
> 	if label in class_map:
> 		test_labels.append(class_map[label])
> 	else:
> 		print(f"label '{label} not found in class_map")
>
> #print(type(test_labels))
> test_labels = tf.one_hot(test_labels, 1001)
>
> image_paths = [os.path.join(images, filename) for filename in os.listdir(images)]
>
> dataset = tf.data.Dataset.from_tensor_slices((image_paths, test_labels))
>
> def preprocess_image(image_path):
> 	image = tf.io.read_file(image_path)
> 	image = tf.image.decode_jpeg(image, channels=3)
> 	image = tf.image.resize(image, [224,224])
> 	#image = tf.image.convert_image_dtype(image, tf.float32)
> 	image = tf.cast(image, tf.float32) / 255.0
> 	return image
>
> dataset = dataset.map(lambda image_path, label: (preprocess_image(image_path), label))
> dataset = dataset.batch(batch_size=32)
>
> model.compile(loss='categorical_crossentropy', optimizer='rmsprop', metrics=['accuracy'])
> loss, accuracy = model.evaluate(dataset)
>
> print('loss: ', loss)
> print('accuracy: ', accuracy)
>
> i get the results as:
>
> loss: 10.239829063415527
> accuracy: 0.0009599999757483602
>
> am i missing something here?
> my val.txt and imagenetlabels.txt file are attached
>
> thanks
> val.txt <https://github.com/tensorflow/tensorflow/files/11127384/val.txt>
> ImageNetLabels.txt
> <https://github.com/tensorflow/tensorflow/files/11127385/ImageNetLabels.txt>
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/tensorflow/tensorflow/issues/60181#issuecomment-1492676976>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AZI5F7EXRWQ6OILLTWHEEF3W65KSXANCNFSM6AAAAAAWN7ZE2M>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>
 @suyash-narain Could you refer to the comment above and let us know if there is any update?
Thank you! This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you. > @suyash-narain Could you refer to the comment above and let us know if there is any update? Thank you!

Hi, @sushreebarsa, apologies for the late reply. 
Is it possible for you to provide with a sample script on how to preprocess data from a subset of imagenet correctly? My current method took into consideration all the bullet points asked by you and yet my accuracy lies around 0-1%, which is surely incorrect.
thanks  @suyash-narain Sorry for the late response!
On subsets with fewer training images, the model needs to be trained for more epochs and the hyper-parameters need to be adjusted as a requirement. This will give a better performance of the model.
For preprocessing data from subset of imagenet kindly have a look at [this](https://medium.com/swlh/subsets-of-imagenet-aca5342eb275) article.

I tried to replicate the issue reported here and faced a different error. Please find the gist and let me know if I am missing something [here](https://colab.research.google.com/gist/sushreebarsa/b0583e20aff5bbf75683cc2aa71a45de/60181.ipynb?pli=1#scrollTo=HtPjd6ONypnv).  
Thank you! This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you. This issue was closed because it has been inactive for 7 days since being marked as stale. Please reopen if you'd like to work on this further. Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/60181">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/60181">No</a>


### System information

-   **Have I written custom code (as opposed to using a stock example script
    provided in TensorFlow)**: No
-   **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: Ubuntu 20.04
-   **TensorFlow installed from (source or binary)**: binary
-   **TensorFlow version (use command below)**:2.11
-   **Python version**: 3.9

### Describe the problem
How can i use a pretrained saved_model and find its accuracy on a test dataset?

I have mobilenet_v2 saved model which is sourced from https://tfhub.dev/google/imagenet/mobilenet_v2_100_224/classification/5

I have an imagenet validation dataset consisting of 50000 images, and a labels.txt file consisting of ground truth labels for those 50000 images. 

I also have ImageNetLabels.txt sourced from https://storage.googleapis.com/download.tensorflow.org/data/ImageNetLabels.txt consisting of 1001 imagenet classes. 

How do I preprocess this data so that i can run evaluate() function to find test_data loss and accuracy of this pretrained model?

I am currently using the below script, but it doesn't seem to work:

```
import tensorflow as tf
import tensorflow_hub as hub
import numpy as np
import os

m = tf.keras.Sequential([hub.KerasLayer("https://tfhub.dev/google/imagenet/mobilenet_v2_100_224/classification/4", output_shape=(1001,))])
m.build([None, 224, 224, 3])# Batch input shape.
images = '/home/Downloads/ILSVRC2012_img_val'
classes = '/home/Documents/ImageNetLabels.txt'
labels ='/home/Documents/val.txt'

with open(labels, 'r') as f:
	label_name = [line.strip() for line in f.readlines()]

class_map = {}
with open(classes, 'r') as f:
	classes = [line.strip() for line in f]
	for i, class_name in enumerate(classes):
		class_map[class_name] = i

test_labels=[]
for label in label_name:
	if label in class_map:
		test_labels.append(class_map[label])
	else:
		print(f"label '{label} not found in class_map")

image_paths = [os.path.join(images, filename) for filename in os.listdir(images)]
dataset = tf.data.Dataset.from_tensor_slices((image_paths, test_labels))
def preprocess_image(image_path):
	image = tf.io.read_file(image_path)
	image = tf.image.decode_jpeg(image, channels=3)
	image = tf.image.resize(image, [224,224])
	image = tf.image.convert_image_dtype(image, tf.float32)
	image /= 255.0
	return image

dataset = dataset.map(lambda image_path, label: (preprocess_image(image_path), label))

dataset = dataset.batch(batch_size=32)
m.compile(loss='categorical_crossentropy', optimizer='rmsprop', metrics=['accuracy'])
loss, accuracy = m.evaluate(dataset)
print('loss: ', loss)
print('accuracy: ', accuracy)
```

I get the below error here:

> Traceback (most recent call last):
>   File "test1234.py", line 74, in <module>
>     loss, accuracy = m.evaluate(dataset)
>   File "/home/mtk/.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 70, in error_handler
>     raise e.with_traceback(filtered_tb) from None
>   File "/tmp/__autograph_generated_filemqiwcebs.py", line 15, in tf__test_function
>     retval_ = ag__.converted_call(ag__.ld(step_function), (ag__.ld(self), ag__.ld(iterator)), None, fscope)
> ValueError: in user code:
> 
>     File "/home/mtk/.local/lib/python3.8/site-packages/keras/engine/training.py", line 1820, in test_function  *
>         return step_function(self, iterator)
>     File "/home/mtk/.local/lib/python3.8/site-packages/keras/engine/training.py", line 1804, in step_function  **
>         outputs = model.distribute_strategy.run(run_step, args=(data,))
>     File "/home/mtk/.local/lib/python3.8/site-packages/keras/engine/training.py", line 1792, in run_step  **
>         outputs = model.test_step(data)
>     File "/home/mtk/.local/lib/python3.8/site-packages/keras/engine/training.py", line 1758, in test_step
>         self.compute_loss(x, y, y_pred, sample_weight)
>     File "/home/mtk/.local/lib/python3.8/site-packages/keras/engine/training.py", line 1082, in compute_loss
>         return self.compiled_loss(
>     File "/home/mtk/.local/lib/python3.8/site-packages/keras/engine/compile_utils.py", line 265, in __call__
>         loss_value = loss_obj(y_t, y_p, sample_weight=sw)
>     File "/home/mtk/.local/lib/python3.8/site-packages/keras/losses.py", line 152, in __call__
>         losses = call_fn(y_true, y_pred)
>     File "/home/mtk/.local/lib/python3.8/site-packages/keras/losses.py", line 284, in call  **
>         return ag_fn(y_true, y_pred, **self._fn_kwargs)
>     File "/home/mtk/.local/lib/python3.8/site-packages/keras/losses.py", line 2004, in categorical_crossentropy
>         return backend.categorical_crossentropy(
>     File "/home/mtk/.local/lib/python3.8/site-packages/keras/backend.py", line 5532, in categorical_crossentropy
>         target.shape.assert_is_compatible_with(output.shape)
> 
>     ValueError: Shapes (None, 1) and (None, 1001) are incompatible

I assume something is wrong in the way I preprocess my data though, but not sure how to go about it. Some insights would be nice
thanks