Correct way to take advantage of Resizing layer in Tensorflow 2.3?
models needs to operate on images of a FIXED size. If you train a model with a fixed size for example (224 X 224) then if you want to use the trained model to make predictions on images you need to resize those images to 224 X 224. Specifically whatever pre-processing you did on the training images you should also do on the images that you wish to predict. For example if your model was trained on RGB images but the images you want to predict are say BGR images (like reading in images with CV2) the results will be incorrect. You would need to convert them to RGB . Similarly if you rescaled you training images by dividing by 255 you should also rescale the images you want to predict.
Tensorflow 2.3 introduced new preprocessing layers, such as tf.keras.layers.experimental.preprocessing.Resizing.
However, the typical flow to train on images with Keras uses tf.keras.preprocessing.image.ImageDataGenerator, which can only take a fixed target_size parameter. As far as I understand, the root cause is that keras is handling images as a numpy array in the background, where all images have to be the same size (is that true?).
While I could then use a model with a resizing layer that was trained on a fixed size to then predict images of arbitrary size, this seems to be risky since the training data and inference data would have systematic differences. One workaround could be to use ImageDataGenerator with a target_size and interpolation method that match the ones of the resizing layer, so that during training the resizing layer basically does nothing, but then it seems that the resizing layer is not really of any benefit.
So the question is, is there a way to train directly on mixed size images to fully take advantage of the resizing layer?