Why are results without Transfer Learning better than with Transfer Learning?
As @fuwiak mentioned, transfer learning may not work if pre-trained model has been fitted on a &quot;very different&quot; dataset. Typically if the pre-trained network extract information that is not relevant for your problem.
Moreover, in the paper License Plate Recognition System Based on Transfer Learning (that you shared with me), they have tried to freeze some layers of a pretrained Xception (based on ImageNet weights) to see the impact on the training. They conclude that ImageNet data and license plate data are too different to freeze layers. So your results are confirmed.
Now changing efnB0_model.trainable = False to True would allow the pre-trained network to update and to be more relevant to your problem. Generally, if you don't have time issues, it seems to be always better (see this post). Will it give better results than initialize the weights randomly ? I think one can presume but cannot know.