Comparing a large/general CNN to a smaller more specialized one?
After reading your question I can relate it to the Representation Learning papers such as SimCLR and SwAV. These models use a &quot;Big Task agnostic CNN&quot; to obtain smaller representations of the images and then they train another CNN for classification. I suggest you read Big Self-Supervised Models are Strong Semi-Supervised Learners by Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi and Geoffrey Hinton. The code for the following can be found here. But I feel that training such a model would take up a lot of computational resources.