Can we apply transfer learning between any two different CNN architectures? There are many types of CNN architectures: LeNet, AlexNet, VGG, GoogLeNet, ResNet, etc. Can we apply transfer learning between any two different CNN architectures? For instance, can we apply transfer learning from AlexNet to GoogLeNet, etc.? Or even just from a &quot;conventional&quot; CNN to one of these other architectures, or the other way around? Is this possible in general?
EDIT: My understanding is that all machine learning models have the ability to perform transfer learning. If this is true, then I guess the question is, as I said, whether we can transfer between two different CNN architectures – for instance, what was learned by a conventional CNN to a different CNN architecture.
No, transfer learning cannot be applied &quot;between&quot; different architectures, as transfer learning is the practice of taking a neural network that has already been trained on one task and retraining it on another task with the same input modality, which means that only the weights (and other trainable parameters) of the network change during transfer learning but not the architecture.
In my understanding, transfer learning is also only really effective in deep learning, but I could be wrong, considering that this Google search seems to yield some results.
You might otherwise be thinking of knowledge distillation, which is a related but different concept, where an already trained network acts as a teacher and teaches another network (a student network) with possibly a different architecture (or a machine learning model not based on neural networks at all) the correct outputs for a bunch of input examples.