Is it possible to train one part of the network with a particular learning rate and the other part with a different one? I have a combined network consisting of two parts: one is for images and the other is for numerical data. Each sample is matched with a numerical case by an ID. For this combined network, a lr of 0.01 was found to be best working via hyperparameter tunings
However, when I trained them as a separate task (binary classification), a lr of 0.001 for images and 0.01 for numerical data were best.
As for an AUC metric, the combined network (0.818) is performing on average of image and numerical networks (0.799 and 0.821, respectively).
Here I thought maybe the combined network's lr is too high for image part and should apply lower lr for that part. But, I don't know if it is possible.
If anyone has any idea of what is what, let me know
Create two different optimizers and split the subnets' parameters into either with different lrs. You will have to call optimizer1.step(), optimizer2.step() with a single backward() call