Is it possible to train one part of the network with a particular learning rate and the other part with a different one?
Create two different optimizers and split the subnets' parameters into either with different lrs. You will have to call optimizer1.step(), optimizer2.step() with a single backward() call