How to train only front part of a neural network?
I'm using pytorch to train part of the network. For example, I have a model structure

hidden1 = Layer1(x)
hidden2 = Layer2(hidden1)
out = Layer3(hidden2)


If I want to train Layer3 only, I can use

hidden1 = Layer1(x)
hidden2 = Layer2(hidden1).detach()
out = Layer3(hidden2)


However, this time I want to train Layer1 only. How can I achieve this? Thanks.

detach will not really "freeze" your layer.
If you don't want to train a layer, you should use requires_grad=False instead.

For example:

hidden2.weight.requires_grad = False
hidden2.bias.requires_grad = False


Then to unfreeze, you do the same with requires_grad=True.
