How to keep model fixed during training?
I am trying to implement a model that uses encoding from multiple pre-trained BERT models on different datasets and gets a combined representation using a fully-connected layer. In this, I want that BERT models should remain fixed and only fully-connected layers should get trained. Is it possible to achieve this in huggingface-transformers? I don't see any flag which allows me to do that. 

PS: I don't want to go by the way of dumping the encoding of inputs for each BERT model and use them as inputs. 

A simple solution to this is to just exclude the parameters related to the BERT model while passing to the optimizer. 

param_optimizer = [x for x in param_optimizer if 'bert' not in x[0]]
optimizer = AdamW(param_optimizer, lr)

