title: How should the learning rate of each module of the pre-training model be set appropriately?
body: <p>When I was reading papers, I found that many papers set a learning rate for pre-trained models (such as BERT) in their experimental settings, and a relatively larger learning rate for the classifier (the last fully connected layer). So I would like to ask two questions:</p>
  <p>(1) What is the benefit of setting the learning rate of the classifier larger than the pretrained model;</p>
  <p>(2) Since the learning rate of the classifier and the pretrained model are different, this leads to finding the number of parameters become more, then how to choose the appropriate learning rate of the classifier and the learning rate of the pre-trained model.</p>
  <blockquote>
  <p>In experiments we use the grid search to find the learning rate for the embeddings within [1 × 10−6, 5 × 10−4]. For the learning rate of the CRF layer, we use a grid search over [0.05, 0.5].</p>
  </blockquote>