Difference between using BERT as a 'feature extractor' and fine tuning BERT with its layers fixed I understand that there are two ways of leveraging BERT for some NLP classification task: 


BERT might perform ‘feature extraction’ and its output is input further to another (classification) model 
The other way is fine-tuning BERT on some text classification task by adding an output layer or layers to pretrained BERT and retraining the whole (with varying number of BERT layers fixed 


However, if in the second case, we fix all the layers and add ALL the layers from the classification model will be added, 1st and 2nd approaches are effectively the same, am I right?
No, approaches 1 and 2 are not the same:

In approach 1 (feature extraction), you not only take BERT's output, but normally take the internal representation of all or some of BERT's layers.

In approach 2, you train not only the classification layers but all BERT's layers also. Normally, you choose a very low learning rate and a triangular learning rate schedule to avoid catastrophic forgetting.


There are many scientific articles studying how to best use BERT in transfer learning scenarios. This one may be a good starting point: https://www.aclweb.org/anthology/W19-4302/