Difference between using BERT as a 'feature extractor' and fine tuning BERT with its layers fixed
No, approaches 1 and 2 are not the same:

In approach 1 (feature extraction), you not only take BERT's output, but normally take the internal representation of all or some of BERT's layers.

In approach 2, you train not only the classification layers but all BERT's layers also. Normally, you choose a very low learning rate and a triangular learning rate schedule to avoid catastrophic forgetting.


There are many scientific articles studying how to best use BERT in transfer learning scenarios. This one may be a good starting point: https://www.aclweb.org/anthology/W19-4302/