Bert: fine-tuning the entire pre-trained model end-to-end vs using contextual token vector In the official github page of BERT, it mentions that: 


  In certain cases, rather than fine-tuning the entire pre-trained model
  end-to-end, it can be beneficial to obtained pre-trained contextual 
      embeddings, which are fixed contextual representations of each input 
      token generated from the hidden layers of the pre-trained model. This 
      should also mitigate most of the out-of-memory issues.


I am wondering in which cases, using only token vectors, will be more beneficial (other than out of memory issue)?
It goes like this: In the case of fine-tuning, you may want to use huge batch size, which may lead to the Out of Memory (OOM) issues. In such case, you may find the embeddings of BERT useful than fine-tuning on BERT itself. Maybe you can think of using BERT embeddings to train CNN or RNN classifier, in such case you can try to obtain the embeddings of BERT using a small batch size (it can be as low as 1), and then use these embeddings to further train your CNN or RNN classifier.