We call these techniques Recurrence over BERT (RoBERT) and Transformer over BERT (ToBERT). We observed that ToBERT outperforms RoBERT on pre-trained BERT features and fine-tuned BERT features for all our tasks.  
 Question: On top of BERT does the RNN layer work better or the transformer layer?
Transformer over BERT (ToBERT)