BERT training on two tasks: what is the order of tasks? I read that BERT has been trained on two tasks: Masked Language Modeling and Next Sentence Prediction. I want to gain clarity how exactly it was done. 

Was it initially trained on  Masked Language Modeling (where we predict masked token) and later on Sentence Prediction (where we predict isnext or not)? It seems like these two tasks would require different architecture. Does BERT has some common architecture for these two tasks with interchanging layer for individual tasks? Still question of ordering remains.
BERT is trained on both task at the same time.


MLM is done using the token representations.
NSP is done using the CLS representation.