text: Fundamentally, BERT is a stack of Transformer 
  encoder layers (Vaswani et al., 2017) which consist 
  of multiple self-attention "heads". For every input 
  token in a sequence, each head computes key, value 
  and query vectors, used to create a weighted repre- 
  sentation. The outputs of all heads in the same layer 
  are combined and run through a fully-connected 
  layer. Each layer is wrapped with a skip connection 
  and followed by layer normalization. 
  
  The conventional workﬂow for BERT consists 
  of two stages: pre-training and ﬁne-tuning. Pre- 
  training uses two self-supervised tasks: masked 
  language modeling (MLM, prediction of randomly 
  masked input tokens) and next sentence prediction 
  (NSP, predicting if two input sentences are adjacent 
  to each other). In ﬁne-tuning for downstream ap- 
  plications, one or more fully-connected layers are 
  typically added on top of the ﬁnal encoder layer. 
  
  The input representations are computed as fol- 
  lows: each word in the input is ﬁrst tokenized into 
  wordpieces (Wu et al., 2016), and then three em- 
  bedding layers (token, position, and segment) are 
  combined to obtain a ﬁxed-length vector. Special 
  token [CLS] is used for classiﬁcation predictions, 
  and [SEP] separates input segments. 