The input ID tensor is processed by a BERT transformer model, where the RoBERTa 15 variant of BERT is adopted. The theories and model architecture of RoBERTa are discussed in the "RoBERTa model" Section of this paper. RoBERTa outputs a set of vectors with the length of 768, and each vector is transformed from a token by RoBERTa. The vectors are passed into a feed-forward neural network, which outputs the output logits. The output logit is the logit transformation applied to its original output, shown in Eq. 1. The output vector of each token remains the size equal to the vocal size, i.e. the total number of tokens, indicating the projection of each tokens into each output logit.