How to use BERT in seq2seq model? I would like to use pretrained BERT as encoder of transformer model. The decoder has the same vocabulary as encoder and I am going to use shared embeddings. But I need &lt;SOS&gt;, &lt;EOS&gt; tokens which are not trained with BERT. How should I get them ? Can I use &lt;CLS&gt; token as &lt;SOS&gt; and &lt;SEP&gt; as &lt;EOS&gt; ? Or I have to create these two embeddings as trainable Variables and concatenate them to the decoder input / labels ?
In principle, it is possible to reuse the special tokens as you describe.
However, according to research, you should not freeze BERT, but fine-tune the whole model with your data, in order to obtain better translation quality.
Another option would be to reuse just the embeddings instead of the whole model.