How does BERT and GPT-2 encoding deal with token such as ,
BERT is not trained with this kind of special tokens, so the tokenizer is not expecting them and therefore it splits them as any other piece of normal text, and they will probably harm the obtained representations if you keep them. You should remove these special tokens from the input text.

In the case of GPT-2, OpenAI trained it only with &lt;|endoftext|&gt;, but it has to be added after the tokenization. Some people mistakenly add it before tokenization, leading to problems.
&lt;|startoftext|&gt; is specific to the library gpt-2-simple.