Why do you need to retrain GPT-2?
I've come to learn about GPT-2 through Robert Miles AI safety Youtube channel and intend to look into it in more detail.

From my current understanding, GPT-2 is pre-trained to "understand" "natural" language (for any definition of the words in quotes). However you would want it to not only understand general text but generate text similar to some specific "genre", e.g. scientific articles, youtube comments, twitter messages, you name it.

So using its pre-trained understanding, it analyzes the structure of sample texts and replicates this structure.
For scientific articles this structure could be:


Abstract
Context of research topic
Introduction of researchers
Explanation of methods/experiments/discoveries
Results and interpretation
Future research and application


For Youtube comments the structure is probably more chaotic but could include a vague reference to former comments, insults, nonsensical bar-grade philosophy, internet slang and smileys.

TL;DR: The domain specific text is only used to tell GPT-2 what you're looking for. You basically hand it context to work with, instead of prompting "Say something clever" (my least favorite line at parties, when I've been introduced as clever).

P.S.: Take this with a grain of salt. It's 90% conjecture from incomplete information.