Pretraining. The hyper-parameters of the LSTM and GPT were tuned with random search for five days on a single NVIDIA A100 40GB GPU. The defined hyper-parameter space is based on previous work (Table ). 40 LSTM and 35 GPT models were optimized within a five-day limit. Hyper-parameter search was conducted to maximize the validity during pre-training.