This issue aligns with the phenomenon of "catastrophic forgetting", wherein knowledge acquired during pretraining is disrupted or overwritten during the subsequent finetuning process. To remedy this issue, we implemented a refined finetuning strategy with continual learning loss. This additional loss term is designed to preserve the pretraining model's knowledge during the finetuning.