Continual Learning of Large Language Models: A Comprehensive Survey
Haizhou Shi Zihao Xu Hengyi Wang Weiyi Qin Wenyuan Wang Yibin Wang Hao Wang



The Directed Landscape is a Black Noise
    [
    April 29, 2024
=======================================














[1]Rutgers University  ^2Wuhan University   ^3Huazhong University of Science and Technology.     ^*Correspondence to: Haizhou Shi <haizhou.shi@rutgers.edu>, Hao Wang <hw488@cs.rutgers.edu>. 




    
    The recent success of large language models (LLMs) trained on static, pre-collected, general datasets has sparked numerous research directions and applications. One such direction addresses the non-trivial challenge of integrating pre-trained LLMs into dynamic data distributions, task structures, and user preferences.
    The primary challenge of this problem lies in balancing model adaptation and knowledge preservation. Pre-trained LLMs, when tailored for specific needs, often experience significant performance degradation in previous knowledge domains – a phenomenon known as “catastrophic forgetting”. While extensively studied in the continual learning (CL) community, it presents new manifestations in the realm of LLMs.
    
    In this survey, we provide a comprehensive overview and detailed discussion of the current research progress on large language models within the context of continual learning.
    Besides the introduction of the preliminary knowledge, this survey is structured into four main sections:
    we first describe an overview of continually learning LLMs, consisting of two directions of continuity: vertical continuity (or vertical continual learning), i.e., continual adaptation from general to specific capabilities, and horizontal continuity  (or horizontal continual learning), i.e., continual adaptation across time and domains (sec:overview). 
    
    
    
    
    Following vertical continuity, we summarize three stages of learning LLMs in the context of modern CL: Continual Pre-Training (CPT), Domain-Adaptive Pre-training (DAP), and Continual Fine-Tuning (CFT) (sec:stages).
    Then we provide an overview of evaluation protocols for continual learning with LLMs, along with the current available data sources (sec:eval-and-data).
    Finally, we discuss intriguing questions pertaining to continual learning for LLMs (sec:discussion).
    This survey sheds light on the relatively understudied domain of continually pre-training, adapting, and fine-tuning large language models, suggesting the necessity for greater attention from the community. Key areas requiring immediate focus include the development of practical and accessible evaluation benchmarks, along with methodologies specifically designed to counter forgetting and enable knowledge transfer within the evolving landscape of LLM learning paradigms.
    The full list of papers examined in this survey is available at https://github.com/Wang-ML-Lab/llm-continual-learning-surveyhttps://github.com/Wang-ML-Lab/llm-continual-learning-survey.




§ INTRODUCTION



    
    
        
        

    
    
    

Recent advances in large language models (LLMs) have demonstrated considerable potential for achieving artificial general intelligence (AGI) <cit.>.
Researchers have observed that complex abilities such as multi-step reasoning, few-shot in-context learning, and instruction following improve as the scale of parameter size increases <cit.>.
The development of LLMs is impactful and revolutionary, prompting machine learning practitioners to reconsider traditional computational paradigms for once-challenging human-level tasks such as question answering, machine translation, and dialogue systems <cit.>.
However, LLMs are typically trained on static, pre-collected datasets encompassing general domains, leading to gradual performance degradation over time <cit.> and across different content domains <cit.>.
Additionally, a single pre-trained large model cannot meet every user need and requires further fine-tuning <cit.>.
While one potential solution is re-collecting pre-training data and re-training models with additional specific needs, this approach is prohibitively expensive and impractical in real-world scenarios. 


To efficiently adapt LLMs to downstream tasks while minimizing performance degradation on previous knowledge domains, researchers employ the methodology of continual learning, also known as lifelong learning or incremental learning <cit.>. Continual learning, inspired by the incremental learning pattern observed in human brains <cit.>, involves training machine learning models sequentially on a series of tasks with the expectation of maintaining performance across all tasks <cit.>. Throughout training, models have limited or no access to previous data, posing a challenge in retaining past knowledge as optimization constraints from unseen previous data are absent during current-task learning <cit.>. This challenge, known as catastrophic forgetting, has been a central focus in continual learning research since its inception.
Over the years, researchers have explored various techniques to mitigate forgetting in machine learning models. These include replay-based methods <cit.>, parameter regularization <cit.>, and model architecture expansion <cit.>. Together, these techniques have significantly advanced the goal of achieving zero forgetting in continual learning across diverse tasks, model architectures, and learning paradigms.


In the context of training and adapting LLMs sequentially, the significance of CL is undergoing semantic shifts of its own as well. 
To better highlight this ongoing shift, in this survey, we provide a comprehensive overview and detailed discussion of the current research progress on LLMs within the context of CL. 
For the general picture of continually learning LLMs, we divide it into two directions of continuity that need to be addressed by practitioners (sec:overview):

    
  * Vertical continuity (or vertical continual learning), which refers to the ongoing adaptation of LLMs as they transition from large-scale general domains to smaller-scale specific domains, involving shifts in learning objectives and entities of execution. 
    For example, healthcare institutions may develop LLMs tailored to the medical domain while retaining their general reasoning and question answering capabilities for users.
    
  * Horizontal continuity (or horizontal continual learning), which refers to continual adaptation across time and domains, often entails multiple training stages and increased vulnerability to catastrophic forgetting. 
    For example, social media platforms continuously update LLMs to reflect recent trends, ensuring accurate targeting of downstream services like advertising and recommendations while maintaining a seamless user experience for existing users.
    

In fig:overview, following vertical continuity, we delineate three key stages of LLM learning within modern CL: Continual Pre-Training (CPT), Domain-Adaptive Pre-training (DAP), and Continual Fine-Tuning (CFT) (sec:stages). 
In CPT, existing research primarily investigates three types of distributional shifts: temporal, content-level, and language-level. Each presents distinct focuses and challenges.
In DAP, while it is primarily seen as the procedure of preparing LLMs for downstream tasks, CL evaluation and techniques are frequently utilized. However, there is a noticeable lack of diversity in these techniques, considering the maturity of the conventional CL community.
In CFT, our focus is on the emerging field of learning LLMs, covering topics such as Continual Instruction Tuning (CIT), Continual Model Refinement (CMR), Continual Model Alignment (CMA), and Continual Multimodal LLMs (CMLLMs).
Next, we present a compilation of publicly available evaluation protocols and benchmarks (sec:eval-and-data).
We conclude our survey with a discussion covering recent emergent properties of continual LLMs, changes in the roles of conventional incremental learning types and memory constraints within the context of continual LLMs, and prospective research directions for this subject (sec:discussion).

In summary, this paper provides a comprehensive view of existing continual learning studies for LLMs in detail, which significantly distinguishes itself from existing literature on related topics <cit.>.
Our survey highlights the underexplored research area of continually developing LLMs, especially in the field of continual pre-training (CPT) and domain adaptive pre-training (DAP). We emphasize the needs for increased attention from the community, with urgent needs including the development of practical, accessible, and widely acknowledged evaluation benchmarks. Additionally, methodologies need to be tailored to address forgetting in emerging large language model learning paradigms.
We hope this survey can provide a systematic and novel view of continual learning in the rapidly-changing field of LLMs and can help the continual learning community contribute to the challenging goals of developing LLMs in a more efficient, reliable, and sustainable manner <cit.>.


Organization.  The rest of this paper is organized as follows. We will first start by introducing the background and preliminaries of large language models and continual learning in sec:preliminary. 
Then we present the overview of continual learning in the modern era of large language models in sec:overview. Vertically, it can be roughly divided into three stages of continual training LLMs, and we will present a one-by-one survey of each stage in sec:stages.
In sec:cft, the unique aspects of continual fine-tuning LLMs will be introduced, including continual instruction tuning (sec:cft-cit), continual model refinement (sec:cft-cmr), continual model alignment (sec:cft-cma), and continual multimodal large language models (sec:cft-cmllm).
In sec:eval-and-data, we give an inclusive introduction to the evaluation protocols and benchmarks of continual learning for LLMs that are publicly available.
Finally, in sec:discussion, we present a series of discussion of the role of continual learning in the era of large language models, including 
emergent abilities in large-scale continual LLMs (sec:discussion-emergent), 
three types of continual learning (sec:discussion-xil),
roles of memory in continual learning of LLMs (sec:discussion-mem), and prospective future directions (sec:discussion-future). 




§ PRELIMINARIES


In this section, we provide an overview of the fundamental concepts of large language models (LLMs) and continual learning (CL), ensuring clarity and comprehensibility for readers unfamiliar with these topics. We begin by introducing the notation used in this paper. Subsequently, we discuss the pre-training and downstream adaptation of LLMs, as well as mainstream LLM families (sec:preliminary-llm), followed by an introduction to basic continual learning techniques studied by the community (sec:preliminary-cl).

Notation.  We denote scalars with lowercase letters, vectors with lowercase boldface letters, and matrices with uppercase boldface letters.
The l_2-norm of vectors and the Frobenius norm of a matrix are represented by ·_2. For a vector = [v_1, v_2, ⋯, v_n]^⊤, _2 = (∑_i=1^n v_i^2)^1/2; for a matrix ∈ℝ^m× n, _2 = (∑_ij A_ij^2)^1/2.
We use ϵ_, _ to denote the error function, and loss function that is deployed for training, respectively, where the subscript is used to denote the error/loss measured by taking the expectation on the data distribution . 
We further use _S to represent the empirical evaluation of the loss function  over the set of examples S.
Probability and expectation are denoted by P and , respectively. 
We use [m] to denote the set of positive integers up to m, {1, ⋯, m}.




 §.§ Large Language Models


In the past two decades, neural language modeling has emerged as the dominant field of deep learning, marked by significant and rapid advancements. Primarily built on the transformer architecture, pre-trained language models (PLMs) like BERT have established a universal hidden embedding space through extensive pre-training on large-scale unlabeled text corpora. 
Following the pre-training and fine-tuning paradigms, PLMs exhibit promising performance across various natural language processing tasks after being fine-tuned upon small amounts of task-specific data <cit.>.
Research on scaling laws indicates that increasing model size enhances the capacity of language modelss <cit.>.
By scaling parameters to billions or even hundreds of billions and training on massive text datasets, PLMs not only demonstrate superior language understanding and generation capabilities but also manifest emergent abilities such as in-context learning, instruction following, and multi-step reasoning, which are absent in small-scale language models like BERT <cit.>. 
These larger models are commonly referred to as Large Language Models (LLMs).




  §.§.§ Pre-Training of LLMs


Pre-training is essential for language models to acquire broad language representations. Decoder-only models typically employ probability language modeling (LM) tasks during pre-training. LM, in this context, specifically refers to auto-regressive LM.
Given a sequence of tokens = [x_1, x_2, ⋯, x_N ], LM predicts the next token x_t autoregressively based on all preceding tokens _<t = [x_1, x_2, ⋯, x_t-1], and trains the entire network by minimizing the negative log-likelihood:

    ℒ_ LM()    ≜ -∑^N_t=1log P( x_t | _<t ),

where P(x_1|_<1)≜ P(x_1) is the unconditional probability estimation of the first token.
The three most popular families of decoder-only models are GPT, PaLM, and LLaMA. The GPT family, developed by OpenAI, includes models such as GPT-2 <cit.>, GPT-3  <cit.>, ChatGPT <cit.>, and GPT-4  <cit.>. Notably, GPT-3 was the first LLM to exhibit emergent abilities not found in smaller PLMs. Another notable family, PaLM (Pathways Language Model), developed by Google, is comparable to the GPT family <cit.>. While both GPT and PaLM families are closed-source, LLaMA, released by Meta, is currently the most popular open-source family of LLMs <cit.>. The weights of these models are made available to the research community under non-commercial licenses.

Masked language modeling (MLM) task serves as a common pre-training objective for encoder-only models like BERT <cit.>. 
In MLM, certain tokens in the input sequence are masked, denoted as m(), and the unmasked parts _\ m() are utilized to predict the masked portions. 
Similar to traditional LM, the overarching goal of MLM is to minimize the negative log-likelihood as represented by the equation:

    ℒ_ MLM()    ≜ -∑_x̂∈ m() log  P( x̂|_\ m() ).


Some encoder-decoder architecture models, such as T5 <cit.>, also utilize Sequence-to-Sequence MLM task as the pre-training objective. They take masked sentences as encoder inputs and utilize the decoder to sequentially predict the masked tokens.




  §.§.§ Adaptation of LLMs


After pre-training, LLMs need to be effectively adapted to better serve downstream tasks. A series of adaptation methods have been proposed for specific objectives. Due to the fact that LLMs primarily focus on generating linguistically coherent text during pre-training, their performance may not necessarily align with the actual needs of human users or conform to human values, preferences, and principles. Additionally, due to issues such as the timeliness of pre-training data, LLMs may also encounter  knowledge cutoff or fallacy issues. Therefore, instruction tuning, model refinement, and model alignment have been proposed to address these issues <cit.>.
Below are the formal definitions of the three adaptation tasks for LLMs.


    
    Let h() be a language model that takes as input data , typically consisting of natural language instructions or queries. Instruction Tuning (IT) is a specialized training approach designed to enhance the model's ability to accurately and effectively respond to specific instructions. The objective of IT is to refine h by adjusting its parameters using a designated set of training examples = {(_i, _i)}_i=1^N, where _i represents the desired output for . This set is curated to target specific tasks or functionalities that require improved performance. 
    
    Formally, the updated model h^' is defined as follows:
    
    h^'(_0)    = _0,   ∀ (_0, _0) ∈.

    
    
    
    
    
    
    


    

    Suppose we have a model h() taking data  (e.g., natural language queries) as inputs. Consider a size-N editing set = {(_e, _e, _e)}_e=1^N, where _e denotes the true label of _e, but the model incorrectly outputs _e for _e. Model Refinement (MR) aims to efficiently update the model from h to h^' such that it correctly predicts the editing set , while preserving the original outputs outside . Formally, we aims to find h^' satisfying
    
    h^'(_0) = 
            _0    if  (_0, _0) ∈, 
    
                
                h(_0)     o.w.

    








Consider a model h() designed to process inputs  in decision-making scenarios. Define an alignment dataset of size M as = {(_a, _a, _a)}_a=1^M, where _a represents the model's original decision for input _a, and _a denotes the aligned decision that adheres to specified ethical guidelines or desired outcomes. The objective of Model Alignment (MA) is to modify h into h^' such that for any _a in the alignment dataset, h^'(_a) yields _a, aligning the model's decisions with the alignment criteria. Formally, 
    
    h^'(_0) = _0,   ∀ (_0, _0) ∈.




It is still an open problem to include the constraint of preventing catastrophic forgetting of the general knowledge for IT, and reducing the Alignment Tax <cit.> in the optimization objective of MA. 
A simple extension from the constraint of model refinement in eq:mr, h^'(_0) = h(_0), ∀ (_0, _0) ∉, might be too strong in this case, as we certainly want the preference represented by  can generalize to other similar while not the same inputs. 


























































 §.§ Continual Learning


Contemporary machine learning models differ from human learning processes. Humans gradually accumulate knowledge and skills across tasks without significant performance decline on previous tasks <cit.>. In contrast, machine learning models are usually data-centric, minimizing the training loss on the subsequent tasks will cause the model fail on the old ones, which phenomenon is phrased as “catastrophic forgetting”. Addressing this challenge is a focal point in continual learning research. The problem of efficiently adapting models on a continuous sequence of tasks without forgetting is extensively studied in the continual learning community <cit.>. These studies are conducted under the famous memory constraint of continual learning, as shown below.



Suppose T sets of observations {S_t∼_t}_t=1^T come in as a sequence, where {_t}_t=1^T denotes the T task distributions . At the learning stage of t>1, the sets of observations {S_i}_i=1^t-1 are not accessible (strong) or partially accessible (relaxed).



    In early stages of continual learning, works mostly focused on the strong memory constraint <cit.>; as the research field progresses, more focus was put on relaxing the memory constraint to a small buffer for replay <cit.>; some modern continual learning works consider the scenario where this constraint is completely discarded but the constraint on the computational budget is present <cit.>.





  §.§.§ Three Types of Continual Learning

There are three outstanding types of continual learning scenarios: task-incremental learning (TIL), domain-incremental learning (DIL), and class-incremental learning (CIL).
To establish a groundwork for subsequent discussions (as illustrated in tab:cft and sec:discussion-xil), we adhere to the conceptual framework proposed by <cit.> and offer formal definitions for these three continual learning scenarios.



Suppose T task distributions {_t}_t=1^T come in as a sequence, where _t denotes the joint distribution over the t-th task's input space and the label space (_t, _t). Denote ≜⋃_t=1^T _t and ≜⋃_t=1^T _t as the union of the input and label spaces, respectively.
Under the memory constraint defined in def:memory, Task-Incremental Learning (TIL) aims to find the optimal hypothesis h^*: × [T] → that satisfies:

    h^*    = min_h∑_t=1^T_(, y)∼_t[ 1_h(, t)≠ y].






Suppose T domain distributions {_t}_t=1^T come in as a sequence, where _t denotes the t-th joint distribution over the shared input space and label space (, ). 
Under the memory constraint defined in def:memory, Domain-Incremental Learning (DIL) aims to find the optimal hypothesis h^*: → that satisfies:

    h^*    = min_h∑_t=1^T_(, y)∼_t[ 1_h()≠ y].





Suppose T task distributions {_t}_t=1^T come in as a sequence, where _t denotes the joint distribution over the t-th task's input space and the label space (_t, _t). Denote ≜⋃_t=1^T _t and ≜⋃_t=1^T _t as the union of the input and label spaces, respectively.
Under the memory constraint defined in def:memory, Class-Incremental Learning (CIL) aims to find the optimal hypothesis h^*: → [T] × that satisfies:

    h^*    = min_h∑_t=1^T_(, y)∼_t[ 1_h() ≠ (t,y)].




    In TIL, it is common to have a shared input space = _t, ∀ t ∈ [T], but the space of the label distribution _t can be distinct (_i ∩_j = ∅, ∀ i≠ j), partially shared (_i ∩_j ≠∅, ∃ i≠ j), or shared across different tasks (= _t, ∀ t ∈ [T]). 
    In DIL, the tasks are defined in the same format, i.e., same input space  and same output space . During the inference, no task IDs are provided for the hypothesis, which means the continual learning model needs to capture the pattern between the domain-invariant features and the labels. DIL is commonly perceived as more difficult than TIL.
    CIL is commonly viewed as the most challenging continual learning scenario, as the model needs to infer the label and the task ID at the same time. Another possible formulation of CIL is to represent it as DIL but the output label spaces are disjoint, _i ∩_j = ∅, ∀ i≠ j.





  §.§.§ Techniques of Continual Learning


As outlined in the three definitions provided earlier, the objective of continual learning is to find a hypothesis that minimizes risk across all tasks/domains. Consider domain-incremental learning as an example <cit.>, at t-th learning stage, the ideal training objective (h) is

    (h)    ≜∑_i=1^t-1__i(h)_past domains + __t(h) _current domain.

The objectives for past domains are often challenging to measure or optimize due to the memory constraints (def:memory). Therefore, the core of designing continual learning algorithms lies in identifying a suitable proxy learning objective for the first term without violating the memory constraint.
Existing continual learning techniques can be roughly categorized into 5 groups: (i) replay-based, (ii) regularization-based, (iii) architecture-based, (iv) optimization-based, and (v) architecture-based <cit.>.
Here, we will provide a concise yet comprehensive introduction to the first three categories of continual learning techniques, as they find extensive application in continually learning large language models.



Replay-Based Methods.  Replay-based continual learning methods adopt the relaxed memory constraint by keeping a small buffer {M_i}_i=1^t-1 for each task _i to retain previously observed data examples. Formally, these methods seek to optimize the following empirical training objective:

    _replay(h)    ≜∑_i=1^t-1_M_i(h)_proxy for past domains + _S_t(h) _current domain,

where _S denotes the empirical loss term evaluated on the set of examples S.
Often regarded as a simplistic solution to continual learning, replay-based methods may theoretically lead to loose generalization bounds <cit.>. Despite this, they are valued for their simplicity, stability, and high performance, even with a small episodic memory <cit.>. For instance, DER++ <cit.> demonstrates consistent performance enhancement by replaying a small set of past examples along with their logits (known as dark experience replay). ESM-ER <cit.> introduces error sensitivity modulation (ESM) to mitigate abrupt representational drift caused by high-error new examples.
A significant focus in replay-based continual learning research is enhancing sample efficiency for buffer maintenance. For instance,  <cit.> prioritizes exemplar selection based on herding to accurately model class mean throughout class-incremental learning. 
<cit.> propose storing low-fidelity examples to achieve memory-efficient exemplar set maintenance. 
RM (Rainbow Memory) <cit.> introduces diversity-aware memory updates based on per-sample uncertainty estimation and data augmentation for class-incremental learning.




Regularization-Based Methods.  Suppose h__t-1 is the hypothesis yielded after the t-1-th stage of training, parameterized by _t-1. Regularization-based methods utilize a regularization term as a proxy for past domain losses, determined by the distance in the parameter space.

    _reg(h_)    ≜λ· - _t-1__proxy for past domains + _S_t(h_) _current domain,

where _ = ^⊤ is the vector norm evaluated on a positive-semi-definite matrix , and λ is the regularization coefficient, a hyper-parameter introduced to balance the past knowledge retention and current knowledge learning. 
The matrix  introduced is to measure the different level of importance of each parameters and their correlations in retaining the past knowledge. 
In practice, to reduce computational overhead, diagonal matrices are often designed to encode only the importance of each parameter.
For example, Elastic Weight Consolidation (EWC) adopts a Bayesian perspective, using diagonal values from the Fisher Information Matrix (FIM) as an approximation for the Hessian matrix of parameters. This forms a sequential Maximize A Posteriori (MAP) optimization for continual learning <cit.>. Memory Aware Synapses (MAS) computes parameter importance in an online and unsupervised manner, defining importance by accumulated absolute gradient during training <cit.>.
It is also worth noting that when = degenerates to an identity matrix, the regularization term simplifies to a basic l_2-penalty term, evenly penalizing each parameter, which can be surprisingly effective in some cases of continuall learning LLMs <cit.>.



Architecture-Based Methods.  Expanding the network architecture dynamically to assimilate new knowledge is deemed the most efficient form of continual learning <cit.>. This method primarily tackles adaptation challenges and can achieve zero-forgetting when task IDs are available during inference or can be correctly inferred <cit.>. However, due to the difficulty of task ID inference, architecture expansion is predominantly utilized in task-incremental learning but is scarcely explored in domain-incremental or class-incremental learning.
Progressive Neural Networks (PNN) proposes learning laterally connected neurons as new tasks arise, ensuring non-forgetting and enabling transfer of previously learned neurons for future tasks <cit.>. In conjunction with pre-trained backbone large models like ViT <cit.>, CoLoR <cit.> trains various low-rank adaptation (LoRA) modules for different tasks. It estimates and stores prototypes for each task and utilizes the natural clustering ability of the pre-trained model during testing to infer task IDs, selecting the corresponding LoRA component for prediction generation.
In the domain of continually learning LLMs, architecture expansion has resurged in popularity following the rise of parameter-efficient fine-tuning (PEFT) applied to large models <cit.>, a topic we will delve into shortly <cit.>.











§ CONTINUAL LEARNING MEETS LARGE LANGUAGE MODELS: AN OVERVIEW



    


Large language models (LLMs) are extensive in various dimensions, including the size of model parameters, pre-training datasets, computational resources, project teams, and development cycles <cit.>. The substantial scale of LLMs presents notable challenges for development teams, particularly in keeping them updated amidst rapid environmental changes <cit.>. To illustrate, in 2023, the average daily influx of new tweets posted by users exceeds 500 million[Source: https://www.omnicoreagency.com/twitter-statisticshttps://www.omnicoreagency.com/twitter-statistics ], and training on even a “small” subset of this large volume of data is not affordable. 
Efficiently and reliably adapting LLMs becomes more critical when considering their cascading impact on downstream applications. Downstream users often lack expertise in collecting and storing large-scale data, maintaining large-scale hardware systems, and training LLMs themselves.

Recyclable Tuning <cit.> is the pioneering study that explicitly outlines the supplier-consumer structure of the modern LLM production pipeline. On the supplier side, the model is continually pre-trained over a sequence of large-scale unlabeled datasets. After every release of the pre-trained model, the consumer needs to utilize the stronger and more up-to-date upstream model for better downstream performance. To enhance the efficiency of fine-tuning for downstream consumers, they initially make several key observations about continually pre-trained LLMs, focusing on mode connectivity and functional similarity. Additionally, they propose reusing the outdated fine-tuned components after a major update of the upstream pre-trained LLM. Building upon the conceptual framework introduced by Recyclable Tuning <cit.>, we present a comprehensive framework for a modern production pipeline encompassing various studies on continual LLM pre-training, adaptation, and deployment, illustrated in fig:overview. What sets our framework in this survey apart from existing studies <cit.> is the incorporation of two directions of continuity: vertical continuity and horizontal continuity.









0.4160.853<cit.>
0.5620.853<cit.>
0.6840.853<cit.>


0.520.8263<cit.> 
0.5470.8071<cit.> 
0.71250.8071<cit.> 
0.79750.8071<cit.> 
0.6850.7880<cit.> 
0.6000.7880<cit.> 
0.490.7880<cit.> 
0.5110.7694<cit.> 
0.4840.75<cit.> 
0.5960.75<cit.> 


0.450.715<cit.> 
0.450.707<cit.> 
0.5420.715<cit.> 
0.5420.707<cit.> 
0.6070.715<cit.> 
0.6070.707<cit.> 
0.6690.715<cit.> 
0.6690.707<cit.> 




 §.§ Vertical Continuity (Vertical Continual Learning)

Definition.  Vertical continuity (or vertical continual learning) has long been studied, either implicitly or explicitly, in existing literature; it involves a sequence of adaptation from general to specific domains and tasks <cit.>. Along this axis, the training task transitions gradually from general pre-training to downstream tasks, typically undertaken by distinct entities within the production pipeline <cit.>. 
Vertical continuity is characterized by a hierarchical structure encompassing data inclusiveness, task scope, and computational resources. 
fig:overview shows a typical pipeline for vertical continuity in LLMs, i.e., “pre-training” → “domain-adaptive training” → “downstream fine-tuning” <cit.>: 



  * Pre-training. During the pre-training stage, a substantial amount of data from diverse domains is required to develop a general-purpose LLM. This phase demands a sizable research and development team dedicated to training and benchmarking the model, along with considerable computational resources. 

  * Domain-Adaptive Pre-training. Subsequently, downstream institutions may opt for domain-adaptive pre-training to tailor the model for specific tasks using domain-specific data unavailable to the upstream supplier. 

  * Finetuning. Finally, the LLM undergoes fine-tuning on annotated data for downstream tasks before deployment.


Throughout the process, the unlabeled domain-specific dataset is smaller in scale than the upstream pre-training phase but larger than the final downstream task fine-tuning phase. This pattern extends to computational resources, team size, and other factors. It is important to note that vertical continuity can involve more than three stages <cit.>. In real-world applications, during domain-adaptive pre-training, additional layers can be added to accommodate multiple entities, such as various departments with distinct objectives but operating within the same domain.  




Vertical Forgetting.  We term the performance degradation on general knowledge of a model undergoing vertical continual learning “vertical forgetting”. As shown in fig:continuity, usually for vertical continual learning, the data distribution of upstream tasks partially covers the downstream, meaning the model might start off at a decent initialization for the subsequent stage of training. However, there are two significant challenges to be addressed to prevent vertical forgetting: 

    
  * Task Heterogeneity. Stemming from the inherent disparity between the formulation of upstream tasks and downstream tasks, task heterogeneity can lead to differences in model structures and training schemes, which has long been recognized as a major hurdle <cit.>. To mitigate this issue, practitioners often employ methodologies like freezing shared parameters during downstream phases or reformulating downstream tasks to match the structure of pre-training tasks <cit.>.
    
  * Inaccessible Upstream Data. This challenge arises primarily from varying levels of confidentiality across entities undertaking vertical continual learning. Data collected and curated under different protocols may not be accessible to some downstream entities. This scenario is even more challenging than the strict memory constraint presented in conventional CL (def:memory),
    as algorithms for latter case rely on access to previous data at specific points for parameter importance measurement <cit.> or for replay <cit.>.
    To address the challenge of inaccessible upstream data, existing methods either use public datasets or generate pseudo-examples to create proxy pre-training dataset <cit.>.
















 §.§ Horizontal Continuity (Horizontal Continual Learning)

Definition.  Horizontal continuity (or horizontal continual learning) refers to continual adaptation across time and domains, a topic extensively explored within the continual learning community. The primary rationale for preserving horizontal continuity lies in the dynamic nature of data distribution over time. To stay updated with these content shifts, an LLM must incrementally learn newly-emerged data. Otherwise, the cost of re-training will become prohibitively expensive and impractical <cit.>.
Empirical evidence has consistently shown that despite their impressive capabilities, LLMs struggle to generalize effectively to future unseen data, particularly in the face of temporal or domain shifts <cit.>. Additionally, they struggle to retain complete knowledge of past experiences when adapting to new temporal domains, although they do demonstrate a higher level of robustness against catastrophic forgetting <cit.>.
The necessity of employing complex continual learning algorithms to address challenges in LLMs remains an open question. For instance, during large-scale continual pre-training, major institutions can typically afford the storage costs associated with retaining all historical data, rendering memory constraints negligible. Several studies have demonstrated that with full access to historical data, simple sparse replay techniques can effectively mitigate forgetting in large models <cit.>. In contrast, numerous continual learning studies have showcased superior performance compared to naive solutions, suggesting the importance of continual learning techniques in LLM training <cit.>. 


Horizontal Forgetting.  We informally define “horizontal forgetting” as the degradation in performance on previous tasks during horizontal continual learning.
We informally define “horizontal forgetting” as the performance degradation on the previous tasks when model is undergoing horizontal continual learning. As illustrated in fig:continuity, horizontal continual learning typically involves training stages of similar scales, with potential distributional overlap among their data. In summary, addressing horizontal forgetting presents two main challenges:

    
  * Longer Task Sequence.
    Horizontal continual learning ideally involves numerous incremental phases, particularly to accommodate temporal shifts in data distribution. A longer task sequence entails more update steps of the model, leading to inevitable forgetting of previously learned tasks. To address this challenge, researchers employ established continual learning techniques with stronger constraints, such as continual model ensemble <cit.>.
    
  * Abrupt Distributional Shift. 
    In contrast to vertical continuity, where distributional shifts are often predictable, horizontal continual learning does not impose constraints on the sequential learning tasks' properties. Evidence suggests that abrupt changes in task distributions can result in significant horizontal forgetting of the model <cit.>.




















§ LEARNING STAGES OF CONTINUAL LARGE LANGUAGE MODELS













fig:overview provides an overview of continually learning large language models. Along the axis of vertical continuity, three major layers of modern continual learning emerge. The top layer, Continual Pre-Training (CPT), involves continuous pre-training of LLMs by the supplier on newly-collected data alongside existing data (sec:cpt). As data volume increases, the general capacity of LLMs naturally evolves. The middle layer, Domain-Adaptive Pre-training (DAP), prepares LLMs for domain-specific applications through additional pre-training on domain-specific unlabeled data (sec:dap). The bottom layer, Continual Fine-Tuning (CFT), targets models for final downstream tasks on the consumer side (sec:cft). Within continual fine-tuning, we further cover topics including continual instruction tuning (sec:cft-cit), model refinement (sec:cft-cmr), model alignment (sec:cft-cma), and multimodal LLMs (sec:cft-cmllm).





 §.§ Continual Pre-Training (CPT)



The recent development of large language models has shattered the glass ceiling in achieving close-to-human levels of natural language understanding and generation. However, effectively adapting these models to the ever-evolving environment remains a fundamental challenge. In tab:cpt-small, we outline the basic properties of existing CPT papers.




  §.§.§ CPT: Effectiveness and Efficiency

Before delving into the detailed introduction of papers on continual pre-training (CPT), it is important to address two fundamental questions: Firstly, regarding effectiveness, can CPT enhance performance on downstream tasks beyond that of the initial training on a wide range of data domains? Extensive studies, including ELLE <cit.>, DEMix <cit.>, CKL <cit.>, TemporalWiki <cit.>, LLPT <cit.>, and Lifelong-MoE <cit.>, have not only demonstrated the necessity of CPT for improved downstream performance, but also shown that when distributional shifts are gradual <cit.> or somewhat correlated <cit.>, CPT can enhance model generalization to unseen data.

After confirming the effectiveness of CPT, the second question regarding efficiency arises: given the large number of parameters in the LLM and the size of both old and new data, achieving adaptation and knowledge retention in a computationally efficient manner becomes crucial. Concerning efficiency, most studies focus on techniques for efficient knowledge retention <cit.>, which significantly overlap with the continual learning literature addressing catastrophic forgetting. As mentioned before, these techniques replay <cit.>, parameter regularization <cit.>, and architecture expansion <cit.>.
In contrast to prior approaches that fully utilize emergent data, some studies recognize the impracticality of this approach in real production environments. Instead, they concentrate on further improving the efficiency of adapting to new distributions. For instance, ELLE <cit.> employs a function-preserved model expansion to facilitate efficient knowledge growth; <cit.> sub-samples training data based on semantic shift levels to enhance training efficiency; <cit.> employs a data sampling strategy that encourages novelty and diversity, achieving superior performance to full-data training. Though underexplored, this aspect of efficient adaptation in continual pre-training is poised to become significant, given recent findings emphasizing data quality over quantity for LLM generalization <cit.>.

























































































  §.§.§ General Observations on CPT

The analysis presented in tab:cpt-small sheds light on the prevailing research trends in continual pre-training (CPT). Firstly, it is evident that the development of advanced techniques tailored specifically for CPT is still at the starting stage and warrants further exploration. This observation is underscored by the fact that only about half of the examined papers propose novel techniques (9 out of 16 papers, represented in the deep gray section of tab:cpt-small), while the remaining half either focus solely on the effects of pure adaptation without considering continual learning techniques (3 out of 16 papers, represented in the white section), or conduct empirical studies on the straightforward application of existing continual learning techniques (4 out of 16 papers, represented in the light gray section).
Secondly, while research extensively covers various continual learning techniques, such as rehearsal, parameter regularization, and architecture expansion (as indicated in the light gray section of tab:cpt-small), the practical incorporation of these techniques in systems remains relatively limited. Most practical implementations primarily focus on architecture expansion of LLMs <cit.>, with only a few explicitly utilizing replay <cit.> and parameter regularization <cit.> explicitly (deep gray section of tab:cpt-small).
Thirdly, there is a pressing need for exploration into longer sequences of incremental phases in continual pre-training. Currently, the longest sequence of domains explored is eight, with content-level distributional shifts <cit.>. However, this falls short of real-world scenarios where continual pre-training may occur more frequently and persist for months or years. The efficacy of continual learning techniques in such prolonged scenarios remains uncertain, as potential performance degradation with longer domain sequences is observed in techniques like EWC <cit.>. Additionally, investigating CPT in a task-boundary-free data stream setting is an important avenue for research as well.





  §.§.§ Distributional Shifts in CPT
 
This survey categorizes distributional shifts of continual pre-training into three main types:
(i) Language Shift: LLMs sequentially learn different language corpora, e.g., English → Chinese, focusing on token and vocabulary distributional shifts <cit.>.
(ii) Content Shift: LLMs sequentially learn corpora from different fields, e.g., chemistry → biology, focusing on token and vocabulary distributional shifts as well as shift of semantic meaning <cit.>.
(iii) Temporal Shift: Distributional shifts occur over time, e.g., news in 2021 → news in 2022, focusing on token and vocabulary shifts, and timestamp-sensitive knowledge retention and update, which aligns with real-world LLM deployment needs <cit.>.




Language Shift.  In contrast to the common approach of pre-training multilingual language models jointly on large corpora from multiple languages, <cit.> focuses on assessing these models' natural ability to learn new languages sequentially (English, Norwegian, and Icelandic). The study does not employ explicit continual learning techniques for preventing horizontal forgetting. Nevertheless, it observes consistent positive forward transfer, facilitating new language acquisition regardless of the presentation order. Forgetting, on the other hand, emerges as a significant challenge, influenced by language order and not mitigated by increasing LLM size.
In <cit.>, the degree of forgetting of previously learned language (English) when adapting LLMs to a new language (Traditional Chinese) is investigated. Various continual learning techniques, including parameter freezing, LoRA <cit.>, and (IA)^3 <cit.>, are evaluated across multiple dimensions, including output language, general knowledge retention, and reliability. Preliminary experimental results presented in this study highlight the non-trivial nature of addressing horizontal forgetting in continually pre-training LLMs under the language shift.
To summarize, research on continual pre-training for language shifts is in its preliminary stages for two main reasons: Firstly, the datasets' scale, including the number of languages and total token count, remains small. Secondly, specific methods targeting language shifts have yet to be proposed; only basic combinations of existing continual learning techniques have been evaluated.

Content Shift.  Without using complex CL techniques, <cit.> continues the pre-training phase of Pythia <cit.> on the newly collected SlimPajama dataset <cit.>. The study focuses on optimizing continual pre-training by learning rate (re)warm-up. They discover that regardless of whether a larger or smaller maximum learning rate is used, models that undergo re-warming consistently exhibit improvements over models trained from scratch, even in terms of adaptation solely.

Another pioneering work, LLPT <cit.>, establishes a comprehensive training and evaluation protocol for a series of content-level distributional shifts, referred to as “domain-incremental data streams” in the paper. They assess multiple continual learning methods based on masked language modeling perplexity for pre-training tasks and downstream task accuracy. Similar to findings in <cit.>, they note that later domains benefit from knowledge learned from earlier ones, yet horizontal forgetting remains a significant challenge for earlier domains. Contrary to the common belief that experience replay (ER, <cit.>) is the most efficient approach to preventing forgetting, the authors find it scarcely improves continual pre-training performance. They speculate that ER's inefficiency may stem from overfitting issues, as replaying with distillation loss can alleviate this problem efficiently <cit.>.
Following LLPT, Recyclable Tuning <cit.> is the first study to consider both upstream LLM suppliers and downstream consumers at the same time. It shows that if the upstream supplier continually pre-trains LLMs – initializing from the previous checkpoint and continuing pre-training on newly collected data, with or without replay, consumer-side efficiency can be boosted by recycling previously learned incremental components. Two CL techniques, initializing from outdated components and knowledge distillation, complement each other to improve recyclable tuning in this context.

Other approaches involve training additional domain-specific experts for new content domains.
DEMix <cit.> addresses continual pre-training by incrementally training and integrating new experts (DEMix layer replacing every FFN layer in the transformer) for new domains. To ensure reasonable inference performance during testing when no domain information is available, DEMix proposes a parameter-free probabilistic approach, distinct from the gating function in MoE <cit.>, to dynamically estimate a weighted mixture of domains. Introducing a new domain variable D_t alongside each word x_t, the authors estimate the next word probability p(x_t|_<t) by marginalizing over all experts[In the marginalization step of the following equation in the original paper, it is p(D_t=j|_t) instead of p(D_t=j|_<t), while we believe this is a minor typo, and hence here we use the updated version.]: 

    p(x_t|_<t) 
           = ∑_j=1^n p(x_t|_<t, D_t=j) · p(D_t=j|_<t) 
       = ∑_j=1^n p(x_t|_<t, D_t=j) ·[ p(_<t|D_t=j)· p(D_t=j)/∑_j^'=1^n p(_<t|D_t=j^') · p(D_t=j^')],

where all the probability terms p(·|·, D_t) conditioned on the domain variable D_t are calculated by using a specific domain expert.
The authors develop a large-scale continual pre-training evaluation benchmark comprising eight semantic domains for sequential training and another set of eight domains for assessing LLMs' generalization ability. The DEMix framework's modularization has been shown to facilitate efficient domain-adaptive pre-training, promote relevant knowledge during inference, and allow for removable components.
Lifelong-MoE <cit.> shares a similar approach to DEMix <cit.> by incrementally training domain experts for each new domain. However, Lifelong-MoE differs in utilizing a token-level gating function to activate multiple experts for intermediate embedding calculation. During training, previous experts' parameters and gating functions remain frozen, while knowledge distillation loss is employed to regulate parameter updates. Although the data distributions for evaluation are extremely large-scale (3 domains, 686 billion tokens in total), the Lifelong-MoE is able to efficiently mitigate the issue of horizontal forgetting.

It is noteworthy that some papers draw almost opposite conclusions regarding the significance of CPT. For instance, <cit.> continually pre-trains BERT <cit.> and RoBERTa <cit.> on five scientific domains and evaluates performance on downstream sentiment analysis. They observe that even baseline sequential pre-training does not exhibit severe forgetting, prompting reasonable questions about the necessity of continual pre-training or its suitable application scenarios.

Temporal Shift.  In the context of continual learning amid content shifts, Multi-Task Learning (MTL) is often regarded as  the upper limit achievable in continual learning scenarios <cit.>. However, this belief does not fully hold when considering continual learning under temporal shifts <cit.>, as  temporal shifts can introduce conflicting information, posing challenges for LLMs. For instance, the statement “Lionel Messi plays for team Barcelona” remains accurate from 2004 to 2021 but becomes false by 2024, as “Lionel Messi plays for team Inter Miami” becomes the correct statement.

Hence, as advocated by CKL <cit.> and TemporalWiki <cit.>, LLMs undergoing continual adaptation to temporal shifts in the corpus must simultaneously achieve three objectives: (i) retention of old knowledge, (ii) acquisition of new knowledge, and (iii) update of the outdated knowledge (as a conflict resolution).
They evaluate the same set of continual learning baseline methods, including parameter regularization (RecAdam <cit.>), rehearsal (Mix-review <cit.>), and parameter expansion (LoRA <cit.> and K-Adapter <cit.>), each highlighting distinct aspects of their impact. CKL <cit.> introduces a unified metric, FUAR (Forgetting / (Updated + Acquired) Ratio), to assess the three learning objectives collectively. They observe that parameter expansion consistently exhibits robust performance across all experimental conditions. In contrast, replay-based methods struggle to efficiently adapt to new knowledge acquisition and outdated knowledge update, leading to rapid forgetting of newly learned information during training.
TemporalWiki <cit.>, in contrast, constructs a series of temporal corpora and their differential sets from sequential snapshots of Wikipedia, investigating the efficacy of adapting LLMs to these differential sets. The study reveals that updating LLMs on these differential sets substantially enhances new knowledge acquisition and updates, requiring significantly less computational resources. Moreover, various continual learning techniques prove effective in mitigating horizontal forgetting during this process. 

Additionally, LLPT <cit.> introduces temporal generalization evaluation for LLMs pre-trained on sequential corpora. Through experiments on a large-scale chronologically-ordered Tweet Stream, the authors demonstrate the superiority of CPT, combined with any continual learning technique, over a single task-specific LM, in terms of both knowledge acquisition and temporal generalization. Nonetheless, these preliminary experiments do not conclusively determine which specific continual learning method is more preferable than the others.

Another line of work, Temporal Language Models (TLMs), takes a different approach to address knowledge retention, acquisition, and update under temporal shifts by integrating temporal information into the learning process <cit.>. During training, they inject temporal information into training examples as prefixes of prompts, using special tokens <cit.>, explicit year information <cit.>, or syntax-guided structural information <cit.>. In sequential training experiments conducted by TempoT5 <cit.>, comparison between continually and jointly pre-trained LMs demonstrates that CPT better balances adaptation and forgetting when the replay rate of past data is appropriately set.


The significance of addressing temporal shifts through continual pre-training is underscored by several industrial studies. For instance, <cit.> employs a dynamic vocabulary expansion algorithm and an efficient sub-sampling procedure to conduct CPT on large-scale emerging tweet data. Conversely, <cit.> adopts continual pre-training without explicit measures to constrain model updates, releasing a series of BERT-based LMs incrementally trained on new tweet data every three months. Preliminary experimental results demonstrate substantial improvements of continually pre-trained LMs over the base BERT model across downstream tasks. While some studies question the necessity of continually adapting LLMs along the temporal axis for environmental reasons, such as reducing CO_2 emissions <cit.>, the community commonly embraces CPT as a more efficient and environmentally friendly learning paradigm compared to the traditional “combine-and-retrain” approach. 

















































































































 §.§ Domain-Adaptive Pre-training (DAP)



Background of DAP.  Institutions, regardless of size, often possess significant amounts of unlabeled, domain-specific data. This data bridges the gap between general-purpose LLMs trained on diverse corpora and fine-tuned LLMs designed for specific downstream tasks. Leveraging this data as a preparatory stage can facilitate effective adaptation of LLMs to downstream tasks.
Such process of “continued/continual/continuous pre-training” <cit.>, 
“further pre-training” <cit.>, 
“domain tuning” <cit.>,
“knowledge enhancement pre-training” <cit.>, 
and “knowledge injection training” <cit.> 
is unified and termed “Domain Adaptive Pre-training (DAP)” <cit.> for clarity and consistency throughout the remainder of this survey.
In the pioneering work of domain-adaptive pre-training (DAPT) <cit.>, the authors continuously pre-train the language models on a larger domain-specific dataset before fine-tuning them to the downstream tasks, resulting in universally improved performance aross various tasks. 
As the observation above has been validated on multiple domains in parallel, including BioMed, CS, News, and Reviews <cit.>, practitioners commonly accept that employing DAP on additional unlabeled domain-specific data benefits downstream tasks. Consequently, this technique has become widely deployed in many modern LLMs. 


Summary of LLMs with DAP.  To illustrate this, we provide a summary of existing studies utilizing domain-adaptive pre-training for LLMs in tab:dap. Each entry in the table is characterized by three main features: (i) training process specifications, encompassing the vertical domain for which LLMs are trained, the training pipeline preceding release, and the LLM architecture employed; (ii) adopted continual learning techniques, including rehearsal, parameter regularization, and architecture expansion; and (iii) evaluation metrics for continual learning, such as backward transfer (forgetting) and forward transfer (adaptation to downstream data). Following this overview, we will present general observations about DAP in sec:dap-obs, followed by a detailed introduction to LLMs developed in vertical domains in sec:dap-domains.






  §.§.§ General Observation on DAP

As depicted in tab:dap, several key observations emerge regarding the current research landscape of DAP. Firstly, DAP predominantly occurs in a single stage. Horizontal Continual DAP which involves more than one stage is seldom explored: among the 34 papers listed, only one paper employs two stages of DAP <cit.>. 
In Code Llama <cit.>, aimed at developing a language model tailored to Python programming, the authors initialize the model from the pre-trained Llama 2 checkpoint. They then conduct the first stage of DAP across multiple programming languages (500 billion tokens) before proceeding to the second stage, focusing solely on Python code (100 billion tokens). Finally, they perform long context fine-tuning (20 billion tokens) to enhance the model's capability in challenging long-context scenarios of code generation. This PT → DAP → DAP → FT pipeline represents the sole example found thus far that strictly adheres to the definition and hierarchical structure of vertical continuity in pre-training and adapting LLMs for final end-use.
Hence, categorizing the 10 studies that solely conduct one stage of DAP and nothing more <cit.> proves challenging. One could also argue that they deploy an additional single stage of CPT rather than DAP. Nevertheless, considering that all these papers aim to adapt a general-purpose LLM to a specific domain, we include them in this section for discussion, aligning with the categorization we have established thus far.

Secondly, the notion of interpreting DAP through the lens of continual learning, whether intentional or not, is widely embraced. As demonstrated in tab:dap, with the exception of the first section (white, 11/33), where papers overlook any potential side effects of DAP leading to vertical forgetting of previously learned general knowledge, the remaining sections (all gray, 22/33) either evaluate the potential negative impacts of DAP or proactively employ continual learning techniques to mitigate the risk of vertical forgetting from the outset.

Thirdly, we observe widespread adoption of CL techniques (14/33) for training domain-specific LLMs. However, the diversity of these techniques is limited, with only replay <cit.> and parameter expansion (LoRA <cit.>) or Layer/Block expansion <cit.> being utilized. This highlights the need for further research to investigate, incorporate, and design more sophisticated CL techniques for not just DAP, but vertical continual learning in general.
In fact, it appears that individuals may not explicitly recognize that DAP should be viewed from the perspective of vertical continuity, as they often employ CL techniques unknowingly. This deduction arises from two observations: 
(i) parameter expansion methods inherently embody implicit CL techniques. For instance, in LoRA <cit.>, the increment of weights Δ= preserves the original performance on previous data distributions, but once adaptation occurs (Δ≠), forgetting on the original data distribution follows. This analysis extends to other parameter expansion techniques such as layer expansion <cit.> and block expansion <cit.>. Authors typically empirically demonstrate the effectiveness of these approaches, attributing forgetting mitigation to the low-rank property and parameter efficiency;
(ii) excluding parameter expansion methods, replay emerges as the only CL technique employed during DAP, except in cases where extensive empirical investigations of CL methods are conducted <cit.>. Furthermore, studies deploying replay often term the technique as “data combination” <cit.> or “data mixing/mixture” <cit.>, without recognizing it as a vertical continual learning problem.











  §.§.§ Different Domains of DAP




Legal Domain.  Given the legal industry's demand for managing ever-growing volumes of legal documents, there's a burgeoning need to harness LLMs to aid legal professionals in navigating, interpreting, and generating high-quality legal materials <cit.>. While general-purpose LLMs may perform adequately on some legal benchmarks <cit.>, customizing LLMs with additional unlabeled resources specific to the legal domain can yield superior results. This is because the high-volume unlabeled legal corpus resembles the conditions under which general-purpose LLMs are pre-trained.
In Layer Llama <cit.>, the authors gathered publicly available legal texts from China Courts websites, including judgment documents, legal articles, judicial interpretations, court news, and law popularization articles, totaling approximately 10 billion tokens as noted in a GitHub issue <cit.>. In SaulLM <cit.>, the authors collected the DAP corpus from various jurisdictions in different countries, such as the U.S., Europe, and Australia, resulting in a corpus of 30 billion tokens to cover diverse aspects of legal texts. When combined with previously available datasets[Detailed web links for data collection in SaulLM are omitted here for brevity. Refer to the original paper for more information.] <cit.>, the total tokens used for legal-domain DAP reach 94 billion. 

The substantial volume of DAP data, while offering valuable insights into specific domains, increases the risk of catastrophic forgetting of the general knowledge due to the large number of update steps involved. To mitigate this, SaulLM incorporates general data from Wikipedia, StackExchange, and GitHub into the DAP data, constituting about 2% of the final dataset <cit.>. Following DAP, SaulLM then employs a combination of general-domain and legal-domain instructions to enhance the model's instruction-following ability (U-IT, see tab:dap).
Similarly, Lawyer Llama also incorporates general-domain data during DAP, but the replay rate is not disclosed <cit.>. After DAP, it undergoes two distinct phases of instruction tuning: first, general-domain instruction tuning (G-IT), followed by domain-specific downstream legal consultation application instruction tuning (D-IT). However, no explanation is provided for why two-stage IT is preferred over consolidation into one as in SaulLM <cit.>, leaving this as an open question for future research.










Medical Domain.  The development of LLMs holds promise for revolutionary changes in the medical industry, offering potential improvements in efficiency and quality across medical communication, disease diagnosis, and decision-making for doctors. While some instances of general-domain LLMs have shown success in providing useful advice and accelerating diagnosis progress for patients <cit.>, direct deployment poses risks. These risks include the potential for sub-optimal solutions, such as imprecise medical advice, and the possibility of harm, such as incorrect drug recommendations and the propagation of medical misinformation <cit.>.
Efforts have been made to develop medical specialists by either collecting medical-domain data and training an LLM from scratch <cit.>, or fine-tuning publicly available LLMs to meet specific medical needs <cit.>. Among these approaches, domain-adaptive pre-training techniques have been extensively utilized to preserve the communication and instruction-following abilities of a general LLM, preparing it for subsequent medical applications <cit.>.

BioMedGPT <cit.> is a multi-modal biomedical language model that integrates representations of human language and the language of life (molecules, proteins, cells, genes, etc.). Prior to final multi-modal supervised fine-tuning, the authors initialize the model from Llama2-Chat <cit.> and conduct DAP using extensive biomedical documents from S2ORC <cit.>, without considering any continual learning techniques or metrics.
In <cit.>, a Chinese medical LLM is developed for medical question answering. DAP is performed using Chinese medical encyclopedias and online expert articles, yielding over 364,000 question-answer pairs, with next-token prediction as the sole training objective. Results show that for both models initialized from Llama-2 <cit.> and Chinese-Llama-2 <cit.>, performance gradually deteriorates on general-domain datasets as training steps increase, while improving on the Chinese medical examination evaluation <cit.>.
To ensure the model possesses sufficient fundamental medical knowledge before the instruction tuning phase, PMC-LLama <cit.> gathers biomedical papers from S2ORC <cit.> and medical textbooks for “knowledge injection training”. During this phase, a general language corpus from RedPajama-Data <cit.> is replayed at a 5% rate within a training batch. However, the paper does not analyze the effectiveness of adding and mixing general-domain data during DAP.


To mitigate vertical forgetting, AF Adapter <cit.> proposes an adapter structure extending the original transformer block. This structure extends the width of Attention layers and FFNs for domain-specific knowledge storage. During DAP, the general-domain parameters inherited from the BERT checkpoint remain frozen, while only the adapters are trained. To gauge the extent of forgetting after DAP, the authors collect a small subset of general-domain samples from WuDao <cit.> and calculate the accuracy of masked word prediction using the same pre-training protocol as BERT and DAP of AF Adapter. They find minimal performance degradation compared to other full-parameter fine-tuning techniques.
HuatuoGPT-II <cit.>, proposes to fuse the DAP into the final SFT, making the two-stage development one unified protocol. The challenge of such process mainly comes from the data heterogeneity of the domain-adaptive pre-training unlabeled corpus. The authors address this challenge by reformulating paragraphs of data into (instruction, output) format using existing large language models. During the unified one-stage SFT, they employ a priority sampling strategy to avoid compromising downstream ability, as seen in fixed-rate data mixing <cit.>. Unlike previous approaches, this paper empirically demonstrates the superiority of unified one-stage SFT over two-stage training, questioning the current computational paradigm of adapting LLMs to specific domains via DAP.
In <cit.>, the authors investigate the effectiveness of continual learning techniques for mitigating catastrophic forgetting in DAP on medical-domain corpus. They find that LMs constrained by continual learning techniques on source domains exhibit greater robustness to future domain shifts. Specifically, they identify that parameter regularization techniques like EWC <cit.>, despite slightly higher computational cost, can facilitate beneficial forward and backward transfer, offering valuable insights to the community.
 

Financial Domain.  Similar to the medical domain, large language models hold immense potential for enhancing financial communication, decision-making processes, and risk assessment for both traders and ordinary individuals <cit.>. However, the financial domain entails high stakes, with even minor errors bearing significant consequences. Thus, integrating LLMs into financial workflows requires heightened caution to avoid inaccuracies or misunderstandings that could lead to substantial financial losses or the dissemination of misleading information. Despite advancements, a gap persists between general-purpose large language models and existing domain-specific smaller-scale language models <cit.>, underscoring the urgent need for more powerful financial-domain experts through the integration of LLMs. Notably, DAP techniques have emerged as crucial tools for tailoring LLMs to the intricacies of the financial domain while mitigating the negative effects of abrupt domain shifts from general to finance <cit.>.

BBT-Fin <cit.> collects a Chinese financial DAP dataset comprising 80 billion tokens sourced from corporate reports, analyst reports, social media, and financial news. In addition to the conventional masked language modeling (MLM) training objective, BBT-Fin further incorporates triplet masking and span masking techniques during DAP. This knowledge enhancement pre-training entails: (i) actively selecting sentences containing specific knowledge triplets (head entity, relation, tail entity), masking one of them, and (ii) simultaneously masking 15% of a random-length span.
CFGPT <cit.> creates CFData, a financial dataset for domain-adaptive pre-training and supervised fine-tuning, comprising 141 billion tokens. The dataset includes corporate prospectuses, announcements, research reports, social media content, financial news, and Wikipedia articles. During DAP, CFGPT does not employ continual learning techniques but utilizes QLoRA <cit.> for preventing overfitting to downstream data and balancing general response ability and domain-specific ability during supervised fine-tuning. These two methods are typical domain-specific LLM studies focusing solely on target domains, without explicit continual learning measures or evaluation of performance degradation on general skills.


In <cit.>, the authors aim to enhance the data efficiency of domain-adaptive pre-training. They propose two data selection techniques: (i) efficient task-similar (ETS) and (ii) efficient task-agnostic (ETA) domain-adaptive pre-training. These methods are inspired by the generalization bound of domain adaptation <cit.>. 
Suppose the source domain data distribution is  and the task data distribution is . The generalization error ϵ_(h) of the hypothesis h on the target task distribution  can be given as

    ϵ_(h)    ≤ϵ_(h) + 1/2d_Δ(, ) + ,

where ϵ is the 0-1 error function for binary classification, h represents the model, d_Δ is the Δ divergence that measures the distributional discrepancy between two distributions based on a hypothesis set  used for discriminating different data distributions <cit.>, and  is a constant.
Therefore, based on the theory, finding a set of examples from the DAP corpus that are similar to the downstream task's data, i.e., d_Δ(, ) is low, should help the final-stage adaptation to the tasks. 
In the context of large language models (LLMs), <cit.> suggests ensuring novelty and diversity in the sampled corpus for DAP. Here, high perplexity of an LLM on a sentence indicates high novelty, while high entropy of part-of-speech (POS) tags on a sentence indicates high diversity <cit.>. This approach significantly enhances DAP efficiency: it utilizes only 10% of the originally collected data yet outperforms models trained on the entire dataset, underscoring the importance of data quality over quantity in DAP. Furthermore, with the reduced number of examples, the authors do not observe any signs of forgetting regarding the model's capacity in open-domain scenarios.
WeaverBird <cit.> introduces an intelligent finance dialogue system, where the encoder is trained on Chinese and English financial documents, alongside expert-annotated financial query-response pairs, using LoRA <cit.>. Xuanyuan 2.0 <cit.>, akin to HuatuoGPT-II <cit.>, proposes the technique of hybrid-tuning, which fuses the stage of DAP and SFT into one, general-domain data and financial-domain data into one. Notably, the distribution of data in hybrid-tuning is non-conventional: financial pre-training data comprises 13%, general pre-training data 20%, and instruction tuning data 67%. Within the instruction tuning data, the financial domain accounts for approximately 30%. This prompts a pertinent question in line with the investigation on efficient DAP in <cit.>: Is a large DAP dataset necessary for developing a domain-specific LLM?





Scientific Domain.  Vertical scientific large language models <cit.> span many subjects, including astronomy <cit.>, mathematics <cit.>, geology <cit.>, chemistry and physics <cit.>, biology <cit.>. 
However, among all the studies listed above, only a small fraction of them adopt the technique of DAP. 


OceanGPT <cit.> is the first LLM tailored specifically for the ocean domain, catering to a range of downstream ocean science applications. It compiles a raw corpus of ocean science literature, prioritizing recent research and historically significant works, and performs DAP on the Llama 2 model.
K2 <cit.> pioneers the development of a foundational language model tailored specifically for geoscience. It aggregates geoscience open access literature and Earth science-related Wikipedia pages for DAP. Following this, it undergoes multi-task instruction tuning utilizing LoRA <cit.> on both a general instruction tuning dataset and the GeoSignal benchmark introduced within the K2 framework.
AstroLlama <cit.> gathers abstracts solely from astronomy papers on arXiv and proceeds with pre-training focused on next token prediction. It observes an improved perplexity on the domain of scholarly astronomy, without providing more quantitative evaluation. 
MarineGPT <cit.> is a multi-modal LLM designed specifically for the marine domain. 

During DAP, MarineGPT incorporates 5 million marine image-text pairs to imbue domain knowledge. This involves training the parameters of a Q-Former <cit.> between the frozen visual encoder <cit.> and text decoder <cit.>. 

The four methods introduced above lack further validation regarding the necessity of DAP deployment. They seem to adopt DAP merely out of convention, likely for the convenience of utilizing domain-specific corpora as pre-training data.

Another branch of methods proactively integrate in the replay of the general-domain data to mitigate vertical forgetting.
GeoGalactica <cit.> introduces a series of large language models tailored for geoscience. In the DAP phase, besides the 52-billion-token geoscience corpus, Arxiv papers and Codedata are incorporated, with a mixing ratio of 8:1:1. The authors believe that the inclusion of the Codedata during the model's pre-training can significantly boost the reasoning ability of the LLMs, which is crucial to fine-tuning on the downstream tasks. 
However, although GeoGalactica pinpoints challenges of DAP, including overfitting, catastrophic forgetting, maintaining the training stability and convergence speed, it does not further provide empirical evidence supporting the inclusion of the Codedata, or deploying specific measures to address the challenges proposed above. 
Llemma <cit.> focuses on mathematics, initialized from Code Llama <cit.>, and undergoes DAP on a blend of the 55 billion mathematical pre-training dataset (Proof-Pile-2 including scientific papers, web data, and mathematical code) and general domain data (Pile <cit.>) at a 19:1 ratio. In contrast, PLlama <cit.>, designed for plant science, mixes domain-specific and general-domain data at a ratio of 9:1.






Code Domain.  The development of LLMs for automatic code filling, debugging, and generation holds significant practical importance, with programmers worldwide benefitting from recent advancements in coding assistants <cit.>. These advancements cover various frameworks, including encoder-only <cit.>, encoder-decoder <cit.>, and decoder-only <cit.>.
As noted in <cit.>, in the era of LLMs, there's a growing trend towards decoder-only architectures, leveraging models pre-trained on general natural language like Llama <cit.>. Additionally, there's a shift in the training objective from utilizing code structures to simpler tasks like next token prediction and infilling.


From a continual learning perspective, the code domain presents unique advantages and challenges for domain-adaptive pre-trainig (DAP) compared to other domains discussed so far. On one hand, its hierarchical structure (general domain corpus → multi-language code → specific programming language) provides an ideal training pipeline for DAPs <cit.>, offering potential for more efficient training strategies. On the other hand, programming languages adhere to strict grammars, unlike the more fuzzy and context-dependent nature of natural language. Consequently, language models should ideally leverage these structures through tailored designs. However, given the vast availability of code for training, particularly in popular languages like Python and Java, adopting the same training objectives as for natural languages may yield sub-optimal results. Therefore, many existing works in the field omit domain-adaptive pre-training <cit.>.
In the following section, we will introduce existing code LLMs that employ DAP before the final downstream tasks, discussing both their common attributes and unique characteristics.

Representing a series of notable works that focus solely on adaptation to target domains, 
CodeGen <cit.> comprises a suite of LLMs designed for natural language (CodeGen-NL), multi-lingual programming languages (CodeGen-Multi), and mono-lingual programming languages (CodeGen-Mono). These models are trained sequentially, with each subsequent model initialized from the previous one trained on more general-domain data. Specifically, CodeGen-NL is trained on Pile <cit.>, while CodeGen-Multi further trains on a subset of the BigQuery dataset[Link to BigQuery: https://cloud.google.com/bigquery/public-datahttps://cloud.google.com/bigquery/public-data.], which includes source code from six popular programming languages (C, C++, Go, Java, JavaScript, and Python). Subsequently, CodeGen-Mono is trained on BigPython, a large-scale corpus of Python code. Throughout the training process, consistent pre-training objectives are employed, focusing on next token prediction.
Comment-Aug <cit.> addresses the challenge of aligning programming languages with natural languages (PL-NL alignment). Recognizing that denser comments in existing code could enhance the model's generation capabilities, Comment-Aug proposes a self-augmentation strategy. Initially, it enhances LLMs with comment generation ability through instruction tuning. Then it augments sparsely commented code by generating additional comments, followed by further pre-training on the enriched code data.
StarCoder <cit.> introduces two models: StarCoderBase and StarCoder. StarCoderBase is initially trained on a mixed dataset comprising various programming languages. With the aim of benefiting a broader user base, it maintains the original data distribution without significant reweighting. Subsequently, StarCoderBase undergoes further fine-tuning on an additional 35 billion tokens of Python code, resulting in the development of StarCoder.
DeepSeek-Coder-v1.5 <cit.> originates from DeepSeek-LLM <cit.> and undergoes pre-training on 2 trillion tokens, comprising 87% source code, 10% English code-related natural language, and 3% Chinese natural language corpus. Unlike Deepseek-Coder, which employs both next token prediction and fill-in-the-middle objectives, DeepSeek-Coder-v1.5 focuses solely on next token prediction during the stage of DAP. Despite this change, initialization from a general-domain LLM results in improved performance across various tasks, including natural language and mathematical reasoning, with minimal performance degradation on coding tasks. This underscores the efficacy of DAP before final downstream tasks in achieving balanced performance across general and domain-specific domains.


As the only work investigated so far that utilizes the general data replay to mitigate vertical forgetting in the code domain,
Code Llama <cit.> introduces a sophisticated training framework tailored for various coding tasks and model sizes, including 7B, 13B, 34B, and 70B variants. Initialized from Llama 2 weights, these models undergo continuous pre-training on a dataset composed of deduplicated public code, discussions about code, and a subset of natural language data. This mix of natural language data serves as a form of pseudo-replay to maintain the models' proficiency in understanding natural language. During DAP, Code Llama optimizes two objectives: autoregressive next token prediction and code infilling prediction (except for the 34B model, which excludes infilling). Subsequently, Code Llama undergoes further refinement through long-context fine-tuning (LC-FT) to enhance its repository-level understanding. Building upon Code Llama, Code Llama-Instruct undergoes additional instruction-tuning, while a Python-specific variant of Code Llama undergoes additional domain-adaptive pre-training on Python language datasets before LC-FT.

Architecture expansion has proven effective in acquiring robust coding abilities and preventing vertical forgetting simultaneously.
IRCoder <cit.> investigates the potential of utilizing compiler intermediate representations to enhance the multilingual transferability of Code LLMs. By conducting DAP on code grounded in intermediate representations with LoRA <cit.>, IRCoder achieves superior multilingual programming instruction following, enhanced multilingual code understanding, and increased robustness to prompt perturbations.
Llama Pro <cit.>, initialized from Llama2, undergoes continuous pre-training on a combination of code and math data. It expands the original Llama2 architecture by dynamically adding multiple identity copies of the transformer blocks. These added blocks initially serve as identity mappings, preserving the original functionality, and will be tuned to adapt to the DAP data. The proposed expansion method is asserted to be more resilient against catastrophic forgetting of previous general knowledge compared to sequential fine-tuning and other parameter-efficient tuning methods like LoRA.

The three aforementioned works highlight the importance of domain-adaptive pre-training in the realm of code. However, it is crucial to note that the problem definition and conventional architectures of existing Code LLMs may present compatibility challenges for DAP deployment. This situation leads to trade-offs between specialization in the code domain and generalization ability in the natural language domain.


Other Domains.  ECONET <cit.> enhances the model's ability to reason about event temporal relations through a dedicated continual pre-training phase. In this phase, temporal and event indicators are masked out, and a contrastive loss is applied to the recovered masked tokens. Results demonstrate that incorporating this domain-adaptive pre-training stage significantly improves performance on final tasks compared to direct fine-tuning.
In Concept-Aware Language Model (CALM) <cit.>, the authors introduce a data-efficient domain-adaptive pre-training approach. This method incorporates both generative and discriminative commonsense reasoning to enhance the concept-centric commonsense reasoning ability of large language models. Similar to ECONET, CALM's DAP phase is specifically tailored for concept-centric reasoning tasks. Consequently, even a small number of data examples for continued pre-training can lead to notable improvements in downstream tasks.

EcomGPT-CT <cit.> is a large language model tailored for the E-commerce through domain-adaptive pre-training. Given that E-commerce data often exhibits a semi-structured format, stored in tables or databases, EcomGPT-CT employs a data mixing strategy. This approach transforms semi-structured data into a set of nodes and edges, samples a cluster of nodes, and then extracts and concatenates them into a training example. During DAP, it combines the general-domain corpus with E-commerce data at a ratio of 2:1, which is significantly lower than the common setting adopted by other works.

Notably, AdaptLLM <cit.> challenges the data efficiency of conventional DAP by adopting a novel approach inspired by human learning patterns in reading comprehension. Following techniques from previous work <cit.>, it transforms raw corpora into (raw text, question, answer) format, creating intrinsic tasks. The model is trained on reading tasks (next token prediction on the raw text) and comprehension tasks (question-answering based on the corpora). During DAP, to ensure instruction diversity beyond predefined templates, it integrates substantial amounts of general instruction tuning data at ratios ranging from 1:1 to 1:2 across various domains like biomedicine, finance, and law. Compared to traditional DAP approaches that utilize raw domain-specific corpora as-is, AdaptLLM's method demonstrates superior domain-specific knowledge adaptation and minimal vertical forgetting.






 §.§ Continual Fine-Tuning (CFT)





















Background of Continual Fine-Tuning (CFT).  Continual Fine-Tuning (CFT) lies at the bottom layer of the vertical continuity, where models are trained on successive homogeneous tasks drawn from an evolving data distribution. As the service-oriented layer of LLM, it doesn't require consideration of further adaptation to another downstream tasks, simplifying optimization objectives to a great extent: better adaptation and less forgetting. In essence, the goal is to improve the overall performance.
The challenge of CFT has been extensively explored in the continual learning community. Essentially, all existing continual learning literature can be seen as a variant of CFT: models are either randomly initialized or initialized from pre-trained weights and undergo CFT thereafter. In the realm where continual learning intersects with natural language processing, we will only briefly outline the most notable works in this domain in <ref>, and direct interested readers to additional survey literature on this topic <cit.>.


In the era of LLMs, new computational paradigms in CFT have emerged and attracted significant attention within the research community. These topics include Continual Instruction Tuning (CIT, sec:cft-cit), Continual Model Refinement (CMR, sec:cft-cmr), Continual Model Alignment (CMA, sec:cft-cma), and Continual Learning for Multimodal Language Models (CMLLMs, sec:cft-cmllm).
While all these fall under the umbrella of CFT, each presents distinct features and challenges. In CIT, models must generalize to new tasks encoded in instructions, requiring semantic understanding <cit.>. CMR demands fine-grained, possibly example-level, operations for model refinement, differing from task-based approaches <cit.>. CMA aligns models with evolving human preferences, challenging due to subjective nature and lack of clear task boundaries <cit.>. In CMLLMs, addressing the composite architectural design and preventing catastrophic forgetting are key challenges <cit.>. Detailed exploration of these sub-categories follows in subsequent chapters.



Summary of Continual Fine-Tuning LLMs.  We have organized existing research on continual fine-tuning in tab:cft, categorizing studies into sub-categories: (i) general continual fine-tuning, (ii) continual instruction tuning (CIT), (iii) continual model refinement (CMR), (iv) continual model alignment (CMA), and (v) continual multimodal LLMs (CMLLMs). The table includes details on incremental learning types (X-IL), LLM architecture, and employed continual learning techniques and evaluation metrics. After discussing general observations on CFT in sec:cft-obs, we'll delve into each sub-category in detail.
















































































































































  §.§.§ General Observations about Continual Fine-Tuning








Examining the landscape of continual learning in the context of large language models (LLMs), and combined with the results shown in tab:cft, we can discern significant trends and shifts within the research community. Firstly, there has been a noticeable transition in focus from class-incremental learning to domain-incremental and task-incremental learning paradigms. For example, in 12 papers of general continual fine-tuning (the first white section), only 3 papers study continually fine-tuning language models in the setting of class-incremental learning. In the remaining four topics, CIT completely belongs to the field of task-incremental learning, as the instruction provided to the models can be seen as a soft encoding of the task information; CMR completely belongs to the field of domain-incremental learning, as different editing examples follows the same problem definition, and no task information is provided during inference. In CMA and CMLLMs, no examples of class-incremental learning has been reported. 

It has been a longstanding common sense in the continual learning community that class-incremental learning, as it requires the model to predict the context label and within-context label at the same time <cit.>, is the most challenging continual learning scenario and hence receives most of the attention from the community. 
However, the evolution observed so far suggests a recognition of the broader spectrum of challenges faced in real-world applications of continually learning LLMs. This growing awareness of the importance of task-incremental and domain-incremental learning underscores the necessity for more comprehensive and specific approaches to continual LLMs, with precedent continual learning literature provided as reference. 

Furthermore, in continual fine-tuning (CFT), continual learning techniques enjoy broader adoption and explicit exploration compared to CPT and DAP. 
In tab:cft, all 34 papers explicitly deploy the continual techniques, 50% of which develop their own new techniques that cannot be easily categorized into the three mainstream sets of continual learning algorithms. 
For example, 

in instruction tuning, SAPT designs a shared attentive learning framework to enable the catastrophic forgetting mitigation and knowledge transfer at the same time <cit.>; 
in model refinement, Larimar proposes to adopt an external memory system, Kanerva Memory, that supports operations of read, write, and generate <cit.>; 
in model alignment, AMA proposes adaptive model averaging method to achieve Pareto-optimal when averaging different models for a reward-tax trade-off (alignment-performance trade-off) <cit.>;
in continual learning multimodal LLMs, <cit.> devises a novel method named Reconstruct before Query (RebQ), harnessing the multi-modal knowledge from a pre-trained model to reconstruct the absent information for the missing modality.

Hence we can further conclude that, beyond mere replication of existing techniques, researchers are actively developing tailored solutions for various scenarios where an LLM needs to be continually updated directly for downstream tasks. 
This emphasis underscores the recognition of continual learning as a pivotal component in the development of resilient and adaptive language models, and further signals a maturation of the field towards more specialized and effective continual learning methodologies for large language models.




  §.§.§ General Continual Fine-Tuning (General CFT)









Researchers have long investigated the phenomenon of forgetting resilience in pre-trained large language models when fine-tuned for various downstream tasks <cit.>. Although the pre-trained weights initially position the model in a flat-loss basin, aiding adaptation to future tasks without severely impacting previous ones <cit.>, zero or near-zero forgetting is only observed at the representation level. This implies that while the model retains its ability to distinguish between task-specific representations, it may still forget specific task details <cit.>. Therefore, additional measures are necessary when deploying these models in real-world applications <cit.>. For instance, <cit.> demonstrate that relying solely on the intrinsic anti-forgetting and generalization abilities of LLMs can be risky in scenarios with changing code and API distributions, and simple baseline continual learning techniques can significantly enhance performance.


Many studies advance beyond naive sequential fine-tuning, leveraging the inherent anti-forgetting nature of LLMs while avoiding overly complex techniques of continual learning. They favor sequential fine-tuning due to its simplicity and predictability in real-world scenarios <cit.>. For instance, LR ADJUST <cit.> proposes a straightforward yet effective method of dynamically adjusting the learning rate to mitigate the overwriting of knowledge from new languages onto old ones. Building on the innate anti-forgetting ability of large language models like Pythia <cit.>, SEQ^* <cit.> introduces several strategies for fine-tuning LLMs on a sequence of downstream classification tasks: (i) freezing the LLMs and old classifiers after the warm-up phase and learning new tasks, respectively; (ii) employing (cosine) linear classifiers when old data is (not) available; and (iii) pre-allocating future classifiers.



Given the minimal forgetting observed at the representation level in continual learning, some studies aim to tackle the misalignment between the representation space and the decision-making layers, such as the final classification layer, by introducing representation-level constraints during continual fine-tuning. NeiAttn <cit.> exemplifies this approach by formulating classification tasks as masked language modeling and proposing a neighboring attention mechanism to counteract negative representation drift. This method seamlessly complements existing continual learning techniques <cit.>, as its focus is independent of mainstream continual learning methods.



Another line of approaches delves into refining the input/output format and network architectures of pre-trained language models, crafting specific structures and training methods to tackle the challenges of continual fine-tuning. 
For instance, CTR <cit.> integrates a pre-trained model at the onset of incremental learning, incorporating two CL-plugin modules within transformer layers. These modules consist of a task-specific module (TSM) for acquiring task-specific knowledge and a knowledge-sharing module (KSM) for selectively transferring previously learned similar knowledge. 


CIRCLE <cit.> manually designs diverse prompt templates for various types of buggy code, unifying them into representations solvable through cloze tasks. It employs difficulty-based example replay to enhance continual program repair, outperforming existing automatic program repair (APR) methods. LFPT5 <cit.> addresses lifelong few-shot language learning by consolidating sequence labeling, text classification, and text generation tasks into a text-to-text format. The model, treated as a generative model, undergoes prompt tuning and data generation, generating pseudo-examples from previous domains during adaptation to new tasks and performing knowledge distillation. 

In <cit.>, the authors propose a method for adaptively adding compositional adapters during continual sequence generation tasks. Before training on new domains, a decision stage determines which trained module can be reused using hidden state mixing. During training, this module addresses the task of fine-tuning while simultaneously regenerating examples.
C3 <cit.> merges parameter-efficient fine-tuning (PEFT) and in-context learning (ICL) in a teacher-student framework. The teacher model undergoes in-context tuning focused solely on the current domain, while the student model, equipped with tunable prompts, minimizes the KL-divergence of both models' autoregressive token distributions per step to retain the teacher model's few-shot capability.











  §.§.§ Continual Instruction Tuning (CIT)


















Instruction Tuning (IT) is a technique used to refine the instruction-following capabilities of LLMs <cit.>. While LLMs are typically pre-trained on extensive and diverse corpora, they may struggle with specific tasks despite their general knowledge. Numerous studies have shown that IT can notably improve LLMs' ability to follow textual instructions for particular tasks <cit.>, leveraging the pre-existing knowledge within LLMs to bridge the gap between general and task-specific performance <cit.>. Additionally, IT enhances the intuitive interaction between humans and LLMs, providing a more natural interface and aligning LLM outputs more closely with human expectations and preferences <cit.>.



Continual Instruction Tuning (CIT).  When the instruction tuning data comes in as a stream, this series of the fine-tuning tasks need to address the challenge of catastrophic forgetting of the general knowledge. 

CT0 <cit.> represents the inaugural study on continual learning in fine-tuned LLMs, utilizing the replay method during the IT process on the base T0 model. This model successfully learns 8 new tasks while maintaining robust performance on previously learned tasks. Subsequent research has focused on enhancing the replay method used during CIT. For instance, <cit.> improve replay efficiency by computing Key-part Information Gain (KPIG) on masked parts to dynamically select replay data, addressing the “half-listening” issue in instruction following. Similarly, SSR <cit.> uses the LLM to generate synthetic instances for replay, achieving superior or comparable performance with greater data efficiency than traditional methods.

Other approaches include combining multiple continual techniques during CIT. DynaInst <cit.> merges parameter regularization with dynamic replay, selectively storing and replaying instances and tasks to enhance outcomes. InstructionSpeak <cit.> employs negative training and replay instructions to improve both forward transfer and backward transfer. Additionally, some methods incorporate Parameter Efficient Tuning (PET). Orthogonal Low-Rank Adaptation (O-LoRA) learns new tasks within an orthogonal subspace while preserving LoRA parameters for previous tasks <cit.>. Shared Attention Framework (SAPT) combines a PET block with a selection module via a Shared Attentive Learning & Selection module, and tackles catastrophic forgetting and knowledge transfer concurrently <cit.>.
While regularization-based and architectural-based methods require additional parameter storage and GPU memory during fine-tuning, which potentially leads to higher training costs, replay-based methods remain popular due to their simplicity and data efficiency <cit.>. These methods are particularly favored in traditional continual learning scenarios for tuning LLMs.

CIT vs Conventional CL.  Both CIT and conventional CL aim to enable LLMs to acquire new tasks or information over time while retaining previously learned knowledge. However, CIT specifically utilizes rich natural language instructions to enhance LLMs' ability to follow human instructions <cit.>. The natural-language encoding of task information enables the potential for positive forward transfer when semantically similar tasks are encountered in the data stream. This scenario is typically challenging to engineer manually in conventional continual learning setups.
In contrast, conventional CL focuses on broader knowledge acquisition and is not limited to instruction-based learning. While CIT is predominantly applied within LLMs, conventional CL is employed across various fields, including vision, multimodal models and robotics. Regarding challenges, both approaches contend with catastrophic forgetting; however, CIT additionally concentrates on refining instruction-following capabilities to better interact with human needs. Conversely, conventional CL emphasizes building robust generalizability to manage variations within and across tasks <cit.>. 




  §.§.§ Continual Model Refinement (CMR)


Like humans, LLMs are prone to errors, such as inaccurate translations or outdated information <cit.>. To keep LLMs updated with the evolving factual knowledge, directly fine-tuning the model to correct these mistakes can be time-consuming and may disrupt its performance on previously learned tasks. To overcome these challenges, Model Refinement (also known as Model Editing) is proposed. This approach aims to rectify the model's errors while preserving its performance on other inputs, with only moderate computing resources <cit.>.
The concept of model editing was initially explored in <cit.>, which introduced a “reliability-locality-efficiency” principle and proposed a gradient descent editor to address it efficiently. Subsequent research, such as <cit.> and <cit.>, extended this principle to edit factual knowledge in BERT-based language models and larger models like GPT-J-6B <cit.> and T5-XXL <cit.>, respectively, using gradient decomposition. These approaches typically update a subset of model parameters to alter the labels of specific inputs. Additionally, memory-based models, as discussed in <cit.> and <cit.>, incorporate editing through retrieval mechanisms.


Model Refinement as a Special Form of Continual Learning.  Although model refinement and continual learning are typically treated as separated fields, they share fundamental similarities. The principle of locality in model refinement, ensuring that new knowledge doesn't disrupt responses to other inputs, aligns with the non-forgetting objective of continual learning. Furthermore, the principles of reliability—effectively updating the model—and efficiency are crucial in both model refinement and continual learning Thus, the problem of model refinement can be viewed as akin to continual learning, where a small batch of updated samples {(_e, ŷ_e)} represents a new task. Alternatively, one can consider the stream of such samples for updating as a specialized form of online continual learning <cit.>.






Continual Model Refinement (CMR).  

Recent works have combined continual learning with model refinement, resulting in a new problem termed continual model refinement (CMR). This concept extends model refinement horizontally, presenting updated sample pairs (_e, y_e, ŷ_e)^e=1_N sequentially as a stream. <cit.> initially introduces this idea as continual model refinement, evaluating various continual learning methods with a dynamic sampling algorithm. To solve CMR problem, many methods employ a retrieval mechanism. For instance, <cit.> uses hidden activations of the language model as a “key” to activate updated parameters only when input x_0 resembles updated sample pairs; 
<cit.> improves this approach's efficiency by integrating LoRA <cit.>; <cit.> augments the LLM with an external episodic memory, modeling CMR as an ongoing memory refresh. 
Meanwhile, some methods focus solely on updating a subset of model parameters. For example, <cit.> addresses the issue of “toxicity buildup and flash” in single-editing methods like ROME <cit.>, adapting it to a continual context with a knowledge-aware layer selection algorithm.

































While all these works pioneer research in CMR, the exploration of continual model refinement for LLMs remains open. <cit.> highlighted a potential problem: the location for storing the fact may not coincide with the best place for editing it. This challenges the classical "locate and edit" paradigm used by several model editing methods <cit.>, and could become a significent concern for CME <cit.>. Other questions, including whether such problem setting fits LLMs and whether more memory/computationally efficient methods of CMR could be developed for LLMs, are yet to be explored and answered.






































  §.§.§ Continual Model Alignment (CMA)


Model Alignment (MA) is a crucial concept in the development and deployment of AI systems, ensuring that their actions and outputs align with human values, ethics, and preferences.
It can be defined as the process of adjusting the objectives and functioning of an AI system to achieve such goals, involving a combination of mathematical models, algorithmic adjustments, and iterative feedback to refine AI behavior <cit.>.
MA can be broadly categorized into two types: Reinforcement Learning-based (RL-based) and Supervised Learning-based (SL-based). In the RL-based approach, as discussed in <cit.>, models are trained to make decisions reinforced by human feedback, using a reward system to guide them towards desirable outcomes. Conversely, the SL-based approach, as seen in <cit.>, directly trains models on datasets of human preferences, aligning their output with demonstrated human values. Both approaches leverage a combination of algorithmic learning techniques and human feedback to progressively refine and align model behaviors.


When language models undergo the phase of model alignment (MA) to align with specific ethical or normative standards, the performance degradation occurs. 
In <cit.>, the authors refer to this phenomenon of general knowledge forgetting induced by model alignment as the “Alignment Tax”. 

Notably, even a single stage of MA can diminish the model's performance capabilities, as it restricts the model's responses to a narrower subset of its training distribution. This paper focuses on understanding and reducing the impact of the Alignment Tax in the process of refining AI models, highlighting the balance between alignment and maintaining broad model capabilities.

Continual Model Alignment (CMA).  Continual Model Alignment (CMA) aims to continuously refine LLMs to align with evolving human values, ethics, and data. 
The significance of CMA lies in its capacity to ensure that LLMs retain their relevance, accuracy, and ethical alignment over time. This ongoing adjustment is essential to navigate the complexities introduced by concept drift, the evolution of data, and shifts in societal values. The static nature of LLM training on historical data sets can lead to discrepancies between the models' outputs and current factual accuracies, societal norms, and standards, making CMA a crucial process for maintaining their adaptability and alignment with contemporary contexts <cit.>. 
Despite its importance, CMA faces several challenges: 
(i) scalability and resource intensity issue: continually updating LLMs requires significant computational resources and human oversight, posing scalability issues for the practitioners; 
(ii) ensuring ethical alignment: balancing model updates to reflect societal changes without introducing or perpetuating biases remains a complex issue; 
(iii) data privacy and security: continuously integrating new data into LLMs raises concerns regarding data privacy, security, and the potential for misuse.

Emerging strategies in CMA emphasize the optimization of LLMs for enhanced adaptability, addressing the necessity for these models to accommodate continuous changes in language use, information validity, and societal expectations. Notably, innovative research efforts such as those detailed in <cit.>, and <cit.>, highlight the development of methodologies designed to minimize the challenges associated with the continuous realignment of LLMs. These studies underscore the importance of implementing specific optimization strategies to foster the adaptability of LLMs, ensuring their outputs remain ethically attuned and factually relevant in the face of dynamic societal and informational landscapes. 

Likewise, there are two types of CMA frameworks: RL-based and SL-based. 

In the realm of RL-based CMA, two significant contributions have been noted: <cit.> explores methodologies for reducing the overhead associated with continual alignment through efficient reinforcement learning techniques, and <cit.>, which suggests a framework for applying continual learning principles to reinforcement learning with human feedback, potentially mitigating the alignment tax over time. Employing reinforcement learning for CMA focus on developing efficient reinforcement learning techniques that integrate continual learning principles and human feedback to dynamically maintain alignment with evolving human values while minimizing computational overhead and mitigating forgetting.
For SL-based CMA, <cit.> presents an innovative approach by integrating supervised continual learning techniques with the process of aligning AI systems through direct training on evolving datasets of human preferences. This methodology promises a more sustainable and adaptable model for maintaining alignment with human values over extended periods and across various contexts. The use of supervised continual learning suggests a focus on preventing catastrophic forgetting, a common challenge where a model loses its ability to perform previously learned tasks upon learning new ones.


In summary, both RL-based and SL-based frameworks need to address forgetting, especially in scenarios where the model continuously integrates new information. Effective strategies might include techniques (such as EWC <cit.>, experience replay <cit.>, and dynamic re-weighting <cit.>), which help the model retain old knowledge while integrating new insights.
Future research in CMA aims to develop more efficient, automated processes for model updates, better mechanisms for ethical oversight, and innovative solutions to balance model relevance with privacy and security concerns. Existing streams of research highlight the importance of developing adaptable, efficient, and robust AI systems that can continually align with human values without substantial losses in performance or increased computational costs. Future research could explore hybrid models that combine RL-based and SL-based approaches, potentially offering a more holistic framework for continual model alignment.


        










































































  §.§.§ Continual Multimodal Large Language Models (CMLLMs)



















Multi-modal LLMs have garnered significant attention for their capacity beyond single modality. These models integrate data from multiple modalities, like texts, images and videos to enhance real-world information comprehension <cit.>. Typically, MLLMs consist of modality-specific sub-modules such as pre-trained vision encoders, large language models, and projectors for cross-model alignment. This alignment is essential for MLLMs to fuse the diverse data types and promote their comprehension. For example, MiniGPT-4 <cit.> utilizes a linear projector to align frozen vision encoders and language models; LLaVA <cit.> employs a simple linear layer to connect image features and instruction into the word embedding space. Currently, all existing MLLMs are pre-trained on large scale multi-modal datasets and then fine-tuned on specific small downstream datasets, which constitutes the training process <cit.>.

Several existing studies have explored the causes of catastrophic forgetting when continually training MLLMs. <cit.> performs singular value decomposition on input embeddings, revealing a significant disparity among different input embeddings. This discrepancy causes the model to learn irrelevant information for previously trained tasks, resulting in catastrophic forgetting and negative forward transfer. 
<cit.> observes that minority collapse may lead to catastrophic forgetting, when the imbalance ratio between majority and minority classes approaches infinity during fine-tuning. It further identifies hallucination as a contributing factor to performance degradation in MLLMs. 

Continual Pre-Training MLLMs.  Training an MLLM to be updated with the changing world from scratch is resource-intensive, requiring considerable time and cost. While continual learning offers a promising solution to this problem, trivially applying past methods is not advisable, given the distinct structure of MLLMs that includes modality-specific sub-modules and cross-model alignment. Currently, there is an apparent lack of continual pre-training for MLLMs, and further exploration into continual and joint pre-training of MLLMs is necessary.

Continual Fine-Tuning MLLMs.  In contrast to traditional continual learning methods that involve full-model fine-tuning for new tasks, continual fine-tuning for MLLMs focuses on refining specific layers when adapting to new tasks <cit.>. Given the strong capabilities of pre-trained models, training specific layers suffices, and can simultaneously reduce computational demands. 
<cit.> additionally considers an continual learning scenario, Continual Missing Modality Learning (CMML), where different modalities are emerging throughout the incremental learning stages.
All the aforementioned studies collectively indicate that MLLMs still suffer from catastrophic forgetting, which manifests in two ways: along the direction of vertical continuity, a performance decline on pre-trained tasks following fine-tuning for downstream tasks; and along the axis of horizontal continuity, a performance degrade on previously fine-tuned tasks after fine-tuning for new tasks. <cit.> also observes negative forward transfer, where the performance of unseen tasks degrades when learning new tasks, indicating a decline in model generalization capability.



Continual Learning MLLMs.  While traditional CL methods are applicable, some may not yield optimal results, as evidenced by various experiments <cit.>. For instance,
<cit.> observes a consistent efficacy of replay-based and model expansion strategies across diverse scenarios of continual fine-tuning MLLMs, but regularization-based methods only perform well on models that have been jointly instruction-tuned on multiple tasks. 
Other works seek to develop ad-hoc solutions for continual learning MLLMs.
<cit.> proposes EProj to expand the projection layer in MLLMs for each new task and utilizes task-similarity-informed regularization (TIR) to enhance performance. <cit.> introduces Fwd-Prompt, a prompt tuning method that projects prompt gradient to both the residual space and the pre-trained subspace to minimize the interference between tasks and reuse pre-trained knowledge respectively, fostering positive forward transfer without relying on previous samples. 
<cit.> focuses on the forgetting of the pre-trained MLLMs after fine-tuned on specific tasks and proposes model tailor to compensate the selected subset that are critical for enhancing target task performance. 
<cit.> presents a novel method named Reconstruct before Query (RebQ), leveraging  the multi-modal knowledge from a pre-trained model to reconstruct the absent information for the missing modality. Recently, MoE (Mixture-of-Experts) framework has gained attention which resembles the architecture-based methods in CL. It provides the model with
the ability to learn different intentions from distinct experts, e.g., 
<cit.> first introduces MoELoRA to fine-tune LLaVA, effectively mitigate the catastrophic forgetting of MLLMs in CoIN and the results demonstrate the effectiveness.


In concluding remarks on Continual Learning for MLLMs, the role of templates in instruction tuning emerges as crucial. As highlighted by <cit.>, employing similar templates across tasks proves more advantageous, aiding in knowledge retention and forgetting mitigation. This approach fosters task-specific learning, reducing reliance on common knowledge prone to forgetting in sequential contexts. In addition, it is noteworthy that the forgetting induced by the gap between tasks is more critical than the forgetting induced by the distributional gap between datasets. According to <cit.>, moderate fine-tuning is advantageous for non-fine-tuned tasks, excessive fine-tuning ultimately leads to catastrophic forgetting in these tasks. <cit.> discovers the multi-task joint instruction tuning at the beginning state can facilitate the model’s continual learning ability and mitigate forgetting. 
Overall, continual learning in MLLMs holds promise, but further research is needed to fully realize its potential.





§ EVALUATION PROTOCOLS AND DATASETS


In this section, we introduce the evaluation protocols and datasets for continually learning large language models. In sec:eval, we discuss common continual learning evaluation metrics adapted for this context, along with metrics designed specifically for continual LLMs. Then, in sec:data, we outline the datasets available for each discussed topic.



 §.§ Evaluation Protocols


In the realm of conventional continual learning, where task streams take the form of classification, many metrics rely on the concept of Accuracy Matrix <cit.>.
Extending this notion to the context of continually learning LLMs, we introduce the Performance Matrix ∈ℝ^T× T, where T represents the total number of training stages.
Each entry of  corresponds to a performance metric evaluated on the models, such as perplexity on pre-training data <cit.>, zero-shot/few-shot evaluation metrics on downstream data without fine-tuning <cit.>, fine-tuned accuracies on downstream tasks <cit.>, and probing accuracies from fine-tuning add-on components evaluated on downstream tasks <cit.>.
In , P_i,j denotes the model's performance after training on task i and evaluating on task j. With this Performance Matrix definition, we introduce the primary evaluation protocols widely adopted.

Overall Performance (OP).  The Overall Performance (OP) <cit.> is a natural extension of the concept of Average Accuracy <cit.>. The OP measured up until training stage t is the average performance of the model trained right after the stage t. Denote it as OP_t and we have: 

    OP_t    ≜1/t∑_i=1^t P_t,i.

As noted in <cit.>, the OP corresponds to the primary optimization objective defined in def:til, <ref>, and <ref>. In much of the continual learning literature, once all T tasks are completed, the final OP (OP_T) is reported, with the subscript _T often omitted for brevity. 
In some works, OP is weighted by the importance of tasks ÕP̃≜1/T∑_i=1^T w_i P_t,i, where w_i = N_i / ∑_j=1^T N_j represents the ratio of data. In some literature, ÕP̃ is referred to as “example accuracy” <cit.>, “whole accuracy” <cit.> or “edit success rate” in  CMR <cit.>. 

Forgetting (F).  Define F_t as the forgetting up to task t, which represents the largest performance drop observed throughout the training process, averaged over t training stages:

    F_t    ≜1/t-1∑_j=1^t-1[ max_l∈ [t-1]{P_l,j - P_t,j}].

Typically, researchers report the average forgetting F=F_T at the end of the entire training process.
Forgetting quantifies the impact of learning new tasks on previously acquired knowledge. Ideally, a robust continual learning framework should achieve Backward Transfer (BWT), where learning new tasks enhances performance on prior tasks. This enhancement is typically measured by negating the forgetting, thus indicating an improvement in performance on earlier tasks. The concepts of Forgetting and Backward Transfer underpin various evaluation metrics, such as knowledge retention <cit.>, performance on unchanged knowledge <cit.>, average increased perplexity (AP^+) <cit.>, and test and edit retention rate in CMR <cit.>.


Forward Transfer (FWT).  Forward Transfer measures the generalization ability of the continual learning algorithms. Formally, forward transfer FWT_t up to training stage t is defined as

    FWT_t    ≜1/t-1∑_i=2^tP_i-1, i - b_i,

where b_i is the baseline performance of the model evaluated on task i before undergoing continual learning. Strictly speaking, the definition of b_i is not the same as defined in the previous work <cit.>, where it is used to denote the performance of a random initialization of the model.
Additionally, we extend the notation of forward transfer in the vertical direction to represent the performance improvement on downstream tasks resulting from domain-adaptive pre-training (see tab:dap). 
Forward Transfer is alternatively referred to as temporal generalization <cit.> or knowledge transfer <cit.> in some literature.

LAnguage Model Analysis (LAMA).  LAnguage Model Analysis (LAMA) is an evaluation framework designed to probe the world knowledge embedded in language models <cit.>.
It converts each world fact into a cloze statement, which is then inputted into the language models to predict the correct answer.
LAMA has been extended for continual pre-training, particularly for those under the temporal shifts <cit.>. In CKL, three LAMA benchmarks are constructed for different dimensions: InvariantLAMA assesses knowledge retention on time-invariant facts, UpdatedLAMA focuses on knowledge update, and NewLAMA evaluates knowledge acquisition <cit.>. 

Forgotten / (Updated + Acquired) Ratio (FUAR).  As the performance of a pre-trained LLM is decomposed into a fine-grained set in CKL <cit.>, OP becomes a too general metric and cannot accurately reflect the balance and trade-offs of the model's behavior. 
To address this issue, CKL proposes a joint evaluation metric FUAR (Forgotten / (Updated + Acquired) Ratio) for continual pre-training. 
A FUAR value of 1 represents an equal trade-off between the knowledge forgetting and knowledge learning: for each piece of updated or acquired knowledge, one piece of time-invariant knowledge is forgotten on average. A FUAR less than 1 suggests high learning efficacy, where more than one piece of knowledge is acquired at the expense of forgetting one piece of time-invariant knowledge.


X-Delta.  In TRACE <cit.>, the authors propose a set of "X-Delta" metrics for continual instruction tuning, quantifying the forward transfer on specific abilities of LLMs. Let's denote a set of M datasets {X_1, X_2, ⋯, X_M} for task X. The baseline performances of the pre-trained LLM evaluated on these tasks are denoted as {b_1^X, ⋯, b_M^X}. The model undergoes continuous fine-tuning on a different set of tasks, distinct from those used for evaluation. Throughout the sequential training process, the performance of the model after learning task t on evaluation tasks X_i is R_t,i^X. The X-Delta Δ R_t^X after learning task t is defined as:

    Δ R_t^X    ≜1/M∑_m=1^M (R_t,i^X - b_i^X).

In the public TRACE benchmark, the authors construct three sets of evaluation tasks to benchmark the ability of LLMs, including general ability, instruction following, and safety <cit.>.




NLG Score.  In continual model alignment, three prominent metrics used to evaluate different aspects of Natural language generation (NLG) are BLEU-4 <cit.>, METEOR <cit.>, and ROUGE-L <cit.>. BLEU-4<cit.>, designed for machine translation (MT), evaluates the precision of n-grams between the machine-generated and reference texts, focusing especially on four-word sequences to gauge fluency and adequacy. METEOR <cit.> also targets MT but aims to improve correlation with human judgment by considering synonyms and stemming, thus providing a more nuanced assessment of translation quality. On the other hand, ROUGE-L <cit.> is commonly applied in summarization tasks, assessing the longest common subsequence between the generated summary and a set of reference summaries, effectively measuring the recall of essential content. Each metric has its strengths and is tailored to specific kinds of language processing tasks, reflecting different dimensions of text generation quality. 

rPMS.  The reference PM score (rPMS) quantifies the alignment of a model's outputs with human preferences as captured by a reference Preference Model (PM). Specifically, for a given task t, the rPMS is defined as:

    rPMS_t = rPM(M_t, D_test, t),

where M_t is the model being evaluated, D_test, trepresents the test dataset for task t, and rPMS is the function implemented by the reference PM that scores model outputs based on their alignment with human preferences. Higher rPMS values indicate that the model M_t aligns closely with human preferences, suggesting effective learning and retention of the desired behaviors across tasks in a continual learning scenario. The metric is crucial for evaluating the extent to which models can maintain or improve performance relative to human preference standards without substantial forgetting over time.





 §.§ Datasets


In this section, we provide a comprehensive review of the datasets available for benchmarking continual LLMs, as illustrated in tab:datasets. We intentionally exclude datasets used for domain-adaptive pre-training LLMs in vertical domains such as legal, medical, and financial, unless they are specifically designed for continual domain-adaptive pre-training. Furthermore, we omit datasets used in general continual fine-tuning, as they have already been extensively studied in existing works <cit.>.


Datasets for Continual Pre-Training (CPT) and Domain Adaptive Pre-Training (DAP).  

Current research lacks a widely recognized benchmark for evaluating continual pre-training LLMs under temporal shifts.
TimeLMs utilizes a series of Twitter corpora collected until 2022, sequentially pre-training RoBERTa models quarterly <cit.>.
CC-RecentNews, adopted as unlabeled pre-training data for LMs in CKL <cit.>, consists of recent news and serves as a single-stage dataset. 
Additionally, CKL introduces InvariantLAMA, NewLAMA, and UpdatedLAMA to assess the principles of continual knowledge learning.
TWiki, a dataset derived from the articles of Wikipedia between August and December 2021, is curated and cleaned in TemporalWiki <cit.>. This dataset facilitates the exploration of incremental learning by providing the Diffsets between neighboring snapshots.

For works that study the content-level distributional shifts in CPT and DAP, researchers often resort to a similar set of publicly available datasets <cit.> to construct their own test beds for continual learning algorithms. 
The ^*DAPT dataset, developed by <cit.>, comprises four domains: BioMed and Computer Science from S2ORC <cit.>, News from <cit.>, and Reviews from <cit.>. In ^*DAPT's original study, each domain undergoes individual domain adaptive pre-training stages to demonstrate the universality of DAP's effectiveness. Subsequent works, such as ELLE <cit.> and Recyclable Tuning <cit.>, follow suit by employing these domains for multi-stage CPT.
DEMix <cit.> presents another large-scale dataset, featuring eight semantic domains with over 73.8 billion tokens. Alongside the training set, it includes eight additional datasets for validating the generalization ability of LLMs. 
On a smaller scale, ^*CPT <cit.> and ^*DAS <cit.> datasets consist of four and eight domains, respectively, with approximately 3.12 million examples and a size of 4.16GB each. These datasets are constructed similarly to the aforementioned ones.





    
    Summary of the existing benchmarks publicly available for Continual Learning LLMs. 
    In the column of Name, we use the superscript “^*” to denote the lack of the dataset name and the name shown is that of the original paper. 
    In this table, we deliberately omit the datasets used for domain-adaptive pre-training the vertical LLMs, as their main focus of development is not on continual learning. We also omit the datasets used for general continual fine-tuning, as they are extensively discussed in other existing surveys <cit.>. 
    
    
    1!

	[0.15em]
    Name     Type     Shift      Domain     #Stages     Scale     Sources     Applications     Comment 

    ^*TimeLMs <cit.>     CPT     Temporal     Social Media     8     #Examples: 123.86M      Tweets     <cit.>     https://github.com/cardiffnlp/timelms.code

    CC-RecentNews <cit.>     CPT     Temporal     News     1     #Tokens: ∼168M     Web     <cit.>     https://github.com/joeljang/continual-knowledge-learningcode

    TWiki <cit.>     CPT     Temporal     General Knowledge     5     #Tokens: 4.7B     Wikipedia     <cit.>     https://github.com/joeljang/temporalwikicode 

    ^*DAPT <cit.>     CPT 
 DAP     Content     Multi-Domain     4     Size: 160GB     BioMed <cit.>, CS <cit.>, News <cit.>, Reviews <cit.>     <cit.> 
 <cit.> 
 <cit.>     https://github.com/allenai/dont-stop-pretrainingcode 

    ^*CPT <cit.>     CPT     Content     Multi-Domain     4     #Examples: 3.12M     Yelp <cit.>, S2ORC <cit.>, AG-News <cit.>     <cit.>     https://github.com/UIC-Liu-Lab/CPTcode 

    ^*DEMix <cit.>     CPT     Content     Multi-Domain     8     #Tokens: 73.8B     1B <cit.>, CS <cit.>, Legal <cit.>, Med <cit.>
 WebText <cit.>, RealNews <cit.>, Reddit <cit.>, Reviews <cit.>     <cit.>     https://github.com/kernelmachine/demixcode 

    
    
    ^*DAS <cit.>     CPT 
 DAP     Content     Multi-Domain     6     Size: 4.16GB     Yelp <cit.>, Reviews <cit.>, Papers <cit.>, PubMed     <cit.>     https://github.com/UIC-Liu-Lab/ContinualLMcode 

    
    SuperNI <cit.>     CIT     Content     Mutli-Domain     16     #Tasks: 1616 
 #Examples: ∼5M     GitHub     <cit.>     https://github.com/allenai/natural-instructionscode 


    CITB <cit.>     CIT     Content     Mutli-Domain     19     #Tasks: 38     SuperNI <cit.>      <cit.>     https://github.com/hyintell/CITBcode 


    CoIN <cit.>     CIT     Content     Multi-Domain     8    #Examples: ∼1.14M     RefCOCO <cit.>,RefCOCO+ <cit.>,RefCOCOg <cit.> 
 ImageNet <cit.>, VQAv2 <cit.>, ScienceQA <cit.> 
 
    TextVQA  <cit.>, GQA <cit.>, VizWiz  <cit.>, OCR-VQA <cit.>     <cit.>     https://github.com/zackschen/CoINcode 

    TRACE <cit.>     CIT     Content     Mutli-Domain     8      #Examples: 56,000      ScienceQA <cit.>, FOMC <cit.>, MeetingBank <cit.>
 
    C-STANCE <cit.>, 20Minuten <cit.>, CodeXGLUE <cit.>, NumGLUE<cit.>     <cit.>     https://github.com/BeyonderXX/TRACEcode 

    NATURAL-INSTRUCTION <cit.>     CIT     Content     Mutli-Domain     6       #Examples: 193k     CosmosQA <cit.>, DROP <cit.>, Essential-Terms <cit.> 
 MCTACO <cit.>, MultiRC <cit.>, QASC <cit.> 
 Quoref<cit.> , ROPES <cit.> , Winogrande <cit.>     <cit.>     https://github.com/allenai/natural-instructions-v1code 

    IMDB <cit.>     CMA     Content    Social Media   1   Size: 217.35 MB    IMDB    <cit.>     https://huggingface.co/datasets/stanfordnlp/imdbcode

    HH-RLHF <cit.>     CMA     Content    General Knowledge   1   Size: 28.1 MB    Human Feedback    <cit.>     https://github.com/anthropics/hh-rlhfcode

    Reddit TL;DR <cit.>     CMA     Content   Social Media   2   Size: 19.6 GB   Reddit     <cit.>    https://zenodo.org/records/1043504code

    Common Sense QA <cit.> 
 Reading Comprehension <cit.>
 
    
    Translation <cit.>     CMA     Content    Multi-Domain   6    #Examples: ∼ 41.16M     ARC Easy and Challenge <cit.>, Race <cit.>, PIQA <cit.> 
 SQuAD <cit.>, DROP <cit.> 

    WMT 2014 French to English <cit.>
        <cit.>     see sources 

    FEVER <cit.>     CMR     Content     General Knowledge     1     #Examples: 420k      Wikipedia     <cit.>     code 

    VitaminC <cit.>     CMR     Content     General Knowledge     1     #Examples: 450k      Wikipedia     <cit.>     code 

    zsRE <cit.>     CMR     Content     General Knowledge     1     #Examples: 120M      Wikireading <cit.>     <cit.>     code 

    T-rex <cit.>     CMR     Content     General Knowledge     1     #Examples: 11M      Dbpedia abstracts <cit.>     <cit.>     code 

    NQ <cit.>     CMR     Content     General Knowledge     1     #Examples: 320k      Google queries, Wikipedia     <cit.>     code 

    CounterFact <cit.>     CMR     Content     General Knowledge     1     #Examples: 22k     zsRE <cit.>     <cit.>     code 

    SCOTUS <cit.>     CMR     Temporal     Law     1     #Examples: 9.2k      Supreme Court Database     <cit.>     code 

    
    
    
    
    
    
    
    
	[0.15em]
	
	




Datasets for Continual Instruction Tuning.  Measuring the effectiveness of CIT is crucial, particularly because traditional evaluation metrics may not be suitable for LLMs: many of them are overly simplistic and fail to comprehensively assess the model's ability to learn continually. New benchmarks and metrics are required to evaluate both the retention of old knowledge and the integration of new instructions. TRACE <cit.> stands as a continual learning benchmark designed specifically for LLMs, encompassing diverse tasks such as multilingual capabilities, code generation, and mathematical reasoning. CITB <cit.> represents another benchmark for CIT, incorporating both learning and evaluation protocols. It in addition demonstrates that replay generally yields the best performance across all methods. CoIN <cit.> extends the benchmark to MLLMs, incorporating a balanced and diverse set of instructions from vision-language datasets. 




Datasets for Continual Model Refinement.  

Most datasets for continual model refinement can be categorized into two types <cit.>: fact checking and question answering. For fact checking, models are asked to verify the truthfulness of certain claims, typically modeled as a classification task. Key datasets include FEVER <cit.> (used by <cit.>) and VitaminC <cit.> (used by <cit.>), both sourced from Wikipedia. For question answering, models are tasked with providing specific answers instead of choices. Zero-shot Relation Extraction (zsRE) <cit.> is the most widely employed dataset for this purpose  <cit.>, alongside Natural Questions (NQ)  <cit.> and T-rex  <cit.>. <cit.> adapted zsRE with additional counterfactuals to create the more challenging CounterFact dataset, used by  <cit.>. Beyond these two categories, SCOTUS <cit.> is also utilized <cit.> in the assessment of continual model refinement through a document classification task for U.S. Supreme Court cases into 11 topics.















Datasets for Continual Model Alignment.  In the domain of reinforcement learning with human feedback (RLHF), several datasets are commonly employed across different studies to evaluate the adaptation and effectiveness of models under varying scenarios and continuous learning conditions. The IMDB <cit.> and HH-RLHF <cit.> dataset, as introduced in <cit.> within their study on continual learning through optimal policy fitting, leverages data gathered from interactive RL scenarios to model human preferences dynamically. Similarly, the Reddit TL;DR dataset <cit.> used by <cit.> is focused on text summarization, providing a robust platform for testing the longevity and adaptability of learning algorithms under evolving conditions. Lastly, Common Sense QA <cit.>, Reading Comprehension <cit.>, and Translation <cit.>, which are utilized in <cit.> are selected to assess the challenges of aligning RL agents with human expectations without incurring significant performance penalties. Each of these datasets is pivotal in advancing the understanding of continual learning and the interplay between human feedback and machine learning adaptation.


Datasets for Continual Multimodal Large Language Models.  Following LLaVA <cit.>, many MLLMs adopt the pattern of instruction tuning to enable assessing alignment with human intention and knowledge preservation for reasoning. Thus, traditional tasks like image classification can be transformed to VQA tasks to evaluate the ability of MLLMs, which are otherwise challenging to assess using conventional methods. Several benchmarks have been proposed to evaluate the CL method for MLLMs. MCIT <cit.> proposes the first continual instruction tuning benchmarks, Benchmark1 and Benchmark2. The difference between benchmark1 and benchmark2 is that benchmark2 includes Multi-task Joint Instruction Tuning, which aims to explore whether multi-task joint instruction tuning improves the model’s continual learning ability.
<cit.> proposes EMT, the first classification evaluation framework to investigate catastrophic forgetting in MLLMs. <cit.> presents a comprehensive benchmark CoIN, spanning 8 task categories and evaluating MLLMs from two perspectives:  Instruction Following and General Knowledge, which assess the alignment with human intention and knowledge preserved for reasoning, respectively. 
<cit.> constructs two datasets, UPMC-Food101-CMML and MM-IMDb-CMML to benchmark the novel CMML task, which means the data of certain modalities is missing during continual fine-tuning. UPMC-Food101-CMM contains 101 food categories and 61,142 training, 6,846 validation, and 22,716 test image-text pairs. MM-IMDb-CMML is a multi-label classification dataset across 27 distinct movie genres, consisting of 15,552 training, 2,608 validation and 7,799 test image-text pairs.




§ DISCUSSION


In this section, we delve into the intersection of conventional computational patterns in continual learning and the training and deployment of large language models (LLMs). We begin by examining intriguing properties that arise during continual learning with LLMs. Next, we explore the evolving roles of three types of incremental learning within the context of LLMs. Following this, we contrast the roles of memory in continual LLMs with those in traditional continual learning. Finally, we conclude with a concise overview of promising directions for future research in this area.





 §.§ Intriguing Properties Emergent in Continual LLMs


Beyond the well-established resilience of pre-trained large language models (LLMs) against catastrophic forgetting compared to downstream-specific models <cit.>, there is a notable lack of exploration into other intriguing properties of LLMs when trained continually. While investigations into the emergent capabilities of continuously trained LLMs have attracted attentions from the community to a certain extent, they remain relatively limited. For instance, in <cit.>, it is observed that when fine-tuned sequentially and cyclically on a series of documents, large models exhibit a phenomenon known as “anticipatory recovering”. This refers to the LLMs' ability to recover forgotten information on documents even before encountering them again. 
This suggests that LLMs may possess the capability of sequential memorization, which could pave the way for research into memory replay and more complex structured learning environments as model parameters scale up.






 §.§ Conventional Types of Incremental Learning


As mentioned in sec:preliminary-cl, three types of incremental learning are prevalent <cit.>. Among them, class-incremental learning (CIL) has historically attracted significant attention from the community <cit.>. However, in the context of continually pre-training and adapting large language models (LLMs), we observe a decreased interest in CIL but an increased focus on task-incremental learning (TIL) and domain-incremental learning (DIL).

Given that language models are inherently designed for content generation and are pre-trained with the pretext generative task of next-word prediction, it is natural to emphasize the patterns of generative tasks and integrate the traditional CIL paradigm into the broader framework of language modeling, discarding the incremental classification head <cit.>. 
For instance, in Vocabulary-Aware Label Generation (VAG), CIL is redefined as the task of continual label generation. This approach utilizes a pre-trained encoder-decoder language model to generate class labels <cit.>. Meanwhile, in the Generative Multi-modal Model (GMM) for CIL <cit.>, image patches and prompts are concatenated and fed into the language model to generate classification results.

However, the declining attention to the conventional CIL paradigm does not suggest that these techniques are not impactful in the field of continual learning for LLMs. Nonetheless, many current research endeavors unwittingly employ such techniques, indicating their widespread adoption in various applications.
For example, techniques such as vocabulary expansion <cit.> can be seen as an extension of expanding the classification head in CIL. These CIL techniques can be further integrated into systems like Lifelong-MoE <cit.>, where adding a new expert to the transformer blocks requires updating the gating function to include the routing of the newly added expert. 
The EProj framework, described in <cit.>, employs a similar architecture, incorporating linear projectors for new domains alongside a selector module trained to route these projectors.
Since the aforementioned sub-modules operate on the principles of CIL, previously validated techniques can be directly applied.

The importance of domain-incremental learning (DIL) is self-evident, given the shared task definition and input-output format in continual pre-training (CPT) and domain-adaptive pre-training (DAP). As dynamically expanding token vocabularies can pose additional challenges, it is natural to focus on understanding distributional shifts within the input corpus while keeping the vocabulary fixed. 
On another front, task-incremental learning (TIL) attracts significant interest due to its potential for personalizing LLM services. For instance, users may desire options for selecting domain-specific experts, thereby making task IDs available throughout inference time <cit.>. Additionally, TIL plays a crucial role in instruction tuning, where instructions can be seen as natural-language-encoded task information <cit.>.
It is worth noting that the boundary between TIL and DIL becomes somewhat blurred in continual instruction tuning. Language models demonstrate the capability to infer domain information for unseen instructions, suggesting a convergence of TIL and DIL in certain contexts.





 §.§ Roles of Memory in Continual LLMs


Previous continual learning research, drawing inspiration from human learning patterns, primarily emphasizes the storage efficiency of past data. The setting of continual learning with limited memory size has garnered significant attention from the community. However, this focus may no longer hold true in the context of continual LLMs. In the direction of relaxing memory constraints, institutions with access to training data may opt to retain full access without restricting memory size, given that the cost of memory storage is more than affordable. In such scenarios, as highlighted in <cit.>, the challenge shifts from storage efficiency to computational efficiency. To achieve continual learning goals, models must efficiently adapt to new data (efficient adaptation) and select key experiences for replay (efficient replay) <cit.>. Therefore, it is essential to reassess the existing memory constraint and prioritize optimizing computational efficiency for continual learning of LLMs by restricting the number of updates and the number of FLOPs <cit.>.

On the other end of the spectrum, studies with tightened memory constraints remain vital in modern continual learning of LLMs. As shown in fig:overview, upstream suppliers of LLMs typically do not provide training data with the released model weights. Consequently, consumers must adapt these models to downstream data without access to the actual replay data. Various rehearsal-free continual strategies are applied in this scenario, such as collecting data examples from alternate sources <cit.>, leveraging the generative capabilities of LLMs to produce pseudo-examples for replay <cit.>, and implementing regularization techniques in the parameter space <cit.>. Continual learning under the strict memory constraint is also driven by data privacy concerns, where preserving data on the server side is prohibited. In these scenarios, researchers must rely on online continual learning methods <cit.>, where data examples are only utilized for training as they arrive in a stream, and numerous efforts are already underway to develop LLMs capable of operating under these constraints <cit.>.





 §.§ Prospective Directions


Theories of Continual LLMs.  It is widely recognized that the continual learning community tends to prioritize empirical research over theoretical exploration. Nevertheless, there are efforts to establish theoretical foundations for CL. In <cit.>, the authors utilize second-order Taylor expansions around optimal parameters to derive an inter-task generalization error bound based on the maximum eigenvalue and l_2-norm of parameter differences. Another line of approaches leverages task/domain discrepancies to construct a multi-task generalization bound. For instance, Unified Domain Incremental Learning (UDIL) in <cit.> proposes upper bounds for intra-domain and cross-domain distillation losses, unifying various replay-based DIL techniques under a single adaptive generalization bound. However, applying these existing theories directly to continual LLMs can be imprudent, given their pre-trained, large-scale nature. Consequently, there is a notable gap in research focusing on continually learning LLMs with robust theoretical guarantees and understanding the forgetting behaviors of LLMs from a theoretical perspective.



Efficient Replay for Knowledge Retention for Continual LLMs.  Computational resources for training large-scale LLMs are often limited. While the storage budget can theoretically be infinite (sec:discussion-mem), replaying past experiences without specific design can lead to inefficient updates in current domain learning, resulting in slow convergence. Beyond sparse replay solutions that control data mixture ratios <cit.>, there is ongoing exploration of efficient replay for continual LLMs. 
For example, KPIG <cit.> enhances replay efficiency by calculating Key-part Information Gain (KPIG) on masked segments, enabling the dynamic selection of replay data. 
In <cit.>, a pioneering effort introduces a forgetting forecasting mechanism based on output changes during adaptation, later used for selective replay in continual model refinement (CMR). It has been verified in this work that filtering replay samples based on their tendency to forget significantly improves knowledge retention rates for continual LLMs. However, more sophisticated and accurate data mixing strategies and efficient replay sample selection mechanisms are much needed, e.g., a dynamic data mixing ratio throughout the training process. Hence we mark this practical direction of efficient replay for LLMs a significant research focus in the future.



Continual LLMs with Controllable Memory.  The long-term memory inherent in the whole set of parameters of LLMs often lacks interpretability and explicit manipulability, which is crucial in certain application areas. For instance, consider a scenario where a supplier collects data from customers under their consent and continually utilizes this data to update LLMs. However, if some users later revoke their consent, the knowledge acquired by the trained model from that portion of data must also be revoked. With a continually pre-trained large-scale LLM, the only solution is to roll back to a previous model version predating the inclusion of these users' data and retrain the model from that point onward.
This example of “machine unlearning” <cit.> vividly illustrates the benefits of equipping LLMs with an external, controllable memory.
As part of continual model refinement (CMR), memory systems for continual learning have been explored in several studies. Larimar <cit.> suggests integrating the Kanerva Machine <cit.> as an episodic memory for multi-fact model editing. This memory system supports basic operations like writing, reading, and generating, as well as advanced operations such as sequential writing and forgetting. It enables one-shot knowledge updates without costly retraining or fine-tuning. Additionally, other memory systems like Hopfield Networks <cit.> hold promise for future investigation.

Continual LLMs with Custom Preferences.  Customizing user preferences is critical for LLMs, especially in service-oriented contexts. Users often require different trade-offs between domain expertise, ethics, values, or tones of expression. Efficiently building customized LLMs for individual users and offering flexible adjustment options is a challenging task. Early attempts in this direction include Imprecise Bayesian Continual Learning (IBCL), which, under certain assumptions, guarantees the generation of Pareto-optimal models based on user preferences by combining two model posteriors in the parameter space <cit.>. While empirical validation is limited in scale, this approach paves the way for future research in this area.



§ CONCLUSION


In this work, we offer a comprehensive survey on continual LLMs, summarizing recent advancements in their training and deployment from a continual learning standpoint. We categorize the problems and tasks based on their positions within our proposed broader framework of modern stratified continual learning of LLMs. While there is a widespread and growing interest in this area across the community, we also note several missing cornerstones, including algorithmic diversity and a fundamental understanding of large models' behaviors such as knowledge forgetting, transfer, and acquisition. With a holistic yet detailed approach, we aim for this survey to inspire more practitioners to explore continual learning techniques, ultimately contributing to the development of robust and self-evolving AI systems.












































































abbrv



