<!DOCTYPE html><html lang="en">
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<title>[2310.06762] TRACE: A Comprehensive Benchmark for Continual Learning in Large Language Models</title><meta property="og:description" content="Aligned large language models (LLMs) demonstrate exceptional capabilities in task-solving, following instructions, and ensuring safety. However, the continual learning aspect of these aligned LLMs has been largely over…">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="twitter:card" content="summary">
<meta name="twitter:title" content="TRACE: A Comprehensive Benchmark for Continual Learning in Large Language Models">
<meta name="twitter:image:src" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta name="twitter:image:alt" content="ar5iv logo">
<meta property="og:title" content="TRACE: A Comprehensive Benchmark for Continual Learning in Large Language Models">
<meta property="og:site_name" content="ar5iv">
<meta property="og:image" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta property="og:type" content="article">
<meta property="og:url" content="https://ar5iv.labs.arxiv.org/html/2310.06762">

<!--Generated on Tue Feb 27 22:57:24 2024 by LaTeXML (version 0.8.8) http://dlmf.nist.gov/LaTeXML/.-->
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">

<script>
  function detectColorScheme(){
    var theme="light";
    var current_theme = localStorage.getItem("ar5iv_theme");
    if(current_theme){
      if(current_theme == "dark"){
        theme = "dark";
      } }
    else if(!window.matchMedia) { return false; }
    else if(window.matchMedia("(prefers-color-scheme: dark)").matches) {
      theme = "dark"; }
    if (theme=="dark") {
      document.documentElement.setAttribute("data-theme", "dark");
    } else {
      document.documentElement.setAttribute("data-theme", "light"); } }

  detectColorScheme();

  function toggleColorScheme(){
    var current_theme = localStorage.getItem("ar5iv_theme");
    if (current_theme) {
      if (current_theme == "light") {
        localStorage.setItem("ar5iv_theme", "dark"); }
      else {
        localStorage.setItem("ar5iv_theme", "light"); } }
    else {
        localStorage.setItem("ar5iv_theme", "dark"); }
    detectColorScheme(); }
</script>
<link media="all" rel="stylesheet" href="/assets/ar5iv-fonts.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv-site.0.2.2.css">
</head>
<body>
<div class="ltx_page_main">
<div class="ltx_page_content">
<article class="ltx_document ltx_authors_1line">
<h1 class="ltx_title ltx_title_document">TRACE: A Comprehensive Benchmark for Continual Learning in Large Language Models</h1>
<div class="ltx_authors">
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">
<span id="id1.1.1" class="ltx_text ltx_font_bold">Xiao Wang<sup id="id1.1.1.1" class="ltx_sup"><span id="id1.1.1.1.1" class="ltx_text ltx_font_medium ltx_font_italic">★</span></sup></span> <span id="id2.2.2" class="ltx_text ltx_font_bold">Yuansen Zhang<sup id="id2.2.2.1" class="ltx_sup"><span id="id2.2.2.1.1" class="ltx_text ltx_font_medium ltx_font_italic">★∗</span></sup></span> <span id="id3.3.3" class="ltx_text ltx_font_bold">Tianze Chen<sup id="id3.3.3.1" class="ltx_sup"><span id="id3.3.3.1.1" class="ltx_text ltx_font_medium ltx_font_italic">★∗</span></sup></span> 
<br class="ltx_break"><span id="id4.4.4" class="ltx_text ltx_font_bold">Songyang Gao<sup id="id4.4.4.1" class="ltx_sup"><span id="id4.4.4.1.1" class="ltx_text ltx_font_medium ltx_font_italic">★</span></sup></span> <span id="id5.5.5" class="ltx_text ltx_font_bold">Senjie Jin<sup id="id5.5.5.1" class="ltx_sup"><span id="id5.5.5.1.1" class="ltx_text ltx_font_medium ltx_font_italic">★</span></sup></span> <span id="id6.6.6" class="ltx_text ltx_font_bold">Xianjun Yang<sup id="id6.6.6.1" class="ltx_sup"><span id="id6.6.6.1.1" class="ltx_text ltx_font_medium ltx_font_italic">♣</span></sup></span> <span id="id7.7.7" class="ltx_text ltx_font_bold">Zhiheng Xi<sup id="id7.7.7.1" class="ltx_sup"><span id="id7.7.7.1.1" class="ltx_text ltx_font_medium ltx_font_italic">★</span></sup></span> 
<br class="ltx_break"><span id="id8.8.8" class="ltx_text ltx_font_bold">Rui Zheng<sup id="id8.8.8.1" class="ltx_sup"><span id="id8.8.8.1.1" class="ltx_text ltx_font_medium ltx_font_italic">★</span></sup></span> <span id="id9.9.9" class="ltx_text ltx_font_bold">Yicheng Zou<sup id="id9.9.9.1" class="ltx_sup"><span id="id9.9.9.1.1" class="ltx_text ltx_font_medium ltx_font_italic">♠</span></sup></span> <span id="id10.10.10" class="ltx_text ltx_font_bold">Tao Gui<sup id="id10.10.10.1" class="ltx_sup"><span id="id10.10.10.1.1" class="ltx_text ltx_font_medium ltx_font_italic">◆</span></sup></span> <span id="id11.11.11" class="ltx_text ltx_font_bold">Qi Zhang<sup id="id11.11.11.1" class="ltx_sup"><span id="id11.11.11.1.1" class="ltx_text ltx_font_medium ltx_font_italic">★†</span></sup></span> <span id="id12.12.12" class="ltx_text ltx_font_bold">Xuanjing Huang<sup id="id12.12.12.1" class="ltx_sup"><span id="id12.12.12.1.1" class="ltx_text ltx_font_medium ltx_font_italic">★</span></sup></span>

<br class="ltx_break"><sup id="id17.17.id1" class="ltx_sup">★</sup> School of Computer Science, Fudan University, Shanghai, China 
<br class="ltx_break"><sup id="id18.18.id2" class="ltx_sup">◆</sup> Institute of Modern Languages and Linguistics, Fudan University, Shanghai, China 
<br class="ltx_break"><sup id="id19.19.id3" class="ltx_sup">♣</sup> University of California, Santa Barbara 
<br class="ltx_break"><sup id="id20.20.id4" class="ltx_sup">♠</sup> Shanghai AI Laboratory
<br class="ltx_break"><span id="id21.21.id5" class="ltx_text ltx_font_typewriter">{xiao_wang20,qz,tgui}@fudan.edu.cn</span>
</span><span class="ltx_author_notes">Equal contributionCorresponding Author</span></span>
</div>

<div class="ltx_abstract">
<h6 class="ltx_title ltx_title_abstract">Abstract</h6>
<p id="id22.id1" class="ltx_p">Aligned large language models (LLMs) demonstrate exceptional capabilities in task-solving, following instructions, and ensuring safety. However, the continual learning aspect of these aligned LLMs has been largely overlooked.
Existing continual learning benchmarks lack sufficient challenge for leading aligned LLMs, owing to both their simplicity and the models’ potential exposure during instruction tuning.
In this paper, we introduce TRACE, a novel benchmark designed to evaluate continual learning in LLMs. TRACE consists of 8 distinct datasets spanning challenging tasks including domain-specific tasks, multilingual capabilities, code generation, and mathematical reasoning.
All datasets are standardized into a unified format, allowing for effortless automatic evaluation of LLMs.
Our experiments show that after training on TRACE, aligned LLMs exhibit significant declines in both general ability and instruction-following capabilities.
For example, the accuracy of llama2-chat 13B on gsm8k dataset declined precipitously from 28.8% to 2% after training on our datasets.
This highlights the challenge of finding a suitable tradeoff between achieving performance on specific tasks while preserving the original prowess of LLMs.
Empirical findings suggest that tasks inherently equipped with reasoning paths contribute significantly to preserving certain capabilities of LLMs against potential declines.
Motivated by this, we introduce the Reasoning-augmented Continual Learning (RCL) approach. RCL integrates task-specific cues with meta-rationales, effectively reducing catastrophic forgetting in LLMs while expediting convergence on novel tasks.
</p>
</div>
<section id="S1" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">1 </span>Introduction</h2>

<div id="S1.p1" class="ltx_para ltx_noindent">
<p id="S1.p1.1" class="ltx_p">Large Language Models (LLMs) <cite class="ltx_cite ltx_citemacro_citep">(OpenAI, <a href="#bib.bib33" title="" class="ltx_ref">2023</a>; Touvron et al., <a href="#bib.bib44" title="" class="ltx_ref">2023</a>)</cite> have revolutionized natural language processing through a two-step process: initial pretraining on extensive corpora, followed by fine-tuning on human-generated instructions and preference data, aligning them with human language and intentions.
Aligned LLMs have showcased impressive capabilities and ensured safer responses. However, as the demands for language models grow, there’s a pressing need to enhance their abilities in areas such as domain-specific knowledge <cite class="ltx_cite ltx_citemacro_citep">(Wu et al., <a href="#bib.bib51" title="" class="ltx_ref">2023</a>; Li et al., <a href="#bib.bib25" title="" class="ltx_ref">2023</a>)</cite>, multilingual proficiency <cite class="ltx_cite ltx_citemacro_citep">(Huang et al., <a href="#bib.bib21" title="" class="ltx_ref">2023</a>)</cite>, complex task-solving <cite class="ltx_cite ltx_citemacro_citep">(Surameery &amp; Shakor, <a href="#bib.bib42" title="" class="ltx_ref">2023</a>)</cite>, and tool usage <cite class="ltx_cite ltx_citemacro_citep">(Qin et al., <a href="#bib.bib34" title="" class="ltx_ref">2023</a>)</cite>.
Yet, retraining and realigning them from scratch to meet these demands is impractical due to prohibitive training costs and the challenge of acquiring high-quality data.
Therefore, incrementally training existing Aligned LLMs through continual learning (CL <cite class="ltx_cite ltx_citemacro_citep">(Wang et al., <a href="#bib.bib47" title="" class="ltx_ref">2023a</a>)</cite>) is crucial.
This prompts the pressing question: <span id="S1.p1.1.1" class="ltx_text ltx_font_italic">To what degree do Aligned LLMs exhibit catastrophic forgetting when subjected to incremental training?</span></p>
</div>
<div id="S1.p2" class="ltx_para ltx_noindent">
<p id="S1.p2.1" class="ltx_p">Existing continual learning benchmarks <cite class="ltx_cite ltx_citemacro_citep">(Zhang et al., <a href="#bib.bib53" title="" class="ltx_ref">2015</a>; Scialom et al., <a href="#bib.bib39" title="" class="ltx_ref">2022</a>; Razdaibiedina et al., <a href="#bib.bib35" title="" class="ltx_ref">2023</a>)</cite> are not suitable for evaluating the state-of-the-art LLMs. Firstly, many of these benchmarks predominantly consist of simplistic natural language understanding datasets. These tasks, due to their inherent simplicity, fail to challenge the capabilities of large-scale models adequately.
Furthermore, a significant drawback lies in the fact that some of these datasets have previously appeared in the instruction tuning <cite class="ltx_cite ltx_citemacro_citep">(Chung et al., <a href="#bib.bib8" title="" class="ltx_ref">2022</a>)</cite> sets of these LLMs, suggesting that the models may have already learned these tasks during training.
Secondly, prior benchmarks have primarily focused on metrics that assess the performance of the models on target sequential tasks.
Yet, for aligned models, aspects like generalization to new tasks, the ability to follow human instructions, and safety preservation are of paramount importance. Regrettably, these dimensions have not been extensively studied or incorporated into assessments.</p>
</div>
<div id="S1.p3" class="ltx_para ltx_noindent">
<p id="S1.p3.1" class="ltx_p">To facilitate further research, we present <span id="S1.p3.1.1" class="ltx_text ltx_font_bold">TRACE</span>, a continual learning benchmark designed for aligned Large Language Models.
Our benchmark consists of 8 distinct datasets spanning challenging tasks including domain-specific tasks, multilingual capabilities, code generation, and mathematical reasoning.
To ensure task balance, we have sampled 5,000 instances for each task, and for classification tasks, we have ensured an equal distribution of samples across all classes.
Additionally, all datasets have been standardized into a unified format, simplifying the evaluation process.
To evaluate continual learning in aligned LLMs, we introduce three metrics: ”General Ability Delta,” ”Instruction Following Delta,” and ”Safety Delta” to assess models’ forgetfulness in such scenarios.</p>
</div>
<div id="S1.p4" class="ltx_para ltx_noindent">
<p id="S1.p4.1" class="ltx_p">We conduct a comprehensive evaluation of 5 aligned LLMs on TRACE.
Evaluation results reveal several key findings: 1) Nearly all models exhibit a significant decline in general abilities, especially in math and reasoning. For instance, when trained on TRACE, the accuracy of llama2-chat 13B on the gsm8k dataset dropped from 28.8% to a mere 2%.
2) Unlike other skills, LLMs’ multilingual abilities generally improve. For example, llama2-chat 7B’s performance on the TydiQA dataset surged from an F1 score of 23.47 to 33.23.
3) Full-parameter training, compared to LoRA training, more easily fits the target tasks, but it also leads to a more pronounced decline in general abilities.
4) LLMs’ instruction-following capabilities also suffer a significant reduction after continual learning.</p>
</div>
<figure id="S1.F1" class="ltx_figure"><img src="/html/2310.06762/assets/x1.png" id="S1.F1.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="422" height="156" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 1: </span>An overview of TRACE benchmark. TRACE consists of two main components: 1) A selection of eight datasets constituting a tailored set of tasks for continual learning, covering challenges in domain-specific tasks, multilingual capabilities, code generation, and mathematical reasoning. 2) A post-training evaluation of LLM capabilities. In addition to traditional continual learning metrics, we introduce General Ability Delta, Instruction Following Delta, and Safety Delta to evaluate shifts in LLM’s inherent abilities.</figcaption>
</figure>
<div id="S1.p5" class="ltx_para ltx_noindent">
<p id="S1.p5.1" class="ltx_p">Through experimentation, we observed that tasks augmented with reasoning paths are notably effective in preserving certain capabilities of LLMs, preventing them from substantial declines. Such findings lead us to ponder on leveraging a model’s inherent strengths for rapid transfer on new tasks, rather than starting the learning curve from scratch. This motivation birthed our novel training strategy: Reasoning-augmented Continual Learning (RCL). RCL prompts the model to generate task analyses and rationales during training. As our results indicate, this approach not only boosts performance on target tasks but also significantly upholds the inherent strengths of LLMs. <span id="footnote1" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">1</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">1</sup><span class="ltx_tag ltx_tag_note">1</span>The dataset, code can be found at <a target="_blank" href="https://github.com/BeyonderXX/TRACE" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/BeyonderXX/TRACE</a></span></span></span>.</p>
</div>
</section>
<section id="S2" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">2 </span>Related Work</h2>

<section id="S2.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.1 </span>Continual Learning</h3>

<div id="S2.SS1.p1" class="ltx_para ltx_noindent">
<p id="S2.SS1.p1.1" class="ltx_p">Continual learning <cite class="ltx_cite ltx_citemacro_citep">(Wang et al., <a href="#bib.bib47" title="" class="ltx_ref">2023a</a>)</cite> aims to develop learning algorithms that can accumulate knowledge on non-stationary data. Existing works can be broadly categorized into rehearsal-based, regularization-based, and architecture-based approaches. Rehearsal-based approaches <cite class="ltx_cite ltx_citemacro_citep">(Lopez-Paz &amp; Ranzato, <a href="#bib.bib27" title="" class="ltx_ref">2017</a>; de Masson D’Autume et al., <a href="#bib.bib13" title="" class="ltx_ref">2019</a>)</cite> leverage a memory buffer that stores examples from previous tasks, training the model jointly with the current task.
Experience replay (ER) <cite class="ltx_cite ltx_citemacro_citep">(Rolnick et al., <a href="#bib.bib37" title="" class="ltx_ref">2019</a>)</cite> is a common strategy employed in rehearsal-based approaches and serves as a strong baseline.
Regularization-based approaches <cite class="ltx_cite ltx_citemacro_citep">(Kirkpatrick et al., <a href="#bib.bib23" title="" class="ltx_ref">2017</a>; Smith et al., <a href="#bib.bib41" title="" class="ltx_ref">2023</a>)</cite> incorporate additional terms into the loss function to penalize changes in crucial weights.
For example, Orthogonal Gradient Descent (OGD) <cite class="ltx_cite ltx_citemacro_citep">(Farajtabar et al., <a href="#bib.bib15" title="" class="ltx_ref">2020</a>)</cite> constrains the parameters to move within the orthogonal space defined by the gradients of previous tasks, Elastic Weight Consolidation (EWC) <cite class="ltx_cite ltx_citemacro_citep">(Kirkpatrick et al., <a href="#bib.bib23" title="" class="ltx_ref">2017</a>)</cite> constraint important parameters to stay close to their previous ones through, Gradient Episodic Memory (GEM) <cite class="ltx_cite ltx_citemacro_cite">Lopez-Paz &amp; Ranzato (<a href="#bib.bib27" title="" class="ltx_ref">2017</a>)</cite> leverages episode memories to avoid forgetting.
Architecture-based approaches <cite class="ltx_cite ltx_citemacro_citep">(Wang et al., <a href="#bib.bib49" title="" class="ltx_ref">2023b</a>; Razdaibiedina et al., <a href="#bib.bib35" title="" class="ltx_ref">2023</a>)</cite> focus on dynamically expanding model capacity or isolating existing model weights to mitigate interference between new and old tasks.
Progressive Prompts <cite class="ltx_cite ltx_citemacro_citep">(Razdaibiedina et al., <a href="#bib.bib35" title="" class="ltx_ref">2023</a>)</cite> learns separate prompts for each incoming task and sequentially concatenates them with previously learned prompts.</p>
</div>
</section>
<section id="S2.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.2 </span>CL Benchmarks in NLP</h3>

<div id="S2.SS2.p1" class="ltx_para ltx_noindent">
<p id="S2.SS2.p1.1" class="ltx_p">The most recognized CL benchmark for NLP encompasses five text classification datasets introduced by <cite class="ltx_cite ltx_citemacro_citet">Zhang et al. (<a href="#bib.bib53" title="" class="ltx_ref">2015</a>)</cite>, including AG News, Amazon Reviews, Yelp Reviews, DBpedia, and Yahoo Answers. Building upon this, <cite class="ltx_cite ltx_citemacro_citet">Razdaibiedina et al. (<a href="#bib.bib35" title="" class="ltx_ref">2023</a>)</cite> proposed a long CL benchmark which fuses the aforementioned five datasets with an additional four from the GLUE benchmark <cite class="ltx_cite ltx_citemacro_citep">(Wang et al., <a href="#bib.bib45" title="" class="ltx_ref">2018</a>)</cite>, five from the SuperGLUE benchmark <cite class="ltx_cite ltx_citemacro_citep">(Wang et al., <a href="#bib.bib46" title="" class="ltx_ref">2019</a>)</cite>, and the IMDB dataset <cite class="ltx_cite ltx_citemacro_citep">(Maas et al., <a href="#bib.bib31" title="" class="ltx_ref">2011</a>)</cite>.
While these datasets have been incorporated into the Flan collection <cite class="ltx_cite ltx_citemacro_citep">(Chung et al., <a href="#bib.bib8" title="" class="ltx_ref">2022</a>)</cite> and are widely used for current SoTA LLMs, their ubiquity has rendered them less suitable as CL benchmarks for LLMs.
Taking a different approach, <cite class="ltx_cite ltx_citemacro_citep">(Scialom et al., <a href="#bib.bib39" title="" class="ltx_ref">2022</a>)</cite> focuses on English language generation tasks. In a subsequent study, <cite class="ltx_cite ltx_citemacro_cite">Luo et al. (<a href="#bib.bib30" title="" class="ltx_ref">2023</a>)</cite> conducted an analysis of catastrophic forgetting on Bloomz <cite class="ltx_cite ltx_citemacro_citep">(Scao et al., <a href="#bib.bib38" title="" class="ltx_ref">2022</a>)</cite> using <cite class="ltx_cite ltx_citemacro_cite">Scialom et al. (<a href="#bib.bib39" title="" class="ltx_ref">2022</a>)</cite>’s datasets. However, this benchmark is limited in scope as it solely emphasizes Natural Language Generation (NLG) tasks and is restricted to English, thus lacking task diversity.
In contrast, TRACE offers a diverse and challenging array of sequential tasks. Additionally, it evaluates aligned models on aspects such as general capability, adherence to instructions, and shifts in safety measures.</p>
</div>
</section>
<section id="S2.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.3 </span>Chain-of-Thought</h3>

<div id="S2.SS3.p1" class="ltx_para ltx_noindent">
<p id="S2.SS3.p1.1" class="ltx_p">LLMs have shown advanced step-by-step reasoning capabilities, known as Chain-of-Thought (CoT) reasoning <cite class="ltx_cite ltx_citemacro_citep">(Wei et al., <a href="#bib.bib50" title="" class="ltx_ref">2022</a>)</cite>.
Zero-shot-CoT <cite class="ltx_cite ltx_citemacro_citep">(Kojima et al., <a href="#bib.bib24" title="" class="ltx_ref">2022</a>)</cite> illustrates the profound impact of simply prefacing a reasoning sequence with the sentence, ”Let’s think step by step.” Least-to-most <cite class="ltx_cite ltx_citemacro_citep">(Zhou et al., <a href="#bib.bib57" title="" class="ltx_ref">2022</a>)</cite>, actively prompts LLMs to segment complex problems into smaller tasks. ScienceQA <cite class="ltx_cite ltx_citemacro_citep">(Lu et al., <a href="#bib.bib28" title="" class="ltx_ref">2022</a>)</cite> highlights the efficacy of CoT in LLMs, particularly beneficial for filling the void in datasets within the scientific realm.
Fine-tune-CoT <cite class="ltx_cite ltx_citemacro_citep">(Ho et al., <a href="#bib.bib18" title="" class="ltx_ref">2022</a>)</cite> exploits the prowess of extensive LLMs to generate reasoning samples, further fine-tuning smaller models.
Despite these advancements, the application of CoT in continual learning remains unexplored. Our benchmark, TRACE, showcases that generating explanations can not only accelerate the learning process but also significantly mitigate the forgetting in their foundational capabilities.</p>
</div>
</section>
</section>
<section id="S3" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">3 </span>Preliminaries</h2>

<div id="S3.p1" class="ltx_para ltx_noindent">
<p id="S3.p1.7" class="ltx_p">Continual learning <cite class="ltx_cite ltx_citemacro_citep">(Ke &amp; Liu, <a href="#bib.bib22" title="" class="ltx_ref">2022</a>; Wang et al., <a href="#bib.bib47" title="" class="ltx_ref">2023a</a>)</cite> focuses on developing learning algorithms to accumulate knowledge on non-stationary data. In supervised continual learning, a sequence of tasks <math id="S3.p1.1.m1.3" class="ltx_Math" alttext="\left\{\mathcal{D}_{1},\ldots,\mathcal{D}_{T}\right\}" display="inline"><semantics id="S3.p1.1.m1.3a"><mrow id="S3.p1.1.m1.3.3.2" xref="S3.p1.1.m1.3.3.3.cmml"><mo id="S3.p1.1.m1.3.3.2.3" xref="S3.p1.1.m1.3.3.3.cmml">{</mo><msub id="S3.p1.1.m1.2.2.1.1" xref="S3.p1.1.m1.2.2.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.p1.1.m1.2.2.1.1.2" xref="S3.p1.1.m1.2.2.1.1.2.cmml">𝒟</mi><mn id="S3.p1.1.m1.2.2.1.1.3" xref="S3.p1.1.m1.2.2.1.1.3.cmml">1</mn></msub><mo id="S3.p1.1.m1.3.3.2.4" xref="S3.p1.1.m1.3.3.3.cmml">,</mo><mi mathvariant="normal" id="S3.p1.1.m1.1.1" xref="S3.p1.1.m1.1.1.cmml">…</mi><mo id="S3.p1.1.m1.3.3.2.5" xref="S3.p1.1.m1.3.3.3.cmml">,</mo><msub id="S3.p1.1.m1.3.3.2.2" xref="S3.p1.1.m1.3.3.2.2.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.p1.1.m1.3.3.2.2.2" xref="S3.p1.1.m1.3.3.2.2.2.cmml">𝒟</mi><mi id="S3.p1.1.m1.3.3.2.2.3" xref="S3.p1.1.m1.3.3.2.2.3.cmml">T</mi></msub><mo id="S3.p1.1.m1.3.3.2.6" xref="S3.p1.1.m1.3.3.3.cmml">}</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.p1.1.m1.3b"><set id="S3.p1.1.m1.3.3.3.cmml" xref="S3.p1.1.m1.3.3.2"><apply id="S3.p1.1.m1.2.2.1.1.cmml" xref="S3.p1.1.m1.2.2.1.1"><csymbol cd="ambiguous" id="S3.p1.1.m1.2.2.1.1.1.cmml" xref="S3.p1.1.m1.2.2.1.1">subscript</csymbol><ci id="S3.p1.1.m1.2.2.1.1.2.cmml" xref="S3.p1.1.m1.2.2.1.1.2">𝒟</ci><cn type="integer" id="S3.p1.1.m1.2.2.1.1.3.cmml" xref="S3.p1.1.m1.2.2.1.1.3">1</cn></apply><ci id="S3.p1.1.m1.1.1.cmml" xref="S3.p1.1.m1.1.1">…</ci><apply id="S3.p1.1.m1.3.3.2.2.cmml" xref="S3.p1.1.m1.3.3.2.2"><csymbol cd="ambiguous" id="S3.p1.1.m1.3.3.2.2.1.cmml" xref="S3.p1.1.m1.3.3.2.2">subscript</csymbol><ci id="S3.p1.1.m1.3.3.2.2.2.cmml" xref="S3.p1.1.m1.3.3.2.2.2">𝒟</ci><ci id="S3.p1.1.m1.3.3.2.2.3.cmml" xref="S3.p1.1.m1.3.3.2.2.3">𝑇</ci></apply></set></annotation-xml><annotation encoding="application/x-tex" id="S3.p1.1.m1.3c">\left\{\mathcal{D}_{1},\ldots,\mathcal{D}_{T}\right\}</annotation></semantics></math> arrive in a streaming fashion. Each task <math id="S3.p1.2.m2.1" class="ltx_Math" alttext="\mathcal{D}_{t}=\left\{\left(\bm{x}_{i}^{t},y_{i}^{t}\right)\right\}_{i=1}^{n_{t}}" display="inline"><semantics id="S3.p1.2.m2.1a"><mrow id="S3.p1.2.m2.1.1" xref="S3.p1.2.m2.1.1.cmml"><msub id="S3.p1.2.m2.1.1.3" xref="S3.p1.2.m2.1.1.3.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.p1.2.m2.1.1.3.2" xref="S3.p1.2.m2.1.1.3.2.cmml">𝒟</mi><mi id="S3.p1.2.m2.1.1.3.3" xref="S3.p1.2.m2.1.1.3.3.cmml">t</mi></msub><mo id="S3.p1.2.m2.1.1.2" xref="S3.p1.2.m2.1.1.2.cmml">=</mo><msubsup id="S3.p1.2.m2.1.1.1" xref="S3.p1.2.m2.1.1.1.cmml"><mrow id="S3.p1.2.m2.1.1.1.1.1.1" xref="S3.p1.2.m2.1.1.1.1.1.2.cmml"><mo id="S3.p1.2.m2.1.1.1.1.1.1.2" xref="S3.p1.2.m2.1.1.1.1.1.2.cmml">{</mo><mrow id="S3.p1.2.m2.1.1.1.1.1.1.1.2" xref="S3.p1.2.m2.1.1.1.1.1.1.1.3.cmml"><mo id="S3.p1.2.m2.1.1.1.1.1.1.1.2.3" xref="S3.p1.2.m2.1.1.1.1.1.1.1.3.cmml">(</mo><msubsup id="S3.p1.2.m2.1.1.1.1.1.1.1.1.1" xref="S3.p1.2.m2.1.1.1.1.1.1.1.1.1.cmml"><mi id="S3.p1.2.m2.1.1.1.1.1.1.1.1.1.2.2" xref="S3.p1.2.m2.1.1.1.1.1.1.1.1.1.2.2.cmml">𝒙</mi><mi id="S3.p1.2.m2.1.1.1.1.1.1.1.1.1.2.3" xref="S3.p1.2.m2.1.1.1.1.1.1.1.1.1.2.3.cmml">i</mi><mi id="S3.p1.2.m2.1.1.1.1.1.1.1.1.1.3" xref="S3.p1.2.m2.1.1.1.1.1.1.1.1.1.3.cmml">t</mi></msubsup><mo id="S3.p1.2.m2.1.1.1.1.1.1.1.2.4" xref="S3.p1.2.m2.1.1.1.1.1.1.1.3.cmml">,</mo><msubsup id="S3.p1.2.m2.1.1.1.1.1.1.1.2.2" xref="S3.p1.2.m2.1.1.1.1.1.1.1.2.2.cmml"><mi id="S3.p1.2.m2.1.1.1.1.1.1.1.2.2.2.2" xref="S3.p1.2.m2.1.1.1.1.1.1.1.2.2.2.2.cmml">y</mi><mi id="S3.p1.2.m2.1.1.1.1.1.1.1.2.2.2.3" xref="S3.p1.2.m2.1.1.1.1.1.1.1.2.2.2.3.cmml">i</mi><mi id="S3.p1.2.m2.1.1.1.1.1.1.1.2.2.3" xref="S3.p1.2.m2.1.1.1.1.1.1.1.2.2.3.cmml">t</mi></msubsup><mo id="S3.p1.2.m2.1.1.1.1.1.1.1.2.5" xref="S3.p1.2.m2.1.1.1.1.1.1.1.3.cmml">)</mo></mrow><mo id="S3.p1.2.m2.1.1.1.1.1.1.3" xref="S3.p1.2.m2.1.1.1.1.1.2.cmml">}</mo></mrow><mrow id="S3.p1.2.m2.1.1.1.1.3" xref="S3.p1.2.m2.1.1.1.1.3.cmml"><mi id="S3.p1.2.m2.1.1.1.1.3.2" xref="S3.p1.2.m2.1.1.1.1.3.2.cmml">i</mi><mo id="S3.p1.2.m2.1.1.1.1.3.1" xref="S3.p1.2.m2.1.1.1.1.3.1.cmml">=</mo><mn id="S3.p1.2.m2.1.1.1.1.3.3" xref="S3.p1.2.m2.1.1.1.1.3.3.cmml">1</mn></mrow><msub id="S3.p1.2.m2.1.1.1.3" xref="S3.p1.2.m2.1.1.1.3.cmml"><mi id="S3.p1.2.m2.1.1.1.3.2" xref="S3.p1.2.m2.1.1.1.3.2.cmml">n</mi><mi id="S3.p1.2.m2.1.1.1.3.3" xref="S3.p1.2.m2.1.1.1.3.3.cmml">t</mi></msub></msubsup></mrow><annotation-xml encoding="MathML-Content" id="S3.p1.2.m2.1b"><apply id="S3.p1.2.m2.1.1.cmml" xref="S3.p1.2.m2.1.1"><eq id="S3.p1.2.m2.1.1.2.cmml" xref="S3.p1.2.m2.1.1.2"></eq><apply id="S3.p1.2.m2.1.1.3.cmml" xref="S3.p1.2.m2.1.1.3"><csymbol cd="ambiguous" id="S3.p1.2.m2.1.1.3.1.cmml" xref="S3.p1.2.m2.1.1.3">subscript</csymbol><ci id="S3.p1.2.m2.1.1.3.2.cmml" xref="S3.p1.2.m2.1.1.3.2">𝒟</ci><ci id="S3.p1.2.m2.1.1.3.3.cmml" xref="S3.p1.2.m2.1.1.3.3">𝑡</ci></apply><apply id="S3.p1.2.m2.1.1.1.cmml" xref="S3.p1.2.m2.1.1.1"><csymbol cd="ambiguous" id="S3.p1.2.m2.1.1.1.2.cmml" xref="S3.p1.2.m2.1.1.1">superscript</csymbol><apply id="S3.p1.2.m2.1.1.1.1.cmml" xref="S3.p1.2.m2.1.1.1"><csymbol cd="ambiguous" id="S3.p1.2.m2.1.1.1.1.2.cmml" xref="S3.p1.2.m2.1.1.1">subscript</csymbol><set id="S3.p1.2.m2.1.1.1.1.1.2.cmml" xref="S3.p1.2.m2.1.1.1.1.1.1"><interval closure="open" id="S3.p1.2.m2.1.1.1.1.1.1.1.3.cmml" xref="S3.p1.2.m2.1.1.1.1.1.1.1.2"><apply id="S3.p1.2.m2.1.1.1.1.1.1.1.1.1.cmml" xref="S3.p1.2.m2.1.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.p1.2.m2.1.1.1.1.1.1.1.1.1.1.cmml" xref="S3.p1.2.m2.1.1.1.1.1.1.1.1.1">superscript</csymbol><apply id="S3.p1.2.m2.1.1.1.1.1.1.1.1.1.2.cmml" xref="S3.p1.2.m2.1.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.p1.2.m2.1.1.1.1.1.1.1.1.1.2.1.cmml" xref="S3.p1.2.m2.1.1.1.1.1.1.1.1.1">subscript</csymbol><ci id="S3.p1.2.m2.1.1.1.1.1.1.1.1.1.2.2.cmml" xref="S3.p1.2.m2.1.1.1.1.1.1.1.1.1.2.2">𝒙</ci><ci id="S3.p1.2.m2.1.1.1.1.1.1.1.1.1.2.3.cmml" xref="S3.p1.2.m2.1.1.1.1.1.1.1.1.1.2.3">𝑖</ci></apply><ci id="S3.p1.2.m2.1.1.1.1.1.1.1.1.1.3.cmml" xref="S3.p1.2.m2.1.1.1.1.1.1.1.1.1.3">𝑡</ci></apply><apply id="S3.p1.2.m2.1.1.1.1.1.1.1.2.2.cmml" xref="S3.p1.2.m2.1.1.1.1.1.1.1.2.2"><csymbol cd="ambiguous" id="S3.p1.2.m2.1.1.1.1.1.1.1.2.2.1.cmml" xref="S3.p1.2.m2.1.1.1.1.1.1.1.2.2">superscript</csymbol><apply id="S3.p1.2.m2.1.1.1.1.1.1.1.2.2.2.cmml" xref="S3.p1.2.m2.1.1.1.1.1.1.1.2.2"><csymbol cd="ambiguous" id="S3.p1.2.m2.1.1.1.1.1.1.1.2.2.2.1.cmml" xref="S3.p1.2.m2.1.1.1.1.1.1.1.2.2">subscript</csymbol><ci id="S3.p1.2.m2.1.1.1.1.1.1.1.2.2.2.2.cmml" xref="S3.p1.2.m2.1.1.1.1.1.1.1.2.2.2.2">𝑦</ci><ci id="S3.p1.2.m2.1.1.1.1.1.1.1.2.2.2.3.cmml" xref="S3.p1.2.m2.1.1.1.1.1.1.1.2.2.2.3">𝑖</ci></apply><ci id="S3.p1.2.m2.1.1.1.1.1.1.1.2.2.3.cmml" xref="S3.p1.2.m2.1.1.1.1.1.1.1.2.2.3">𝑡</ci></apply></interval></set><apply id="S3.p1.2.m2.1.1.1.1.3.cmml" xref="S3.p1.2.m2.1.1.1.1.3"><eq id="S3.p1.2.m2.1.1.1.1.3.1.cmml" xref="S3.p1.2.m2.1.1.1.1.3.1"></eq><ci id="S3.p1.2.m2.1.1.1.1.3.2.cmml" xref="S3.p1.2.m2.1.1.1.1.3.2">𝑖</ci><cn type="integer" id="S3.p1.2.m2.1.1.1.1.3.3.cmml" xref="S3.p1.2.m2.1.1.1.1.3.3">1</cn></apply></apply><apply id="S3.p1.2.m2.1.1.1.3.cmml" xref="S3.p1.2.m2.1.1.1.3"><csymbol cd="ambiguous" id="S3.p1.2.m2.1.1.1.3.1.cmml" xref="S3.p1.2.m2.1.1.1.3">subscript</csymbol><ci id="S3.p1.2.m2.1.1.1.3.2.cmml" xref="S3.p1.2.m2.1.1.1.3.2">𝑛</ci><ci id="S3.p1.2.m2.1.1.1.3.3.cmml" xref="S3.p1.2.m2.1.1.1.3.3">𝑡</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.p1.2.m2.1c">\mathcal{D}_{t}=\left\{\left(\bm{x}_{i}^{t},y_{i}^{t}\right)\right\}_{i=1}^{n_{t}}</annotation></semantics></math> contains a separate target dataset, where <math id="S3.p1.3.m3.1" class="ltx_Math" alttext="\bm{x}_{i}^{t}\in\mathcal{X}_{t}" display="inline"><semantics id="S3.p1.3.m3.1a"><mrow id="S3.p1.3.m3.1.1" xref="S3.p1.3.m3.1.1.cmml"><msubsup id="S3.p1.3.m3.1.1.2" xref="S3.p1.3.m3.1.1.2.cmml"><mi id="S3.p1.3.m3.1.1.2.2.2" xref="S3.p1.3.m3.1.1.2.2.2.cmml">𝒙</mi><mi id="S3.p1.3.m3.1.1.2.2.3" xref="S3.p1.3.m3.1.1.2.2.3.cmml">i</mi><mi id="S3.p1.3.m3.1.1.2.3" xref="S3.p1.3.m3.1.1.2.3.cmml">t</mi></msubsup><mo id="S3.p1.3.m3.1.1.1" xref="S3.p1.3.m3.1.1.1.cmml">∈</mo><msub id="S3.p1.3.m3.1.1.3" xref="S3.p1.3.m3.1.1.3.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.p1.3.m3.1.1.3.2" xref="S3.p1.3.m3.1.1.3.2.cmml">𝒳</mi><mi id="S3.p1.3.m3.1.1.3.3" xref="S3.p1.3.m3.1.1.3.3.cmml">t</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="S3.p1.3.m3.1b"><apply id="S3.p1.3.m3.1.1.cmml" xref="S3.p1.3.m3.1.1"><in id="S3.p1.3.m3.1.1.1.cmml" xref="S3.p1.3.m3.1.1.1"></in><apply id="S3.p1.3.m3.1.1.2.cmml" xref="S3.p1.3.m3.1.1.2"><csymbol cd="ambiguous" id="S3.p1.3.m3.1.1.2.1.cmml" xref="S3.p1.3.m3.1.1.2">superscript</csymbol><apply id="S3.p1.3.m3.1.1.2.2.cmml" xref="S3.p1.3.m3.1.1.2"><csymbol cd="ambiguous" id="S3.p1.3.m3.1.1.2.2.1.cmml" xref="S3.p1.3.m3.1.1.2">subscript</csymbol><ci id="S3.p1.3.m3.1.1.2.2.2.cmml" xref="S3.p1.3.m3.1.1.2.2.2">𝒙</ci><ci id="S3.p1.3.m3.1.1.2.2.3.cmml" xref="S3.p1.3.m3.1.1.2.2.3">𝑖</ci></apply><ci id="S3.p1.3.m3.1.1.2.3.cmml" xref="S3.p1.3.m3.1.1.2.3">𝑡</ci></apply><apply id="S3.p1.3.m3.1.1.3.cmml" xref="S3.p1.3.m3.1.1.3"><csymbol cd="ambiguous" id="S3.p1.3.m3.1.1.3.1.cmml" xref="S3.p1.3.m3.1.1.3">subscript</csymbol><ci id="S3.p1.3.m3.1.1.3.2.cmml" xref="S3.p1.3.m3.1.1.3.2">𝒳</ci><ci id="S3.p1.3.m3.1.1.3.3.cmml" xref="S3.p1.3.m3.1.1.3.3">𝑡</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.p1.3.m3.1c">\bm{x}_{i}^{t}\in\mathcal{X}_{t}</annotation></semantics></math> , <math id="S3.p1.4.m4.1" class="ltx_Math" alttext="\bm{y}_{i}^{t}\in\mathcal{Y}_{t}" display="inline"><semantics id="S3.p1.4.m4.1a"><mrow id="S3.p1.4.m4.1.1" xref="S3.p1.4.m4.1.1.cmml"><msubsup id="S3.p1.4.m4.1.1.2" xref="S3.p1.4.m4.1.1.2.cmml"><mi id="S3.p1.4.m4.1.1.2.2.2" xref="S3.p1.4.m4.1.1.2.2.2.cmml">𝒚</mi><mi id="S3.p1.4.m4.1.1.2.2.3" xref="S3.p1.4.m4.1.1.2.2.3.cmml">i</mi><mi id="S3.p1.4.m4.1.1.2.3" xref="S3.p1.4.m4.1.1.2.3.cmml">t</mi></msubsup><mo id="S3.p1.4.m4.1.1.1" xref="S3.p1.4.m4.1.1.1.cmml">∈</mo><msub id="S3.p1.4.m4.1.1.3" xref="S3.p1.4.m4.1.1.3.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.p1.4.m4.1.1.3.2" xref="S3.p1.4.m4.1.1.3.2.cmml">𝒴</mi><mi id="S3.p1.4.m4.1.1.3.3" xref="S3.p1.4.m4.1.1.3.3.cmml">t</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="S3.p1.4.m4.1b"><apply id="S3.p1.4.m4.1.1.cmml" xref="S3.p1.4.m4.1.1"><in id="S3.p1.4.m4.1.1.1.cmml" xref="S3.p1.4.m4.1.1.1"></in><apply id="S3.p1.4.m4.1.1.2.cmml" xref="S3.p1.4.m4.1.1.2"><csymbol cd="ambiguous" id="S3.p1.4.m4.1.1.2.1.cmml" xref="S3.p1.4.m4.1.1.2">superscript</csymbol><apply id="S3.p1.4.m4.1.1.2.2.cmml" xref="S3.p1.4.m4.1.1.2"><csymbol cd="ambiguous" id="S3.p1.4.m4.1.1.2.2.1.cmml" xref="S3.p1.4.m4.1.1.2">subscript</csymbol><ci id="S3.p1.4.m4.1.1.2.2.2.cmml" xref="S3.p1.4.m4.1.1.2.2.2">𝒚</ci><ci id="S3.p1.4.m4.1.1.2.2.3.cmml" xref="S3.p1.4.m4.1.1.2.2.3">𝑖</ci></apply><ci id="S3.p1.4.m4.1.1.2.3.cmml" xref="S3.p1.4.m4.1.1.2.3">𝑡</ci></apply><apply id="S3.p1.4.m4.1.1.3.cmml" xref="S3.p1.4.m4.1.1.3"><csymbol cd="ambiguous" id="S3.p1.4.m4.1.1.3.1.cmml" xref="S3.p1.4.m4.1.1.3">subscript</csymbol><ci id="S3.p1.4.m4.1.1.3.2.cmml" xref="S3.p1.4.m4.1.1.3.2">𝒴</ci><ci id="S3.p1.4.m4.1.1.3.3.cmml" xref="S3.p1.4.m4.1.1.3.3">𝑡</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.p1.4.m4.1c">\bm{y}_{i}^{t}\in\mathcal{Y}_{t}</annotation></semantics></math>.
A single model needs to adapt to them sequentially, with only access to <math id="S3.p1.5.m5.1" class="ltx_Math" alttext="\mathcal{D}_{t}" display="inline"><semantics id="S3.p1.5.m5.1a"><msub id="S3.p1.5.m5.1.1" xref="S3.p1.5.m5.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.p1.5.m5.1.1.2" xref="S3.p1.5.m5.1.1.2.cmml">𝒟</mi><mi id="S3.p1.5.m5.1.1.3" xref="S3.p1.5.m5.1.1.3.cmml">t</mi></msub><annotation-xml encoding="MathML-Content" id="S3.p1.5.m5.1b"><apply id="S3.p1.5.m5.1.1.cmml" xref="S3.p1.5.m5.1.1"><csymbol cd="ambiguous" id="S3.p1.5.m5.1.1.1.cmml" xref="S3.p1.5.m5.1.1">subscript</csymbol><ci id="S3.p1.5.m5.1.1.2.cmml" xref="S3.p1.5.m5.1.1.2">𝒟</ci><ci id="S3.p1.5.m5.1.1.3.cmml" xref="S3.p1.5.m5.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.p1.5.m5.1c">\mathcal{D}_{t}</annotation></semantics></math> at the t-th task.
In general, given a prediction model <math id="S3.p1.6.m6.1" class="ltx_Math" alttext="h_{\Theta}" display="inline"><semantics id="S3.p1.6.m6.1a"><msub id="S3.p1.6.m6.1.1" xref="S3.p1.6.m6.1.1.cmml"><mi id="S3.p1.6.m6.1.1.2" xref="S3.p1.6.m6.1.1.2.cmml">h</mi><mi mathvariant="normal" id="S3.p1.6.m6.1.1.3" xref="S3.p1.6.m6.1.1.3.cmml">Θ</mi></msub><annotation-xml encoding="MathML-Content" id="S3.p1.6.m6.1b"><apply id="S3.p1.6.m6.1.1.cmml" xref="S3.p1.6.m6.1.1"><csymbol cd="ambiguous" id="S3.p1.6.m6.1.1.1.cmml" xref="S3.p1.6.m6.1.1">subscript</csymbol><ci id="S3.p1.6.m6.1.1.2.cmml" xref="S3.p1.6.m6.1.1.2">ℎ</ci><ci id="S3.p1.6.m6.1.1.3.cmml" xref="S3.p1.6.m6.1.1.3">Θ</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.p1.6.m6.1c">h_{\Theta}</annotation></semantics></math> parameterized by <math id="S3.p1.7.m7.1" class="ltx_Math" alttext="\Theta" display="inline"><semantics id="S3.p1.7.m7.1a"><mi mathvariant="normal" id="S3.p1.7.m7.1.1" xref="S3.p1.7.m7.1.1.cmml">Θ</mi><annotation-xml encoding="MathML-Content" id="S3.p1.7.m7.1b"><ci id="S3.p1.7.m7.1.1.cmml" xref="S3.p1.7.m7.1.1">Θ</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.p1.7.m7.1c">\Theta</annotation></semantics></math>, continual learning seeks to optimize for the following objective across all tasks:</p>
<table id="S3.E1" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S3.E1.m1.3" class="ltx_Math" alttext="\max_{\Theta}\sum_{k=1}^{T}\sum_{x,y\in\mathcal{D}_{k}}\log p_{\Theta}(y\mid x)" display="block"><semantics id="S3.E1.m1.3a"><mrow id="S3.E1.m1.3.3" xref="S3.E1.m1.3.3.cmml"><munder id="S3.E1.m1.3.3.3" xref="S3.E1.m1.3.3.3.cmml"><mi id="S3.E1.m1.3.3.3.2" xref="S3.E1.m1.3.3.3.2.cmml">max</mi><mi mathvariant="normal" id="S3.E1.m1.3.3.3.3" xref="S3.E1.m1.3.3.3.3.cmml">Θ</mi></munder><mo lspace="0em" rspace="0em" id="S3.E1.m1.3.3.2" xref="S3.E1.m1.3.3.2.cmml">​</mo><mrow id="S3.E1.m1.3.3.1" xref="S3.E1.m1.3.3.1.cmml"><munderover id="S3.E1.m1.3.3.1.2" xref="S3.E1.m1.3.3.1.2.cmml"><mo movablelimits="false" rspace="0em" id="S3.E1.m1.3.3.1.2.2.2" xref="S3.E1.m1.3.3.1.2.2.2.cmml">∑</mo><mrow id="S3.E1.m1.3.3.1.2.2.3" xref="S3.E1.m1.3.3.1.2.2.3.cmml"><mi id="S3.E1.m1.3.3.1.2.2.3.2" xref="S3.E1.m1.3.3.1.2.2.3.2.cmml">k</mi><mo id="S3.E1.m1.3.3.1.2.2.3.1" xref="S3.E1.m1.3.3.1.2.2.3.1.cmml">=</mo><mn id="S3.E1.m1.3.3.1.2.2.3.3" xref="S3.E1.m1.3.3.1.2.2.3.3.cmml">1</mn></mrow><mi id="S3.E1.m1.3.3.1.2.3" xref="S3.E1.m1.3.3.1.2.3.cmml">T</mi></munderover><mrow id="S3.E1.m1.3.3.1.1" xref="S3.E1.m1.3.3.1.1.cmml"><munder id="S3.E1.m1.3.3.1.1.2" xref="S3.E1.m1.3.3.1.1.2.cmml"><mo movablelimits="false" id="S3.E1.m1.3.3.1.1.2.2" xref="S3.E1.m1.3.3.1.1.2.2.cmml">∑</mo><mrow id="S3.E1.m1.2.2.2" xref="S3.E1.m1.2.2.2.cmml"><mrow id="S3.E1.m1.2.2.2.4.2" xref="S3.E1.m1.2.2.2.4.1.cmml"><mi id="S3.E1.m1.1.1.1.1" xref="S3.E1.m1.1.1.1.1.cmml">x</mi><mo id="S3.E1.m1.2.2.2.4.2.1" xref="S3.E1.m1.2.2.2.4.1.cmml">,</mo><mi id="S3.E1.m1.2.2.2.2" xref="S3.E1.m1.2.2.2.2.cmml">y</mi></mrow><mo id="S3.E1.m1.2.2.2.3" xref="S3.E1.m1.2.2.2.3.cmml">∈</mo><msub id="S3.E1.m1.2.2.2.5" xref="S3.E1.m1.2.2.2.5.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.E1.m1.2.2.2.5.2" xref="S3.E1.m1.2.2.2.5.2.cmml">𝒟</mi><mi id="S3.E1.m1.2.2.2.5.3" xref="S3.E1.m1.2.2.2.5.3.cmml">k</mi></msub></mrow></munder><mrow id="S3.E1.m1.3.3.1.1.1" xref="S3.E1.m1.3.3.1.1.1.cmml"><mrow id="S3.E1.m1.3.3.1.1.1.3" xref="S3.E1.m1.3.3.1.1.1.3.cmml"><mi id="S3.E1.m1.3.3.1.1.1.3.1" xref="S3.E1.m1.3.3.1.1.1.3.1.cmml">log</mi><mo lspace="0.167em" id="S3.E1.m1.3.3.1.1.1.3a" xref="S3.E1.m1.3.3.1.1.1.3.cmml">⁡</mo><msub id="S3.E1.m1.3.3.1.1.1.3.2" xref="S3.E1.m1.3.3.1.1.1.3.2.cmml"><mi id="S3.E1.m1.3.3.1.1.1.3.2.2" xref="S3.E1.m1.3.3.1.1.1.3.2.2.cmml">p</mi><mi mathvariant="normal" id="S3.E1.m1.3.3.1.1.1.3.2.3" xref="S3.E1.m1.3.3.1.1.1.3.2.3.cmml">Θ</mi></msub></mrow><mo lspace="0em" rspace="0em" id="S3.E1.m1.3.3.1.1.1.2" xref="S3.E1.m1.3.3.1.1.1.2.cmml">​</mo><mrow id="S3.E1.m1.3.3.1.1.1.1.1" xref="S3.E1.m1.3.3.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.E1.m1.3.3.1.1.1.1.1.2" xref="S3.E1.m1.3.3.1.1.1.1.1.1.cmml">(</mo><mrow id="S3.E1.m1.3.3.1.1.1.1.1.1" xref="S3.E1.m1.3.3.1.1.1.1.1.1.cmml"><mi id="S3.E1.m1.3.3.1.1.1.1.1.1.2" xref="S3.E1.m1.3.3.1.1.1.1.1.1.2.cmml">y</mi><mo id="S3.E1.m1.3.3.1.1.1.1.1.1.1" xref="S3.E1.m1.3.3.1.1.1.1.1.1.1.cmml">∣</mo><mi id="S3.E1.m1.3.3.1.1.1.1.1.1.3" xref="S3.E1.m1.3.3.1.1.1.1.1.1.3.cmml">x</mi></mrow><mo stretchy="false" id="S3.E1.m1.3.3.1.1.1.1.1.3" xref="S3.E1.m1.3.3.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.E1.m1.3b"><apply id="S3.E1.m1.3.3.cmml" xref="S3.E1.m1.3.3"><times id="S3.E1.m1.3.3.2.cmml" xref="S3.E1.m1.3.3.2"></times><apply id="S3.E1.m1.3.3.3.cmml" xref="S3.E1.m1.3.3.3"><csymbol cd="ambiguous" id="S3.E1.m1.3.3.3.1.cmml" xref="S3.E1.m1.3.3.3">subscript</csymbol><max id="S3.E1.m1.3.3.3.2.cmml" xref="S3.E1.m1.3.3.3.2"></max><ci id="S3.E1.m1.3.3.3.3.cmml" xref="S3.E1.m1.3.3.3.3">Θ</ci></apply><apply id="S3.E1.m1.3.3.1.cmml" xref="S3.E1.m1.3.3.1"><apply id="S3.E1.m1.3.3.1.2.cmml" xref="S3.E1.m1.3.3.1.2"><csymbol cd="ambiguous" id="S3.E1.m1.3.3.1.2.1.cmml" xref="S3.E1.m1.3.3.1.2">superscript</csymbol><apply id="S3.E1.m1.3.3.1.2.2.cmml" xref="S3.E1.m1.3.3.1.2"><csymbol cd="ambiguous" id="S3.E1.m1.3.3.1.2.2.1.cmml" xref="S3.E1.m1.3.3.1.2">subscript</csymbol><sum id="S3.E1.m1.3.3.1.2.2.2.cmml" xref="S3.E1.m1.3.3.1.2.2.2"></sum><apply id="S3.E1.m1.3.3.1.2.2.3.cmml" xref="S3.E1.m1.3.3.1.2.2.3"><eq id="S3.E1.m1.3.3.1.2.2.3.1.cmml" xref="S3.E1.m1.3.3.1.2.2.3.1"></eq><ci id="S3.E1.m1.3.3.1.2.2.3.2.cmml" xref="S3.E1.m1.3.3.1.2.2.3.2">𝑘</ci><cn type="integer" id="S3.E1.m1.3.3.1.2.2.3.3.cmml" xref="S3.E1.m1.3.3.1.2.2.3.3">1</cn></apply></apply><ci id="S3.E1.m1.3.3.1.2.3.cmml" xref="S3.E1.m1.3.3.1.2.3">𝑇</ci></apply><apply id="S3.E1.m1.3.3.1.1.cmml" xref="S3.E1.m1.3.3.1.1"><apply id="S3.E1.m1.3.3.1.1.2.cmml" xref="S3.E1.m1.3.3.1.1.2"><csymbol cd="ambiguous" id="S3.E1.m1.3.3.1.1.2.1.cmml" xref="S3.E1.m1.3.3.1.1.2">subscript</csymbol><sum id="S3.E1.m1.3.3.1.1.2.2.cmml" xref="S3.E1.m1.3.3.1.1.2.2"></sum><apply id="S3.E1.m1.2.2.2.cmml" xref="S3.E1.m1.2.2.2"><in id="S3.E1.m1.2.2.2.3.cmml" xref="S3.E1.m1.2.2.2.3"></in><list id="S3.E1.m1.2.2.2.4.1.cmml" xref="S3.E1.m1.2.2.2.4.2"><ci id="S3.E1.m1.1.1.1.1.cmml" xref="S3.E1.m1.1.1.1.1">𝑥</ci><ci id="S3.E1.m1.2.2.2.2.cmml" xref="S3.E1.m1.2.2.2.2">𝑦</ci></list><apply id="S3.E1.m1.2.2.2.5.cmml" xref="S3.E1.m1.2.2.2.5"><csymbol cd="ambiguous" id="S3.E1.m1.2.2.2.5.1.cmml" xref="S3.E1.m1.2.2.2.5">subscript</csymbol><ci id="S3.E1.m1.2.2.2.5.2.cmml" xref="S3.E1.m1.2.2.2.5.2">𝒟</ci><ci id="S3.E1.m1.2.2.2.5.3.cmml" xref="S3.E1.m1.2.2.2.5.3">𝑘</ci></apply></apply></apply><apply id="S3.E1.m1.3.3.1.1.1.cmml" xref="S3.E1.m1.3.3.1.1.1"><times id="S3.E1.m1.3.3.1.1.1.2.cmml" xref="S3.E1.m1.3.3.1.1.1.2"></times><apply id="S3.E1.m1.3.3.1.1.1.3.cmml" xref="S3.E1.m1.3.3.1.1.1.3"><log id="S3.E1.m1.3.3.1.1.1.3.1.cmml" xref="S3.E1.m1.3.3.1.1.1.3.1"></log><apply id="S3.E1.m1.3.3.1.1.1.3.2.cmml" xref="S3.E1.m1.3.3.1.1.1.3.2"><csymbol cd="ambiguous" id="S3.E1.m1.3.3.1.1.1.3.2.1.cmml" xref="S3.E1.m1.3.3.1.1.1.3.2">subscript</csymbol><ci id="S3.E1.m1.3.3.1.1.1.3.2.2.cmml" xref="S3.E1.m1.3.3.1.1.1.3.2.2">𝑝</ci><ci id="S3.E1.m1.3.3.1.1.1.3.2.3.cmml" xref="S3.E1.m1.3.3.1.1.1.3.2.3">Θ</ci></apply></apply><apply id="S3.E1.m1.3.3.1.1.1.1.1.1.cmml" xref="S3.E1.m1.3.3.1.1.1.1.1"><csymbol cd="latexml" id="S3.E1.m1.3.3.1.1.1.1.1.1.1.cmml" xref="S3.E1.m1.3.3.1.1.1.1.1.1.1">conditional</csymbol><ci id="S3.E1.m1.3.3.1.1.1.1.1.1.2.cmml" xref="S3.E1.m1.3.3.1.1.1.1.1.1.2">𝑦</ci><ci id="S3.E1.m1.3.3.1.1.1.1.1.1.3.cmml" xref="S3.E1.m1.3.3.1.1.1.1.1.1.3">𝑥</ci></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E1.m1.3c">\max_{\Theta}\sum_{k=1}^{T}\sum_{x,y\in\mathcal{D}_{k}}\log p_{\Theta}(y\mid x)</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(1)</span></td>
</tr></tbody>
</table>
</div>
<div id="S3.p2" class="ltx_para ltx_noindent">
<p id="S3.p2.2" class="ltx_p">In this paper, we utilize overall performance (OP <cite class="ltx_cite ltx_citemacro_citep">(Chaudhry et al., <a href="#bib.bib6" title="" class="ltx_ref">2018</a>)</cite>) and backward transfer (BWT <cite class="ltx_cite ltx_citemacro_citep">(Lopez-Paz &amp; Ranzato, <a href="#bib.bib27" title="" class="ltx_ref">2017</a>)</cite>) scores as the main metrics. After incrementally learning the t-th task, the model’s score on the i-th task (where <math id="S3.p2.1.m1.1" class="ltx_Math" alttext="i\geq t" display="inline"><semantics id="S3.p2.1.m1.1a"><mrow id="S3.p2.1.m1.1.1" xref="S3.p2.1.m1.1.1.cmml"><mi id="S3.p2.1.m1.1.1.2" xref="S3.p2.1.m1.1.1.2.cmml">i</mi><mo id="S3.p2.1.m1.1.1.1" xref="S3.p2.1.m1.1.1.1.cmml">≥</mo><mi id="S3.p2.1.m1.1.1.3" xref="S3.p2.1.m1.1.1.3.cmml">t</mi></mrow><annotation-xml encoding="MathML-Content" id="S3.p2.1.m1.1b"><apply id="S3.p2.1.m1.1.1.cmml" xref="S3.p2.1.m1.1.1"><geq id="S3.p2.1.m1.1.1.1.cmml" xref="S3.p2.1.m1.1.1.1"></geq><ci id="S3.p2.1.m1.1.1.2.cmml" xref="S3.p2.1.m1.1.1.2">𝑖</ci><ci id="S3.p2.1.m1.1.1.3.cmml" xref="S3.p2.1.m1.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.p2.1.m1.1c">i\geq t</annotation></semantics></math>) is denoted as <math id="S3.p2.2.m2.2" class="ltx_Math" alttext="R_{t,i}^{D}" display="inline"><semantics id="S3.p2.2.m2.2a"><msubsup id="S3.p2.2.m2.2.3" xref="S3.p2.2.m2.2.3.cmml"><mi id="S3.p2.2.m2.2.3.2.2" xref="S3.p2.2.m2.2.3.2.2.cmml">R</mi><mrow id="S3.p2.2.m2.2.2.2.4" xref="S3.p2.2.m2.2.2.2.3.cmml"><mi id="S3.p2.2.m2.1.1.1.1" xref="S3.p2.2.m2.1.1.1.1.cmml">t</mi><mo id="S3.p2.2.m2.2.2.2.4.1" xref="S3.p2.2.m2.2.2.2.3.cmml">,</mo><mi id="S3.p2.2.m2.2.2.2.2" xref="S3.p2.2.m2.2.2.2.2.cmml">i</mi></mrow><mi id="S3.p2.2.m2.2.3.3" xref="S3.p2.2.m2.2.3.3.cmml">D</mi></msubsup><annotation-xml encoding="MathML-Content" id="S3.p2.2.m2.2b"><apply id="S3.p2.2.m2.2.3.cmml" xref="S3.p2.2.m2.2.3"><csymbol cd="ambiguous" id="S3.p2.2.m2.2.3.1.cmml" xref="S3.p2.2.m2.2.3">superscript</csymbol><apply id="S3.p2.2.m2.2.3.2.cmml" xref="S3.p2.2.m2.2.3"><csymbol cd="ambiguous" id="S3.p2.2.m2.2.3.2.1.cmml" xref="S3.p2.2.m2.2.3">subscript</csymbol><ci id="S3.p2.2.m2.2.3.2.2.cmml" xref="S3.p2.2.m2.2.3.2.2">𝑅</ci><list id="S3.p2.2.m2.2.2.2.3.cmml" xref="S3.p2.2.m2.2.2.2.4"><ci id="S3.p2.2.m2.1.1.1.1.cmml" xref="S3.p2.2.m2.1.1.1.1">𝑡</ci><ci id="S3.p2.2.m2.2.2.2.2.cmml" xref="S3.p2.2.m2.2.2.2.2">𝑖</ci></list></apply><ci id="S3.p2.2.m2.2.3.3.cmml" xref="S3.p2.2.m2.2.3.3">𝐷</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.p2.2.m2.2c">R_{t,i}^{D}</annotation></semantics></math>. The overall performance and backward transfer score are calculated using the following formulas:</p>
<table id="S3.E2" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S3.E2.m1.2" class="ltx_Math" alttext="OP_{t}=\frac{1}{t}\sum_{i=1}^{t}R_{t,i}^{D}" display="block"><semantics id="S3.E2.m1.2a"><mrow id="S3.E2.m1.2.3" xref="S3.E2.m1.2.3.cmml"><mrow id="S3.E2.m1.2.3.2" xref="S3.E2.m1.2.3.2.cmml"><mi id="S3.E2.m1.2.3.2.2" xref="S3.E2.m1.2.3.2.2.cmml">O</mi><mo lspace="0em" rspace="0em" id="S3.E2.m1.2.3.2.1" xref="S3.E2.m1.2.3.2.1.cmml">​</mo><msub id="S3.E2.m1.2.3.2.3" xref="S3.E2.m1.2.3.2.3.cmml"><mi id="S3.E2.m1.2.3.2.3.2" xref="S3.E2.m1.2.3.2.3.2.cmml">P</mi><mi id="S3.E2.m1.2.3.2.3.3" xref="S3.E2.m1.2.3.2.3.3.cmml">t</mi></msub></mrow><mo id="S3.E2.m1.2.3.1" xref="S3.E2.m1.2.3.1.cmml">=</mo><mrow id="S3.E2.m1.2.3.3" xref="S3.E2.m1.2.3.3.cmml"><mfrac id="S3.E2.m1.2.3.3.2" xref="S3.E2.m1.2.3.3.2.cmml"><mn id="S3.E2.m1.2.3.3.2.2" xref="S3.E2.m1.2.3.3.2.2.cmml">1</mn><mi id="S3.E2.m1.2.3.3.2.3" xref="S3.E2.m1.2.3.3.2.3.cmml">t</mi></mfrac><mo lspace="0em" rspace="0em" id="S3.E2.m1.2.3.3.1" xref="S3.E2.m1.2.3.3.1.cmml">​</mo><mrow id="S3.E2.m1.2.3.3.3" xref="S3.E2.m1.2.3.3.3.cmml"><munderover id="S3.E2.m1.2.3.3.3.1" xref="S3.E2.m1.2.3.3.3.1.cmml"><mo movablelimits="false" id="S3.E2.m1.2.3.3.3.1.2.2" xref="S3.E2.m1.2.3.3.3.1.2.2.cmml">∑</mo><mrow id="S3.E2.m1.2.3.3.3.1.2.3" xref="S3.E2.m1.2.3.3.3.1.2.3.cmml"><mi id="S3.E2.m1.2.3.3.3.1.2.3.2" xref="S3.E2.m1.2.3.3.3.1.2.3.2.cmml">i</mi><mo id="S3.E2.m1.2.3.3.3.1.2.3.1" xref="S3.E2.m1.2.3.3.3.1.2.3.1.cmml">=</mo><mn id="S3.E2.m1.2.3.3.3.1.2.3.3" xref="S3.E2.m1.2.3.3.3.1.2.3.3.cmml">1</mn></mrow><mi id="S3.E2.m1.2.3.3.3.1.3" xref="S3.E2.m1.2.3.3.3.1.3.cmml">t</mi></munderover><msubsup id="S3.E2.m1.2.3.3.3.2" xref="S3.E2.m1.2.3.3.3.2.cmml"><mi id="S3.E2.m1.2.3.3.3.2.2.2" xref="S3.E2.m1.2.3.3.3.2.2.2.cmml">R</mi><mrow id="S3.E2.m1.2.2.2.4" xref="S3.E2.m1.2.2.2.3.cmml"><mi id="S3.E2.m1.1.1.1.1" xref="S3.E2.m1.1.1.1.1.cmml">t</mi><mo id="S3.E2.m1.2.2.2.4.1" xref="S3.E2.m1.2.2.2.3.cmml">,</mo><mi id="S3.E2.m1.2.2.2.2" xref="S3.E2.m1.2.2.2.2.cmml">i</mi></mrow><mi id="S3.E2.m1.2.3.3.3.2.3" xref="S3.E2.m1.2.3.3.3.2.3.cmml">D</mi></msubsup></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.E2.m1.2b"><apply id="S3.E2.m1.2.3.cmml" xref="S3.E2.m1.2.3"><eq id="S3.E2.m1.2.3.1.cmml" xref="S3.E2.m1.2.3.1"></eq><apply id="S3.E2.m1.2.3.2.cmml" xref="S3.E2.m1.2.3.2"><times id="S3.E2.m1.2.3.2.1.cmml" xref="S3.E2.m1.2.3.2.1"></times><ci id="S3.E2.m1.2.3.2.2.cmml" xref="S3.E2.m1.2.3.2.2">𝑂</ci><apply id="S3.E2.m1.2.3.2.3.cmml" xref="S3.E2.m1.2.3.2.3"><csymbol cd="ambiguous" id="S3.E2.m1.2.3.2.3.1.cmml" xref="S3.E2.m1.2.3.2.3">subscript</csymbol><ci id="S3.E2.m1.2.3.2.3.2.cmml" xref="S3.E2.m1.2.3.2.3.2">𝑃</ci><ci id="S3.E2.m1.2.3.2.3.3.cmml" xref="S3.E2.m1.2.3.2.3.3">𝑡</ci></apply></apply><apply id="S3.E2.m1.2.3.3.cmml" xref="S3.E2.m1.2.3.3"><times id="S3.E2.m1.2.3.3.1.cmml" xref="S3.E2.m1.2.3.3.1"></times><apply id="S3.E2.m1.2.3.3.2.cmml" xref="S3.E2.m1.2.3.3.2"><divide id="S3.E2.m1.2.3.3.2.1.cmml" xref="S3.E2.m1.2.3.3.2"></divide><cn type="integer" id="S3.E2.m1.2.3.3.2.2.cmml" xref="S3.E2.m1.2.3.3.2.2">1</cn><ci id="S3.E2.m1.2.3.3.2.3.cmml" xref="S3.E2.m1.2.3.3.2.3">𝑡</ci></apply><apply id="S3.E2.m1.2.3.3.3.cmml" xref="S3.E2.m1.2.3.3.3"><apply id="S3.E2.m1.2.3.3.3.1.cmml" xref="S3.E2.m1.2.3.3.3.1"><csymbol cd="ambiguous" id="S3.E2.m1.2.3.3.3.1.1.cmml" xref="S3.E2.m1.2.3.3.3.1">superscript</csymbol><apply id="S3.E2.m1.2.3.3.3.1.2.cmml" xref="S3.E2.m1.2.3.3.3.1"><csymbol cd="ambiguous" id="S3.E2.m1.2.3.3.3.1.2.1.cmml" xref="S3.E2.m1.2.3.3.3.1">subscript</csymbol><sum id="S3.E2.m1.2.3.3.3.1.2.2.cmml" xref="S3.E2.m1.2.3.3.3.1.2.2"></sum><apply id="S3.E2.m1.2.3.3.3.1.2.3.cmml" xref="S3.E2.m1.2.3.3.3.1.2.3"><eq id="S3.E2.m1.2.3.3.3.1.2.3.1.cmml" xref="S3.E2.m1.2.3.3.3.1.2.3.1"></eq><ci id="S3.E2.m1.2.3.3.3.1.2.3.2.cmml" xref="S3.E2.m1.2.3.3.3.1.2.3.2">𝑖</ci><cn type="integer" id="S3.E2.m1.2.3.3.3.1.2.3.3.cmml" xref="S3.E2.m1.2.3.3.3.1.2.3.3">1</cn></apply></apply><ci id="S3.E2.m1.2.3.3.3.1.3.cmml" xref="S3.E2.m1.2.3.3.3.1.3">𝑡</ci></apply><apply id="S3.E2.m1.2.3.3.3.2.cmml" xref="S3.E2.m1.2.3.3.3.2"><csymbol cd="ambiguous" id="S3.E2.m1.2.3.3.3.2.1.cmml" xref="S3.E2.m1.2.3.3.3.2">superscript</csymbol><apply id="S3.E2.m1.2.3.3.3.2.2.cmml" xref="S3.E2.m1.2.3.3.3.2"><csymbol cd="ambiguous" id="S3.E2.m1.2.3.3.3.2.2.1.cmml" xref="S3.E2.m1.2.3.3.3.2">subscript</csymbol><ci id="S3.E2.m1.2.3.3.3.2.2.2.cmml" xref="S3.E2.m1.2.3.3.3.2.2.2">𝑅</ci><list id="S3.E2.m1.2.2.2.3.cmml" xref="S3.E2.m1.2.2.2.4"><ci id="S3.E2.m1.1.1.1.1.cmml" xref="S3.E2.m1.1.1.1.1">𝑡</ci><ci id="S3.E2.m1.2.2.2.2.cmml" xref="S3.E2.m1.2.2.2.2">𝑖</ci></list></apply><ci id="S3.E2.m1.2.3.3.3.2.3.cmml" xref="S3.E2.m1.2.3.3.3.2.3">𝐷</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E2.m1.2c">OP_{t}=\frac{1}{t}\sum_{i=1}^{t}R_{t,i}^{D}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(2)</span></td>
</tr></tbody>
</table>
</div>
<div id="S3.p3" class="ltx_para ltx_noindent">
<table id="S3.E3" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S3.E3.m1.5" class="ltx_Math" alttext="BWT_{t}=\frac{1}{t}\sum_{i=1}^{t-1}\left(R_{t,i}^{D}-R_{i,i}^{D}\right)" display="block"><semantics id="S3.E3.m1.5a"><mrow id="S3.E3.m1.5.5" xref="S3.E3.m1.5.5.cmml"><mrow id="S3.E3.m1.5.5.3" xref="S3.E3.m1.5.5.3.cmml"><mi id="S3.E3.m1.5.5.3.2" xref="S3.E3.m1.5.5.3.2.cmml">B</mi><mo lspace="0em" rspace="0em" id="S3.E3.m1.5.5.3.1" xref="S3.E3.m1.5.5.3.1.cmml">​</mo><mi id="S3.E3.m1.5.5.3.3" xref="S3.E3.m1.5.5.3.3.cmml">W</mi><mo lspace="0em" rspace="0em" id="S3.E3.m1.5.5.3.1a" xref="S3.E3.m1.5.5.3.1.cmml">​</mo><msub id="S3.E3.m1.5.5.3.4" xref="S3.E3.m1.5.5.3.4.cmml"><mi id="S3.E3.m1.5.5.3.4.2" xref="S3.E3.m1.5.5.3.4.2.cmml">T</mi><mi id="S3.E3.m1.5.5.3.4.3" xref="S3.E3.m1.5.5.3.4.3.cmml">t</mi></msub></mrow><mo id="S3.E3.m1.5.5.2" xref="S3.E3.m1.5.5.2.cmml">=</mo><mrow id="S3.E3.m1.5.5.1" xref="S3.E3.m1.5.5.1.cmml"><mfrac id="S3.E3.m1.5.5.1.3" xref="S3.E3.m1.5.5.1.3.cmml"><mn id="S3.E3.m1.5.5.1.3.2" xref="S3.E3.m1.5.5.1.3.2.cmml">1</mn><mi id="S3.E3.m1.5.5.1.3.3" xref="S3.E3.m1.5.5.1.3.3.cmml">t</mi></mfrac><mo lspace="0em" rspace="0em" id="S3.E3.m1.5.5.1.2" xref="S3.E3.m1.5.5.1.2.cmml">​</mo><mrow id="S3.E3.m1.5.5.1.1" xref="S3.E3.m1.5.5.1.1.cmml"><munderover id="S3.E3.m1.5.5.1.1.2" xref="S3.E3.m1.5.5.1.1.2.cmml"><mo movablelimits="false" rspace="0em" id="S3.E3.m1.5.5.1.1.2.2.2" xref="S3.E3.m1.5.5.1.1.2.2.2.cmml">∑</mo><mrow id="S3.E3.m1.5.5.1.1.2.2.3" xref="S3.E3.m1.5.5.1.1.2.2.3.cmml"><mi id="S3.E3.m1.5.5.1.1.2.2.3.2" xref="S3.E3.m1.5.5.1.1.2.2.3.2.cmml">i</mi><mo id="S3.E3.m1.5.5.1.1.2.2.3.1" xref="S3.E3.m1.5.5.1.1.2.2.3.1.cmml">=</mo><mn id="S3.E3.m1.5.5.1.1.2.2.3.3" xref="S3.E3.m1.5.5.1.1.2.2.3.3.cmml">1</mn></mrow><mrow id="S3.E3.m1.5.5.1.1.2.3" xref="S3.E3.m1.5.5.1.1.2.3.cmml"><mi id="S3.E3.m1.5.5.1.1.2.3.2" xref="S3.E3.m1.5.5.1.1.2.3.2.cmml">t</mi><mo id="S3.E3.m1.5.5.1.1.2.3.1" xref="S3.E3.m1.5.5.1.1.2.3.1.cmml">−</mo><mn id="S3.E3.m1.5.5.1.1.2.3.3" xref="S3.E3.m1.5.5.1.1.2.3.3.cmml">1</mn></mrow></munderover><mrow id="S3.E3.m1.5.5.1.1.1.1" xref="S3.E3.m1.5.5.1.1.1.1.1.cmml"><mo id="S3.E3.m1.5.5.1.1.1.1.2" xref="S3.E3.m1.5.5.1.1.1.1.1.cmml">(</mo><mrow id="S3.E3.m1.5.5.1.1.1.1.1" xref="S3.E3.m1.5.5.1.1.1.1.1.cmml"><msubsup id="S3.E3.m1.5.5.1.1.1.1.1.2" xref="S3.E3.m1.5.5.1.1.1.1.1.2.cmml"><mi id="S3.E3.m1.5.5.1.1.1.1.1.2.2.2" xref="S3.E3.m1.5.5.1.1.1.1.1.2.2.2.cmml">R</mi><mrow id="S3.E3.m1.2.2.2.4" xref="S3.E3.m1.2.2.2.3.cmml"><mi id="S3.E3.m1.1.1.1.1" xref="S3.E3.m1.1.1.1.1.cmml">t</mi><mo id="S3.E3.m1.2.2.2.4.1" xref="S3.E3.m1.2.2.2.3.cmml">,</mo><mi id="S3.E3.m1.2.2.2.2" xref="S3.E3.m1.2.2.2.2.cmml">i</mi></mrow><mi id="S3.E3.m1.5.5.1.1.1.1.1.2.3" xref="S3.E3.m1.5.5.1.1.1.1.1.2.3.cmml">D</mi></msubsup><mo id="S3.E3.m1.5.5.1.1.1.1.1.1" xref="S3.E3.m1.5.5.1.1.1.1.1.1.cmml">−</mo><msubsup id="S3.E3.m1.5.5.1.1.1.1.1.3" xref="S3.E3.m1.5.5.1.1.1.1.1.3.cmml"><mi id="S3.E3.m1.5.5.1.1.1.1.1.3.2.2" xref="S3.E3.m1.5.5.1.1.1.1.1.3.2.2.cmml">R</mi><mrow id="S3.E3.m1.4.4.2.4" xref="S3.E3.m1.4.4.2.3.cmml"><mi id="S3.E3.m1.3.3.1.1" xref="S3.E3.m1.3.3.1.1.cmml">i</mi><mo id="S3.E3.m1.4.4.2.4.1" xref="S3.E3.m1.4.4.2.3.cmml">,</mo><mi id="S3.E3.m1.4.4.2.2" xref="S3.E3.m1.4.4.2.2.cmml">i</mi></mrow><mi id="S3.E3.m1.5.5.1.1.1.1.1.3.3" xref="S3.E3.m1.5.5.1.1.1.1.1.3.3.cmml">D</mi></msubsup></mrow><mo id="S3.E3.m1.5.5.1.1.1.1.3" xref="S3.E3.m1.5.5.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.E3.m1.5b"><apply id="S3.E3.m1.5.5.cmml" xref="S3.E3.m1.5.5"><eq id="S3.E3.m1.5.5.2.cmml" xref="S3.E3.m1.5.5.2"></eq><apply id="S3.E3.m1.5.5.3.cmml" xref="S3.E3.m1.5.5.3"><times id="S3.E3.m1.5.5.3.1.cmml" xref="S3.E3.m1.5.5.3.1"></times><ci id="S3.E3.m1.5.5.3.2.cmml" xref="S3.E3.m1.5.5.3.2">𝐵</ci><ci id="S3.E3.m1.5.5.3.3.cmml" xref="S3.E3.m1.5.5.3.3">𝑊</ci><apply id="S3.E3.m1.5.5.3.4.cmml" xref="S3.E3.m1.5.5.3.4"><csymbol cd="ambiguous" id="S3.E3.m1.5.5.3.4.1.cmml" xref="S3.E3.m1.5.5.3.4">subscript</csymbol><ci id="S3.E3.m1.5.5.3.4.2.cmml" xref="S3.E3.m1.5.5.3.4.2">𝑇</ci><ci id="S3.E3.m1.5.5.3.4.3.cmml" xref="S3.E3.m1.5.5.3.4.3">𝑡</ci></apply></apply><apply id="S3.E3.m1.5.5.1.cmml" xref="S3.E3.m1.5.5.1"><times id="S3.E3.m1.5.5.1.2.cmml" xref="S3.E3.m1.5.5.1.2"></times><apply id="S3.E3.m1.5.5.1.3.cmml" xref="S3.E3.m1.5.5.1.3"><divide id="S3.E3.m1.5.5.1.3.1.cmml" xref="S3.E3.m1.5.5.1.3"></divide><cn type="integer" id="S3.E3.m1.5.5.1.3.2.cmml" xref="S3.E3.m1.5.5.1.3.2">1</cn><ci id="S3.E3.m1.5.5.1.3.3.cmml" xref="S3.E3.m1.5.5.1.3.3">𝑡</ci></apply><apply id="S3.E3.m1.5.5.1.1.cmml" xref="S3.E3.m1.5.5.1.1"><apply id="S3.E3.m1.5.5.1.1.2.cmml" xref="S3.E3.m1.5.5.1.1.2"><csymbol cd="ambiguous" id="S3.E3.m1.5.5.1.1.2.1.cmml" xref="S3.E3.m1.5.5.1.1.2">superscript</csymbol><apply id="S3.E3.m1.5.5.1.1.2.2.cmml" xref="S3.E3.m1.5.5.1.1.2"><csymbol cd="ambiguous" id="S3.E3.m1.5.5.1.1.2.2.1.cmml" xref="S3.E3.m1.5.5.1.1.2">subscript</csymbol><sum id="S3.E3.m1.5.5.1.1.2.2.2.cmml" xref="S3.E3.m1.5.5.1.1.2.2.2"></sum><apply id="S3.E3.m1.5.5.1.1.2.2.3.cmml" xref="S3.E3.m1.5.5.1.1.2.2.3"><eq id="S3.E3.m1.5.5.1.1.2.2.3.1.cmml" xref="S3.E3.m1.5.5.1.1.2.2.3.1"></eq><ci id="S3.E3.m1.5.5.1.1.2.2.3.2.cmml" xref="S3.E3.m1.5.5.1.1.2.2.3.2">𝑖</ci><cn type="integer" id="S3.E3.m1.5.5.1.1.2.2.3.3.cmml" xref="S3.E3.m1.5.5.1.1.2.2.3.3">1</cn></apply></apply><apply id="S3.E3.m1.5.5.1.1.2.3.cmml" xref="S3.E3.m1.5.5.1.1.2.3"><minus id="S3.E3.m1.5.5.1.1.2.3.1.cmml" xref="S3.E3.m1.5.5.1.1.2.3.1"></minus><ci id="S3.E3.m1.5.5.1.1.2.3.2.cmml" xref="S3.E3.m1.5.5.1.1.2.3.2">𝑡</ci><cn type="integer" id="S3.E3.m1.5.5.1.1.2.3.3.cmml" xref="S3.E3.m1.5.5.1.1.2.3.3">1</cn></apply></apply><apply id="S3.E3.m1.5.5.1.1.1.1.1.cmml" xref="S3.E3.m1.5.5.1.1.1.1"><minus id="S3.E3.m1.5.5.1.1.1.1.1.1.cmml" xref="S3.E3.m1.5.5.1.1.1.1.1.1"></minus><apply id="S3.E3.m1.5.5.1.1.1.1.1.2.cmml" xref="S3.E3.m1.5.5.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S3.E3.m1.5.5.1.1.1.1.1.2.1.cmml" xref="S3.E3.m1.5.5.1.1.1.1.1.2">superscript</csymbol><apply id="S3.E3.m1.5.5.1.1.1.1.1.2.2.cmml" xref="S3.E3.m1.5.5.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S3.E3.m1.5.5.1.1.1.1.1.2.2.1.cmml" xref="S3.E3.m1.5.5.1.1.1.1.1.2">subscript</csymbol><ci id="S3.E3.m1.5.5.1.1.1.1.1.2.2.2.cmml" xref="S3.E3.m1.5.5.1.1.1.1.1.2.2.2">𝑅</ci><list id="S3.E3.m1.2.2.2.3.cmml" xref="S3.E3.m1.2.2.2.4"><ci id="S3.E3.m1.1.1.1.1.cmml" xref="S3.E3.m1.1.1.1.1">𝑡</ci><ci id="S3.E3.m1.2.2.2.2.cmml" xref="S3.E3.m1.2.2.2.2">𝑖</ci></list></apply><ci id="S3.E3.m1.5.5.1.1.1.1.1.2.3.cmml" xref="S3.E3.m1.5.5.1.1.1.1.1.2.3">𝐷</ci></apply><apply id="S3.E3.m1.5.5.1.1.1.1.1.3.cmml" xref="S3.E3.m1.5.5.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.E3.m1.5.5.1.1.1.1.1.3.1.cmml" xref="S3.E3.m1.5.5.1.1.1.1.1.3">superscript</csymbol><apply id="S3.E3.m1.5.5.1.1.1.1.1.3.2.cmml" xref="S3.E3.m1.5.5.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.E3.m1.5.5.1.1.1.1.1.3.2.1.cmml" xref="S3.E3.m1.5.5.1.1.1.1.1.3">subscript</csymbol><ci id="S3.E3.m1.5.5.1.1.1.1.1.3.2.2.cmml" xref="S3.E3.m1.5.5.1.1.1.1.1.3.2.2">𝑅</ci><list id="S3.E3.m1.4.4.2.3.cmml" xref="S3.E3.m1.4.4.2.4"><ci id="S3.E3.m1.3.3.1.1.cmml" xref="S3.E3.m1.3.3.1.1">𝑖</ci><ci id="S3.E3.m1.4.4.2.2.cmml" xref="S3.E3.m1.4.4.2.2">𝑖</ci></list></apply><ci id="S3.E3.m1.5.5.1.1.1.1.1.3.3.cmml" xref="S3.E3.m1.5.5.1.1.1.1.1.3.3">𝐷</ci></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E3.m1.5c">BWT_{t}=\frac{1}{t}\sum_{i=1}^{t-1}\left(R_{t,i}^{D}-R_{i,i}^{D}\right)</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(3)</span></td>
</tr></tbody>
</table>
</div>
</section>
<section id="S4" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">4 </span>TRACE: A Comprehensive Benchmark for CL in LLMs</h2>

<div id="S4.p1" class="ltx_para ltx_noindent">
<p id="S4.p1.1" class="ltx_p">TRACE is designed to offer a comprehensive continual learning evaluation for LLMs. Illustrated in Figure <a href="#S1.F1" title="Figure 1 ‣ 1 Introduction ‣ TRACE: A Comprehensive Benchmark for Continual Learning in Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>, TRACE encompasses two primary components: a curated set of tasks tailored for continual learning, followed by an in-depth evaluation of an LLM’s post-training capabilities.
In this section, we detail TRACE’s sequential tasks and introduce our evaluation metrics. In Section <a href="#S4.SS4" title="4.4 Main Results ‣ 4 TRACE: A Comprehensive Benchmark for CL in LLMs ‣ TRACE: A Comprehensive Benchmark for Continual Learning in Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4.4</span></a>, we evaluate five models using TRACE and present our key findings.</p>
</div>
<section id="S4.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.1 </span>Data Creation</h3>

<div id="S4.SS1.p1" class="ltx_para ltx_noindent">
<p id="S4.SS1.p1.1" class="ltx_p">There are three principles for the creation of TRACE. First, the datasets should be novel enough that most LLMs have not been trained on them. Second, they should be challenging for large language models. Third, a variety of tasks should be covered in our benchmark.</p>
</div>
<div id="S4.SS1.p2" class="ltx_para ltx_noindent">
<p id="S4.SS1.p2.1" class="ltx_p">According to these three principles, in this section, we will provide a detailed introduction to the data collection process for each dataset. As these datasets have varying sizes, we create a balanced version by randomly sampling 5000 training examples and 2000 testing examples from the original datasets. As shown in Table <a href="#Ax1.T4" title="Table 4 ‣ .2 Trace dataset statistics ‣ Appendix ‣ TRACE: A Comprehensive Benchmark for Continual Learning in Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>, we get 40,000 training examples and 16,000 testing examples in total.</p>
</div>
<div id="S4.SS1.p3" class="ltx_para ltx_noindent">
<p id="S4.SS1.p3.1" class="ltx_p"><span id="S4.SS1.p3.1.1" class="ltx_text ltx_font_bold">Domain-Specific.</span> These tasks require specific knowledge, so models may perform poorly if they have not appeared frequently enough in the training data. We select three datasets, ScienceQA <cite class="ltx_cite ltx_citemacro_citep">(Lu et al., <a href="#bib.bib28" title="" class="ltx_ref">2022</a>)</cite>, FOMC <cite class="ltx_cite ltx_citemacro_citep">(Shah et al., <a href="#bib.bib40" title="" class="ltx_ref">2023</a>)</cite> and MeetingBank <cite class="ltx_cite ltx_citemacro_citep">(Hu et al., <a href="#bib.bib20" title="" class="ltx_ref">2023</a>)</cite>.
ScienceQA is a multi-hop QA dataset collected from elementary and high school science curricula, with a rich domain diversity from natural science, social science, and language science, requiring the model of reasoning ability and science knowledge.
As we only test the performance of language models, only the examples without multi-modal contexts are included in TRACE. FOMC is a hawkish-dovish classification task, which is novel in the financial domain. The dataset is divided into three subsets: data on meeting minutes, press conference data, and speech data. We use a combination of them. MeetingBank is a new benchmark dataset for city council meeting summarization, an unstudied domain. It demands a global understanding of the whole long context.</p>
</div>
<div id="S4.SS1.p4" class="ltx_para ltx_noindent">
<p id="S4.SS1.p4.1" class="ltx_p"><span id="S4.SS1.p4.1.1" class="ltx_text ltx_font_bold">Multi-lingual.</span>
The cross-lingual ability of large language models is limited due to vocabulary and pre-training corpus. For instance, LLaMA’s vocabulary contains few Chinese tokens, affecting its efficiency with Chinese text. <cite class="ltx_cite ltx_citemacro_citep">(Cui et al., <a href="#bib.bib12" title="" class="ltx_ref">2023</a>)</cite> expand LLaMA’s Chinese vocabulary and fine-tune with additional Chinese corpus. Yet, capabilities for other languages can be forgotten after training in a specific language, making it vital to evaluate cross-lingual ability in our benchmark.
We select C-STANCE<cite class="ltx_cite ltx_citemacro_citep">(Zhao et al., <a href="#bib.bib54" title="" class="ltx_ref">2023</a>)</cite> and 20Minuten<cite class="ltx_cite ltx_citemacro_citep">(Rios et al., <a href="#bib.bib36" title="" class="ltx_ref">2021</a>)</cite> as multi-lingual datasets. C-STANCE is the first Chinese dataset for zero-shot stance detection collected from Sina Weibo, one of the most popular Chinese social media sites. It includes two challenging subtasks: target-based stance detection and domain-based stance detection. In TRACE, we include the target-based one, which means the targets in testing examples are unseen during training. 20Minuten is a text simplification dataset consisting of full articles paired with shortened, simplified summaries from the Swiss news magazine. We use this dataset to evaluate the ability to generate German text.</p>
</div>
<div id="S4.SS1.p5" class="ltx_para ltx_noindent">
<p id="S4.SS1.p5.1" class="ltx_p"><span id="S4.SS1.p5.1.1" class="ltx_text ltx_font_bold">Code completion.</span> Code completion is another challenging task to evaluate long context modeling ability<cite class="ltx_cite ltx_citemacro_citep">(Bai et al., <a href="#bib.bib1" title="" class="ltx_ref">2023</a>)</cite>, and it is one of the most widely used features in software development through IDEs. We select the line-level code completion task of CodeXGLUE<cite class="ltx_cite ltx_citemacro_citep">(Lu et al., <a href="#bib.bib29" title="" class="ltx_ref">2021</a>)</cite>, which requires the model to generate the next line given the lengthy code input. The corpus Py150 contains 150,000 Python programs collected from GitHub repositories. Since the golden labels of the testing dataset are not available by <cite class="ltx_cite ltx_citemacro_cite">Lu et al. (<a href="#bib.bib29" title="" class="ltx_ref">2021</a>)</cite>, we randomly divide each Python code in Py150 into two parts, taking the first part as inputs and the next line as labels.</p>
</div>
<div id="S4.SS1.p6" class="ltx_para ltx_noindent">
<p id="S4.SS1.p6.1" class="ltx_p"><span id="S4.SS1.p6.1.1" class="ltx_text ltx_font_bold">Mathematical reasoning.</span> Mathematical problems are always used to evaluate the reasoning ability of models. NumGLUE<cite class="ltx_cite ltx_citemacro_citep">(Mishra et al., <a href="#bib.bib32" title="" class="ltx_ref">2022</a>)</cite> is an 8-task benchmark far from solved including state-of-the-art large-scale language models performing significantly worse than humans. We include the first two tasks of NumGLUE because they are freshly collected data and intelligent modifications of already existing datasets. Both of the two tasks require arithmetic reasoning ability.
It is worth noting that both datasets have original labels consisting only of numbers, without associated inference processes.</p>
</div>
</section>
<section id="S4.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.2 </span>CL Metrics Design</h3>

<div id="S4.SS2.p1" class="ltx_para ltx_noindent">
<p id="S4.SS2.p1.1" class="ltx_p">Unlike traditional continual learning benchmarks focused on sequential target tasks, evaluating aligned LLMs should also account for the preservation of their inherent capabilities. SoTA LLMs, through instruction tuning, exhibit impressive task-solving abilities. Aligning these models with human preferences further boosts their safety and usefulness. Hence, TRACE introduces a broader evaluation, including three unique metrics: ”General Ability Delta,” ”Instruction Following Delta,” and ”Safety Delta.” These measure the changes in LLM’s general capability, instruction adherence, and response safety post-continual learning.
</p>
</div>
<div id="S4.SS2.p2" class="ltx_para ltx_noindent">
<p id="S4.SS2.p2.4" class="ltx_p"><span id="S4.SS2.p2.4.1" class="ltx_text ltx_font_bold">General Ability Delta</span> is designed to assess the change in performance of an LLM on generic tasks after training on sequential target tasks.
Let’s consider a set of general tasks denoted as <math id="S4.SS2.p2.1.m1.3" class="ltx_Math" alttext="\{G_{1},...,G_{M}\}" display="inline"><semantics id="S4.SS2.p2.1.m1.3a"><mrow id="S4.SS2.p2.1.m1.3.3.2" xref="S4.SS2.p2.1.m1.3.3.3.cmml"><mo stretchy="false" id="S4.SS2.p2.1.m1.3.3.2.3" xref="S4.SS2.p2.1.m1.3.3.3.cmml">{</mo><msub id="S4.SS2.p2.1.m1.2.2.1.1" xref="S4.SS2.p2.1.m1.2.2.1.1.cmml"><mi id="S4.SS2.p2.1.m1.2.2.1.1.2" xref="S4.SS2.p2.1.m1.2.2.1.1.2.cmml">G</mi><mn id="S4.SS2.p2.1.m1.2.2.1.1.3" xref="S4.SS2.p2.1.m1.2.2.1.1.3.cmml">1</mn></msub><mo id="S4.SS2.p2.1.m1.3.3.2.4" xref="S4.SS2.p2.1.m1.3.3.3.cmml">,</mo><mi mathvariant="normal" id="S4.SS2.p2.1.m1.1.1" xref="S4.SS2.p2.1.m1.1.1.cmml">…</mi><mo id="S4.SS2.p2.1.m1.3.3.2.5" xref="S4.SS2.p2.1.m1.3.3.3.cmml">,</mo><msub id="S4.SS2.p2.1.m1.3.3.2.2" xref="S4.SS2.p2.1.m1.3.3.2.2.cmml"><mi id="S4.SS2.p2.1.m1.3.3.2.2.2" xref="S4.SS2.p2.1.m1.3.3.2.2.2.cmml">G</mi><mi id="S4.SS2.p2.1.m1.3.3.2.2.3" xref="S4.SS2.p2.1.m1.3.3.2.2.3.cmml">M</mi></msub><mo stretchy="false" id="S4.SS2.p2.1.m1.3.3.2.6" xref="S4.SS2.p2.1.m1.3.3.3.cmml">}</mo></mrow><annotation-xml encoding="MathML-Content" id="S4.SS2.p2.1.m1.3b"><set id="S4.SS2.p2.1.m1.3.3.3.cmml" xref="S4.SS2.p2.1.m1.3.3.2"><apply id="S4.SS2.p2.1.m1.2.2.1.1.cmml" xref="S4.SS2.p2.1.m1.2.2.1.1"><csymbol cd="ambiguous" id="S4.SS2.p2.1.m1.2.2.1.1.1.cmml" xref="S4.SS2.p2.1.m1.2.2.1.1">subscript</csymbol><ci id="S4.SS2.p2.1.m1.2.2.1.1.2.cmml" xref="S4.SS2.p2.1.m1.2.2.1.1.2">𝐺</ci><cn type="integer" id="S4.SS2.p2.1.m1.2.2.1.1.3.cmml" xref="S4.SS2.p2.1.m1.2.2.1.1.3">1</cn></apply><ci id="S4.SS2.p2.1.m1.1.1.cmml" xref="S4.SS2.p2.1.m1.1.1">…</ci><apply id="S4.SS2.p2.1.m1.3.3.2.2.cmml" xref="S4.SS2.p2.1.m1.3.3.2.2"><csymbol cd="ambiguous" id="S4.SS2.p2.1.m1.3.3.2.2.1.cmml" xref="S4.SS2.p2.1.m1.3.3.2.2">subscript</csymbol><ci id="S4.SS2.p2.1.m1.3.3.2.2.2.cmml" xref="S4.SS2.p2.1.m1.3.3.2.2.2">𝐺</ci><ci id="S4.SS2.p2.1.m1.3.3.2.2.3.cmml" xref="S4.SS2.p2.1.m1.3.3.2.2.3">𝑀</ci></apply></set></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p2.1.m1.3c">\{G_{1},...,G_{M}\}</annotation></semantics></math>. The baseline performance of the initial LLM on the i-th general task is represented by <math id="S4.SS2.p2.2.m2.2" class="ltx_Math" alttext="R_{0,i}^{G}" display="inline"><semantics id="S4.SS2.p2.2.m2.2a"><msubsup id="S4.SS2.p2.2.m2.2.3" xref="S4.SS2.p2.2.m2.2.3.cmml"><mi id="S4.SS2.p2.2.m2.2.3.2.2" xref="S4.SS2.p2.2.m2.2.3.2.2.cmml">R</mi><mrow id="S4.SS2.p2.2.m2.2.2.2.4" xref="S4.SS2.p2.2.m2.2.2.2.3.cmml"><mn id="S4.SS2.p2.2.m2.1.1.1.1" xref="S4.SS2.p2.2.m2.1.1.1.1.cmml">0</mn><mo id="S4.SS2.p2.2.m2.2.2.2.4.1" xref="S4.SS2.p2.2.m2.2.2.2.3.cmml">,</mo><mi id="S4.SS2.p2.2.m2.2.2.2.2" xref="S4.SS2.p2.2.m2.2.2.2.2.cmml">i</mi></mrow><mi id="S4.SS2.p2.2.m2.2.3.3" xref="S4.SS2.p2.2.m2.2.3.3.cmml">G</mi></msubsup><annotation-xml encoding="MathML-Content" id="S4.SS2.p2.2.m2.2b"><apply id="S4.SS2.p2.2.m2.2.3.cmml" xref="S4.SS2.p2.2.m2.2.3"><csymbol cd="ambiguous" id="S4.SS2.p2.2.m2.2.3.1.cmml" xref="S4.SS2.p2.2.m2.2.3">superscript</csymbol><apply id="S4.SS2.p2.2.m2.2.3.2.cmml" xref="S4.SS2.p2.2.m2.2.3"><csymbol cd="ambiguous" id="S4.SS2.p2.2.m2.2.3.2.1.cmml" xref="S4.SS2.p2.2.m2.2.3">subscript</csymbol><ci id="S4.SS2.p2.2.m2.2.3.2.2.cmml" xref="S4.SS2.p2.2.m2.2.3.2.2">𝑅</ci><list id="S4.SS2.p2.2.m2.2.2.2.3.cmml" xref="S4.SS2.p2.2.m2.2.2.2.4"><cn type="integer" id="S4.SS2.p2.2.m2.1.1.1.1.cmml" xref="S4.SS2.p2.2.m2.1.1.1.1">0</cn><ci id="S4.SS2.p2.2.m2.2.2.2.2.cmml" xref="S4.SS2.p2.2.m2.2.2.2.2">𝑖</ci></list></apply><ci id="S4.SS2.p2.2.m2.2.3.3.cmml" xref="S4.SS2.p2.2.m2.2.3.3">𝐺</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p2.2.m2.2c">R_{0,i}^{G}</annotation></semantics></math>. After incrementally learning up to the t-th task, the score on the i-th general task becomes <math id="S4.SS2.p2.3.m3.2" class="ltx_Math" alttext="R_{t,i}^{G}" display="inline"><semantics id="S4.SS2.p2.3.m3.2a"><msubsup id="S4.SS2.p2.3.m3.2.3" xref="S4.SS2.p2.3.m3.2.3.cmml"><mi id="S4.SS2.p2.3.m3.2.3.2.2" xref="S4.SS2.p2.3.m3.2.3.2.2.cmml">R</mi><mrow id="S4.SS2.p2.3.m3.2.2.2.4" xref="S4.SS2.p2.3.m3.2.2.2.3.cmml"><mi id="S4.SS2.p2.3.m3.1.1.1.1" xref="S4.SS2.p2.3.m3.1.1.1.1.cmml">t</mi><mo id="S4.SS2.p2.3.m3.2.2.2.4.1" xref="S4.SS2.p2.3.m3.2.2.2.3.cmml">,</mo><mi id="S4.SS2.p2.3.m3.2.2.2.2" xref="S4.SS2.p2.3.m3.2.2.2.2.cmml">i</mi></mrow><mi id="S4.SS2.p2.3.m3.2.3.3" xref="S4.SS2.p2.3.m3.2.3.3.cmml">G</mi></msubsup><annotation-xml encoding="MathML-Content" id="S4.SS2.p2.3.m3.2b"><apply id="S4.SS2.p2.3.m3.2.3.cmml" xref="S4.SS2.p2.3.m3.2.3"><csymbol cd="ambiguous" id="S4.SS2.p2.3.m3.2.3.1.cmml" xref="S4.SS2.p2.3.m3.2.3">superscript</csymbol><apply id="S4.SS2.p2.3.m3.2.3.2.cmml" xref="S4.SS2.p2.3.m3.2.3"><csymbol cd="ambiguous" id="S4.SS2.p2.3.m3.2.3.2.1.cmml" xref="S4.SS2.p2.3.m3.2.3">subscript</csymbol><ci id="S4.SS2.p2.3.m3.2.3.2.2.cmml" xref="S4.SS2.p2.3.m3.2.3.2.2">𝑅</ci><list id="S4.SS2.p2.3.m3.2.2.2.3.cmml" xref="S4.SS2.p2.3.m3.2.2.2.4"><ci id="S4.SS2.p2.3.m3.1.1.1.1.cmml" xref="S4.SS2.p2.3.m3.1.1.1.1">𝑡</ci><ci id="S4.SS2.p2.3.m3.2.2.2.2.cmml" xref="S4.SS2.p2.3.m3.2.2.2.2">𝑖</ci></list></apply><ci id="S4.SS2.p2.3.m3.2.3.3.cmml" xref="S4.SS2.p2.3.m3.2.3.3">𝐺</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p2.3.m3.2c">R_{t,i}^{G}</annotation></semantics></math>. The ”General Ability Delta” after training on the t-th task, represented as <math id="S4.SS2.p2.4.m4.1" class="ltx_Math" alttext="\Delta R_{t}^{G}" display="inline"><semantics id="S4.SS2.p2.4.m4.1a"><mrow id="S4.SS2.p2.4.m4.1.1" xref="S4.SS2.p2.4.m4.1.1.cmml"><mi mathvariant="normal" id="S4.SS2.p2.4.m4.1.1.2" xref="S4.SS2.p2.4.m4.1.1.2.cmml">Δ</mi><mo lspace="0em" rspace="0em" id="S4.SS2.p2.4.m4.1.1.1" xref="S4.SS2.p2.4.m4.1.1.1.cmml">​</mo><msubsup id="S4.SS2.p2.4.m4.1.1.3" xref="S4.SS2.p2.4.m4.1.1.3.cmml"><mi id="S4.SS2.p2.4.m4.1.1.3.2.2" xref="S4.SS2.p2.4.m4.1.1.3.2.2.cmml">R</mi><mi id="S4.SS2.p2.4.m4.1.1.3.2.3" xref="S4.SS2.p2.4.m4.1.1.3.2.3.cmml">t</mi><mi id="S4.SS2.p2.4.m4.1.1.3.3" xref="S4.SS2.p2.4.m4.1.1.3.3.cmml">G</mi></msubsup></mrow><annotation-xml encoding="MathML-Content" id="S4.SS2.p2.4.m4.1b"><apply id="S4.SS2.p2.4.m4.1.1.cmml" xref="S4.SS2.p2.4.m4.1.1"><times id="S4.SS2.p2.4.m4.1.1.1.cmml" xref="S4.SS2.p2.4.m4.1.1.1"></times><ci id="S4.SS2.p2.4.m4.1.1.2.cmml" xref="S4.SS2.p2.4.m4.1.1.2">Δ</ci><apply id="S4.SS2.p2.4.m4.1.1.3.cmml" xref="S4.SS2.p2.4.m4.1.1.3"><csymbol cd="ambiguous" id="S4.SS2.p2.4.m4.1.1.3.1.cmml" xref="S4.SS2.p2.4.m4.1.1.3">superscript</csymbol><apply id="S4.SS2.p2.4.m4.1.1.3.2.cmml" xref="S4.SS2.p2.4.m4.1.1.3"><csymbol cd="ambiguous" id="S4.SS2.p2.4.m4.1.1.3.2.1.cmml" xref="S4.SS2.p2.4.m4.1.1.3">subscript</csymbol><ci id="S4.SS2.p2.4.m4.1.1.3.2.2.cmml" xref="S4.SS2.p2.4.m4.1.1.3.2.2">𝑅</ci><ci id="S4.SS2.p2.4.m4.1.1.3.2.3.cmml" xref="S4.SS2.p2.4.m4.1.1.3.2.3">𝑡</ci></apply><ci id="S4.SS2.p2.4.m4.1.1.3.3.cmml" xref="S4.SS2.p2.4.m4.1.1.3.3">𝐺</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p2.4.m4.1c">\Delta R_{t}^{G}</annotation></semantics></math>, is given by:</p>
<table id="S4.E4" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S4.E4.m1.5" class="ltx_Math" alttext="\Delta R_{t}^{G}=\frac{1}{M}\sum_{i=1}^{M}(R_{t,i}^{G}-R_{0,i}^{G})" display="block"><semantics id="S4.E4.m1.5a"><mrow id="S4.E4.m1.5.5" xref="S4.E4.m1.5.5.cmml"><mrow id="S4.E4.m1.5.5.3" xref="S4.E4.m1.5.5.3.cmml"><mi mathvariant="normal" id="S4.E4.m1.5.5.3.2" xref="S4.E4.m1.5.5.3.2.cmml">Δ</mi><mo lspace="0em" rspace="0em" id="S4.E4.m1.5.5.3.1" xref="S4.E4.m1.5.5.3.1.cmml">​</mo><msubsup id="S4.E4.m1.5.5.3.3" xref="S4.E4.m1.5.5.3.3.cmml"><mi id="S4.E4.m1.5.5.3.3.2.2" xref="S4.E4.m1.5.5.3.3.2.2.cmml">R</mi><mi id="S4.E4.m1.5.5.3.3.2.3" xref="S4.E4.m1.5.5.3.3.2.3.cmml">t</mi><mi id="S4.E4.m1.5.5.3.3.3" xref="S4.E4.m1.5.5.3.3.3.cmml">G</mi></msubsup></mrow><mo id="S4.E4.m1.5.5.2" xref="S4.E4.m1.5.5.2.cmml">=</mo><mrow id="S4.E4.m1.5.5.1" xref="S4.E4.m1.5.5.1.cmml"><mfrac id="S4.E4.m1.5.5.1.3" xref="S4.E4.m1.5.5.1.3.cmml"><mn id="S4.E4.m1.5.5.1.3.2" xref="S4.E4.m1.5.5.1.3.2.cmml">1</mn><mi id="S4.E4.m1.5.5.1.3.3" xref="S4.E4.m1.5.5.1.3.3.cmml">M</mi></mfrac><mo lspace="0em" rspace="0em" id="S4.E4.m1.5.5.1.2" xref="S4.E4.m1.5.5.1.2.cmml">​</mo><mrow id="S4.E4.m1.5.5.1.1" xref="S4.E4.m1.5.5.1.1.cmml"><munderover id="S4.E4.m1.5.5.1.1.2" xref="S4.E4.m1.5.5.1.1.2.cmml"><mo movablelimits="false" rspace="0em" id="S4.E4.m1.5.5.1.1.2.2.2" xref="S4.E4.m1.5.5.1.1.2.2.2.cmml">∑</mo><mrow id="S4.E4.m1.5.5.1.1.2.2.3" xref="S4.E4.m1.5.5.1.1.2.2.3.cmml"><mi id="S4.E4.m1.5.5.1.1.2.2.3.2" xref="S4.E4.m1.5.5.1.1.2.2.3.2.cmml">i</mi><mo id="S4.E4.m1.5.5.1.1.2.2.3.1" xref="S4.E4.m1.5.5.1.1.2.2.3.1.cmml">=</mo><mn id="S4.E4.m1.5.5.1.1.2.2.3.3" xref="S4.E4.m1.5.5.1.1.2.2.3.3.cmml">1</mn></mrow><mi id="S4.E4.m1.5.5.1.1.2.3" xref="S4.E4.m1.5.5.1.1.2.3.cmml">M</mi></munderover><mrow id="S4.E4.m1.5.5.1.1.1.1" xref="S4.E4.m1.5.5.1.1.1.1.1.cmml"><mo stretchy="false" id="S4.E4.m1.5.5.1.1.1.1.2" xref="S4.E4.m1.5.5.1.1.1.1.1.cmml">(</mo><mrow id="S4.E4.m1.5.5.1.1.1.1.1" xref="S4.E4.m1.5.5.1.1.1.1.1.cmml"><msubsup id="S4.E4.m1.5.5.1.1.1.1.1.2" xref="S4.E4.m1.5.5.1.1.1.1.1.2.cmml"><mi id="S4.E4.m1.5.5.1.1.1.1.1.2.2.2" xref="S4.E4.m1.5.5.1.1.1.1.1.2.2.2.cmml">R</mi><mrow id="S4.E4.m1.2.2.2.4" xref="S4.E4.m1.2.2.2.3.cmml"><mi id="S4.E4.m1.1.1.1.1" xref="S4.E4.m1.1.1.1.1.cmml">t</mi><mo id="S4.E4.m1.2.2.2.4.1" xref="S4.E4.m1.2.2.2.3.cmml">,</mo><mi id="S4.E4.m1.2.2.2.2" xref="S4.E4.m1.2.2.2.2.cmml">i</mi></mrow><mi id="S4.E4.m1.5.5.1.1.1.1.1.2.3" xref="S4.E4.m1.5.5.1.1.1.1.1.2.3.cmml">G</mi></msubsup><mo id="S4.E4.m1.5.5.1.1.1.1.1.1" xref="S4.E4.m1.5.5.1.1.1.1.1.1.cmml">−</mo><msubsup id="S4.E4.m1.5.5.1.1.1.1.1.3" xref="S4.E4.m1.5.5.1.1.1.1.1.3.cmml"><mi id="S4.E4.m1.5.5.1.1.1.1.1.3.2.2" xref="S4.E4.m1.5.5.1.1.1.1.1.3.2.2.cmml">R</mi><mrow id="S4.E4.m1.4.4.2.4" xref="S4.E4.m1.4.4.2.3.cmml"><mn id="S4.E4.m1.3.3.1.1" xref="S4.E4.m1.3.3.1.1.cmml">0</mn><mo id="S4.E4.m1.4.4.2.4.1" xref="S4.E4.m1.4.4.2.3.cmml">,</mo><mi id="S4.E4.m1.4.4.2.2" xref="S4.E4.m1.4.4.2.2.cmml">i</mi></mrow><mi id="S4.E4.m1.5.5.1.1.1.1.1.3.3" xref="S4.E4.m1.5.5.1.1.1.1.1.3.3.cmml">G</mi></msubsup></mrow><mo stretchy="false" id="S4.E4.m1.5.5.1.1.1.1.3" xref="S4.E4.m1.5.5.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.E4.m1.5b"><apply id="S4.E4.m1.5.5.cmml" xref="S4.E4.m1.5.5"><eq id="S4.E4.m1.5.5.2.cmml" xref="S4.E4.m1.5.5.2"></eq><apply id="S4.E4.m1.5.5.3.cmml" xref="S4.E4.m1.5.5.3"><times id="S4.E4.m1.5.5.3.1.cmml" xref="S4.E4.m1.5.5.3.1"></times><ci id="S4.E4.m1.5.5.3.2.cmml" xref="S4.E4.m1.5.5.3.2">Δ</ci><apply id="S4.E4.m1.5.5.3.3.cmml" xref="S4.E4.m1.5.5.3.3"><csymbol cd="ambiguous" id="S4.E4.m1.5.5.3.3.1.cmml" xref="S4.E4.m1.5.5.3.3">superscript</csymbol><apply id="S4.E4.m1.5.5.3.3.2.cmml" xref="S4.E4.m1.5.5.3.3"><csymbol cd="ambiguous" id="S4.E4.m1.5.5.3.3.2.1.cmml" xref="S4.E4.m1.5.5.3.3">subscript</csymbol><ci id="S4.E4.m1.5.5.3.3.2.2.cmml" xref="S4.E4.m1.5.5.3.3.2.2">𝑅</ci><ci id="S4.E4.m1.5.5.3.3.2.3.cmml" xref="S4.E4.m1.5.5.3.3.2.3">𝑡</ci></apply><ci id="S4.E4.m1.5.5.3.3.3.cmml" xref="S4.E4.m1.5.5.3.3.3">𝐺</ci></apply></apply><apply id="S4.E4.m1.5.5.1.cmml" xref="S4.E4.m1.5.5.1"><times id="S4.E4.m1.5.5.1.2.cmml" xref="S4.E4.m1.5.5.1.2"></times><apply id="S4.E4.m1.5.5.1.3.cmml" xref="S4.E4.m1.5.5.1.3"><divide id="S4.E4.m1.5.5.1.3.1.cmml" xref="S4.E4.m1.5.5.1.3"></divide><cn type="integer" id="S4.E4.m1.5.5.1.3.2.cmml" xref="S4.E4.m1.5.5.1.3.2">1</cn><ci id="S4.E4.m1.5.5.1.3.3.cmml" xref="S4.E4.m1.5.5.1.3.3">𝑀</ci></apply><apply id="S4.E4.m1.5.5.1.1.cmml" xref="S4.E4.m1.5.5.1.1"><apply id="S4.E4.m1.5.5.1.1.2.cmml" xref="S4.E4.m1.5.5.1.1.2"><csymbol cd="ambiguous" id="S4.E4.m1.5.5.1.1.2.1.cmml" xref="S4.E4.m1.5.5.1.1.2">superscript</csymbol><apply id="S4.E4.m1.5.5.1.1.2.2.cmml" xref="S4.E4.m1.5.5.1.1.2"><csymbol cd="ambiguous" id="S4.E4.m1.5.5.1.1.2.2.1.cmml" xref="S4.E4.m1.5.5.1.1.2">subscript</csymbol><sum id="S4.E4.m1.5.5.1.1.2.2.2.cmml" xref="S4.E4.m1.5.5.1.1.2.2.2"></sum><apply id="S4.E4.m1.5.5.1.1.2.2.3.cmml" xref="S4.E4.m1.5.5.1.1.2.2.3"><eq id="S4.E4.m1.5.5.1.1.2.2.3.1.cmml" xref="S4.E4.m1.5.5.1.1.2.2.3.1"></eq><ci id="S4.E4.m1.5.5.1.1.2.2.3.2.cmml" xref="S4.E4.m1.5.5.1.1.2.2.3.2">𝑖</ci><cn type="integer" id="S4.E4.m1.5.5.1.1.2.2.3.3.cmml" xref="S4.E4.m1.5.5.1.1.2.2.3.3">1</cn></apply></apply><ci id="S4.E4.m1.5.5.1.1.2.3.cmml" xref="S4.E4.m1.5.5.1.1.2.3">𝑀</ci></apply><apply id="S4.E4.m1.5.5.1.1.1.1.1.cmml" xref="S4.E4.m1.5.5.1.1.1.1"><minus id="S4.E4.m1.5.5.1.1.1.1.1.1.cmml" xref="S4.E4.m1.5.5.1.1.1.1.1.1"></minus><apply id="S4.E4.m1.5.5.1.1.1.1.1.2.cmml" xref="S4.E4.m1.5.5.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S4.E4.m1.5.5.1.1.1.1.1.2.1.cmml" xref="S4.E4.m1.5.5.1.1.1.1.1.2">superscript</csymbol><apply id="S4.E4.m1.5.5.1.1.1.1.1.2.2.cmml" xref="S4.E4.m1.5.5.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S4.E4.m1.5.5.1.1.1.1.1.2.2.1.cmml" xref="S4.E4.m1.5.5.1.1.1.1.1.2">subscript</csymbol><ci id="S4.E4.m1.5.5.1.1.1.1.1.2.2.2.cmml" xref="S4.E4.m1.5.5.1.1.1.1.1.2.2.2">𝑅</ci><list id="S4.E4.m1.2.2.2.3.cmml" xref="S4.E4.m1.2.2.2.4"><ci id="S4.E4.m1.1.1.1.1.cmml" xref="S4.E4.m1.1.1.1.1">𝑡</ci><ci id="S4.E4.m1.2.2.2.2.cmml" xref="S4.E4.m1.2.2.2.2">𝑖</ci></list></apply><ci id="S4.E4.m1.5.5.1.1.1.1.1.2.3.cmml" xref="S4.E4.m1.5.5.1.1.1.1.1.2.3">𝐺</ci></apply><apply id="S4.E4.m1.5.5.1.1.1.1.1.3.cmml" xref="S4.E4.m1.5.5.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S4.E4.m1.5.5.1.1.1.1.1.3.1.cmml" xref="S4.E4.m1.5.5.1.1.1.1.1.3">superscript</csymbol><apply id="S4.E4.m1.5.5.1.1.1.1.1.3.2.cmml" xref="S4.E4.m1.5.5.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S4.E4.m1.5.5.1.1.1.1.1.3.2.1.cmml" xref="S4.E4.m1.5.5.1.1.1.1.1.3">subscript</csymbol><ci id="S4.E4.m1.5.5.1.1.1.1.1.3.2.2.cmml" xref="S4.E4.m1.5.5.1.1.1.1.1.3.2.2">𝑅</ci><list id="S4.E4.m1.4.4.2.3.cmml" xref="S4.E4.m1.4.4.2.4"><cn type="integer" id="S4.E4.m1.3.3.1.1.cmml" xref="S4.E4.m1.3.3.1.1">0</cn><ci id="S4.E4.m1.4.4.2.2.cmml" xref="S4.E4.m1.4.4.2.2">𝑖</ci></list></apply><ci id="S4.E4.m1.5.5.1.1.1.1.1.3.3.cmml" xref="S4.E4.m1.5.5.1.1.1.1.1.3.3">𝐺</ci></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.E4.m1.5c">\Delta R_{t}^{G}=\frac{1}{M}\sum_{i=1}^{M}(R_{t,i}^{G}-R_{0,i}^{G})</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(4)</span></td>
</tr></tbody>
</table>
</div>
<div id="S4.SS2.p3" class="ltx_para ltx_noindent">
<p id="S4.SS2.p3.4" class="ltx_p"><span id="S4.SS2.p3.4.1" class="ltx_text ltx_font_bold">Instruction Following Delta</span> measures the change in a model’s ability to follow instructions after training on sequential tasks. Using a set of datasets, represented as <math id="S4.SS2.p3.1.m1.3" class="ltx_Math" alttext="{I_{1},...,I_{N}}" display="inline"><semantics id="S4.SS2.p3.1.m1.3a"><mrow id="S4.SS2.p3.1.m1.3.3.2" xref="S4.SS2.p3.1.m1.3.3.3.cmml"><msub id="S4.SS2.p3.1.m1.2.2.1.1" xref="S4.SS2.p3.1.m1.2.2.1.1.cmml"><mi id="S4.SS2.p3.1.m1.2.2.1.1.2" xref="S4.SS2.p3.1.m1.2.2.1.1.2.cmml">I</mi><mn id="S4.SS2.p3.1.m1.2.2.1.1.3" xref="S4.SS2.p3.1.m1.2.2.1.1.3.cmml">1</mn></msub><mo id="S4.SS2.p3.1.m1.3.3.2.3" xref="S4.SS2.p3.1.m1.3.3.3.cmml">,</mo><mi mathvariant="normal" id="S4.SS2.p3.1.m1.1.1" xref="S4.SS2.p3.1.m1.1.1.cmml">…</mi><mo id="S4.SS2.p3.1.m1.3.3.2.4" xref="S4.SS2.p3.1.m1.3.3.3.cmml">,</mo><msub id="S4.SS2.p3.1.m1.3.3.2.2" xref="S4.SS2.p3.1.m1.3.3.2.2.cmml"><mi id="S4.SS2.p3.1.m1.3.3.2.2.2" xref="S4.SS2.p3.1.m1.3.3.2.2.2.cmml">I</mi><mi id="S4.SS2.p3.1.m1.3.3.2.2.3" xref="S4.SS2.p3.1.m1.3.3.2.2.3.cmml">N</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="S4.SS2.p3.1.m1.3b"><list id="S4.SS2.p3.1.m1.3.3.3.cmml" xref="S4.SS2.p3.1.m1.3.3.2"><apply id="S4.SS2.p3.1.m1.2.2.1.1.cmml" xref="S4.SS2.p3.1.m1.2.2.1.1"><csymbol cd="ambiguous" id="S4.SS2.p3.1.m1.2.2.1.1.1.cmml" xref="S4.SS2.p3.1.m1.2.2.1.1">subscript</csymbol><ci id="S4.SS2.p3.1.m1.2.2.1.1.2.cmml" xref="S4.SS2.p3.1.m1.2.2.1.1.2">𝐼</ci><cn type="integer" id="S4.SS2.p3.1.m1.2.2.1.1.3.cmml" xref="S4.SS2.p3.1.m1.2.2.1.1.3">1</cn></apply><ci id="S4.SS2.p3.1.m1.1.1.cmml" xref="S4.SS2.p3.1.m1.1.1">…</ci><apply id="S4.SS2.p3.1.m1.3.3.2.2.cmml" xref="S4.SS2.p3.1.m1.3.3.2.2"><csymbol cd="ambiguous" id="S4.SS2.p3.1.m1.3.3.2.2.1.cmml" xref="S4.SS2.p3.1.m1.3.3.2.2">subscript</csymbol><ci id="S4.SS2.p3.1.m1.3.3.2.2.2.cmml" xref="S4.SS2.p3.1.m1.3.3.2.2.2">𝐼</ci><ci id="S4.SS2.p3.1.m1.3.3.2.2.3.cmml" xref="S4.SS2.p3.1.m1.3.3.2.2.3">𝑁</ci></apply></list></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p3.1.m1.3c">{I_{1},...,I_{N}}</annotation></semantics></math>, the initial LLM performance on the i-th task is <math id="S4.SS2.p3.2.m2.2" class="ltx_Math" alttext="R_{0,i}^{I}" display="inline"><semantics id="S4.SS2.p3.2.m2.2a"><msubsup id="S4.SS2.p3.2.m2.2.3" xref="S4.SS2.p3.2.m2.2.3.cmml"><mi id="S4.SS2.p3.2.m2.2.3.2.2" xref="S4.SS2.p3.2.m2.2.3.2.2.cmml">R</mi><mrow id="S4.SS2.p3.2.m2.2.2.2.4" xref="S4.SS2.p3.2.m2.2.2.2.3.cmml"><mn id="S4.SS2.p3.2.m2.1.1.1.1" xref="S4.SS2.p3.2.m2.1.1.1.1.cmml">0</mn><mo id="S4.SS2.p3.2.m2.2.2.2.4.1" xref="S4.SS2.p3.2.m2.2.2.2.3.cmml">,</mo><mi id="S4.SS2.p3.2.m2.2.2.2.2" xref="S4.SS2.p3.2.m2.2.2.2.2.cmml">i</mi></mrow><mi id="S4.SS2.p3.2.m2.2.3.3" xref="S4.SS2.p3.2.m2.2.3.3.cmml">I</mi></msubsup><annotation-xml encoding="MathML-Content" id="S4.SS2.p3.2.m2.2b"><apply id="S4.SS2.p3.2.m2.2.3.cmml" xref="S4.SS2.p3.2.m2.2.3"><csymbol cd="ambiguous" id="S4.SS2.p3.2.m2.2.3.1.cmml" xref="S4.SS2.p3.2.m2.2.3">superscript</csymbol><apply id="S4.SS2.p3.2.m2.2.3.2.cmml" xref="S4.SS2.p3.2.m2.2.3"><csymbol cd="ambiguous" id="S4.SS2.p3.2.m2.2.3.2.1.cmml" xref="S4.SS2.p3.2.m2.2.3">subscript</csymbol><ci id="S4.SS2.p3.2.m2.2.3.2.2.cmml" xref="S4.SS2.p3.2.m2.2.3.2.2">𝑅</ci><list id="S4.SS2.p3.2.m2.2.2.2.3.cmml" xref="S4.SS2.p3.2.m2.2.2.2.4"><cn type="integer" id="S4.SS2.p3.2.m2.1.1.1.1.cmml" xref="S4.SS2.p3.2.m2.1.1.1.1">0</cn><ci id="S4.SS2.p3.2.m2.2.2.2.2.cmml" xref="S4.SS2.p3.2.m2.2.2.2.2">𝑖</ci></list></apply><ci id="S4.SS2.p3.2.m2.2.3.3.cmml" xref="S4.SS2.p3.2.m2.2.3.3">𝐼</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p3.2.m2.2c">R_{0,i}^{I}</annotation></semantics></math>. After incremental learning to the t-th task, its score on the i-th task is <math id="S4.SS2.p3.3.m3.2" class="ltx_Math" alttext="R_{t,i}^{I}" display="inline"><semantics id="S4.SS2.p3.3.m3.2a"><msubsup id="S4.SS2.p3.3.m3.2.3" xref="S4.SS2.p3.3.m3.2.3.cmml"><mi id="S4.SS2.p3.3.m3.2.3.2.2" xref="S4.SS2.p3.3.m3.2.3.2.2.cmml">R</mi><mrow id="S4.SS2.p3.3.m3.2.2.2.4" xref="S4.SS2.p3.3.m3.2.2.2.3.cmml"><mi id="S4.SS2.p3.3.m3.1.1.1.1" xref="S4.SS2.p3.3.m3.1.1.1.1.cmml">t</mi><mo id="S4.SS2.p3.3.m3.2.2.2.4.1" xref="S4.SS2.p3.3.m3.2.2.2.3.cmml">,</mo><mi id="S4.SS2.p3.3.m3.2.2.2.2" xref="S4.SS2.p3.3.m3.2.2.2.2.cmml">i</mi></mrow><mi id="S4.SS2.p3.3.m3.2.3.3" xref="S4.SS2.p3.3.m3.2.3.3.cmml">I</mi></msubsup><annotation-xml encoding="MathML-Content" id="S4.SS2.p3.3.m3.2b"><apply id="S4.SS2.p3.3.m3.2.3.cmml" xref="S4.SS2.p3.3.m3.2.3"><csymbol cd="ambiguous" id="S4.SS2.p3.3.m3.2.3.1.cmml" xref="S4.SS2.p3.3.m3.2.3">superscript</csymbol><apply id="S4.SS2.p3.3.m3.2.3.2.cmml" xref="S4.SS2.p3.3.m3.2.3"><csymbol cd="ambiguous" id="S4.SS2.p3.3.m3.2.3.2.1.cmml" xref="S4.SS2.p3.3.m3.2.3">subscript</csymbol><ci id="S4.SS2.p3.3.m3.2.3.2.2.cmml" xref="S4.SS2.p3.3.m3.2.3.2.2">𝑅</ci><list id="S4.SS2.p3.3.m3.2.2.2.3.cmml" xref="S4.SS2.p3.3.m3.2.2.2.4"><ci id="S4.SS2.p3.3.m3.1.1.1.1.cmml" xref="S4.SS2.p3.3.m3.1.1.1.1">𝑡</ci><ci id="S4.SS2.p3.3.m3.2.2.2.2.cmml" xref="S4.SS2.p3.3.m3.2.2.2.2">𝑖</ci></list></apply><ci id="S4.SS2.p3.3.m3.2.3.3.cmml" xref="S4.SS2.p3.3.m3.2.3.3">𝐼</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p3.3.m3.2c">R_{t,i}^{I}</annotation></semantics></math>. The change, represented by <math id="S4.SS2.p3.4.m4.1" class="ltx_Math" alttext="\Delta R_{t}^{I}" display="inline"><semantics id="S4.SS2.p3.4.m4.1a"><mrow id="S4.SS2.p3.4.m4.1.1" xref="S4.SS2.p3.4.m4.1.1.cmml"><mi mathvariant="normal" id="S4.SS2.p3.4.m4.1.1.2" xref="S4.SS2.p3.4.m4.1.1.2.cmml">Δ</mi><mo lspace="0em" rspace="0em" id="S4.SS2.p3.4.m4.1.1.1" xref="S4.SS2.p3.4.m4.1.1.1.cmml">​</mo><msubsup id="S4.SS2.p3.4.m4.1.1.3" xref="S4.SS2.p3.4.m4.1.1.3.cmml"><mi id="S4.SS2.p3.4.m4.1.1.3.2.2" xref="S4.SS2.p3.4.m4.1.1.3.2.2.cmml">R</mi><mi id="S4.SS2.p3.4.m4.1.1.3.2.3" xref="S4.SS2.p3.4.m4.1.1.3.2.3.cmml">t</mi><mi id="S4.SS2.p3.4.m4.1.1.3.3" xref="S4.SS2.p3.4.m4.1.1.3.3.cmml">I</mi></msubsup></mrow><annotation-xml encoding="MathML-Content" id="S4.SS2.p3.4.m4.1b"><apply id="S4.SS2.p3.4.m4.1.1.cmml" xref="S4.SS2.p3.4.m4.1.1"><times id="S4.SS2.p3.4.m4.1.1.1.cmml" xref="S4.SS2.p3.4.m4.1.1.1"></times><ci id="S4.SS2.p3.4.m4.1.1.2.cmml" xref="S4.SS2.p3.4.m4.1.1.2">Δ</ci><apply id="S4.SS2.p3.4.m4.1.1.3.cmml" xref="S4.SS2.p3.4.m4.1.1.3"><csymbol cd="ambiguous" id="S4.SS2.p3.4.m4.1.1.3.1.cmml" xref="S4.SS2.p3.4.m4.1.1.3">superscript</csymbol><apply id="S4.SS2.p3.4.m4.1.1.3.2.cmml" xref="S4.SS2.p3.4.m4.1.1.3"><csymbol cd="ambiguous" id="S4.SS2.p3.4.m4.1.1.3.2.1.cmml" xref="S4.SS2.p3.4.m4.1.1.3">subscript</csymbol><ci id="S4.SS2.p3.4.m4.1.1.3.2.2.cmml" xref="S4.SS2.p3.4.m4.1.1.3.2.2">𝑅</ci><ci id="S4.SS2.p3.4.m4.1.1.3.2.3.cmml" xref="S4.SS2.p3.4.m4.1.1.3.2.3">𝑡</ci></apply><ci id="S4.SS2.p3.4.m4.1.1.3.3.cmml" xref="S4.SS2.p3.4.m4.1.1.3.3">𝐼</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p3.4.m4.1c">\Delta R_{t}^{I}</annotation></semantics></math>, is computed as:</p>
<table id="S4.E5" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S4.E5.m1.5" class="ltx_Math" alttext="\Delta R_{t}^{I}=\frac{1}{N}\sum_{i=1}^{N}(R_{t,i}^{I}-R_{0,i}^{I})" display="block"><semantics id="S4.E5.m1.5a"><mrow id="S4.E5.m1.5.5" xref="S4.E5.m1.5.5.cmml"><mrow id="S4.E5.m1.5.5.3" xref="S4.E5.m1.5.5.3.cmml"><mi mathvariant="normal" id="S4.E5.m1.5.5.3.2" xref="S4.E5.m1.5.5.3.2.cmml">Δ</mi><mo lspace="0em" rspace="0em" id="S4.E5.m1.5.5.3.1" xref="S4.E5.m1.5.5.3.1.cmml">​</mo><msubsup id="S4.E5.m1.5.5.3.3" xref="S4.E5.m1.5.5.3.3.cmml"><mi id="S4.E5.m1.5.5.3.3.2.2" xref="S4.E5.m1.5.5.3.3.2.2.cmml">R</mi><mi id="S4.E5.m1.5.5.3.3.2.3" xref="S4.E5.m1.5.5.3.3.2.3.cmml">t</mi><mi id="S4.E5.m1.5.5.3.3.3" xref="S4.E5.m1.5.5.3.3.3.cmml">I</mi></msubsup></mrow><mo id="S4.E5.m1.5.5.2" xref="S4.E5.m1.5.5.2.cmml">=</mo><mrow id="S4.E5.m1.5.5.1" xref="S4.E5.m1.5.5.1.cmml"><mfrac id="S4.E5.m1.5.5.1.3" xref="S4.E5.m1.5.5.1.3.cmml"><mn id="S4.E5.m1.5.5.1.3.2" xref="S4.E5.m1.5.5.1.3.2.cmml">1</mn><mi id="S4.E5.m1.5.5.1.3.3" xref="S4.E5.m1.5.5.1.3.3.cmml">N</mi></mfrac><mo lspace="0em" rspace="0em" id="S4.E5.m1.5.5.1.2" xref="S4.E5.m1.5.5.1.2.cmml">​</mo><mrow id="S4.E5.m1.5.5.1.1" xref="S4.E5.m1.5.5.1.1.cmml"><munderover id="S4.E5.m1.5.5.1.1.2" xref="S4.E5.m1.5.5.1.1.2.cmml"><mo movablelimits="false" rspace="0em" id="S4.E5.m1.5.5.1.1.2.2.2" xref="S4.E5.m1.5.5.1.1.2.2.2.cmml">∑</mo><mrow id="S4.E5.m1.5.5.1.1.2.2.3" xref="S4.E5.m1.5.5.1.1.2.2.3.cmml"><mi id="S4.E5.m1.5.5.1.1.2.2.3.2" xref="S4.E5.m1.5.5.1.1.2.2.3.2.cmml">i</mi><mo id="S4.E5.m1.5.5.1.1.2.2.3.1" xref="S4.E5.m1.5.5.1.1.2.2.3.1.cmml">=</mo><mn id="S4.E5.m1.5.5.1.1.2.2.3.3" xref="S4.E5.m1.5.5.1.1.2.2.3.3.cmml">1</mn></mrow><mi id="S4.E5.m1.5.5.1.1.2.3" xref="S4.E5.m1.5.5.1.1.2.3.cmml">N</mi></munderover><mrow id="S4.E5.m1.5.5.1.1.1.1" xref="S4.E5.m1.5.5.1.1.1.1.1.cmml"><mo stretchy="false" id="S4.E5.m1.5.5.1.1.1.1.2" xref="S4.E5.m1.5.5.1.1.1.1.1.cmml">(</mo><mrow id="S4.E5.m1.5.5.1.1.1.1.1" xref="S4.E5.m1.5.5.1.1.1.1.1.cmml"><msubsup id="S4.E5.m1.5.5.1.1.1.1.1.2" xref="S4.E5.m1.5.5.1.1.1.1.1.2.cmml"><mi id="S4.E5.m1.5.5.1.1.1.1.1.2.2.2" xref="S4.E5.m1.5.5.1.1.1.1.1.2.2.2.cmml">R</mi><mrow id="S4.E5.m1.2.2.2.4" xref="S4.E5.m1.2.2.2.3.cmml"><mi id="S4.E5.m1.1.1.1.1" xref="S4.E5.m1.1.1.1.1.cmml">t</mi><mo id="S4.E5.m1.2.2.2.4.1" xref="S4.E5.m1.2.2.2.3.cmml">,</mo><mi id="S4.E5.m1.2.2.2.2" xref="S4.E5.m1.2.2.2.2.cmml">i</mi></mrow><mi id="S4.E5.m1.5.5.1.1.1.1.1.2.3" xref="S4.E5.m1.5.5.1.1.1.1.1.2.3.cmml">I</mi></msubsup><mo id="S4.E5.m1.5.5.1.1.1.1.1.1" xref="S4.E5.m1.5.5.1.1.1.1.1.1.cmml">−</mo><msubsup id="S4.E5.m1.5.5.1.1.1.1.1.3" xref="S4.E5.m1.5.5.1.1.1.1.1.3.cmml"><mi id="S4.E5.m1.5.5.1.1.1.1.1.3.2.2" xref="S4.E5.m1.5.5.1.1.1.1.1.3.2.2.cmml">R</mi><mrow id="S4.E5.m1.4.4.2.4" xref="S4.E5.m1.4.4.2.3.cmml"><mn id="S4.E5.m1.3.3.1.1" xref="S4.E5.m1.3.3.1.1.cmml">0</mn><mo id="S4.E5.m1.4.4.2.4.1" xref="S4.E5.m1.4.4.2.3.cmml">,</mo><mi id="S4.E5.m1.4.4.2.2" xref="S4.E5.m1.4.4.2.2.cmml">i</mi></mrow><mi id="S4.E5.m1.5.5.1.1.1.1.1.3.3" xref="S4.E5.m1.5.5.1.1.1.1.1.3.3.cmml">I</mi></msubsup></mrow><mo stretchy="false" id="S4.E5.m1.5.5.1.1.1.1.3" xref="S4.E5.m1.5.5.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.E5.m1.5b"><apply id="S4.E5.m1.5.5.cmml" xref="S4.E5.m1.5.5"><eq id="S4.E5.m1.5.5.2.cmml" xref="S4.E5.m1.5.5.2"></eq><apply id="S4.E5.m1.5.5.3.cmml" xref="S4.E5.m1.5.5.3"><times id="S4.E5.m1.5.5.3.1.cmml" xref="S4.E5.m1.5.5.3.1"></times><ci id="S4.E5.m1.5.5.3.2.cmml" xref="S4.E5.m1.5.5.3.2">Δ</ci><apply id="S4.E5.m1.5.5.3.3.cmml" xref="S4.E5.m1.5.5.3.3"><csymbol cd="ambiguous" id="S4.E5.m1.5.5.3.3.1.cmml" xref="S4.E5.m1.5.5.3.3">superscript</csymbol><apply id="S4.E5.m1.5.5.3.3.2.cmml" xref="S4.E5.m1.5.5.3.3"><csymbol cd="ambiguous" id="S4.E5.m1.5.5.3.3.2.1.cmml" xref="S4.E5.m1.5.5.3.3">subscript</csymbol><ci id="S4.E5.m1.5.5.3.3.2.2.cmml" xref="S4.E5.m1.5.5.3.3.2.2">𝑅</ci><ci id="S4.E5.m1.5.5.3.3.2.3.cmml" xref="S4.E5.m1.5.5.3.3.2.3">𝑡</ci></apply><ci id="S4.E5.m1.5.5.3.3.3.cmml" xref="S4.E5.m1.5.5.3.3.3">𝐼</ci></apply></apply><apply id="S4.E5.m1.5.5.1.cmml" xref="S4.E5.m1.5.5.1"><times id="S4.E5.m1.5.5.1.2.cmml" xref="S4.E5.m1.5.5.1.2"></times><apply id="S4.E5.m1.5.5.1.3.cmml" xref="S4.E5.m1.5.5.1.3"><divide id="S4.E5.m1.5.5.1.3.1.cmml" xref="S4.E5.m1.5.5.1.3"></divide><cn type="integer" id="S4.E5.m1.5.5.1.3.2.cmml" xref="S4.E5.m1.5.5.1.3.2">1</cn><ci id="S4.E5.m1.5.5.1.3.3.cmml" xref="S4.E5.m1.5.5.1.3.3">𝑁</ci></apply><apply id="S4.E5.m1.5.5.1.1.cmml" xref="S4.E5.m1.5.5.1.1"><apply id="S4.E5.m1.5.5.1.1.2.cmml" xref="S4.E5.m1.5.5.1.1.2"><csymbol cd="ambiguous" id="S4.E5.m1.5.5.1.1.2.1.cmml" xref="S4.E5.m1.5.5.1.1.2">superscript</csymbol><apply id="S4.E5.m1.5.5.1.1.2.2.cmml" xref="S4.E5.m1.5.5.1.1.2"><csymbol cd="ambiguous" id="S4.E5.m1.5.5.1.1.2.2.1.cmml" xref="S4.E5.m1.5.5.1.1.2">subscript</csymbol><sum id="S4.E5.m1.5.5.1.1.2.2.2.cmml" xref="S4.E5.m1.5.5.1.1.2.2.2"></sum><apply id="S4.E5.m1.5.5.1.1.2.2.3.cmml" xref="S4.E5.m1.5.5.1.1.2.2.3"><eq id="S4.E5.m1.5.5.1.1.2.2.3.1.cmml" xref="S4.E5.m1.5.5.1.1.2.2.3.1"></eq><ci id="S4.E5.m1.5.5.1.1.2.2.3.2.cmml" xref="S4.E5.m1.5.5.1.1.2.2.3.2">𝑖</ci><cn type="integer" id="S4.E5.m1.5.5.1.1.2.2.3.3.cmml" xref="S4.E5.m1.5.5.1.1.2.2.3.3">1</cn></apply></apply><ci id="S4.E5.m1.5.5.1.1.2.3.cmml" xref="S4.E5.m1.5.5.1.1.2.3">𝑁</ci></apply><apply id="S4.E5.m1.5.5.1.1.1.1.1.cmml" xref="S4.E5.m1.5.5.1.1.1.1"><minus id="S4.E5.m1.5.5.1.1.1.1.1.1.cmml" xref="S4.E5.m1.5.5.1.1.1.1.1.1"></minus><apply id="S4.E5.m1.5.5.1.1.1.1.1.2.cmml" xref="S4.E5.m1.5.5.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S4.E5.m1.5.5.1.1.1.1.1.2.1.cmml" xref="S4.E5.m1.5.5.1.1.1.1.1.2">superscript</csymbol><apply id="S4.E5.m1.5.5.1.1.1.1.1.2.2.cmml" xref="S4.E5.m1.5.5.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S4.E5.m1.5.5.1.1.1.1.1.2.2.1.cmml" xref="S4.E5.m1.5.5.1.1.1.1.1.2">subscript</csymbol><ci id="S4.E5.m1.5.5.1.1.1.1.1.2.2.2.cmml" xref="S4.E5.m1.5.5.1.1.1.1.1.2.2.2">𝑅</ci><list id="S4.E5.m1.2.2.2.3.cmml" xref="S4.E5.m1.2.2.2.4"><ci id="S4.E5.m1.1.1.1.1.cmml" xref="S4.E5.m1.1.1.1.1">𝑡</ci><ci id="S4.E5.m1.2.2.2.2.cmml" xref="S4.E5.m1.2.2.2.2">𝑖</ci></list></apply><ci id="S4.E5.m1.5.5.1.1.1.1.1.2.3.cmml" xref="S4.E5.m1.5.5.1.1.1.1.1.2.3">𝐼</ci></apply><apply id="S4.E5.m1.5.5.1.1.1.1.1.3.cmml" xref="S4.E5.m1.5.5.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S4.E5.m1.5.5.1.1.1.1.1.3.1.cmml" xref="S4.E5.m1.5.5.1.1.1.1.1.3">superscript</csymbol><apply id="S4.E5.m1.5.5.1.1.1.1.1.3.2.cmml" xref="S4.E5.m1.5.5.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S4.E5.m1.5.5.1.1.1.1.1.3.2.1.cmml" xref="S4.E5.m1.5.5.1.1.1.1.1.3">subscript</csymbol><ci id="S4.E5.m1.5.5.1.1.1.1.1.3.2.2.cmml" xref="S4.E5.m1.5.5.1.1.1.1.1.3.2.2">𝑅</ci><list id="S4.E5.m1.4.4.2.3.cmml" xref="S4.E5.m1.4.4.2.4"><cn type="integer" id="S4.E5.m1.3.3.1.1.cmml" xref="S4.E5.m1.3.3.1.1">0</cn><ci id="S4.E5.m1.4.4.2.2.cmml" xref="S4.E5.m1.4.4.2.2">𝑖</ci></list></apply><ci id="S4.E5.m1.5.5.1.1.1.1.1.3.3.cmml" xref="S4.E5.m1.5.5.1.1.1.1.1.3.3">𝐼</ci></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.E5.m1.5c">\Delta R_{t}^{I}=\frac{1}{N}\sum_{i=1}^{N}(R_{t,i}^{I}-R_{0,i}^{I})</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(5)</span></td>
</tr></tbody>
</table>
</div>
<div id="S4.SS2.p4" class="ltx_para ltx_noindent">
<p id="S4.SS2.p4.4" class="ltx_p"><span id="S4.SS2.p4.4.1" class="ltx_text ltx_font_bold">Safety Delta</span> quantifies the change in a model’s response safety after sequential training. Using a set of datasets designed for safety evaluation, denoted as <math id="S4.SS2.p4.1.m1.3" class="ltx_Math" alttext="{S_{1},...,S_{L}}" display="inline"><semantics id="S4.SS2.p4.1.m1.3a"><mrow id="S4.SS2.p4.1.m1.3.3.2" xref="S4.SS2.p4.1.m1.3.3.3.cmml"><msub id="S4.SS2.p4.1.m1.2.2.1.1" xref="S4.SS2.p4.1.m1.2.2.1.1.cmml"><mi id="S4.SS2.p4.1.m1.2.2.1.1.2" xref="S4.SS2.p4.1.m1.2.2.1.1.2.cmml">S</mi><mn id="S4.SS2.p4.1.m1.2.2.1.1.3" xref="S4.SS2.p4.1.m1.2.2.1.1.3.cmml">1</mn></msub><mo id="S4.SS2.p4.1.m1.3.3.2.3" xref="S4.SS2.p4.1.m1.3.3.3.cmml">,</mo><mi mathvariant="normal" id="S4.SS2.p4.1.m1.1.1" xref="S4.SS2.p4.1.m1.1.1.cmml">…</mi><mo id="S4.SS2.p4.1.m1.3.3.2.4" xref="S4.SS2.p4.1.m1.3.3.3.cmml">,</mo><msub id="S4.SS2.p4.1.m1.3.3.2.2" xref="S4.SS2.p4.1.m1.3.3.2.2.cmml"><mi id="S4.SS2.p4.1.m1.3.3.2.2.2" xref="S4.SS2.p4.1.m1.3.3.2.2.2.cmml">S</mi><mi id="S4.SS2.p4.1.m1.3.3.2.2.3" xref="S4.SS2.p4.1.m1.3.3.2.2.3.cmml">L</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="S4.SS2.p4.1.m1.3b"><list id="S4.SS2.p4.1.m1.3.3.3.cmml" xref="S4.SS2.p4.1.m1.3.3.2"><apply id="S4.SS2.p4.1.m1.2.2.1.1.cmml" xref="S4.SS2.p4.1.m1.2.2.1.1"><csymbol cd="ambiguous" id="S4.SS2.p4.1.m1.2.2.1.1.1.cmml" xref="S4.SS2.p4.1.m1.2.2.1.1">subscript</csymbol><ci id="S4.SS2.p4.1.m1.2.2.1.1.2.cmml" xref="S4.SS2.p4.1.m1.2.2.1.1.2">𝑆</ci><cn type="integer" id="S4.SS2.p4.1.m1.2.2.1.1.3.cmml" xref="S4.SS2.p4.1.m1.2.2.1.1.3">1</cn></apply><ci id="S4.SS2.p4.1.m1.1.1.cmml" xref="S4.SS2.p4.1.m1.1.1">…</ci><apply id="S4.SS2.p4.1.m1.3.3.2.2.cmml" xref="S4.SS2.p4.1.m1.3.3.2.2"><csymbol cd="ambiguous" id="S4.SS2.p4.1.m1.3.3.2.2.1.cmml" xref="S4.SS2.p4.1.m1.3.3.2.2">subscript</csymbol><ci id="S4.SS2.p4.1.m1.3.3.2.2.2.cmml" xref="S4.SS2.p4.1.m1.3.3.2.2.2">𝑆</ci><ci id="S4.SS2.p4.1.m1.3.3.2.2.3.cmml" xref="S4.SS2.p4.1.m1.3.3.2.2.3">𝐿</ci></apply></list></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p4.1.m1.3c">{S_{1},...,S_{L}}</annotation></semantics></math>, the initial safety metric on the i-th dataset is <math id="S4.SS2.p4.2.m2.2" class="ltx_Math" alttext="R_{0,i}^{S}" display="inline"><semantics id="S4.SS2.p4.2.m2.2a"><msubsup id="S4.SS2.p4.2.m2.2.3" xref="S4.SS2.p4.2.m2.2.3.cmml"><mi id="S4.SS2.p4.2.m2.2.3.2.2" xref="S4.SS2.p4.2.m2.2.3.2.2.cmml">R</mi><mrow id="S4.SS2.p4.2.m2.2.2.2.4" xref="S4.SS2.p4.2.m2.2.2.2.3.cmml"><mn id="S4.SS2.p4.2.m2.1.1.1.1" xref="S4.SS2.p4.2.m2.1.1.1.1.cmml">0</mn><mo id="S4.SS2.p4.2.m2.2.2.2.4.1" xref="S4.SS2.p4.2.m2.2.2.2.3.cmml">,</mo><mi id="S4.SS2.p4.2.m2.2.2.2.2" xref="S4.SS2.p4.2.m2.2.2.2.2.cmml">i</mi></mrow><mi id="S4.SS2.p4.2.m2.2.3.3" xref="S4.SS2.p4.2.m2.2.3.3.cmml">S</mi></msubsup><annotation-xml encoding="MathML-Content" id="S4.SS2.p4.2.m2.2b"><apply id="S4.SS2.p4.2.m2.2.3.cmml" xref="S4.SS2.p4.2.m2.2.3"><csymbol cd="ambiguous" id="S4.SS2.p4.2.m2.2.3.1.cmml" xref="S4.SS2.p4.2.m2.2.3">superscript</csymbol><apply id="S4.SS2.p4.2.m2.2.3.2.cmml" xref="S4.SS2.p4.2.m2.2.3"><csymbol cd="ambiguous" id="S4.SS2.p4.2.m2.2.3.2.1.cmml" xref="S4.SS2.p4.2.m2.2.3">subscript</csymbol><ci id="S4.SS2.p4.2.m2.2.3.2.2.cmml" xref="S4.SS2.p4.2.m2.2.3.2.2">𝑅</ci><list id="S4.SS2.p4.2.m2.2.2.2.3.cmml" xref="S4.SS2.p4.2.m2.2.2.2.4"><cn type="integer" id="S4.SS2.p4.2.m2.1.1.1.1.cmml" xref="S4.SS2.p4.2.m2.1.1.1.1">0</cn><ci id="S4.SS2.p4.2.m2.2.2.2.2.cmml" xref="S4.SS2.p4.2.m2.2.2.2.2">𝑖</ci></list></apply><ci id="S4.SS2.p4.2.m2.2.3.3.cmml" xref="S4.SS2.p4.2.m2.2.3.3">𝑆</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p4.2.m2.2c">R_{0,i}^{S}</annotation></semantics></math>. After training up to the t-th task, its score on the i-th dataset is <math id="S4.SS2.p4.3.m3.2" class="ltx_Math" alttext="R_{t,i}^{S}" display="inline"><semantics id="S4.SS2.p4.3.m3.2a"><msubsup id="S4.SS2.p4.3.m3.2.3" xref="S4.SS2.p4.3.m3.2.3.cmml"><mi id="S4.SS2.p4.3.m3.2.3.2.2" xref="S4.SS2.p4.3.m3.2.3.2.2.cmml">R</mi><mrow id="S4.SS2.p4.3.m3.2.2.2.4" xref="S4.SS2.p4.3.m3.2.2.2.3.cmml"><mi id="S4.SS2.p4.3.m3.1.1.1.1" xref="S4.SS2.p4.3.m3.1.1.1.1.cmml">t</mi><mo id="S4.SS2.p4.3.m3.2.2.2.4.1" xref="S4.SS2.p4.3.m3.2.2.2.3.cmml">,</mo><mi id="S4.SS2.p4.3.m3.2.2.2.2" xref="S4.SS2.p4.3.m3.2.2.2.2.cmml">i</mi></mrow><mi id="S4.SS2.p4.3.m3.2.3.3" xref="S4.SS2.p4.3.m3.2.3.3.cmml">S</mi></msubsup><annotation-xml encoding="MathML-Content" id="S4.SS2.p4.3.m3.2b"><apply id="S4.SS2.p4.3.m3.2.3.cmml" xref="S4.SS2.p4.3.m3.2.3"><csymbol cd="ambiguous" id="S4.SS2.p4.3.m3.2.3.1.cmml" xref="S4.SS2.p4.3.m3.2.3">superscript</csymbol><apply id="S4.SS2.p4.3.m3.2.3.2.cmml" xref="S4.SS2.p4.3.m3.2.3"><csymbol cd="ambiguous" id="S4.SS2.p4.3.m3.2.3.2.1.cmml" xref="S4.SS2.p4.3.m3.2.3">subscript</csymbol><ci id="S4.SS2.p4.3.m3.2.3.2.2.cmml" xref="S4.SS2.p4.3.m3.2.3.2.2">𝑅</ci><list id="S4.SS2.p4.3.m3.2.2.2.3.cmml" xref="S4.SS2.p4.3.m3.2.2.2.4"><ci id="S4.SS2.p4.3.m3.1.1.1.1.cmml" xref="S4.SS2.p4.3.m3.1.1.1.1">𝑡</ci><ci id="S4.SS2.p4.3.m3.2.2.2.2.cmml" xref="S4.SS2.p4.3.m3.2.2.2.2">𝑖</ci></list></apply><ci id="S4.SS2.p4.3.m3.2.3.3.cmml" xref="S4.SS2.p4.3.m3.2.3.3">𝑆</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p4.3.m3.2c">R_{t,i}^{S}</annotation></semantics></math>. The change after the t-th task, represented by <math id="S4.SS2.p4.4.m4.1" class="ltx_Math" alttext="\Delta R_{t}^{S}" display="inline"><semantics id="S4.SS2.p4.4.m4.1a"><mrow id="S4.SS2.p4.4.m4.1.1" xref="S4.SS2.p4.4.m4.1.1.cmml"><mi mathvariant="normal" id="S4.SS2.p4.4.m4.1.1.2" xref="S4.SS2.p4.4.m4.1.1.2.cmml">Δ</mi><mo lspace="0em" rspace="0em" id="S4.SS2.p4.4.m4.1.1.1" xref="S4.SS2.p4.4.m4.1.1.1.cmml">​</mo><msubsup id="S4.SS2.p4.4.m4.1.1.3" xref="S4.SS2.p4.4.m4.1.1.3.cmml"><mi id="S4.SS2.p4.4.m4.1.1.3.2.2" xref="S4.SS2.p4.4.m4.1.1.3.2.2.cmml">R</mi><mi id="S4.SS2.p4.4.m4.1.1.3.2.3" xref="S4.SS2.p4.4.m4.1.1.3.2.3.cmml">t</mi><mi id="S4.SS2.p4.4.m4.1.1.3.3" xref="S4.SS2.p4.4.m4.1.1.3.3.cmml">S</mi></msubsup></mrow><annotation-xml encoding="MathML-Content" id="S4.SS2.p4.4.m4.1b"><apply id="S4.SS2.p4.4.m4.1.1.cmml" xref="S4.SS2.p4.4.m4.1.1"><times id="S4.SS2.p4.4.m4.1.1.1.cmml" xref="S4.SS2.p4.4.m4.1.1.1"></times><ci id="S4.SS2.p4.4.m4.1.1.2.cmml" xref="S4.SS2.p4.4.m4.1.1.2">Δ</ci><apply id="S4.SS2.p4.4.m4.1.1.3.cmml" xref="S4.SS2.p4.4.m4.1.1.3"><csymbol cd="ambiguous" id="S4.SS2.p4.4.m4.1.1.3.1.cmml" xref="S4.SS2.p4.4.m4.1.1.3">superscript</csymbol><apply id="S4.SS2.p4.4.m4.1.1.3.2.cmml" xref="S4.SS2.p4.4.m4.1.1.3"><csymbol cd="ambiguous" id="S4.SS2.p4.4.m4.1.1.3.2.1.cmml" xref="S4.SS2.p4.4.m4.1.1.3">subscript</csymbol><ci id="S4.SS2.p4.4.m4.1.1.3.2.2.cmml" xref="S4.SS2.p4.4.m4.1.1.3.2.2">𝑅</ci><ci id="S4.SS2.p4.4.m4.1.1.3.2.3.cmml" xref="S4.SS2.p4.4.m4.1.1.3.2.3">𝑡</ci></apply><ci id="S4.SS2.p4.4.m4.1.1.3.3.cmml" xref="S4.SS2.p4.4.m4.1.1.3.3">𝑆</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p4.4.m4.1c">\Delta R_{t}^{S}</annotation></semantics></math>, is computed as:</p>
<table id="S4.E6" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S4.E6.m1.5" class="ltx_Math" alttext="\Delta R_{t}^{S}=\frac{1}{L}\sum_{i=1}^{L}(R_{t,i}^{S}-R_{0,i}^{S})" display="block"><semantics id="S4.E6.m1.5a"><mrow id="S4.E6.m1.5.5" xref="S4.E6.m1.5.5.cmml"><mrow id="S4.E6.m1.5.5.3" xref="S4.E6.m1.5.5.3.cmml"><mi mathvariant="normal" id="S4.E6.m1.5.5.3.2" xref="S4.E6.m1.5.5.3.2.cmml">Δ</mi><mo lspace="0em" rspace="0em" id="S4.E6.m1.5.5.3.1" xref="S4.E6.m1.5.5.3.1.cmml">​</mo><msubsup id="S4.E6.m1.5.5.3.3" xref="S4.E6.m1.5.5.3.3.cmml"><mi id="S4.E6.m1.5.5.3.3.2.2" xref="S4.E6.m1.5.5.3.3.2.2.cmml">R</mi><mi id="S4.E6.m1.5.5.3.3.2.3" xref="S4.E6.m1.5.5.3.3.2.3.cmml">t</mi><mi id="S4.E6.m1.5.5.3.3.3" xref="S4.E6.m1.5.5.3.3.3.cmml">S</mi></msubsup></mrow><mo id="S4.E6.m1.5.5.2" xref="S4.E6.m1.5.5.2.cmml">=</mo><mrow id="S4.E6.m1.5.5.1" xref="S4.E6.m1.5.5.1.cmml"><mfrac id="S4.E6.m1.5.5.1.3" xref="S4.E6.m1.5.5.1.3.cmml"><mn id="S4.E6.m1.5.5.1.3.2" xref="S4.E6.m1.5.5.1.3.2.cmml">1</mn><mi id="S4.E6.m1.5.5.1.3.3" xref="S4.E6.m1.5.5.1.3.3.cmml">L</mi></mfrac><mo lspace="0em" rspace="0em" id="S4.E6.m1.5.5.1.2" xref="S4.E6.m1.5.5.1.2.cmml">​</mo><mrow id="S4.E6.m1.5.5.1.1" xref="S4.E6.m1.5.5.1.1.cmml"><munderover id="S4.E6.m1.5.5.1.1.2" xref="S4.E6.m1.5.5.1.1.2.cmml"><mo movablelimits="false" rspace="0em" id="S4.E6.m1.5.5.1.1.2.2.2" xref="S4.E6.m1.5.5.1.1.2.2.2.cmml">∑</mo><mrow id="S4.E6.m1.5.5.1.1.2.2.3" xref="S4.E6.m1.5.5.1.1.2.2.3.cmml"><mi id="S4.E6.m1.5.5.1.1.2.2.3.2" xref="S4.E6.m1.5.5.1.1.2.2.3.2.cmml">i</mi><mo id="S4.E6.m1.5.5.1.1.2.2.3.1" xref="S4.E6.m1.5.5.1.1.2.2.3.1.cmml">=</mo><mn id="S4.E6.m1.5.5.1.1.2.2.3.3" xref="S4.E6.m1.5.5.1.1.2.2.3.3.cmml">1</mn></mrow><mi id="S4.E6.m1.5.5.1.1.2.3" xref="S4.E6.m1.5.5.1.1.2.3.cmml">L</mi></munderover><mrow id="S4.E6.m1.5.5.1.1.1.1" xref="S4.E6.m1.5.5.1.1.1.1.1.cmml"><mo stretchy="false" id="S4.E6.m1.5.5.1.1.1.1.2" xref="S4.E6.m1.5.5.1.1.1.1.1.cmml">(</mo><mrow id="S4.E6.m1.5.5.1.1.1.1.1" xref="S4.E6.m1.5.5.1.1.1.1.1.cmml"><msubsup id="S4.E6.m1.5.5.1.1.1.1.1.2" xref="S4.E6.m1.5.5.1.1.1.1.1.2.cmml"><mi id="S4.E6.m1.5.5.1.1.1.1.1.2.2.2" xref="S4.E6.m1.5.5.1.1.1.1.1.2.2.2.cmml">R</mi><mrow id="S4.E6.m1.2.2.2.4" xref="S4.E6.m1.2.2.2.3.cmml"><mi id="S4.E6.m1.1.1.1.1" xref="S4.E6.m1.1.1.1.1.cmml">t</mi><mo id="S4.E6.m1.2.2.2.4.1" xref="S4.E6.m1.2.2.2.3.cmml">,</mo><mi id="S4.E6.m1.2.2.2.2" xref="S4.E6.m1.2.2.2.2.cmml">i</mi></mrow><mi id="S4.E6.m1.5.5.1.1.1.1.1.2.3" xref="S4.E6.m1.5.5.1.1.1.1.1.2.3.cmml">S</mi></msubsup><mo id="S4.E6.m1.5.5.1.1.1.1.1.1" xref="S4.E6.m1.5.5.1.1.1.1.1.1.cmml">−</mo><msubsup id="S4.E6.m1.5.5.1.1.1.1.1.3" xref="S4.E6.m1.5.5.1.1.1.1.1.3.cmml"><mi id="S4.E6.m1.5.5.1.1.1.1.1.3.2.2" xref="S4.E6.m1.5.5.1.1.1.1.1.3.2.2.cmml">R</mi><mrow id="S4.E6.m1.4.4.2.4" xref="S4.E6.m1.4.4.2.3.cmml"><mn id="S4.E6.m1.3.3.1.1" xref="S4.E6.m1.3.3.1.1.cmml">0</mn><mo id="S4.E6.m1.4.4.2.4.1" xref="S4.E6.m1.4.4.2.3.cmml">,</mo><mi id="S4.E6.m1.4.4.2.2" xref="S4.E6.m1.4.4.2.2.cmml">i</mi></mrow><mi id="S4.E6.m1.5.5.1.1.1.1.1.3.3" xref="S4.E6.m1.5.5.1.1.1.1.1.3.3.cmml">S</mi></msubsup></mrow><mo stretchy="false" id="S4.E6.m1.5.5.1.1.1.1.3" xref="S4.E6.m1.5.5.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.E6.m1.5b"><apply id="S4.E6.m1.5.5.cmml" xref="S4.E6.m1.5.5"><eq id="S4.E6.m1.5.5.2.cmml" xref="S4.E6.m1.5.5.2"></eq><apply id="S4.E6.m1.5.5.3.cmml" xref="S4.E6.m1.5.5.3"><times id="S4.E6.m1.5.5.3.1.cmml" xref="S4.E6.m1.5.5.3.1"></times><ci id="S4.E6.m1.5.5.3.2.cmml" xref="S4.E6.m1.5.5.3.2">Δ</ci><apply id="S4.E6.m1.5.5.3.3.cmml" xref="S4.E6.m1.5.5.3.3"><csymbol cd="ambiguous" id="S4.E6.m1.5.5.3.3.1.cmml" xref="S4.E6.m1.5.5.3.3">superscript</csymbol><apply id="S4.E6.m1.5.5.3.3.2.cmml" xref="S4.E6.m1.5.5.3.3"><csymbol cd="ambiguous" id="S4.E6.m1.5.5.3.3.2.1.cmml" xref="S4.E6.m1.5.5.3.3">subscript</csymbol><ci id="S4.E6.m1.5.5.3.3.2.2.cmml" xref="S4.E6.m1.5.5.3.3.2.2">𝑅</ci><ci id="S4.E6.m1.5.5.3.3.2.3.cmml" xref="S4.E6.m1.5.5.3.3.2.3">𝑡</ci></apply><ci id="S4.E6.m1.5.5.3.3.3.cmml" xref="S4.E6.m1.5.5.3.3.3">𝑆</ci></apply></apply><apply id="S4.E6.m1.5.5.1.cmml" xref="S4.E6.m1.5.5.1"><times id="S4.E6.m1.5.5.1.2.cmml" xref="S4.E6.m1.5.5.1.2"></times><apply id="S4.E6.m1.5.5.1.3.cmml" xref="S4.E6.m1.5.5.1.3"><divide id="S4.E6.m1.5.5.1.3.1.cmml" xref="S4.E6.m1.5.5.1.3"></divide><cn type="integer" id="S4.E6.m1.5.5.1.3.2.cmml" xref="S4.E6.m1.5.5.1.3.2">1</cn><ci id="S4.E6.m1.5.5.1.3.3.cmml" xref="S4.E6.m1.5.5.1.3.3">𝐿</ci></apply><apply id="S4.E6.m1.5.5.1.1.cmml" xref="S4.E6.m1.5.5.1.1"><apply id="S4.E6.m1.5.5.1.1.2.cmml" xref="S4.E6.m1.5.5.1.1.2"><csymbol cd="ambiguous" id="S4.E6.m1.5.5.1.1.2.1.cmml" xref="S4.E6.m1.5.5.1.1.2">superscript</csymbol><apply id="S4.E6.m1.5.5.1.1.2.2.cmml" xref="S4.E6.m1.5.5.1.1.2"><csymbol cd="ambiguous" id="S4.E6.m1.5.5.1.1.2.2.1.cmml" xref="S4.E6.m1.5.5.1.1.2">subscript</csymbol><sum id="S4.E6.m1.5.5.1.1.2.2.2.cmml" xref="S4.E6.m1.5.5.1.1.2.2.2"></sum><apply id="S4.E6.m1.5.5.1.1.2.2.3.cmml" xref="S4.E6.m1.5.5.1.1.2.2.3"><eq id="S4.E6.m1.5.5.1.1.2.2.3.1.cmml" xref="S4.E6.m1.5.5.1.1.2.2.3.1"></eq><ci id="S4.E6.m1.5.5.1.1.2.2.3.2.cmml" xref="S4.E6.m1.5.5.1.1.2.2.3.2">𝑖</ci><cn type="integer" id="S4.E6.m1.5.5.1.1.2.2.3.3.cmml" xref="S4.E6.m1.5.5.1.1.2.2.3.3">1</cn></apply></apply><ci id="S4.E6.m1.5.5.1.1.2.3.cmml" xref="S4.E6.m1.5.5.1.1.2.3">𝐿</ci></apply><apply id="S4.E6.m1.5.5.1.1.1.1.1.cmml" xref="S4.E6.m1.5.5.1.1.1.1"><minus id="S4.E6.m1.5.5.1.1.1.1.1.1.cmml" xref="S4.E6.m1.5.5.1.1.1.1.1.1"></minus><apply id="S4.E6.m1.5.5.1.1.1.1.1.2.cmml" xref="S4.E6.m1.5.5.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S4.E6.m1.5.5.1.1.1.1.1.2.1.cmml" xref="S4.E6.m1.5.5.1.1.1.1.1.2">superscript</csymbol><apply id="S4.E6.m1.5.5.1.1.1.1.1.2.2.cmml" xref="S4.E6.m1.5.5.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S4.E6.m1.5.5.1.1.1.1.1.2.2.1.cmml" xref="S4.E6.m1.5.5.1.1.1.1.1.2">subscript</csymbol><ci id="S4.E6.m1.5.5.1.1.1.1.1.2.2.2.cmml" xref="S4.E6.m1.5.5.1.1.1.1.1.2.2.2">𝑅</ci><list id="S4.E6.m1.2.2.2.3.cmml" xref="S4.E6.m1.2.2.2.4"><ci id="S4.E6.m1.1.1.1.1.cmml" xref="S4.E6.m1.1.1.1.1">𝑡</ci><ci id="S4.E6.m1.2.2.2.2.cmml" xref="S4.E6.m1.2.2.2.2">𝑖</ci></list></apply><ci id="S4.E6.m1.5.5.1.1.1.1.1.2.3.cmml" xref="S4.E6.m1.5.5.1.1.1.1.1.2.3">𝑆</ci></apply><apply id="S4.E6.m1.5.5.1.1.1.1.1.3.cmml" xref="S4.E6.m1.5.5.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S4.E6.m1.5.5.1.1.1.1.1.3.1.cmml" xref="S4.E6.m1.5.5.1.1.1.1.1.3">superscript</csymbol><apply id="S4.E6.m1.5.5.1.1.1.1.1.3.2.cmml" xref="S4.E6.m1.5.5.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S4.E6.m1.5.5.1.1.1.1.1.3.2.1.cmml" xref="S4.E6.m1.5.5.1.1.1.1.1.3">subscript</csymbol><ci id="S4.E6.m1.5.5.1.1.1.1.1.3.2.2.cmml" xref="S4.E6.m1.5.5.1.1.1.1.1.3.2.2">𝑅</ci><list id="S4.E6.m1.4.4.2.3.cmml" xref="S4.E6.m1.4.4.2.4"><cn type="integer" id="S4.E6.m1.3.3.1.1.cmml" xref="S4.E6.m1.3.3.1.1">0</cn><ci id="S4.E6.m1.4.4.2.2.cmml" xref="S4.E6.m1.4.4.2.2">𝑖</ci></list></apply><ci id="S4.E6.m1.5.5.1.1.1.1.1.3.3.cmml" xref="S4.E6.m1.5.5.1.1.1.1.1.3.3">𝑆</ci></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.E6.m1.5c">\Delta R_{t}^{S}=\frac{1}{L}\sum_{i=1}^{L}(R_{t,i}^{S}-R_{0,i}^{S})</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(6)</span></td>
</tr></tbody>
</table>
</div>
</section>
<section id="S4.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.3 </span>Experimental Setup</h3>

<section id="S4.SS3.SSS1" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">4.3.1 </span>Baselines</h4>

<div id="S4.SS3.SSS1.p1" class="ltx_para ltx_noindent">
<p id="S4.SS3.SSS1.p1.1" class="ltx_p">We evaluate the performance of LLMs in a continual learning setting using four approaches—three requiring training and one not:</p>
</div>
<div id="S4.SS3.SSS1.p2" class="ltx_para ltx_noindent">
<p id="S4.SS3.SSS1.p2.1" class="ltx_p"><span id="S4.SS3.SSS1.p2.1.1" class="ltx_text ltx_font_bold">Sequential Full-Parameter Fine-Tuning (SeqFT)</span>: This method involves training all model parameters in sequence.</p>
</div>
<div id="S4.SS3.SSS1.p3" class="ltx_para ltx_noindent">
<p id="S4.SS3.SSS1.p3.1" class="ltx_p"><span id="S4.SS3.SSS1.p3.1.1" class="ltx_text ltx_font_bold">LoRA-based Sequential Fine-Tuning (LoraSeqFT)</span>: Only the low-rank LoRA matrices are fine-tuned, leaving the LLM backbone fixed <cite class="ltx_cite ltx_citemacro_citep">(Hu et al., <a href="#bib.bib19" title="" class="ltx_ref">2021</a>)</cite>. This method is chosen based on prior findings of reduced forgetting with ”Efficient Tuning” <cite class="ltx_cite ltx_citemacro_citep">(Liu &amp; Huang, <a href="#bib.bib26" title="" class="ltx_ref">2023</a>)</cite>.</p>
</div>
<div id="S4.SS3.SSS1.p4" class="ltx_para ltx_noindent">
<p id="S4.SS3.SSS1.p4.1" class="ltx_p"><span id="S4.SS3.SSS1.p4.1.1" class="ltx_text ltx_font_bold">Replay-based Sequential Fine-Tuning (Replay)</span>: Replay, a common continual learning strategy, is employed for its simplicity and effectiveness. We incorporate alignment data from LIMA into the replay memory, replaying 10% of historical data.</p>
</div>
<div id="S4.SS3.SSS1.p5" class="ltx_para ltx_noindent">
<p id="S4.SS3.SSS1.p5.1" class="ltx_p"><span id="S4.SS3.SSS1.p5.1.1" class="ltx_text ltx_font_bold">In-Context Learning (ICL)</span>: Task demonstrations are supplied as part of the language prompt, acting as a form of prompt engineering <cite class="ltx_cite ltx_citemacro_citep">(Brown et al., <a href="#bib.bib5" title="" class="ltx_ref">2020</a>)</cite>. A 6-shot setting is used for our experiments.</p>
</div>
<div id="S4.SS3.SSS1.p6" class="ltx_para ltx_noindent">
<p id="S4.SS3.SSS1.p6.1" class="ltx_p">To evaluate the resilience of safety alignment models from diverse training backgrounds and strategies, we select five aligned models from three organizations: Meta: LLaMa-2-7B-Chat, LLaMa-2-13B-Chat <cite class="ltx_cite ltx_citemacro_citep">(Touvron et al., <a href="#bib.bib44" title="" class="ltx_ref">2023</a>)</cite>, BaiChuan: Baichuan 2-7B-Chat <cite class="ltx_cite ltx_citemacro_citep">(Baichuan, <a href="#bib.bib2" title="" class="ltx_ref">2023</a>)</cite>, and Large Model Systems Organization: Vicuna-13B-V1.5, Vicuna-7B-V1.5 <cite class="ltx_cite ltx_citemacro_citep">(Chiang et al., <a href="#bib.bib7" title="" class="ltx_ref">2023</a>)</cite>.</p>
</div>
</section>
<section id="S4.SS3.SSS2" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">4.3.2 </span>Datasets</h4>

<div id="S4.SS3.SSS2.p1" class="ltx_para ltx_noindent">
<p id="S4.SS3.SSS2.p1.1" class="ltx_p">To evaluate a model’s <span id="S4.SS3.SSS2.p1.1.1" class="ltx_text ltx_font_italic">general ability</span>, we assess across five key dimensions: Factual Knowledge, General Reasoning, Multilinguality, Commonsense Reasoning, and Reading Comprehension.</p>
</div>
<div id="S4.SS3.SSS2.p2" class="ltx_para ltx_noindent">
<p id="S4.SS3.SSS2.p2.1" class="ltx_p"><span id="S4.SS3.SSS2.p2.1.1" class="ltx_text ltx_font_bold">Factual Knowledge</span>: We use the Massive Multitask Language Understanding dataset (MMLU <cite class="ltx_cite ltx_citemacro_citep">(Hendrycks et al., <a href="#bib.bib17" title="" class="ltx_ref">2020</a>)</cite>) with questions on 57 subjects, from elementary to professional levels. Following LLaMa-2 <cite class="ltx_cite ltx_citemacro_citep">(Touvron et al., <a href="#bib.bib44" title="" class="ltx_ref">2023</a>)</cite>, we report 5-shot accuracy based on answer perplexity.</p>
</div>
<div id="S4.SS3.SSS2.p3" class="ltx_para ltx_noindent">
<p id="S4.SS3.SSS2.p3.1" class="ltx_p"><span id="S4.SS3.SSS2.p3.1.1" class="ltx_text ltx_font_bold">General Reasoning</span>: Evaluated using Big-Bench-Hard (BBH <cite class="ltx_cite ltx_citemacro_citep">(Suzgun et al., <a href="#bib.bib43" title="" class="ltx_ref">2022</a>)</cite>) with 23 tasks from Big-Bench <cite class="ltx_cite ltx_citemacro_citep">(Ghazal et al., <a href="#bib.bib16" title="" class="ltx_ref">2013</a>)</cite>. The evaluation uses chain-of-thought prompts with 3-shot in-context examples, and EM scores are reported.</p>
</div>
<div id="S4.SS3.SSS2.p4" class="ltx_para ltx_noindent">
<p id="S4.SS3.SSS2.p4.1" class="ltx_p"><span id="S4.SS3.SSS2.p4.1.1" class="ltx_text ltx_font_bold">Multilinguality</span>: We use TyDiQA <cite class="ltx_cite ltx_citemacro_citep">(Clark et al., <a href="#bib.bib10" title="" class="ltx_ref">2020</a>)</cite>, a multilingual QA benchmark with 11 languages. Using the gold-passage setup, 0-shot F1 scores are reported.</p>
</div>
<div id="S4.SS3.SSS2.p5" class="ltx_para ltx_noindent">
<p id="S4.SS3.SSS2.p5.1" class="ltx_p"><span id="S4.SS3.SSS2.p5.1.1" class="ltx_text ltx_font_bold">Commonsense Reasoning</span>: Evaluated using PIQA <cite class="ltx_cite ltx_citemacro_citep">(Bisk et al., <a href="#bib.bib4" title="" class="ltx_ref">2020</a>)</cite>. Following LLaMa-2 <cite class="ltx_cite ltx_citemacro_citep">(Touvron et al., <a href="#bib.bib44" title="" class="ltx_ref">2023</a>)</cite>, we report 0-shot accuracy based on answer perplexity.</p>
</div>
<div id="S4.SS3.SSS2.p6" class="ltx_para ltx_noindent">
<p id="S4.SS3.SSS2.p6.1" class="ltx_p"><span id="S4.SS3.SSS2.p6.1.1" class="ltx_text ltx_font_bold">Reading Comprehension</span>: Assessed with BoolQ <cite class="ltx_cite ltx_citemacro_citep">(Clark et al., <a href="#bib.bib9" title="" class="ltx_ref">2019</a>)</cite>, containing 15942 questions. We report 0-shot accuracy based on answer perplexity.</p>
</div>
<div id="S4.SS3.SSS2.p7" class="ltx_para ltx_noindent">
<p id="S4.SS3.SSS2.p7.1" class="ltx_p">For <span id="S4.SS3.SSS2.p7.1.1" class="ltx_text ltx_font_italic">instruction-following capability</span>, we use Self-instruct dataset <cite class="ltx_cite ltx_citemacro_citep">(Wang et al., <a href="#bib.bib48" title="" class="ltx_ref">2022</a>)</cite>, which is user-oriented and comprises a diverse set of 175 prompts, encompassing areas like email writing, social media, productivity tools, entertainment, and programming. We also employ LIMA dataset <cite class="ltx_cite ltx_citemacro_citep">(Zhou et al., <a href="#bib.bib56" title="" class="ltx_ref">2023</a>)</cite> LIMA assembles 300 prompts, primarily sourced from community Q&amp;A forums and supplemented by manually authored examples.</p>
</div>
<div id="S4.SS3.SSS2.p8" class="ltx_para ltx_noindent">
<p id="S4.SS3.SSS2.p8.1" class="ltx_p">For assessing changes in <span id="S4.SS3.SSS2.p8.1.1" class="ltx_text ltx_font_italic">safety</span>, we leverage the CoNa dataset <cite class="ltx_cite ltx_citemacro_citep">(Bianchi et al., <a href="#bib.bib3" title="" class="ltx_ref">2023</a>)</cite>. This corpus encompasses 178 expert-annotated samples, specifically curated to address instructions associated with hateful speech generation.</p>
</div>
<figure id="S4.T1" class="ltx_table">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 1: </span>OP(BWT) for all the baseline models and 3 baseline methods.</figcaption>
<table id="S4.T1.20" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="S4.T1.20.21" class="ltx_tr">
<td id="S4.T1.20.21.1" class="ltx_td ltx_border_r ltx_border_t"></td>
<td id="S4.T1.20.21.2" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">ICL</td>
<td id="S4.T1.20.21.3" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">SeqFT</td>
<td id="S4.T1.20.21.4" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">LoraSeqFT</td>
<td id="S4.T1.20.21.5" class="ltx_td ltx_align_center ltx_border_t">Replay</td>
</tr>
<tr id="S4.T1.4.4" class="ltx_tr">
<td id="S4.T1.4.4.5" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">LLaMA-2-7B-Chat</td>
<td id="S4.T1.1.1.1" class="ltx_td ltx_align_center ltx_border_r ltx_border_t"><math id="S4.T1.1.1.1.m1.1" class="ltx_Math" alttext="38.9" display="inline"><semantics id="S4.T1.1.1.1.m1.1a"><mn id="S4.T1.1.1.1.m1.1.1" xref="S4.T1.1.1.1.m1.1.1.cmml">38.9</mn><annotation-xml encoding="MathML-Content" id="S4.T1.1.1.1.m1.1b"><cn type="float" id="S4.T1.1.1.1.m1.1.1.cmml" xref="S4.T1.1.1.1.m1.1.1">38.9</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.1.1.1.m1.1c">38.9</annotation></semantics></math></td>
<td id="S4.T1.2.2.2" class="ltx_td ltx_align_center ltx_border_r ltx_border_t"><math id="S4.T1.2.2.2.m1.1" class="ltx_Math" alttext="48.7(-8.3\%)" display="inline"><semantics id="S4.T1.2.2.2.m1.1a"><mrow id="S4.T1.2.2.2.m1.1.1" xref="S4.T1.2.2.2.m1.1.1.cmml"><mn id="S4.T1.2.2.2.m1.1.1.3" xref="S4.T1.2.2.2.m1.1.1.3.cmml">48.7</mn><mo lspace="0em" rspace="0em" id="S4.T1.2.2.2.m1.1.1.2" xref="S4.T1.2.2.2.m1.1.1.2.cmml">​</mo><mrow id="S4.T1.2.2.2.m1.1.1.1.1" xref="S4.T1.2.2.2.m1.1.1.1.1.1.cmml"><mo stretchy="false" id="S4.T1.2.2.2.m1.1.1.1.1.2" xref="S4.T1.2.2.2.m1.1.1.1.1.1.cmml">(</mo><mrow id="S4.T1.2.2.2.m1.1.1.1.1.1" xref="S4.T1.2.2.2.m1.1.1.1.1.1.cmml"><mo id="S4.T1.2.2.2.m1.1.1.1.1.1a" xref="S4.T1.2.2.2.m1.1.1.1.1.1.cmml">−</mo><mrow id="S4.T1.2.2.2.m1.1.1.1.1.1.2" xref="S4.T1.2.2.2.m1.1.1.1.1.1.2.cmml"><mn id="S4.T1.2.2.2.m1.1.1.1.1.1.2.2" xref="S4.T1.2.2.2.m1.1.1.1.1.1.2.2.cmml">8.3</mn><mo id="S4.T1.2.2.2.m1.1.1.1.1.1.2.1" xref="S4.T1.2.2.2.m1.1.1.1.1.1.2.1.cmml">%</mo></mrow></mrow><mo stretchy="false" id="S4.T1.2.2.2.m1.1.1.1.1.3" xref="S4.T1.2.2.2.m1.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.T1.2.2.2.m1.1b"><apply id="S4.T1.2.2.2.m1.1.1.cmml" xref="S4.T1.2.2.2.m1.1.1"><times id="S4.T1.2.2.2.m1.1.1.2.cmml" xref="S4.T1.2.2.2.m1.1.1.2"></times><cn type="float" id="S4.T1.2.2.2.m1.1.1.3.cmml" xref="S4.T1.2.2.2.m1.1.1.3">48.7</cn><apply id="S4.T1.2.2.2.m1.1.1.1.1.1.cmml" xref="S4.T1.2.2.2.m1.1.1.1.1"><minus id="S4.T1.2.2.2.m1.1.1.1.1.1.1.cmml" xref="S4.T1.2.2.2.m1.1.1.1.1"></minus><apply id="S4.T1.2.2.2.m1.1.1.1.1.1.2.cmml" xref="S4.T1.2.2.2.m1.1.1.1.1.1.2"><csymbol cd="latexml" id="S4.T1.2.2.2.m1.1.1.1.1.1.2.1.cmml" xref="S4.T1.2.2.2.m1.1.1.1.1.1.2.1">percent</csymbol><cn type="float" id="S4.T1.2.2.2.m1.1.1.1.1.1.2.2.cmml" xref="S4.T1.2.2.2.m1.1.1.1.1.1.2.2">8.3</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.2.2.2.m1.1c">48.7(-8.3\%)</annotation></semantics></math></td>
<td id="S4.T1.3.3.3" class="ltx_td ltx_align_center ltx_border_r ltx_border_t"><math id="S4.T1.3.3.3.m1.1" class="ltx_Math" alttext="12.7(-45.7\%)" display="inline"><semantics id="S4.T1.3.3.3.m1.1a"><mrow id="S4.T1.3.3.3.m1.1.1" xref="S4.T1.3.3.3.m1.1.1.cmml"><mn id="S4.T1.3.3.3.m1.1.1.3" xref="S4.T1.3.3.3.m1.1.1.3.cmml">12.7</mn><mo lspace="0em" rspace="0em" id="S4.T1.3.3.3.m1.1.1.2" xref="S4.T1.3.3.3.m1.1.1.2.cmml">​</mo><mrow id="S4.T1.3.3.3.m1.1.1.1.1" xref="S4.T1.3.3.3.m1.1.1.1.1.1.cmml"><mo stretchy="false" id="S4.T1.3.3.3.m1.1.1.1.1.2" xref="S4.T1.3.3.3.m1.1.1.1.1.1.cmml">(</mo><mrow id="S4.T1.3.3.3.m1.1.1.1.1.1" xref="S4.T1.3.3.3.m1.1.1.1.1.1.cmml"><mo id="S4.T1.3.3.3.m1.1.1.1.1.1a" xref="S4.T1.3.3.3.m1.1.1.1.1.1.cmml">−</mo><mrow id="S4.T1.3.3.3.m1.1.1.1.1.1.2" xref="S4.T1.3.3.3.m1.1.1.1.1.1.2.cmml"><mn id="S4.T1.3.3.3.m1.1.1.1.1.1.2.2" xref="S4.T1.3.3.3.m1.1.1.1.1.1.2.2.cmml">45.7</mn><mo id="S4.T1.3.3.3.m1.1.1.1.1.1.2.1" xref="S4.T1.3.3.3.m1.1.1.1.1.1.2.1.cmml">%</mo></mrow></mrow><mo stretchy="false" id="S4.T1.3.3.3.m1.1.1.1.1.3" xref="S4.T1.3.3.3.m1.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.T1.3.3.3.m1.1b"><apply id="S4.T1.3.3.3.m1.1.1.cmml" xref="S4.T1.3.3.3.m1.1.1"><times id="S4.T1.3.3.3.m1.1.1.2.cmml" xref="S4.T1.3.3.3.m1.1.1.2"></times><cn type="float" id="S4.T1.3.3.3.m1.1.1.3.cmml" xref="S4.T1.3.3.3.m1.1.1.3">12.7</cn><apply id="S4.T1.3.3.3.m1.1.1.1.1.1.cmml" xref="S4.T1.3.3.3.m1.1.1.1.1"><minus id="S4.T1.3.3.3.m1.1.1.1.1.1.1.cmml" xref="S4.T1.3.3.3.m1.1.1.1.1"></minus><apply id="S4.T1.3.3.3.m1.1.1.1.1.1.2.cmml" xref="S4.T1.3.3.3.m1.1.1.1.1.1.2"><csymbol cd="latexml" id="S4.T1.3.3.3.m1.1.1.1.1.1.2.1.cmml" xref="S4.T1.3.3.3.m1.1.1.1.1.1.2.1">percent</csymbol><cn type="float" id="S4.T1.3.3.3.m1.1.1.1.1.1.2.2.cmml" xref="S4.T1.3.3.3.m1.1.1.1.1.1.2.2">45.7</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.3.3.3.m1.1c">12.7(-45.7\%)</annotation></semantics></math></td>
<td id="S4.T1.4.4.4" class="ltx_td ltx_align_center ltx_border_t"><math id="S4.T1.4.4.4.m1.1" class="ltx_Math" alttext="55.5(2.6\%)" display="inline"><semantics id="S4.T1.4.4.4.m1.1a"><mrow id="S4.T1.4.4.4.m1.1.1" xref="S4.T1.4.4.4.m1.1.1.cmml"><mn id="S4.T1.4.4.4.m1.1.1.3" xref="S4.T1.4.4.4.m1.1.1.3.cmml">55.5</mn><mo lspace="0em" rspace="0em" id="S4.T1.4.4.4.m1.1.1.2" xref="S4.T1.4.4.4.m1.1.1.2.cmml">​</mo><mrow id="S4.T1.4.4.4.m1.1.1.1.1" xref="S4.T1.4.4.4.m1.1.1.1.1.1.cmml"><mo stretchy="false" id="S4.T1.4.4.4.m1.1.1.1.1.2" xref="S4.T1.4.4.4.m1.1.1.1.1.1.cmml">(</mo><mrow id="S4.T1.4.4.4.m1.1.1.1.1.1" xref="S4.T1.4.4.4.m1.1.1.1.1.1.cmml"><mn id="S4.T1.4.4.4.m1.1.1.1.1.1.2" xref="S4.T1.4.4.4.m1.1.1.1.1.1.2.cmml">2.6</mn><mo id="S4.T1.4.4.4.m1.1.1.1.1.1.1" xref="S4.T1.4.4.4.m1.1.1.1.1.1.1.cmml">%</mo></mrow><mo stretchy="false" id="S4.T1.4.4.4.m1.1.1.1.1.3" xref="S4.T1.4.4.4.m1.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.T1.4.4.4.m1.1b"><apply id="S4.T1.4.4.4.m1.1.1.cmml" xref="S4.T1.4.4.4.m1.1.1"><times id="S4.T1.4.4.4.m1.1.1.2.cmml" xref="S4.T1.4.4.4.m1.1.1.2"></times><cn type="float" id="S4.T1.4.4.4.m1.1.1.3.cmml" xref="S4.T1.4.4.4.m1.1.1.3">55.5</cn><apply id="S4.T1.4.4.4.m1.1.1.1.1.1.cmml" xref="S4.T1.4.4.4.m1.1.1.1.1"><csymbol cd="latexml" id="S4.T1.4.4.4.m1.1.1.1.1.1.1.cmml" xref="S4.T1.4.4.4.m1.1.1.1.1.1.1">percent</csymbol><cn type="float" id="S4.T1.4.4.4.m1.1.1.1.1.1.2.cmml" xref="S4.T1.4.4.4.m1.1.1.1.1.1.2">2.6</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.4.4.4.m1.1c">55.5(2.6\%)</annotation></semantics></math></td>
</tr>
<tr id="S4.T1.8.8" class="ltx_tr">
<td id="S4.T1.8.8.5" class="ltx_td ltx_align_center ltx_border_r">LLaMA-2-13B-Chat</td>
<td id="S4.T1.5.5.1" class="ltx_td ltx_align_center ltx_border_r"><math id="S4.T1.5.5.1.m1.1" class="ltx_Math" alttext="41.9" display="inline"><semantics id="S4.T1.5.5.1.m1.1a"><mn id="S4.T1.5.5.1.m1.1.1" xref="S4.T1.5.5.1.m1.1.1.cmml">41.9</mn><annotation-xml encoding="MathML-Content" id="S4.T1.5.5.1.m1.1b"><cn type="float" id="S4.T1.5.5.1.m1.1.1.cmml" xref="S4.T1.5.5.1.m1.1.1">41.9</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.5.5.1.m1.1c">41.9</annotation></semantics></math></td>
<td id="S4.T1.6.6.2" class="ltx_td ltx_align_center ltx_border_r"><math id="S4.T1.6.6.2.m1.1" class="ltx_Math" alttext="49.9(-7.0\%)" display="inline"><semantics id="S4.T1.6.6.2.m1.1a"><mrow id="S4.T1.6.6.2.m1.1.1" xref="S4.T1.6.6.2.m1.1.1.cmml"><mn id="S4.T1.6.6.2.m1.1.1.3" xref="S4.T1.6.6.2.m1.1.1.3.cmml">49.9</mn><mo lspace="0em" rspace="0em" id="S4.T1.6.6.2.m1.1.1.2" xref="S4.T1.6.6.2.m1.1.1.2.cmml">​</mo><mrow id="S4.T1.6.6.2.m1.1.1.1.1" xref="S4.T1.6.6.2.m1.1.1.1.1.1.cmml"><mo stretchy="false" id="S4.T1.6.6.2.m1.1.1.1.1.2" xref="S4.T1.6.6.2.m1.1.1.1.1.1.cmml">(</mo><mrow id="S4.T1.6.6.2.m1.1.1.1.1.1" xref="S4.T1.6.6.2.m1.1.1.1.1.1.cmml"><mo id="S4.T1.6.6.2.m1.1.1.1.1.1a" xref="S4.T1.6.6.2.m1.1.1.1.1.1.cmml">−</mo><mrow id="S4.T1.6.6.2.m1.1.1.1.1.1.2" xref="S4.T1.6.6.2.m1.1.1.1.1.1.2.cmml"><mn id="S4.T1.6.6.2.m1.1.1.1.1.1.2.2" xref="S4.T1.6.6.2.m1.1.1.1.1.1.2.2.cmml">7.0</mn><mo id="S4.T1.6.6.2.m1.1.1.1.1.1.2.1" xref="S4.T1.6.6.2.m1.1.1.1.1.1.2.1.cmml">%</mo></mrow></mrow><mo stretchy="false" id="S4.T1.6.6.2.m1.1.1.1.1.3" xref="S4.T1.6.6.2.m1.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.T1.6.6.2.m1.1b"><apply id="S4.T1.6.6.2.m1.1.1.cmml" xref="S4.T1.6.6.2.m1.1.1"><times id="S4.T1.6.6.2.m1.1.1.2.cmml" xref="S4.T1.6.6.2.m1.1.1.2"></times><cn type="float" id="S4.T1.6.6.2.m1.1.1.3.cmml" xref="S4.T1.6.6.2.m1.1.1.3">49.9</cn><apply id="S4.T1.6.6.2.m1.1.1.1.1.1.cmml" xref="S4.T1.6.6.2.m1.1.1.1.1"><minus id="S4.T1.6.6.2.m1.1.1.1.1.1.1.cmml" xref="S4.T1.6.6.2.m1.1.1.1.1"></minus><apply id="S4.T1.6.6.2.m1.1.1.1.1.1.2.cmml" xref="S4.T1.6.6.2.m1.1.1.1.1.1.2"><csymbol cd="latexml" id="S4.T1.6.6.2.m1.1.1.1.1.1.2.1.cmml" xref="S4.T1.6.6.2.m1.1.1.1.1.1.2.1">percent</csymbol><cn type="float" id="S4.T1.6.6.2.m1.1.1.1.1.1.2.2.cmml" xref="S4.T1.6.6.2.m1.1.1.1.1.1.2.2">7.0</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.6.6.2.m1.1c">49.9(-7.0\%)</annotation></semantics></math></td>
<td id="S4.T1.7.7.3" class="ltx_td ltx_align_center ltx_border_r"><math id="S4.T1.7.7.3.m1.1" class="ltx_Math" alttext="28.0(-36.5\%)" display="inline"><semantics id="S4.T1.7.7.3.m1.1a"><mrow id="S4.T1.7.7.3.m1.1.1" xref="S4.T1.7.7.3.m1.1.1.cmml"><mn id="S4.T1.7.7.3.m1.1.1.3" xref="S4.T1.7.7.3.m1.1.1.3.cmml">28.0</mn><mo lspace="0em" rspace="0em" id="S4.T1.7.7.3.m1.1.1.2" xref="S4.T1.7.7.3.m1.1.1.2.cmml">​</mo><mrow id="S4.T1.7.7.3.m1.1.1.1.1" xref="S4.T1.7.7.3.m1.1.1.1.1.1.cmml"><mo stretchy="false" id="S4.T1.7.7.3.m1.1.1.1.1.2" xref="S4.T1.7.7.3.m1.1.1.1.1.1.cmml">(</mo><mrow id="S4.T1.7.7.3.m1.1.1.1.1.1" xref="S4.T1.7.7.3.m1.1.1.1.1.1.cmml"><mo id="S4.T1.7.7.3.m1.1.1.1.1.1a" xref="S4.T1.7.7.3.m1.1.1.1.1.1.cmml">−</mo><mrow id="S4.T1.7.7.3.m1.1.1.1.1.1.2" xref="S4.T1.7.7.3.m1.1.1.1.1.1.2.cmml"><mn id="S4.T1.7.7.3.m1.1.1.1.1.1.2.2" xref="S4.T1.7.7.3.m1.1.1.1.1.1.2.2.cmml">36.5</mn><mo id="S4.T1.7.7.3.m1.1.1.1.1.1.2.1" xref="S4.T1.7.7.3.m1.1.1.1.1.1.2.1.cmml">%</mo></mrow></mrow><mo stretchy="false" id="S4.T1.7.7.3.m1.1.1.1.1.3" xref="S4.T1.7.7.3.m1.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.T1.7.7.3.m1.1b"><apply id="S4.T1.7.7.3.m1.1.1.cmml" xref="S4.T1.7.7.3.m1.1.1"><times id="S4.T1.7.7.3.m1.1.1.2.cmml" xref="S4.T1.7.7.3.m1.1.1.2"></times><cn type="float" id="S4.T1.7.7.3.m1.1.1.3.cmml" xref="S4.T1.7.7.3.m1.1.1.3">28.0</cn><apply id="S4.T1.7.7.3.m1.1.1.1.1.1.cmml" xref="S4.T1.7.7.3.m1.1.1.1.1"><minus id="S4.T1.7.7.3.m1.1.1.1.1.1.1.cmml" xref="S4.T1.7.7.3.m1.1.1.1.1"></minus><apply id="S4.T1.7.7.3.m1.1.1.1.1.1.2.cmml" xref="S4.T1.7.7.3.m1.1.1.1.1.1.2"><csymbol cd="latexml" id="S4.T1.7.7.3.m1.1.1.1.1.1.2.1.cmml" xref="S4.T1.7.7.3.m1.1.1.1.1.1.2.1">percent</csymbol><cn type="float" id="S4.T1.7.7.3.m1.1.1.1.1.1.2.2.cmml" xref="S4.T1.7.7.3.m1.1.1.1.1.1.2.2">36.5</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.7.7.3.m1.1c">28.0(-36.5\%)</annotation></semantics></math></td>
<td id="S4.T1.8.8.4" class="ltx_td ltx_align_center"><math id="S4.T1.8.8.4.m1.1" class="ltx_Math" alttext="56.6(0.4\%)" display="inline"><semantics id="S4.T1.8.8.4.m1.1a"><mrow id="S4.T1.8.8.4.m1.1.1" xref="S4.T1.8.8.4.m1.1.1.cmml"><mn id="S4.T1.8.8.4.m1.1.1.3" xref="S4.T1.8.8.4.m1.1.1.3.cmml">56.6</mn><mo lspace="0em" rspace="0em" id="S4.T1.8.8.4.m1.1.1.2" xref="S4.T1.8.8.4.m1.1.1.2.cmml">​</mo><mrow id="S4.T1.8.8.4.m1.1.1.1.1" xref="S4.T1.8.8.4.m1.1.1.1.1.1.cmml"><mo stretchy="false" id="S4.T1.8.8.4.m1.1.1.1.1.2" xref="S4.T1.8.8.4.m1.1.1.1.1.1.cmml">(</mo><mrow id="S4.T1.8.8.4.m1.1.1.1.1.1" xref="S4.T1.8.8.4.m1.1.1.1.1.1.cmml"><mn id="S4.T1.8.8.4.m1.1.1.1.1.1.2" xref="S4.T1.8.8.4.m1.1.1.1.1.1.2.cmml">0.4</mn><mo id="S4.T1.8.8.4.m1.1.1.1.1.1.1" xref="S4.T1.8.8.4.m1.1.1.1.1.1.1.cmml">%</mo></mrow><mo stretchy="false" id="S4.T1.8.8.4.m1.1.1.1.1.3" xref="S4.T1.8.8.4.m1.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.T1.8.8.4.m1.1b"><apply id="S4.T1.8.8.4.m1.1.1.cmml" xref="S4.T1.8.8.4.m1.1.1"><times id="S4.T1.8.8.4.m1.1.1.2.cmml" xref="S4.T1.8.8.4.m1.1.1.2"></times><cn type="float" id="S4.T1.8.8.4.m1.1.1.3.cmml" xref="S4.T1.8.8.4.m1.1.1.3">56.6</cn><apply id="S4.T1.8.8.4.m1.1.1.1.1.1.cmml" xref="S4.T1.8.8.4.m1.1.1.1.1"><csymbol cd="latexml" id="S4.T1.8.8.4.m1.1.1.1.1.1.1.cmml" xref="S4.T1.8.8.4.m1.1.1.1.1.1.1">percent</csymbol><cn type="float" id="S4.T1.8.8.4.m1.1.1.1.1.1.2.cmml" xref="S4.T1.8.8.4.m1.1.1.1.1.1.2">0.4</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.8.8.4.m1.1c">56.6(0.4\%)</annotation></semantics></math></td>
</tr>
<tr id="S4.T1.12.12" class="ltx_tr">
<td id="S4.T1.12.12.5" class="ltx_td ltx_align_center ltx_border_r">Vicuna-7B-V1.5</td>
<td id="S4.T1.9.9.1" class="ltx_td ltx_align_center ltx_border_r"><math id="S4.T1.9.9.1.m1.1" class="ltx_Math" alttext="42.2" display="inline"><semantics id="S4.T1.9.9.1.m1.1a"><mn id="S4.T1.9.9.1.m1.1.1" xref="S4.T1.9.9.1.m1.1.1.cmml">42.2</mn><annotation-xml encoding="MathML-Content" id="S4.T1.9.9.1.m1.1b"><cn type="float" id="S4.T1.9.9.1.m1.1.1.cmml" xref="S4.T1.9.9.1.m1.1.1">42.2</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.9.9.1.m1.1c">42.2</annotation></semantics></math></td>
<td id="S4.T1.10.10.2" class="ltx_td ltx_align_center ltx_border_r"><math id="S4.T1.10.10.2.m1.1" class="ltx_Math" alttext="49.2(-8.4\%)" display="inline"><semantics id="S4.T1.10.10.2.m1.1a"><mrow id="S4.T1.10.10.2.m1.1.1" xref="S4.T1.10.10.2.m1.1.1.cmml"><mn id="S4.T1.10.10.2.m1.1.1.3" xref="S4.T1.10.10.2.m1.1.1.3.cmml">49.2</mn><mo lspace="0em" rspace="0em" id="S4.T1.10.10.2.m1.1.1.2" xref="S4.T1.10.10.2.m1.1.1.2.cmml">​</mo><mrow id="S4.T1.10.10.2.m1.1.1.1.1" xref="S4.T1.10.10.2.m1.1.1.1.1.1.cmml"><mo stretchy="false" id="S4.T1.10.10.2.m1.1.1.1.1.2" xref="S4.T1.10.10.2.m1.1.1.1.1.1.cmml">(</mo><mrow id="S4.T1.10.10.2.m1.1.1.1.1.1" xref="S4.T1.10.10.2.m1.1.1.1.1.1.cmml"><mo id="S4.T1.10.10.2.m1.1.1.1.1.1a" xref="S4.T1.10.10.2.m1.1.1.1.1.1.cmml">−</mo><mrow id="S4.T1.10.10.2.m1.1.1.1.1.1.2" xref="S4.T1.10.10.2.m1.1.1.1.1.1.2.cmml"><mn id="S4.T1.10.10.2.m1.1.1.1.1.1.2.2" xref="S4.T1.10.10.2.m1.1.1.1.1.1.2.2.cmml">8.4</mn><mo id="S4.T1.10.10.2.m1.1.1.1.1.1.2.1" xref="S4.T1.10.10.2.m1.1.1.1.1.1.2.1.cmml">%</mo></mrow></mrow><mo stretchy="false" id="S4.T1.10.10.2.m1.1.1.1.1.3" xref="S4.T1.10.10.2.m1.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.T1.10.10.2.m1.1b"><apply id="S4.T1.10.10.2.m1.1.1.cmml" xref="S4.T1.10.10.2.m1.1.1"><times id="S4.T1.10.10.2.m1.1.1.2.cmml" xref="S4.T1.10.10.2.m1.1.1.2"></times><cn type="float" id="S4.T1.10.10.2.m1.1.1.3.cmml" xref="S4.T1.10.10.2.m1.1.1.3">49.2</cn><apply id="S4.T1.10.10.2.m1.1.1.1.1.1.cmml" xref="S4.T1.10.10.2.m1.1.1.1.1"><minus id="S4.T1.10.10.2.m1.1.1.1.1.1.1.cmml" xref="S4.T1.10.10.2.m1.1.1.1.1"></minus><apply id="S4.T1.10.10.2.m1.1.1.1.1.1.2.cmml" xref="S4.T1.10.10.2.m1.1.1.1.1.1.2"><csymbol cd="latexml" id="S4.T1.10.10.2.m1.1.1.1.1.1.2.1.cmml" xref="S4.T1.10.10.2.m1.1.1.1.1.1.2.1">percent</csymbol><cn type="float" id="S4.T1.10.10.2.m1.1.1.1.1.1.2.2.cmml" xref="S4.T1.10.10.2.m1.1.1.1.1.1.2.2">8.4</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.10.10.2.m1.1c">49.2(-8.4\%)</annotation></semantics></math></td>
<td id="S4.T1.11.11.3" class="ltx_td ltx_align_center ltx_border_r"><math id="S4.T1.11.11.3.m1.1" class="ltx_Math" alttext="33.4(-23.7\%)" display="inline"><semantics id="S4.T1.11.11.3.m1.1a"><mrow id="S4.T1.11.11.3.m1.1.1" xref="S4.T1.11.11.3.m1.1.1.cmml"><mn id="S4.T1.11.11.3.m1.1.1.3" xref="S4.T1.11.11.3.m1.1.1.3.cmml">33.4</mn><mo lspace="0em" rspace="0em" id="S4.T1.11.11.3.m1.1.1.2" xref="S4.T1.11.11.3.m1.1.1.2.cmml">​</mo><mrow id="S4.T1.11.11.3.m1.1.1.1.1" xref="S4.T1.11.11.3.m1.1.1.1.1.1.cmml"><mo stretchy="false" id="S4.T1.11.11.3.m1.1.1.1.1.2" xref="S4.T1.11.11.3.m1.1.1.1.1.1.cmml">(</mo><mrow id="S4.T1.11.11.3.m1.1.1.1.1.1" xref="S4.T1.11.11.3.m1.1.1.1.1.1.cmml"><mo id="S4.T1.11.11.3.m1.1.1.1.1.1a" xref="S4.T1.11.11.3.m1.1.1.1.1.1.cmml">−</mo><mrow id="S4.T1.11.11.3.m1.1.1.1.1.1.2" xref="S4.T1.11.11.3.m1.1.1.1.1.1.2.cmml"><mn id="S4.T1.11.11.3.m1.1.1.1.1.1.2.2" xref="S4.T1.11.11.3.m1.1.1.1.1.1.2.2.cmml">23.7</mn><mo id="S4.T1.11.11.3.m1.1.1.1.1.1.2.1" xref="S4.T1.11.11.3.m1.1.1.1.1.1.2.1.cmml">%</mo></mrow></mrow><mo stretchy="false" id="S4.T1.11.11.3.m1.1.1.1.1.3" xref="S4.T1.11.11.3.m1.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.T1.11.11.3.m1.1b"><apply id="S4.T1.11.11.3.m1.1.1.cmml" xref="S4.T1.11.11.3.m1.1.1"><times id="S4.T1.11.11.3.m1.1.1.2.cmml" xref="S4.T1.11.11.3.m1.1.1.2"></times><cn type="float" id="S4.T1.11.11.3.m1.1.1.3.cmml" xref="S4.T1.11.11.3.m1.1.1.3">33.4</cn><apply id="S4.T1.11.11.3.m1.1.1.1.1.1.cmml" xref="S4.T1.11.11.3.m1.1.1.1.1"><minus id="S4.T1.11.11.3.m1.1.1.1.1.1.1.cmml" xref="S4.T1.11.11.3.m1.1.1.1.1"></minus><apply id="S4.T1.11.11.3.m1.1.1.1.1.1.2.cmml" xref="S4.T1.11.11.3.m1.1.1.1.1.1.2"><csymbol cd="latexml" id="S4.T1.11.11.3.m1.1.1.1.1.1.2.1.cmml" xref="S4.T1.11.11.3.m1.1.1.1.1.1.2.1">percent</csymbol><cn type="float" id="S4.T1.11.11.3.m1.1.1.1.1.1.2.2.cmml" xref="S4.T1.11.11.3.m1.1.1.1.1.1.2.2">23.7</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.11.11.3.m1.1c">33.4(-23.7\%)</annotation></semantics></math></td>
<td id="S4.T1.12.12.4" class="ltx_td ltx_align_center"><math id="S4.T1.12.12.4.m1.1" class="ltx_Math" alttext="55.3(0.2\%)" display="inline"><semantics id="S4.T1.12.12.4.m1.1a"><mrow id="S4.T1.12.12.4.m1.1.1" xref="S4.T1.12.12.4.m1.1.1.cmml"><mn id="S4.T1.12.12.4.m1.1.1.3" xref="S4.T1.12.12.4.m1.1.1.3.cmml">55.3</mn><mo lspace="0em" rspace="0em" id="S4.T1.12.12.4.m1.1.1.2" xref="S4.T1.12.12.4.m1.1.1.2.cmml">​</mo><mrow id="S4.T1.12.12.4.m1.1.1.1.1" xref="S4.T1.12.12.4.m1.1.1.1.1.1.cmml"><mo stretchy="false" id="S4.T1.12.12.4.m1.1.1.1.1.2" xref="S4.T1.12.12.4.m1.1.1.1.1.1.cmml">(</mo><mrow id="S4.T1.12.12.4.m1.1.1.1.1.1" xref="S4.T1.12.12.4.m1.1.1.1.1.1.cmml"><mn id="S4.T1.12.12.4.m1.1.1.1.1.1.2" xref="S4.T1.12.12.4.m1.1.1.1.1.1.2.cmml">0.2</mn><mo id="S4.T1.12.12.4.m1.1.1.1.1.1.1" xref="S4.T1.12.12.4.m1.1.1.1.1.1.1.cmml">%</mo></mrow><mo stretchy="false" id="S4.T1.12.12.4.m1.1.1.1.1.3" xref="S4.T1.12.12.4.m1.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.T1.12.12.4.m1.1b"><apply id="S4.T1.12.12.4.m1.1.1.cmml" xref="S4.T1.12.12.4.m1.1.1"><times id="S4.T1.12.12.4.m1.1.1.2.cmml" xref="S4.T1.12.12.4.m1.1.1.2"></times><cn type="float" id="S4.T1.12.12.4.m1.1.1.3.cmml" xref="S4.T1.12.12.4.m1.1.1.3">55.3</cn><apply id="S4.T1.12.12.4.m1.1.1.1.1.1.cmml" xref="S4.T1.12.12.4.m1.1.1.1.1"><csymbol cd="latexml" id="S4.T1.12.12.4.m1.1.1.1.1.1.1.cmml" xref="S4.T1.12.12.4.m1.1.1.1.1.1.1">percent</csymbol><cn type="float" id="S4.T1.12.12.4.m1.1.1.1.1.1.2.cmml" xref="S4.T1.12.12.4.m1.1.1.1.1.1.2">0.2</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.12.12.4.m1.1c">55.3(0.2\%)</annotation></semantics></math></td>
</tr>
<tr id="S4.T1.16.16" class="ltx_tr">
<td id="S4.T1.16.16.5" class="ltx_td ltx_align_center ltx_border_r">Vicuna-13B-V1.5</td>
<td id="S4.T1.13.13.1" class="ltx_td ltx_align_center ltx_border_r"><math id="S4.T1.13.13.1.m1.1" class="ltx_Math" alttext="46.9" display="inline"><semantics id="S4.T1.13.13.1.m1.1a"><mn id="S4.T1.13.13.1.m1.1.1" xref="S4.T1.13.13.1.m1.1.1.cmml">46.9</mn><annotation-xml encoding="MathML-Content" id="S4.T1.13.13.1.m1.1b"><cn type="float" id="S4.T1.13.13.1.m1.1.1.cmml" xref="S4.T1.13.13.1.m1.1.1">46.9</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.13.13.1.m1.1c">46.9</annotation></semantics></math></td>
<td id="S4.T1.14.14.2" class="ltx_td ltx_align_center ltx_border_r"><math id="S4.T1.14.14.2.m1.1" class="ltx_Math" alttext="51.7(-5.9\%)" display="inline"><semantics id="S4.T1.14.14.2.m1.1a"><mrow id="S4.T1.14.14.2.m1.1.1" xref="S4.T1.14.14.2.m1.1.1.cmml"><mn id="S4.T1.14.14.2.m1.1.1.3" xref="S4.T1.14.14.2.m1.1.1.3.cmml">51.7</mn><mo lspace="0em" rspace="0em" id="S4.T1.14.14.2.m1.1.1.2" xref="S4.T1.14.14.2.m1.1.1.2.cmml">​</mo><mrow id="S4.T1.14.14.2.m1.1.1.1.1" xref="S4.T1.14.14.2.m1.1.1.1.1.1.cmml"><mo stretchy="false" id="S4.T1.14.14.2.m1.1.1.1.1.2" xref="S4.T1.14.14.2.m1.1.1.1.1.1.cmml">(</mo><mrow id="S4.T1.14.14.2.m1.1.1.1.1.1" xref="S4.T1.14.14.2.m1.1.1.1.1.1.cmml"><mo id="S4.T1.14.14.2.m1.1.1.1.1.1a" xref="S4.T1.14.14.2.m1.1.1.1.1.1.cmml">−</mo><mrow id="S4.T1.14.14.2.m1.1.1.1.1.1.2" xref="S4.T1.14.14.2.m1.1.1.1.1.1.2.cmml"><mn id="S4.T1.14.14.2.m1.1.1.1.1.1.2.2" xref="S4.T1.14.14.2.m1.1.1.1.1.1.2.2.cmml">5.9</mn><mo id="S4.T1.14.14.2.m1.1.1.1.1.1.2.1" xref="S4.T1.14.14.2.m1.1.1.1.1.1.2.1.cmml">%</mo></mrow></mrow><mo stretchy="false" id="S4.T1.14.14.2.m1.1.1.1.1.3" xref="S4.T1.14.14.2.m1.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.T1.14.14.2.m1.1b"><apply id="S4.T1.14.14.2.m1.1.1.cmml" xref="S4.T1.14.14.2.m1.1.1"><times id="S4.T1.14.14.2.m1.1.1.2.cmml" xref="S4.T1.14.14.2.m1.1.1.2"></times><cn type="float" id="S4.T1.14.14.2.m1.1.1.3.cmml" xref="S4.T1.14.14.2.m1.1.1.3">51.7</cn><apply id="S4.T1.14.14.2.m1.1.1.1.1.1.cmml" xref="S4.T1.14.14.2.m1.1.1.1.1"><minus id="S4.T1.14.14.2.m1.1.1.1.1.1.1.cmml" xref="S4.T1.14.14.2.m1.1.1.1.1"></minus><apply id="S4.T1.14.14.2.m1.1.1.1.1.1.2.cmml" xref="S4.T1.14.14.2.m1.1.1.1.1.1.2"><csymbol cd="latexml" id="S4.T1.14.14.2.m1.1.1.1.1.1.2.1.cmml" xref="S4.T1.14.14.2.m1.1.1.1.1.1.2.1">percent</csymbol><cn type="float" id="S4.T1.14.14.2.m1.1.1.1.1.1.2.2.cmml" xref="S4.T1.14.14.2.m1.1.1.1.1.1.2.2">5.9</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.14.14.2.m1.1c">51.7(-5.9\%)</annotation></semantics></math></td>
<td id="S4.T1.15.15.3" class="ltx_td ltx_align_center ltx_border_r"><math id="S4.T1.15.15.3.m1.1" class="ltx_Math" alttext="31.6(-28.4\%)" display="inline"><semantics id="S4.T1.15.15.3.m1.1a"><mrow id="S4.T1.15.15.3.m1.1.1" xref="S4.T1.15.15.3.m1.1.1.cmml"><mn id="S4.T1.15.15.3.m1.1.1.3" xref="S4.T1.15.15.3.m1.1.1.3.cmml">31.6</mn><mo lspace="0em" rspace="0em" id="S4.T1.15.15.3.m1.1.1.2" xref="S4.T1.15.15.3.m1.1.1.2.cmml">​</mo><mrow id="S4.T1.15.15.3.m1.1.1.1.1" xref="S4.T1.15.15.3.m1.1.1.1.1.1.cmml"><mo stretchy="false" id="S4.T1.15.15.3.m1.1.1.1.1.2" xref="S4.T1.15.15.3.m1.1.1.1.1.1.cmml">(</mo><mrow id="S4.T1.15.15.3.m1.1.1.1.1.1" xref="S4.T1.15.15.3.m1.1.1.1.1.1.cmml"><mo id="S4.T1.15.15.3.m1.1.1.1.1.1a" xref="S4.T1.15.15.3.m1.1.1.1.1.1.cmml">−</mo><mrow id="S4.T1.15.15.3.m1.1.1.1.1.1.2" xref="S4.T1.15.15.3.m1.1.1.1.1.1.2.cmml"><mn id="S4.T1.15.15.3.m1.1.1.1.1.1.2.2" xref="S4.T1.15.15.3.m1.1.1.1.1.1.2.2.cmml">28.4</mn><mo id="S4.T1.15.15.3.m1.1.1.1.1.1.2.1" xref="S4.T1.15.15.3.m1.1.1.1.1.1.2.1.cmml">%</mo></mrow></mrow><mo stretchy="false" id="S4.T1.15.15.3.m1.1.1.1.1.3" xref="S4.T1.15.15.3.m1.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.T1.15.15.3.m1.1b"><apply id="S4.T1.15.15.3.m1.1.1.cmml" xref="S4.T1.15.15.3.m1.1.1"><times id="S4.T1.15.15.3.m1.1.1.2.cmml" xref="S4.T1.15.15.3.m1.1.1.2"></times><cn type="float" id="S4.T1.15.15.3.m1.1.1.3.cmml" xref="S4.T1.15.15.3.m1.1.1.3">31.6</cn><apply id="S4.T1.15.15.3.m1.1.1.1.1.1.cmml" xref="S4.T1.15.15.3.m1.1.1.1.1"><minus id="S4.T1.15.15.3.m1.1.1.1.1.1.1.cmml" xref="S4.T1.15.15.3.m1.1.1.1.1"></minus><apply id="S4.T1.15.15.3.m1.1.1.1.1.1.2.cmml" xref="S4.T1.15.15.3.m1.1.1.1.1.1.2"><csymbol cd="latexml" id="S4.T1.15.15.3.m1.1.1.1.1.1.2.1.cmml" xref="S4.T1.15.15.3.m1.1.1.1.1.1.2.1">percent</csymbol><cn type="float" id="S4.T1.15.15.3.m1.1.1.1.1.1.2.2.cmml" xref="S4.T1.15.15.3.m1.1.1.1.1.1.2.2">28.4</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.15.15.3.m1.1c">31.6(-28.4\%)</annotation></semantics></math></td>
<td id="S4.T1.16.16.4" class="ltx_td ltx_align_center"><math id="S4.T1.16.16.4.m1.1" class="ltx_Math" alttext="56.9(0.6\%)" display="inline"><semantics id="S4.T1.16.16.4.m1.1a"><mrow id="S4.T1.16.16.4.m1.1.1" xref="S4.T1.16.16.4.m1.1.1.cmml"><mn id="S4.T1.16.16.4.m1.1.1.3" xref="S4.T1.16.16.4.m1.1.1.3.cmml">56.9</mn><mo lspace="0em" rspace="0em" id="S4.T1.16.16.4.m1.1.1.2" xref="S4.T1.16.16.4.m1.1.1.2.cmml">​</mo><mrow id="S4.T1.16.16.4.m1.1.1.1.1" xref="S4.T1.16.16.4.m1.1.1.1.1.1.cmml"><mo stretchy="false" id="S4.T1.16.16.4.m1.1.1.1.1.2" xref="S4.T1.16.16.4.m1.1.1.1.1.1.cmml">(</mo><mrow id="S4.T1.16.16.4.m1.1.1.1.1.1" xref="S4.T1.16.16.4.m1.1.1.1.1.1.cmml"><mn id="S4.T1.16.16.4.m1.1.1.1.1.1.2" xref="S4.T1.16.16.4.m1.1.1.1.1.1.2.cmml">0.6</mn><mo id="S4.T1.16.16.4.m1.1.1.1.1.1.1" xref="S4.T1.16.16.4.m1.1.1.1.1.1.1.cmml">%</mo></mrow><mo stretchy="false" id="S4.T1.16.16.4.m1.1.1.1.1.3" xref="S4.T1.16.16.4.m1.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.T1.16.16.4.m1.1b"><apply id="S4.T1.16.16.4.m1.1.1.cmml" xref="S4.T1.16.16.4.m1.1.1"><times id="S4.T1.16.16.4.m1.1.1.2.cmml" xref="S4.T1.16.16.4.m1.1.1.2"></times><cn type="float" id="S4.T1.16.16.4.m1.1.1.3.cmml" xref="S4.T1.16.16.4.m1.1.1.3">56.9</cn><apply id="S4.T1.16.16.4.m1.1.1.1.1.1.cmml" xref="S4.T1.16.16.4.m1.1.1.1.1"><csymbol cd="latexml" id="S4.T1.16.16.4.m1.1.1.1.1.1.1.cmml" xref="S4.T1.16.16.4.m1.1.1.1.1.1.1">percent</csymbol><cn type="float" id="S4.T1.16.16.4.m1.1.1.1.1.1.2.cmml" xref="S4.T1.16.16.4.m1.1.1.1.1.1.2">0.6</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.16.16.4.m1.1c">56.9(0.6\%)</annotation></semantics></math></td>
</tr>
<tr id="S4.T1.20.20" class="ltx_tr">
<td id="S4.T1.20.20.5" class="ltx_td ltx_align_center ltx_border_b ltx_border_r">Baichuan2-7B-Instruct</td>
<td id="S4.T1.17.17.1" class="ltx_td ltx_align_center ltx_border_b ltx_border_r"><math id="S4.T1.17.17.1.m1.1" class="ltx_Math" alttext="44.6" display="inline"><semantics id="S4.T1.17.17.1.m1.1a"><mn id="S4.T1.17.17.1.m1.1.1" xref="S4.T1.17.17.1.m1.1.1.cmml">44.6</mn><annotation-xml encoding="MathML-Content" id="S4.T1.17.17.1.m1.1b"><cn type="float" id="S4.T1.17.17.1.m1.1.1.cmml" xref="S4.T1.17.17.1.m1.1.1">44.6</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.17.17.1.m1.1c">44.6</annotation></semantics></math></td>
<td id="S4.T1.18.18.2" class="ltx_td ltx_align_center ltx_border_b ltx_border_r"><math id="S4.T1.18.18.2.m1.1" class="ltx_Math" alttext="43.4(-15.4\%)" display="inline"><semantics id="S4.T1.18.18.2.m1.1a"><mrow id="S4.T1.18.18.2.m1.1.1" xref="S4.T1.18.18.2.m1.1.1.cmml"><mn id="S4.T1.18.18.2.m1.1.1.3" xref="S4.T1.18.18.2.m1.1.1.3.cmml">43.4</mn><mo lspace="0em" rspace="0em" id="S4.T1.18.18.2.m1.1.1.2" xref="S4.T1.18.18.2.m1.1.1.2.cmml">​</mo><mrow id="S4.T1.18.18.2.m1.1.1.1.1" xref="S4.T1.18.18.2.m1.1.1.1.1.1.cmml"><mo stretchy="false" id="S4.T1.18.18.2.m1.1.1.1.1.2" xref="S4.T1.18.18.2.m1.1.1.1.1.1.cmml">(</mo><mrow id="S4.T1.18.18.2.m1.1.1.1.1.1" xref="S4.T1.18.18.2.m1.1.1.1.1.1.cmml"><mo id="S4.T1.18.18.2.m1.1.1.1.1.1a" xref="S4.T1.18.18.2.m1.1.1.1.1.1.cmml">−</mo><mrow id="S4.T1.18.18.2.m1.1.1.1.1.1.2" xref="S4.T1.18.18.2.m1.1.1.1.1.1.2.cmml"><mn id="S4.T1.18.18.2.m1.1.1.1.1.1.2.2" xref="S4.T1.18.18.2.m1.1.1.1.1.1.2.2.cmml">15.4</mn><mo id="S4.T1.18.18.2.m1.1.1.1.1.1.2.1" xref="S4.T1.18.18.2.m1.1.1.1.1.1.2.1.cmml">%</mo></mrow></mrow><mo stretchy="false" id="S4.T1.18.18.2.m1.1.1.1.1.3" xref="S4.T1.18.18.2.m1.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.T1.18.18.2.m1.1b"><apply id="S4.T1.18.18.2.m1.1.1.cmml" xref="S4.T1.18.18.2.m1.1.1"><times id="S4.T1.18.18.2.m1.1.1.2.cmml" xref="S4.T1.18.18.2.m1.1.1.2"></times><cn type="float" id="S4.T1.18.18.2.m1.1.1.3.cmml" xref="S4.T1.18.18.2.m1.1.1.3">43.4</cn><apply id="S4.T1.18.18.2.m1.1.1.1.1.1.cmml" xref="S4.T1.18.18.2.m1.1.1.1.1"><minus id="S4.T1.18.18.2.m1.1.1.1.1.1.1.cmml" xref="S4.T1.18.18.2.m1.1.1.1.1"></minus><apply id="S4.T1.18.18.2.m1.1.1.1.1.1.2.cmml" xref="S4.T1.18.18.2.m1.1.1.1.1.1.2"><csymbol cd="latexml" id="S4.T1.18.18.2.m1.1.1.1.1.1.2.1.cmml" xref="S4.T1.18.18.2.m1.1.1.1.1.1.2.1">percent</csymbol><cn type="float" id="S4.T1.18.18.2.m1.1.1.1.1.1.2.2.cmml" xref="S4.T1.18.18.2.m1.1.1.1.1.1.2.2">15.4</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.18.18.2.m1.1c">43.4(-15.4\%)</annotation></semantics></math></td>
<td id="S4.T1.19.19.3" class="ltx_td ltx_align_center ltx_border_b ltx_border_r"><math id="S4.T1.19.19.3.m1.1" class="ltx_Math" alttext="43.8(-9.0\%)" display="inline"><semantics id="S4.T1.19.19.3.m1.1a"><mrow id="S4.T1.19.19.3.m1.1.1" xref="S4.T1.19.19.3.m1.1.1.cmml"><mn id="S4.T1.19.19.3.m1.1.1.3" xref="S4.T1.19.19.3.m1.1.1.3.cmml">43.8</mn><mo lspace="0em" rspace="0em" id="S4.T1.19.19.3.m1.1.1.2" xref="S4.T1.19.19.3.m1.1.1.2.cmml">​</mo><mrow id="S4.T1.19.19.3.m1.1.1.1.1" xref="S4.T1.19.19.3.m1.1.1.1.1.1.cmml"><mo stretchy="false" id="S4.T1.19.19.3.m1.1.1.1.1.2" xref="S4.T1.19.19.3.m1.1.1.1.1.1.cmml">(</mo><mrow id="S4.T1.19.19.3.m1.1.1.1.1.1" xref="S4.T1.19.19.3.m1.1.1.1.1.1.cmml"><mo id="S4.T1.19.19.3.m1.1.1.1.1.1a" xref="S4.T1.19.19.3.m1.1.1.1.1.1.cmml">−</mo><mrow id="S4.T1.19.19.3.m1.1.1.1.1.1.2" xref="S4.T1.19.19.3.m1.1.1.1.1.1.2.cmml"><mn id="S4.T1.19.19.3.m1.1.1.1.1.1.2.2" xref="S4.T1.19.19.3.m1.1.1.1.1.1.2.2.cmml">9.0</mn><mo id="S4.T1.19.19.3.m1.1.1.1.1.1.2.1" xref="S4.T1.19.19.3.m1.1.1.1.1.1.2.1.cmml">%</mo></mrow></mrow><mo stretchy="false" id="S4.T1.19.19.3.m1.1.1.1.1.3" xref="S4.T1.19.19.3.m1.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.T1.19.19.3.m1.1b"><apply id="S4.T1.19.19.3.m1.1.1.cmml" xref="S4.T1.19.19.3.m1.1.1"><times id="S4.T1.19.19.3.m1.1.1.2.cmml" xref="S4.T1.19.19.3.m1.1.1.2"></times><cn type="float" id="S4.T1.19.19.3.m1.1.1.3.cmml" xref="S4.T1.19.19.3.m1.1.1.3">43.8</cn><apply id="S4.T1.19.19.3.m1.1.1.1.1.1.cmml" xref="S4.T1.19.19.3.m1.1.1.1.1"><minus id="S4.T1.19.19.3.m1.1.1.1.1.1.1.cmml" xref="S4.T1.19.19.3.m1.1.1.1.1"></minus><apply id="S4.T1.19.19.3.m1.1.1.1.1.1.2.cmml" xref="S4.T1.19.19.3.m1.1.1.1.1.1.2"><csymbol cd="latexml" id="S4.T1.19.19.3.m1.1.1.1.1.1.2.1.cmml" xref="S4.T1.19.19.3.m1.1.1.1.1.1.2.1">percent</csymbol><cn type="float" id="S4.T1.19.19.3.m1.1.1.1.1.1.2.2.cmml" xref="S4.T1.19.19.3.m1.1.1.1.1.1.2.2">9.0</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.19.19.3.m1.1c">43.8(-9.0\%)</annotation></semantics></math></td>
<td id="S4.T1.20.20.4" class="ltx_td ltx_align_center ltx_border_b"><math id="S4.T1.20.20.4.m1.1" class="ltx_Math" alttext="51.7(1.1\%)" display="inline"><semantics id="S4.T1.20.20.4.m1.1a"><mrow id="S4.T1.20.20.4.m1.1.1" xref="S4.T1.20.20.4.m1.1.1.cmml"><mn id="S4.T1.20.20.4.m1.1.1.3" xref="S4.T1.20.20.4.m1.1.1.3.cmml">51.7</mn><mo lspace="0em" rspace="0em" id="S4.T1.20.20.4.m1.1.1.2" xref="S4.T1.20.20.4.m1.1.1.2.cmml">​</mo><mrow id="S4.T1.20.20.4.m1.1.1.1.1" xref="S4.T1.20.20.4.m1.1.1.1.1.1.cmml"><mo stretchy="false" id="S4.T1.20.20.4.m1.1.1.1.1.2" xref="S4.T1.20.20.4.m1.1.1.1.1.1.cmml">(</mo><mrow id="S4.T1.20.20.4.m1.1.1.1.1.1" xref="S4.T1.20.20.4.m1.1.1.1.1.1.cmml"><mn id="S4.T1.20.20.4.m1.1.1.1.1.1.2" xref="S4.T1.20.20.4.m1.1.1.1.1.1.2.cmml">1.1</mn><mo id="S4.T1.20.20.4.m1.1.1.1.1.1.1" xref="S4.T1.20.20.4.m1.1.1.1.1.1.1.cmml">%</mo></mrow><mo stretchy="false" id="S4.T1.20.20.4.m1.1.1.1.1.3" xref="S4.T1.20.20.4.m1.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.T1.20.20.4.m1.1b"><apply id="S4.T1.20.20.4.m1.1.1.cmml" xref="S4.T1.20.20.4.m1.1.1"><times id="S4.T1.20.20.4.m1.1.1.2.cmml" xref="S4.T1.20.20.4.m1.1.1.2"></times><cn type="float" id="S4.T1.20.20.4.m1.1.1.3.cmml" xref="S4.T1.20.20.4.m1.1.1.3">51.7</cn><apply id="S4.T1.20.20.4.m1.1.1.1.1.1.cmml" xref="S4.T1.20.20.4.m1.1.1.1.1"><csymbol cd="latexml" id="S4.T1.20.20.4.m1.1.1.1.1.1.1.cmml" xref="S4.T1.20.20.4.m1.1.1.1.1.1.1">percent</csymbol><cn type="float" id="S4.T1.20.20.4.m1.1.1.1.1.1.2.cmml" xref="S4.T1.20.20.4.m1.1.1.1.1.1.2">1.1</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.20.20.4.m1.1c">51.7(1.1\%)</annotation></semantics></math></td>
</tr>
</table>
</figure>
</section>
<section id="S4.SS3.SSS3" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">4.3.3 </span>Metrics</h4>

<div id="S4.SS3.SSS3.p1" class="ltx_para ltx_noindent">
<p id="S4.SS3.SSS3.p1.1" class="ltx_p">As mentioned in Section <a href="#S3" title="3 Preliminaries ‣ TRACE: A Comprehensive Benchmark for Continual Learning in Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>, we measure the performance of LLMs in continual learning tasks using Overall Performance and Backward Transfer Score. To check how well LLMs keep their original abilities, we use General Ability Delta, Instruction Following Delta, and Safety Delta. More details are in Section <a href="#S4.SS2" title="4.2 CL Metrics Design ‣ 4 TRACE: A Comprehensive Benchmark for CL in LLMs ‣ TRACE: A Comprehensive Benchmark for Continual Learning in Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4.2</span></a>. For evaluating instruction-following and safety, we score with GPT-4 <cite class="ltx_cite ltx_citemacro_citep">(OpenAI, <a href="#bib.bib33" title="" class="ltx_ref">2023</a>)</cite>. More details about this scoring can be found in the Appendix <a href="#Ax1.SS7" title=".7 Model evaluation ‣ Appendix ‣ TRACE: A Comprehensive Benchmark for Continual Learning in Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">.7</span></a>.</p>
</div>
</section>
<section id="S4.SS3.SSS4" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">4.3.4 </span>Implementation Details</h4>

<div id="S4.SS3.SSS4.p1" class="ltx_para ltx_noindent">
<p id="S4.SS3.SSS4.p1.1" class="ltx_p">The detailed settings can be found in Appendix <a href="#Ax1.SS1" title=".1 Implementation Details ‣ Appendix ‣ TRACE: A Comprehensive Benchmark for Continual Learning in Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">.1</span></a>.</p>
</div>
</section>
</section>
<section id="S4.SS4" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.4 </span>Main Results</h3>

<section id="S4.SS4.SSS1" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">4.4.1 </span>Performance of Target Sequential Tasks</h4>

<div id="S4.SS4.SSS1.p1" class="ltx_para ltx_noindent">
<p id="S4.SS4.SSS1.p1.1" class="ltx_p">Table <a href="#S4.T1" title="Table 1 ‣ 4.3.2 Datasets ‣ 4.3 Experimental Setup ‣ 4 TRACE: A Comprehensive Benchmark for CL in LLMs ‣ TRACE: A Comprehensive Benchmark for Continual Learning in Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a> showcases the performance of five distinct LLMs on TRACE benchmark, after their continual learning phase. From this evaluation, we can draw the following conclusions:</p>
</div>
<div id="S4.SS4.SSS1.p2" class="ltx_para ltx_noindent">
<p id="S4.SS4.SSS1.p2.1" class="ltx_p"><span id="S4.SS4.SSS1.p2.1.1" class="ltx_text ltx_font_bold">In-Context Learning (ICL) Performance</span>: ICL methods generally perform lower than SeqFT and Replay methods. This suggests that the TRACE benchmark is indeed challenging, and LLMs can’t readily identify solutions just through simple demonstrations.</p>
</div>
<div id="S4.SS4.SSS1.p3" class="ltx_para ltx_noindent">
<p id="S4.SS4.SSS1.p3.1" class="ltx_p"><span id="S4.SS4.SSS1.p3.1.1" class="ltx_text ltx_font_bold">Replay Performance</span>: Among all the baselines, Replay achieved the highest OP score. With its BWT score being positive, it indicates that Replay effectively retains its performance on sequential tasks without significant forgetting. This makes Replay a straightforward and efficient strategy in a continual learning context.</p>
</div>
<div id="S4.SS4.SSS1.p4" class="ltx_para ltx_noindent">
<p id="S4.SS4.SSS1.p4.1" class="ltx_p"><span id="S4.SS4.SSS1.p4.1.1" class="ltx_text ltx_font_bold">Full Parameter Training vs. LoRA</span>: Full parameter training demonstrates better task-specific adaptability compared to LoRA, with a smaller BWT score. For instance, LLaMA-2-7B-Chat’s SeqFT OP(BWT) is 48.7 (8.3%), while LoRASeqFT stands at 12.7 (45.7%). This suggests that when the focus is primarily on sequential tasks, full parameter fine-tuning should be prioritized over parameter-efficient methods like LoRA.</p>
</div>
<figure id="S4.T2" class="ltx_table">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 2: </span>Comparison of the general language understanding and reasoning abilities. <span id="S4.T2.144.1" class="ltx_text" style="color:#0000FF;">blue</span> means increase, while <span id="S4.T2.145.2" class="ltx_text" style="color:#FF0000;">red</span> means decrease.</figcaption>
<div id="S4.T2.141" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:397.5pt;height:285.1pt;vertical-align:-0.6pt;"><span class="ltx_transformed_inner" style="transform:translate(-110.6pt,79.2pt) scale(0.642427716888915,0.642427716888915) ;">
<table id="S4.T2.141.141" class="ltx_tabular ltx_align_middle">
<tr id="S4.T2.1.1.1" class="ltx_tr">
<td id="S4.T2.1.1.1.2" class="ltx_td ltx_border_tt" style="padding-left:2.0pt;padding-right:2.0pt;"></td>
<td id="S4.T2.1.1.1.3" class="ltx_td ltx_align_center ltx_border_tt" style="padding-left:2.0pt;padding-right:2.0pt;">
<table id="S4.T2.1.1.1.3.1" class="ltx_tabular ltx_align_middle">
<tr id="S4.T2.1.1.1.3.1.1" class="ltx_tr">
<td id="S4.T2.1.1.1.3.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;"><span id="S4.T2.1.1.1.3.1.1.1.1" class="ltx_text ltx_font_bold">MMLU</span></td>
</tr>
<tr id="S4.T2.1.1.1.3.1.2" class="ltx_tr">
<td id="S4.T2.1.1.1.3.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;"><span id="S4.T2.1.1.1.3.1.2.1.1" class="ltx_text ltx_font_bold">(factuality)</span></td>
</tr>
</table>
</td>
<td id="S4.T2.1.1.1.4" class="ltx_td ltx_align_center ltx_border_tt" style="padding-left:2.0pt;padding-right:2.0pt;">
<table id="S4.T2.1.1.1.4.1" class="ltx_tabular ltx_align_middle">
<tr id="S4.T2.1.1.1.4.1.1" class="ltx_tr">
<td id="S4.T2.1.1.1.4.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;"><span id="S4.T2.1.1.1.4.1.1.1.1" class="ltx_text ltx_font_bold">GSM</span></td>
</tr>
<tr id="S4.T2.1.1.1.4.1.2" class="ltx_tr">
<td id="S4.T2.1.1.1.4.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;"><span id="S4.T2.1.1.1.4.1.2.1.1" class="ltx_text ltx_font_bold">(math)</span></td>
</tr>
</table>
</td>
<td id="S4.T2.1.1.1.5" class="ltx_td ltx_align_center ltx_border_tt" style="padding-left:2.0pt;padding-right:2.0pt;">
<table id="S4.T2.1.1.1.5.1" class="ltx_tabular ltx_align_middle">
<tr id="S4.T2.1.1.1.5.1.1" class="ltx_tr">
<td id="S4.T2.1.1.1.5.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;"><span id="S4.T2.1.1.1.5.1.1.1.1" class="ltx_text ltx_font_bold">BBH</span></td>
</tr>
<tr id="S4.T2.1.1.1.5.1.2" class="ltx_tr">
<td id="S4.T2.1.1.1.5.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;"><span id="S4.T2.1.1.1.5.1.2.1.1" class="ltx_text ltx_font_bold">(reasoning)</span></td>
</tr>
</table>
</td>
<td id="S4.T2.1.1.1.6" class="ltx_td ltx_align_center ltx_border_tt" style="padding-left:2.0pt;padding-right:2.0pt;">
<table id="S4.T2.1.1.1.6.1" class="ltx_tabular ltx_align_middle">
<tr id="S4.T2.1.1.1.6.1.1" class="ltx_tr">
<td id="S4.T2.1.1.1.6.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;"><span id="S4.T2.1.1.1.6.1.1.1.1" class="ltx_text ltx_font_bold">TydiQA</span></td>
</tr>
<tr id="S4.T2.1.1.1.6.1.2" class="ltx_tr">
<td id="S4.T2.1.1.1.6.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;"><span id="S4.T2.1.1.1.6.1.2.1.1" class="ltx_text ltx_font_bold">(multilinguality)</span></td>
</tr>
</table>
</td>
<td id="S4.T2.1.1.1.7" class="ltx_td ltx_align_left ltx_border_tt" style="padding-left:2.0pt;padding-right:2.0pt;">
<table id="S4.T2.1.1.1.7.1" class="ltx_tabular ltx_align_middle">
<tr id="S4.T2.1.1.1.7.1.1" class="ltx_tr">
<td id="S4.T2.1.1.1.7.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;"><span id="S4.T2.1.1.1.7.1.1.1.1" class="ltx_text ltx_font_bold">BoolQ</span></td>
</tr>
<tr id="S4.T2.1.1.1.7.1.2" class="ltx_tr">
<td id="S4.T2.1.1.1.7.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;"><span id="S4.T2.1.1.1.7.1.2.1.1" class="ltx_text ltx_font_bold">(comprehension)</span></td>
</tr>
</table>
</td>
<td id="S4.T2.1.1.1.8" class="ltx_td ltx_align_left ltx_border_tt" style="padding-left:2.0pt;padding-right:2.0pt;">
<table id="S4.T2.1.1.1.8.1" class="ltx_tabular ltx_align_middle">
<tr id="S4.T2.1.1.1.8.1.1" class="ltx_tr">
<td id="S4.T2.1.1.1.8.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;"><span id="S4.T2.1.1.1.8.1.1.1.1" class="ltx_text ltx_font_bold">PIQA</span></td>
</tr>
<tr id="S4.T2.1.1.1.8.1.2" class="ltx_tr">
<td id="S4.T2.1.1.1.8.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;"><span id="S4.T2.1.1.1.8.1.2.1.1" class="ltx_text ltx_font_bold">(commonsense)</span></td>
</tr>
</table>
</td>
<td id="S4.T2.1.1.1.1" class="ltx_td ltx_align_center ltx_border_tt" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.1.1.1.1.m1.1" class="ltx_Math" alttext="\Delta R_{t}^{G}" display="inline"><semantics id="S4.T2.1.1.1.1.m1.1a"><mrow id="S4.T2.1.1.1.1.m1.1.1" xref="S4.T2.1.1.1.1.m1.1.1.cmml"><mi mathvariant="normal" id="S4.T2.1.1.1.1.m1.1.1.2" xref="S4.T2.1.1.1.1.m1.1.1.2.cmml">Δ</mi><mo lspace="0em" rspace="0em" id="S4.T2.1.1.1.1.m1.1.1.1" xref="S4.T2.1.1.1.1.m1.1.1.1.cmml">​</mo><msubsup id="S4.T2.1.1.1.1.m1.1.1.3" xref="S4.T2.1.1.1.1.m1.1.1.3.cmml"><mi id="S4.T2.1.1.1.1.m1.1.1.3.2.2" xref="S4.T2.1.1.1.1.m1.1.1.3.2.2.cmml">R</mi><mi id="S4.T2.1.1.1.1.m1.1.1.3.2.3" xref="S4.T2.1.1.1.1.m1.1.1.3.2.3.cmml">t</mi><mi id="S4.T2.1.1.1.1.m1.1.1.3.3" xref="S4.T2.1.1.1.1.m1.1.1.3.3.cmml">G</mi></msubsup></mrow><annotation-xml encoding="MathML-Content" id="S4.T2.1.1.1.1.m1.1b"><apply id="S4.T2.1.1.1.1.m1.1.1.cmml" xref="S4.T2.1.1.1.1.m1.1.1"><times id="S4.T2.1.1.1.1.m1.1.1.1.cmml" xref="S4.T2.1.1.1.1.m1.1.1.1"></times><ci id="S4.T2.1.1.1.1.m1.1.1.2.cmml" xref="S4.T2.1.1.1.1.m1.1.1.2">Δ</ci><apply id="S4.T2.1.1.1.1.m1.1.1.3.cmml" xref="S4.T2.1.1.1.1.m1.1.1.3"><csymbol cd="ambiguous" id="S4.T2.1.1.1.1.m1.1.1.3.1.cmml" xref="S4.T2.1.1.1.1.m1.1.1.3">superscript</csymbol><apply id="S4.T2.1.1.1.1.m1.1.1.3.2.cmml" xref="S4.T2.1.1.1.1.m1.1.1.3"><csymbol cd="ambiguous" id="S4.T2.1.1.1.1.m1.1.1.3.2.1.cmml" xref="S4.T2.1.1.1.1.m1.1.1.3">subscript</csymbol><ci id="S4.T2.1.1.1.1.m1.1.1.3.2.2.cmml" xref="S4.T2.1.1.1.1.m1.1.1.3.2.2">𝑅</ci><ci id="S4.T2.1.1.1.1.m1.1.1.3.2.3.cmml" xref="S4.T2.1.1.1.1.m1.1.1.3.2.3">𝑡</ci></apply><ci id="S4.T2.1.1.1.1.m1.1.1.3.3.cmml" xref="S4.T2.1.1.1.1.m1.1.1.3.3">𝐺</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.1.1.1.1.m1.1c">\Delta R_{t}^{G}</annotation></semantics></math></td>
</tr>
<tr id="S4.T2.141.141.142" class="ltx_tr">
<td id="S4.T2.141.141.142.1" class="ltx_td" style="padding-left:2.0pt;padding-right:2.0pt;"></td>
<td id="S4.T2.141.141.142.2" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:2.0pt;padding-right:2.0pt;">
<table id="S4.T2.141.141.142.2.1" class="ltx_tabular ltx_align_middle">
<tr id="S4.T2.141.141.142.2.1.1" class="ltx_tr">
<td id="S4.T2.141.141.142.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;"><span id="S4.T2.141.141.142.2.1.1.1.1" class="ltx_text ltx_font_bold">ACC</span></td>
</tr>
<tr id="S4.T2.141.141.142.2.1.2" class="ltx_tr">
<td id="S4.T2.141.141.142.2.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;"><span id="S4.T2.141.141.142.2.1.2.1.1" class="ltx_text ltx_font_bold">(5-shot)</span></td>
</tr>
</table>
</td>
<td id="S4.T2.141.141.142.3" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:2.0pt;padding-right:2.0pt;">
<table id="S4.T2.141.141.142.3.1" class="ltx_tabular ltx_align_middle">
<tr id="S4.T2.141.141.142.3.1.1" class="ltx_tr">
<td id="S4.T2.141.141.142.3.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;"><span id="S4.T2.141.141.142.3.1.1.1.1" class="ltx_text ltx_font_bold">EM</span></td>
</tr>
<tr id="S4.T2.141.141.142.3.1.2" class="ltx_tr">
<td id="S4.T2.141.141.142.3.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;"><span id="S4.T2.141.141.142.3.1.2.1.1" class="ltx_text ltx_font_bold">(8-shot, CoT)</span></td>
</tr>
</table>
</td>
<td id="S4.T2.141.141.142.4" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:2.0pt;padding-right:2.0pt;">
<table id="S4.T2.141.141.142.4.1" class="ltx_tabular ltx_align_middle">
<tr id="S4.T2.141.141.142.4.1.1" class="ltx_tr">
<td id="S4.T2.141.141.142.4.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;"><span id="S4.T2.141.141.142.4.1.1.1.1" class="ltx_text ltx_font_bold">EM</span></td>
</tr>
<tr id="S4.T2.141.141.142.4.1.2" class="ltx_tr">
<td id="S4.T2.141.141.142.4.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;"><span id="S4.T2.141.141.142.4.1.2.1.1" class="ltx_text ltx_font_bold">(3-shot, CoT)</span></td>
</tr>
</table>
</td>
<td id="S4.T2.141.141.142.5" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:2.0pt;padding-right:2.0pt;">
<table id="S4.T2.141.141.142.5.1" class="ltx_tabular ltx_align_middle">
<tr id="S4.T2.141.141.142.5.1.1" class="ltx_tr">
<td id="S4.T2.141.141.142.5.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;"><span id="S4.T2.141.141.142.5.1.1.1.1" class="ltx_text ltx_font_bold">F1</span></td>
</tr>
<tr id="S4.T2.141.141.142.5.1.2" class="ltx_tr">
<td id="S4.T2.141.141.142.5.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;"><span id="S4.T2.141.141.142.5.1.2.1.1" class="ltx_text ltx_font_bold">(1-shot, GP)</span></td>
</tr>
</table>
</td>
<td id="S4.T2.141.141.142.6" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:2.0pt;padding-right:2.0pt;">
<table id="S4.T2.141.141.142.6.1" class="ltx_tabular ltx_align_middle">
<tr id="S4.T2.141.141.142.6.1.1" class="ltx_tr">
<td id="S4.T2.141.141.142.6.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;"><span id="S4.T2.141.141.142.6.1.1.1.1" class="ltx_text ltx_font_bold">ACC</span></td>
</tr>
<tr id="S4.T2.141.141.142.6.1.2" class="ltx_tr">
<td id="S4.T2.141.141.142.6.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;"><span id="S4.T2.141.141.142.6.1.2.1.1" class="ltx_text ltx_font_bold">(0-shot)</span></td>
</tr>
</table>
</td>
<td id="S4.T2.141.141.142.7" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:2.0pt;padding-right:2.0pt;">
<table id="S4.T2.141.141.142.7.1" class="ltx_tabular ltx_align_middle">
<tr id="S4.T2.141.141.142.7.1.1" class="ltx_tr">
<td id="S4.T2.141.141.142.7.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;"><span id="S4.T2.141.141.142.7.1.1.1.1" class="ltx_text ltx_font_bold">ACC</span></td>
</tr>
<tr id="S4.T2.141.141.142.7.1.2" class="ltx_tr">
<td id="S4.T2.141.141.142.7.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;"><span id="S4.T2.141.141.142.7.1.2.1.1" class="ltx_text ltx_font_bold">(0-shot)</span></td>
</tr>
</table>
</td>
<td id="S4.T2.141.141.142.8" class="ltx_td ltx_border_t" style="padding-left:2.0pt;padding-right:2.0pt;"></td>
</tr>
<tr id="S4.T2.8.8.8" class="ltx_tr">
<td id="S4.T2.8.8.8.8" class="ltx_td ltx_align_left ltx_border_t" style="padding-left:2.0pt;padding-right:2.0pt;">LLaMA-2-7B-Chat</td>
<td id="S4.T2.2.2.2.1" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.2.2.2.1.m1.1" class="ltx_Math" alttext="46.56" display="inline"><semantics id="S4.T2.2.2.2.1.m1.1a"><mn id="S4.T2.2.2.2.1.m1.1.1" xref="S4.T2.2.2.2.1.m1.1.1.cmml">46.56</mn><annotation-xml encoding="MathML-Content" id="S4.T2.2.2.2.1.m1.1b"><cn type="float" id="S4.T2.2.2.2.1.m1.1.1.cmml" xref="S4.T2.2.2.2.1.m1.1.1">46.56</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.2.2.2.1.m1.1c">46.56</annotation></semantics></math></td>
<td id="S4.T2.3.3.3.2" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.3.3.3.2.m1.1" class="ltx_Math" alttext="26.08" display="inline"><semantics id="S4.T2.3.3.3.2.m1.1a"><mn id="S4.T2.3.3.3.2.m1.1.1" xref="S4.T2.3.3.3.2.m1.1.1.cmml">26.08</mn><annotation-xml encoding="MathML-Content" id="S4.T2.3.3.3.2.m1.1b"><cn type="float" id="S4.T2.3.3.3.2.m1.1.1.cmml" xref="S4.T2.3.3.3.2.m1.1.1">26.08</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.3.3.3.2.m1.1c">26.08</annotation></semantics></math></td>
<td id="S4.T2.4.4.4.3" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.4.4.4.3.m1.1" class="ltx_Math" alttext="40.23" display="inline"><semantics id="S4.T2.4.4.4.3.m1.1a"><mn id="S4.T2.4.4.4.3.m1.1.1" xref="S4.T2.4.4.4.3.m1.1.1.cmml">40.23</mn><annotation-xml encoding="MathML-Content" id="S4.T2.4.4.4.3.m1.1b"><cn type="float" id="S4.T2.4.4.4.3.m1.1.1.cmml" xref="S4.T2.4.4.4.3.m1.1.1">40.23</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.4.4.4.3.m1.1c">40.23</annotation></semantics></math></td>
<td id="S4.T2.5.5.5.4" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.5.5.5.4.m1.1" class="ltx_Math" alttext="23.47" display="inline"><semantics id="S4.T2.5.5.5.4.m1.1a"><mn id="S4.T2.5.5.5.4.m1.1.1" xref="S4.T2.5.5.5.4.m1.1.1.cmml">23.47</mn><annotation-xml encoding="MathML-Content" id="S4.T2.5.5.5.4.m1.1b"><cn type="float" id="S4.T2.5.5.5.4.m1.1.1.cmml" xref="S4.T2.5.5.5.4.m1.1.1">23.47</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.5.5.5.4.m1.1c">23.47</annotation></semantics></math></td>
<td id="S4.T2.6.6.6.5" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.6.6.6.5.m1.1" class="ltx_Math" alttext="70.55" display="inline"><semantics id="S4.T2.6.6.6.5.m1.1a"><mn id="S4.T2.6.6.6.5.m1.1.1" xref="S4.T2.6.6.6.5.m1.1.1.cmml">70.55</mn><annotation-xml encoding="MathML-Content" id="S4.T2.6.6.6.5.m1.1b"><cn type="float" id="S4.T2.6.6.6.5.m1.1.1.cmml" xref="S4.T2.6.6.6.5.m1.1.1">70.55</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.6.6.6.5.m1.1c">70.55</annotation></semantics></math></td>
<td id="S4.T2.7.7.7.6" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.7.7.7.6.m1.1" class="ltx_Math" alttext="76.22" display="inline"><semantics id="S4.T2.7.7.7.6.m1.1a"><mn id="S4.T2.7.7.7.6.m1.1.1" xref="S4.T2.7.7.7.6.m1.1.1.cmml">76.22</mn><annotation-xml encoding="MathML-Content" id="S4.T2.7.7.7.6.m1.1b"><cn type="float" id="S4.T2.7.7.7.6.m1.1.1.cmml" xref="S4.T2.7.7.7.6.m1.1.1">76.22</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.7.7.7.6.m1.1c">76.22</annotation></semantics></math></td>
<td id="S4.T2.8.8.8.7" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_t" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.8.8.8.7.m1.1" class="ltx_Math" alttext="0" display="inline"><semantics id="S4.T2.8.8.8.7.m1.1a"><mn id="S4.T2.8.8.8.7.m1.1.1" xref="S4.T2.8.8.8.7.m1.1.1.cmml">0</mn><annotation-xml encoding="MathML-Content" id="S4.T2.8.8.8.7.m1.1b"><cn type="integer" id="S4.T2.8.8.8.7.m1.1.1.cmml" xref="S4.T2.8.8.8.7.m1.1.1">0</cn></annotation-xml></semantics></math></td>
</tr>
<tr id="S4.T2.15.15.15" class="ltx_tr">
<td id="S4.T2.15.15.15.8" class="ltx_td ltx_align_left" style="padding-left:2.0pt;padding-right:2.0pt;">LLaMA-2-7B-Chat-Seq</td>
<td id="S4.T2.9.9.9.1" class="ltx_td ltx_align_center" style="background-color:#FFF7F7;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.9.9.9.1.m1.1" class="ltx_Math" style="background-color:#FFF7F7;" alttext="46.43" display="inline"><semantics id="S4.T2.9.9.9.1.m1.1a"><mn mathbackground="#FFF7F7" id="S4.T2.9.9.9.1.m1.1.1" xref="S4.T2.9.9.9.1.m1.1.1.cmml">46.43</mn><annotation-xml encoding="MathML-Content" id="S4.T2.9.9.9.1.m1.1b"><cn type="float" id="S4.T2.9.9.9.1.m1.1.1.cmml" xref="S4.T2.9.9.9.1.m1.1.1">46.43</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.9.9.9.1.m1.1c">46.43</annotation></semantics></math></td>
<td id="S4.T2.10.10.10.2" class="ltx_td ltx_align_center" style="background-color:#FF8F8F;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.10.10.10.2.m1.1" class="ltx_Math" style="background-color:#FF8F8F;" alttext="3.49" display="inline"><semantics id="S4.T2.10.10.10.2.m1.1a"><mn mathbackground="#FF8F8F" id="S4.T2.10.10.10.2.m1.1.1" xref="S4.T2.10.10.10.2.m1.1.1.cmml">3.49</mn><annotation-xml encoding="MathML-Content" id="S4.T2.10.10.10.2.m1.1b"><cn type="float" id="S4.T2.10.10.10.2.m1.1.1.cmml" xref="S4.T2.10.10.10.2.m1.1.1">3.49</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.10.10.10.2.m1.1c">3.49</annotation></semantics></math></td>
<td id="S4.T2.11.11.11.3" class="ltx_td ltx_align_center" style="background-color:#FFB5B5;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.11.11.11.3.m1.1" class="ltx_Math" style="background-color:#FFB5B5;" alttext="30.11" display="inline"><semantics id="S4.T2.11.11.11.3.m1.1a"><mn mathbackground="#FFB5B5" id="S4.T2.11.11.11.3.m1.1.1" xref="S4.T2.11.11.11.3.m1.1.1.cmml">30.11</mn><annotation-xml encoding="MathML-Content" id="S4.T2.11.11.11.3.m1.1b"><cn type="float" id="S4.T2.11.11.11.3.m1.1.1.cmml" xref="S4.T2.11.11.11.3.m1.1.1">30.11</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.11.11.11.3.m1.1c">30.11</annotation></semantics></math></td>
<td id="S4.T2.12.12.12.4" class="ltx_td ltx_align_center" style="background-color:#D6D6FF;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.12.12.12.4.m1.1" class="ltx_Math" style="background-color:#D6D6FF;" alttext="33.23" display="inline"><semantics id="S4.T2.12.12.12.4.m1.1a"><mn mathbackground="#D6D6FF" id="S4.T2.12.12.12.4.m1.1.1" xref="S4.T2.12.12.12.4.m1.1.1.cmml">33.23</mn><annotation-xml encoding="MathML-Content" id="S4.T2.12.12.12.4.m1.1b"><cn type="float" id="S4.T2.12.12.12.4.m1.1.1.cmml" xref="S4.T2.12.12.12.4.m1.1.1">33.23</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.12.12.12.4.m1.1c">33.23</annotation></semantics></math></td>
<td id="S4.T2.13.13.13.5" class="ltx_td ltx_align_center" style="background-color:#DCDCFF;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.13.13.13.5.m1.1" class="ltx_Math" style="background-color:#DCDCFF;" alttext="77.89" display="inline"><semantics id="S4.T2.13.13.13.5.m1.1a"><mn mathbackground="#DCDCFF" id="S4.T2.13.13.13.5.m1.1.1" xref="S4.T2.13.13.13.5.m1.1.1.cmml">77.89</mn><annotation-xml encoding="MathML-Content" id="S4.T2.13.13.13.5.m1.1b"><cn type="float" id="S4.T2.13.13.13.5.m1.1.1.cmml" xref="S4.T2.13.13.13.5.m1.1.1">77.89</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.13.13.13.5.m1.1c">77.89</annotation></semantics></math></td>
<td id="S4.T2.14.14.14.6" class="ltx_td ltx_align_center" style="background-color:#F7F7FF;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.14.14.14.6.m1.1" class="ltx_Math" style="background-color:#F7F7FF;" alttext="76.5" display="inline"><semantics id="S4.T2.14.14.14.6.m1.1a"><mn mathbackground="#F7F7FF" id="S4.T2.14.14.14.6.m1.1.1" xref="S4.T2.14.14.14.6.m1.1.1.cmml">76.5</mn><annotation-xml encoding="MathML-Content" id="S4.T2.14.14.14.6.m1.1b"><cn type="float" id="S4.T2.14.14.14.6.m1.1.1.cmml" xref="S4.T2.14.14.14.6.m1.1.1">76.5</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.14.14.14.6.m1.1c">76.5</annotation></semantics></math></td>
<td id="S4.T2.15.15.15.7" class="ltx_td ltx_nopad_r ltx_align_center" style="background-color:#FFDFDF;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.15.15.15.7.m1.1" class="ltx_Math" style="background-color:#FFDFDF;" alttext="-2.58" display="inline"><semantics id="S4.T2.15.15.15.7.m1.1a"><mrow id="S4.T2.15.15.15.7.m1.1.1" xref="S4.T2.15.15.15.7.m1.1.1.cmml"><mo mathbackground="#FFDFDF" id="S4.T2.15.15.15.7.m1.1.1a" xref="S4.T2.15.15.15.7.m1.1.1.cmml">−</mo><mn mathbackground="#FFDFDF" id="S4.T2.15.15.15.7.m1.1.1.2" xref="S4.T2.15.15.15.7.m1.1.1.2.cmml">2.58</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.T2.15.15.15.7.m1.1b"><apply id="S4.T2.15.15.15.7.m1.1.1.cmml" xref="S4.T2.15.15.15.7.m1.1.1"><minus id="S4.T2.15.15.15.7.m1.1.1.1.cmml" xref="S4.T2.15.15.15.7.m1.1.1"></minus><cn type="float" id="S4.T2.15.15.15.7.m1.1.1.2.cmml" xref="S4.T2.15.15.15.7.m1.1.1.2">2.58</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.15.15.15.7.m1.1c">-2.58</annotation></semantics></math></td>
</tr>
<tr id="S4.T2.22.22.22" class="ltx_tr">
<td id="S4.T2.22.22.22.8" class="ltx_td ltx_align_left" style="padding-left:2.0pt;padding-right:2.0pt;">LLaMA-2-7B-Chat-LoraSeq</td>
<td id="S4.T2.16.16.16.1" class="ltx_td ltx_align_center" style="background-color:#FFD6D6;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.16.16.16.1.m1.1" class="ltx_Math" style="background-color:#FFD6D6;" alttext="42.28" display="inline"><semantics id="S4.T2.16.16.16.1.m1.1a"><mn mathbackground="#FFD6D6" id="S4.T2.16.16.16.1.m1.1.1" xref="S4.T2.16.16.16.1.m1.1.1.cmml">42.28</mn><annotation-xml encoding="MathML-Content" id="S4.T2.16.16.16.1.m1.1b"><cn type="float" id="S4.T2.16.16.16.1.m1.1.1.cmml" xref="S4.T2.16.16.16.1.m1.1.1">42.28</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.16.16.16.1.m1.1c">42.28</annotation></semantics></math></td>
<td id="S4.T2.17.17.17.2" class="ltx_td ltx_align_center" style="background-color:#FF9F9F;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.17.17.17.2.m1.1" class="ltx_Math" style="background-color:#FF9F9F;" alttext="14.71" display="inline"><semantics id="S4.T2.17.17.17.2.m1.1a"><mn mathbackground="#FF9F9F" id="S4.T2.17.17.17.2.m1.1.1" xref="S4.T2.17.17.17.2.m1.1.1.cmml">14.71</mn><annotation-xml encoding="MathML-Content" id="S4.T2.17.17.17.2.m1.1b"><cn type="float" id="S4.T2.17.17.17.2.m1.1.1.cmml" xref="S4.T2.17.17.17.2.m1.1.1">14.71</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.17.17.17.2.m1.1c">14.71</annotation></semantics></math></td>
<td id="S4.T2.18.18.18.3" class="ltx_td ltx_align_center" style="background-color:#FFBBBB;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.18.18.18.3.m1.1" class="ltx_Math" style="background-color:#FFBBBB;" alttext="33.61" display="inline"><semantics id="S4.T2.18.18.18.3.m1.1a"><mn mathbackground="#FFBBBB" id="S4.T2.18.18.18.3.m1.1.1" xref="S4.T2.18.18.18.3.m1.1.1.cmml">33.61</mn><annotation-xml encoding="MathML-Content" id="S4.T2.18.18.18.3.m1.1b"><cn type="float" id="S4.T2.18.18.18.3.m1.1.1.cmml" xref="S4.T2.18.18.18.3.m1.1.1">33.61</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.18.18.18.3.m1.1c">33.61</annotation></semantics></math></td>
<td id="S4.T2.19.19.19.4" class="ltx_td ltx_align_center" style="background-color:#FFF1F1;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.19.19.19.4.m1.1" class="ltx_Math" style="background-color:#FFF1F1;" alttext="21.72" display="inline"><semantics id="S4.T2.19.19.19.4.m1.1a"><mn mathbackground="#FFF1F1" id="S4.T2.19.19.19.4.m1.1.1" xref="S4.T2.19.19.19.4.m1.1.1.cmml">21.72</mn><annotation-xml encoding="MathML-Content" id="S4.T2.19.19.19.4.m1.1b"><cn type="float" id="S4.T2.19.19.19.4.m1.1.1.cmml" xref="S4.T2.19.19.19.4.m1.1.1">21.72</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.19.19.19.4.m1.1c">21.72</annotation></semantics></math></td>
<td id="S4.T2.20.20.20.5" class="ltx_td ltx_align_center" style="background-color:#FF8A8A;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.20.20.20.5.m1.1" class="ltx_Math" style="background-color:#FF8A8A;" alttext="53.43" display="inline"><semantics id="S4.T2.20.20.20.5.m1.1a"><mn mathbackground="#FF8A8A" id="S4.T2.20.20.20.5.m1.1.1" xref="S4.T2.20.20.20.5.m1.1.1.cmml">53.43</mn><annotation-xml encoding="MathML-Content" id="S4.T2.20.20.20.5.m1.1b"><cn type="float" id="S4.T2.20.20.20.5.m1.1.1.cmml" xref="S4.T2.20.20.20.5.m1.1.1">53.43</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.20.20.20.5.m1.1c">53.43</annotation></semantics></math></td>
<td id="S4.T2.21.21.21.6" class="ltx_td ltx_align_center" style="background-color:#FFF9F9;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.21.21.21.6.m1.1" class="ltx_Math" style="background-color:#FFF9F9;" alttext="75.19" display="inline"><semantics id="S4.T2.21.21.21.6.m1.1a"><mn mathbackground="#FFF9F9" id="S4.T2.21.21.21.6.m1.1.1" xref="S4.T2.21.21.21.6.m1.1.1.cmml">75.19</mn><annotation-xml encoding="MathML-Content" id="S4.T2.21.21.21.6.m1.1b"><cn type="float" id="S4.T2.21.21.21.6.m1.1.1.cmml" xref="S4.T2.21.21.21.6.m1.1.1">75.19</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.21.21.21.6.m1.1c">75.19</annotation></semantics></math></td>
<td id="S4.T2.22.22.22.7" class="ltx_td ltx_nopad_r ltx_align_center" style="background-color:#FFB2B2;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.22.22.22.7.m1.1" class="ltx_Math" style="background-color:#FFB2B2;" alttext="-7.03" display="inline"><semantics id="S4.T2.22.22.22.7.m1.1a"><mrow id="S4.T2.22.22.22.7.m1.1.1" xref="S4.T2.22.22.22.7.m1.1.1.cmml"><mo mathbackground="#FFB2B2" id="S4.T2.22.22.22.7.m1.1.1a" xref="S4.T2.22.22.22.7.m1.1.1.cmml">−</mo><mn mathbackground="#FFB2B2" id="S4.T2.22.22.22.7.m1.1.1.2" xref="S4.T2.22.22.22.7.m1.1.1.2.cmml">7.03</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.T2.22.22.22.7.m1.1b"><apply id="S4.T2.22.22.22.7.m1.1.1.cmml" xref="S4.T2.22.22.22.7.m1.1.1"><minus id="S4.T2.22.22.22.7.m1.1.1.1.cmml" xref="S4.T2.22.22.22.7.m1.1.1"></minus><cn type="float" id="S4.T2.22.22.22.7.m1.1.1.2.cmml" xref="S4.T2.22.22.22.7.m1.1.1.2">7.03</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.22.22.22.7.m1.1c">-7.03</annotation></semantics></math></td>
</tr>
<tr id="S4.T2.29.29.29" class="ltx_tr">
<td id="S4.T2.29.29.29.8" class="ltx_td ltx_align_left" style="padding-left:2.0pt;padding-right:2.0pt;">LLaMA-2-7B-Chat-Replay</td>
<td id="S4.T2.23.23.23.1" class="ltx_td ltx_align_center" style="background-color:#F7F7FF;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.23.23.23.1.m1.1" class="ltx_Math" style="background-color:#F7F7FF;" alttext="47.04" display="inline"><semantics id="S4.T2.23.23.23.1.m1.1a"><mn mathbackground="#F7F7FF" id="S4.T2.23.23.23.1.m1.1.1" xref="S4.T2.23.23.23.1.m1.1.1.cmml">47.04</mn><annotation-xml encoding="MathML-Content" id="S4.T2.23.23.23.1.m1.1b"><cn type="float" id="S4.T2.23.23.23.1.m1.1.1.cmml" xref="S4.T2.23.23.23.1.m1.1.1">47.04</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.23.23.23.1.m1.1c">47.04</annotation></semantics></math></td>
<td id="S4.T2.24.24.24.2" class="ltx_td ltx_align_center" style="background-color:#FF8888;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.24.24.24.2.m1.1" class="ltx_Math" style="background-color:#FF8888;" alttext="3.03" display="inline"><semantics id="S4.T2.24.24.24.2.m1.1a"><mn mathbackground="#FF8888" id="S4.T2.24.24.24.2.m1.1.1" xref="S4.T2.24.24.24.2.m1.1.1.cmml">3.03</mn><annotation-xml encoding="MathML-Content" id="S4.T2.24.24.24.2.m1.1b"><cn type="float" id="S4.T2.24.24.24.2.m1.1.1.cmml" xref="S4.T2.24.24.24.2.m1.1.1">3.03</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.24.24.24.2.m1.1c">3.03</annotation></semantics></math></td>
<td id="S4.T2.25.25.25.3" class="ltx_td ltx_align_center" style="background-color:#FFDADA;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.25.25.25.3.m1.1" class="ltx_Math" style="background-color:#FFDADA;" alttext="36.61" display="inline"><semantics id="S4.T2.25.25.25.3.m1.1a"><mn mathbackground="#FFDADA" id="S4.T2.25.25.25.3.m1.1.1" xref="S4.T2.25.25.25.3.m1.1.1.cmml">36.61</mn><annotation-xml encoding="MathML-Content" id="S4.T2.25.25.25.3.m1.1b"><cn type="float" id="S4.T2.25.25.25.3.m1.1.1.cmml" xref="S4.T2.25.25.25.3.m1.1.1">36.61</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.25.25.25.3.m1.1c">36.61</annotation></semantics></math></td>
<td id="S4.T2.26.26.26.4" class="ltx_td ltx_align_center" style="background-color:#DADAFF;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.26.26.26.4.m1.1" class="ltx_Math" style="background-color:#DADAFF;" alttext="31.57" display="inline"><semantics id="S4.T2.26.26.26.4.m1.1a"><mn mathbackground="#DADAFF" id="S4.T2.26.26.26.4.m1.1.1" xref="S4.T2.26.26.26.4.m1.1.1.cmml">31.57</mn><annotation-xml encoding="MathML-Content" id="S4.T2.26.26.26.4.m1.1b"><cn type="float" id="S4.T2.26.26.26.4.m1.1.1.cmml" xref="S4.T2.26.26.26.4.m1.1.1">31.57</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.26.26.26.4.m1.1c">31.57</annotation></semantics></math></td>
<td id="S4.T2.27.27.27.5" class="ltx_td ltx_align_center" style="background-color:#E0E0FF;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.27.27.27.5.m1.1" class="ltx_Math" style="background-color:#E0E0FF;" alttext="75.75" display="inline"><semantics id="S4.T2.27.27.27.5.m1.1a"><mn mathbackground="#E0E0FF" id="S4.T2.27.27.27.5.m1.1.1" xref="S4.T2.27.27.27.5.m1.1.1.cmml">75.75</mn><annotation-xml encoding="MathML-Content" id="S4.T2.27.27.27.5.m1.1b"><cn type="float" id="S4.T2.27.27.27.5.m1.1.1.cmml" xref="S4.T2.27.27.27.5.m1.1.1">75.75</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.27.27.27.5.m1.1c">75.75</annotation></semantics></math></td>
<td id="S4.T2.28.28.28.6" class="ltx_td ltx_align_center" style="background-color:#FFF8F8;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.28.28.28.6.m1.1" class="ltx_Math" style="background-color:#FFF8F8;" alttext="75.3" display="inline"><semantics id="S4.T2.28.28.28.6.m1.1a"><mn mathbackground="#FFF8F8" id="S4.T2.28.28.28.6.m1.1.1" xref="S4.T2.28.28.28.6.m1.1.1.cmml">75.3</mn><annotation-xml encoding="MathML-Content" id="S4.T2.28.28.28.6.m1.1b"><cn type="float" id="S4.T2.28.28.28.6.m1.1.1.cmml" xref="S4.T2.28.28.28.6.m1.1.1">75.3</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.28.28.28.6.m1.1c">75.3</annotation></semantics></math></td>
<td id="S4.T2.29.29.29.7" class="ltx_td ltx_nopad_r ltx_align_center" style="background-color:#FFDCDC;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.29.29.29.7.m1.1" class="ltx_Math" style="background-color:#FFDCDC;" alttext="-2.31" display="inline"><semantics id="S4.T2.29.29.29.7.m1.1a"><mrow id="S4.T2.29.29.29.7.m1.1.1" xref="S4.T2.29.29.29.7.m1.1.1.cmml"><mo mathbackground="#FFDCDC" id="S4.T2.29.29.29.7.m1.1.1a" xref="S4.T2.29.29.29.7.m1.1.1.cmml">−</mo><mn mathbackground="#FFDCDC" id="S4.T2.29.29.29.7.m1.1.1.2" xref="S4.T2.29.29.29.7.m1.1.1.2.cmml">2.31</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.T2.29.29.29.7.m1.1b"><apply id="S4.T2.29.29.29.7.m1.1.1.cmml" xref="S4.T2.29.29.29.7.m1.1.1"><minus id="S4.T2.29.29.29.7.m1.1.1.1.cmml" xref="S4.T2.29.29.29.7.m1.1.1"></minus><cn type="float" id="S4.T2.29.29.29.7.m1.1.1.2.cmml" xref="S4.T2.29.29.29.7.m1.1.1.2">2.31</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.29.29.29.7.m1.1c">-2.31</annotation></semantics></math></td>
</tr>
<tr id="S4.T2.36.36.36" class="ltx_tr">
<td id="S4.T2.36.36.36.8" class="ltx_td ltx_align_left ltx_border_t" style="padding-left:2.0pt;padding-right:2.0pt;">LLaMA-2-13B-Chat</td>
<td id="S4.T2.30.30.30.1" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.30.30.30.1.m1.1" class="ltx_Math" alttext="54.61" display="inline"><semantics id="S4.T2.30.30.30.1.m1.1a"><mn id="S4.T2.30.30.30.1.m1.1.1" xref="S4.T2.30.30.30.1.m1.1.1.cmml">54.61</mn><annotation-xml encoding="MathML-Content" id="S4.T2.30.30.30.1.m1.1b"><cn type="float" id="S4.T2.30.30.30.1.m1.1.1.cmml" xref="S4.T2.30.30.30.1.m1.1.1">54.61</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.30.30.30.1.m1.1c">54.61</annotation></semantics></math></td>
<td id="S4.T2.31.31.31.2" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.31.31.31.2.m1.1" class="ltx_Math" alttext="43.14" display="inline"><semantics id="S4.T2.31.31.31.2.m1.1a"><mn id="S4.T2.31.31.31.2.m1.1.1" xref="S4.T2.31.31.31.2.m1.1.1.cmml">43.14</mn><annotation-xml encoding="MathML-Content" id="S4.T2.31.31.31.2.m1.1b"><cn type="float" id="S4.T2.31.31.31.2.m1.1.1.cmml" xref="S4.T2.31.31.31.2.m1.1.1">43.14</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.31.31.31.2.m1.1c">43.14</annotation></semantics></math></td>
<td id="S4.T2.32.32.32.3" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.32.32.32.3.m1.1" class="ltx_Math" alttext="49.70" display="inline"><semantics id="S4.T2.32.32.32.3.m1.1a"><mn id="S4.T2.32.32.32.3.m1.1.1" xref="S4.T2.32.32.32.3.m1.1.1.cmml">49.70</mn><annotation-xml encoding="MathML-Content" id="S4.T2.32.32.32.3.m1.1b"><cn type="float" id="S4.T2.32.32.32.3.m1.1.1.cmml" xref="S4.T2.32.32.32.3.m1.1.1">49.70</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.32.32.32.3.m1.1c">49.70</annotation></semantics></math></td>
<td id="S4.T2.33.33.33.4" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.33.33.33.4.m1.1" class="ltx_Math" alttext="27.65" display="inline"><semantics id="S4.T2.33.33.33.4.m1.1a"><mn id="S4.T2.33.33.33.4.m1.1.1" xref="S4.T2.33.33.33.4.m1.1.1.cmml">27.65</mn><annotation-xml encoding="MathML-Content" id="S4.T2.33.33.33.4.m1.1b"><cn type="float" id="S4.T2.33.33.33.4.m1.1.1.cmml" xref="S4.T2.33.33.33.4.m1.1.1">27.65</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.33.33.33.4.m1.1c">27.65</annotation></semantics></math></td>
<td id="S4.T2.34.34.34.5" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.34.34.34.5.m1.1" class="ltx_Math" alttext="81.5" display="inline"><semantics id="S4.T2.34.34.34.5.m1.1a"><mn id="S4.T2.34.34.34.5.m1.1.1" xref="S4.T2.34.34.34.5.m1.1.1.cmml">81.5</mn><annotation-xml encoding="MathML-Content" id="S4.T2.34.34.34.5.m1.1b"><cn type="float" id="S4.T2.34.34.34.5.m1.1.1.cmml" xref="S4.T2.34.34.34.5.m1.1.1">81.5</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.34.34.34.5.m1.1c">81.5</annotation></semantics></math></td>
<td id="S4.T2.35.35.35.6" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.35.35.35.6.m1.1" class="ltx_Math" alttext="78.24" display="inline"><semantics id="S4.T2.35.35.35.6.m1.1a"><mn id="S4.T2.35.35.35.6.m1.1.1" xref="S4.T2.35.35.35.6.m1.1.1.cmml">78.24</mn><annotation-xml encoding="MathML-Content" id="S4.T2.35.35.35.6.m1.1b"><cn type="float" id="S4.T2.35.35.35.6.m1.1.1.cmml" xref="S4.T2.35.35.35.6.m1.1.1">78.24</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.35.35.35.6.m1.1c">78.24</annotation></semantics></math></td>
<td id="S4.T2.36.36.36.7" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_t" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.36.36.36.7.m1.1" class="ltx_Math" alttext="0" display="inline"><semantics id="S4.T2.36.36.36.7.m1.1a"><mn id="S4.T2.36.36.36.7.m1.1.1" xref="S4.T2.36.36.36.7.m1.1.1.cmml">0</mn><annotation-xml encoding="MathML-Content" id="S4.T2.36.36.36.7.m1.1b"><cn type="integer" id="S4.T2.36.36.36.7.m1.1.1.cmml" xref="S4.T2.36.36.36.7.m1.1.1">0</cn></annotation-xml></semantics></math></td>
</tr>
<tr id="S4.T2.43.43.43" class="ltx_tr">
<td id="S4.T2.43.43.43.8" class="ltx_td ltx_align_left" style="padding-left:2.0pt;padding-right:2.0pt;">LLaMA-2-13B-Chat-Seq</td>
<td id="S4.T2.37.37.37.1" class="ltx_td ltx_align_center" style="background-color:#FFB0B0;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.37.37.37.1.m1.1" class="ltx_Math" style="background-color:#FFB0B0;" alttext="41.88" display="inline"><semantics id="S4.T2.37.37.37.1.m1.1a"><mn mathbackground="#FFB0B0" id="S4.T2.37.37.37.1.m1.1.1" xref="S4.T2.37.37.37.1.m1.1.1.cmml">41.88</mn><annotation-xml encoding="MathML-Content" id="S4.T2.37.37.37.1.m1.1b"><cn type="float" id="S4.T2.37.37.37.1.m1.1.1.cmml" xref="S4.T2.37.37.37.1.m1.1.1">41.88</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.37.37.37.1.m1.1c">41.88</annotation></semantics></math></td>
<td id="S4.T2.38.38.38.2" class="ltx_td ltx_align_center" style="background-color:#FF7070;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.38.38.38.2.m1.1" class="ltx_Math" style="background-color:#FF7070;" alttext="2.12" display="inline"><semantics id="S4.T2.38.38.38.2.m1.1a"><mn mathbackground="#FF7070" id="S4.T2.38.38.38.2.m1.1.1" xref="S4.T2.38.38.38.2.m1.1.1.cmml">2.12</mn><annotation-xml encoding="MathML-Content" id="S4.T2.38.38.38.2.m1.1b"><cn type="float" id="S4.T2.38.38.38.2.m1.1.1.cmml" xref="S4.T2.38.38.38.2.m1.1.1">2.12</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.38.38.38.2.m1.1c">2.12</annotation></semantics></math></td>
<td id="S4.T2.39.39.39.3" class="ltx_td ltx_align_center" style="background-color:#FF9090;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.39.39.39.3.m1.1" class="ltx_Math" style="background-color:#FF9090;" alttext="19.47" display="inline"><semantics id="S4.T2.39.39.39.3.m1.1a"><mn mathbackground="#FF9090" id="S4.T2.39.39.39.3.m1.1.1" xref="S4.T2.39.39.39.3.m1.1.1.cmml">19.47</mn><annotation-xml encoding="MathML-Content" id="S4.T2.39.39.39.3.m1.1b"><cn type="float" id="S4.T2.39.39.39.3.m1.1.1.cmml" xref="S4.T2.39.39.39.3.m1.1.1">19.47</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.39.39.39.3.m1.1c">19.47</annotation></semantics></math></td>
<td id="S4.T2.40.40.40.4" class="ltx_td ltx_align_center" style="background-color:#EAEAFF;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.40.40.40.4.m1.1" class="ltx_Math" style="background-color:#EAEAFF;" alttext="32.27" display="inline"><semantics id="S4.T2.40.40.40.4.m1.1a"><mn mathbackground="#EAEAFF" id="S4.T2.40.40.40.4.m1.1.1" xref="S4.T2.40.40.40.4.m1.1.1.cmml">32.27</mn><annotation-xml encoding="MathML-Content" id="S4.T2.40.40.40.4.m1.1b"><cn type="float" id="S4.T2.40.40.40.4.m1.1.1.cmml" xref="S4.T2.40.40.40.4.m1.1.1">32.27</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.40.40.40.4.m1.1c">32.27</annotation></semantics></math></td>
<td id="S4.T2.41.41.41.5" class="ltx_td ltx_align_center" style="background-color:#F7F7FF;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.41.41.41.5.m1.1" class="ltx_Math" style="background-color:#F7F7FF;" alttext="82.08" display="inline"><semantics id="S4.T2.41.41.41.5.m1.1a"><mn mathbackground="#F7F7FF" id="S4.T2.41.41.41.5.m1.1.1" xref="S4.T2.41.41.41.5.m1.1.1.cmml">82.08</mn><annotation-xml encoding="MathML-Content" id="S4.T2.41.41.41.5.m1.1b"><cn type="float" id="S4.T2.41.41.41.5.m1.1.1.cmml" xref="S4.T2.41.41.41.5.m1.1.1">82.08</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.41.41.41.5.m1.1c">82.08</annotation></semantics></math></td>
<td id="S4.T2.42.42.42.6" class="ltx_td ltx_align_center" style="background-color:#FFF8F8;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.42.42.42.6.m1.1" class="ltx_Math" style="background-color:#FFF8F8;" alttext="77.15" display="inline"><semantics id="S4.T2.42.42.42.6.m1.1a"><mn mathbackground="#FFF8F8" id="S4.T2.42.42.42.6.m1.1.1" xref="S4.T2.42.42.42.6.m1.1.1.cmml">77.15</mn><annotation-xml encoding="MathML-Content" id="S4.T2.42.42.42.6.m1.1b"><cn type="float" id="S4.T2.42.42.42.6.m1.1.1.cmml" xref="S4.T2.42.42.42.6.m1.1.1">77.15</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.42.42.42.6.m1.1c">77.15</annotation></semantics></math></td>
<td id="S4.T2.43.43.43.7" class="ltx_td ltx_nopad_r ltx_align_center" style="background-color:#FFADAD;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.43.43.43.7.m1.1" class="ltx_Math" style="background-color:#FFADAD;" alttext="-13.32" display="inline"><semantics id="S4.T2.43.43.43.7.m1.1a"><mrow id="S4.T2.43.43.43.7.m1.1.1" xref="S4.T2.43.43.43.7.m1.1.1.cmml"><mo mathbackground="#FFADAD" id="S4.T2.43.43.43.7.m1.1.1a" xref="S4.T2.43.43.43.7.m1.1.1.cmml">−</mo><mn mathbackground="#FFADAD" id="S4.T2.43.43.43.7.m1.1.1.2" xref="S4.T2.43.43.43.7.m1.1.1.2.cmml">13.32</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.T2.43.43.43.7.m1.1b"><apply id="S4.T2.43.43.43.7.m1.1.1.cmml" xref="S4.T2.43.43.43.7.m1.1.1"><minus id="S4.T2.43.43.43.7.m1.1.1.1.cmml" xref="S4.T2.43.43.43.7.m1.1.1"></minus><cn type="float" id="S4.T2.43.43.43.7.m1.1.1.2.cmml" xref="S4.T2.43.43.43.7.m1.1.1.2">13.32</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.43.43.43.7.m1.1c">-13.32</annotation></semantics></math></td>
</tr>
<tr id="S4.T2.50.50.50" class="ltx_tr">
<td id="S4.T2.50.50.50.8" class="ltx_td ltx_align_left" style="padding-left:2.0pt;padding-right:2.0pt;">LLaMA-2-13B-Chat-LoraSeq</td>
<td id="S4.T2.44.44.44.1" class="ltx_td ltx_align_center" style="background-color:#FFDADA;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.44.44.44.1.m1.1" class="ltx_Math" style="background-color:#FFDADA;" alttext="50.63" display="inline"><semantics id="S4.T2.44.44.44.1.m1.1a"><mn mathbackground="#FFDADA" id="S4.T2.44.44.44.1.m1.1.1" xref="S4.T2.44.44.44.1.m1.1.1.cmml">50.63</mn><annotation-xml encoding="MathML-Content" id="S4.T2.44.44.44.1.m1.1b"><cn type="float" id="S4.T2.44.44.44.1.m1.1.1.cmml" xref="S4.T2.44.44.44.1.m1.1.1">50.63</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.44.44.44.1.m1.1c">50.63</annotation></semantics></math></td>
<td id="S4.T2.45.45.45.2" class="ltx_td ltx_align_center" style="background-color:#FF8F8F;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.45.45.45.2.m1.1" class="ltx_Math" style="background-color:#FF8F8F;" alttext="24.72" display="inline"><semantics id="S4.T2.45.45.45.2.m1.1a"><mn mathbackground="#FF8F8F" id="S4.T2.45.45.45.2.m1.1.1" xref="S4.T2.45.45.45.2.m1.1.1.cmml">24.72</mn><annotation-xml encoding="MathML-Content" id="S4.T2.45.45.45.2.m1.1b"><cn type="float" id="S4.T2.45.45.45.2.m1.1.1.cmml" xref="S4.T2.45.45.45.2.m1.1.1">24.72</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.45.45.45.2.m1.1c">24.72</annotation></semantics></math></td>
<td id="S4.T2.46.46.46.3" class="ltx_td ltx_align_center" style="background-color:#FFB5B5;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.46.46.46.3.m1.1" class="ltx_Math" style="background-color:#FFB5B5;" alttext="38.98" display="inline"><semantics id="S4.T2.46.46.46.3.m1.1a"><mn mathbackground="#FFB5B5" id="S4.T2.46.46.46.3.m1.1.1" xref="S4.T2.46.46.46.3.m1.1.1.cmml">38.98</mn><annotation-xml encoding="MathML-Content" id="S4.T2.46.46.46.3.m1.1b"><cn type="float" id="S4.T2.46.46.46.3.m1.1.1.cmml" xref="S4.T2.46.46.46.3.m1.1.1">38.98</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.46.46.46.3.m1.1c">38.98</annotation></semantics></math></td>
<td id="S4.T2.47.47.47.4" class="ltx_td ltx_align_center" style="background-color:#FFF8F8;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.47.47.47.4.m1.1" class="ltx_Math" style="background-color:#FFF8F8;" alttext="26.93" display="inline"><semantics id="S4.T2.47.47.47.4.m1.1a"><mn mathbackground="#FFF8F8" id="S4.T2.47.47.47.4.m1.1.1" xref="S4.T2.47.47.47.4.m1.1.1.cmml">26.93</mn><annotation-xml encoding="MathML-Content" id="S4.T2.47.47.47.4.m1.1b"><cn type="float" id="S4.T2.47.47.47.4.m1.1.1.cmml" xref="S4.T2.47.47.47.4.m1.1.1">26.93</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.47.47.47.4.m1.1c">26.93</annotation></semantics></math></td>
<td id="S4.T2.48.48.48.5" class="ltx_td ltx_align_center" style="background-color:#FFAAAA;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.48.48.48.5.m1.1" class="ltx_Math" style="background-color:#FFAAAA;" alttext="68.96" display="inline"><semantics id="S4.T2.48.48.48.5.m1.1a"><mn mathbackground="#FFAAAA" id="S4.T2.48.48.48.5.m1.1.1" xref="S4.T2.48.48.48.5.m1.1.1.cmml">68.96</mn><annotation-xml encoding="MathML-Content" id="S4.T2.48.48.48.5.m1.1b"><cn type="float" id="S4.T2.48.48.48.5.m1.1.1.cmml" xref="S4.T2.48.48.48.5.m1.1.1">68.96</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.48.48.48.5.m1.1c">68.96</annotation></semantics></math></td>
<td id="S4.T2.49.49.49.6" class="ltx_td ltx_align_center" style="background-color:#FFFBFB;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.49.49.49.6.m1.1" class="ltx_Math" style="background-color:#FFFBFB;" alttext="78.02" display="inline"><semantics id="S4.T2.49.49.49.6.m1.1a"><mn mathbackground="#FFFBFB" id="S4.T2.49.49.49.6.m1.1.1" xref="S4.T2.49.49.49.6.m1.1.1.cmml">78.02</mn><annotation-xml encoding="MathML-Content" id="S4.T2.49.49.49.6.m1.1b"><cn type="float" id="S4.T2.49.49.49.6.m1.1.1.cmml" xref="S4.T2.49.49.49.6.m1.1.1">78.02</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.49.49.49.6.m1.1c">78.02</annotation></semantics></math></td>
<td id="S4.T2.50.50.50.7" class="ltx_td ltx_nopad_r ltx_align_center" style="background-color:#FFCACA;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.50.50.50.7.m1.1" class="ltx_Math" style="background-color:#FFCACA;" alttext="-7.78" display="inline"><semantics id="S4.T2.50.50.50.7.m1.1a"><mrow id="S4.T2.50.50.50.7.m1.1.1" xref="S4.T2.50.50.50.7.m1.1.1.cmml"><mo mathbackground="#FFCACA" id="S4.T2.50.50.50.7.m1.1.1a" xref="S4.T2.50.50.50.7.m1.1.1.cmml">−</mo><mn mathbackground="#FFCACA" id="S4.T2.50.50.50.7.m1.1.1.2" xref="S4.T2.50.50.50.7.m1.1.1.2.cmml">7.78</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.T2.50.50.50.7.m1.1b"><apply id="S4.T2.50.50.50.7.m1.1.1.cmml" xref="S4.T2.50.50.50.7.m1.1.1"><minus id="S4.T2.50.50.50.7.m1.1.1.1.cmml" xref="S4.T2.50.50.50.7.m1.1.1"></minus><cn type="float" id="S4.T2.50.50.50.7.m1.1.1.2.cmml" xref="S4.T2.50.50.50.7.m1.1.1.2">7.78</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.50.50.50.7.m1.1c">-7.78</annotation></semantics></math></td>
</tr>
<tr id="S4.T2.57.57.57" class="ltx_tr">
<td id="S4.T2.57.57.57.8" class="ltx_td ltx_align_left" style="padding-left:2.0pt;padding-right:2.0pt;">LLaMA-2-13B-Chat-Replay</td>
<td id="S4.T2.51.51.51.1" class="ltx_td ltx_align_center" style="background-color:#FFD4D4;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.51.51.51.1.m1.1" class="ltx_Math" style="background-color:#FFD4D4;" alttext="47.72" display="inline"><semantics id="S4.T2.51.51.51.1.m1.1a"><mn mathbackground="#FFD4D4" id="S4.T2.51.51.51.1.m1.1.1" xref="S4.T2.51.51.51.1.m1.1.1.cmml">47.72</mn><annotation-xml encoding="MathML-Content" id="S4.T2.51.51.51.1.m1.1b"><cn type="float" id="S4.T2.51.51.51.1.m1.1.1.cmml" xref="S4.T2.51.51.51.1.m1.1.1">47.72</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.51.51.51.1.m1.1c">47.72</annotation></semantics></math></td>
<td id="S4.T2.52.52.52.2" class="ltx_td ltx_align_center" style="background-color:#FF7373;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.52.52.52.2.m1.1" class="ltx_Math" style="background-color:#FF7373;" alttext="2.96" display="inline"><semantics id="S4.T2.52.52.52.2.m1.1a"><mn mathbackground="#FF7373" id="S4.T2.52.52.52.2.m1.1.1" xref="S4.T2.52.52.52.2.m1.1.1.cmml">2.96</mn><annotation-xml encoding="MathML-Content" id="S4.T2.52.52.52.2.m1.1b"><cn type="float" id="S4.T2.52.52.52.2.m1.1.1.cmml" xref="S4.T2.52.52.52.2.m1.1.1">2.96</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.52.52.52.2.m1.1c">2.96</annotation></semantics></math></td>
<td id="S4.T2.53.53.53.3" class="ltx_td ltx_align_center" style="background-color:#FFAEAE;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.53.53.53.3.m1.1" class="ltx_Math" style="background-color:#FFAEAE;" alttext="36.52" display="inline"><semantics id="S4.T2.53.53.53.3.m1.1a"><mn mathbackground="#FFAEAE" id="S4.T2.53.53.53.3.m1.1.1" xref="S4.T2.53.53.53.3.m1.1.1.cmml">36.52</mn><annotation-xml encoding="MathML-Content" id="S4.T2.53.53.53.3.m1.1b"><cn type="float" id="S4.T2.53.53.53.3.m1.1.1.cmml" xref="S4.T2.53.53.53.3.m1.1.1">36.52</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.53.53.53.3.m1.1c">36.52</annotation></semantics></math></td>
<td id="S4.T2.54.54.54.4" class="ltx_td ltx_align_center" style="background-color:#EAEAFF;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.54.54.54.4.m1.1" class="ltx_Math" style="background-color:#EAEAFF;" alttext="32.52" display="inline"><semantics id="S4.T2.54.54.54.4.m1.1a"><mn mathbackground="#EAEAFF" id="S4.T2.54.54.54.4.m1.1.1" xref="S4.T2.54.54.54.4.m1.1.1.cmml">32.52</mn><annotation-xml encoding="MathML-Content" id="S4.T2.54.54.54.4.m1.1b"><cn type="float" id="S4.T2.54.54.54.4.m1.1.1.cmml" xref="S4.T2.54.54.54.4.m1.1.1">32.52</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.54.54.54.4.m1.1c">32.52</annotation></semantics></math></td>
<td id="S4.T2.55.55.55.5" class="ltx_td ltx_align_center" style="background-color:#F7F7FF;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.55.55.55.5.m1.1" class="ltx_Math" style="background-color:#F7F7FF;" alttext="82.45" display="inline"><semantics id="S4.T2.55.55.55.5.m1.1a"><mn mathbackground="#F7F7FF" id="S4.T2.55.55.55.5.m1.1.1" xref="S4.T2.55.55.55.5.m1.1.1.cmml">82.45</mn><annotation-xml encoding="MathML-Content" id="S4.T2.55.55.55.5.m1.1b"><cn type="float" id="S4.T2.55.55.55.5.m1.1.1.cmml" xref="S4.T2.55.55.55.5.m1.1.1">82.45</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.55.55.55.5.m1.1c">82.45</annotation></semantics></math></td>
<td id="S4.T2.56.56.56.6" class="ltx_td ltx_align_center" style="background-color:#FFF6F6;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.56.56.56.6.m1.1" class="ltx_Math" style="background-color:#FFF6F6;" alttext="76.88" display="inline"><semantics id="S4.T2.56.56.56.6.m1.1a"><mn mathbackground="#FFF6F6" id="S4.T2.56.56.56.6.m1.1.1" xref="S4.T2.56.56.56.6.m1.1.1.cmml">76.88</mn><annotation-xml encoding="MathML-Content" id="S4.T2.56.56.56.6.m1.1b"><cn type="float" id="S4.T2.56.56.56.6.m1.1.1.cmml" xref="S4.T2.56.56.56.6.m1.1.1">76.88</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.56.56.56.6.m1.1c">76.88</annotation></semantics></math></td>
<td id="S4.T2.57.57.57.7" class="ltx_td ltx_nopad_r ltx_align_center" style="background-color:#FFC6C6;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.57.57.57.7.m1.1" class="ltx_Math" style="background-color:#FFC6C6;" alttext="-9.3" display="inline"><semantics id="S4.T2.57.57.57.7.m1.1a"><mrow id="S4.T2.57.57.57.7.m1.1.1" xref="S4.T2.57.57.57.7.m1.1.1.cmml"><mo mathbackground="#FFC6C6" id="S4.T2.57.57.57.7.m1.1.1a" xref="S4.T2.57.57.57.7.m1.1.1.cmml">−</mo><mn mathbackground="#FFC6C6" id="S4.T2.57.57.57.7.m1.1.1.2" xref="S4.T2.57.57.57.7.m1.1.1.2.cmml">9.3</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.T2.57.57.57.7.m1.1b"><apply id="S4.T2.57.57.57.7.m1.1.1.cmml" xref="S4.T2.57.57.57.7.m1.1.1"><minus id="S4.T2.57.57.57.7.m1.1.1.1.cmml" xref="S4.T2.57.57.57.7.m1.1.1"></minus><cn type="float" id="S4.T2.57.57.57.7.m1.1.1.2.cmml" xref="S4.T2.57.57.57.7.m1.1.1.2">9.3</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.57.57.57.7.m1.1c">-9.3</annotation></semantics></math></td>
</tr>
<tr id="S4.T2.64.64.64" class="ltx_tr">
<td id="S4.T2.64.64.64.8" class="ltx_td ltx_align_left ltx_border_t" style="padding-left:2.0pt;padding-right:2.0pt;">Baichuan2-7B-Instruct</td>
<td id="S4.T2.58.58.58.1" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.58.58.58.1.m1.1" class="ltx_Math" alttext="53.80" display="inline"><semantics id="S4.T2.58.58.58.1.m1.1a"><mn id="S4.T2.58.58.58.1.m1.1.1" xref="S4.T2.58.58.58.1.m1.1.1.cmml">53.80</mn><annotation-xml encoding="MathML-Content" id="S4.T2.58.58.58.1.m1.1b"><cn type="float" id="S4.T2.58.58.58.1.m1.1.1.cmml" xref="S4.T2.58.58.58.1.m1.1.1">53.80</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.58.58.58.1.m1.1c">53.80</annotation></semantics></math></td>
<td id="S4.T2.59.59.59.2" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.59.59.59.2.m1.1" class="ltx_Math" alttext="33.21" display="inline"><semantics id="S4.T2.59.59.59.2.m1.1a"><mn id="S4.T2.59.59.59.2.m1.1.1" xref="S4.T2.59.59.59.2.m1.1.1.cmml">33.21</mn><annotation-xml encoding="MathML-Content" id="S4.T2.59.59.59.2.m1.1b"><cn type="float" id="S4.T2.59.59.59.2.m1.1.1.cmml" xref="S4.T2.59.59.59.2.m1.1.1">33.21</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.59.59.59.2.m1.1c">33.21</annotation></semantics></math></td>
<td id="S4.T2.60.60.60.3" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.60.60.60.3.m1.1" class="ltx_Math" alttext="35.66" display="inline"><semantics id="S4.T2.60.60.60.3.m1.1a"><mn id="S4.T2.60.60.60.3.m1.1.1" xref="S4.T2.60.60.60.3.m1.1.1.cmml">35.66</mn><annotation-xml encoding="MathML-Content" id="S4.T2.60.60.60.3.m1.1b"><cn type="float" id="S4.T2.60.60.60.3.m1.1.1.cmml" xref="S4.T2.60.60.60.3.m1.1.1">35.66</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.60.60.60.3.m1.1c">35.66</annotation></semantics></math></td>
<td id="S4.T2.61.61.61.4" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.61.61.61.4.m1.1" class="ltx_Math" alttext="20.64" display="inline"><semantics id="S4.T2.61.61.61.4.m1.1a"><mn id="S4.T2.61.61.61.4.m1.1.1" xref="S4.T2.61.61.61.4.m1.1.1.cmml">20.64</mn><annotation-xml encoding="MathML-Content" id="S4.T2.61.61.61.4.m1.1b"><cn type="float" id="S4.T2.61.61.61.4.m1.1.1.cmml" xref="S4.T2.61.61.61.4.m1.1.1">20.64</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.61.61.61.4.m1.1c">20.64</annotation></semantics></math></td>
<td id="S4.T2.62.62.62.5" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.62.62.62.5.m1.1" class="ltx_Math" alttext="77.09" display="inline"><semantics id="S4.T2.62.62.62.5.m1.1a"><mn id="S4.T2.62.62.62.5.m1.1.1" xref="S4.T2.62.62.62.5.m1.1.1.cmml">77.09</mn><annotation-xml encoding="MathML-Content" id="S4.T2.62.62.62.5.m1.1b"><cn type="float" id="S4.T2.62.62.62.5.m1.1.1.cmml" xref="S4.T2.62.62.62.5.m1.1.1">77.09</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.62.62.62.5.m1.1c">77.09</annotation></semantics></math></td>
<td id="S4.T2.63.63.63.6" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.63.63.63.6.m1.1" class="ltx_Math" alttext="74.05" display="inline"><semantics id="S4.T2.63.63.63.6.m1.1a"><mn id="S4.T2.63.63.63.6.m1.1.1" xref="S4.T2.63.63.63.6.m1.1.1.cmml">74.05</mn><annotation-xml encoding="MathML-Content" id="S4.T2.63.63.63.6.m1.1b"><cn type="float" id="S4.T2.63.63.63.6.m1.1.1.cmml" xref="S4.T2.63.63.63.6.m1.1.1">74.05</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.63.63.63.6.m1.1c">74.05</annotation></semantics></math></td>
<td id="S4.T2.64.64.64.7" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_t" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.64.64.64.7.m1.1" class="ltx_Math" alttext="0" display="inline"><semantics id="S4.T2.64.64.64.7.m1.1a"><mn id="S4.T2.64.64.64.7.m1.1.1" xref="S4.T2.64.64.64.7.m1.1.1.cmml">0</mn><annotation-xml encoding="MathML-Content" id="S4.T2.64.64.64.7.m1.1b"><cn type="integer" id="S4.T2.64.64.64.7.m1.1.1.cmml" xref="S4.T2.64.64.64.7.m1.1.1">0</cn></annotation-xml></semantics></math></td>
</tr>
<tr id="S4.T2.71.71.71" class="ltx_tr">
<td id="S4.T2.71.71.71.8" class="ltx_td ltx_align_left" style="padding-left:2.0pt;padding-right:2.0pt;">Baichuan2-7B-Instruct-Seq</td>
<td id="S4.T2.65.65.65.1" class="ltx_td ltx_align_center" style="background-color:#FFDDDD;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.65.65.65.1.m1.1" class="ltx_Math" style="background-color:#FFDDDD;" alttext="46.92" display="inline"><semantics id="S4.T2.65.65.65.1.m1.1a"><mn mathbackground="#FFDDDD" id="S4.T2.65.65.65.1.m1.1.1" xref="S4.T2.65.65.65.1.m1.1.1.cmml">46.92</mn><annotation-xml encoding="MathML-Content" id="S4.T2.65.65.65.1.m1.1b"><cn type="float" id="S4.T2.65.65.65.1.m1.1.1.cmml" xref="S4.T2.65.65.65.1.m1.1.1">46.92</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.65.65.65.1.m1.1c">46.92</annotation></semantics></math></td>
<td id="S4.T2.66.66.66.2" class="ltx_td ltx_align_center" style="background-color:#FF8B8B;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.66.66.66.2.m1.1" class="ltx_Math" style="background-color:#FF8B8B;" alttext="4.25" display="inline"><semantics id="S4.T2.66.66.66.2.m1.1a"><mn mathbackground="#FF8B8B" id="S4.T2.66.66.66.2.m1.1.1" xref="S4.T2.66.66.66.2.m1.1.1.cmml">4.25</mn><annotation-xml encoding="MathML-Content" id="S4.T2.66.66.66.2.m1.1b"><cn type="float" id="S4.T2.66.66.66.2.m1.1.1.cmml" xref="S4.T2.66.66.66.2.m1.1.1">4.25</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.66.66.66.2.m1.1c">4.25</annotation></semantics></math></td>
<td id="S4.T2.67.67.67.3" class="ltx_td ltx_align_center" style="background-color:#F0F0FF;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.67.67.67.3.m1.1" class="ltx_Math" style="background-color:#F0F0FF;" alttext="37.45" display="inline"><semantics id="S4.T2.67.67.67.3.m1.1a"><mn mathbackground="#F0F0FF" id="S4.T2.67.67.67.3.m1.1.1" xref="S4.T2.67.67.67.3.m1.1.1.cmml">37.45</mn><annotation-xml encoding="MathML-Content" id="S4.T2.67.67.67.3.m1.1b"><cn type="float" id="S4.T2.67.67.67.3.m1.1.1.cmml" xref="S4.T2.67.67.67.3.m1.1.1">37.45</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.67.67.67.3.m1.1c">37.45</annotation></semantics></math></td>
<td id="S4.T2.68.68.68.4" class="ltx_td ltx_align_center" style="background-color:#B5B5FF;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.68.68.68.4.m1.1" class="ltx_Math" style="background-color:#B5B5FF;" alttext="35.20" display="inline"><semantics id="S4.T2.68.68.68.4.m1.1a"><mn mathbackground="#B5B5FF" id="S4.T2.68.68.68.4.m1.1.1" xref="S4.T2.68.68.68.4.m1.1.1.cmml">35.20</mn><annotation-xml encoding="MathML-Content" id="S4.T2.68.68.68.4.m1.1b"><cn type="float" id="S4.T2.68.68.68.4.m1.1.1.cmml" xref="S4.T2.68.68.68.4.m1.1.1">35.20</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.68.68.68.4.m1.1c">35.20</annotation></semantics></math></td>
<td id="S4.T2.69.69.69.5" class="ltx_td ltx_align_center" style="background-color:#EDEFFF;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.69.69.69.5.m1.1" class="ltx_Math" style="background-color:#EDEFFF;" alttext="79.08" display="inline"><semantics id="S4.T2.69.69.69.5.m1.1a"><mn mathbackground="#EDEFFF" id="S4.T2.69.69.69.5.m1.1.1" xref="S4.T2.69.69.69.5.m1.1.1.cmml">79.08</mn><annotation-xml encoding="MathML-Content" id="S4.T2.69.69.69.5.m1.1b"><cn type="float" id="S4.T2.69.69.69.5.m1.1.1.cmml" xref="S4.T2.69.69.69.5.m1.1.1">79.08</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.69.69.69.5.m1.1c">79.08</annotation></semantics></math></td>
<td id="S4.T2.70.70.70.6" class="ltx_td ltx_align_center" style="background-color:#FAFAFF;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.70.70.70.6.m1.1" class="ltx_Math" style="background-color:#FAFAFF;" alttext="74.21" display="inline"><semantics id="S4.T2.70.70.70.6.m1.1a"><mn mathbackground="#FAFAFF" id="S4.T2.70.70.70.6.m1.1.1" xref="S4.T2.70.70.70.6.m1.1.1.cmml">74.21</mn><annotation-xml encoding="MathML-Content" id="S4.T2.70.70.70.6.m1.1b"><cn type="float" id="S4.T2.70.70.70.6.m1.1.1.cmml" xref="S4.T2.70.70.70.6.m1.1.1">74.21</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.70.70.70.6.m1.1c">74.21</annotation></semantics></math></td>
<td id="S4.T2.71.71.71.7" class="ltx_td ltx_nopad_r ltx_align_center" style="background-color:#FFD7D7;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.71.71.71.7.m1.1" class="ltx_Math" style="background-color:#FFD7D7;" alttext="-2.89" display="inline"><semantics id="S4.T2.71.71.71.7.m1.1a"><mrow id="S4.T2.71.71.71.7.m1.1.1" xref="S4.T2.71.71.71.7.m1.1.1.cmml"><mo mathbackground="#FFD7D7" id="S4.T2.71.71.71.7.m1.1.1a" xref="S4.T2.71.71.71.7.m1.1.1.cmml">−</mo><mn mathbackground="#FFD7D7" id="S4.T2.71.71.71.7.m1.1.1.2" xref="S4.T2.71.71.71.7.m1.1.1.2.cmml">2.89</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.T2.71.71.71.7.m1.1b"><apply id="S4.T2.71.71.71.7.m1.1.1.cmml" xref="S4.T2.71.71.71.7.m1.1.1"><minus id="S4.T2.71.71.71.7.m1.1.1.1.cmml" xref="S4.T2.71.71.71.7.m1.1.1"></minus><cn type="float" id="S4.T2.71.71.71.7.m1.1.1.2.cmml" xref="S4.T2.71.71.71.7.m1.1.1.2">2.89</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.71.71.71.7.m1.1c">-2.89</annotation></semantics></math></td>
</tr>
<tr id="S4.T2.78.78.78" class="ltx_tr">
<td id="S4.T2.78.78.78.8" class="ltx_td ltx_align_left" style="padding-left:2.0pt;padding-right:2.0pt;">Baichuan2-7B-Instruct-LoraSeq</td>
<td id="S4.T2.72.72.72.1" class="ltx_td ltx_align_center" style="background-color:#FFD9D9;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.72.72.72.1.m1.1" class="ltx_Math" style="background-color:#FFD9D9;" alttext="52.14" display="inline"><semantics id="S4.T2.72.72.72.1.m1.1a"><mn mathbackground="#FFD9D9" id="S4.T2.72.72.72.1.m1.1.1" xref="S4.T2.72.72.72.1.m1.1.1.cmml">52.14</mn><annotation-xml encoding="MathML-Content" id="S4.T2.72.72.72.1.m1.1b"><cn type="float" id="S4.T2.72.72.72.1.m1.1.1.cmml" xref="S4.T2.72.72.72.1.m1.1.1">52.14</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.72.72.72.1.m1.1c">52.14</annotation></semantics></math></td>
<td id="S4.T2.73.73.73.2" class="ltx_td ltx_align_center" style="background-color:#FFB8B8;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.73.73.73.2.m1.1" class="ltx_Math" style="background-color:#FFB8B8;" alttext="22.74" display="inline"><semantics id="S4.T2.73.73.73.2.m1.1a"><mn mathbackground="#FFB8B8" id="S4.T2.73.73.73.2.m1.1.1" xref="S4.T2.73.73.73.2.m1.1.1.cmml">22.74</mn><annotation-xml encoding="MathML-Content" id="S4.T2.73.73.73.2.m1.1b"><cn type="float" id="S4.T2.73.73.73.2.m1.1.1.cmml" xref="S4.T2.73.73.73.2.m1.1.1">22.74</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.73.73.73.2.m1.1c">22.74</annotation></semantics></math></td>
<td id="S4.T2.74.74.74.3" class="ltx_td ltx_align_center" style="background-color:#FFCDCD;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.74.74.74.3.m1.1" class="ltx_Math" style="background-color:#FFCDCD;" alttext="27.53" display="inline"><semantics id="S4.T2.74.74.74.3.m1.1a"><mn mathbackground="#FFCDCD" id="S4.T2.74.74.74.3.m1.1.1" xref="S4.T2.74.74.74.3.m1.1.1.cmml">27.53</mn><annotation-xml encoding="MathML-Content" id="S4.T2.74.74.74.3.m1.1b"><cn type="float" id="S4.T2.74.74.74.3.m1.1.1.cmml" xref="S4.T2.74.74.74.3.m1.1.1">27.53</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.74.74.74.3.m1.1c">27.53</annotation></semantics></math></td>
<td id="S4.T2.75.75.75.4" class="ltx_td ltx_align_center" style="background-color:#D8D8FF;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.75.75.75.4.m1.1" class="ltx_Math" style="background-color:#D8D8FF;" alttext="30.99" display="inline"><semantics id="S4.T2.75.75.75.4.m1.1a"><mn mathbackground="#D8D8FF" id="S4.T2.75.75.75.4.m1.1.1" xref="S4.T2.75.75.75.4.m1.1.1.cmml">30.99</mn><annotation-xml encoding="MathML-Content" id="S4.T2.75.75.75.4.m1.1b"><cn type="float" id="S4.T2.75.75.75.4.m1.1.1.cmml" xref="S4.T2.75.75.75.4.m1.1.1">30.99</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.75.75.75.4.m1.1c">30.99</annotation></semantics></math></td>
<td id="S4.T2.76.76.76.5" class="ltx_td ltx_align_center" style="background-color:#FFD7D7;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.76.76.76.5.m1.1" class="ltx_Math" style="background-color:#FFD7D7;" alttext="75.23" display="inline"><semantics id="S4.T2.76.76.76.5.m1.1a"><mn mathbackground="#FFD7D7" id="S4.T2.76.76.76.5.m1.1.1" xref="S4.T2.76.76.76.5.m1.1.1.cmml">75.23</mn><annotation-xml encoding="MathML-Content" id="S4.T2.76.76.76.5.m1.1b"><cn type="float" id="S4.T2.76.76.76.5.m1.1.1.cmml" xref="S4.T2.76.76.76.5.m1.1.1">75.23</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.76.76.76.5.m1.1c">75.23</annotation></semantics></math></td>
<td id="S4.T2.77.77.77.6" class="ltx_td ltx_align_center" style="background-color:#F5F5FF;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.77.77.77.6.m1.1" class="ltx_Math" style="background-color:#F5F5FF;" alttext="74.86" display="inline"><semantics id="S4.T2.77.77.77.6.m1.1a"><mn mathbackground="#F5F5FF" id="S4.T2.77.77.77.6.m1.1.1" xref="S4.T2.77.77.77.6.m1.1.1.cmml">74.86</mn><annotation-xml encoding="MathML-Content" id="S4.T2.77.77.77.6.m1.1b"><cn type="float" id="S4.T2.77.77.77.6.m1.1.1.cmml" xref="S4.T2.77.77.77.6.m1.1.1">74.86</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.77.77.77.6.m1.1c">74.86</annotation></semantics></math></td>
<td id="S4.T2.78.78.78.7" class="ltx_td ltx_nopad_r ltx_align_center" style="background-color:#FFE5E5;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.78.78.78.7.m1.1" class="ltx_Math" style="background-color:#FFE5E5;" alttext="-1.83" display="inline"><semantics id="S4.T2.78.78.78.7.m1.1a"><mrow id="S4.T2.78.78.78.7.m1.1.1" xref="S4.T2.78.78.78.7.m1.1.1.cmml"><mo mathbackground="#FFE5E5" id="S4.T2.78.78.78.7.m1.1.1a" xref="S4.T2.78.78.78.7.m1.1.1.cmml">−</mo><mn mathbackground="#FFE5E5" id="S4.T2.78.78.78.7.m1.1.1.2" xref="S4.T2.78.78.78.7.m1.1.1.2.cmml">1.83</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.T2.78.78.78.7.m1.1b"><apply id="S4.T2.78.78.78.7.m1.1.1.cmml" xref="S4.T2.78.78.78.7.m1.1.1"><minus id="S4.T2.78.78.78.7.m1.1.1.1.cmml" xref="S4.T2.78.78.78.7.m1.1.1"></minus><cn type="float" id="S4.T2.78.78.78.7.m1.1.1.2.cmml" xref="S4.T2.78.78.78.7.m1.1.1.2">1.83</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.78.78.78.7.m1.1c">-1.83</annotation></semantics></math></td>
</tr>
<tr id="S4.T2.85.85.85" class="ltx_tr">
<td id="S4.T2.85.85.85.8" class="ltx_td ltx_align_left" style="padding-left:2.0pt;padding-right:2.0pt;">Baichuan2-7B-Instruct-Replay</td>
<td id="S4.T2.79.79.79.1" class="ltx_td ltx_align_center" style="background-color:#FFD6D6;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.79.79.79.1.m1.1" class="ltx_Math" style="background-color:#FFD6D6;" alttext="45.72" display="inline"><semantics id="S4.T2.79.79.79.1.m1.1a"><mn mathbackground="#FFD6D6" id="S4.T2.79.79.79.1.m1.1.1" xref="S4.T2.79.79.79.1.m1.1.1.cmml">45.72</mn><annotation-xml encoding="MathML-Content" id="S4.T2.79.79.79.1.m1.1b"><cn type="float" id="S4.T2.79.79.79.1.m1.1.1.cmml" xref="S4.T2.79.79.79.1.m1.1.1">45.72</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.79.79.79.1.m1.1c">45.72</annotation></semantics></math></td>
<td id="S4.T2.80.80.80.2" class="ltx_td ltx_align_center" style="background-color:#FFA1A1;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.80.80.80.2.m1.1" class="ltx_Math" style="background-color:#FFA1A1;" alttext="8.19" display="inline"><semantics id="S4.T2.80.80.80.2.m1.1a"><mn mathbackground="#FFA1A1" id="S4.T2.80.80.80.2.m1.1.1" xref="S4.T2.80.80.80.2.m1.1.1.cmml">8.19</mn><annotation-xml encoding="MathML-Content" id="S4.T2.80.80.80.2.m1.1b"><cn type="float" id="S4.T2.80.80.80.2.m1.1.1.cmml" xref="S4.T2.80.80.80.2.m1.1.1">8.19</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.80.80.80.2.m1.1c">8.19</annotation></semantics></math></td>
<td id="S4.T2.81.81.81.3" class="ltx_td ltx_align_center" style="background-color:#FFFFFF;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.81.81.81.3.m1.1" class="ltx_Math" style="background-color:#FFFFFF;" alttext="35.61" display="inline"><semantics id="S4.T2.81.81.81.3.m1.1a"><mn mathbackground="#FFFFFF" id="S4.T2.81.81.81.3.m1.1.1" xref="S4.T2.81.81.81.3.m1.1.1.cmml">35.61</mn><annotation-xml encoding="MathML-Content" id="S4.T2.81.81.81.3.m1.1b"><cn type="float" id="S4.T2.81.81.81.3.m1.1.1.cmml" xref="S4.T2.81.81.81.3.m1.1.1">35.61</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.81.81.81.3.m1.1c">35.61</annotation></semantics></math></td>
<td id="S4.T2.82.82.82.4" class="ltx_td ltx_align_center" style="background-color:#B6B6FF;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.82.82.82.4.m1.1" class="ltx_Math" style="background-color:#B6B6FF;" alttext="34.65" display="inline"><semantics id="S4.T2.82.82.82.4.m1.1a"><mn mathbackground="#B6B6FF" id="S4.T2.82.82.82.4.m1.1.1" xref="S4.T2.82.82.82.4.m1.1.1.cmml">34.65</mn><annotation-xml encoding="MathML-Content" id="S4.T2.82.82.82.4.m1.1b"><cn type="float" id="S4.T2.82.82.82.4.m1.1.1.cmml" xref="S4.T2.82.82.82.4.m1.1.1">34.65</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.82.82.82.4.m1.1c">34.65</annotation></semantics></math></td>
<td id="S4.T2.83.83.83.5" class="ltx_td ltx_align_center" style="background-color:#E8E8FF;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.83.83.83.5.m1.1" class="ltx_Math" style="background-color:#E8E8FF;" alttext="80.06" display="inline"><semantics id="S4.T2.83.83.83.5.m1.1a"><mn mathbackground="#E8E8FF" id="S4.T2.83.83.83.5.m1.1.1" xref="S4.T2.83.83.83.5.m1.1.1.cmml">80.06</mn><annotation-xml encoding="MathML-Content" id="S4.T2.83.83.83.5.m1.1b"><cn type="float" id="S4.T2.83.83.83.5.m1.1.1.cmml" xref="S4.T2.83.83.83.5.m1.1.1">80.06</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.83.83.83.5.m1.1c">80.06</annotation></semantics></math></td>
<td id="S4.T2.84.84.84.6" class="ltx_td ltx_align_center" style="background-color:#FFEBEB;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.84.84.84.6.m1.1" class="ltx_Math" style="background-color:#FFEBEB;" alttext="72.69" display="inline"><semantics id="S4.T2.84.84.84.6.m1.1a"><mn mathbackground="#FFEBEB" id="S4.T2.84.84.84.6.m1.1.1" xref="S4.T2.84.84.84.6.m1.1.1.cmml">72.69</mn><annotation-xml encoding="MathML-Content" id="S4.T2.84.84.84.6.m1.1b"><cn type="float" id="S4.T2.84.84.84.6.m1.1.1.cmml" xref="S4.T2.84.84.84.6.m1.1.1">72.69</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.84.84.84.6.m1.1c">72.69</annotation></semantics></math></td>
<td id="S4.T2.85.85.85.7" class="ltx_td ltx_nopad_r ltx_align_center" style="background-color:#FFD7D7;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.85.85.85.7.m1.1" class="ltx_Math" style="background-color:#FFD7D7;" alttext="-2.93" display="inline"><semantics id="S4.T2.85.85.85.7.m1.1a"><mrow id="S4.T2.85.85.85.7.m1.1.1" xref="S4.T2.85.85.85.7.m1.1.1.cmml"><mo mathbackground="#FFD7D7" id="S4.T2.85.85.85.7.m1.1.1a" xref="S4.T2.85.85.85.7.m1.1.1.cmml">−</mo><mn mathbackground="#FFD7D7" id="S4.T2.85.85.85.7.m1.1.1.2" xref="S4.T2.85.85.85.7.m1.1.1.2.cmml">2.93</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.T2.85.85.85.7.m1.1b"><apply id="S4.T2.85.85.85.7.m1.1.1.cmml" xref="S4.T2.85.85.85.7.m1.1.1"><minus id="S4.T2.85.85.85.7.m1.1.1.1.cmml" xref="S4.T2.85.85.85.7.m1.1.1"></minus><cn type="float" id="S4.T2.85.85.85.7.m1.1.1.2.cmml" xref="S4.T2.85.85.85.7.m1.1.1.2">2.93</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.85.85.85.7.m1.1c">-2.93</annotation></semantics></math></td>
</tr>
<tr id="S4.T2.92.92.92" class="ltx_tr">
<td id="S4.T2.92.92.92.8" class="ltx_td ltx_align_left ltx_border_t" style="padding-left:2.0pt;padding-right:2.0pt;">Vicuna-7B-V1.5</td>
<td id="S4.T2.86.86.86.1" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.86.86.86.1.m1.1" class="ltx_Math" alttext="51.28" display="inline"><semantics id="S4.T2.86.86.86.1.m1.1a"><mn id="S4.T2.86.86.86.1.m1.1.1" xref="S4.T2.86.86.86.1.m1.1.1.cmml">51.28</mn><annotation-xml encoding="MathML-Content" id="S4.T2.86.86.86.1.m1.1b"><cn type="float" id="S4.T2.86.86.86.1.m1.1.1.cmml" xref="S4.T2.86.86.86.1.m1.1.1">51.28</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.86.86.86.1.m1.1c">51.28</annotation></semantics></math></td>
<td id="S4.T2.87.87.87.2" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.87.87.87.2.m1.1" class="ltx_Math" alttext="23.65" display="inline"><semantics id="S4.T2.87.87.87.2.m1.1a"><mn id="S4.T2.87.87.87.2.m1.1.1" xref="S4.T2.87.87.87.2.m1.1.1.cmml">23.65</mn><annotation-xml encoding="MathML-Content" id="S4.T2.87.87.87.2.m1.1b"><cn type="float" id="S4.T2.87.87.87.2.m1.1.1.cmml" xref="S4.T2.87.87.87.2.m1.1.1">23.65</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.87.87.87.2.m1.1c">23.65</annotation></semantics></math></td>
<td id="S4.T2.88.88.88.3" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.88.88.88.3.m1.1" class="ltx_Math" alttext="43.32" display="inline"><semantics id="S4.T2.88.88.88.3.m1.1a"><mn id="S4.T2.88.88.88.3.m1.1.1" xref="S4.T2.88.88.88.3.m1.1.1.cmml">43.32</mn><annotation-xml encoding="MathML-Content" id="S4.T2.88.88.88.3.m1.1b"><cn type="float" id="S4.T2.88.88.88.3.m1.1.1.cmml" xref="S4.T2.88.88.88.3.m1.1.1">43.32</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.88.88.88.3.m1.1c">43.32</annotation></semantics></math></td>
<td id="S4.T2.89.89.89.4" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.89.89.89.4.m1.1" class="ltx_Math" alttext="22.38" display="inline"><semantics id="S4.T2.89.89.89.4.m1.1a"><mn id="S4.T2.89.89.89.4.m1.1.1" xref="S4.T2.89.89.89.4.m1.1.1.cmml">22.38</mn><annotation-xml encoding="MathML-Content" id="S4.T2.89.89.89.4.m1.1b"><cn type="float" id="S4.T2.89.89.89.4.m1.1.1.cmml" xref="S4.T2.89.89.89.4.m1.1.1">22.38</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.89.89.89.4.m1.1c">22.38</annotation></semantics></math></td>
<td id="S4.T2.90.90.90.5" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.90.90.90.5.m1.1" class="ltx_Math" alttext="78.56" display="inline"><semantics id="S4.T2.90.90.90.5.m1.1a"><mn id="S4.T2.90.90.90.5.m1.1.1" xref="S4.T2.90.90.90.5.m1.1.1.cmml">78.56</mn><annotation-xml encoding="MathML-Content" id="S4.T2.90.90.90.5.m1.1b"><cn type="float" id="S4.T2.90.90.90.5.m1.1.1.cmml" xref="S4.T2.90.90.90.5.m1.1.1">78.56</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.90.90.90.5.m1.1c">78.56</annotation></semantics></math></td>
<td id="S4.T2.91.91.91.6" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.91.91.91.6.m1.1" class="ltx_Math" alttext="77.42" display="inline"><semantics id="S4.T2.91.91.91.6.m1.1a"><mn id="S4.T2.91.91.91.6.m1.1.1" xref="S4.T2.91.91.91.6.m1.1.1.cmml">77.42</mn><annotation-xml encoding="MathML-Content" id="S4.T2.91.91.91.6.m1.1b"><cn type="float" id="S4.T2.91.91.91.6.m1.1.1.cmml" xref="S4.T2.91.91.91.6.m1.1.1">77.42</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.91.91.91.6.m1.1c">77.42</annotation></semantics></math></td>
<td id="S4.T2.92.92.92.7" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_t" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.92.92.92.7.m1.1" class="ltx_Math" alttext="0" display="inline"><semantics id="S4.T2.92.92.92.7.m1.1a"><mn id="S4.T2.92.92.92.7.m1.1.1" xref="S4.T2.92.92.92.7.m1.1.1.cmml">0</mn><annotation-xml encoding="MathML-Content" id="S4.T2.92.92.92.7.m1.1b"><cn type="integer" id="S4.T2.92.92.92.7.m1.1.1.cmml" xref="S4.T2.92.92.92.7.m1.1.1">0</cn></annotation-xml></semantics></math></td>
</tr>
<tr id="S4.T2.99.99.99" class="ltx_tr">
<td id="S4.T2.99.99.99.8" class="ltx_td ltx_align_left" style="padding-left:2.0pt;padding-right:2.0pt;">Vicuna-7B-V1.5-Seq</td>
<td id="S4.T2.93.93.93.1" class="ltx_td ltx_align_center" style="background-color:#FFEBEB;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.93.93.93.1.m1.1" class="ltx_Math" style="background-color:#FFEBEB;" alttext="49.46" display="inline"><semantics id="S4.T2.93.93.93.1.m1.1a"><mn mathbackground="#FFEBEB" id="S4.T2.93.93.93.1.m1.1.1" xref="S4.T2.93.93.93.1.m1.1.1.cmml">49.46</mn><annotation-xml encoding="MathML-Content" id="S4.T2.93.93.93.1.m1.1b"><cn type="float" id="S4.T2.93.93.93.1.m1.1.1.cmml" xref="S4.T2.93.93.93.1.m1.1.1">49.46</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.93.93.93.1.m1.1c">49.46</annotation></semantics></math></td>
<td id="S4.T2.94.94.94.2" class="ltx_td ltx_align_center" style="background-color:#FF8E8E;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.94.94.94.2.m1.1" class="ltx_Math" style="background-color:#FF8E8E;" alttext="3.87" display="inline"><semantics id="S4.T2.94.94.94.2.m1.1a"><mn mathbackground="#FF8E8E" id="S4.T2.94.94.94.2.m1.1.1" xref="S4.T2.94.94.94.2.m1.1.1.cmml">3.87</mn><annotation-xml encoding="MathML-Content" id="S4.T2.94.94.94.2.m1.1b"><cn type="float" id="S4.T2.94.94.94.2.m1.1.1.cmml" xref="S4.T2.94.94.94.2.m1.1.1">3.87</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.94.94.94.2.m1.1c">3.87</annotation></semantics></math></td>
<td id="S4.T2.95.95.95.3" class="ltx_td ltx_align_center" style="background-color:#FFD5D5;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.95.95.95.3.m1.1" class="ltx_Math" style="background-color:#FFD5D5;" alttext="39.25" display="inline"><semantics id="S4.T2.95.95.95.3.m1.1a"><mn mathbackground="#FFD5D5" id="S4.T2.95.95.95.3.m1.1.1" xref="S4.T2.95.95.95.3.m1.1.1.cmml">39.25</mn><annotation-xml encoding="MathML-Content" id="S4.T2.95.95.95.3.m1.1b"><cn type="float" id="S4.T2.95.95.95.3.m1.1.1.cmml" xref="S4.T2.95.95.95.3.m1.1.1">39.25</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.95.95.95.3.m1.1c">39.25</annotation></semantics></math></td>
<td id="S4.T2.96.96.96.4" class="ltx_td ltx_align_center" style="background-color:#D0D0FF;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.96.96.96.4.m1.1" class="ltx_Math" style="background-color:#D0D0FF;" alttext="33.92" display="inline"><semantics id="S4.T2.96.96.96.4.m1.1a"><mn mathbackground="#D0D0FF" id="S4.T2.96.96.96.4.m1.1.1" xref="S4.T2.96.96.96.4.m1.1.1.cmml">33.92</mn><annotation-xml encoding="MathML-Content" id="S4.T2.96.96.96.4.m1.1b"><cn type="float" id="S4.T2.96.96.96.4.m1.1.1.cmml" xref="S4.T2.96.96.96.4.m1.1.1">33.92</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.96.96.96.4.m1.1c">33.92</annotation></semantics></math></td>
<td id="S4.T2.97.97.97.5" class="ltx_td ltx_align_center" style="background-color:#FFF7F7;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.97.97.97.5.m1.1" class="ltx_Math" style="background-color:#FFF7F7;" alttext="77.74" display="inline"><semantics id="S4.T2.97.97.97.5.m1.1a"><mn mathbackground="#FFF7F7" id="S4.T2.97.97.97.5.m1.1.1" xref="S4.T2.97.97.97.5.m1.1.1.cmml">77.74</mn><annotation-xml encoding="MathML-Content" id="S4.T2.97.97.97.5.m1.1b"><cn type="float" id="S4.T2.97.97.97.5.m1.1.1.cmml" xref="S4.T2.97.97.97.5.m1.1.1">77.74</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.97.97.97.5.m1.1c">77.74</annotation></semantics></math></td>
<td id="S4.T2.98.98.98.6" class="ltx_td ltx_align_center" style="background-color:#FFEBEB;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.98.98.98.6.m1.1" class="ltx_Math" style="background-color:#FFEBEB;" alttext="75.73" display="inline"><semantics id="S4.T2.98.98.98.6.m1.1a"><mn mathbackground="#FFEBEB" id="S4.T2.98.98.98.6.m1.1.1" xref="S4.T2.98.98.98.6.m1.1.1.cmml">75.73</mn><annotation-xml encoding="MathML-Content" id="S4.T2.98.98.98.6.m1.1b"><cn type="float" id="S4.T2.98.98.98.6.m1.1.1.cmml" xref="S4.T2.98.98.98.6.m1.1.1">75.73</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.98.98.98.6.m1.1c">75.73</annotation></semantics></math></td>
<td id="S4.T2.99.99.99.7" class="ltx_td ltx_nopad_r ltx_align_center" style="background-color:#FFDFDF;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.99.99.99.7.m1.1" class="ltx_Math" style="background-color:#FFDFDF;" alttext="-2.77" display="inline"><semantics id="S4.T2.99.99.99.7.m1.1a"><mrow id="S4.T2.99.99.99.7.m1.1.1" xref="S4.T2.99.99.99.7.m1.1.1.cmml"><mo mathbackground="#FFDFDF" id="S4.T2.99.99.99.7.m1.1.1a" xref="S4.T2.99.99.99.7.m1.1.1.cmml">−</mo><mn mathbackground="#FFDFDF" id="S4.T2.99.99.99.7.m1.1.1.2" xref="S4.T2.99.99.99.7.m1.1.1.2.cmml">2.77</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.T2.99.99.99.7.m1.1b"><apply id="S4.T2.99.99.99.7.m1.1.1.cmml" xref="S4.T2.99.99.99.7.m1.1.1"><minus id="S4.T2.99.99.99.7.m1.1.1.1.cmml" xref="S4.T2.99.99.99.7.m1.1.1"></minus><cn type="float" id="S4.T2.99.99.99.7.m1.1.1.2.cmml" xref="S4.T2.99.99.99.7.m1.1.1.2">2.77</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.99.99.99.7.m1.1c">-2.77</annotation></semantics></math></td>
</tr>
<tr id="S4.T2.106.106.106" class="ltx_tr">
<td id="S4.T2.106.106.106.8" class="ltx_td ltx_align_left" style="padding-left:2.0pt;padding-right:2.0pt;">Vicuna-7B-V1.5-LoraSeq</td>
<td id="S4.T2.100.100.100.1" class="ltx_td ltx_align_center" style="background-color:#FFDCDC;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.100.100.100.1.m1.1" class="ltx_Math" style="background-color:#FFDCDC;" alttext="48.37" display="inline"><semantics id="S4.T2.100.100.100.1.m1.1a"><mn mathbackground="#FFDCDC" id="S4.T2.100.100.100.1.m1.1.1" xref="S4.T2.100.100.100.1.m1.1.1.cmml">48.37</mn><annotation-xml encoding="MathML-Content" id="S4.T2.100.100.100.1.m1.1b"><cn type="float" id="S4.T2.100.100.100.1.m1.1.1.cmml" xref="S4.T2.100.100.100.1.m1.1.1">48.37</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.100.100.100.1.m1.1c">48.37</annotation></semantics></math></td>
<td id="S4.T2.101.101.101.2" class="ltx_td ltx_align_center" style="background-color:#FFD2D2;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.101.101.101.2.m1.1" class="ltx_Math" style="background-color:#FFD2D2;" alttext="18.89" display="inline"><semantics id="S4.T2.101.101.101.2.m1.1a"><mn mathbackground="#FFD2D2" id="S4.T2.101.101.101.2.m1.1.1" xref="S4.T2.101.101.101.2.m1.1.1.cmml">18.89</mn><annotation-xml encoding="MathML-Content" id="S4.T2.101.101.101.2.m1.1b"><cn type="float" id="S4.T2.101.101.101.2.m1.1.1.cmml" xref="S4.T2.101.101.101.2.m1.1.1">18.89</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.101.101.101.2.m1.1c">18.89</annotation></semantics></math></td>
<td id="S4.T2.102.102.102.3" class="ltx_td ltx_align_center" style="background-color:#FFA5A5;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.102.102.102.3.m1.1" class="ltx_Math" style="background-color:#FFA5A5;" alttext="28.16" display="inline"><semantics id="S4.T2.102.102.102.3.m1.1a"><mn mathbackground="#FFA5A5" id="S4.T2.102.102.102.3.m1.1.1" xref="S4.T2.102.102.102.3.m1.1.1.cmml">28.16</mn><annotation-xml encoding="MathML-Content" id="S4.T2.102.102.102.3.m1.1b"><cn type="float" id="S4.T2.102.102.102.3.m1.1.1.cmml" xref="S4.T2.102.102.102.3.m1.1.1">28.16</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.102.102.102.3.m1.1c">28.16</annotation></semantics></math></td>
<td id="S4.T2.103.103.103.4" class="ltx_td ltx_align_center" style="background-color:#F2F2FF;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.103.103.103.4.m1.1" class="ltx_Math" style="background-color:#F2F2FF;" alttext="25.84" display="inline"><semantics id="S4.T2.103.103.103.4.m1.1a"><mn mathbackground="#F2F2FF" id="S4.T2.103.103.103.4.m1.1.1" xref="S4.T2.103.103.103.4.m1.1.1.cmml">25.84</mn><annotation-xml encoding="MathML-Content" id="S4.T2.103.103.103.4.m1.1b"><cn type="float" id="S4.T2.103.103.103.4.m1.1.1.cmml" xref="S4.T2.103.103.103.4.m1.1.1">25.84</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.103.103.103.4.m1.1c">25.84</annotation></semantics></math></td>
<td id="S4.T2.104.104.104.5" class="ltx_td ltx_align_center" style="background-color:#FFC7C7;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.104.104.104.5.m1.1" class="ltx_Math" style="background-color:#FFC7C7;" alttext="67.24" display="inline"><semantics id="S4.T2.104.104.104.5.m1.1a"><mn mathbackground="#FFC7C7" id="S4.T2.104.104.104.5.m1.1.1" xref="S4.T2.104.104.104.5.m1.1.1.cmml">67.24</mn><annotation-xml encoding="MathML-Content" id="S4.T2.104.104.104.5.m1.1b"><cn type="float" id="S4.T2.104.104.104.5.m1.1.1.cmml" xref="S4.T2.104.104.104.5.m1.1.1">67.24</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.104.104.104.5.m1.1c">67.24</annotation></semantics></math></td>
<td id="S4.T2.105.105.105.6" class="ltx_td ltx_align_center" style="background-color:#FFF2F2;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.105.105.105.6.m1.1" class="ltx_Math" style="background-color:#FFF2F2;" alttext="76.23" display="inline"><semantics id="S4.T2.105.105.105.6.m1.1a"><mn mathbackground="#FFF2F2" id="S4.T2.105.105.105.6.m1.1.1" xref="S4.T2.105.105.105.6.m1.1.1.cmml">76.23</mn><annotation-xml encoding="MathML-Content" id="S4.T2.105.105.105.6.m1.1b"><cn type="float" id="S4.T2.105.105.105.6.m1.1.1.cmml" xref="S4.T2.105.105.105.6.m1.1.1">76.23</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.105.105.105.6.m1.1c">76.23</annotation></semantics></math></td>
<td id="S4.T2.106.106.106.7" class="ltx_td ltx_nopad_r ltx_align_center" style="background-color:#FFCFCF;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.106.106.106.7.m1.1" class="ltx_Math" style="background-color:#FFCFCF;" alttext="-5.13" display="inline"><semantics id="S4.T2.106.106.106.7.m1.1a"><mrow id="S4.T2.106.106.106.7.m1.1.1" xref="S4.T2.106.106.106.7.m1.1.1.cmml"><mo mathbackground="#FFCFCF" id="S4.T2.106.106.106.7.m1.1.1a" xref="S4.T2.106.106.106.7.m1.1.1.cmml">−</mo><mn mathbackground="#FFCFCF" id="S4.T2.106.106.106.7.m1.1.1.2" xref="S4.T2.106.106.106.7.m1.1.1.2.cmml">5.13</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.T2.106.106.106.7.m1.1b"><apply id="S4.T2.106.106.106.7.m1.1.1.cmml" xref="S4.T2.106.106.106.7.m1.1.1"><minus id="S4.T2.106.106.106.7.m1.1.1.1.cmml" xref="S4.T2.106.106.106.7.m1.1.1"></minus><cn type="float" id="S4.T2.106.106.106.7.m1.1.1.2.cmml" xref="S4.T2.106.106.106.7.m1.1.1.2">5.13</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.106.106.106.7.m1.1c">-5.13</annotation></semantics></math></td>
</tr>
<tr id="S4.T2.113.113.113" class="ltx_tr">
<td id="S4.T2.113.113.113.8" class="ltx_td ltx_align_left" style="padding-left:2.0pt;padding-right:2.0pt;">Vicuna-7B-V1.5-Replay</td>
<td id="S4.T2.107.107.107.1" class="ltx_td ltx_align_center" style="background-color:#FFD5D5;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.107.107.107.1.m1.1" class="ltx_Math" style="background-color:#FFD5D5;" alttext="47.20" display="inline"><semantics id="S4.T2.107.107.107.1.m1.1a"><mn mathbackground="#FFD5D5" id="S4.T2.107.107.107.1.m1.1.1" xref="S4.T2.107.107.107.1.m1.1.1.cmml">47.20</mn><annotation-xml encoding="MathML-Content" id="S4.T2.107.107.107.1.m1.1b"><cn type="float" id="S4.T2.107.107.107.1.m1.1.1.cmml" xref="S4.T2.107.107.107.1.m1.1.1">47.20</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.107.107.107.1.m1.1c">47.20</annotation></semantics></math></td>
<td id="S4.T2.108.108.108.2" class="ltx_td ltx_align_center" style="background-color:#FF9292;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.108.108.108.2.m1.1" class="ltx_Math" style="background-color:#FF9292;" alttext="4.78" display="inline"><semantics id="S4.T2.108.108.108.2.m1.1a"><mn mathbackground="#FF9292" id="S4.T2.108.108.108.2.m1.1.1" xref="S4.T2.108.108.108.2.m1.1.1.cmml">4.78</mn><annotation-xml encoding="MathML-Content" id="S4.T2.108.108.108.2.m1.1b"><cn type="float" id="S4.T2.108.108.108.2.m1.1.1.cmml" xref="S4.T2.108.108.108.2.m1.1.1">4.78</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.108.108.108.2.m1.1c">4.78</annotation></semantics></math></td>
<td id="S4.T2.109.109.109.3" class="ltx_td ltx_align_center" style="background-color:#FFD6D6;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.109.109.109.3.m1.1" class="ltx_Math" style="background-color:#FFD6D6;" alttext="39.26" display="inline"><semantics id="S4.T2.109.109.109.3.m1.1a"><mn mathbackground="#FFD6D6" id="S4.T2.109.109.109.3.m1.1.1" xref="S4.T2.109.109.109.3.m1.1.1.cmml">39.26</mn><annotation-xml encoding="MathML-Content" id="S4.T2.109.109.109.3.m1.1b"><cn type="float" id="S4.T2.109.109.109.3.m1.1.1.cmml" xref="S4.T2.109.109.109.3.m1.1.1">39.26</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.109.109.109.3.m1.1c">39.26</annotation></semantics></math></td>
<td id="S4.T2.110.110.110.4" class="ltx_td ltx_align_center" style="background-color:#D9D9FF;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.110.110.110.4.m1.1" class="ltx_Math" style="background-color:#D9D9FF;" alttext="31.86" display="inline"><semantics id="S4.T2.110.110.110.4.m1.1a"><mn mathbackground="#D9D9FF" id="S4.T2.110.110.110.4.m1.1.1" xref="S4.T2.110.110.110.4.m1.1.1.cmml">31.86</mn><annotation-xml encoding="MathML-Content" id="S4.T2.110.110.110.4.m1.1b"><cn type="float" id="S4.T2.110.110.110.4.m1.1.1.cmml" xref="S4.T2.110.110.110.4.m1.1.1">31.86</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.110.110.110.4.m1.1c">31.86</annotation></semantics></math></td>
<td id="S4.T2.111.111.111.5" class="ltx_td ltx_align_center" style="background-color:#F9F9FF;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.111.111.111.5.m1.1" class="ltx_Math" style="background-color:#F9F9FF;" alttext="78.92" display="inline"><semantics id="S4.T2.111.111.111.5.m1.1a"><mn mathbackground="#F9F9FF" id="S4.T2.111.111.111.5.m1.1.1" xref="S4.T2.111.111.111.5.m1.1.1.cmml">78.92</mn><annotation-xml encoding="MathML-Content" id="S4.T2.111.111.111.5.m1.1b"><cn type="float" id="S4.T2.111.111.111.5.m1.1.1.cmml" xref="S4.T2.111.111.111.5.m1.1.1">78.92</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.111.111.111.5.m1.1c">78.92</annotation></semantics></math></td>
<td id="S4.T2.112.112.112.6" class="ltx_td ltx_align_center" style="background-color:#F6F6FF;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.112.112.112.6.m1.1" class="ltx_Math" style="background-color:#F6F6FF;" alttext="80.13" display="inline"><semantics id="S4.T2.112.112.112.6.m1.1a"><mn mathbackground="#F6F6FF" id="S4.T2.112.112.112.6.m1.1.1" xref="S4.T2.112.112.112.6.m1.1.1.cmml">80.13</mn><annotation-xml encoding="MathML-Content" id="S4.T2.112.112.112.6.m1.1b"><cn type="float" id="S4.T2.112.112.112.6.m1.1.1.cmml" xref="S4.T2.112.112.112.6.m1.1.1">80.13</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.112.112.112.6.m1.1c">80.13</annotation></semantics></math></td>
<td id="S4.T2.113.113.113.7" class="ltx_td ltx_nopad_r ltx_align_center" style="background-color:#FFDEDE;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.113.113.113.7.m1.1" class="ltx_Math" style="background-color:#FFDEDE;" alttext="-2.41" display="inline"><semantics id="S4.T2.113.113.113.7.m1.1a"><mrow id="S4.T2.113.113.113.7.m1.1.1" xref="S4.T2.113.113.113.7.m1.1.1.cmml"><mo mathbackground="#FFDEDE" id="S4.T2.113.113.113.7.m1.1.1a" xref="S4.T2.113.113.113.7.m1.1.1.cmml">−</mo><mn mathbackground="#FFDEDE" id="S4.T2.113.113.113.7.m1.1.1.2" xref="S4.T2.113.113.113.7.m1.1.1.2.cmml">2.41</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.T2.113.113.113.7.m1.1b"><apply id="S4.T2.113.113.113.7.m1.1.1.cmml" xref="S4.T2.113.113.113.7.m1.1.1"><minus id="S4.T2.113.113.113.7.m1.1.1.1.cmml" xref="S4.T2.113.113.113.7.m1.1.1"></minus><cn type="float" id="S4.T2.113.113.113.7.m1.1.1.2.cmml" xref="S4.T2.113.113.113.7.m1.1.1.2">2.41</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.113.113.113.7.m1.1c">-2.41</annotation></semantics></math></td>
</tr>
<tr id="S4.T2.120.120.120" class="ltx_tr">
<td id="S4.T2.120.120.120.8" class="ltx_td ltx_align_left ltx_border_t" style="padding-left:2.0pt;padding-right:2.0pt;">Vicuna-13B-V1.5</td>
<td id="S4.T2.114.114.114.1" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.114.114.114.1.m1.1" class="ltx_Math" alttext="56.16" display="inline"><semantics id="S4.T2.114.114.114.1.m1.1a"><mn id="S4.T2.114.114.114.1.m1.1.1" xref="S4.T2.114.114.114.1.m1.1.1.cmml">56.16</mn><annotation-xml encoding="MathML-Content" id="S4.T2.114.114.114.1.m1.1b"><cn type="float" id="S4.T2.114.114.114.1.m1.1.1.cmml" xref="S4.T2.114.114.114.1.m1.1.1">56.16</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.114.114.114.1.m1.1c">56.16</annotation></semantics></math></td>
<td id="S4.T2.115.115.115.2" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.115.115.115.2.m1.1" class="ltx_Math" alttext="36.09" display="inline"><semantics id="S4.T2.115.115.115.2.m1.1a"><mn id="S4.T2.115.115.115.2.m1.1.1" xref="S4.T2.115.115.115.2.m1.1.1.cmml">36.09</mn><annotation-xml encoding="MathML-Content" id="S4.T2.115.115.115.2.m1.1b"><cn type="float" id="S4.T2.115.115.115.2.m1.1.1.cmml" xref="S4.T2.115.115.115.2.m1.1.1">36.09</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.115.115.115.2.m1.1c">36.09</annotation></semantics></math></td>
<td id="S4.T2.116.116.116.3" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.116.116.116.3.m1.1" class="ltx_Math" alttext="51.29" display="inline"><semantics id="S4.T2.116.116.116.3.m1.1a"><mn id="S4.T2.116.116.116.3.m1.1.1" xref="S4.T2.116.116.116.3.m1.1.1.cmml">51.29</mn><annotation-xml encoding="MathML-Content" id="S4.T2.116.116.116.3.m1.1b"><cn type="float" id="S4.T2.116.116.116.3.m1.1.1.cmml" xref="S4.T2.116.116.116.3.m1.1.1">51.29</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.116.116.116.3.m1.1c">51.29</annotation></semantics></math></td>
<td id="S4.T2.117.117.117.4" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.117.117.117.4.m1.1" class="ltx_Math" alttext="24.89" display="inline"><semantics id="S4.T2.117.117.117.4.m1.1a"><mn id="S4.T2.117.117.117.4.m1.1.1" xref="S4.T2.117.117.117.4.m1.1.1.cmml">24.89</mn><annotation-xml encoding="MathML-Content" id="S4.T2.117.117.117.4.m1.1b"><cn type="float" id="S4.T2.117.117.117.4.m1.1.1.cmml" xref="S4.T2.117.117.117.4.m1.1.1">24.89</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.117.117.117.4.m1.1c">24.89</annotation></semantics></math></td>
<td id="S4.T2.118.118.118.5" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.118.118.118.5.m1.1" class="ltx_Math" alttext="82.45" display="inline"><semantics id="S4.T2.118.118.118.5.m1.1a"><mn id="S4.T2.118.118.118.5.m1.1.1" xref="S4.T2.118.118.118.5.m1.1.1.cmml">82.45</mn><annotation-xml encoding="MathML-Content" id="S4.T2.118.118.118.5.m1.1b"><cn type="float" id="S4.T2.118.118.118.5.m1.1.1.cmml" xref="S4.T2.118.118.118.5.m1.1.1">82.45</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.118.118.118.5.m1.1c">82.45</annotation></semantics></math></td>
<td id="S4.T2.119.119.119.6" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.119.119.119.6.m1.1" class="ltx_Math" alttext="78.89" display="inline"><semantics id="S4.T2.119.119.119.6.m1.1a"><mn id="S4.T2.119.119.119.6.m1.1.1" xref="S4.T2.119.119.119.6.m1.1.1.cmml">78.89</mn><annotation-xml encoding="MathML-Content" id="S4.T2.119.119.119.6.m1.1b"><cn type="float" id="S4.T2.119.119.119.6.m1.1.1.cmml" xref="S4.T2.119.119.119.6.m1.1.1">78.89</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.119.119.119.6.m1.1c">78.89</annotation></semantics></math></td>
<td id="S4.T2.120.120.120.7" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_t" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.120.120.120.7.m1.1" class="ltx_Math" alttext="0" display="inline"><semantics id="S4.T2.120.120.120.7.m1.1a"><mn id="S4.T2.120.120.120.7.m1.1.1" xref="S4.T2.120.120.120.7.m1.1.1.cmml">0</mn><annotation-xml encoding="MathML-Content" id="S4.T2.120.120.120.7.m1.1b"><cn type="integer" id="S4.T2.120.120.120.7.m1.1.1.cmml" xref="S4.T2.120.120.120.7.m1.1.1">0</cn></annotation-xml></semantics></math></td>
</tr>
<tr id="S4.T2.127.127.127" class="ltx_tr">
<td id="S4.T2.127.127.127.8" class="ltx_td ltx_align_left" style="padding-left:2.0pt;padding-right:2.0pt;">Vicuna-13B-V1.5-Seq</td>
<td id="S4.T2.121.121.121.1" class="ltx_td ltx_align_center" style="background-color:#FFB0B0;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.121.121.121.1.m1.1" class="ltx_Math" style="background-color:#FFB0B0;" alttext="37.93" display="inline"><semantics id="S4.T2.121.121.121.1.m1.1a"><mn mathbackground="#FFB0B0" id="S4.T2.121.121.121.1.m1.1.1" xref="S4.T2.121.121.121.1.m1.1.1.cmml">37.93</mn><annotation-xml encoding="MathML-Content" id="S4.T2.121.121.121.1.m1.1b"><cn type="float" id="S4.T2.121.121.121.1.m1.1.1.cmml" xref="S4.T2.121.121.121.1.m1.1.1">37.93</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.121.121.121.1.m1.1c">37.93</annotation></semantics></math></td>
<td id="S4.T2.122.122.122.2" class="ltx_td ltx_align_center" style="background-color:#FF8080;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.122.122.122.2.m1.1" class="ltx_Math" style="background-color:#FF8080;" alttext="2.81" display="inline"><semantics id="S4.T2.122.122.122.2.m1.1a"><mn mathbackground="#FF8080" id="S4.T2.122.122.122.2.m1.1.1" xref="S4.T2.122.122.122.2.m1.1.1.cmml">2.81</mn><annotation-xml encoding="MathML-Content" id="S4.T2.122.122.122.2.m1.1b"><cn type="float" id="S4.T2.122.122.122.2.m1.1.1.cmml" xref="S4.T2.122.122.122.2.m1.1.1">2.81</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.122.122.122.2.m1.1c">2.81</annotation></semantics></math></td>
<td id="S4.T2.123.123.123.3" class="ltx_td ltx_align_center" style="background-color:#FFBABA;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.123.123.123.3.m1.1" class="ltx_Math" style="background-color:#FFBABA;" alttext="35.23" display="inline"><semantics id="S4.T2.123.123.123.3.m1.1a"><mn mathbackground="#FFBABA" id="S4.T2.123.123.123.3.m1.1.1" xref="S4.T2.123.123.123.3.m1.1.1.cmml">35.23</mn><annotation-xml encoding="MathML-Content" id="S4.T2.123.123.123.3.m1.1b"><cn type="float" id="S4.T2.123.123.123.3.m1.1.1.cmml" xref="S4.T2.123.123.123.3.m1.1.1">35.23</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.123.123.123.3.m1.1c">35.23</annotation></semantics></math></td>
<td id="S4.T2.124.124.124.4" class="ltx_td ltx_align_center" style="background-color:#E0E0FF;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.124.124.124.4.m1.1" class="ltx_Math" style="background-color:#E0E0FF;" alttext="36.86" display="inline"><semantics id="S4.T2.124.124.124.4.m1.1a"><mn mathbackground="#E0E0FF" id="S4.T2.124.124.124.4.m1.1.1" xref="S4.T2.124.124.124.4.m1.1.1.cmml">36.86</mn><annotation-xml encoding="MathML-Content" id="S4.T2.124.124.124.4.m1.1b"><cn type="float" id="S4.T2.124.124.124.4.m1.1.1.cmml" xref="S4.T2.124.124.124.4.m1.1.1">36.86</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.124.124.124.4.m1.1c">36.86</annotation></semantics></math></td>
<td id="S4.T2.125.125.125.5" class="ltx_td ltx_align_center" style="background-color:#F8F8FF;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.125.125.125.5.m1.1" class="ltx_Math" style="background-color:#F8F8FF;" alttext="83.43" display="inline"><semantics id="S4.T2.125.125.125.5.m1.1a"><mn mathbackground="#F8F8FF" id="S4.T2.125.125.125.5.m1.1.1" xref="S4.T2.125.125.125.5.m1.1.1.cmml">83.43</mn><annotation-xml encoding="MathML-Content" id="S4.T2.125.125.125.5.m1.1b"><cn type="float" id="S4.T2.125.125.125.5.m1.1.1.cmml" xref="S4.T2.125.125.125.5.m1.1.1">83.43</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.125.125.125.5.m1.1c">83.43</annotation></semantics></math></td>
<td id="S4.T2.126.126.126.6" class="ltx_td ltx_align_center" style="background-color:#FFF7F7;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.126.126.126.6.m1.1" class="ltx_Math" style="background-color:#FFF7F7;" alttext="77.86" display="inline"><semantics id="S4.T2.126.126.126.6.m1.1a"><mn mathbackground="#FFF7F7" id="S4.T2.126.126.126.6.m1.1.1" xref="S4.T2.126.126.126.6.m1.1.1.cmml">77.86</mn><annotation-xml encoding="MathML-Content" id="S4.T2.126.126.126.6.m1.1b"><cn type="float" id="S4.T2.126.126.126.6.m1.1.1.cmml" xref="S4.T2.126.126.126.6.m1.1.1">77.86</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.126.126.126.6.m1.1c">77.86</annotation></semantics></math></td>
<td id="S4.T2.127.127.127.7" class="ltx_td ltx_nopad_r ltx_align_center" style="background-color:#FFC1C1;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.127.127.127.7.m1.1" class="ltx_Math" style="background-color:#FFC1C1;" alttext="-9.27" display="inline"><semantics id="S4.T2.127.127.127.7.m1.1a"><mrow id="S4.T2.127.127.127.7.m1.1.1" xref="S4.T2.127.127.127.7.m1.1.1.cmml"><mo mathbackground="#FFC1C1" id="S4.T2.127.127.127.7.m1.1.1a" xref="S4.T2.127.127.127.7.m1.1.1.cmml">−</mo><mn mathbackground="#FFC1C1" id="S4.T2.127.127.127.7.m1.1.1.2" xref="S4.T2.127.127.127.7.m1.1.1.2.cmml">9.27</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.T2.127.127.127.7.m1.1b"><apply id="S4.T2.127.127.127.7.m1.1.1.cmml" xref="S4.T2.127.127.127.7.m1.1.1"><minus id="S4.T2.127.127.127.7.m1.1.1.1.cmml" xref="S4.T2.127.127.127.7.m1.1.1"></minus><cn type="float" id="S4.T2.127.127.127.7.m1.1.1.2.cmml" xref="S4.T2.127.127.127.7.m1.1.1.2">9.27</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.127.127.127.7.m1.1c">-9.27</annotation></semantics></math></td>
</tr>
<tr id="S4.T2.134.134.134" class="ltx_tr">
<td id="S4.T2.134.134.134.8" class="ltx_td ltx_align_left" style="padding-left:2.0pt;padding-right:2.0pt;">Vicuna-13B-V1.5-LoraSeq</td>
<td id="S4.T2.128.128.128.1" class="ltx_td ltx_align_center" style="background-color:#FFDADA;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.128.128.128.1.m1.1" class="ltx_Math" style="background-color:#FFDADA;" alttext="52.46" display="inline"><semantics id="S4.T2.128.128.128.1.m1.1a"><mn mathbackground="#FFDADA" id="S4.T2.128.128.128.1.m1.1.1" xref="S4.T2.128.128.128.1.m1.1.1.cmml">52.46</mn><annotation-xml encoding="MathML-Content" id="S4.T2.128.128.128.1.m1.1b"><cn type="float" id="S4.T2.128.128.128.1.m1.1.1.cmml" xref="S4.T2.128.128.128.1.m1.1.1">52.46</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.128.128.128.1.m1.1c">52.46</annotation></semantics></math></td>
<td id="S4.T2.129.129.129.2" class="ltx_td ltx_align_center" style="background-color:#FF8F8F;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.129.129.129.2.m1.1" class="ltx_Math" style="background-color:#FF8F8F;" alttext="22.14" display="inline"><semantics id="S4.T2.129.129.129.2.m1.1a"><mn mathbackground="#FF8F8F" id="S4.T2.129.129.129.2.m1.1.1" xref="S4.T2.129.129.129.2.m1.1.1.cmml">22.14</mn><annotation-xml encoding="MathML-Content" id="S4.T2.129.129.129.2.m1.1b"><cn type="float" id="S4.T2.129.129.129.2.m1.1.1.cmml" xref="S4.T2.129.129.129.2.m1.1.1">22.14</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.129.129.129.2.m1.1c">22.14</annotation></semantics></math></td>
<td id="S4.T2.130.130.130.3" class="ltx_td ltx_align_center" style="background-color:#FFB5B5;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.130.130.130.3.m1.1" class="ltx_Math" style="background-color:#FFB5B5;" alttext="41.22" display="inline"><semantics id="S4.T2.130.130.130.3.m1.1a"><mn mathbackground="#FFB5B5" id="S4.T2.130.130.130.3.m1.1.1" xref="S4.T2.130.130.130.3.m1.1.1.cmml">41.22</mn><annotation-xml encoding="MathML-Content" id="S4.T2.130.130.130.3.m1.1b"><cn type="float" id="S4.T2.130.130.130.3.m1.1.1.cmml" xref="S4.T2.130.130.130.3.m1.1.1">41.22</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.130.130.130.3.m1.1c">41.22</annotation></semantics></math></td>
<td id="S4.T2.131.131.131.4" class="ltx_td ltx_align_center" style="background-color:#F2F2FF;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.131.131.131.4.m1.1" class="ltx_Math" style="background-color:#F2F2FF;" alttext="27.86" display="inline"><semantics id="S4.T2.131.131.131.4.m1.1a"><mn mathbackground="#F2F2FF" id="S4.T2.131.131.131.4.m1.1.1" xref="S4.T2.131.131.131.4.m1.1.1.cmml">27.86</mn><annotation-xml encoding="MathML-Content" id="S4.T2.131.131.131.4.m1.1b"><cn type="float" id="S4.T2.131.131.131.4.m1.1.1.cmml" xref="S4.T2.131.131.131.4.m1.1.1">27.86</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.131.131.131.4.m1.1c">27.86</annotation></semantics></math></td>
<td id="S4.T2.132.132.132.5" class="ltx_td ltx_align_center" style="background-color:#FF8888;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.132.132.132.5.m1.1" class="ltx_Math" style="background-color:#FF8888;" alttext="67.71" display="inline"><semantics id="S4.T2.132.132.132.5.m1.1a"><mn mathbackground="#FF8888" id="S4.T2.132.132.132.5.m1.1.1" xref="S4.T2.132.132.132.5.m1.1.1.cmml">67.71</mn><annotation-xml encoding="MathML-Content" id="S4.T2.132.132.132.5.m1.1b"><cn type="float" id="S4.T2.132.132.132.5.m1.1.1.cmml" xref="S4.T2.132.132.132.5.m1.1.1">67.71</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.132.132.132.5.m1.1c">67.71</annotation></semantics></math></td>
<td id="S4.T2.133.133.133.6" class="ltx_td ltx_align_center" style="background-color:#FFF8F8;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.133.133.133.6.m1.1" class="ltx_Math" style="background-color:#FFF8F8;" alttext="77.53" display="inline"><semantics id="S4.T2.133.133.133.6.m1.1a"><mn mathbackground="#FFF8F8" id="S4.T2.133.133.133.6.m1.1.1" xref="S4.T2.133.133.133.6.m1.1.1.cmml">77.53</mn><annotation-xml encoding="MathML-Content" id="S4.T2.133.133.133.6.m1.1b"><cn type="float" id="S4.T2.133.133.133.6.m1.1.1.cmml" xref="S4.T2.133.133.133.6.m1.1.1">77.53</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.133.133.133.6.m1.1c">77.53</annotation></semantics></math></td>
<td id="S4.T2.134.134.134.7" class="ltx_td ltx_nopad_r ltx_align_center" style="background-color:#FFD6D6;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.134.134.134.7.m1.1" class="ltx_Math" style="background-color:#FFD6D6;" alttext="-6.18" display="inline"><semantics id="S4.T2.134.134.134.7.m1.1a"><mrow id="S4.T2.134.134.134.7.m1.1.1" xref="S4.T2.134.134.134.7.m1.1.1.cmml"><mo mathbackground="#FFD6D6" id="S4.T2.134.134.134.7.m1.1.1a" xref="S4.T2.134.134.134.7.m1.1.1.cmml">−</mo><mn mathbackground="#FFD6D6" id="S4.T2.134.134.134.7.m1.1.1.2" xref="S4.T2.134.134.134.7.m1.1.1.2.cmml">6.18</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.T2.134.134.134.7.m1.1b"><apply id="S4.T2.134.134.134.7.m1.1.1.cmml" xref="S4.T2.134.134.134.7.m1.1.1"><minus id="S4.T2.134.134.134.7.m1.1.1.1.cmml" xref="S4.T2.134.134.134.7.m1.1.1"></minus><cn type="float" id="S4.T2.134.134.134.7.m1.1.1.2.cmml" xref="S4.T2.134.134.134.7.m1.1.1.2">6.18</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.134.134.134.7.m1.1c">-6.18</annotation></semantics></math></td>
</tr>
<tr id="S4.T2.141.141.141" class="ltx_tr">
<td id="S4.T2.141.141.141.8" class="ltx_td ltx_align_left ltx_border_bb" style="padding-left:2.0pt;padding-right:2.0pt;">Vicuna-13B-V1.5-Replay</td>
<td id="S4.T2.135.135.135.1" class="ltx_td ltx_align_center ltx_border_bb" style="background-color:#FFDADA;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.135.135.135.1.m1.1" class="ltx_Math" style="background-color:#FFDADA;" alttext="48.73" display="inline"><semantics id="S4.T2.135.135.135.1.m1.1a"><mn mathbackground="#FFDADA" id="S4.T2.135.135.135.1.m1.1.1" xref="S4.T2.135.135.135.1.m1.1.1.cmml">48.73</mn><annotation-xml encoding="MathML-Content" id="S4.T2.135.135.135.1.m1.1b"><cn type="float" id="S4.T2.135.135.135.1.m1.1.1.cmml" xref="S4.T2.135.135.135.1.m1.1.1">48.73</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.135.135.135.1.m1.1c">48.73</annotation></semantics></math></td>
<td id="S4.T2.136.136.136.2" class="ltx_td ltx_align_center ltx_border_bb" style="background-color:#FF8282;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.136.136.136.2.m1.1" class="ltx_Math" style="background-color:#FF8282;" alttext="3.11" display="inline"><semantics id="S4.T2.136.136.136.2.m1.1a"><mn mathbackground="#FF8282" id="S4.T2.136.136.136.2.m1.1.1" xref="S4.T2.136.136.136.2.m1.1.1.cmml">3.11</mn><annotation-xml encoding="MathML-Content" id="S4.T2.136.136.136.2.m1.1b"><cn type="float" id="S4.T2.136.136.136.2.m1.1.1.cmml" xref="S4.T2.136.136.136.2.m1.1.1">3.11</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.136.136.136.2.m1.1c">3.11</annotation></semantics></math></td>
<td id="S4.T2.137.137.137.3" class="ltx_td ltx_align_center ltx_border_bb" style="background-color:#FFD4D4;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.137.137.137.3.m1.1" class="ltx_Math" style="background-color:#FFD4D4;" alttext="42.94" display="inline"><semantics id="S4.T2.137.137.137.3.m1.1a"><mn mathbackground="#FFD4D4" id="S4.T2.137.137.137.3.m1.1.1" xref="S4.T2.137.137.137.3.m1.1.1.cmml">42.94</mn><annotation-xml encoding="MathML-Content" id="S4.T2.137.137.137.3.m1.1b"><cn type="float" id="S4.T2.137.137.137.3.m1.1.1.cmml" xref="S4.T2.137.137.137.3.m1.1.1">42.94</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.137.137.137.3.m1.1c">42.94</annotation></semantics></math></td>
<td id="S4.T2.138.138.138.4" class="ltx_td ltx_align_center ltx_border_bb" style="background-color:#D2D2FF;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.138.138.138.4.m1.1" class="ltx_Math" style="background-color:#D2D2FF;" alttext="39.60" display="inline"><semantics id="S4.T2.138.138.138.4.m1.1a"><mn mathbackground="#D2D2FF" id="S4.T2.138.138.138.4.m1.1.1" xref="S4.T2.138.138.138.4.m1.1.1.cmml">39.60</mn><annotation-xml encoding="MathML-Content" id="S4.T2.138.138.138.4.m1.1b"><cn type="float" id="S4.T2.138.138.138.4.m1.1.1.cmml" xref="S4.T2.138.138.138.4.m1.1.1">39.60</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.138.138.138.4.m1.1c">39.60</annotation></semantics></math></td>
<td id="S4.T2.139.139.139.5" class="ltx_td ltx_align_center ltx_border_bb" style="background-color:#F5F5FF;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.139.139.139.5.m1.1" class="ltx_Math" style="background-color:#F5F5FF;" alttext="84.71" display="inline"><semantics id="S4.T2.139.139.139.5.m1.1a"><mn mathbackground="#F5F5FF" id="S4.T2.139.139.139.5.m1.1.1" xref="S4.T2.139.139.139.5.m1.1.1.cmml">84.71</mn><annotation-xml encoding="MathML-Content" id="S4.T2.139.139.139.5.m1.1b"><cn type="float" id="S4.T2.139.139.139.5.m1.1.1.cmml" xref="S4.T2.139.139.139.5.m1.1.1">84.71</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.139.139.139.5.m1.1c">84.71</annotation></semantics></math></td>
<td id="S4.T2.140.140.140.6" class="ltx_td ltx_align_center ltx_border_bb" style="background-color:#FFF6F6;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.140.140.140.6.m1.1" class="ltx_Math" style="background-color:#FFF6F6;" alttext="77.53" display="inline"><semantics id="S4.T2.140.140.140.6.m1.1a"><mn mathbackground="#FFF6F6" id="S4.T2.140.140.140.6.m1.1.1" xref="S4.T2.140.140.140.6.m1.1.1.cmml">77.53</mn><annotation-xml encoding="MathML-Content" id="S4.T2.140.140.140.6.m1.1b"><cn type="float" id="S4.T2.140.140.140.6.m1.1.1.cmml" xref="S4.T2.140.140.140.6.m1.1.1">77.53</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.140.140.140.6.m1.1c">77.53</annotation></semantics></math></td>
<td id="S4.T2.141.141.141.7" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_bb" style="background-color:#FFE5E5;padding-left:2.0pt;padding-right:2.0pt;"><math id="S4.T2.141.141.141.7.m1.1" class="ltx_Math" style="background-color:#FFE5E5;" alttext="-5.52" display="inline"><semantics id="S4.T2.141.141.141.7.m1.1a"><mrow id="S4.T2.141.141.141.7.m1.1.1" xref="S4.T2.141.141.141.7.m1.1.1.cmml"><mo mathbackground="#FFE5E5" id="S4.T2.141.141.141.7.m1.1.1a" xref="S4.T2.141.141.141.7.m1.1.1.cmml">−</mo><mn mathbackground="#FFE5E5" id="S4.T2.141.141.141.7.m1.1.1.2" xref="S4.T2.141.141.141.7.m1.1.1.2.cmml">5.52</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.T2.141.141.141.7.m1.1b"><apply id="S4.T2.141.141.141.7.m1.1.1.cmml" xref="S4.T2.141.141.141.7.m1.1.1"><minus id="S4.T2.141.141.141.7.m1.1.1.1.cmml" xref="S4.T2.141.141.141.7.m1.1.1"></minus><cn type="float" id="S4.T2.141.141.141.7.m1.1.1.2.cmml" xref="S4.T2.141.141.141.7.m1.1.1.2">5.52</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.141.141.141.7.m1.1c">-5.52</annotation></semantics></math></td>
</tr>
</table>
</span></div>
</figure>
<figure id="S4.F2" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="S4.F2.sf1" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2310.06762/assets/x2.png" id="S4.F2.sf1.g1" class="ltx_graphics ltx_img_landscape" width="422" height="186" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(a) </span>Helpful evaluation</figcaption>
</figure>
</div>
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="S4.F2.sf2" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2310.06762/assets/x3.png" id="S4.F2.sf2.g1" class="ltx_graphics ltx_img_landscape" width="422" height="186" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(b) </span>Safety evaluation</figcaption>
</figure>
</div>
</div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 2: </span>GPT-4 evaluation with llama-13b-chat, comparing 3 different baselines (Replay, LoRA and Sequential) to the base model across tasks including helpful and safety.</figcaption>
</figure>
</section>
<section id="S4.SS4.SSS2" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">4.4.2 </span>Variation of General Ability</h4>

<div id="S4.SS4.SSS2.p1" class="ltx_para ltx_noindent">
<p id="S4.SS4.SSS2.p1.1" class="ltx_p">Table <a href="#S4.T2" title="Table 2 ‣ 4.4.1 Performance of Target Sequential Tasks ‣ 4.4 Main Results ‣ 4 TRACE: A Comprehensive Benchmark for CL in LLMs ‣ TRACE: A Comprehensive Benchmark for Continual Learning in Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a> presents the evaluations of various LLM models concerning general abilities. The degree of general ability forgetting in LLMs can be analyzed from three perspectives. For a more detailed evaluation, refer to the Appendix.</p>
</div>
<div id="S4.SS4.SSS2.p2" class="ltx_para ltx_noindent">
<p id="S4.SS4.SSS2.p2.1" class="ltx_p">From the Model Perspective:
<span id="S4.SS4.SSS2.p2.1.1" class="ltx_text ltx_font_bold">1)</span> Nearly all models display a negative General Ability Delta, indicating a general decline in overall capabilities after continual learning.
<span id="S4.SS4.SSS2.p2.1.2" class="ltx_text ltx_font_bold">2)</span> Larger models, in comparison to their smaller counterparts, show a more pronounced forgetting in factual knowledge and reasoning tasks. For instance, the General Ability Delta for llama2-13B-chat-Seq stands at 41.88, whereas the SeqFT version for llama2-7B is 46.43.</p>
</div>
<div id="S4.SS4.SSS2.p3" class="ltx_para ltx_noindent">
<p id="S4.SS4.SSS2.p3.1" class="ltx_p">From the Task Perspective:
<span id="S4.SS4.SSS2.p3.1.1" class="ltx_text ltx_font_bold">1)</span> Despite the presence of CoT prompts, there is a noticeable decline in math and reasoning abilities across all models, suggesting that these abilities are highly sensitive to new task learning.
<span id="S4.SS4.SSS2.p3.1.2" class="ltx_text ltx_font_bold">2)</span> Excluding the llama2-7b model, most models exhibit a significant drop in performance on MMLU, suggesting a gradual loss of factual knowledge through continual learning.
<span id="S4.SS4.SSS2.p3.1.3" class="ltx_text ltx_font_bold">3)</span> TydiQA task sees a general boost post-training, possibly due to the inclusion of Chinese and German datasets in our sequential tasks. Even more intriguing is the observed enhancement (and some declines) in other languages on TydiQA, suggesting potential cross-linguistic transfer characteristics.
<span id="S4.SS4.SSS2.p3.1.4" class="ltx_text ltx_font_bold">4)</span> Performance shifts on PIQA for most models are subtle, indicating the relative robustness of commonsense knowledge during continual learning.</p>
</div>
<div id="S4.SS4.SSS2.p4" class="ltx_para ltx_noindent">
<p id="S4.SS4.SSS2.p4.1" class="ltx_p">From the Methodological Perspective:
<span id="S4.SS4.SSS2.p4.1.1" class="ltx_text ltx_font_bold">1)</span> The Replay method proves beneficial in preserving reasoning and factuality skills. Especially for larger models, the mitigation of forgetting through Replay is more pronounced. For instance, for LLaMA-2-7B-Chat, Replay offers a 6.5 EM score boost compared to methods without Replay, while for LLaMA-2-13B-Chat, the increase is 17.1 EM score.</p>
</div>
</section>
<section id="S4.SS4.SSS3" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">4.4.3 </span>Instruction Following Ability Analysis</h4>

<div id="S4.SS4.SSS3.p1" class="ltx_para ltx_noindent">
<p id="S4.SS4.SSS3.p1.1" class="ltx_p">We evaluated the instruction-following ability of models based on two foundation models: LLaMA-2-7B-Chat and LLaMA-2-13B-Chat. Figure <a href="#S4.F2" title="Figure 2 ‣ 4.4.1 Performance of Target Sequential Tasks ‣ 4.4 Main Results ‣ 4 TRACE: A Comprehensive Benchmark for CL in LLMs ‣ TRACE: A Comprehensive Benchmark for Continual Learning in Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a> (a) illustrates the win rate % for instruction following sequentially trained LLMs and their original versions. Here, the win rate can be approximated as an indicator for the Instruction-following delta. It’s evident that all three training methods exhibit a marked decline in instruction-following capabilities compared to their initial versions, with the decline being most pronounced in the LoRA method. Therefore, be cautious when exploring approaches like LoRA for continual learning in LLMs.</p>
</div>
</section>
<section id="S4.SS4.SSS4" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">4.4.4 </span>Safety Analysis</h4>

<div id="S4.SS4.SSS4.p1" class="ltx_para ltx_noindent">
<p id="S4.SS4.SSS4.p1.1" class="ltx_p">We tested the safety of answers from models LLaMA-2-7B-Chat and LLaMA-2-13B-Chat. Figure <a href="#S4.F2" title="Figure 2 ‣ 4.4.1 Performance of Target Sequential Tasks ‣ 4.4 Main Results ‣ 4 TRACE: A Comprehensive Benchmark for CL in LLMs ‣ TRACE: A Comprehensive Benchmark for Continual Learning in Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a> (b) shows the win rate % for instruction following between the new LLMs and their starting versions. Here, the win rate can be used as a measure for the Safety Delta. Compared to the original models, most answers were rated as ’Tie’. This suggests that the safety of the model’s answers is largely unaffected by continual learning on general tasks.</p>
</div>
</section>
</section>
<section id="S4.SS5" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.5 </span>Influencing Factors of Forgetting in LLMs</h3>

<section id="S4.SS5.SSS1" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">4.5.1 </span>Data Quantity &amp; Training Steps</h4>

<div id="S4.SS5.SSS1.p1" class="ltx_para ltx_noindent">
<p id="S4.SS5.SSS1.p1.1" class="ltx_p">Figure <a href="#S4.F4" title="Figure 4 ‣ 4.5.1 Data Quantity &amp; Training Steps ‣ 4.5 Influencing Factors of Forgetting in LLMs ‣ 4 TRACE: A Comprehensive Benchmark for CL in LLMs ‣ TRACE: A Comprehensive Benchmark for Continual Learning in Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a> shows the performance on target tasks for continual learning datasets with different data volumes and training steps. For LLaMA-2-7B-Chat’s SeqFT, we tested with 500, 1000, and 5000 samples from each dataset, training them for 1, 3, 5, 10 epochs. Performance improves as data volume grows, indicating at least 5000 samples from the TRACE-selected datasets are needed for full fitting. Additionally, performance improves with up to 5 training epochs, confirming our baseline epoch setting balances target task optimization and retaining existing capabilities.</p>
</div>
<figure id="S4.F4" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="S4.F4.1" class="ltx_figure ltx_figure_panel ltx_minipage ltx_align_center ltx_align_top" style="width:178.9pt;"><img src="/html/2310.06762/assets/x4.png" id="S4.F4.1.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="181" height="113" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 3: </span>Performance evaluation of LLaMA-2-7B-Chat’s SeqFT on the TRACE benchmark across varying sample sizes (500, 1000, 5000) and training epochs (1, 3, 5, 10 (except for 5000)).</figcaption>
</figure>
</div>
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="S4.F4.2" class="ltx_figure ltx_figure_panel ltx_minipage ltx_align_center ltx_align_top" style="width:178.9pt;"><img src="/html/2310.06762/assets/x5.png" id="S4.F4.2.g1" class="ltx_graphics ltx_img_landscape" width="181" height="113" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 4: </span>Evolution of LLMs’ reasoning capabilities post-training on different tasks, measured using the BBH performance metric. We report the results of LLaMA-2-7B-chat and LLaMA-2-13B-chat.</figcaption>
</figure>
</div>
</div>
</figure>
</section>
<section id="S4.SS5.SSS2" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">4.5.2 </span>Decoupling the Impact of Different Tasks</h4>

<div id="S4.SS5.SSS2.p1" class="ltx_para ltx_noindent">
<p id="S4.SS5.SSS2.p1.1" class="ltx_p">From the results in section <a href="#S4.SS4.SSS2" title="4.4.2 Variation of General Ability ‣ 4.4 Main Results ‣ 4 TRACE: A Comprehensive Benchmark for CL in LLMs ‣ TRACE: A Comprehensive Benchmark for Continual Learning in Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4.4.2</span></a>, it’s evident that post-training LLMs on our benchmark, their innate reasoning and mathematical abilities see a significant dip. This brings forth the question: How exactly does the reasoning capability of LLMs transform during the continual learning process?</p>
</div>
<div id="S4.SS5.SSS2.p2" class="ltx_para ltx_noindent">
<p id="S4.SS5.SSS2.p2.1" class="ltx_p">Figure <a href="#S4.F4" title="Figure 4 ‣ 4.5.1 Data Quantity &amp; Training Steps ‣ 4.5 Influencing Factors of Forgetting in LLMs ‣ 4 TRACE: A Comprehensive Benchmark for CL in LLMs ‣ TRACE: A Comprehensive Benchmark for Continual Learning in Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a> tracks the reasoning ability (assessed via BBH performance) after the completion of each training task. Intriguingly, we observed a surge in the model’s reasoning prowess post-training on the ScienceQA task, while it declined for other tasks. Notably, even though the two tasks from NumGLUE are mathematically inclined, their answers don’t provide a clear reasoning path. In contrast, ScienceQA does offer such a pathway in its answers. This observation suggests the potential advantage of incorporating reasoning paths during training to preserve and perhaps even enhance the model’s reasoning capability.</p>
</div>
</section>
</section>
</section>
<section id="S5" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">5 </span>Reasoning-augmented Continual learning</h2>

<div id="S5.p1" class="ltx_para ltx_noindent">
<p id="S5.p1.1" class="ltx_p">Drawing from our earlier findings, which underscored the unique capabilities of LLMs, we were inspired to reconsider how we approach their training. Instead of treating LLMs as traditional models and inundating them with large volumes of data to fit a task’s distribution, might we leverage their inherent abilities for rapid task transfer? With these insights as our foundation, we formulated the Reasoning-augmented Continual Learning (RCL) approach. RCL forms reasoning paths on new datasets, aiming to not only preserve LLMs’ reasoning capabilities but also to enhance their task transfer and output clarity.</p>
</div>
<div id="S5.p2" class="ltx_para ltx_noindent">
<p id="S5.p2.1" class="ltx_p">As depicted in Figure <a href="#S5.F5" title="Figure 5 ‣ 5.1 Performance on Target Sequential Tasks ‣ 5 Reasoning-augmented Continual learning ‣ TRACE: A Comprehensive Benchmark for Continual Learning in Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a>, RCL has two phases: automated reasoning annotation and sequential training on the augmented dataset. Domain experts created prompts for each task. Three samples per task were manually annotated. GPT-4, using these prompts, generated reasoning paths for every entry. Reasoning was verified against ground truth and underwent human checks. We relied on machine-generated answers due to cost concerns and the consistency of the LM-generated text. To validate reasoning quality, we manually inspected outputs, achieving a 94% approval rate on a 100-sample check, highlighting GPT-4’s reliability. Following this, supervised training was conducted on the target LLM, keeping hyperparameter settings consistent with baselines.</p>
</div>
<section id="S5.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">5.1 </span>Performance on Target Sequential Tasks</h3>

<div id="S5.SS1.p1" class="ltx_para ltx_noindent">
<p id="S5.SS1.p1.1" class="ltx_p">Table <a href="#S5.T3" title="Table 3 ‣ 5.1 Performance on Target Sequential Tasks ‣ 5 Reasoning-augmented Continual learning ‣ TRACE: A Comprehensive Benchmark for Continual Learning in Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a> provides a side-by-side comparison of the performance of our RCL method against other techniques, using LLaMA-2-7B-Chat as the foundational model and limiting training samples to 500 for each task. Through an ablation study contrasting single-task training (SingleFT) with multi-task training, and assessing the impact of reasoning-augmented data, we observed that integrating reasoning pathways into the data consistently boosts performance over the original dataset.</p>
</div>
<div id="S5.SS1.p2" class="ltx_para">
<p id="S5.SS1.p2.1" class="ltx_p">Moreover, our approach, even when trained with just 500 samples, achieves comparable results to the SeqFT method trained on 5000 samples. Furthermore, by leveraging fewer datasets and training steps in our method, we would mitigate the decline in LLMs’ inherent capabilities.</p>
</div>
<figure id="S5.F5" class="ltx_figure"><img src="/html/2310.06762/assets/x6.png" id="S5.F5.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="414" height="217" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 5: </span>An overview of Reasoning-augmented continual learning method. Our method unfolds in two stages: 1) Automatic annotation of sample reasoning paths using GPT-4. We guide GPT-4 through in-context learning and validate the generated paths via answer verification. 2) Continual learning on reasoning-augmented dataset.</figcaption>
</figure>
<figure id="S5.SS1.15" class="ltx_table ltx_align_floatright">
<figure id="S5.SS1.15.15" class="ltx_figure ltx_minipage ltx_align_middle" style="width:178.9pt;">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="S5.T3" class="ltx_table ltx_figure_panel">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table">Table 3: </span>OP and BWT for different baselines. <span id="S5.T3.8.1" class="ltx_text ltx_font_bold">O-Lora<cite class="ltx_cite ltx_citemacro_citep"><span id="S5.T3.8.1.1.1" class="ltx_text ltx_font_medium">(</span>Xiao &amp; Tianze<span id="S5.T3.8.1.2.2.1.1" class="ltx_text ltx_font_medium">, </span><a href="#bib.bib52" title="" class="ltx_ref">2023</a><span id="S5.T3.8.1.3.3" class="ltx_text ltx_font_medium">)</span></cite></span> and <span id="S5.T3.9.2" class="ltx_text ltx_font_bold">PP<cite class="ltx_cite ltx_citemacro_citep"><span id="S5.T3.9.2.1.1" class="ltx_text ltx_font_medium">(</span>Razdaibiedina et al.<span id="S5.T3.9.2.2.2.1.1" class="ltx_text ltx_font_medium">, </span><a href="#bib.bib35" title="" class="ltx_ref">2023</a><span id="S5.T3.9.2.3.3" class="ltx_text ltx_font_medium">)</span></cite></span> refer to two SoTA continual learning methods. All methods except <span id="S5.T3.10.3" class="ltx_text ltx_font_bold">Seq(5k)</span> and <span id="S5.T3.11.4" class="ltx_text ltx_font_bold">ICL</span> are training with 0.5k samples. <span id="S5.T3.12.5" class="ltx_text ltx_font_bold">Re</span> refers to reasoning-augmented, <span id="S5.T3.13.6" class="ltx_text ltx_font_bold">Single FT</span> refers to fine-tuning the model on single task and <span id="S5.T3.14.7" class="ltx_text ltx_font_bold">MT</span> refers to Multi-task training.</figcaption>
</figure>
</div>
<div class="ltx_flex_cell ltx_flex_size_2">
<div id="S5.SS1.15.15.15" class="ltx_inline-block ltx_figure_panel ltx_align_center ltx_transformed_outer" style="width:179.5pt;height:216pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(0.0pt,0.0pt) scale(1.0,1.0) ;">
<table id="S5.SS1.15.15.15.15" class="ltx_tabular ltx_align_middle">
<tr id="S5.SS1.15.15.15.15.16" class="ltx_tr">
<td id="S5.SS1.15.15.15.15.16.1" class="ltx_td ltx_border_r ltx_border_t"></td>
<td id="S5.SS1.15.15.15.15.16.2" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">Overall avg.</td>
<td id="S5.SS1.15.15.15.15.16.3" class="ltx_td ltx_align_center ltx_border_t">BWT</td>
</tr>
<tr id="S5.SS1.1.1.1.1.1" class="ltx_tr">
<td id="S5.SS1.1.1.1.1.1.2" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">ICL</td>
<td id="S5.SS1.1.1.1.1.1.1" class="ltx_td ltx_align_center ltx_border_r ltx_border_t"><math id="S5.SS1.1.1.1.1.1.1.m1.1" class="ltx_Math" alttext="39.5" display="inline"><semantics id="S5.SS1.1.1.1.1.1.1.m1.1a"><mn id="S5.SS1.1.1.1.1.1.1.m1.1.1" xref="S5.SS1.1.1.1.1.1.1.m1.1.1.cmml">39.5</mn><annotation-xml encoding="MathML-Content" id="S5.SS1.1.1.1.1.1.1.m1.1b"><cn type="float" id="S5.SS1.1.1.1.1.1.1.m1.1.1.cmml" xref="S5.SS1.1.1.1.1.1.1.m1.1.1">39.5</cn></annotation-xml><annotation encoding="application/x-tex" id="S5.SS1.1.1.1.1.1.1.m1.1c">39.5</annotation></semantics></math></td>
<td id="S5.SS1.1.1.1.1.1.3" class="ltx_td ltx_align_center ltx_border_t">-</td>
</tr>
<tr id="S5.SS1.2.2.2.2.2" class="ltx_tr">
<td id="S5.SS1.2.2.2.2.2.2" class="ltx_td ltx_align_center ltx_border_r">ICL+Re</td>
<td id="S5.SS1.2.2.2.2.2.1" class="ltx_td ltx_align_center ltx_border_r"><math id="S5.SS1.2.2.2.2.2.1.m1.1" class="ltx_Math" alttext="41.1" display="inline"><semantics id="S5.SS1.2.2.2.2.2.1.m1.1a"><mn id="S5.SS1.2.2.2.2.2.1.m1.1.1" xref="S5.SS1.2.2.2.2.2.1.m1.1.1.cmml">41.1</mn><annotation-xml encoding="MathML-Content" id="S5.SS1.2.2.2.2.2.1.m1.1b"><cn type="float" id="S5.SS1.2.2.2.2.2.1.m1.1.1.cmml" xref="S5.SS1.2.2.2.2.2.1.m1.1.1">41.1</cn></annotation-xml><annotation encoding="application/x-tex" id="S5.SS1.2.2.2.2.2.1.m1.1c">41.1</annotation></semantics></math></td>
<td id="S5.SS1.2.2.2.2.2.3" class="ltx_td ltx_align_center">-</td>
</tr>
<tr id="S5.SS1.4.4.4.4.4" class="ltx_tr">
<td id="S5.SS1.4.4.4.4.4.3" class="ltx_td ltx_align_center ltx_border_r">O-Lora</td>
<td id="S5.SS1.3.3.3.3.3.1" class="ltx_td ltx_align_center ltx_border_r"><math id="S5.SS1.3.3.3.3.3.1.m1.1" class="ltx_Math" alttext="41.3" display="inline"><semantics id="S5.SS1.3.3.3.3.3.1.m1.1a"><mn id="S5.SS1.3.3.3.3.3.1.m1.1.1" xref="S5.SS1.3.3.3.3.3.1.m1.1.1.cmml">41.3</mn><annotation-xml encoding="MathML-Content" id="S5.SS1.3.3.3.3.3.1.m1.1b"><cn type="float" id="S5.SS1.3.3.3.3.3.1.m1.1.1.cmml" xref="S5.SS1.3.3.3.3.3.1.m1.1.1">41.3</cn></annotation-xml><annotation encoding="application/x-tex" id="S5.SS1.3.3.3.3.3.1.m1.1c">41.3</annotation></semantics></math></td>
<td id="S5.SS1.4.4.4.4.4.2" class="ltx_td ltx_align_center"><math id="S5.SS1.4.4.4.4.4.2.m1.1" class="ltx_Math" alttext="6.2\%" display="inline"><semantics id="S5.SS1.4.4.4.4.4.2.m1.1a"><mrow id="S5.SS1.4.4.4.4.4.2.m1.1.1" xref="S5.SS1.4.4.4.4.4.2.m1.1.1.cmml"><mn id="S5.SS1.4.4.4.4.4.2.m1.1.1.2" xref="S5.SS1.4.4.4.4.4.2.m1.1.1.2.cmml">6.2</mn><mo id="S5.SS1.4.4.4.4.4.2.m1.1.1.1" xref="S5.SS1.4.4.4.4.4.2.m1.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S5.SS1.4.4.4.4.4.2.m1.1b"><apply id="S5.SS1.4.4.4.4.4.2.m1.1.1.cmml" xref="S5.SS1.4.4.4.4.4.2.m1.1.1"><csymbol cd="latexml" id="S5.SS1.4.4.4.4.4.2.m1.1.1.1.cmml" xref="S5.SS1.4.4.4.4.4.2.m1.1.1.1">percent</csymbol><cn type="float" id="S5.SS1.4.4.4.4.4.2.m1.1.1.2.cmml" xref="S5.SS1.4.4.4.4.4.2.m1.1.1.2">6.2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS1.4.4.4.4.4.2.m1.1c">6.2\%</annotation></semantics></math></td>
</tr>
<tr id="S5.SS1.5.5.5.5.5" class="ltx_tr">
<td id="S5.SS1.5.5.5.5.5.2" class="ltx_td ltx_align_center ltx_border_r">PP</td>
<td id="S5.SS1.5.5.5.5.5.1" class="ltx_td ltx_align_center ltx_border_r"><math id="S5.SS1.5.5.5.5.5.1.m1.1" class="ltx_Math" alttext="46.2" display="inline"><semantics id="S5.SS1.5.5.5.5.5.1.m1.1a"><mn id="S5.SS1.5.5.5.5.5.1.m1.1.1" xref="S5.SS1.5.5.5.5.5.1.m1.1.1.cmml">46.2</mn><annotation-xml encoding="MathML-Content" id="S5.SS1.5.5.5.5.5.1.m1.1b"><cn type="float" id="S5.SS1.5.5.5.5.5.1.m1.1.1.cmml" xref="S5.SS1.5.5.5.5.5.1.m1.1.1">46.2</cn></annotation-xml><annotation encoding="application/x-tex" id="S5.SS1.5.5.5.5.5.1.m1.1c">46.2</annotation></semantics></math></td>
<td id="S5.SS1.5.5.5.5.5.3" class="ltx_td ltx_align_center">2.3%</td>
</tr>
<tr id="S5.SS1.7.7.7.7.7" class="ltx_tr">
<td id="S5.SS1.7.7.7.7.7.3" class="ltx_td ltx_align_center ltx_border_r">SeqFT</td>
<td id="S5.SS1.6.6.6.6.6.1" class="ltx_td ltx_align_center ltx_border_r"><math id="S5.SS1.6.6.6.6.6.1.m1.1" class="ltx_Math" alttext="23.0" display="inline"><semantics id="S5.SS1.6.6.6.6.6.1.m1.1a"><mn id="S5.SS1.6.6.6.6.6.1.m1.1.1" xref="S5.SS1.6.6.6.6.6.1.m1.1.1.cmml">23.0</mn><annotation-xml encoding="MathML-Content" id="S5.SS1.6.6.6.6.6.1.m1.1b"><cn type="float" id="S5.SS1.6.6.6.6.6.1.m1.1.1.cmml" xref="S5.SS1.6.6.6.6.6.1.m1.1.1">23.0</cn></annotation-xml><annotation encoding="application/x-tex" id="S5.SS1.6.6.6.6.6.1.m1.1c">23.0</annotation></semantics></math></td>
<td id="S5.SS1.7.7.7.7.7.2" class="ltx_td ltx_align_center"><math id="S5.SS1.7.7.7.7.7.2.m1.1" class="ltx_Math" alttext="19\%" display="inline"><semantics id="S5.SS1.7.7.7.7.7.2.m1.1a"><mrow id="S5.SS1.7.7.7.7.7.2.m1.1.1" xref="S5.SS1.7.7.7.7.7.2.m1.1.1.cmml"><mn id="S5.SS1.7.7.7.7.7.2.m1.1.1.2" xref="S5.SS1.7.7.7.7.7.2.m1.1.1.2.cmml">19</mn><mo id="S5.SS1.7.7.7.7.7.2.m1.1.1.1" xref="S5.SS1.7.7.7.7.7.2.m1.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S5.SS1.7.7.7.7.7.2.m1.1b"><apply id="S5.SS1.7.7.7.7.7.2.m1.1.1.cmml" xref="S5.SS1.7.7.7.7.7.2.m1.1.1"><csymbol cd="latexml" id="S5.SS1.7.7.7.7.7.2.m1.1.1.1.cmml" xref="S5.SS1.7.7.7.7.7.2.m1.1.1.1">percent</csymbol><cn type="integer" id="S5.SS1.7.7.7.7.7.2.m1.1.1.2.cmml" xref="S5.SS1.7.7.7.7.7.2.m1.1.1.2">19</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS1.7.7.7.7.7.2.m1.1c">19\%</annotation></semantics></math></td>
</tr>
<tr id="S5.SS1.9.9.9.9.9" class="ltx_tr">
<td id="S5.SS1.9.9.9.9.9.3" class="ltx_td ltx_align_center ltx_border_r">SeqFT(5k)</td>
<td id="S5.SS1.8.8.8.8.8.1" class="ltx_td ltx_align_center ltx_border_r"><span id="S5.SS1.8.8.8.8.8.1.1" class="ltx_text ltx_markedasmath ltx_font_bold">48.7</span></td>
<td id="S5.SS1.9.9.9.9.9.2" class="ltx_td ltx_align_center"><math id="S5.SS1.9.9.9.9.9.2.m1.1" class="ltx_Math" alttext="8.3\%" display="inline"><semantics id="S5.SS1.9.9.9.9.9.2.m1.1a"><mrow id="S5.SS1.9.9.9.9.9.2.m1.1.1" xref="S5.SS1.9.9.9.9.9.2.m1.1.1.cmml"><mn id="S5.SS1.9.9.9.9.9.2.m1.1.1.2" xref="S5.SS1.9.9.9.9.9.2.m1.1.1.2.cmml">8.3</mn><mo id="S5.SS1.9.9.9.9.9.2.m1.1.1.1" xref="S5.SS1.9.9.9.9.9.2.m1.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S5.SS1.9.9.9.9.9.2.m1.1b"><apply id="S5.SS1.9.9.9.9.9.2.m1.1.1.cmml" xref="S5.SS1.9.9.9.9.9.2.m1.1.1"><csymbol cd="latexml" id="S5.SS1.9.9.9.9.9.2.m1.1.1.1.cmml" xref="S5.SS1.9.9.9.9.9.2.m1.1.1.1">percent</csymbol><cn type="float" id="S5.SS1.9.9.9.9.9.2.m1.1.1.2.cmml" xref="S5.SS1.9.9.9.9.9.2.m1.1.1.2">8.3</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS1.9.9.9.9.9.2.m1.1c">8.3\%</annotation></semantics></math></td>
</tr>
<tr id="S5.SS1.11.11.11.11.11" class="ltx_tr">
<td id="S5.SS1.11.11.11.11.11.3" class="ltx_td ltx_align_center ltx_border_r" style="background-color:#90EE90;"><span id="S5.SS1.11.11.11.11.11.3.1" class="ltx_text" style="background-color:#90EE90;">RCL</span></td>
<td id="S5.SS1.10.10.10.10.10.1" class="ltx_td ltx_align_center ltx_border_r"><math id="S5.SS1.10.10.10.10.10.1.m1.1" class="ltx_Math" alttext="46.6" display="inline"><semantics id="S5.SS1.10.10.10.10.10.1.m1.1a"><mn id="S5.SS1.10.10.10.10.10.1.m1.1.1" xref="S5.SS1.10.10.10.10.10.1.m1.1.1.cmml">46.6</mn><annotation-xml encoding="MathML-Content" id="S5.SS1.10.10.10.10.10.1.m1.1b"><cn type="float" id="S5.SS1.10.10.10.10.10.1.m1.1.1.cmml" xref="S5.SS1.10.10.10.10.10.1.m1.1.1">46.6</cn></annotation-xml><annotation encoding="application/x-tex" id="S5.SS1.10.10.10.10.10.1.m1.1c">46.6</annotation></semantics></math></td>
<td id="S5.SS1.11.11.11.11.11.2" class="ltx_td ltx_align_center"><math id="S5.SS1.11.11.11.11.11.2.m1.1" class="ltx_Math" alttext="13\%" display="inline"><semantics id="S5.SS1.11.11.11.11.11.2.m1.1a"><mrow id="S5.SS1.11.11.11.11.11.2.m1.1.1" xref="S5.SS1.11.11.11.11.11.2.m1.1.1.cmml"><mn id="S5.SS1.11.11.11.11.11.2.m1.1.1.2" xref="S5.SS1.11.11.11.11.11.2.m1.1.1.2.cmml">13</mn><mo id="S5.SS1.11.11.11.11.11.2.m1.1.1.1" xref="S5.SS1.11.11.11.11.11.2.m1.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S5.SS1.11.11.11.11.11.2.m1.1b"><apply id="S5.SS1.11.11.11.11.11.2.m1.1.1.cmml" xref="S5.SS1.11.11.11.11.11.2.m1.1.1"><csymbol cd="latexml" id="S5.SS1.11.11.11.11.11.2.m1.1.1.1.cmml" xref="S5.SS1.11.11.11.11.11.2.m1.1.1.1">percent</csymbol><cn type="integer" id="S5.SS1.11.11.11.11.11.2.m1.1.1.2.cmml" xref="S5.SS1.11.11.11.11.11.2.m1.1.1.2">13</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS1.11.11.11.11.11.2.m1.1c">13\%</annotation></semantics></math></td>
</tr>
<tr id="S5.SS1.12.12.12.12.12" class="ltx_tr">
<td id="S5.SS1.12.12.12.12.12.2" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">SingleFT</td>
<td id="S5.SS1.12.12.12.12.12.1" class="ltx_td ltx_align_center ltx_border_r ltx_border_t"><math id="S5.SS1.12.12.12.12.12.1.m1.1" class="ltx_Math" alttext="57.6" display="inline"><semantics id="S5.SS1.12.12.12.12.12.1.m1.1a"><mn id="S5.SS1.12.12.12.12.12.1.m1.1.1" xref="S5.SS1.12.12.12.12.12.1.m1.1.1.cmml">57.6</mn><annotation-xml encoding="MathML-Content" id="S5.SS1.12.12.12.12.12.1.m1.1b"><cn type="float" id="S5.SS1.12.12.12.12.12.1.m1.1.1.cmml" xref="S5.SS1.12.12.12.12.12.1.m1.1.1">57.6</cn></annotation-xml><annotation encoding="application/x-tex" id="S5.SS1.12.12.12.12.12.1.m1.1c">57.6</annotation></semantics></math></td>
<td id="S5.SS1.12.12.12.12.12.3" class="ltx_td ltx_align_center ltx_border_t">-</td>
</tr>
<tr id="S5.SS1.13.13.13.13.13" class="ltx_tr">
<td id="S5.SS1.13.13.13.13.13.2" class="ltx_td ltx_align_center ltx_border_r">SingleFT+Re</td>
<td id="S5.SS1.13.13.13.13.13.1" class="ltx_td ltx_align_center ltx_border_r"><math id="S5.SS1.13.13.13.13.13.1.m1.1" class="ltx_Math" alttext="58.1" display="inline"><semantics id="S5.SS1.13.13.13.13.13.1.m1.1a"><mn id="S5.SS1.13.13.13.13.13.1.m1.1.1" xref="S5.SS1.13.13.13.13.13.1.m1.1.1.cmml">58.1</mn><annotation-xml encoding="MathML-Content" id="S5.SS1.13.13.13.13.13.1.m1.1b"><cn type="float" id="S5.SS1.13.13.13.13.13.1.m1.1.1.cmml" xref="S5.SS1.13.13.13.13.13.1.m1.1.1">58.1</cn></annotation-xml><annotation encoding="application/x-tex" id="S5.SS1.13.13.13.13.13.1.m1.1c">58.1</annotation></semantics></math></td>
<td id="S5.SS1.13.13.13.13.13.3" class="ltx_td ltx_align_center">-</td>
</tr>
<tr id="S5.SS1.14.14.14.14.14" class="ltx_tr">
<td id="S5.SS1.14.14.14.14.14.2" class="ltx_td ltx_align_center ltx_border_r">MT w/o. Re</td>
<td id="S5.SS1.14.14.14.14.14.1" class="ltx_td ltx_align_center ltx_border_r"><math id="S5.SS1.14.14.14.14.14.1.m1.1" class="ltx_Math" alttext="52.3" display="inline"><semantics id="S5.SS1.14.14.14.14.14.1.m1.1a"><mn id="S5.SS1.14.14.14.14.14.1.m1.1.1" xref="S5.SS1.14.14.14.14.14.1.m1.1.1.cmml">52.3</mn><annotation-xml encoding="MathML-Content" id="S5.SS1.14.14.14.14.14.1.m1.1b"><cn type="float" id="S5.SS1.14.14.14.14.14.1.m1.1.1.cmml" xref="S5.SS1.14.14.14.14.14.1.m1.1.1">52.3</cn></annotation-xml><annotation encoding="application/x-tex" id="S5.SS1.14.14.14.14.14.1.m1.1c">52.3</annotation></semantics></math></td>
<td id="S5.SS1.14.14.14.14.14.3" class="ltx_td ltx_align_center">-</td>
</tr>
<tr id="S5.SS1.15.15.15.15.15" class="ltx_tr">
<td id="S5.SS1.15.15.15.15.15.2" class="ltx_td ltx_align_center ltx_border_b ltx_border_r">MT w. Re</td>
<td id="S5.SS1.15.15.15.15.15.1" class="ltx_td ltx_align_center ltx_border_b ltx_border_r"><math id="S5.SS1.15.15.15.15.15.1.m1.1" class="ltx_Math" alttext="58.2" display="inline"><semantics id="S5.SS1.15.15.15.15.15.1.m1.1a"><mn id="S5.SS1.15.15.15.15.15.1.m1.1.1" xref="S5.SS1.15.15.15.15.15.1.m1.1.1.cmml">58.2</mn><annotation-xml encoding="MathML-Content" id="S5.SS1.15.15.15.15.15.1.m1.1b"><cn type="float" id="S5.SS1.15.15.15.15.15.1.m1.1.1.cmml" xref="S5.SS1.15.15.15.15.15.1.m1.1.1">58.2</cn></annotation-xml><annotation encoding="application/x-tex" id="S5.SS1.15.15.15.15.15.1.m1.1c">58.2</annotation></semantics></math></td>
<td id="S5.SS1.15.15.15.15.15.3" class="ltx_td ltx_align_center ltx_border_b">-</td>
</tr>
</table>
</span></div>
</div>
</div>
</figure>
</figure>
</section>
<section id="S5.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">5.2 </span>Impacts on General Ability</h3>

<div id="S5.SS2.p1" class="ltx_para ltx_noindent">
<p id="S5.SS2.p1.1" class="ltx_p">We report the performance of general abilities in Figure <a href="#S6.F6" title="Figure 6 ‣ 6 Discussion ‣ TRACE: A Comprehensive Benchmark for Continual Learning in Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a>. We can conclude that RCL reaches comparable performance with SeqFT and Replay method on MMLU, TydiQA, BoolQA and PIQA though only. However, RCL stands out in reasoning tasks such as GSM and BBH. For instance, for the GSM task, RCL outperforms SeqFT and Replay by 12.7 and 13.2 points respectively, showing the advantages of providing reasoning paths in maintaining the reasoning abilities of models. Besides, combining RCL with replay further improves its performance on reasoning tasks.</p>
</div>
</section>
<section id="S5.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">5.3 </span>Impacts on Instruction-Following and Safety</h3>

<div id="S5.SS3.p1" class="ltx_para ltx_noindent">
<p id="S5.SS3.p1.1" class="ltx_p">The impact of incorporating RCL on instruction-following capabilities is presented in Table <a href="#Ax1.T5" title="Table 5 ‣ .3 Supplementary Tables ‣ Appendix ‣ TRACE: A Comprehensive Benchmark for Continual Learning in Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a>. It’s evident that RCL enhances the model’s ability to follow instructions by 8% and 5% compared to SeqFT and Replay, respectively.</p>
</div>
</section>
</section>
<section id="S6" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">6 </span>Discussion</h2>

<div id="S6.p1" class="ltx_para ltx_noindent">
<p id="S6.p1.1" class="ltx_p"><span id="S6.p1.1.1" class="ltx_text ltx_font_bold">Can traditional continual learning methods be effectively applied to LLMs?</span></p>
</div>
<div id="S6.p2" class="ltx_para ltx_noindent">
<p id="S6.p2.1" class="ltx_p">In section <a href="#S2" title="2 Related Work ‣ TRACE: A Comprehensive Benchmark for Continual Learning in Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>, we introduce various traditional continual learning methods, including replay-based, regularization-based, and architecture-based approaches.
Unfortunately, several characteristics of LLMs challenge the straightforward adoption of these approaches:</p>
<ol id="S6.I1" class="ltx_enumerate">
<li id="S6.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">1.</span> 
<div id="S6.I1.i1.p1" class="ltx_para ltx_noindent">
<p id="S6.I1.i1.p1.1" class="ltx_p"><span id="S6.I1.i1.p1.1.1" class="ltx_text ltx_font_bold">High Training Cost</span>: LLMs require significant data for both pre-training and alignment, leading to a high training cost. Using simple replay to maintain past capabilities can be very expensive. Therefore, selecting key data from past training to keep LLMs’ diverse predictive abilities is essential.</p>
</div>
</li>
<li id="S6.I1.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">2.</span> 
<div id="S6.I1.i2.p1" class="ltx_para ltx_noindent">
<p id="S6.I1.i2.p1.1" class="ltx_p"><span id="S6.I1.i2.p1.1.1" class="ltx_text ltx_font_bold">Large Number of Parameters</span>: The huge parameter size of LLMs demands advanced hardware for training. Many regularization techniques <cite class="ltx_cite ltx_citemacro_citep">(Kirkpatrick et al., <a href="#bib.bib23" title="" class="ltx_ref">2017</a>; Farajtabar et al., <a href="#bib.bib15" title="" class="ltx_ref">2020</a>; Lopez-Paz &amp; Ranzato, <a href="#bib.bib27" title="" class="ltx_ref">2017</a>)</cite>
need to store gradients from past tasks, which is a big challenge for both CPU and GPU memory.</p>
</div>
</li>
<li id="S6.I1.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">3.</span> 
<div id="S6.I1.i3.p1" class="ltx_para ltx_noindent">
<p id="S6.I1.i3.p1.1" class="ltx_p"><span id="S6.I1.i3.p1.1.1" class="ltx_text ltx_font_bold">One-for-All Deployment of LLMs</span>: LLMs are designed for a wide range of tasks, meaning tailoring parameters for specific tasks might limit their ability to generalize to new tasks. Additionally, methods that adjust the network dynamically can complicate deployment, as it becomes tricky to handle multiple task queries at once.</p>
</div>
</li>
</ol>
</div>
<figure id="S6.F6" class="ltx_figure"><img src="/html/2310.06762/assets/x7.png" id="S6.F6.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="422" height="215" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 6: </span>OpenCompass Evaluation Results.<span id="S6.F6.2.1" class="ltx_text ltx_font_bold">RCL+Replay</span> refers to combining our RCL method with replay method.</figcaption>
</figure>
<div id="S6.p3" class="ltx_para ltx_noindent">
<p id="S6.p3.1" class="ltx_p"><span id="S6.p3.1.1" class="ltx_text ltx_font_bold">How should LLMs approach continual learning?</span></p>
</div>
<div id="S6.p4" class="ltx_para ltx_noindent">
<p id="S6.p4.1" class="ltx_p">Our experiments with TRACE show that direct end-to-end training might compel LLMs to focus myopically on specific patterns of the target task, such as shortcuts, thereby undermining their capacities in more universal scenarios.
Intrinsically, LLMs are trained on large and varied datasets. So, they already have the skills to handle many tasks, and can even learn with very few examples.
Based on the ideas from LIMA’s Superficial Alignment Hypothesis <cite class="ltx_cite ltx_citemacro_citep">(Zhou et al., <a href="#bib.bib56" title="" class="ltx_ref">2023</a>)</cite>, our focus should perhaps pivot more towards adeptly aligning LLMs’ existing capabilities to novel tasks rather than embarking on learning from scratch.
Consequently, strategies like our RCL approach, which capitalize on the LLMs’ inherent abilities for quick transfer to new tasks, might also serve as potent tools in mitigating catastrophic forgetting.</p>
</div>
</section>
<section id="S7" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">7 </span>Conclusion</h2>

<div id="S7.p1" class="ltx_para ltx_noindent">
<p id="S7.p1.1" class="ltx_p">Existing benchmarks often fall short in thoroughly evaluating LLMs, either due to their simplistic nature or neglect of critical capabilities like instruction following and safety.
To tackle this, we introduced TRACE, a comprehensive benchmark with diverse challenging tasks and well-rounded metrics. Our experiments showed the real challenges LLMs face, especially a clear drop in their general abilities during continual learning. At the same time, our Reasoning-augmented Continual Learning (RCL) method highlights the importance of using reasoning in training, even though it’s not a complete solution. We believe this area is very important and hope our work lays a solid foundation for future studies.</p>
</div>
</section>
<section id="bib" class="ltx_bibliography">
<h2 class="ltx_title ltx_title_bibliography">References</h2>

<ul class="ltx_biblist">
<li id="bib.bib1" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Bai et al. (2023)</span>
<span class="ltx_bibblock">
Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li.

</span>
<span class="ltx_bibblock">Longbench: A bilingual, multitask benchmark for long context understanding, 2023.

</span>
</li>
<li id="bib.bib2" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Baichuan (2023)</span>
<span class="ltx_bibblock">
Baichuan.

</span>
<span class="ltx_bibblock">Baichuan 2: Open large-scale language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib2.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2309.10305</em>, 2023.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://arxiv.org/abs/2309.10305" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://arxiv.org/abs/2309.10305</a>.

</span>
</li>
<li id="bib.bib3" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Bianchi et al. (2023)</span>
<span class="ltx_bibblock">
Federico Bianchi, Mirac Suzgun, Giuseppe Attanasio, Paul Röttger, Dan Jurafsky, Tatsunori Hashimoto, and James Zou.

</span>
<span class="ltx_bibblock">Safety-tuned llamas: Lessons from improving the safety of large language models that follow instructions.

</span>
<span class="ltx_bibblock"><em id="bib.bib3.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2309.07875</em>, 2023.

</span>
</li>
<li id="bib.bib4" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Bisk et al. (2020)</span>
<span class="ltx_bibblock">
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al.

</span>
<span class="ltx_bibblock">Piqa: Reasoning about physical commonsense in natural language.

</span>
<span class="ltx_bibblock">In <em id="bib.bib4.1.1" class="ltx_emph ltx_font_italic">Proceedings of the AAAI conference on artificial intelligence</em>, volume 34, pp.  7432–7439, 2020.

</span>
</li>
<li id="bib.bib5" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Brown et al. (2020)</span>
<span class="ltx_bibblock">
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, T. J. Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.

</span>
<span class="ltx_bibblock">Language models are few-shot learners.

</span>
<span class="ltx_bibblock"><em id="bib.bib5.1.1" class="ltx_emph ltx_font_italic">ArXiv</em>, abs/2005.14165, 2020.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://api.semanticscholar.org/CorpusID:218971783" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://api.semanticscholar.org/CorpusID:218971783</a>.

</span>
</li>
<li id="bib.bib6" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Chaudhry et al. (2018)</span>
<span class="ltx_bibblock">
Arslan Chaudhry, Puneet K Dokania, Thalaiyasingam Ajanthan, and Philip HS Torr.

</span>
<span class="ltx_bibblock">Riemannian walk for incremental learning: Understanding forgetting and intransigence.

</span>
<span class="ltx_bibblock">In <em id="bib.bib6.1.1" class="ltx_emph ltx_font_italic">Proceedings of the European conference on computer vision (ECCV)</em>, pp.  532–547, 2018.

</span>
</li>
<li id="bib.bib7" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Chiang et al. (2023)</span>
<span class="ltx_bibblock">
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing.

</span>
<span class="ltx_bibblock">Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://lmsys.org/blog/2023-03-30-vicuna/" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://lmsys.org/blog/2023-03-30-vicuna/</a>.

</span>
</li>
<li id="bib.bib8" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Chung et al. (2022)</span>
<span class="ltx_bibblock">
Hyung Won Chung, Le Hou, S. Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Wei Yu, Vincent Zhao, Yanping Huang, Andrew M. Dai, Hongkun Yu, Slav Petrov, Ed Huai hsin Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei.

</span>
<span class="ltx_bibblock">Scaling instruction-finetuned language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib8.1.1" class="ltx_emph ltx_font_italic">ArXiv</em>, abs/2210.11416, 2022.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://api.semanticscholar.org/CorpusID:253018554" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://api.semanticscholar.org/CorpusID:253018554</a>.

</span>
</li>
<li id="bib.bib9" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Clark et al. (2019)</span>
<span class="ltx_bibblock">
Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova.

</span>
<span class="ltx_bibblock">Boolq: Exploring the surprising difficulty of natural yes/no questions.

</span>
<span class="ltx_bibblock"><em id="bib.bib9.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:1905.10044</em>, 2019.

</span>
</li>
<li id="bib.bib10" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Clark et al. (2020)</span>
<span class="ltx_bibblock">
Jonathan H Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki.

</span>
<span class="ltx_bibblock">Tydi qa: A benchmark for information-seeking question answering in ty pologically di verse languages.

</span>
<span class="ltx_bibblock"><em id="bib.bib10.1.1" class="ltx_emph ltx_font_italic">Transactions of the Association for Computational Linguistics</em>, 8:454–470, 2020.

</span>
</li>
<li id="bib.bib11" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Contributors (2023)</span>
<span class="ltx_bibblock">
OpenCompass Contributors.

</span>
<span class="ltx_bibblock">Opencompass: A universal evaluation platform for foundation models.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://github.com/open-compass/opencompass" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/open-compass/opencompass</a>, 2023.

</span>
</li>
<li id="bib.bib12" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Cui et al. (2023)</span>
<span class="ltx_bibblock">
Yiming Cui, Ziqing Yang, and Xin Yao.

</span>
<span class="ltx_bibblock">Efficient and effective text encoding for chinese llama and alpaca, 2023.

</span>
</li>
<li id="bib.bib13" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">de Masson D’Autume et al. (2019)</span>
<span class="ltx_bibblock">
Cyprien de Masson D’Autume, Sebastian Ruder, Lingpeng Kong, and Dani Yogatama.

</span>
<span class="ltx_bibblock">Episodic memory in lifelong language learning.

</span>
<span class="ltx_bibblock"><em id="bib.bib13.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>, 32, 2019.

</span>
</li>
<li id="bib.bib14" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Dubois et al. (2023)</span>
<span class="ltx_bibblock">
Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto.

</span>
<span class="ltx_bibblock">Alpacafarm: A simulation framework for methods that learn from human feedback.

</span>
<span class="ltx_bibblock"><em id="bib.bib14.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2305.14387</em>, 2023.

</span>
</li>
<li id="bib.bib15" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Farajtabar et al. (2020)</span>
<span class="ltx_bibblock">
Mehrdad Farajtabar, Navid Azizan, Alex Mott, and Ang Li.

</span>
<span class="ltx_bibblock">Orthogonal gradient descent for continual learning.

</span>
<span class="ltx_bibblock">In <em id="bib.bib15.1.1" class="ltx_emph ltx_font_italic">International Conference on Artificial Intelligence and Statistics</em>, pp.  3762–3773. PMLR, 2020.

</span>
</li>
<li id="bib.bib16" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ghazal et al. (2013)</span>
<span class="ltx_bibblock">
Ahmad Ghazal, Tilmann Rabl, Minqing Hu, Francois Raab, Meikel Poess, Alain Crolotte, and Hans-Arno Jacobsen.

</span>
<span class="ltx_bibblock">Bigbench: towards an industry standard benchmark for big data analytics.

</span>
<span class="ltx_bibblock">In <em id="bib.bib16.1.1" class="ltx_emph ltx_font_italic">ACM SIGMOD Conference</em>, 2013.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://api.semanticscholar.org/CorpusID:207202897" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://api.semanticscholar.org/CorpusID:207202897</a>.

</span>
</li>
<li id="bib.bib17" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hendrycks et al. (2020)</span>
<span class="ltx_bibblock">
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt.

</span>
<span class="ltx_bibblock">Measuring massive multitask language understanding.

</span>
<span class="ltx_bibblock">In <em id="bib.bib17.1.1" class="ltx_emph ltx_font_italic">International Conference on Learning Representations</em>, 2020.

</span>
</li>
<li id="bib.bib18" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ho et al. (2022)</span>
<span class="ltx_bibblock">
Namgyu Ho, Laura Schmid, and Se-Young Yun.

</span>
<span class="ltx_bibblock">Large language models are reasoning teachers.

</span>
<span class="ltx_bibblock"><em id="bib.bib18.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2212.10071</em>, 2022.

</span>
</li>
<li id="bib.bib19" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hu et al. (2021)</span>
<span class="ltx_bibblock">
J. Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen.

</span>
<span class="ltx_bibblock">Lora: Low-rank adaptation of large language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib19.1.1" class="ltx_emph ltx_font_italic">ArXiv</em>, abs/2106.09685, 2021.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://api.semanticscholar.org/CorpusID:235458009" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://api.semanticscholar.org/CorpusID:235458009</a>.

</span>
</li>
<li id="bib.bib20" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hu et al. (2023)</span>
<span class="ltx_bibblock">
Yebowen Hu, Tim Ganter, Hanieh Deilamsalehy, Franck Dernoncourt, Hassan Foroosh, and Fei Liu.

</span>
<span class="ltx_bibblock">Meetingbank: A benchmark dataset for meeting summarization, 2023.

</span>
</li>
<li id="bib.bib21" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Huang et al. (2023)</span>
<span class="ltx_bibblock">
Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Jiayi Lei, et al.

</span>
<span class="ltx_bibblock">C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models.

</span>
<span class="ltx_bibblock"><em id="bib.bib21.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2305.08322</em>, 2023.

</span>
</li>
<li id="bib.bib22" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ke &amp; Liu (2022)</span>
<span class="ltx_bibblock">
Zixuan Ke and Bin Liu.

</span>
<span class="ltx_bibblock">Continual learning of natural language processing tasks: A survey.

</span>
<span class="ltx_bibblock"><em id="bib.bib22.1.1" class="ltx_emph ltx_font_italic">ArXiv</em>, abs/2211.12701, 2022.

</span>
</li>
<li id="bib.bib23" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Kirkpatrick et al. (2017)</span>
<span class="ltx_bibblock">
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al.

</span>
<span class="ltx_bibblock">Overcoming catastrophic forgetting in neural networks.

</span>
<span class="ltx_bibblock"><em id="bib.bib23.1.1" class="ltx_emph ltx_font_italic">Proceedings of the national academy of sciences</em>, 114(13):3521–3526, 2017.

</span>
</li>
<li id="bib.bib24" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Kojima et al. (2022)</span>
<span class="ltx_bibblock">
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa.

</span>
<span class="ltx_bibblock">Large language models are zero-shot reasoners.

</span>
<span class="ltx_bibblock"><em id="bib.bib24.1.1" class="ltx_emph ltx_font_italic">Advances in neural information processing systems</em>, 35:22199–22213, 2022.

</span>
</li>
<li id="bib.bib25" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Li et al. (2023)</span>
<span class="ltx_bibblock">
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al.

</span>
<span class="ltx_bibblock">Starcoder: may the source be with you!

</span>
<span class="ltx_bibblock"><em id="bib.bib25.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2305.06161</em>, 2023.

</span>
</li>
<li id="bib.bib26" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Liu &amp; Huang (2023)</span>
<span class="ltx_bibblock">
Lei Liu and J. Huang.

</span>
<span class="ltx_bibblock">Prompt learning to mitigate catastrophic forgetting in cross-lingual transfer for open-domain dialogue generation.

</span>
<span class="ltx_bibblock"><em id="bib.bib26.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval</em>, 2023.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://api.semanticscholar.org/CorpusID:258676338" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://api.semanticscholar.org/CorpusID:258676338</a>.

</span>
</li>
<li id="bib.bib27" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lopez-Paz &amp; Ranzato (2017)</span>
<span class="ltx_bibblock">
David Lopez-Paz and Marc’Aurelio Ranzato.

</span>
<span class="ltx_bibblock">Gradient episodic memory for continual learning.

</span>
<span class="ltx_bibblock"><em id="bib.bib27.1.1" class="ltx_emph ltx_font_italic">Advances in neural information processing systems</em>, 30, 2017.

</span>
</li>
<li id="bib.bib28" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lu et al. (2022)</span>
<span class="ltx_bibblock">
Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan.

</span>
<span class="ltx_bibblock">Learn to explain: Multimodal reasoning via thought chains for science question answering, 2022.

</span>
</li>
<li id="bib.bib29" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lu et al. (2021)</span>
<span class="ltx_bibblock">
Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, Ge Li, Lidong Zhou, Linjun Shou, Long Zhou, Michele Tufano, Ming Gong, Ming Zhou, Nan Duan, Neel Sundaresan, Shao Kun Deng, Shengyu Fu, and Shujie Liu.

</span>
<span class="ltx_bibblock">Codexglue: A machine learning benchmark dataset for code understanding and generation, 2021.

</span>
</li>
<li id="bib.bib30" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Luo et al. (2023)</span>
<span class="ltx_bibblock">
Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yuechen Zhang.

</span>
<span class="ltx_bibblock">An empirical study of catastrophic forgetting in large language models during continual fine-tuning.

</span>
<span class="ltx_bibblock"><em id="bib.bib30.1.1" class="ltx_emph ltx_font_italic">ArXiv</em>, abs/2308.08747, 2023.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://api.semanticscholar.org/CorpusID:261031244" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://api.semanticscholar.org/CorpusID:261031244</a>.

</span>
</li>
<li id="bib.bib31" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Maas et al. (2011)</span>
<span class="ltx_bibblock">
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, A. Ng, and Christopher Potts.

</span>
<span class="ltx_bibblock">Learning word vectors for sentiment analysis.

</span>
<span class="ltx_bibblock">In <em id="bib.bib31.1.1" class="ltx_emph ltx_font_italic">Annual Meeting of the Association for Computational Linguistics</em>, 2011.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://api.semanticscholar.org/CorpusID:1428702" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://api.semanticscholar.org/CorpusID:1428702</a>.

</span>
</li>
<li id="bib.bib32" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Mishra et al. (2022)</span>
<span class="ltx_bibblock">
Swaroop Mishra, Arindam Mitra, Neeraj Varshney, Bhavdeep Sachdeva, Peter Clark, Chitta Baral, and Ashwin Kalyan.

</span>
<span class="ltx_bibblock">Numglue: A suite of fundamental yet challenging mathematical reasoning tasks, 2022.

</span>
</li>
<li id="bib.bib33" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">OpenAI (2023)</span>
<span class="ltx_bibblock">
OpenAI.

</span>
<span class="ltx_bibblock">Gpt-4 technical report.

</span>
<span class="ltx_bibblock"><em id="bib.bib33.1.1" class="ltx_emph ltx_font_italic">ArXiv</em>, abs/2303.08774, 2023.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://api.semanticscholar.org/CorpusID:257532815" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://api.semanticscholar.org/CorpusID:257532815</a>.

</span>
</li>
<li id="bib.bib34" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Qin et al. (2023)</span>
<span class="ltx_bibblock">
Yujia Qin, Shi Liang, Yining Ye, Kunlun Zhu, Lan Yan, Ya-Ting Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Runchu Tian, Ruobing Xie, Jie Zhou, Marc H. Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun.

</span>
<span class="ltx_bibblock">Toolllm: Facilitating large language models to master 16000+ real-world apis.

</span>
<span class="ltx_bibblock"><em id="bib.bib34.1.1" class="ltx_emph ltx_font_italic">ArXiv</em>, abs/2307.16789, 2023.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://api.semanticscholar.org/CorpusID:260334759" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://api.semanticscholar.org/CorpusID:260334759</a>.

</span>
</li>
<li id="bib.bib35" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Razdaibiedina et al. (2023)</span>
<span class="ltx_bibblock">
Anastasia Razdaibiedina, Yuning Mao, Rui Hou, Madian Khabsa, Mike Lewis, and Amjad Almahairi.

</span>
<span class="ltx_bibblock">Progressive prompts: Continual learning for language models.

</span>
<span class="ltx_bibblock">In <em id="bib.bib35.1.1" class="ltx_emph ltx_font_italic">The Eleventh International Conference on Learning Representations</em>, 2023.

</span>
</li>
<li id="bib.bib36" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Rios et al. (2021)</span>
<span class="ltx_bibblock">
Annette Rios, Nicolas Spring, Tannon Kew, Marek Kostrzewa, Andreas Säuberli, Mathias Müller, and Sarah Ebling.

</span>
<span class="ltx_bibblock">A new dataset and efficient baselines for document-level text simplification in German.

</span>
<span class="ltx_bibblock">In <em id="bib.bib36.1.1" class="ltx_emph ltx_font_italic">Proceedings of the Third Workshop on New Frontiers in Summarization</em>, pp.  152–161, Online and in Dominican Republic, November 2021. Association for Computational Linguistics.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.18653/v1/2021.newsum-1.16</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://aclanthology.org/2021.newsum-1.16" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://aclanthology.org/2021.newsum-1.16</a>.

</span>
</li>
<li id="bib.bib37" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Rolnick et al. (2019)</span>
<span class="ltx_bibblock">
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne.

</span>
<span class="ltx_bibblock">Experience replay for continual learning.

</span>
<span class="ltx_bibblock"><em id="bib.bib37.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>, 32, 2019.

</span>
</li>
<li id="bib.bib38" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Scao et al. (2022)</span>
<span class="ltx_bibblock">
Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al.

</span>
<span class="ltx_bibblock">Bloom: A 176b-parameter open-access multilingual language model.

</span>
<span class="ltx_bibblock"><em id="bib.bib38.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2211.05100</em>, 2022.

</span>
</li>
<li id="bib.bib39" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Scialom et al. (2022)</span>
<span class="ltx_bibblock">
Thomas Scialom, Tuhin Chakrabarty, and Smaranda Muresan.

</span>
<span class="ltx_bibblock">Fine-tuned language models are continual learners.

</span>
<span class="ltx_bibblock">In <em id="bib.bib39.1.1" class="ltx_emph ltx_font_italic">Conference on Empirical Methods in Natural Language Processing</em>, 2022.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://api.semanticscholar.org/CorpusID:252815378" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://api.semanticscholar.org/CorpusID:252815378</a>.

</span>
</li>
<li id="bib.bib40" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Shah et al. (2023)</span>
<span class="ltx_bibblock">
Agam Shah, Suvan Paturi, and Sudheer Chava.

</span>
<span class="ltx_bibblock">Trillion dollar words: A new financial dataset, task &amp; market analysis, 2023.

</span>
</li>
<li id="bib.bib41" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Smith et al. (2023)</span>
<span class="ltx_bibblock">
James Seale Smith, Yen-Chang Hsu, Lingyu Zhang, Ting Hua, Zsolt Kira, Yilin Shen, and Hongxia Jin.

</span>
<span class="ltx_bibblock">Continual diffusion: Continual customization of text-to-image diffusion with c-lora.

</span>
<span class="ltx_bibblock"><em id="bib.bib41.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2304.06027</em>, 2023.

</span>
</li>
<li id="bib.bib42" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Surameery &amp; Shakor (2023)</span>
<span class="ltx_bibblock">
Nigar M Shafiq Surameery and Mohammed Y Shakor.

</span>
<span class="ltx_bibblock">Use chat gpt to solve programming bugs.

</span>
<span class="ltx_bibblock"><em id="bib.bib42.1.1" class="ltx_emph ltx_font_italic">International Journal of Information Technology &amp; Computer Engineering (IJITC) ISSN: 2455-5290</em>, 3(01):17–22, 2023.

</span>
</li>
<li id="bib.bib43" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Suzgun et al. (2022)</span>
<span class="ltx_bibblock">
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al.

</span>
<span class="ltx_bibblock">Challenging big-bench tasks and whether chain-of-thought can solve them.

</span>
<span class="ltx_bibblock"><em id="bib.bib43.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2210.09261</em>, 2022.

</span>
</li>
<li id="bib.bib44" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Touvron et al. (2023)</span>
<span class="ltx_bibblock">
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.

</span>
<span class="ltx_bibblock">Llama: Open and efficient foundation language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib44.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2302.13971</em>, 2023.

</span>
</li>
<li id="bib.bib45" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wang et al. (2018)</span>
<span class="ltx_bibblock">
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman.

</span>
<span class="ltx_bibblock">Glue: A multi-task benchmark and analysis platform for natural language understanding.

</span>
<span class="ltx_bibblock"><em id="bib.bib45.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:1804.07461</em>, 2018.

</span>
</li>
<li id="bib.bib46" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wang et al. (2019)</span>
<span class="ltx_bibblock">
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman.

</span>
<span class="ltx_bibblock">Superglue: A stickier benchmark for general-purpose language understanding systems.

</span>
<span class="ltx_bibblock"><em id="bib.bib46.1.1" class="ltx_emph ltx_font_italic">Advances in neural information processing systems</em>, 32, 2019.

</span>
</li>
<li id="bib.bib47" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wang et al. (2023a)</span>
<span class="ltx_bibblock">
Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu.

</span>
<span class="ltx_bibblock">A comprehensive survey of continual learning: Theory, method and application.

</span>
<span class="ltx_bibblock"><em id="bib.bib47.1.1" class="ltx_emph ltx_font_italic">ArXiv</em>, abs/2302.00487, 2023a.

</span>
</li>
<li id="bib.bib48" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wang et al. (2022)</span>
<span class="ltx_bibblock">
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi.

</span>
<span class="ltx_bibblock">Self-instruct: Aligning language model with self generated instructions.

</span>
<span class="ltx_bibblock"><em id="bib.bib48.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2212.10560</em>, 2022.

</span>
</li>
<li id="bib.bib49" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wang et al. (2023b)</span>
<span class="ltx_bibblock">
Zhicheng Wang, Yufang Liu, Tao Ji, Xiaoling Wang, Yuanbin Wu, Congcong Jiang, Ye Chao, Zhencong Han, Ling Wang, Xu Shao, and Wenqiu Zeng.

</span>
<span class="ltx_bibblock">Rehearsal-free continual language learning via efficient parameter isolation.

</span>
<span class="ltx_bibblock"><em id="bib.bib49.1.1" class="ltx_emph ltx_font_italic">ArXiv</em>, 2023b.

</span>
</li>
<li id="bib.bib50" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wei et al. (2022)</span>
<span class="ltx_bibblock">
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al.

</span>
<span class="ltx_bibblock">Chain-of-thought prompting elicits reasoning in large language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib50.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>, 35:24824–24837, 2022.

</span>
</li>
<li id="bib.bib51" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wu et al. (2023)</span>
<span class="ltx_bibblock">
Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prabhanjan Kambadur, David Rosenberg, and Gideon Mann.

</span>
<span class="ltx_bibblock">Bloomberggpt: A large language model for finance.

</span>
<span class="ltx_bibblock"><em id="bib.bib51.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2303.17564</em>, 2023.

</span>
</li>
<li id="bib.bib52" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Xiao &amp; Tianze (2023)</span>
<span class="ltx_bibblock">
Wang Xiao and Chen Tianze.

</span>
<span class="ltx_bibblock">Orthogonal subspace learning for language model continual learning.

</span>
<span class="ltx_bibblock"><em id="bib.bib52.1.1" class="ltx_emph ltx_font_italic">ArXiv</em>, abs/2310.06174, 2023.

</span>
</li>
<li id="bib.bib53" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhang et al. (2015)</span>
<span class="ltx_bibblock">
Xiang Zhang, Junbo Jake Zhao, and Yann LeCun.

</span>
<span class="ltx_bibblock">Character-level convolutional networks for text classification.

</span>
<span class="ltx_bibblock">In <em id="bib.bib53.1.1" class="ltx_emph ltx_font_italic">NIPS</em>, 2015.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://api.semanticscholar.org/CorpusID:368182" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://api.semanticscholar.org/CorpusID:368182</a>.

</span>
</li>
<li id="bib.bib54" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhao et al. (2023)</span>
<span class="ltx_bibblock">
Chenye Zhao, Yingjie Li, and Cornelia Caragea.

</span>
<span class="ltx_bibblock">C-STANCE: A large dataset for Chinese zero-shot stance detection.

</span>
<span class="ltx_bibblock">In <em id="bib.bib54.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)</em>, pp.  13369–13385, Toronto, Canada, July 2023. Association for Computational Linguistics.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.18653/v1/2023.acl-long.747</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://aclanthology.org/2023.acl-long.747" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://aclanthology.org/2023.acl-long.747</a>.

</span>
</li>
<li id="bib.bib55" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zheng et al. (2023)</span>
<span class="ltx_bibblock">
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al.

</span>
<span class="ltx_bibblock">Judging llm-as-a-judge with mt-bench and chatbot arena.

</span>
<span class="ltx_bibblock"><em id="bib.bib55.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2306.05685</em>, 2023.

</span>
</li>
<li id="bib.bib56" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhou et al. (2023)</span>
<span class="ltx_bibblock">
Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al.

</span>
<span class="ltx_bibblock">Lima: Less is more for alignment.

</span>
<span class="ltx_bibblock"><em id="bib.bib56.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2305.11206</em>, 2023.

</span>
</li>
<li id="bib.bib57" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhou et al. (2022)</span>
<span class="ltx_bibblock">
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc Le, et al.

</span>
<span class="ltx_bibblock">Least-to-most prompting enables complex reasoning in large language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib57.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2205.10625</em>, 2022.

</span>
</li>
</ul>
</section>
<section id="Ax1" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">Appendix</h2>

<section id="Ax1.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">.1 </span>Implementation Details</h3>

<div id="Ax1.SS1.p1" class="ltx_para ltx_noindent">
<p id="Ax1.SS1.p1.1" class="ltx_p">During the training phase, for the baselines without LoRA adapters, we consistently trained with 5000 samples with a constant learning rate of 1e-5. For the datasets we used (C-STANCE, FOMC, MeetingBank, Py150, ScienceQA, NumGLUE-cm, NumGLUE-ds, 20Minuten), we train for 1, 1, 5, 5, 1, 5, 5, 5 epochs respectively. While for the baselines with LoRA adapters, we trained with 5000 samples with a constant learning rate of 1e-4 for 5, 3, 7, 5, 3, 5, 5, 7 epochs respectively. Our training settings incorporated a weight decay set to 0, and a batch size of 128.
For the testing phase, we use a temperature of 0.1.
All our training and inference experiments were conducted on a machine equipped with 8x80G Nvidia A100, and were implemented using DeepSpeed repository.
All models are trained on 8 A100 GPUs with 80G memory with full parameters fine-tuning. We leave the exploration of large models like LLaMa-2-65B-Chat for future work due to the current hardware limits.</p>
</div>
<div id="Ax1.SS1.p2" class="ltx_para ltx_noindent">
<p id="Ax1.SS1.p2.1" class="ltx_p">All general benchmark evaluations were conducted using the Open-Compass toolkit <cite class="ltx_cite ltx_citemacro_citep">(Contributors, <a href="#bib.bib11" title="" class="ltx_ref">2023</a>)</cite>, adopting its default configuration.</p>
</div>
</section>
<section id="Ax1.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">.2 </span>Trace dataset statistics</h3>

<div id="Ax1.SS2.p1" class="ltx_para">
<p id="Ax1.SS2.p1.1" class="ltx_p">In this section, we represent the overview of dataset statistics, including source, average lenght, metric, language and number of samples of each dataset in TRACE benchmark.</p>
</div>
<figure id="Ax1.T4" class="ltx_table">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table">Table 4: </span>An overview of dataset statistics in TRACE.
’Source’ indicates the context’s origin. ’Avg len’ represents word count for English, German, and code datasets, and character count for Chinese. ’SARI’ is a score specific to simplification.</figcaption>
<table id="Ax1.T4.1" class="ltx_tabular ltx_align_middle">
<tr id="Ax1.T4.1.1" class="ltx_tr">
<td id="Ax1.T4.1.1.1" class="ltx_td ltx_align_left ltx_border_tt">Dataset</td>
<td id="Ax1.T4.1.1.2" class="ltx_td ltx_align_left ltx_border_tt">Source</td>
<td id="Ax1.T4.1.1.3" class="ltx_td ltx_align_left ltx_border_tt">Avg len</td>
<td id="Ax1.T4.1.1.4" class="ltx_td ltx_align_left ltx_border_tt">Metric</td>
<td id="Ax1.T4.1.1.5" class="ltx_td ltx_align_left ltx_border_tt">Language</td>
<td id="Ax1.T4.1.1.6" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_tt">#data</td>
</tr>
<tr id="Ax1.T4.1.2" class="ltx_tr">
<td id="Ax1.T4.1.2.1" class="ltx_td ltx_align_left ltx_border_t"><span id="Ax1.T4.1.2.1.1" class="ltx_text ltx_font_italic">Domain-specific</span></td>
<td id="Ax1.T4.1.2.2" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T4.1.2.3" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T4.1.2.4" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T4.1.2.5" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T4.1.2.6" class="ltx_td ltx_nopad_r ltx_border_t"></td>
</tr>
<tr id="Ax1.T4.1.3" class="ltx_tr">
<td id="Ax1.T4.1.3.1" class="ltx_td ltx_align_left">ScienceQA</td>
<td id="Ax1.T4.1.3.2" class="ltx_td ltx_align_left">Science</td>
<td id="Ax1.T4.1.3.3" class="ltx_td ltx_align_left">210</td>
<td id="Ax1.T4.1.3.4" class="ltx_td ltx_align_left">Accuracy</td>
<td id="Ax1.T4.1.3.5" class="ltx_td ltx_align_left">English</td>
<td id="Ax1.T4.1.3.6" class="ltx_td ltx_nopad_r ltx_align_left">5,000</td>
</tr>
<tr id="Ax1.T4.1.4" class="ltx_tr">
<td id="Ax1.T4.1.4.1" class="ltx_td ltx_align_left">FOMC</td>
<td id="Ax1.T4.1.4.2" class="ltx_td ltx_align_left">Finance</td>
<td id="Ax1.T4.1.4.3" class="ltx_td ltx_align_left">51</td>
<td id="Ax1.T4.1.4.4" class="ltx_td ltx_align_left">Accuracy</td>
<td id="Ax1.T4.1.4.5" class="ltx_td ltx_align_left">English</td>
<td id="Ax1.T4.1.4.6" class="ltx_td ltx_nopad_r ltx_align_left">5,000</td>
</tr>
<tr id="Ax1.T4.1.5" class="ltx_tr">
<td id="Ax1.T4.1.5.1" class="ltx_td ltx_align_left">MeetingBank</td>
<td id="Ax1.T4.1.5.2" class="ltx_td ltx_align_left">Meeting</td>
<td id="Ax1.T4.1.5.3" class="ltx_td ltx_align_left">2853</td>
<td id="Ax1.T4.1.5.4" class="ltx_td ltx_align_left">ROUGE-L</td>
<td id="Ax1.T4.1.5.5" class="ltx_td ltx_align_left">English</td>
<td id="Ax1.T4.1.5.6" class="ltx_td ltx_nopad_r ltx_align_left">5,000</td>
</tr>
<tr id="Ax1.T4.1.6" class="ltx_tr">
<td id="Ax1.T4.1.6.1" class="ltx_td ltx_align_left ltx_border_t"><span id="Ax1.T4.1.6.1.1" class="ltx_text ltx_font_italic">Multi-lingual</span></td>
<td id="Ax1.T4.1.6.2" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T4.1.6.3" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T4.1.6.4" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T4.1.6.5" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T4.1.6.6" class="ltx_td ltx_nopad_r ltx_border_t"></td>
</tr>
<tr id="Ax1.T4.1.7" class="ltx_tr">
<td id="Ax1.T4.1.7.1" class="ltx_td ltx_align_left">C-STANCE</td>
<td id="Ax1.T4.1.7.2" class="ltx_td ltx_align_left">Social media</td>
<td id="Ax1.T4.1.7.3" class="ltx_td ltx_align_left">127</td>
<td id="Ax1.T4.1.7.4" class="ltx_td ltx_align_left">Accuracy</td>
<td id="Ax1.T4.1.7.5" class="ltx_td ltx_align_left">Chinese</td>
<td id="Ax1.T4.1.7.6" class="ltx_td ltx_nopad_r ltx_align_left">5,000</td>
</tr>
<tr id="Ax1.T4.1.8" class="ltx_tr">
<td id="Ax1.T4.1.8.1" class="ltx_td ltx_align_left">20Minuten</td>
<td id="Ax1.T4.1.8.2" class="ltx_td ltx_align_left">News</td>
<td id="Ax1.T4.1.8.3" class="ltx_td ltx_align_left">382</td>
<td id="Ax1.T4.1.8.4" class="ltx_td ltx_align_left">SARI</td>
<td id="Ax1.T4.1.8.5" class="ltx_td ltx_align_left">Germany</td>
<td id="Ax1.T4.1.8.6" class="ltx_td ltx_nopad_r ltx_align_left">5,000</td>
</tr>
<tr id="Ax1.T4.1.9" class="ltx_tr">
<td id="Ax1.T4.1.9.1" class="ltx_td ltx_align_left ltx_border_t"><span id="Ax1.T4.1.9.1.1" class="ltx_text ltx_font_italic">Code completion</span></td>
<td id="Ax1.T4.1.9.2" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T4.1.9.3" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T4.1.9.4" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T4.1.9.5" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T4.1.9.6" class="ltx_td ltx_nopad_r ltx_border_t"></td>
</tr>
<tr id="Ax1.T4.1.10" class="ltx_tr">
<td id="Ax1.T4.1.10.1" class="ltx_td ltx_align_left">Py150</td>
<td id="Ax1.T4.1.10.2" class="ltx_td ltx_align_left">Github</td>
<td id="Ax1.T4.1.10.3" class="ltx_td ltx_align_left">422</td>
<td id="Ax1.T4.1.10.4" class="ltx_td ltx_align_left">Edim similarity</td>
<td id="Ax1.T4.1.10.5" class="ltx_td ltx_align_left">Python</td>
<td id="Ax1.T4.1.10.6" class="ltx_td ltx_nopad_r ltx_align_left">5,000</td>
</tr>
<tr id="Ax1.T4.1.11" class="ltx_tr">
<td id="Ax1.T4.1.11.1" class="ltx_td ltx_align_left ltx_border_t"><span id="Ax1.T4.1.11.1.1" class="ltx_text ltx_font_italic">Mathematical reasoning</span></td>
<td id="Ax1.T4.1.11.2" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T4.1.11.3" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T4.1.11.4" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T4.1.11.5" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T4.1.11.6" class="ltx_td ltx_nopad_r ltx_border_t"></td>
</tr>
<tr id="Ax1.T4.1.12" class="ltx_tr">
<td id="Ax1.T4.1.12.1" class="ltx_td ltx_align_left">NumGLUE-cm</td>
<td id="Ax1.T4.1.12.2" class="ltx_td ltx_align_left">Math</td>
<td id="Ax1.T4.1.12.3" class="ltx_td ltx_align_left">32</td>
<td id="Ax1.T4.1.12.4" class="ltx_td ltx_align_left">Accuracy</td>
<td id="Ax1.T4.1.12.5" class="ltx_td ltx_align_left">English</td>
<td id="Ax1.T4.1.12.6" class="ltx_td ltx_nopad_r ltx_align_left">5,000</td>
</tr>
<tr id="Ax1.T4.1.13" class="ltx_tr">
<td id="Ax1.T4.1.13.1" class="ltx_td ltx_align_left ltx_border_bb">NumGLUE-ds</td>
<td id="Ax1.T4.1.13.2" class="ltx_td ltx_align_left ltx_border_bb">Math</td>
<td id="Ax1.T4.1.13.3" class="ltx_td ltx_align_left ltx_border_bb">21</td>
<td id="Ax1.T4.1.13.4" class="ltx_td ltx_align_left ltx_border_bb">Accuracy</td>
<td id="Ax1.T4.1.13.5" class="ltx_td ltx_align_left ltx_border_bb">English</td>
<td id="Ax1.T4.1.13.6" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_bb">5,000</td>
</tr>
</table>
</figure>
</section>
<section id="Ax1.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">.3 </span>Supplementary Tables</h3>

<figure id="Ax1.T5" class="ltx_table">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 5: </span>Instruction-following abilities of SeqFT, Replay and RCL</figcaption>
<table id="Ax1.T5.1" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="Ax1.T5.1.1" class="ltx_tr">
<td id="Ax1.T5.1.1.1" class="ltx_td ltx_border_r ltx_border_t"></td>
<td id="Ax1.T5.1.1.2" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">Win</td>
<td id="Ax1.T5.1.1.3" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">Tie</td>
<td id="Ax1.T5.1.1.4" class="ltx_td ltx_align_center ltx_border_t">Loss</td>
</tr>
<tr id="Ax1.T5.1.2" class="ltx_tr">
<td id="Ax1.T5.1.2.1" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">SeqFT</td>
<td id="Ax1.T5.1.2.2" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">12%</td>
<td id="Ax1.T5.1.2.3" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">30%</td>
<td id="Ax1.T5.1.2.4" class="ltx_td ltx_align_center ltx_border_t">58%</td>
</tr>
<tr id="Ax1.T5.1.3" class="ltx_tr">
<td id="Ax1.T5.1.3.1" class="ltx_td ltx_align_center ltx_border_r">Replay</td>
<td id="Ax1.T5.1.3.2" class="ltx_td ltx_align_center ltx_border_r">15%</td>
<td id="Ax1.T5.1.3.3" class="ltx_td ltx_align_center ltx_border_r">31%</td>
<td id="Ax1.T5.1.3.4" class="ltx_td ltx_align_center">55%</td>
</tr>
<tr id="Ax1.T5.1.4" class="ltx_tr">
<td id="Ax1.T5.1.4.1" class="ltx_td ltx_align_center ltx_border_b ltx_border_r">RCL</td>
<td id="Ax1.T5.1.4.2" class="ltx_td ltx_align_center ltx_border_b ltx_border_r">20%</td>
<td id="Ax1.T5.1.4.3" class="ltx_td ltx_align_center ltx_border_b ltx_border_r">30%</td>
<td id="Ax1.T5.1.4.4" class="ltx_td ltx_align_center ltx_border_b">50%</td>
</tr>
</table>
</figure>
</section>
<section id="Ax1.SS4" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">.4 </span>Detailed Experiments Results</h3>

<div id="Ax1.SS4.p1" class="ltx_para ltx_noindent">
<p id="Ax1.SS4.p1.1" class="ltx_p">In this section, we report the detailed experiment results in our paper. The model includes Baichuan-7b,LLaMA2-7b-chat, LLaMA2-13b-chat, Vicuna-7b and Vicuna-13b. The results are shown in Table <a href="#Ax1.T6" title="Table 6 ‣ .4.1 In-Context Learning ‣ .4 Detailed Experiments Results ‣ Appendix ‣ TRACE: A Comprehensive Benchmark for Continual Learning in Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a> <a href="#Ax1.T27" title="Table 27 ‣ .4.8 Detailed results of Table 3 ‣ .4 Detailed Experiments Results ‣ Appendix ‣ TRACE: A Comprehensive Benchmark for Continual Learning in Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">27</span></a>.</p>
</div>
<section id="Ax1.SS4.SSS1" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">.4.1 </span>In-Context Learning</h4>

<div id="Ax1.SS4.SSS1.p1" class="ltx_para">
<p id="Ax1.SS4.SSS1.p1.1" class="ltx_p">Table <a href="#Ax1.T6" title="Table 6 ‣ .4.1 In-Context Learning ‣ .4 Detailed Experiments Results ‣ Appendix ‣ TRACE: A Comprehensive Benchmark for Continual Learning in Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a> represents the performance of different models with in-context learning.</p>
</div>
<figure id="Ax1.T6" class="ltx_table">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 6: </span>Detailed results of in-context learning of different large language models.</figcaption>
<table id="Ax1.T6.1" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="Ax1.T6.1.1" class="ltx_tr">
<td id="Ax1.T6.1.1.1" class="ltx_td ltx_align_left ltx_border_tt">Task/Model</td>
<td id="Ax1.T6.1.1.2" class="ltx_td ltx_align_left ltx_border_tt">Baichuan-7b</td>
<td id="Ax1.T6.1.1.3" class="ltx_td ltx_align_left ltx_border_tt">LLaMA2-7b-chat</td>
<td id="Ax1.T6.1.1.4" class="ltx_td ltx_align_left ltx_border_tt">LLaMA2-13b-chat</td>
<td id="Ax1.T6.1.1.5" class="ltx_td ltx_align_left ltx_border_tt">Vicuna-7b</td>
<td id="Ax1.T6.1.1.6" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_tt">Vicuna-13b</td>
</tr>
<tr id="Ax1.T6.1.2" class="ltx_tr">
<td id="Ax1.T6.1.2.1" class="ltx_td ltx_align_left ltx_border_t">C-STANCE</td>
<td id="Ax1.T6.1.2.2" class="ltx_td ltx_align_left ltx_border_t">0.58</td>
<td id="Ax1.T6.1.2.3" class="ltx_td ltx_align_left ltx_border_t">0.4</td>
<td id="Ax1.T6.1.2.4" class="ltx_td ltx_align_left ltx_border_t">0.366</td>
<td id="Ax1.T6.1.2.5" class="ltx_td ltx_align_left ltx_border_t">0.403</td>
<td id="Ax1.T6.1.2.6" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_t">0.57</td>
</tr>
<tr id="Ax1.T6.1.3" class="ltx_tr">
<td id="Ax1.T6.1.3.1" class="ltx_td ltx_align_left">FOMC</td>
<td id="Ax1.T6.1.3.2" class="ltx_td ltx_align_left">0.63</td>
<td id="Ax1.T6.1.3.3" class="ltx_td ltx_align_left">0.483</td>
<td id="Ax1.T6.1.3.4" class="ltx_td ltx_align_left">0.519</td>
<td id="Ax1.T6.1.3.5" class="ltx_td ltx_align_left">0.551</td>
<td id="Ax1.T6.1.3.6" class="ltx_td ltx_nopad_r ltx_align_left"><span id="Ax1.T6.1.3.6.1" class="ltx_text" style="color:#1F2329;">0.61</span></td>
</tr>
<tr id="Ax1.T6.1.4" class="ltx_tr">
<td id="Ax1.T6.1.4.1" class="ltx_td ltx_align_left">MeetingBank</td>
<td id="Ax1.T6.1.4.2" class="ltx_td ltx_align_left">0.225</td>
<td id="Ax1.T6.1.4.3" class="ltx_td ltx_align_left">0.198</td>
<td id="Ax1.T6.1.4.4" class="ltx_td ltx_align_left">0.221</td>
<td id="Ax1.T6.1.4.5" class="ltx_td ltx_align_left">0.223</td>
<td id="Ax1.T6.1.4.6" class="ltx_td ltx_nopad_r ltx_align_left">0.229</td>
</tr>
<tr id="Ax1.T6.1.5" class="ltx_tr">
<td id="Ax1.T6.1.5.1" class="ltx_td ltx_align_left">Py150</td>
<td id="Ax1.T6.1.5.2" class="ltx_td ltx_align_left">0.586</td>
<td id="Ax1.T6.1.5.3" class="ltx_td ltx_align_left">0.522</td>
<td id="Ax1.T6.1.5.4" class="ltx_td ltx_align_left">0.539</td>
<td id="Ax1.T6.1.5.5" class="ltx_td ltx_align_left">0.529</td>
<td id="Ax1.T6.1.5.6" class="ltx_td ltx_nopad_r ltx_align_left">0.585</td>
</tr>
<tr id="Ax1.T6.1.6" class="ltx_tr">
<td id="Ax1.T6.1.6.1" class="ltx_td ltx_align_left">ScienceQA</td>
<td id="Ax1.T6.1.6.2" class="ltx_td ltx_align_left">0.68</td>
<td id="Ax1.T6.1.6.3" class="ltx_td ltx_align_left"><span id="Ax1.T6.1.6.3.1" class="ltx_text" style="color:#1F2329;">0.628</span></td>
<td id="Ax1.T6.1.6.4" class="ltx_td ltx_align_left">0.689</td>
<td id="Ax1.T6.1.6.5" class="ltx_td ltx_align_left">0.695</td>
<td id="Ax1.T6.1.6.6" class="ltx_td ltx_nopad_r ltx_align_left">0.7</td>
</tr>
<tr id="Ax1.T6.1.7" class="ltx_tr">
<td id="Ax1.T6.1.7.1" class="ltx_td ltx_align_left">NumGLUE-cm</td>
<td id="Ax1.T6.1.7.2" class="ltx_td ltx_align_left">0.271</td>
<td id="Ax1.T6.1.7.3" class="ltx_td ltx_align_left">0.284</td>
<td id="Ax1.T6.1.7.4" class="ltx_td ltx_align_left">0.407</td>
<td id="Ax1.T6.1.7.5" class="ltx_td ltx_align_left">0.284</td>
<td id="Ax1.T6.1.7.6" class="ltx_td ltx_nopad_r ltx_align_left">0.347</td>
</tr>
<tr id="Ax1.T6.1.8" class="ltx_tr">
<td id="Ax1.T6.1.8.1" class="ltx_td ltx_align_left">NumGLUE-ds</td>
<td id="Ax1.T6.1.8.2" class="ltx_td ltx_align_left">0.23</td>
<td id="Ax1.T6.1.8.3" class="ltx_td ltx_align_left">0.203</td>
<td id="Ax1.T6.1.8.4" class="ltx_td ltx_align_left">0.218</td>
<td id="Ax1.T6.1.8.5" class="ltx_td ltx_align_left">0.302</td>
<td id="Ax1.T6.1.8.6" class="ltx_td ltx_nopad_r ltx_align_left">0.33</td>
</tr>
<tr id="Ax1.T6.1.9" class="ltx_tr">
<td id="Ax1.T6.1.9.1" class="ltx_td ltx_align_left">20Minuten</td>
<td id="Ax1.T6.1.9.2" class="ltx_td ltx_align_left">0.366</td>
<td id="Ax1.T6.1.9.3" class="ltx_td ltx_align_left">0.395</td>
<td id="Ax1.T6.1.9.4" class="ltx_td ltx_align_left">0.395</td>
<td id="Ax1.T6.1.9.5" class="ltx_td ltx_align_left">0.392</td>
<td id="Ax1.T6.1.9.6" class="ltx_td ltx_nopad_r ltx_align_left">0.378</td>
</tr>
<tr id="Ax1.T6.1.10" class="ltx_tr">
<td id="Ax1.T6.1.10.1" class="ltx_td ltx_align_left ltx_border_bb ltx_border_t">average</td>
<td id="Ax1.T6.1.10.2" class="ltx_td ltx_align_left ltx_border_bb ltx_border_t">0.446</td>
<td id="Ax1.T6.1.10.3" class="ltx_td ltx_align_left ltx_border_bb ltx_border_t">0.389</td>
<td id="Ax1.T6.1.10.4" class="ltx_td ltx_align_left ltx_border_bb ltx_border_t">0.419</td>
<td id="Ax1.T6.1.10.5" class="ltx_td ltx_align_left ltx_border_bb ltx_border_t">0.422</td>
<td id="Ax1.T6.1.10.6" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_bb ltx_border_t">0.469</td>
</tr>
</table>
</figure>
</section>
<section id="Ax1.SS4.SSS2" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">.4.2 </span>SeqFT method</h4>

<div id="Ax1.SS4.SSS2.p1" class="ltx_para">
<p id="Ax1.SS4.SSS2.p1.1" class="ltx_p">Table <a href="#Ax1.T7" title="Table 7 ‣ .4.2 SeqFT method ‣ .4 Detailed Experiments Results ‣ Appendix ‣ TRACE: A Comprehensive Benchmark for Continual Learning in Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">7</span></a> - <a href="#Ax1.T11" title="Table 11 ‣ .4.2 SeqFT method ‣ .4 Detailed Experiments Results ‣ Appendix ‣ TRACE: A Comprehensive Benchmark for Continual Learning in Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">11</span></a> shows the detailed performance of different models of each round during the continual learning. SeqFT represents sequential fine-tuning.</p>
</div>
<figure id="Ax1.T7" class="ltx_table">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table">Table 7: </span>Detailed results of continual learning of Baichuan-7b.</figcaption>
<table id="Ax1.T7.1" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="Ax1.T7.1.1" class="ltx_tr">
<td id="Ax1.T7.1.1.1" class="ltx_td ltx_align_left ltx_border_tt">Task\Round</td>
<td id="Ax1.T7.1.1.2" class="ltx_td ltx_align_left ltx_border_tt">1</td>
<td id="Ax1.T7.1.1.3" class="ltx_td ltx_align_left ltx_border_tt">2</td>
<td id="Ax1.T7.1.1.4" class="ltx_td ltx_align_left ltx_border_tt">3</td>
<td id="Ax1.T7.1.1.5" class="ltx_td ltx_align_left ltx_border_tt">4</td>
<td id="Ax1.T7.1.1.6" class="ltx_td ltx_align_left ltx_border_tt">5</td>
<td id="Ax1.T7.1.1.7" class="ltx_td ltx_align_left ltx_border_tt">6</td>
<td id="Ax1.T7.1.1.8" class="ltx_td ltx_align_left ltx_border_tt">7</td>
<td id="Ax1.T7.1.1.9" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_tt">8</td>
</tr>
<tr id="Ax1.T7.1.2" class="ltx_tr">
<td id="Ax1.T7.1.2.1" class="ltx_td ltx_align_left ltx_border_t">C-STANCE</td>
<td id="Ax1.T7.1.2.2" class="ltx_td ltx_align_left ltx_border_t">0.62</td>
<td id="Ax1.T7.1.2.3" class="ltx_td ltx_align_left ltx_border_t">0.629</td>
<td id="Ax1.T7.1.2.4" class="ltx_td ltx_align_left ltx_border_t">0.663</td>
<td id="Ax1.T7.1.2.5" class="ltx_td ltx_align_left ltx_border_t">0.621</td>
<td id="Ax1.T7.1.2.6" class="ltx_td ltx_align_left ltx_border_t">0.531</td>
<td id="Ax1.T7.1.2.7" class="ltx_td ltx_align_left ltx_border_t">0.55</td>
<td id="Ax1.T7.1.2.8" class="ltx_td ltx_align_left ltx_border_t">0.588</td>
<td id="Ax1.T7.1.2.9" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_t">0.579</td>
</tr>
<tr id="Ax1.T7.1.3" class="ltx_tr">
<td id="Ax1.T7.1.3.1" class="ltx_td ltx_align_left">FOMC</td>
<td id="Ax1.T7.1.3.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T7.1.3.3" class="ltx_td ltx_align_left">0.681</td>
<td id="Ax1.T7.1.3.4" class="ltx_td ltx_align_left">0.353</td>
<td id="Ax1.T7.1.3.5" class="ltx_td ltx_align_left">0.318</td>
<td id="Ax1.T7.1.3.6" class="ltx_td ltx_align_left"><span id="Ax1.T7.1.3.6.1" class="ltx_text" style="color:#1F2329;">0.02</span></td>
<td id="Ax1.T7.1.3.7" class="ltx_td ltx_align_left">0.363</td>
<td id="Ax1.T7.1.3.8" class="ltx_td ltx_align_left">0.347</td>
<td id="Ax1.T7.1.3.9" class="ltx_td ltx_nopad_r ltx_align_left">0.335</td>
</tr>
<tr id="Ax1.T7.1.4" class="ltx_tr">
<td id="Ax1.T7.1.4.1" class="ltx_td ltx_align_left">MeetingBank</td>
<td id="Ax1.T7.1.4.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T7.1.4.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T7.1.4.4" class="ltx_td ltx_align_left">0.442</td>
<td id="Ax1.T7.1.4.5" class="ltx_td ltx_align_left">0.351</td>
<td id="Ax1.T7.1.4.6" class="ltx_td ltx_align_left">0.371</td>
<td id="Ax1.T7.1.4.7" class="ltx_td ltx_align_left">0.379</td>
<td id="Ax1.T7.1.4.8" class="ltx_td ltx_align_left">0.389</td>
<td id="Ax1.T7.1.4.9" class="ltx_td ltx_nopad_r ltx_align_left">0.364</td>
</tr>
<tr id="Ax1.T7.1.5" class="ltx_tr">
<td id="Ax1.T7.1.5.1" class="ltx_td ltx_align_left">Py150</td>
<td id="Ax1.T7.1.5.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T7.1.5.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T7.1.5.4" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T7.1.5.5" class="ltx_td ltx_align_left">0.626</td>
<td id="Ax1.T7.1.5.6" class="ltx_td ltx_align_left">0.562</td>
<td id="Ax1.T7.1.5.7" class="ltx_td ltx_align_left">0.586</td>
<td id="Ax1.T7.1.5.8" class="ltx_td ltx_align_left">0.589</td>
<td id="Ax1.T7.1.5.9" class="ltx_td ltx_nopad_r ltx_align_left">0.58</td>
</tr>
<tr id="Ax1.T7.1.6" class="ltx_tr">
<td id="Ax1.T7.1.6.1" class="ltx_td ltx_align_left">ScienceQA</td>
<td id="Ax1.T7.1.6.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T7.1.6.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T7.1.6.4" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T7.1.6.5" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T7.1.6.6" class="ltx_td ltx_align_left">0.77</td>
<td id="Ax1.T7.1.6.7" class="ltx_td ltx_align_left">0.68</td>
<td id="Ax1.T7.1.6.8" class="ltx_td ltx_align_left">0.5</td>
<td id="Ax1.T7.1.6.9" class="ltx_td ltx_nopad_r ltx_align_left">0.44</td>
</tr>
<tr id="Ax1.T7.1.7" class="ltx_tr">
<td id="Ax1.T7.1.7.1" class="ltx_td ltx_align_left">NumGLUE-cm</td>
<td id="Ax1.T7.1.7.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T7.1.7.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T7.1.7.4" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T7.1.7.5" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T7.1.7.6" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T7.1.7.7" class="ltx_td ltx_align_left">0.358</td>
<td id="Ax1.T7.1.7.8" class="ltx_td ltx_align_left">0.247</td>
<td id="Ax1.T7.1.7.9" class="ltx_td ltx_nopad_r ltx_align_left">0.284</td>
</tr>
<tr id="Ax1.T7.1.8" class="ltx_tr">
<td id="Ax1.T7.1.8.1" class="ltx_td ltx_align_left">NumGLUE-ds</td>
<td id="Ax1.T7.1.8.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T7.1.8.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T7.1.8.4" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T7.1.8.5" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T7.1.8.6" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T7.1.8.7" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T7.1.8.8" class="ltx_td ltx_align_left">0.64</td>
<td id="Ax1.T7.1.8.9" class="ltx_td ltx_nopad_r ltx_align_left">0.475</td>
</tr>
<tr id="Ax1.T7.1.9" class="ltx_tr">
<td id="Ax1.T7.1.9.1" class="ltx_td ltx_align_left">20Minuten</td>
<td id="Ax1.T7.1.9.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T7.1.9.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T7.1.9.4" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T7.1.9.5" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T7.1.9.6" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T7.1.9.7" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T7.1.9.8" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T7.1.9.9" class="ltx_td ltx_nopad_r ltx_align_left">0.415</td>
</tr>
<tr id="Ax1.T7.1.10" class="ltx_tr">
<td id="Ax1.T7.1.10.1" class="ltx_td ltx_align_left ltx_border_t">average</td>
<td id="Ax1.T7.1.10.2" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T7.1.10.3" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T7.1.10.4" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T7.1.10.5" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T7.1.10.6" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T7.1.10.7" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T7.1.10.8" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T7.1.10.9" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_t">0.434</td>
</tr>
<tr id="Ax1.T7.1.11" class="ltx_tr">
<td id="Ax1.T7.1.11.1" class="ltx_td ltx_align_left ltx_border_bb">BWT</td>
<td id="Ax1.T7.1.11.2" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T7.1.11.3" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T7.1.11.4" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T7.1.11.5" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T7.1.11.6" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T7.1.11.7" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T7.1.11.8" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T7.1.11.9" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_bb">-0.154</td>
</tr>
</table>
</figure>
<figure id="Ax1.T8" class="ltx_table">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table">Table 8: </span>Detailed results of continual learning of LLaMA-7b-chat.</figcaption>
<table id="Ax1.T8.1" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="Ax1.T8.1.1" class="ltx_tr">
<td id="Ax1.T8.1.1.1" class="ltx_td ltx_align_left ltx_border_tt">Task\Round</td>
<td id="Ax1.T8.1.1.2" class="ltx_td ltx_align_left ltx_border_tt">1</td>
<td id="Ax1.T8.1.1.3" class="ltx_td ltx_align_left ltx_border_tt">2</td>
<td id="Ax1.T8.1.1.4" class="ltx_td ltx_align_left ltx_border_tt">3</td>
<td id="Ax1.T8.1.1.5" class="ltx_td ltx_align_left ltx_border_tt">4</td>
<td id="Ax1.T8.1.1.6" class="ltx_td ltx_align_left ltx_border_tt">5</td>
<td id="Ax1.T8.1.1.7" class="ltx_td ltx_align_left ltx_border_tt">6</td>
<td id="Ax1.T8.1.1.8" class="ltx_td ltx_align_left ltx_border_tt">7</td>
<td id="Ax1.T8.1.1.9" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_tt">8</td>
</tr>
<tr id="Ax1.T8.1.2" class="ltx_tr">
<td id="Ax1.T8.1.2.1" class="ltx_td ltx_align_left ltx_border_t">C-STANCE</td>
<td id="Ax1.T8.1.2.2" class="ltx_td ltx_align_left ltx_border_t">0.5</td>
<td id="Ax1.T8.1.2.3" class="ltx_td ltx_align_left ltx_border_t">0.456</td>
<td id="Ax1.T8.1.2.4" class="ltx_td ltx_align_left ltx_border_t">0.448</td>
<td id="Ax1.T8.1.2.5" class="ltx_td ltx_align_left ltx_border_t">0.453</td>
<td id="Ax1.T8.1.2.6" class="ltx_td ltx_align_left ltx_border_t">0.435</td>
<td id="Ax1.T8.1.2.7" class="ltx_td ltx_align_left ltx_border_t">0.442</td>
<td id="Ax1.T8.1.2.8" class="ltx_td ltx_align_left ltx_border_t">0.436</td>
<td id="Ax1.T8.1.2.9" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_t">0.454</td>
</tr>
<tr id="Ax1.T8.1.3" class="ltx_tr">
<td id="Ax1.T8.1.3.1" class="ltx_td ltx_align_left">FOMC</td>
<td id="Ax1.T8.1.3.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T8.1.3.3" class="ltx_td ltx_align_left">0.735</td>
<td id="Ax1.T8.1.3.4" class="ltx_td ltx_align_left">0.67</td>
<td id="Ax1.T8.1.3.5" class="ltx_td ltx_align_left">0.658</td>
<td id="Ax1.T8.1.3.6" class="ltx_td ltx_align_left"><span id="Ax1.T8.1.3.6.1" class="ltx_text" style="color:#1F2329;">0</span></td>
<td id="Ax1.T8.1.3.7" class="ltx_td ltx_align_left">0.595</td>
<td id="Ax1.T8.1.3.8" class="ltx_td ltx_align_left">0.577</td>
<td id="Ax1.T8.1.3.9" class="ltx_td ltx_nopad_r ltx_align_left">0.609</td>
</tr>
<tr id="Ax1.T8.1.4" class="ltx_tr">
<td id="Ax1.T8.1.4.1" class="ltx_td ltx_align_left">MeetingBank</td>
<td id="Ax1.T8.1.4.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T8.1.4.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T8.1.4.4" class="ltx_td ltx_align_left">0.523</td>
<td id="Ax1.T8.1.4.5" class="ltx_td ltx_align_left">0.459</td>
<td id="Ax1.T8.1.4.6" class="ltx_td ltx_align_left">0.433</td>
<td id="Ax1.T8.1.4.7" class="ltx_td ltx_align_left">0.446</td>
<td id="Ax1.T8.1.4.8" class="ltx_td ltx_align_left">0.442</td>
<td id="Ax1.T8.1.4.9" class="ltx_td ltx_nopad_r ltx_align_left">0.457</td>
</tr>
<tr id="Ax1.T8.1.5" class="ltx_tr">
<td id="Ax1.T8.1.5.1" class="ltx_td ltx_align_left">Py150</td>
<td id="Ax1.T8.1.5.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T8.1.5.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T8.1.5.4" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T8.1.5.5" class="ltx_td ltx_align_left">0.58</td>
<td id="Ax1.T8.1.5.6" class="ltx_td ltx_align_left">0.459</td>
<td id="Ax1.T8.1.5.7" class="ltx_td ltx_align_left">0.509</td>
<td id="Ax1.T8.1.5.8" class="ltx_td ltx_align_left">0.508</td>
<td id="Ax1.T8.1.5.9" class="ltx_td ltx_nopad_r ltx_align_left">0.512</td>
</tr>
<tr id="Ax1.T8.1.6" class="ltx_tr">
<td id="Ax1.T8.1.6.1" class="ltx_td ltx_align_left">ScienceQA</td>
<td id="Ax1.T8.1.6.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T8.1.6.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T8.1.6.4" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T8.1.6.5" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T8.1.6.6" class="ltx_td ltx_align_left">0.764</td>
<td id="Ax1.T8.1.6.7" class="ltx_td ltx_align_left">0.636</td>
<td id="Ax1.T8.1.6.8" class="ltx_td ltx_align_left">0.45</td>
<td id="Ax1.T8.1.6.9" class="ltx_td ltx_nopad_r ltx_align_left">0.637</td>
</tr>
<tr id="Ax1.T8.1.7" class="ltx_tr">
<td id="Ax1.T8.1.7.1" class="ltx_td ltx_align_left">NumGLUE-cm</td>
<td id="Ax1.T8.1.7.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T8.1.7.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T8.1.7.4" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T8.1.7.5" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T8.1.7.6" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T8.1.7.7" class="ltx_td ltx_align_left">0.383</td>
<td id="Ax1.T8.1.7.8" class="ltx_td ltx_align_left">0.247</td>
<td id="Ax1.T8.1.7.9" class="ltx_td ltx_nopad_r ltx_align_left">0.272</td>
</tr>
<tr id="Ax1.T8.1.8" class="ltx_tr">
<td id="Ax1.T8.1.8.1" class="ltx_td ltx_align_left">NumGLUE-ds</td>
<td id="Ax1.T8.1.8.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T8.1.8.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T8.1.8.4" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T8.1.8.5" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T8.1.8.6" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T8.1.8.7" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T8.1.8.8" class="ltx_td ltx_align_left">0.582</td>
<td id="Ax1.T8.1.8.9" class="ltx_td ltx_nopad_r ltx_align_left">0.548</td>
</tr>
<tr id="Ax1.T8.1.9" class="ltx_tr">
<td id="Ax1.T8.1.9.1" class="ltx_td ltx_align_left">20Minuten</td>
<td id="Ax1.T8.1.9.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T8.1.9.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T8.1.9.4" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T8.1.9.5" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T8.1.9.6" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T8.1.9.7" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T8.1.9.8" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T8.1.9.9" class="ltx_td ltx_nopad_r ltx_align_left">0.408</td>
</tr>
<tr id="Ax1.T8.1.10" class="ltx_tr">
<td id="Ax1.T8.1.10.1" class="ltx_td ltx_align_left ltx_border_t">average</td>
<td id="Ax1.T8.1.10.2" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T8.1.10.3" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T8.1.10.4" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T8.1.10.5" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T8.1.10.6" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T8.1.10.7" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T8.1.10.8" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T8.1.10.9" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_t">0.487</td>
</tr>
<tr id="Ax1.T8.1.11" class="ltx_tr">
<td id="Ax1.T8.1.11.1" class="ltx_td ltx_align_left ltx_border_bb">BWT</td>
<td id="Ax1.T8.1.11.2" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T8.1.11.3" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T8.1.11.4" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T8.1.11.5" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T8.1.11.6" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T8.1.11.7" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T8.1.11.8" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T8.1.11.9" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_bb">-0.083</td>
</tr>
</table>
</figure>
<figure id="Ax1.T9" class="ltx_table">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table">Table 9: </span>Detailed results of continual learning of LLaMA-13b-chat.</figcaption>
<table id="Ax1.T9.1" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="Ax1.T9.1.1" class="ltx_tr">
<td id="Ax1.T9.1.1.1" class="ltx_td ltx_align_left ltx_border_tt">Task\Round</td>
<td id="Ax1.T9.1.1.2" class="ltx_td ltx_align_left ltx_border_tt">1</td>
<td id="Ax1.T9.1.1.3" class="ltx_td ltx_align_left ltx_border_tt">2</td>
<td id="Ax1.T9.1.1.4" class="ltx_td ltx_align_left ltx_border_tt">3</td>
<td id="Ax1.T9.1.1.5" class="ltx_td ltx_align_left ltx_border_tt">4</td>
<td id="Ax1.T9.1.1.6" class="ltx_td ltx_align_left ltx_border_tt">5</td>
<td id="Ax1.T9.1.1.7" class="ltx_td ltx_align_left ltx_border_tt">6</td>
<td id="Ax1.T9.1.1.8" class="ltx_td ltx_align_left ltx_border_tt">7</td>
<td id="Ax1.T9.1.1.9" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_tt">8</td>
</tr>
<tr id="Ax1.T9.1.2" class="ltx_tr">
<td id="Ax1.T9.1.2.1" class="ltx_td ltx_align_left ltx_border_t">C-STANCE</td>
<td id="Ax1.T9.1.2.2" class="ltx_td ltx_align_left ltx_border_t">0.469</td>
<td id="Ax1.T9.1.2.3" class="ltx_td ltx_align_left ltx_border_t">0.465</td>
<td id="Ax1.T9.1.2.4" class="ltx_td ltx_align_left ltx_border_t">0.47</td>
<td id="Ax1.T9.1.2.5" class="ltx_td ltx_align_left ltx_border_t">0.477</td>
<td id="Ax1.T9.1.2.6" class="ltx_td ltx_align_left ltx_border_t">0.463</td>
<td id="Ax1.T9.1.2.7" class="ltx_td ltx_align_left ltx_border_t">0.466</td>
<td id="Ax1.T9.1.2.8" class="ltx_td ltx_align_left ltx_border_t">0.45</td>
<td id="Ax1.T9.1.2.9" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_t">0.5</td>
</tr>
<tr id="Ax1.T9.1.3" class="ltx_tr">
<td id="Ax1.T9.1.3.1" class="ltx_td ltx_align_left">FOMC</td>
<td id="Ax1.T9.1.3.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T9.1.3.3" class="ltx_td ltx_align_left">0.754</td>
<td id="Ax1.T9.1.3.4" class="ltx_td ltx_align_left">0.738</td>
<td id="Ax1.T9.1.3.5" class="ltx_td ltx_align_left">0.748</td>
<td id="Ax1.T9.1.3.6" class="ltx_td ltx_align_left"><span id="Ax1.T9.1.3.6.1" class="ltx_text" style="color:#1F2329;">0.03</span></td>
<td id="Ax1.T9.1.3.7" class="ltx_td ltx_align_left">0.721</td>
<td id="Ax1.T9.1.3.8" class="ltx_td ltx_align_left">0.71</td>
<td id="Ax1.T9.1.3.9" class="ltx_td ltx_nopad_r ltx_align_left">0.717</td>
</tr>
<tr id="Ax1.T9.1.4" class="ltx_tr">
<td id="Ax1.T9.1.4.1" class="ltx_td ltx_align_left">MeetingBank</td>
<td id="Ax1.T9.1.4.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T9.1.4.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T9.1.4.4" class="ltx_td ltx_align_left">0.533</td>
<td id="Ax1.T9.1.4.5" class="ltx_td ltx_align_left">0.51</td>
<td id="Ax1.T9.1.4.6" class="ltx_td ltx_align_left">0.375</td>
<td id="Ax1.T9.1.4.7" class="ltx_td ltx_align_left">0.421</td>
<td id="Ax1.T9.1.4.8" class="ltx_td ltx_align_left">0.385</td>
<td id="Ax1.T9.1.4.9" class="ltx_td ltx_nopad_r ltx_align_left">0.351</td>
</tr>
<tr id="Ax1.T9.1.5" class="ltx_tr">
<td id="Ax1.T9.1.5.1" class="ltx_td ltx_align_left">Py150</td>
<td id="Ax1.T9.1.5.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T9.1.5.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T9.1.5.4" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T9.1.5.5" class="ltx_td ltx_align_left">0.568</td>
<td id="Ax1.T9.1.5.6" class="ltx_td ltx_align_left">0.538</td>
<td id="Ax1.T9.1.5.7" class="ltx_td ltx_align_left">0.537</td>
<td id="Ax1.T9.1.5.8" class="ltx_td ltx_align_left">0.541</td>
<td id="Ax1.T9.1.5.9" class="ltx_td ltx_nopad_r ltx_align_left">0.547</td>
</tr>
<tr id="Ax1.T9.1.6" class="ltx_tr">
<td id="Ax1.T9.1.6.1" class="ltx_td ltx_align_left">ScienceQA</td>
<td id="Ax1.T9.1.6.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T9.1.6.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T9.1.6.4" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T9.1.6.5" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T9.1.6.6" class="ltx_td ltx_align_left">0.8</td>
<td id="Ax1.T9.1.6.7" class="ltx_td ltx_align_left">0.655</td>
<td id="Ax1.T9.1.6.8" class="ltx_td ltx_align_left">0.241</td>
<td id="Ax1.T9.1.6.9" class="ltx_td ltx_nopad_r ltx_align_left">0.55</td>
</tr>
<tr id="Ax1.T9.1.7" class="ltx_tr">
<td id="Ax1.T9.1.7.1" class="ltx_td ltx_align_left">NumGLUE-cm</td>
<td id="Ax1.T9.1.7.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T9.1.7.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T9.1.7.4" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T9.1.7.5" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T9.1.7.6" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T9.1.7.7" class="ltx_td ltx_align_left">0.333</td>
<td id="Ax1.T9.1.7.8" class="ltx_td ltx_align_left">0.284</td>
<td id="Ax1.T9.1.7.9" class="ltx_td ltx_nopad_r ltx_align_left">0.296</td>
</tr>
<tr id="Ax1.T9.1.8" class="ltx_tr">
<td id="Ax1.T9.1.8.1" class="ltx_td ltx_align_left">NumGLUE-ds</td>
<td id="Ax1.T9.1.8.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T9.1.8.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T9.1.8.4" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T9.1.8.5" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T9.1.8.6" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T9.1.8.7" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T9.1.8.8" class="ltx_td ltx_align_left">0.618</td>
<td id="Ax1.T9.1.8.9" class="ltx_td ltx_nopad_r ltx_align_left">0.622</td>
</tr>
<tr id="Ax1.T9.1.9" class="ltx_tr">
<td id="Ax1.T9.1.9.1" class="ltx_td ltx_align_left">20Minuten</td>
<td id="Ax1.T9.1.9.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T9.1.9.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T9.1.9.4" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T9.1.9.5" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T9.1.9.6" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T9.1.9.7" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T9.1.9.8" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T9.1.9.9" class="ltx_td ltx_nopad_r ltx_align_left">0.408</td>
</tr>
<tr id="Ax1.T9.1.10" class="ltx_tr">
<td id="Ax1.T9.1.10.1" class="ltx_td ltx_align_left ltx_border_t">average</td>
<td id="Ax1.T9.1.10.2" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T9.1.10.3" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T9.1.10.4" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T9.1.10.5" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T9.1.10.6" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T9.1.10.7" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T9.1.10.8" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T9.1.10.9" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_t">0.499</td>
</tr>
<tr id="Ax1.T9.1.11" class="ltx_tr">
<td id="Ax1.T9.1.11.1" class="ltx_td ltx_align_left ltx_border_bb">BWT</td>
<td id="Ax1.T9.1.11.2" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T9.1.11.3" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T9.1.11.4" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T9.1.11.5" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T9.1.11.6" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T9.1.11.7" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T9.1.11.8" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T9.1.11.9" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_bb">-0.07</td>
</tr>
</table>
</figure>
<figure id="Ax1.T10" class="ltx_table">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table">Table 10: </span>Detailed results of continual learning of Vicuna-7b.</figcaption>
<table id="Ax1.T10.1" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="Ax1.T10.1.1" class="ltx_tr">
<td id="Ax1.T10.1.1.1" class="ltx_td ltx_align_left ltx_border_tt">Task\Round</td>
<td id="Ax1.T10.1.1.2" class="ltx_td ltx_align_left ltx_border_tt">1</td>
<td id="Ax1.T10.1.1.3" class="ltx_td ltx_align_left ltx_border_tt">2</td>
<td id="Ax1.T10.1.1.4" class="ltx_td ltx_align_left ltx_border_tt">3</td>
<td id="Ax1.T10.1.1.5" class="ltx_td ltx_align_left ltx_border_tt">4</td>
<td id="Ax1.T10.1.1.6" class="ltx_td ltx_align_left ltx_border_tt">5</td>
<td id="Ax1.T10.1.1.7" class="ltx_td ltx_align_left ltx_border_tt">6</td>
<td id="Ax1.T10.1.1.8" class="ltx_td ltx_align_left ltx_border_tt">7</td>
<td id="Ax1.T10.1.1.9" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_tt">8</td>
</tr>
<tr id="Ax1.T10.1.2" class="ltx_tr">
<td id="Ax1.T10.1.2.1" class="ltx_td ltx_align_left ltx_border_t">C-STANCE</td>
<td id="Ax1.T10.1.2.2" class="ltx_td ltx_align_left ltx_border_t">0.532</td>
<td id="Ax1.T10.1.2.3" class="ltx_td ltx_align_left ltx_border_t">0.451</td>
<td id="Ax1.T10.1.2.4" class="ltx_td ltx_align_left ltx_border_t">0.439</td>
<td id="Ax1.T10.1.2.5" class="ltx_td ltx_align_left ltx_border_t">0.448</td>
<td id="Ax1.T10.1.2.6" class="ltx_td ltx_align_left ltx_border_t">0.149</td>
<td id="Ax1.T10.1.2.7" class="ltx_td ltx_align_left ltx_border_t">0.477</td>
<td id="Ax1.T10.1.2.8" class="ltx_td ltx_align_left ltx_border_t">0.47</td>
<td id="Ax1.T10.1.2.9" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_t">0.476</td>
</tr>
<tr id="Ax1.T10.1.3" class="ltx_tr">
<td id="Ax1.T10.1.3.1" class="ltx_td ltx_align_left">FOMC</td>
<td id="Ax1.T10.1.3.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T10.1.3.3" class="ltx_td ltx_align_left">0.738</td>
<td id="Ax1.T10.1.3.4" class="ltx_td ltx_align_left">0.732</td>
<td id="Ax1.T10.1.3.5" class="ltx_td ltx_align_left">0.744</td>
<td id="Ax1.T10.1.3.6" class="ltx_td ltx_align_left"><span id="Ax1.T10.1.3.6.1" class="ltx_text" style="color:#1F2329;">0</span></td>
<td id="Ax1.T10.1.3.7" class="ltx_td ltx_align_left">0.605</td>
<td id="Ax1.T10.1.3.8" class="ltx_td ltx_align_left">0.567</td>
<td id="Ax1.T10.1.3.9" class="ltx_td ltx_nopad_r ltx_align_left">0.675</td>
</tr>
<tr id="Ax1.T10.1.4" class="ltx_tr">
<td id="Ax1.T10.1.4.1" class="ltx_td ltx_align_left">MeetingBank</td>
<td id="Ax1.T10.1.4.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T10.1.4.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T10.1.4.4" class="ltx_td ltx_align_left">0.519</td>
<td id="Ax1.T10.1.4.5" class="ltx_td ltx_align_left">0.447</td>
<td id="Ax1.T10.1.4.6" class="ltx_td ltx_align_left">0.443</td>
<td id="Ax1.T10.1.4.7" class="ltx_td ltx_align_left">0.427</td>
<td id="Ax1.T10.1.4.8" class="ltx_td ltx_align_left">0.417</td>
<td id="Ax1.T10.1.4.9" class="ltx_td ltx_nopad_r ltx_align_left">0.439</td>
</tr>
<tr id="Ax1.T10.1.5" class="ltx_tr">
<td id="Ax1.T10.1.5.1" class="ltx_td ltx_align_left">Py150</td>
<td id="Ax1.T10.1.5.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T10.1.5.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T10.1.5.4" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T10.1.5.5" class="ltx_td ltx_align_left">0.577</td>
<td id="Ax1.T10.1.5.6" class="ltx_td ltx_align_left">0.384</td>
<td id="Ax1.T10.1.5.7" class="ltx_td ltx_align_left">0.486</td>
<td id="Ax1.T10.1.5.8" class="ltx_td ltx_align_left">0.482</td>
<td id="Ax1.T10.1.5.9" class="ltx_td ltx_nopad_r ltx_align_left">0.482</td>
</tr>
<tr id="Ax1.T10.1.6" class="ltx_tr">
<td id="Ax1.T10.1.6.1" class="ltx_td ltx_align_left">ScienceQA</td>
<td id="Ax1.T10.1.6.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T10.1.6.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T10.1.6.4" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T10.1.6.5" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T10.1.6.6" class="ltx_td ltx_align_left">0.773</td>
<td id="Ax1.T10.1.6.7" class="ltx_td ltx_align_left">0.7</td>
<td id="Ax1.T10.1.6.8" class="ltx_td ltx_align_left">0.608</td>
<td id="Ax1.T10.1.6.9" class="ltx_td ltx_nopad_r ltx_align_left">0.649</td>
</tr>
<tr id="Ax1.T10.1.7" class="ltx_tr">
<td id="Ax1.T10.1.7.1" class="ltx_td ltx_align_left">NumGLUE-cm</td>
<td id="Ax1.T10.1.7.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T10.1.7.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T10.1.7.4" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T10.1.7.5" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T10.1.7.6" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T10.1.7.7" class="ltx_td ltx_align_left">0.407</td>
<td id="Ax1.T10.1.7.8" class="ltx_td ltx_align_left">0.247</td>
<td id="Ax1.T10.1.7.9" class="ltx_td ltx_nopad_r ltx_align_left">0.296</td>
</tr>
<tr id="Ax1.T10.1.8" class="ltx_tr">
<td id="Ax1.T10.1.8.1" class="ltx_td ltx_align_left">NumGLUE-ds</td>
<td id="Ax1.T10.1.8.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T10.1.8.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T10.1.8.4" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T10.1.8.5" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T10.1.8.6" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T10.1.8.7" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T10.1.8.8" class="ltx_td ltx_align_left">0.578</td>
<td id="Ax1.T10.1.8.9" class="ltx_td ltx_nopad_r ltx_align_left">0.517</td>
</tr>
<tr id="Ax1.T10.1.9" class="ltx_tr">
<td id="Ax1.T10.1.9.1" class="ltx_td ltx_align_left">20Minuten</td>
<td id="Ax1.T10.1.9.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T10.1.9.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T10.1.9.4" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T10.1.9.5" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T10.1.9.6" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T10.1.9.7" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T10.1.9.8" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T10.1.9.9" class="ltx_td ltx_nopad_r ltx_align_left">0.403</td>
</tr>
<tr id="Ax1.T10.1.10" class="ltx_tr">
<td id="Ax1.T10.1.10.1" class="ltx_td ltx_align_left ltx_border_t">average</td>
<td id="Ax1.T10.1.10.2" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T10.1.10.3" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T10.1.10.4" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T10.1.10.5" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T10.1.10.6" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T10.1.10.7" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T10.1.10.8" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T10.1.10.9" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_t">0.492</td>
</tr>
<tr id="Ax1.T10.1.11" class="ltx_tr">
<td id="Ax1.T10.1.11.1" class="ltx_td ltx_align_left ltx_border_bb">BWT</td>
<td id="Ax1.T10.1.11.2" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T10.1.11.3" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T10.1.11.4" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T10.1.11.5" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T10.1.11.6" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T10.1.11.7" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T10.1.11.8" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T10.1.11.9" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_bb">-0.084</td>
</tr>
</table>
</figure>
<figure id="Ax1.T11" class="ltx_table">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table">Table 11: </span>Detailed results of continual learning of Vicuna-13b.</figcaption>
<table id="Ax1.T11.1" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="Ax1.T11.1.1" class="ltx_tr">
<td id="Ax1.T11.1.1.1" class="ltx_td ltx_align_left ltx_border_tt">Task\Round</td>
<td id="Ax1.T11.1.1.2" class="ltx_td ltx_align_left ltx_border_tt">1</td>
<td id="Ax1.T11.1.1.3" class="ltx_td ltx_align_left ltx_border_tt">2</td>
<td id="Ax1.T11.1.1.4" class="ltx_td ltx_align_left ltx_border_tt">3</td>
<td id="Ax1.T11.1.1.5" class="ltx_td ltx_align_left ltx_border_tt">4</td>
<td id="Ax1.T11.1.1.6" class="ltx_td ltx_align_left ltx_border_tt">5</td>
<td id="Ax1.T11.1.1.7" class="ltx_td ltx_align_left ltx_border_tt">6</td>
<td id="Ax1.T11.1.1.8" class="ltx_td ltx_align_left ltx_border_tt">7</td>
<td id="Ax1.T11.1.1.9" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_tt">8</td>
</tr>
<tr id="Ax1.T11.1.2" class="ltx_tr">
<td id="Ax1.T11.1.2.1" class="ltx_td ltx_align_left ltx_border_t">C-STANCE</td>
<td id="Ax1.T11.1.2.2" class="ltx_td ltx_align_left ltx_border_t">0.527</td>
<td id="Ax1.T11.1.2.3" class="ltx_td ltx_align_left ltx_border_t">0.43</td>
<td id="Ax1.T11.1.2.4" class="ltx_td ltx_align_left ltx_border_t">0.471</td>
<td id="Ax1.T11.1.2.5" class="ltx_td ltx_align_left ltx_border_t">0.497</td>
<td id="Ax1.T11.1.2.6" class="ltx_td ltx_align_left ltx_border_t">0.374</td>
<td id="Ax1.T11.1.2.7" class="ltx_td ltx_align_left ltx_border_t">0.468</td>
<td id="Ax1.T11.1.2.8" class="ltx_td ltx_align_left ltx_border_t">0.469</td>
<td id="Ax1.T11.1.2.9" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_t">0.484</td>
</tr>
<tr id="Ax1.T11.1.3" class="ltx_tr">
<td id="Ax1.T11.1.3.1" class="ltx_td ltx_align_left">FOMC</td>
<td id="Ax1.T11.1.3.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T11.1.3.3" class="ltx_td ltx_align_left">0.741</td>
<td id="Ax1.T11.1.3.4" class="ltx_td ltx_align_left">0.739</td>
<td id="Ax1.T11.1.3.5" class="ltx_td ltx_align_left">0.731</td>
<td id="Ax1.T11.1.3.6" class="ltx_td ltx_align_left"><span id="Ax1.T11.1.3.6.1" class="ltx_text" style="color:#1F2329;">0</span></td>
<td id="Ax1.T11.1.3.7" class="ltx_td ltx_align_left">0.754</td>
<td id="Ax1.T11.1.3.8" class="ltx_td ltx_align_left">0.678</td>
<td id="Ax1.T11.1.3.9" class="ltx_td ltx_nopad_r ltx_align_left">0.714</td>
</tr>
<tr id="Ax1.T11.1.4" class="ltx_tr">
<td id="Ax1.T11.1.4.1" class="ltx_td ltx_align_left">MeetingBank</td>
<td id="Ax1.T11.1.4.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T11.1.4.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T11.1.4.4" class="ltx_td ltx_align_left">0.549</td>
<td id="Ax1.T11.1.4.5" class="ltx_td ltx_align_left">0.532</td>
<td id="Ax1.T11.1.4.6" class="ltx_td ltx_align_left">0.53</td>
<td id="Ax1.T11.1.4.7" class="ltx_td ltx_align_left">0.491</td>
<td id="Ax1.T11.1.4.8" class="ltx_td ltx_align_left">0.427</td>
<td id="Ax1.T11.1.4.9" class="ltx_td ltx_nopad_r ltx_align_left">0.412</td>
</tr>
<tr id="Ax1.T11.1.5" class="ltx_tr">
<td id="Ax1.T11.1.5.1" class="ltx_td ltx_align_left">Py150</td>
<td id="Ax1.T11.1.5.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T11.1.5.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T11.1.5.4" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T11.1.5.5" class="ltx_td ltx_align_left">0.564</td>
<td id="Ax1.T11.1.5.6" class="ltx_td ltx_align_left">0.54</td>
<td id="Ax1.T11.1.5.7" class="ltx_td ltx_align_left">0.546</td>
<td id="Ax1.T11.1.5.8" class="ltx_td ltx_align_left">0.538</td>
<td id="Ax1.T11.1.5.9" class="ltx_td ltx_nopad_r ltx_align_left">0.552</td>
</tr>
<tr id="Ax1.T11.1.6" class="ltx_tr">
<td id="Ax1.T11.1.6.1" class="ltx_td ltx_align_left">ScienceQA</td>
<td id="Ax1.T11.1.6.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T11.1.6.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T11.1.6.4" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T11.1.6.5" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T11.1.6.6" class="ltx_td ltx_align_left">0.79</td>
<td id="Ax1.T11.1.6.7" class="ltx_td ltx_align_left">0.616</td>
<td id="Ax1.T11.1.6.8" class="ltx_td ltx_align_left">0.586</td>
<td id="Ax1.T11.1.6.9" class="ltx_td ltx_nopad_r ltx_align_left">0.633</td>
</tr>
<tr id="Ax1.T11.1.7" class="ltx_tr">
<td id="Ax1.T11.1.7.1" class="ltx_td ltx_align_left">NumGLUE-cm</td>
<td id="Ax1.T11.1.7.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T11.1.7.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T11.1.7.4" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T11.1.7.5" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T11.1.7.6" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T11.1.7.7" class="ltx_td ltx_align_left">0.346</td>
<td id="Ax1.T11.1.7.8" class="ltx_td ltx_align_left">0.309</td>
<td id="Ax1.T11.1.7.9" class="ltx_td ltx_nopad_r ltx_align_left">0.358</td>
</tr>
<tr id="Ax1.T11.1.8" class="ltx_tr">
<td id="Ax1.T11.1.8.1" class="ltx_td ltx_align_left">NumGLUE-ds</td>
<td id="Ax1.T11.1.8.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T11.1.8.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T11.1.8.4" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T11.1.8.5" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T11.1.8.6" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T11.1.8.7" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T11.1.8.8" class="ltx_td ltx_align_left">0.622</td>
<td id="Ax1.T11.1.8.9" class="ltx_td ltx_nopad_r ltx_align_left">0.572</td>
</tr>
<tr id="Ax1.T11.1.9" class="ltx_tr">
<td id="Ax1.T11.1.9.1" class="ltx_td ltx_align_left">20Minuten</td>
<td id="Ax1.T11.1.9.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T11.1.9.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T11.1.9.4" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T11.1.9.5" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T11.1.9.6" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T11.1.9.7" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T11.1.9.8" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T11.1.9.9" class="ltx_td ltx_nopad_r ltx_align_left">0.41</td>
</tr>
<tr id="Ax1.T11.1.10" class="ltx_tr">
<td id="Ax1.T11.1.10.1" class="ltx_td ltx_align_left ltx_border_t">average</td>
<td id="Ax1.T11.1.10.2" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T11.1.10.3" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T11.1.10.4" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T11.1.10.5" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T11.1.10.6" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T11.1.10.7" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T11.1.10.8" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T11.1.10.9" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_t">0.517</td>
</tr>
<tr id="Ax1.T11.1.11" class="ltx_tr">
<td id="Ax1.T11.1.11.1" class="ltx_td ltx_align_left ltx_border_bb">BWT</td>
<td id="Ax1.T11.1.11.2" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T11.1.11.3" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T11.1.11.4" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T11.1.11.5" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T11.1.11.6" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T11.1.11.7" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T11.1.11.8" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T11.1.11.9" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_bb">-0.059</td>
</tr>
</table>
</figure>
</section>
<section id="Ax1.SS4.SSS3" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">.4.3 </span>SeqLoraFT method</h4>

<div id="Ax1.SS4.SSS3.p1" class="ltx_para">
<p id="Ax1.SS4.SSS3.p1.1" class="ltx_p">Table 12 16 shows the detailed performance of different models of each round during the continual learning. SeqLoRAFT represents sequential fine-tuning with LoRA adapters.</p>
</div>
<figure id="Ax1.T12" class="ltx_table">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table">Table 12: </span>Detailed results of continual learning of Baichuan-7b with LoRA adapters.</figcaption>
<table id="Ax1.T12.1" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="Ax1.T12.1.1" class="ltx_tr">
<td id="Ax1.T12.1.1.1" class="ltx_td ltx_align_left ltx_border_tt">Task\Round</td>
<td id="Ax1.T12.1.1.2" class="ltx_td ltx_align_left ltx_border_tt">1</td>
<td id="Ax1.T12.1.1.3" class="ltx_td ltx_align_left ltx_border_tt">2</td>
<td id="Ax1.T12.1.1.4" class="ltx_td ltx_align_left ltx_border_tt">3</td>
<td id="Ax1.T12.1.1.5" class="ltx_td ltx_align_left ltx_border_tt">4</td>
<td id="Ax1.T12.1.1.6" class="ltx_td ltx_align_left ltx_border_tt">5</td>
<td id="Ax1.T12.1.1.7" class="ltx_td ltx_align_left ltx_border_tt">6</td>
<td id="Ax1.T12.1.1.8" class="ltx_td ltx_align_left ltx_border_tt">7</td>
<td id="Ax1.T12.1.1.9" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_tt">8</td>
</tr>
<tr id="Ax1.T12.1.2" class="ltx_tr">
<td id="Ax1.T12.1.2.1" class="ltx_td ltx_align_left ltx_border_t">C-STANCE</td>
<td id="Ax1.T12.1.2.2" class="ltx_td ltx_align_left ltx_border_t">0.613</td>
<td id="Ax1.T12.1.2.3" class="ltx_td ltx_align_left ltx_border_t">0.601</td>
<td id="Ax1.T12.1.2.4" class="ltx_td ltx_align_left ltx_border_t">0.597</td>
<td id="Ax1.T12.1.2.5" class="ltx_td ltx_align_left ltx_border_t">0.584</td>
<td id="Ax1.T12.1.2.6" class="ltx_td ltx_align_left ltx_border_t">0.506</td>
<td id="Ax1.T12.1.2.7" class="ltx_td ltx_align_left ltx_border_t">0.504</td>
<td id="Ax1.T12.1.2.8" class="ltx_td ltx_align_left ltx_border_t">0.53</td>
<td id="Ax1.T12.1.2.9" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_t">0.477</td>
</tr>
<tr id="Ax1.T12.1.3" class="ltx_tr">
<td id="Ax1.T12.1.3.1" class="ltx_td ltx_align_left">FOMC</td>
<td id="Ax1.T12.1.3.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T12.1.3.3" class="ltx_td ltx_align_left">0.652</td>
<td id="Ax1.T12.1.3.4" class="ltx_td ltx_align_left">0.604</td>
<td id="Ax1.T12.1.3.5" class="ltx_td ltx_align_left">0.591</td>
<td id="Ax1.T12.1.3.6" class="ltx_td ltx_align_left"><span id="Ax1.T12.1.3.6.1" class="ltx_text" style="color:#1F2329;">0.602</span></td>
<td id="Ax1.T12.1.3.7" class="ltx_td ltx_align_left">0.588</td>
<td id="Ax1.T12.1.3.8" class="ltx_td ltx_align_left">0.587</td>
<td id="Ax1.T12.1.3.9" class="ltx_td ltx_nopad_r ltx_align_left">0.417</td>
</tr>
<tr id="Ax1.T12.1.4" class="ltx_tr">
<td id="Ax1.T12.1.4.1" class="ltx_td ltx_align_left">MeetingBank</td>
<td id="Ax1.T12.1.4.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T12.1.4.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T12.1.4.4" class="ltx_td ltx_align_left">0.345</td>
<td id="Ax1.T12.1.4.5" class="ltx_td ltx_align_left">0.334</td>
<td id="Ax1.T12.1.4.6" class="ltx_td ltx_align_left">0.333</td>
<td id="Ax1.T12.1.4.7" class="ltx_td ltx_align_left">0.343</td>
<td id="Ax1.T12.1.4.8" class="ltx_td ltx_align_left">0.34</td>
<td id="Ax1.T12.1.4.9" class="ltx_td ltx_nopad_r ltx_align_left">0.337</td>
</tr>
<tr id="Ax1.T12.1.5" class="ltx_tr">
<td id="Ax1.T12.1.5.1" class="ltx_td ltx_align_left">Py150</td>
<td id="Ax1.T12.1.5.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T12.1.5.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T12.1.5.4" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T12.1.5.5" class="ltx_td ltx_align_left">0.588</td>
<td id="Ax1.T12.1.5.6" class="ltx_td ltx_align_left">0.472</td>
<td id="Ax1.T12.1.5.7" class="ltx_td ltx_align_left">0.539</td>
<td id="Ax1.T12.1.5.8" class="ltx_td ltx_align_left">0.517</td>
<td id="Ax1.T12.1.5.9" class="ltx_td ltx_nopad_r ltx_align_left">0.472</td>
</tr>
<tr id="Ax1.T12.1.6" class="ltx_tr">
<td id="Ax1.T12.1.6.1" class="ltx_td ltx_align_left">ScienceQA</td>
<td id="Ax1.T12.1.6.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T12.1.6.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T12.1.6.4" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T12.1.6.5" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T12.1.6.6" class="ltx_td ltx_align_left">0.641</td>
<td id="Ax1.T12.1.6.7" class="ltx_td ltx_align_left">0.68</td>
<td id="Ax1.T12.1.6.8" class="ltx_td ltx_align_left">0.625</td>
<td id="Ax1.T12.1.6.9" class="ltx_td ltx_nopad_r ltx_align_left">0.63</td>
</tr>
<tr id="Ax1.T12.1.7" class="ltx_tr">
<td id="Ax1.T12.1.7.1" class="ltx_td ltx_align_left">NumGLUE-cm</td>
<td id="Ax1.T12.1.7.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T12.1.7.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T12.1.7.4" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T12.1.7.5" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T12.1.7.6" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T12.1.7.7" class="ltx_td ltx_align_left">0.457</td>
<td id="Ax1.T12.1.7.8" class="ltx_td ltx_align_left">0.432</td>
<td id="Ax1.T12.1.7.9" class="ltx_td ltx_nopad_r ltx_align_left">0.407</td>
</tr>
<tr id="Ax1.T12.1.8" class="ltx_tr">
<td id="Ax1.T12.1.8.1" class="ltx_td ltx_align_left">NumGLUE-ds</td>
<td id="Ax1.T12.1.8.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T12.1.8.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T12.1.8.4" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T12.1.8.5" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T12.1.8.6" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T12.1.8.7" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T12.1.8.8" class="ltx_td ltx_align_left">0.43</td>
<td id="Ax1.T12.1.8.9" class="ltx_td ltx_nopad_r ltx_align_left">0.36</td>
</tr>
<tr id="Ax1.T12.1.9" class="ltx_tr">
<td id="Ax1.T12.1.9.1" class="ltx_td ltx_align_left">20Minuten</td>
<td id="Ax1.T12.1.9.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T12.1.9.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T12.1.9.4" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T12.1.9.5" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T12.1.9.6" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T12.1.9.7" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T12.1.9.8" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T12.1.9.9" class="ltx_td ltx_nopad_r ltx_align_left">0.407</td>
</tr>
<tr id="Ax1.T12.1.10" class="ltx_tr">
<td id="Ax1.T12.1.10.1" class="ltx_td ltx_align_left ltx_border_t">average</td>
<td id="Ax1.T12.1.10.2" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T12.1.10.3" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T12.1.10.4" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T12.1.10.5" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T12.1.10.6" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T12.1.10.7" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T12.1.10.8" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T12.1.10.9" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_t">0.438</td>
</tr>
<tr id="Ax1.T12.1.11" class="ltx_tr">
<td id="Ax1.T12.1.11.1" class="ltx_td ltx_align_left ltx_border_bb">BWT</td>
<td id="Ax1.T12.1.11.2" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T12.1.11.3" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T12.1.11.4" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T12.1.11.5" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T12.1.11.6" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T12.1.11.7" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T12.1.11.8" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T12.1.11.9" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_bb">-0.090</td>
</tr>
</table>
</figure>
<figure id="Ax1.T13" class="ltx_table">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table">Table 13: </span>Detailed results of continual learning of LLaMA-7b-chat with LoRA adapters.</figcaption>
<table id="Ax1.T13.1" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="Ax1.T13.1.1" class="ltx_tr">
<td id="Ax1.T13.1.1.1" class="ltx_td ltx_align_left ltx_border_tt">Task\Round</td>
<td id="Ax1.T13.1.1.2" class="ltx_td ltx_align_left ltx_border_tt">1</td>
<td id="Ax1.T13.1.1.3" class="ltx_td ltx_align_left ltx_border_tt">2</td>
<td id="Ax1.T13.1.1.4" class="ltx_td ltx_align_left ltx_border_tt">3</td>
<td id="Ax1.T13.1.1.5" class="ltx_td ltx_align_left ltx_border_tt">4</td>
<td id="Ax1.T13.1.1.6" class="ltx_td ltx_align_left ltx_border_tt">5</td>
<td id="Ax1.T13.1.1.7" class="ltx_td ltx_align_left ltx_border_tt">6</td>
<td id="Ax1.T13.1.1.8" class="ltx_td ltx_align_left ltx_border_tt">7</td>
<td id="Ax1.T13.1.1.9" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_tt">8</td>
</tr>
<tr id="Ax1.T13.1.2" class="ltx_tr">
<td id="Ax1.T13.1.2.1" class="ltx_td ltx_align_left ltx_border_t">C-STANCE</td>
<td id="Ax1.T13.1.2.2" class="ltx_td ltx_align_left ltx_border_t">0.511</td>
<td id="Ax1.T13.1.2.3" class="ltx_td ltx_align_left ltx_border_t">0.45</td>
<td id="Ax1.T13.1.2.4" class="ltx_td ltx_align_left ltx_border_t">0.412</td>
<td id="Ax1.T13.1.2.5" class="ltx_td ltx_align_left ltx_border_t">0.373</td>
<td id="Ax1.T13.1.2.6" class="ltx_td ltx_align_left ltx_border_t">0.133</td>
<td id="Ax1.T13.1.2.7" class="ltx_td ltx_align_left ltx_border_t">0.391</td>
<td id="Ax1.T13.1.2.8" class="ltx_td ltx_align_left ltx_border_t">0.294</td>
<td id="Ax1.T13.1.2.9" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_t">0.277</td>
</tr>
<tr id="Ax1.T13.1.3" class="ltx_tr">
<td id="Ax1.T13.1.3.1" class="ltx_td ltx_align_left">FOMC</td>
<td id="Ax1.T13.1.3.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T13.1.3.3" class="ltx_td ltx_align_left">0.713</td>
<td id="Ax1.T13.1.3.4" class="ltx_td ltx_align_left">0.55</td>
<td id="Ax1.T13.1.3.5" class="ltx_td ltx_align_left">0.452</td>
<td id="Ax1.T13.1.3.6" class="ltx_td ltx_align_left"><span id="Ax1.T13.1.3.6.1" class="ltx_text" style="color:#1F2329;">0</span></td>
<td id="Ax1.T13.1.3.7" class="ltx_td ltx_align_left">0.421</td>
<td id="Ax1.T13.1.3.8" class="ltx_td ltx_align_left">0.341</td>
<td id="Ax1.T13.1.3.9" class="ltx_td ltx_nopad_r ltx_align_left">0.24</td>
</tr>
<tr id="Ax1.T13.1.4" class="ltx_tr">
<td id="Ax1.T13.1.4.1" class="ltx_td ltx_align_left">MeetingBank</td>
<td id="Ax1.T13.1.4.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T13.1.4.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T13.1.4.4" class="ltx_td ltx_align_left">0.51</td>
<td id="Ax1.T13.1.4.5" class="ltx_td ltx_align_left">0.212</td>
<td id="Ax1.T13.1.4.6" class="ltx_td ltx_align_left">0.151</td>
<td id="Ax1.T13.1.4.7" class="ltx_td ltx_align_left">0.067</td>
<td id="Ax1.T13.1.4.8" class="ltx_td ltx_align_left">0.037</td>
<td id="Ax1.T13.1.4.9" class="ltx_td ltx_nopad_r ltx_align_left">0.121</td>
</tr>
<tr id="Ax1.T13.1.5" class="ltx_tr">
<td id="Ax1.T13.1.5.1" class="ltx_td ltx_align_left">Py150</td>
<td id="Ax1.T13.1.5.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T13.1.5.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T13.1.5.4" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T13.1.5.5" class="ltx_td ltx_align_left">0.578</td>
<td id="Ax1.T13.1.5.6" class="ltx_td ltx_align_left">0.004</td>
<td id="Ax1.T13.1.5.7" class="ltx_td ltx_align_left">0.495</td>
<td id="Ax1.T13.1.5.8" class="ltx_td ltx_align_left">0.452</td>
<td id="Ax1.T13.1.5.9" class="ltx_td ltx_nopad_r ltx_align_left">0.004</td>
</tr>
<tr id="Ax1.T13.1.6" class="ltx_tr">
<td id="Ax1.T13.1.6.1" class="ltx_td ltx_align_left">ScienceQA</td>
<td id="Ax1.T13.1.6.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T13.1.6.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T13.1.6.4" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T13.1.6.5" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T13.1.6.6" class="ltx_td ltx_align_left">0.68</td>
<td id="Ax1.T13.1.6.7" class="ltx_td ltx_align_left">0.645</td>
<td id="Ax1.T13.1.6.8" class="ltx_td ltx_align_left">0.535</td>
<td id="Ax1.T13.1.6.9" class="ltx_td ltx_nopad_r ltx_align_left">0</td>
</tr>
<tr id="Ax1.T13.1.7" class="ltx_tr">
<td id="Ax1.T13.1.7.1" class="ltx_td ltx_align_left">NumGLUE-cm</td>
<td id="Ax1.T13.1.7.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T13.1.7.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T13.1.7.4" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T13.1.7.5" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T13.1.7.6" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T13.1.7.7" class="ltx_td ltx_align_left">0.37</td>
<td id="Ax1.T13.1.7.8" class="ltx_td ltx_align_left">0.235</td>
<td id="Ax1.T13.1.7.9" class="ltx_td ltx_nopad_r ltx_align_left">0</td>
</tr>
<tr id="Ax1.T13.1.8" class="ltx_tr">
<td id="Ax1.T13.1.8.1" class="ltx_td ltx_align_left">NumGLUE-ds</td>
<td id="Ax1.T13.1.8.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T13.1.8.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T13.1.8.4" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T13.1.8.5" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T13.1.8.6" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T13.1.8.7" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T13.1.8.8" class="ltx_td ltx_align_left">0.486</td>
<td id="Ax1.T13.1.8.9" class="ltx_td ltx_nopad_r ltx_align_left">0</td>
</tr>
<tr id="Ax1.T13.1.9" class="ltx_tr">
<td id="Ax1.T13.1.9.1" class="ltx_td ltx_align_left">20Minuten</td>
<td id="Ax1.T13.1.9.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T13.1.9.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T13.1.9.4" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T13.1.9.5" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T13.1.9.6" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T13.1.9.7" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T13.1.9.8" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T13.1.9.9" class="ltx_td ltx_nopad_r ltx_align_left">0.37</td>
</tr>
<tr id="Ax1.T13.1.10" class="ltx_tr">
<td id="Ax1.T13.1.10.1" class="ltx_td ltx_align_left ltx_border_t">average</td>
<td id="Ax1.T13.1.10.2" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T13.1.10.3" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T13.1.10.4" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T13.1.10.5" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T13.1.10.6" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T13.1.10.7" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T13.1.10.8" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T13.1.10.9" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_t">0.127</td>
</tr>
<tr id="Ax1.T13.1.11" class="ltx_tr">
<td id="Ax1.T13.1.11.1" class="ltx_td ltx_align_left ltx_border_bb">BWT</td>
<td id="Ax1.T13.1.11.2" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T13.1.11.3" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T13.1.11.4" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T13.1.11.5" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T13.1.11.6" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T13.1.11.7" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T13.1.11.8" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T13.1.11.9" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_bb">-0.457</td>
</tr>
</table>
</figure>
<figure id="Ax1.T14" class="ltx_table">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table">Table 14: </span>Detailed results of continual learning of LLaMA-13b with LoRA adapters.</figcaption>
<table id="Ax1.T14.1" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="Ax1.T14.1.1" class="ltx_tr">
<td id="Ax1.T14.1.1.1" class="ltx_td ltx_align_left ltx_border_tt">Task\Round</td>
<td id="Ax1.T14.1.1.2" class="ltx_td ltx_align_left ltx_border_tt">1</td>
<td id="Ax1.T14.1.1.3" class="ltx_td ltx_align_left ltx_border_tt">2</td>
<td id="Ax1.T14.1.1.4" class="ltx_td ltx_align_left ltx_border_tt">3</td>
<td id="Ax1.T14.1.1.5" class="ltx_td ltx_align_left ltx_border_tt">4</td>
<td id="Ax1.T14.1.1.6" class="ltx_td ltx_align_left ltx_border_tt">5</td>
<td id="Ax1.T14.1.1.7" class="ltx_td ltx_align_left ltx_border_tt">6</td>
<td id="Ax1.T14.1.1.8" class="ltx_td ltx_align_left ltx_border_tt">7</td>
<td id="Ax1.T14.1.1.9" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_tt">8</td>
</tr>
<tr id="Ax1.T14.1.2" class="ltx_tr">
<td id="Ax1.T14.1.2.1" class="ltx_td ltx_align_left ltx_border_t">C-STANCE</td>
<td id="Ax1.T14.1.2.2" class="ltx_td ltx_align_left ltx_border_t">0.62</td>
<td id="Ax1.T14.1.2.3" class="ltx_td ltx_align_left ltx_border_t">0.36</td>
<td id="Ax1.T14.1.2.4" class="ltx_td ltx_align_left ltx_border_t">0.432</td>
<td id="Ax1.T14.1.2.5" class="ltx_td ltx_align_left ltx_border_t">0.491</td>
<td id="Ax1.T14.1.2.6" class="ltx_td ltx_align_left ltx_border_t">0.18</td>
<td id="Ax1.T14.1.2.7" class="ltx_td ltx_align_left ltx_border_t">0.42</td>
<td id="Ax1.T14.1.2.8" class="ltx_td ltx_align_left ltx_border_t">0.411</td>
<td id="Ax1.T14.1.2.9" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_t">0.124</td>
</tr>
<tr id="Ax1.T14.1.3" class="ltx_tr">
<td id="Ax1.T14.1.3.1" class="ltx_td ltx_align_left">FOMC</td>
<td id="Ax1.T14.1.3.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T14.1.3.3" class="ltx_td ltx_align_left">0.743</td>
<td id="Ax1.T14.1.3.4" class="ltx_td ltx_align_left">0.681</td>
<td id="Ax1.T14.1.3.5" class="ltx_td ltx_align_left">0.63</td>
<td id="Ax1.T14.1.3.6" class="ltx_td ltx_align_left"><span id="Ax1.T14.1.3.6.1" class="ltx_text" style="color:#1F2329;">0.53</span></td>
<td id="Ax1.T14.1.3.7" class="ltx_td ltx_align_left">0.605</td>
<td id="Ax1.T14.1.3.8" class="ltx_td ltx_align_left">0.579</td>
<td id="Ax1.T14.1.3.9" class="ltx_td ltx_nopad_r ltx_align_left">0</td>
</tr>
<tr id="Ax1.T14.1.4" class="ltx_tr">
<td id="Ax1.T14.1.4.1" class="ltx_td ltx_align_left">MeetingBank</td>
<td id="Ax1.T14.1.4.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T14.1.4.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T14.1.4.4" class="ltx_td ltx_align_left">0.484</td>
<td id="Ax1.T14.1.4.5" class="ltx_td ltx_align_left">0.264</td>
<td id="Ax1.T14.1.4.6" class="ltx_td ltx_align_left">0.201</td>
<td id="Ax1.T14.1.4.7" class="ltx_td ltx_align_left">0.147</td>
<td id="Ax1.T14.1.4.8" class="ltx_td ltx_align_left">0.032</td>
<td id="Ax1.T14.1.4.9" class="ltx_td ltx_nopad_r ltx_align_left">0.122</td>
</tr>
<tr id="Ax1.T14.1.5" class="ltx_tr">
<td id="Ax1.T14.1.5.1" class="ltx_td ltx_align_left">Py150</td>
<td id="Ax1.T14.1.5.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T14.1.5.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T14.1.5.4" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T14.1.5.5" class="ltx_td ltx_align_left">0.581</td>
<td id="Ax1.T14.1.5.6" class="ltx_td ltx_align_left">0.397</td>
<td id="Ax1.T14.1.5.7" class="ltx_td ltx_align_left">0.488</td>
<td id="Ax1.T14.1.5.8" class="ltx_td ltx_align_left">0.497</td>
<td id="Ax1.T14.1.5.9" class="ltx_td ltx_nopad_r ltx_align_left">0.249</td>
</tr>
<tr id="Ax1.T14.1.6" class="ltx_tr">
<td id="Ax1.T14.1.6.1" class="ltx_td ltx_align_left">ScienceQA</td>
<td id="Ax1.T14.1.6.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T14.1.6.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T14.1.6.4" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T14.1.6.5" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T14.1.6.6" class="ltx_td ltx_align_left">0.75</td>
<td id="Ax1.T14.1.6.7" class="ltx_td ltx_align_left">0.729</td>
<td id="Ax1.T14.1.6.8" class="ltx_td ltx_align_left">0.714</td>
<td id="Ax1.T14.1.6.9" class="ltx_td ltx_nopad_r ltx_align_left">0.68</td>
</tr>
<tr id="Ax1.T14.1.7" class="ltx_tr">
<td id="Ax1.T14.1.7.1" class="ltx_td ltx_align_left">NumGLUE-cm</td>
<td id="Ax1.T14.1.7.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T14.1.7.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T14.1.7.4" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T14.1.7.5" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T14.1.7.6" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T14.1.7.7" class="ltx_td ltx_align_left">0.58</td>
<td id="Ax1.T14.1.7.8" class="ltx_td ltx_align_left">0.296</td>
<td id="Ax1.T14.1.7.9" class="ltx_td ltx_nopad_r ltx_align_left">0.259</td>
</tr>
<tr id="Ax1.T14.1.8" class="ltx_tr">
<td id="Ax1.T14.1.8.1" class="ltx_td ltx_align_left">NumGLUE-ds</td>
<td id="Ax1.T14.1.8.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T14.1.8.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T14.1.8.4" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T14.1.8.5" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T14.1.8.6" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T14.1.8.7" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T14.1.8.8" class="ltx_td ltx_align_left">0.62</td>
<td id="Ax1.T14.1.8.9" class="ltx_td ltx_nopad_r ltx_align_left">0.386</td>
</tr>
<tr id="Ax1.T14.1.9" class="ltx_tr">
<td id="Ax1.T14.1.9.1" class="ltx_td ltx_align_left">20Minuten</td>
<td id="Ax1.T14.1.9.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T14.1.9.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T14.1.9.4" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T14.1.9.5" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T14.1.9.6" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T14.1.9.7" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T14.1.9.8" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T14.1.9.9" class="ltx_td ltx_nopad_r ltx_align_left">0.417</td>
</tr>
<tr id="Ax1.T14.1.10" class="ltx_tr">
<td id="Ax1.T14.1.10.1" class="ltx_td ltx_align_left ltx_border_t">average</td>
<td id="Ax1.T14.1.10.2" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T14.1.10.3" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T14.1.10.4" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T14.1.10.5" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T14.1.10.6" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T14.1.10.7" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T14.1.10.8" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T14.1.10.9" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_t">0.28</td>
</tr>
<tr id="Ax1.T14.1.11" class="ltx_tr">
<td id="Ax1.T14.1.11.1" class="ltx_td ltx_align_left ltx_border_bb">BWT</td>
<td id="Ax1.T14.1.11.2" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T14.1.11.3" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T14.1.11.4" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T14.1.11.5" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T14.1.11.6" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T14.1.11.7" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T14.1.11.8" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T14.1.11.9" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_bb">-0.365</td>
</tr>
</table>
</figure>
<figure id="Ax1.T15" class="ltx_table">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table">Table 15: </span>Detailed results of continual learning of Vicuna-7b with LoRA adapters.</figcaption>
<table id="Ax1.T15.1" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="Ax1.T15.1.1" class="ltx_tr">
<td id="Ax1.T15.1.1.1" class="ltx_td ltx_align_left ltx_border_tt">Task\Round</td>
<td id="Ax1.T15.1.1.2" class="ltx_td ltx_align_left ltx_border_tt">1</td>
<td id="Ax1.T15.1.1.3" class="ltx_td ltx_align_left ltx_border_tt">2</td>
<td id="Ax1.T15.1.1.4" class="ltx_td ltx_align_left ltx_border_tt">3</td>
<td id="Ax1.T15.1.1.5" class="ltx_td ltx_align_left ltx_border_tt">4</td>
<td id="Ax1.T15.1.1.6" class="ltx_td ltx_align_left ltx_border_tt">5</td>
<td id="Ax1.T15.1.1.7" class="ltx_td ltx_align_left ltx_border_tt">6</td>
<td id="Ax1.T15.1.1.8" class="ltx_td ltx_align_left ltx_border_tt">7</td>
<td id="Ax1.T15.1.1.9" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_tt">8</td>
</tr>
<tr id="Ax1.T15.1.2" class="ltx_tr">
<td id="Ax1.T15.1.2.1" class="ltx_td ltx_align_left ltx_border_t">C-STANCE</td>
<td id="Ax1.T15.1.2.2" class="ltx_td ltx_align_left ltx_border_t">0.514</td>
<td id="Ax1.T15.1.2.3" class="ltx_td ltx_align_left ltx_border_t">0.452</td>
<td id="Ax1.T15.1.2.4" class="ltx_td ltx_align_left ltx_border_t">0.433</td>
<td id="Ax1.T15.1.2.5" class="ltx_td ltx_align_left ltx_border_t">0.446</td>
<td id="Ax1.T15.1.2.6" class="ltx_td ltx_align_left ltx_border_t">0</td>
<td id="Ax1.T15.1.2.7" class="ltx_td ltx_align_left ltx_border_t">0.344</td>
<td id="Ax1.T15.1.2.8" class="ltx_td ltx_align_left ltx_border_t">0.089</td>
<td id="Ax1.T15.1.2.9" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_t">0.141</td>
</tr>
<tr id="Ax1.T15.1.3" class="ltx_tr">
<td id="Ax1.T15.1.3.1" class="ltx_td ltx_align_left">FOMC</td>
<td id="Ax1.T15.1.3.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T15.1.3.3" class="ltx_td ltx_align_left">0.715</td>
<td id="Ax1.T15.1.3.4" class="ltx_td ltx_align_left">0.48</td>
<td id="Ax1.T15.1.3.5" class="ltx_td ltx_align_left">0.427</td>
<td id="Ax1.T15.1.3.6" class="ltx_td ltx_align_left"><span id="Ax1.T15.1.3.6.1" class="ltx_text" style="color:#1F2329;">0</span></td>
<td id="Ax1.T15.1.3.7" class="ltx_td ltx_align_left">0.272</td>
<td id="Ax1.T15.1.3.8" class="ltx_td ltx_align_left">0.304</td>
<td id="Ax1.T15.1.3.9" class="ltx_td ltx_nopad_r ltx_align_left">0.29</td>
</tr>
<tr id="Ax1.T15.1.4" class="ltx_tr">
<td id="Ax1.T15.1.4.1" class="ltx_td ltx_align_left">MeetingBank</td>
<td id="Ax1.T15.1.4.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T15.1.4.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T15.1.4.4" class="ltx_td ltx_align_left">0.5</td>
<td id="Ax1.T15.1.4.5" class="ltx_td ltx_align_left">0.113</td>
<td id="Ax1.T15.1.4.6" class="ltx_td ltx_align_left">0.144</td>
<td id="Ax1.T15.1.4.7" class="ltx_td ltx_align_left">0.026</td>
<td id="Ax1.T15.1.4.8" class="ltx_td ltx_align_left">0.011</td>
<td id="Ax1.T15.1.4.9" class="ltx_td ltx_nopad_r ltx_align_left">0.07</td>
</tr>
<tr id="Ax1.T15.1.5" class="ltx_tr">
<td id="Ax1.T15.1.5.1" class="ltx_td ltx_align_left">Py150</td>
<td id="Ax1.T15.1.5.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T15.1.5.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T15.1.5.4" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T15.1.5.5" class="ltx_td ltx_align_left">0.573</td>
<td id="Ax1.T15.1.5.6" class="ltx_td ltx_align_left">0.222</td>
<td id="Ax1.T15.1.5.7" class="ltx_td ltx_align_left">0.47</td>
<td id="Ax1.T15.1.5.8" class="ltx_td ltx_align_left">0.452</td>
<td id="Ax1.T15.1.5.9" class="ltx_td ltx_nopad_r ltx_align_left">0.413</td>
</tr>
<tr id="Ax1.T15.1.6" class="ltx_tr">
<td id="Ax1.T15.1.6.1" class="ltx_td ltx_align_left">ScienceQA</td>
<td id="Ax1.T15.1.6.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T15.1.6.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T15.1.6.4" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T15.1.6.5" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T15.1.6.6" class="ltx_td ltx_align_left">0.67</td>
<td id="Ax1.T15.1.6.7" class="ltx_td ltx_align_left">0.632</td>
<td id="Ax1.T15.1.6.8" class="ltx_td ltx_align_left">0.53</td>
<td id="Ax1.T15.1.6.9" class="ltx_td ltx_nopad_r ltx_align_left">0.6</td>
</tr>
<tr id="Ax1.T15.1.7" class="ltx_tr">
<td id="Ax1.T15.1.7.1" class="ltx_td ltx_align_left">NumGLUE-cm</td>
<td id="Ax1.T15.1.7.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T15.1.7.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T15.1.7.4" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T15.1.7.5" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T15.1.7.6" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T15.1.7.7" class="ltx_td ltx_align_left">0.407</td>
<td id="Ax1.T15.1.7.8" class="ltx_td ltx_align_left">0.37</td>
<td id="Ax1.T15.1.7.9" class="ltx_td ltx_nopad_r ltx_align_left">0.259</td>
</tr>
<tr id="Ax1.T15.1.8" class="ltx_tr">
<td id="Ax1.T15.1.8.1" class="ltx_td ltx_align_left">NumGLUE-ds</td>
<td id="Ax1.T15.1.8.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T15.1.8.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T15.1.8.4" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T15.1.8.5" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T15.1.8.6" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T15.1.8.7" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T15.1.8.8" class="ltx_td ltx_align_left">0.545</td>
<td id="Ax1.T15.1.8.9" class="ltx_td ltx_nopad_r ltx_align_left">0.492</td>
</tr>
<tr id="Ax1.T15.1.9" class="ltx_tr">
<td id="Ax1.T15.1.9.1" class="ltx_td ltx_align_left">20Minuten</td>
<td id="Ax1.T15.1.9.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T15.1.9.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T15.1.9.4" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T15.1.9.5" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T15.1.9.6" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T15.1.9.7" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T15.1.9.8" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T15.1.9.9" class="ltx_td ltx_nopad_r ltx_align_left">0.409</td>
</tr>
<tr id="Ax1.T15.1.10" class="ltx_tr">
<td id="Ax1.T15.1.10.1" class="ltx_td ltx_align_left ltx_border_t">average</td>
<td id="Ax1.T15.1.10.2" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T15.1.10.3" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T15.1.10.4" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T15.1.10.5" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T15.1.10.6" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T15.1.10.7" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T15.1.10.8" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T15.1.10.9" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_t">0.334</td>
</tr>
<tr id="Ax1.T15.1.11" class="ltx_tr">
<td id="Ax1.T15.1.11.1" class="ltx_td ltx_align_left ltx_border_bb">BWT</td>
<td id="Ax1.T15.1.11.2" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T15.1.11.3" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T15.1.11.4" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T15.1.11.5" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T15.1.11.6" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T15.1.11.7" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T15.1.11.8" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T15.1.11.9" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_bb">-0.237</td>
</tr>
</table>
</figure>
<figure id="Ax1.T16" class="ltx_table">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table">Table 16: </span>Detailed results of continual learning of Vicuna-13b with LoRA adapters.</figcaption>
<table id="Ax1.T16.1" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="Ax1.T16.1.1" class="ltx_tr">
<td id="Ax1.T16.1.1.1" class="ltx_td ltx_align_left ltx_border_tt">Task\Round</td>
<td id="Ax1.T16.1.1.2" class="ltx_td ltx_align_left ltx_border_tt">1</td>
<td id="Ax1.T16.1.1.3" class="ltx_td ltx_align_left ltx_border_tt">2</td>
<td id="Ax1.T16.1.1.4" class="ltx_td ltx_align_left ltx_border_tt">3</td>
<td id="Ax1.T16.1.1.5" class="ltx_td ltx_align_left ltx_border_tt">4</td>
<td id="Ax1.T16.1.1.6" class="ltx_td ltx_align_left ltx_border_tt">5</td>
<td id="Ax1.T16.1.1.7" class="ltx_td ltx_align_left ltx_border_tt">6</td>
<td id="Ax1.T16.1.1.8" class="ltx_td ltx_align_left ltx_border_tt">7</td>
<td id="Ax1.T16.1.1.9" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_tt">8</td>
</tr>
<tr id="Ax1.T16.1.2" class="ltx_tr">
<td id="Ax1.T16.1.2.1" class="ltx_td ltx_align_left ltx_border_t">C-STANCE</td>
<td id="Ax1.T16.1.2.2" class="ltx_td ltx_align_left ltx_border_t">0.524</td>
<td id="Ax1.T16.1.2.3" class="ltx_td ltx_align_left ltx_border_t">0.504</td>
<td id="Ax1.T16.1.2.4" class="ltx_td ltx_align_left ltx_border_t">0.394</td>
<td id="Ax1.T16.1.2.5" class="ltx_td ltx_align_left ltx_border_t">0.385</td>
<td id="Ax1.T16.1.2.6" class="ltx_td ltx_align_left ltx_border_t">0.389</td>
<td id="Ax1.T16.1.2.7" class="ltx_td ltx_align_left ltx_border_t">0.347</td>
<td id="Ax1.T16.1.2.8" class="ltx_td ltx_align_left ltx_border_t">0.329</td>
<td id="Ax1.T16.1.2.9" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_t">0.07</td>
</tr>
<tr id="Ax1.T16.1.3" class="ltx_tr">
<td id="Ax1.T16.1.3.1" class="ltx_td ltx_align_left">FOMC</td>
<td id="Ax1.T16.1.3.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T16.1.3.3" class="ltx_td ltx_align_left">0.74</td>
<td id="Ax1.T16.1.3.4" class="ltx_td ltx_align_left">0.68</td>
<td id="Ax1.T16.1.3.5" class="ltx_td ltx_align_left">0.616</td>
<td id="Ax1.T16.1.3.6" class="ltx_td ltx_align_left"><span id="Ax1.T16.1.3.6.1" class="ltx_text" style="color:#1F2329;">0.188</span></td>
<td id="Ax1.T16.1.3.7" class="ltx_td ltx_align_left">0.62</td>
<td id="Ax1.T16.1.3.8" class="ltx_td ltx_align_left">0.438</td>
<td id="Ax1.T16.1.3.9" class="ltx_td ltx_nopad_r ltx_align_left">0.04</td>
</tr>
<tr id="Ax1.T16.1.4" class="ltx_tr">
<td id="Ax1.T16.1.4.1" class="ltx_td ltx_align_left">MeetingBank</td>
<td id="Ax1.T16.1.4.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T16.1.4.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T16.1.4.4" class="ltx_td ltx_align_left">0.495</td>
<td id="Ax1.T16.1.4.5" class="ltx_td ltx_align_left">0.24</td>
<td id="Ax1.T16.1.4.6" class="ltx_td ltx_align_left">0.157</td>
<td id="Ax1.T16.1.4.7" class="ltx_td ltx_align_left">0.132</td>
<td id="Ax1.T16.1.4.8" class="ltx_td ltx_align_left">0.08</td>
<td id="Ax1.T16.1.4.9" class="ltx_td ltx_nopad_r ltx_align_left">0.14</td>
</tr>
<tr id="Ax1.T16.1.5" class="ltx_tr">
<td id="Ax1.T16.1.5.1" class="ltx_td ltx_align_left">Py150</td>
<td id="Ax1.T16.1.5.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T16.1.5.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T16.1.5.4" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T16.1.5.5" class="ltx_td ltx_align_left">0.6</td>
<td id="Ax1.T16.1.5.6" class="ltx_td ltx_align_left">0.368</td>
<td id="Ax1.T16.1.5.7" class="ltx_td ltx_align_left">0.52</td>
<td id="Ax1.T16.1.5.8" class="ltx_td ltx_align_left">0.491</td>
<td id="Ax1.T16.1.5.9" class="ltx_td ltx_nopad_r ltx_align_left">0.256</td>
</tr>
<tr id="Ax1.T16.1.6" class="ltx_tr">
<td id="Ax1.T16.1.6.1" class="ltx_td ltx_align_left">ScienceQA</td>
<td id="Ax1.T16.1.6.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T16.1.6.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T16.1.6.4" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T16.1.6.5" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T16.1.6.6" class="ltx_td ltx_align_left">0.77</td>
<td id="Ax1.T16.1.6.7" class="ltx_td ltx_align_left">0.75</td>
<td id="Ax1.T16.1.6.8" class="ltx_td ltx_align_left">0.732</td>
<td id="Ax1.T16.1.6.9" class="ltx_td ltx_nopad_r ltx_align_left">0.74</td>
</tr>
<tr id="Ax1.T16.1.7" class="ltx_tr">
<td id="Ax1.T16.1.7.1" class="ltx_td ltx_align_left">NumGLUE-cm</td>
<td id="Ax1.T16.1.7.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T16.1.7.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T16.1.7.4" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T16.1.7.5" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T16.1.7.6" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T16.1.7.7" class="ltx_td ltx_align_left">0.407</td>
<td id="Ax1.T16.1.7.8" class="ltx_td ltx_align_left">0.346</td>
<td id="Ax1.T16.1.7.9" class="ltx_td ltx_nopad_r ltx_align_left">0.346</td>
</tr>
<tr id="Ax1.T16.1.8" class="ltx_tr">
<td id="Ax1.T16.1.8.1" class="ltx_td ltx_align_left">NumGLUE-ds</td>
<td id="Ax1.T16.1.8.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T16.1.8.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T16.1.8.4" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T16.1.8.5" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T16.1.8.6" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T16.1.8.7" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T16.1.8.8" class="ltx_td ltx_align_left">0.569</td>
<td id="Ax1.T16.1.8.9" class="ltx_td ltx_nopad_r ltx_align_left">0.52</td>
</tr>
<tr id="Ax1.T16.1.9" class="ltx_tr">
<td id="Ax1.T16.1.9.1" class="ltx_td ltx_align_left">20Minuten</td>
<td id="Ax1.T16.1.9.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T16.1.9.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T16.1.9.4" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T16.1.9.5" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T16.1.9.6" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T16.1.9.7" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T16.1.9.8" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T16.1.9.9" class="ltx_td ltx_nopad_r ltx_align_left">0.413</td>
</tr>
<tr id="Ax1.T16.1.10" class="ltx_tr">
<td id="Ax1.T16.1.10.1" class="ltx_td ltx_align_left ltx_border_t">average</td>
<td id="Ax1.T16.1.10.2" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T16.1.10.3" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T16.1.10.4" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T16.1.10.5" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T16.1.10.6" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T16.1.10.7" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T16.1.10.8" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T16.1.10.9" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_t">0.316</td>
</tr>
<tr id="Ax1.T16.1.11" class="ltx_tr">
<td id="Ax1.T16.1.11.1" class="ltx_td ltx_align_left ltx_border_bb">BWT</td>
<td id="Ax1.T16.1.11.2" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T16.1.11.3" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T16.1.11.4" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T16.1.11.5" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T16.1.11.6" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T16.1.11.7" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T16.1.11.8" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T16.1.11.9" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_bb">-0.284</td>
</tr>
</table>
</figure>
</section>
<section id="Ax1.SS4.SSS4" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">.4.4 </span>Replay method</h4>

<div id="Ax1.SS4.SSS4.p1" class="ltx_para">
<p id="Ax1.SS4.SSS4.p1.1" class="ltx_p">Table 17 21 shows the detailed performance of different models of each round during the continual learning with replay data.</p>
</div>
<figure id="Ax1.T17" class="ltx_table">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table">Table 17: </span>Detailed results of continual learning of Baichuan-7b with replay data.</figcaption>
<table id="Ax1.T17.1" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="Ax1.T17.1.1" class="ltx_tr">
<td id="Ax1.T17.1.1.1" class="ltx_td ltx_align_left ltx_border_tt">Task\Round</td>
<td id="Ax1.T17.1.1.2" class="ltx_td ltx_align_left ltx_border_tt">1</td>
<td id="Ax1.T17.1.1.3" class="ltx_td ltx_align_left ltx_border_tt">2</td>
<td id="Ax1.T17.1.1.4" class="ltx_td ltx_align_left ltx_border_tt">3</td>
<td id="Ax1.T17.1.1.5" class="ltx_td ltx_align_left ltx_border_tt">4</td>
<td id="Ax1.T17.1.1.6" class="ltx_td ltx_align_left ltx_border_tt">5</td>
<td id="Ax1.T17.1.1.7" class="ltx_td ltx_align_left ltx_border_tt">6</td>
<td id="Ax1.T17.1.1.8" class="ltx_td ltx_align_left ltx_border_tt">7</td>
<td id="Ax1.T17.1.1.9" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_tt">8</td>
</tr>
<tr id="Ax1.T17.1.2" class="ltx_tr">
<td id="Ax1.T17.1.2.1" class="ltx_td ltx_align_left ltx_border_t">C-STANCE</td>
<td id="Ax1.T17.1.2.2" class="ltx_td ltx_align_left ltx_border_t">0.57</td>
<td id="Ax1.T17.1.2.3" class="ltx_td ltx_align_left ltx_border_t">0.55</td>
<td id="Ax1.T17.1.2.4" class="ltx_td ltx_align_left ltx_border_t">0.56</td>
<td id="Ax1.T17.1.2.5" class="ltx_td ltx_align_left ltx_border_t">0.63</td>
<td id="Ax1.T17.1.2.6" class="ltx_td ltx_align_left ltx_border_t">0.6</td>
<td id="Ax1.T17.1.2.7" class="ltx_td ltx_align_left ltx_border_t">0.64</td>
<td id="Ax1.T17.1.2.8" class="ltx_td ltx_align_left ltx_border_t">0.62</td>
<td id="Ax1.T17.1.2.9" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_t">0.61</td>
</tr>
<tr id="Ax1.T17.1.3" class="ltx_tr">
<td id="Ax1.T17.1.3.1" class="ltx_td ltx_align_left">FOMC</td>
<td id="Ax1.T17.1.3.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T17.1.3.3" class="ltx_td ltx_align_left">0.69</td>
<td id="Ax1.T17.1.3.4" class="ltx_td ltx_align_left">0.64</td>
<td id="Ax1.T17.1.3.5" class="ltx_td ltx_align_left">0.64</td>
<td id="Ax1.T17.1.3.6" class="ltx_td ltx_align_left"><span id="Ax1.T17.1.3.6.1" class="ltx_text" style="color:#1F2329;">0.65</span></td>
<td id="Ax1.T17.1.3.7" class="ltx_td ltx_align_left">0.65</td>
<td id="Ax1.T17.1.3.8" class="ltx_td ltx_align_left">0.66</td>
<td id="Ax1.T17.1.3.9" class="ltx_td ltx_nopad_r ltx_align_left">0.61</td>
</tr>
<tr id="Ax1.T17.1.4" class="ltx_tr">
<td id="Ax1.T17.1.4.1" class="ltx_td ltx_align_left">MeetingBank</td>
<td id="Ax1.T17.1.4.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T17.1.4.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T17.1.4.4" class="ltx_td ltx_align_left">0.445</td>
<td id="Ax1.T17.1.4.5" class="ltx_td ltx_align_left">0.457</td>
<td id="Ax1.T17.1.4.6" class="ltx_td ltx_align_left">0.449</td>
<td id="Ax1.T17.1.4.7" class="ltx_td ltx_align_left">0.466</td>
<td id="Ax1.T17.1.4.8" class="ltx_td ltx_align_left">0.461</td>
<td id="Ax1.T17.1.4.9" class="ltx_td ltx_nopad_r ltx_align_left">0.482</td>
</tr>
<tr id="Ax1.T17.1.5" class="ltx_tr">
<td id="Ax1.T17.1.5.1" class="ltx_td ltx_align_left">Py150</td>
<td id="Ax1.T17.1.5.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T17.1.5.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T17.1.5.4" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T17.1.5.5" class="ltx_td ltx_align_left">0.546</td>
<td id="Ax1.T17.1.5.6" class="ltx_td ltx_align_left">0.577</td>
<td id="Ax1.T17.1.5.7" class="ltx_td ltx_align_left">0.577</td>
<td id="Ax1.T17.1.5.8" class="ltx_td ltx_align_left">0.613</td>
<td id="Ax1.T17.1.5.9" class="ltx_td ltx_nopad_r ltx_align_left">0.583</td>
</tr>
<tr id="Ax1.T17.1.6" class="ltx_tr">
<td id="Ax1.T17.1.6.1" class="ltx_td ltx_align_left">ScienceQA</td>
<td id="Ax1.T17.1.6.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T17.1.6.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T17.1.6.4" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T17.1.6.5" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T17.1.6.6" class="ltx_td ltx_align_left">0.58</td>
<td id="Ax1.T17.1.6.7" class="ltx_td ltx_align_left">0.51</td>
<td id="Ax1.T17.1.6.8" class="ltx_td ltx_align_left">0.54</td>
<td id="Ax1.T17.1.6.9" class="ltx_td ltx_nopad_r ltx_align_left">0.57</td>
</tr>
<tr id="Ax1.T17.1.7" class="ltx_tr">
<td id="Ax1.T17.1.7.1" class="ltx_td ltx_align_left">NumGLUE-cm</td>
<td id="Ax1.T17.1.7.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T17.1.7.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T17.1.7.4" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T17.1.7.5" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T17.1.7.6" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T17.1.7.7" class="ltx_td ltx_align_left">0.321</td>
<td id="Ax1.T17.1.7.8" class="ltx_td ltx_align_left">0.346</td>
<td id="Ax1.T17.1.7.9" class="ltx_td ltx_nopad_r ltx_align_left">0.333</td>
</tr>
<tr id="Ax1.T17.1.8" class="ltx_tr">
<td id="Ax1.T17.1.8.1" class="ltx_td ltx_align_left">NumGLUE-ds</td>
<td id="Ax1.T17.1.8.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T17.1.8.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T17.1.8.4" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T17.1.8.5" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T17.1.8.6" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T17.1.8.7" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T17.1.8.8" class="ltx_td ltx_align_left">0.5</td>
<td id="Ax1.T17.1.8.9" class="ltx_td ltx_nopad_r ltx_align_left">0.55</td>
</tr>
<tr id="Ax1.T17.1.9" class="ltx_tr">
<td id="Ax1.T17.1.9.1" class="ltx_td ltx_align_left">20Minuten</td>
<td id="Ax1.T17.1.9.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T17.1.9.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T17.1.9.4" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T17.1.9.5" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T17.1.9.6" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T17.1.9.7" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T17.1.9.8" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T17.1.9.9" class="ltx_td ltx_nopad_r ltx_align_left">0.405</td>
</tr>
<tr id="Ax1.T17.1.10" class="ltx_tr">
<td id="Ax1.T17.1.10.1" class="ltx_td ltx_align_left ltx_border_t">average</td>
<td id="Ax1.T17.1.10.2" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T17.1.10.3" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T17.1.10.4" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T17.1.10.5" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T17.1.10.6" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T17.1.10.7" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T17.1.10.8" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T17.1.10.9" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_t">0.517</td>
</tr>
<tr id="Ax1.T17.1.11" class="ltx_tr">
<td id="Ax1.T17.1.11.1" class="ltx_td ltx_align_left ltx_border_bb">BWT</td>
<td id="Ax1.T17.1.11.2" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T17.1.11.3" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T17.1.11.4" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T17.1.11.5" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T17.1.11.6" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T17.1.11.7" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T17.1.11.8" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T17.1.11.9" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_bb">0.011</td>
</tr>
</table>
</figure>
<figure id="Ax1.T18" class="ltx_table">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table">Table 18: </span>Detailed results of continual learning of LLaMA-7b-chat with replay data.</figcaption>
<table id="Ax1.T18.1" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="Ax1.T18.1.1" class="ltx_tr">
<td id="Ax1.T18.1.1.1" class="ltx_td ltx_align_left ltx_border_tt">Task\Round</td>
<td id="Ax1.T18.1.1.2" class="ltx_td ltx_align_left ltx_border_tt">1</td>
<td id="Ax1.T18.1.1.3" class="ltx_td ltx_align_left ltx_border_tt">2</td>
<td id="Ax1.T18.1.1.4" class="ltx_td ltx_align_left ltx_border_tt">3</td>
<td id="Ax1.T18.1.1.5" class="ltx_td ltx_align_left ltx_border_tt">4</td>
<td id="Ax1.T18.1.1.6" class="ltx_td ltx_align_left ltx_border_tt">5</td>
<td id="Ax1.T18.1.1.7" class="ltx_td ltx_align_left ltx_border_tt">6</td>
<td id="Ax1.T18.1.1.8" class="ltx_td ltx_align_left ltx_border_tt">7</td>
<td id="Ax1.T18.1.1.9" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_tt">8</td>
</tr>
<tr id="Ax1.T18.1.2" class="ltx_tr">
<td id="Ax1.T18.1.2.1" class="ltx_td ltx_align_left ltx_border_t">C-STANCE</td>
<td id="Ax1.T18.1.2.2" class="ltx_td ltx_align_left ltx_border_t">0.471</td>
<td id="Ax1.T18.1.2.3" class="ltx_td ltx_align_left ltx_border_t">0.487</td>
<td id="Ax1.T18.1.2.4" class="ltx_td ltx_align_left ltx_border_t">0.485</td>
<td id="Ax1.T18.1.2.5" class="ltx_td ltx_align_left ltx_border_t">0.5</td>
<td id="Ax1.T18.1.2.6" class="ltx_td ltx_align_left ltx_border_t">0.486</td>
<td id="Ax1.T18.1.2.7" class="ltx_td ltx_align_left ltx_border_t">0.475</td>
<td id="Ax1.T18.1.2.8" class="ltx_td ltx_align_left ltx_border_t">0.493</td>
<td id="Ax1.T18.1.2.9" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_t">0.5</td>
</tr>
<tr id="Ax1.T18.1.3" class="ltx_tr">
<td id="Ax1.T18.1.3.1" class="ltx_td ltx_align_left">FOMC</td>
<td id="Ax1.T18.1.3.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T18.1.3.3" class="ltx_td ltx_align_left">0.734</td>
<td id="Ax1.T18.1.3.4" class="ltx_td ltx_align_left">0.769</td>
<td id="Ax1.T18.1.3.5" class="ltx_td ltx_align_left">0.785</td>
<td id="Ax1.T18.1.3.6" class="ltx_td ltx_align_left"><span id="Ax1.T18.1.3.6.1" class="ltx_text" style="color:#1F2329;">0.807</span></td>
<td id="Ax1.T18.1.3.7" class="ltx_td ltx_align_left">0.781</td>
<td id="Ax1.T18.1.3.8" class="ltx_td ltx_align_left">0.785</td>
<td id="Ax1.T18.1.3.9" class="ltx_td ltx_nopad_r ltx_align_left">0.8</td>
</tr>
<tr id="Ax1.T18.1.4" class="ltx_tr">
<td id="Ax1.T18.1.4.1" class="ltx_td ltx_align_left">MeetingBank</td>
<td id="Ax1.T18.1.4.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T18.1.4.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T18.1.4.4" class="ltx_td ltx_align_left">0.499</td>
<td id="Ax1.T18.1.4.5" class="ltx_td ltx_align_left">0.496</td>
<td id="Ax1.T18.1.4.6" class="ltx_td ltx_align_left">0.507</td>
<td id="Ax1.T18.1.4.7" class="ltx_td ltx_align_left">0.494</td>
<td id="Ax1.T18.1.4.8" class="ltx_td ltx_align_left">0.492</td>
<td id="Ax1.T18.1.4.9" class="ltx_td ltx_nopad_r ltx_align_left">0.51</td>
</tr>
<tr id="Ax1.T18.1.5" class="ltx_tr">
<td id="Ax1.T18.1.5.1" class="ltx_td ltx_align_left">Py150</td>
<td id="Ax1.T18.1.5.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T18.1.5.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T18.1.5.4" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T18.1.5.5" class="ltx_td ltx_align_left">0.543</td>
<td id="Ax1.T18.1.5.6" class="ltx_td ltx_align_left">0.561</td>
<td id="Ax1.T18.1.5.7" class="ltx_td ltx_align_left">0.546</td>
<td id="Ax1.T18.1.5.8" class="ltx_td ltx_align_left">0.552</td>
<td id="Ax1.T18.1.5.9" class="ltx_td ltx_nopad_r ltx_align_left">0.55</td>
</tr>
<tr id="Ax1.T18.1.6" class="ltx_tr">
<td id="Ax1.T18.1.6.1" class="ltx_td ltx_align_left">ScienceQA</td>
<td id="Ax1.T18.1.6.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T18.1.6.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T18.1.6.4" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T18.1.6.5" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T18.1.6.6" class="ltx_td ltx_align_left">0.763</td>
<td id="Ax1.T18.1.6.7" class="ltx_td ltx_align_left">0.78</td>
<td id="Ax1.T18.1.6.8" class="ltx_td ltx_align_left">0.78</td>
<td id="Ax1.T18.1.6.9" class="ltx_td ltx_nopad_r ltx_align_left">0.785</td>
</tr>
<tr id="Ax1.T18.1.7" class="ltx_tr">
<td id="Ax1.T18.1.7.1" class="ltx_td ltx_align_left">NumGLUE-cm</td>
<td id="Ax1.T18.1.7.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T18.1.7.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T18.1.7.4" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T18.1.7.5" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T18.1.7.6" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T18.1.7.7" class="ltx_td ltx_align_left">0.358</td>
<td id="Ax1.T18.1.7.8" class="ltx_td ltx_align_left">0.309</td>
<td id="Ax1.T18.1.7.9" class="ltx_td ltx_nopad_r ltx_align_left">0.37</td>
</tr>
<tr id="Ax1.T18.1.8" class="ltx_tr">
<td id="Ax1.T18.1.8.1" class="ltx_td ltx_align_left">NumGLUE-ds</td>
<td id="Ax1.T18.1.8.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T18.1.8.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T18.1.8.4" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T18.1.8.5" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T18.1.8.6" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T18.1.8.7" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T18.1.8.8" class="ltx_td ltx_align_left">0.486</td>
<td id="Ax1.T18.1.8.9" class="ltx_td ltx_nopad_r ltx_align_left">0.52</td>
</tr>
<tr id="Ax1.T18.1.9" class="ltx_tr">
<td id="Ax1.T18.1.9.1" class="ltx_td ltx_align_left">20Minuten</td>
<td id="Ax1.T18.1.9.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T18.1.9.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T18.1.9.4" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T18.1.9.5" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T18.1.9.6" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T18.1.9.7" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T18.1.9.8" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T18.1.9.9" class="ltx_td ltx_nopad_r ltx_align_left">0.406</td>
</tr>
<tr id="Ax1.T18.1.10" class="ltx_tr">
<td id="Ax1.T18.1.10.1" class="ltx_td ltx_align_left ltx_border_t">average</td>
<td id="Ax1.T18.1.10.2" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T18.1.10.3" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T18.1.10.4" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T18.1.10.5" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T18.1.10.6" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T18.1.10.7" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T18.1.10.8" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T18.1.10.9" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_t">0.555</td>
</tr>
<tr id="Ax1.T18.1.11" class="ltx_tr">
<td id="Ax1.T18.1.11.1" class="ltx_td ltx_align_left ltx_border_bb">BWT</td>
<td id="Ax1.T18.1.11.2" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T18.1.11.3" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T18.1.11.4" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T18.1.11.5" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T18.1.11.6" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T18.1.11.7" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T18.1.11.8" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T18.1.11.9" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_bb">0.026</td>
</tr>
</table>
</figure>
<figure id="Ax1.T19" class="ltx_table">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table">Table 19: </span>Detailed results of continual learning of LLaMA-13b-chat with replay data.</figcaption>
<table id="Ax1.T19.1" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="Ax1.T19.1.1" class="ltx_tr">
<td id="Ax1.T19.1.1.1" class="ltx_td ltx_align_left ltx_border_tt">Task\Round</td>
<td id="Ax1.T19.1.1.2" class="ltx_td ltx_align_left ltx_border_tt">1</td>
<td id="Ax1.T19.1.1.3" class="ltx_td ltx_align_left ltx_border_tt">2</td>
<td id="Ax1.T19.1.1.4" class="ltx_td ltx_align_left ltx_border_tt">3</td>
<td id="Ax1.T19.1.1.5" class="ltx_td ltx_align_left ltx_border_tt">4</td>
<td id="Ax1.T19.1.1.6" class="ltx_td ltx_align_left ltx_border_tt">5</td>
<td id="Ax1.T19.1.1.7" class="ltx_td ltx_align_left ltx_border_tt">6</td>
<td id="Ax1.T19.1.1.8" class="ltx_td ltx_align_left ltx_border_tt">7</td>
<td id="Ax1.T19.1.1.9" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_tt">8</td>
</tr>
<tr id="Ax1.T19.1.2" class="ltx_tr">
<td id="Ax1.T19.1.2.1" class="ltx_td ltx_align_left ltx_border_t">C-STANCE</td>
<td id="Ax1.T19.1.2.2" class="ltx_td ltx_align_left ltx_border_t">0.5</td>
<td id="Ax1.T19.1.2.3" class="ltx_td ltx_align_left ltx_border_t">0.496</td>
<td id="Ax1.T19.1.2.4" class="ltx_td ltx_align_left ltx_border_t">0.497</td>
<td id="Ax1.T19.1.2.5" class="ltx_td ltx_align_left ltx_border_t">0.493</td>
<td id="Ax1.T19.1.2.6" class="ltx_td ltx_align_left ltx_border_t">0.52</td>
<td id="Ax1.T19.1.2.7" class="ltx_td ltx_align_left ltx_border_t">0.503</td>
<td id="Ax1.T19.1.2.8" class="ltx_td ltx_align_left ltx_border_t">0.5</td>
<td id="Ax1.T19.1.2.9" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_t">0.51</td>
</tr>
<tr id="Ax1.T19.1.3" class="ltx_tr">
<td id="Ax1.T19.1.3.1" class="ltx_td ltx_align_left">FOMC</td>
<td id="Ax1.T19.1.3.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T19.1.3.3" class="ltx_td ltx_align_left">0.778</td>
<td id="Ax1.T19.1.3.4" class="ltx_td ltx_align_left">0.803</td>
<td id="Ax1.T19.1.3.5" class="ltx_td ltx_align_left">0.805</td>
<td id="Ax1.T19.1.3.6" class="ltx_td ltx_align_left"><span id="Ax1.T19.1.3.6.1" class="ltx_text" style="color:#1F2329;">0.792</span></td>
<td id="Ax1.T19.1.3.7" class="ltx_td ltx_align_left">0.789</td>
<td id="Ax1.T19.1.3.8" class="ltx_td ltx_align_left">0.785</td>
<td id="Ax1.T19.1.3.9" class="ltx_td ltx_nopad_r ltx_align_left">0.813</td>
</tr>
<tr id="Ax1.T19.1.4" class="ltx_tr">
<td id="Ax1.T19.1.4.1" class="ltx_td ltx_align_left">MeetingBank</td>
<td id="Ax1.T19.1.4.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T19.1.4.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T19.1.4.4" class="ltx_td ltx_align_left">0.484</td>
<td id="Ax1.T19.1.4.5" class="ltx_td ltx_align_left">0.495</td>
<td id="Ax1.T19.1.4.6" class="ltx_td ltx_align_left">0.515</td>
<td id="Ax1.T19.1.4.7" class="ltx_td ltx_align_left">0.499</td>
<td id="Ax1.T19.1.4.8" class="ltx_td ltx_align_left">0.503</td>
<td id="Ax1.T19.1.4.9" class="ltx_td ltx_nopad_r ltx_align_left">0.482</td>
</tr>
<tr id="Ax1.T19.1.5" class="ltx_tr">
<td id="Ax1.T19.1.5.1" class="ltx_td ltx_align_left">Py150</td>
<td id="Ax1.T19.1.5.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T19.1.5.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T19.1.5.4" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T19.1.5.5" class="ltx_td ltx_align_left">0.523</td>
<td id="Ax1.T19.1.5.6" class="ltx_td ltx_align_left">0.549</td>
<td id="Ax1.T19.1.5.7" class="ltx_td ltx_align_left">0.532</td>
<td id="Ax1.T19.1.5.8" class="ltx_td ltx_align_left">0.534</td>
<td id="Ax1.T19.1.5.9" class="ltx_td ltx_nopad_r ltx_align_left">0.523</td>
</tr>
<tr id="Ax1.T19.1.6" class="ltx_tr">
<td id="Ax1.T19.1.6.1" class="ltx_td ltx_align_left">ScienceQA</td>
<td id="Ax1.T19.1.6.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T19.1.6.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T19.1.6.4" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T19.1.6.5" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T19.1.6.6" class="ltx_td ltx_align_left">0.816</td>
<td id="Ax1.T19.1.6.7" class="ltx_td ltx_align_left">0.8</td>
<td id="Ax1.T19.1.6.8" class="ltx_td ltx_align_left">0.804</td>
<td id="Ax1.T19.1.6.9" class="ltx_td ltx_nopad_r ltx_align_left">0.792</td>
</tr>
<tr id="Ax1.T19.1.7" class="ltx_tr">
<td id="Ax1.T19.1.7.1" class="ltx_td ltx_align_left">NumGLUE-cm</td>
<td id="Ax1.T19.1.7.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T19.1.7.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T19.1.7.4" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T19.1.7.5" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T19.1.7.6" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T19.1.7.7" class="ltx_td ltx_align_left">0.358</td>
<td id="Ax1.T19.1.7.8" class="ltx_td ltx_align_left">0.407</td>
<td id="Ax1.T19.1.7.9" class="ltx_td ltx_nopad_r ltx_align_left">0.396</td>
</tr>
<tr id="Ax1.T19.1.8" class="ltx_tr">
<td id="Ax1.T19.1.8.1" class="ltx_td ltx_align_left">NumGLUE-ds</td>
<td id="Ax1.T19.1.8.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T19.1.8.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T19.1.8.4" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T19.1.8.5" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T19.1.8.6" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T19.1.8.7" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T19.1.8.8" class="ltx_td ltx_align_left">0.628</td>
<td id="Ax1.T19.1.8.9" class="ltx_td ltx_nopad_r ltx_align_left">0.606</td>
</tr>
<tr id="Ax1.T19.1.9" class="ltx_tr">
<td id="Ax1.T19.1.9.1" class="ltx_td ltx_align_left">20Minuten</td>
<td id="Ax1.T19.1.9.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T19.1.9.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T19.1.9.4" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T19.1.9.5" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T19.1.9.6" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T19.1.9.7" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T19.1.9.8" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T19.1.9.9" class="ltx_td ltx_nopad_r ltx_align_left">0.407</td>
</tr>
<tr id="Ax1.T19.1.10" class="ltx_tr">
<td id="Ax1.T19.1.10.1" class="ltx_td ltx_align_left ltx_border_t">average</td>
<td id="Ax1.T19.1.10.2" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T19.1.10.3" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T19.1.10.4" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T19.1.10.5" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T19.1.10.6" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T19.1.10.7" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T19.1.10.8" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T19.1.10.9" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_t">0.566</td>
</tr>
<tr id="Ax1.T19.1.11" class="ltx_tr">
<td id="Ax1.T19.1.11.1" class="ltx_td ltx_align_left ltx_border_bb">BWT</td>
<td id="Ax1.T19.1.11.2" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T19.1.11.3" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T19.1.11.4" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T19.1.11.5" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T19.1.11.6" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T19.1.11.7" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T19.1.11.8" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T19.1.11.9" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_bb">0.004</td>
</tr>
</table>
</figure>
<figure id="Ax1.T20" class="ltx_table">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table">Table 20: </span>Detailed results of continual learning of Vicuna-7b with replay data.</figcaption>
<table id="Ax1.T20.1" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="Ax1.T20.1.1" class="ltx_tr">
<td id="Ax1.T20.1.1.1" class="ltx_td ltx_align_left ltx_border_tt">Task\Round</td>
<td id="Ax1.T20.1.1.2" class="ltx_td ltx_align_left ltx_border_tt">1</td>
<td id="Ax1.T20.1.1.3" class="ltx_td ltx_align_left ltx_border_tt">2</td>
<td id="Ax1.T20.1.1.4" class="ltx_td ltx_align_left ltx_border_tt">3</td>
<td id="Ax1.T20.1.1.5" class="ltx_td ltx_align_left ltx_border_tt">4</td>
<td id="Ax1.T20.1.1.6" class="ltx_td ltx_align_left ltx_border_tt">5</td>
<td id="Ax1.T20.1.1.7" class="ltx_td ltx_align_left ltx_border_tt">6</td>
<td id="Ax1.T20.1.1.8" class="ltx_td ltx_align_left ltx_border_tt">7</td>
<td id="Ax1.T20.1.1.9" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_tt">8</td>
</tr>
<tr id="Ax1.T20.1.2" class="ltx_tr">
<td id="Ax1.T20.1.2.1" class="ltx_td ltx_align_left ltx_border_t">C-STANCE</td>
<td id="Ax1.T20.1.2.2" class="ltx_td ltx_align_left ltx_border_t">0.5</td>
<td id="Ax1.T20.1.2.3" class="ltx_td ltx_align_left ltx_border_t">0.528</td>
<td id="Ax1.T20.1.2.4" class="ltx_td ltx_align_left ltx_border_t">0.512</td>
<td id="Ax1.T20.1.2.5" class="ltx_td ltx_align_left ltx_border_t">0.519</td>
<td id="Ax1.T20.1.2.6" class="ltx_td ltx_align_left ltx_border_t">0.518</td>
<td id="Ax1.T20.1.2.7" class="ltx_td ltx_align_left ltx_border_t">0.519</td>
<td id="Ax1.T20.1.2.8" class="ltx_td ltx_align_left ltx_border_t">0.515</td>
<td id="Ax1.T20.1.2.9" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_t">0.524</td>
</tr>
<tr id="Ax1.T20.1.3" class="ltx_tr">
<td id="Ax1.T20.1.3.1" class="ltx_td ltx_align_left">FOMC</td>
<td id="Ax1.T20.1.3.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T20.1.3.3" class="ltx_td ltx_align_left">0.747</td>
<td id="Ax1.T20.1.3.4" class="ltx_td ltx_align_left">0.803</td>
<td id="Ax1.T20.1.3.5" class="ltx_td ltx_align_left">0.794</td>
<td id="Ax1.T20.1.3.6" class="ltx_td ltx_align_left"><span id="Ax1.T20.1.3.6.1" class="ltx_text" style="color:#1F2329;">0.805</span></td>
<td id="Ax1.T20.1.3.7" class="ltx_td ltx_align_left">0.795</td>
<td id="Ax1.T20.1.3.8" class="ltx_td ltx_align_left">0.801</td>
<td id="Ax1.T20.1.3.9" class="ltx_td ltx_nopad_r ltx_align_left">0.806</td>
</tr>
<tr id="Ax1.T20.1.4" class="ltx_tr">
<td id="Ax1.T20.1.4.1" class="ltx_td ltx_align_left">MeetingBank</td>
<td id="Ax1.T20.1.4.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T20.1.4.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T20.1.4.4" class="ltx_td ltx_align_left">0.512</td>
<td id="Ax1.T20.1.4.5" class="ltx_td ltx_align_left">0.483</td>
<td id="Ax1.T20.1.4.6" class="ltx_td ltx_align_left">0.516</td>
<td id="Ax1.T20.1.4.7" class="ltx_td ltx_align_left">0.516</td>
<td id="Ax1.T20.1.4.8" class="ltx_td ltx_align_left">0.492</td>
<td id="Ax1.T20.1.4.9" class="ltx_td ltx_nopad_r ltx_align_left">0.496</td>
</tr>
<tr id="Ax1.T20.1.5" class="ltx_tr">
<td id="Ax1.T20.1.5.1" class="ltx_td ltx_align_left">Py150</td>
<td id="Ax1.T20.1.5.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T20.1.5.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T20.1.5.4" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T20.1.5.5" class="ltx_td ltx_align_left">0.525</td>
<td id="Ax1.T20.1.5.6" class="ltx_td ltx_align_left">0.569</td>
<td id="Ax1.T20.1.5.7" class="ltx_td ltx_align_left">0.553</td>
<td id="Ax1.T20.1.5.8" class="ltx_td ltx_align_left">0.551</td>
<td id="Ax1.T20.1.5.9" class="ltx_td ltx_nopad_r ltx_align_left">0.551</td>
</tr>
<tr id="Ax1.T20.1.6" class="ltx_tr">
<td id="Ax1.T20.1.6.1" class="ltx_td ltx_align_left">ScienceQA</td>
<td id="Ax1.T20.1.6.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T20.1.6.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T20.1.6.4" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T20.1.6.5" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T20.1.6.6" class="ltx_td ltx_align_left">0.77</td>
<td id="Ax1.T20.1.6.7" class="ltx_td ltx_align_left">0.776</td>
<td id="Ax1.T20.1.6.8" class="ltx_td ltx_align_left">0.772</td>
<td id="Ax1.T20.1.6.9" class="ltx_td ltx_nopad_r ltx_align_left">0.767</td>
</tr>
<tr id="Ax1.T20.1.7" class="ltx_tr">
<td id="Ax1.T20.1.7.1" class="ltx_td ltx_align_left">NumGLUE-cm</td>
<td id="Ax1.T20.1.7.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T20.1.7.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T20.1.7.4" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T20.1.7.5" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T20.1.7.6" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T20.1.7.7" class="ltx_td ltx_align_left">0.396</td>
<td id="Ax1.T20.1.7.8" class="ltx_td ltx_align_left">0.322</td>
<td id="Ax1.T20.1.7.9" class="ltx_td ltx_nopad_r ltx_align_left">0.309</td>
</tr>
<tr id="Ax1.T20.1.8" class="ltx_tr">
<td id="Ax1.T20.1.8.1" class="ltx_td ltx_align_left">NumGLUE-ds</td>
<td id="Ax1.T20.1.8.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T20.1.8.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T20.1.8.4" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T20.1.8.5" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T20.1.8.6" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T20.1.8.7" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T20.1.8.8" class="ltx_td ltx_align_left">0.554</td>
<td id="Ax1.T20.1.8.9" class="ltx_td ltx_nopad_r ltx_align_left">0.563</td>
</tr>
<tr id="Ax1.T20.1.9" class="ltx_tr">
<td id="Ax1.T20.1.9.1" class="ltx_td ltx_align_left">20Minuten</td>
<td id="Ax1.T20.1.9.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T20.1.9.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T20.1.9.4" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T20.1.9.5" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T20.1.9.6" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T20.1.9.7" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T20.1.9.8" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T20.1.9.9" class="ltx_td ltx_nopad_r ltx_align_left">0.405</td>
</tr>
<tr id="Ax1.T20.1.10" class="ltx_tr">
<td id="Ax1.T20.1.10.1" class="ltx_td ltx_align_left ltx_border_t">average</td>
<td id="Ax1.T20.1.10.2" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T20.1.10.3" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T20.1.10.4" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T20.1.10.5" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T20.1.10.6" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T20.1.10.7" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T20.1.10.8" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T20.1.10.9" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_t">0.553</td>
</tr>
<tr id="Ax1.T20.1.11" class="ltx_tr">
<td id="Ax1.T20.1.11.1" class="ltx_td ltx_align_left ltx_border_bb">BWT</td>
<td id="Ax1.T20.1.11.2" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T20.1.11.3" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T20.1.11.4" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T20.1.11.5" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T20.1.11.6" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T20.1.11.7" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T20.1.11.8" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T20.1.11.9" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_bb">0.002</td>
</tr>
</table>
</figure>
<figure id="Ax1.T21" class="ltx_table">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table">Table 21: </span>Detailed results of continual learning of Vicuna-13b with replay data.</figcaption>
<table id="Ax1.T21.1" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="Ax1.T21.1.1" class="ltx_tr">
<td id="Ax1.T21.1.1.1" class="ltx_td ltx_align_left ltx_border_tt">Task\Round</td>
<td id="Ax1.T21.1.1.2" class="ltx_td ltx_align_left ltx_border_tt">1</td>
<td id="Ax1.T21.1.1.3" class="ltx_td ltx_align_left ltx_border_tt">2</td>
<td id="Ax1.T21.1.1.4" class="ltx_td ltx_align_left ltx_border_tt">3</td>
<td id="Ax1.T21.1.1.5" class="ltx_td ltx_align_left ltx_border_tt">4</td>
<td id="Ax1.T21.1.1.6" class="ltx_td ltx_align_left ltx_border_tt">5</td>
<td id="Ax1.T21.1.1.7" class="ltx_td ltx_align_left ltx_border_tt">6</td>
<td id="Ax1.T21.1.1.8" class="ltx_td ltx_align_left ltx_border_tt">7</td>
<td id="Ax1.T21.1.1.9" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_tt">8</td>
</tr>
<tr id="Ax1.T21.1.2" class="ltx_tr">
<td id="Ax1.T21.1.2.1" class="ltx_td ltx_align_left ltx_border_t">C-STANCE</td>
<td id="Ax1.T21.1.2.2" class="ltx_td ltx_align_left ltx_border_t">0.56</td>
<td id="Ax1.T21.1.2.3" class="ltx_td ltx_align_left ltx_border_t">0.58</td>
<td id="Ax1.T21.1.2.4" class="ltx_td ltx_align_left ltx_border_t">0.616</td>
<td id="Ax1.T21.1.2.5" class="ltx_td ltx_align_left ltx_border_t">0.62</td>
<td id="Ax1.T21.1.2.6" class="ltx_td ltx_align_left ltx_border_t">0.616</td>
<td id="Ax1.T21.1.2.7" class="ltx_td ltx_align_left ltx_border_t">0.637</td>
<td id="Ax1.T21.1.2.8" class="ltx_td ltx_align_left ltx_border_t">0.629</td>
<td id="Ax1.T21.1.2.9" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_t">0.629</td>
</tr>
<tr id="Ax1.T21.1.3" class="ltx_tr">
<td id="Ax1.T21.1.3.1" class="ltx_td ltx_align_left">FOMC</td>
<td id="Ax1.T21.1.3.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T21.1.3.3" class="ltx_td ltx_align_left">0.736</td>
<td id="Ax1.T21.1.3.4" class="ltx_td ltx_align_left">0.76</td>
<td id="Ax1.T21.1.3.5" class="ltx_td ltx_align_left">0.76</td>
<td id="Ax1.T21.1.3.6" class="ltx_td ltx_align_left"><span id="Ax1.T21.1.3.6.1" class="ltx_text" style="color:#1F2329;">0.788</span></td>
<td id="Ax1.T21.1.3.7" class="ltx_td ltx_align_left">0.771</td>
<td id="Ax1.T21.1.3.8" class="ltx_td ltx_align_left">0.76</td>
<td id="Ax1.T21.1.3.9" class="ltx_td ltx_nopad_r ltx_align_left">0.76</td>
</tr>
<tr id="Ax1.T21.1.4" class="ltx_tr">
<td id="Ax1.T21.1.4.1" class="ltx_td ltx_align_left">MeetingBank</td>
<td id="Ax1.T21.1.4.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T21.1.4.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T21.1.4.4" class="ltx_td ltx_align_left">0.464</td>
<td id="Ax1.T21.1.4.5" class="ltx_td ltx_align_left">0.505</td>
<td id="Ax1.T21.1.4.6" class="ltx_td ltx_align_left">0.468</td>
<td id="Ax1.T21.1.4.7" class="ltx_td ltx_align_left">0.441</td>
<td id="Ax1.T21.1.4.8" class="ltx_td ltx_align_left">0.473</td>
<td id="Ax1.T21.1.4.9" class="ltx_td ltx_nopad_r ltx_align_left">0.451</td>
</tr>
<tr id="Ax1.T21.1.5" class="ltx_tr">
<td id="Ax1.T21.1.5.1" class="ltx_td ltx_align_left">Py150</td>
<td id="Ax1.T21.1.5.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T21.1.5.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T21.1.5.4" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T21.1.5.5" class="ltx_td ltx_align_left">0.544</td>
<td id="Ax1.T21.1.5.6" class="ltx_td ltx_align_left">0.559</td>
<td id="Ax1.T21.1.5.7" class="ltx_td ltx_align_left">0.563</td>
<td id="Ax1.T21.1.5.8" class="ltx_td ltx_align_left">0.591</td>
<td id="Ax1.T21.1.5.9" class="ltx_td ltx_nopad_r ltx_align_left">0.554</td>
</tr>
<tr id="Ax1.T21.1.6" class="ltx_tr">
<td id="Ax1.T21.1.6.1" class="ltx_td ltx_align_left">ScienceQA</td>
<td id="Ax1.T21.1.6.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T21.1.6.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T21.1.6.4" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T21.1.6.5" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T21.1.6.6" class="ltx_td ltx_align_left">0.71</td>
<td id="Ax1.T21.1.6.7" class="ltx_td ltx_align_left">0.699</td>
<td id="Ax1.T21.1.6.8" class="ltx_td ltx_align_left">0.674</td>
<td id="Ax1.T21.1.6.9" class="ltx_td ltx_nopad_r ltx_align_left">0.71</td>
</tr>
<tr id="Ax1.T21.1.7" class="ltx_tr">
<td id="Ax1.T21.1.7.1" class="ltx_td ltx_align_left">NumGLUE-cm</td>
<td id="Ax1.T21.1.7.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T21.1.7.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T21.1.7.4" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T21.1.7.5" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T21.1.7.6" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T21.1.7.7" class="ltx_td ltx_align_left">0.42</td>
<td id="Ax1.T21.1.7.8" class="ltx_td ltx_align_left">0.358</td>
<td id="Ax1.T21.1.7.9" class="ltx_td ltx_nopad_r ltx_align_left">0.358</td>
</tr>
<tr id="Ax1.T21.1.8" class="ltx_tr">
<td id="Ax1.T21.1.8.1" class="ltx_td ltx_align_left">NumGLUE-ds</td>
<td id="Ax1.T21.1.8.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T21.1.8.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T21.1.8.4" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T21.1.8.5" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T21.1.8.6" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T21.1.8.7" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T21.1.8.8" class="ltx_td ltx_align_left">0.667</td>
<td id="Ax1.T21.1.8.9" class="ltx_td ltx_nopad_r ltx_align_left">0.68</td>
</tr>
<tr id="Ax1.T21.1.9" class="ltx_tr">
<td id="Ax1.T21.1.9.1" class="ltx_td ltx_align_left">20Minuten</td>
<td id="Ax1.T21.1.9.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T21.1.9.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T21.1.9.4" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T21.1.9.5" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T21.1.9.6" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T21.1.9.7" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T21.1.9.8" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T21.1.9.9" class="ltx_td ltx_nopad_r ltx_align_left">0.41</td>
</tr>
<tr id="Ax1.T21.1.10" class="ltx_tr">
<td id="Ax1.T21.1.10.1" class="ltx_td ltx_align_left ltx_border_t">average</td>
<td id="Ax1.T21.1.10.2" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T21.1.10.3" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T21.1.10.4" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T21.1.10.5" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T21.1.10.6" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T21.1.10.7" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T21.1.10.8" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T21.1.10.9" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_t">0.569</td>
</tr>
<tr id="Ax1.T21.1.11" class="ltx_tr">
<td id="Ax1.T21.1.11.1" class="ltx_td ltx_align_left ltx_border_bb">BWT</td>
<td id="Ax1.T21.1.11.2" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T21.1.11.3" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T21.1.11.4" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T21.1.11.5" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T21.1.11.6" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T21.1.11.7" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T21.1.11.8" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T21.1.11.9" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_bb">0.006</td>
</tr>
</table>
</figure>
</section>
<section id="Ax1.SS4.SSS5" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">.4.5 </span>RCL method</h4>

<div id="Ax1.SS4.SSS5.p1" class="ltx_para">
<p id="Ax1.SS4.SSS5.p1.1" class="ltx_p">Table 22 shows the detailed performance of LLaMA2-7b-chat of each round during the continual learning. RCL represents reasoning-based continual learning.</p>
</div>
<figure id="Ax1.T22" class="ltx_table">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table">Table 22: </span>Detailed results of RCL learning of LLaMA2-7b-chat.</figcaption>
<table id="Ax1.T22.1" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="Ax1.T22.1.1" class="ltx_tr">
<td id="Ax1.T22.1.1.1" class="ltx_td ltx_align_left ltx_border_tt">Task\Round</td>
<td id="Ax1.T22.1.1.2" class="ltx_td ltx_align_left ltx_border_tt">1</td>
<td id="Ax1.T22.1.1.3" class="ltx_td ltx_align_left ltx_border_tt">2</td>
<td id="Ax1.T22.1.1.4" class="ltx_td ltx_align_left ltx_border_tt">3</td>
<td id="Ax1.T22.1.1.5" class="ltx_td ltx_align_left ltx_border_tt">4</td>
<td id="Ax1.T22.1.1.6" class="ltx_td ltx_align_left ltx_border_tt">5</td>
<td id="Ax1.T22.1.1.7" class="ltx_td ltx_align_left ltx_border_tt">6</td>
<td id="Ax1.T22.1.1.8" class="ltx_td ltx_align_left ltx_border_tt">7</td>
<td id="Ax1.T22.1.1.9" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_tt">8</td>
</tr>
<tr id="Ax1.T22.1.2" class="ltx_tr">
<td id="Ax1.T22.1.2.1" class="ltx_td ltx_align_left ltx_border_t">C-STANCE</td>
<td id="Ax1.T22.1.2.2" class="ltx_td ltx_align_left ltx_border_t">0.614</td>
<td id="Ax1.T22.1.2.3" class="ltx_td ltx_align_left ltx_border_t">0.428</td>
<td id="Ax1.T22.1.2.4" class="ltx_td ltx_align_left ltx_border_t">0.464</td>
<td id="Ax1.T22.1.2.5" class="ltx_td ltx_align_left ltx_border_t">0.486</td>
<td id="Ax1.T22.1.2.6" class="ltx_td ltx_align_left ltx_border_t">0.5</td>
<td id="Ax1.T22.1.2.7" class="ltx_td ltx_align_left ltx_border_t">0.472</td>
<td id="Ax1.T22.1.2.8" class="ltx_td ltx_align_left ltx_border_t">0.452</td>
<td id="Ax1.T22.1.2.9" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_t">0.522</td>
</tr>
<tr id="Ax1.T22.1.3" class="ltx_tr">
<td id="Ax1.T22.1.3.1" class="ltx_td ltx_align_left">FOMC</td>
<td id="Ax1.T22.1.3.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T22.1.3.3" class="ltx_td ltx_align_left">0.621</td>
<td id="Ax1.T22.1.3.4" class="ltx_td ltx_align_left">0.476</td>
<td id="Ax1.T22.1.3.5" class="ltx_td ltx_align_left">0.002</td>
<td id="Ax1.T22.1.3.6" class="ltx_td ltx_align_left"><span id="Ax1.T22.1.3.6.1" class="ltx_text" style="color:#1F2329;">0.563</span></td>
<td id="Ax1.T22.1.3.7" class="ltx_td ltx_align_left">0.542</td>
<td id="Ax1.T22.1.3.8" class="ltx_td ltx_align_left">0.534</td>
<td id="Ax1.T22.1.3.9" class="ltx_td ltx_nopad_r ltx_align_left">0.516</td>
</tr>
<tr id="Ax1.T22.1.4" class="ltx_tr">
<td id="Ax1.T22.1.4.1" class="ltx_td ltx_align_left">MeetingBank</td>
<td id="Ax1.T22.1.4.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T22.1.4.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T22.1.4.4" class="ltx_td ltx_align_left">0.497</td>
<td id="Ax1.T22.1.4.5" class="ltx_td ltx_align_left">0.431</td>
<td id="Ax1.T22.1.4.6" class="ltx_td ltx_align_left">0.329</td>
<td id="Ax1.T22.1.4.7" class="ltx_td ltx_align_left">0.363</td>
<td id="Ax1.T22.1.4.8" class="ltx_td ltx_align_left">0.332</td>
<td id="Ax1.T22.1.4.9" class="ltx_td ltx_nopad_r ltx_align_left">0.343</td>
</tr>
<tr id="Ax1.T22.1.5" class="ltx_tr">
<td id="Ax1.T22.1.5.1" class="ltx_td ltx_align_left">Py150</td>
<td id="Ax1.T22.1.5.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T22.1.5.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T22.1.5.4" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T22.1.5.5" class="ltx_td ltx_align_left">0.563</td>
<td id="Ax1.T22.1.5.6" class="ltx_td ltx_align_left">0.513</td>
<td id="Ax1.T22.1.5.7" class="ltx_td ltx_align_left">0.521</td>
<td id="Ax1.T22.1.5.8" class="ltx_td ltx_align_left">0.528</td>
<td id="Ax1.T22.1.5.9" class="ltx_td ltx_nopad_r ltx_align_left">0.527</td>
</tr>
<tr id="Ax1.T22.1.6" class="ltx_tr">
<td id="Ax1.T22.1.6.1" class="ltx_td ltx_align_left">ScienceQA</td>
<td id="Ax1.T22.1.6.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T22.1.6.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T22.1.6.4" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T22.1.6.5" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T22.1.6.6" class="ltx_td ltx_align_left">0.72</td>
<td id="Ax1.T22.1.6.7" class="ltx_td ltx_align_left">0.624</td>
<td id="Ax1.T22.1.6.8" class="ltx_td ltx_align_left">0.6</td>
<td id="Ax1.T22.1.6.9" class="ltx_td ltx_nopad_r ltx_align_left">0.598</td>
</tr>
<tr id="Ax1.T22.1.7" class="ltx_tr">
<td id="Ax1.T22.1.7.1" class="ltx_td ltx_align_left">NumGLUE-cm</td>
<td id="Ax1.T22.1.7.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T22.1.7.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T22.1.7.4" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T22.1.7.5" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T22.1.7.6" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T22.1.7.7" class="ltx_td ltx_align_left">0.691</td>
<td id="Ax1.T22.1.7.8" class="ltx_td ltx_align_left">0.494</td>
<td id="Ax1.T22.1.7.9" class="ltx_td ltx_nopad_r ltx_align_left">0.469</td>
</tr>
<tr id="Ax1.T22.1.8" class="ltx_tr">
<td id="Ax1.T22.1.8.1" class="ltx_td ltx_align_left">NumGLUE-ds</td>
<td id="Ax1.T22.1.8.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T22.1.8.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T22.1.8.4" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T22.1.8.5" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T22.1.8.6" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T22.1.8.7" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T22.1.8.8" class="ltx_td ltx_align_left">0.566</td>
<td id="Ax1.T22.1.8.9" class="ltx_td ltx_nopad_r ltx_align_left">0.354</td>
</tr>
<tr id="Ax1.T22.1.9" class="ltx_tr">
<td id="Ax1.T22.1.9.1" class="ltx_td ltx_align_left">20Minuten</td>
<td id="Ax1.T22.1.9.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T22.1.9.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T22.1.9.4" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T22.1.9.5" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T22.1.9.6" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T22.1.9.7" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T22.1.9.8" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T22.1.9.9" class="ltx_td ltx_nopad_r ltx_align_left">0.402</td>
</tr>
<tr id="Ax1.T22.1.10" class="ltx_tr">
<td id="Ax1.T22.1.10.1" class="ltx_td ltx_align_left ltx_border_t">average</td>
<td id="Ax1.T22.1.10.2" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T22.1.10.3" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T22.1.10.4" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T22.1.10.5" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T22.1.10.6" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T22.1.10.7" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T22.1.10.8" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T22.1.10.9" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_t">0.466</td>
</tr>
<tr id="Ax1.T22.1.11" class="ltx_tr">
<td id="Ax1.T22.1.11.1" class="ltx_td ltx_align_left ltx_border_bb">BWT</td>
<td id="Ax1.T22.1.11.2" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T22.1.11.3" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T22.1.11.4" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T22.1.11.5" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T22.1.11.6" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T22.1.11.7" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T22.1.11.8" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T22.1.11.9" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_bb">-0.135</td>
</tr>
</table>
</figure>
</section>
<section id="Ax1.SS4.SSS6" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">.4.6 </span>Different amounts of data and training steps</h4>

<div id="Ax1.SS4.SSS6.p1" class="ltx_para">
<p id="Ax1.SS4.SSS6.p1.1" class="ltx_p">Table 23-25 shows the performance of LLaMA2-7b-chat with different number of data and training epochs.</p>
</div>
<figure id="Ax1.T23" class="ltx_table">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table">Table 23: </span>Performance of LLaMA-7b-chat after training on all of the sequential tasks for different epochs. Each dataset is sampled with 500 examples.</figcaption>
<table id="Ax1.T23.1" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="Ax1.T23.1.1" class="ltx_tr">
<td id="Ax1.T23.1.1.1" class="ltx_td ltx_align_left ltx_border_tt">Task/Number of epochs</td>
<td id="Ax1.T23.1.1.2" class="ltx_td ltx_align_left ltx_border_tt">1</td>
<td id="Ax1.T23.1.1.3" class="ltx_td ltx_align_left ltx_border_tt">3</td>
<td id="Ax1.T23.1.1.4" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_tt">5</td>
</tr>
<tr id="Ax1.T23.1.2" class="ltx_tr">
<td id="Ax1.T23.1.2.1" class="ltx_td ltx_align_left ltx_border_t">C-STANCE</td>
<td id="Ax1.T23.1.2.2" class="ltx_td ltx_align_left ltx_border_t">0.24</td>
<td id="Ax1.T23.1.2.3" class="ltx_td ltx_align_left ltx_border_t">0.38</td>
<td id="Ax1.T23.1.2.4" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_t">0.5</td>
</tr>
<tr id="Ax1.T23.1.3" class="ltx_tr">
<td id="Ax1.T23.1.3.1" class="ltx_td ltx_align_left">FOMC</td>
<td id="Ax1.T23.1.3.2" class="ltx_td ltx_align_left">0</td>
<td id="Ax1.T23.1.3.3" class="ltx_td ltx_align_left">0</td>
<td id="Ax1.T23.1.3.4" class="ltx_td ltx_nopad_r ltx_align_left">0.43</td>
</tr>
<tr id="Ax1.T23.1.4" class="ltx_tr">
<td id="Ax1.T23.1.4.1" class="ltx_td ltx_align_left">MeetingBank</td>
<td id="Ax1.T23.1.4.2" class="ltx_td ltx_align_left">0.215</td>
<td id="Ax1.T23.1.4.3" class="ltx_td ltx_align_left">0.255</td>
<td id="Ax1.T23.1.4.4" class="ltx_td ltx_nopad_r ltx_align_left">0.269</td>
</tr>
<tr id="Ax1.T23.1.5" class="ltx_tr">
<td id="Ax1.T23.1.5.1" class="ltx_td ltx_align_left">Py150</td>
<td id="Ax1.T23.1.5.2" class="ltx_td ltx_align_left">0.293</td>
<td id="Ax1.T23.1.5.3" class="ltx_td ltx_align_left">0.428</td>
<td id="Ax1.T23.1.5.4" class="ltx_td ltx_nopad_r ltx_align_left">0.49</td>
</tr>
<tr id="Ax1.T23.1.6" class="ltx_tr">
<td id="Ax1.T23.1.6.1" class="ltx_td ltx_align_left">ScienceQA</td>
<td id="Ax1.T23.1.6.2" class="ltx_td ltx_align_left">0.57</td>
<td id="Ax1.T23.1.6.3" class="ltx_td ltx_align_left"><span id="Ax1.T23.1.6.3.1" class="ltx_text" style="color:#1F2329;">0.24</span></td>
<td id="Ax1.T23.1.6.4" class="ltx_td ltx_nopad_r ltx_align_left">0.4</td>
</tr>
<tr id="Ax1.T23.1.7" class="ltx_tr">
<td id="Ax1.T23.1.7.1" class="ltx_td ltx_align_left">NumGLUE-cm</td>
<td id="Ax1.T23.1.7.2" class="ltx_td ltx_align_left">0.148</td>
<td id="Ax1.T23.1.7.3" class="ltx_td ltx_align_left">0.06</td>
<td id="Ax1.T23.1.7.4" class="ltx_td ltx_nopad_r ltx_align_left">0.21</td>
</tr>
<tr id="Ax1.T23.1.8" class="ltx_tr">
<td id="Ax1.T23.1.8.1" class="ltx_td ltx_align_left">NumGLUE-ds</td>
<td id="Ax1.T23.1.8.2" class="ltx_td ltx_align_left">0.04</td>
<td id="Ax1.T23.1.8.3" class="ltx_td ltx_align_left">0</td>
<td id="Ax1.T23.1.8.4" class="ltx_td ltx_nopad_r ltx_align_left">0.42</td>
</tr>
<tr id="Ax1.T23.1.9" class="ltx_tr">
<td id="Ax1.T23.1.9.1" class="ltx_td ltx_align_left">20Minuten</td>
<td id="Ax1.T23.1.9.2" class="ltx_td ltx_align_left">0.39</td>
<td id="Ax1.T23.1.9.3" class="ltx_td ltx_align_left">0.403</td>
<td id="Ax1.T23.1.9.4" class="ltx_td ltx_nopad_r ltx_align_left">0.41</td>
</tr>
<tr id="Ax1.T23.1.10" class="ltx_tr">
<td id="Ax1.T23.1.10.1" class="ltx_td ltx_align_left ltx_border_bb ltx_border_t">average</td>
<td id="Ax1.T23.1.10.2" class="ltx_td ltx_align_left ltx_border_bb ltx_border_t">0.237</td>
<td id="Ax1.T23.1.10.3" class="ltx_td ltx_align_left ltx_border_bb ltx_border_t">0.22</td>
<td id="Ax1.T23.1.10.4" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_bb ltx_border_t">0.391</td>
</tr>
</table>
</figure>
<figure id="Ax1.T24" class="ltx_table">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table">Table 24: </span>Performance of LLaMA-7b-chat after training on all of the sequential tasks for different epochs. Each dataset is sampled with 1000 examples.</figcaption>
<table id="Ax1.T24.1" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="Ax1.T24.1.1" class="ltx_tr">
<td id="Ax1.T24.1.1.1" class="ltx_td ltx_align_left ltx_border_tt">Task/Number of epochs</td>
<td id="Ax1.T24.1.1.2" class="ltx_td ltx_align_left ltx_border_tt">1</td>
<td id="Ax1.T24.1.1.3" class="ltx_td ltx_align_left ltx_border_tt">3</td>
<td id="Ax1.T24.1.1.4" class="ltx_td ltx_align_left ltx_border_tt">5</td>
<td id="Ax1.T24.1.1.5" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_tt">10</td>
</tr>
<tr id="Ax1.T24.1.2" class="ltx_tr">
<td id="Ax1.T24.1.2.1" class="ltx_td ltx_align_left ltx_border_t">C-STANCE</td>
<td id="Ax1.T24.1.2.2" class="ltx_td ltx_align_left ltx_border_t">0.14</td>
<td id="Ax1.T24.1.2.3" class="ltx_td ltx_align_left ltx_border_t">0.301</td>
<td id="Ax1.T24.1.2.4" class="ltx_td ltx_align_left ltx_border_t">0.57</td>
<td id="Ax1.T24.1.2.5" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_t">0.6</td>
</tr>
<tr id="Ax1.T24.1.3" class="ltx_tr">
<td id="Ax1.T24.1.3.1" class="ltx_td ltx_align_left">FOMC</td>
<td id="Ax1.T24.1.3.2" class="ltx_td ltx_align_left">0.19</td>
<td id="Ax1.T24.1.3.3" class="ltx_td ltx_align_left">0.097</td>
<td id="Ax1.T24.1.3.4" class="ltx_td ltx_align_left">0.31</td>
<td id="Ax1.T24.1.3.5" class="ltx_td ltx_nopad_r ltx_align_left">0.29</td>
</tr>
<tr id="Ax1.T24.1.4" class="ltx_tr">
<td id="Ax1.T24.1.4.1" class="ltx_td ltx_align_left">MeetingBank</td>
<td id="Ax1.T24.1.4.2" class="ltx_td ltx_align_left">0.194</td>
<td id="Ax1.T24.1.4.3" class="ltx_td ltx_align_left">0.248</td>
<td id="Ax1.T24.1.4.4" class="ltx_td ltx_align_left">0.357</td>
<td id="Ax1.T24.1.4.5" class="ltx_td ltx_nopad_r ltx_align_left">0.387</td>
</tr>
<tr id="Ax1.T24.1.5" class="ltx_tr">
<td id="Ax1.T24.1.5.1" class="ltx_td ltx_align_left">Py150</td>
<td id="Ax1.T24.1.5.2" class="ltx_td ltx_align_left">0.283</td>
<td id="Ax1.T24.1.5.3" class="ltx_td ltx_align_left">0.32</td>
<td id="Ax1.T24.1.5.4" class="ltx_td ltx_align_left">0.55</td>
<td id="Ax1.T24.1.5.5" class="ltx_td ltx_nopad_r ltx_align_left">0.54</td>
</tr>
<tr id="Ax1.T24.1.6" class="ltx_tr">
<td id="Ax1.T24.1.6.1" class="ltx_td ltx_align_left">ScienceQA</td>
<td id="Ax1.T24.1.6.2" class="ltx_td ltx_align_left">0.26</td>
<td id="Ax1.T24.1.6.3" class="ltx_td ltx_align_left"><span id="Ax1.T24.1.6.3.1" class="ltx_text" style="color:#1F2329;">0.36</span></td>
<td id="Ax1.T24.1.6.4" class="ltx_td ltx_align_left">0.41</td>
<td id="Ax1.T24.1.6.5" class="ltx_td ltx_nopad_r ltx_align_left">0.52</td>
</tr>
<tr id="Ax1.T24.1.7" class="ltx_tr">
<td id="Ax1.T24.1.7.1" class="ltx_td ltx_align_left">NumGLUE-cm</td>
<td id="Ax1.T24.1.7.2" class="ltx_td ltx_align_left">0.309</td>
<td id="Ax1.T24.1.7.3" class="ltx_td ltx_align_left">0.346</td>
<td id="Ax1.T24.1.7.4" class="ltx_td ltx_align_left">0.21</td>
<td id="Ax1.T24.1.7.5" class="ltx_td ltx_nopad_r ltx_align_left">0.24</td>
</tr>
<tr id="Ax1.T24.1.8" class="ltx_tr">
<td id="Ax1.T24.1.8.1" class="ltx_td ltx_align_left">NumGLUE-ds</td>
<td id="Ax1.T24.1.8.2" class="ltx_td ltx_align_left">0.51</td>
<td id="Ax1.T24.1.8.3" class="ltx_td ltx_align_left">0.5</td>
<td id="Ax1.T24.1.8.4" class="ltx_td ltx_align_left">0.42</td>
<td id="Ax1.T24.1.8.5" class="ltx_td ltx_nopad_r ltx_align_left">0.45</td>
</tr>
<tr id="Ax1.T24.1.9" class="ltx_tr">
<td id="Ax1.T24.1.9.1" class="ltx_td ltx_align_left">20Minuten</td>
<td id="Ax1.T24.1.9.2" class="ltx_td ltx_align_left">0.405</td>
<td id="Ax1.T24.1.9.3" class="ltx_td ltx_align_left">0.411</td>
<td id="Ax1.T24.1.9.4" class="ltx_td ltx_align_left">0.407</td>
<td id="Ax1.T24.1.9.5" class="ltx_td ltx_nopad_r ltx_align_left">0.387</td>
</tr>
<tr id="Ax1.T24.1.10" class="ltx_tr">
<td id="Ax1.T24.1.10.1" class="ltx_td ltx_align_left ltx_border_bb ltx_border_t">average</td>
<td id="Ax1.T24.1.10.2" class="ltx_td ltx_align_left ltx_border_bb ltx_border_t">0.286</td>
<td id="Ax1.T24.1.10.3" class="ltx_td ltx_align_left ltx_border_bb ltx_border_t">0.329</td>
<td id="Ax1.T24.1.10.4" class="ltx_td ltx_align_left ltx_border_bb ltx_border_t">0.404</td>
<td id="Ax1.T24.1.10.5" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_bb ltx_border_t">0.426</td>
</tr>
</table>
</figure>
<figure id="Ax1.T25" class="ltx_table">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table">Table 25: </span>Performance of LLaMA-7b-chat after training on all of the sequential tasks for different epochs. Each dataset is sampled with 5000 examples.</figcaption>
<table id="Ax1.T25.1" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="Ax1.T25.1.1" class="ltx_tr">
<td id="Ax1.T25.1.1.1" class="ltx_td ltx_align_left ltx_border_tt">Task/Number of epochs</td>
<td id="Ax1.T25.1.1.2" class="ltx_td ltx_align_left ltx_border_tt">1</td>
<td id="Ax1.T25.1.1.3" class="ltx_td ltx_align_left ltx_border_tt">3</td>
<td id="Ax1.T25.1.1.4" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_tt">5</td>
</tr>
<tr id="Ax1.T25.1.2" class="ltx_tr">
<td id="Ax1.T25.1.2.1" class="ltx_td ltx_align_left ltx_border_t">C-STANCE</td>
<td id="Ax1.T25.1.2.2" class="ltx_td ltx_align_left ltx_border_t">0.4</td>
<td id="Ax1.T25.1.2.3" class="ltx_td ltx_align_left ltx_border_t">0.48</td>
<td id="Ax1.T25.1.2.4" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_t">0.454</td>
</tr>
<tr id="Ax1.T25.1.3" class="ltx_tr">
<td id="Ax1.T25.1.3.1" class="ltx_td ltx_align_left">FOMC</td>
<td id="Ax1.T25.1.3.2" class="ltx_td ltx_align_left">0.63</td>
<td id="Ax1.T25.1.3.3" class="ltx_td ltx_align_left">0.28</td>
<td id="Ax1.T25.1.3.4" class="ltx_td ltx_nopad_r ltx_align_left">0.609</td>
</tr>
<tr id="Ax1.T25.1.4" class="ltx_tr">
<td id="Ax1.T25.1.4.1" class="ltx_td ltx_align_left">MeetingBank</td>
<td id="Ax1.T25.1.4.2" class="ltx_td ltx_align_left">0.393</td>
<td id="Ax1.T25.1.4.3" class="ltx_td ltx_align_left">0.388</td>
<td id="Ax1.T25.1.4.4" class="ltx_td ltx_nopad_r ltx_align_left">0.457</td>
</tr>
<tr id="Ax1.T25.1.5" class="ltx_tr">
<td id="Ax1.T25.1.5.1" class="ltx_td ltx_align_left">Py150</td>
<td id="Ax1.T25.1.5.2" class="ltx_td ltx_align_left">0.596</td>
<td id="Ax1.T25.1.5.3" class="ltx_td ltx_align_left">0.474</td>
<td id="Ax1.T25.1.5.4" class="ltx_td ltx_nopad_r ltx_align_left">0.512</td>
</tr>
<tr id="Ax1.T25.1.6" class="ltx_tr">
<td id="Ax1.T25.1.6.1" class="ltx_td ltx_align_left">ScienceQA</td>
<td id="Ax1.T25.1.6.2" class="ltx_td ltx_align_left">0.48</td>
<td id="Ax1.T25.1.6.3" class="ltx_td ltx_align_left"><span id="Ax1.T25.1.6.3.1" class="ltx_text" style="color:#1F2329;">0.62</span></td>
<td id="Ax1.T25.1.6.4" class="ltx_td ltx_nopad_r ltx_align_left">0.637</td>
</tr>
<tr id="Ax1.T25.1.7" class="ltx_tr">
<td id="Ax1.T25.1.7.1" class="ltx_td ltx_align_left">NumGLUE-cm</td>
<td id="Ax1.T25.1.7.2" class="ltx_td ltx_align_left">0.272</td>
<td id="Ax1.T25.1.7.3" class="ltx_td ltx_align_left">0.309</td>
<td id="Ax1.T25.1.7.4" class="ltx_td ltx_nopad_r ltx_align_left">0.272</td>
</tr>
<tr id="Ax1.T25.1.8" class="ltx_tr">
<td id="Ax1.T25.1.8.1" class="ltx_td ltx_align_left">NumGLUE-ds</td>
<td id="Ax1.T25.1.8.2" class="ltx_td ltx_align_left">0.32</td>
<td id="Ax1.T25.1.8.3" class="ltx_td ltx_align_left">0.49</td>
<td id="Ax1.T25.1.8.4" class="ltx_td ltx_nopad_r ltx_align_left">0.548</td>
</tr>
<tr id="Ax1.T25.1.9" class="ltx_tr">
<td id="Ax1.T25.1.9.1" class="ltx_td ltx_align_left">20Minuten</td>
<td id="Ax1.T25.1.9.2" class="ltx_td ltx_align_left">0.416</td>
<td id="Ax1.T25.1.9.3" class="ltx_td ltx_align_left">0.413</td>
<td id="Ax1.T25.1.9.4" class="ltx_td ltx_nopad_r ltx_align_left">0.408</td>
</tr>
<tr id="Ax1.T25.1.10" class="ltx_tr">
<td id="Ax1.T25.1.10.1" class="ltx_td ltx_align_left ltx_border_bb ltx_border_t">average</td>
<td id="Ax1.T25.1.10.2" class="ltx_td ltx_align_left ltx_border_bb ltx_border_t">0.438</td>
<td id="Ax1.T25.1.10.3" class="ltx_td ltx_align_left ltx_border_bb ltx_border_t">0.432</td>
<td id="Ax1.T25.1.10.4" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_bb ltx_border_t">0.487</td>
</tr>
</table>
</figure>
</section>
<section id="Ax1.SS4.SSS7" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">.4.7 </span>Different Order</h4>

<div id="Ax1.SS4.SSS7.p1" class="ltx_para">
<p id="Ax1.SS4.SSS7.p1.1" class="ltx_p">To migrate the influence of different order of tasks, we experiment with one different sequence of tasks: NumGLUE-cm, NumGLUE-ds, FOMC,20Minuten, C-STANCE, Py150, MeetingBank, ScienceQA. We report the results in Table <a href="#Ax1.T26" title="Table 26 ‣ .4.7 Different Order ‣ .4 Detailed Experiments Results ‣ Appendix ‣ TRACE: A Comprehensive Benchmark for Continual Learning in Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">26</span></a>.</p>
</div>
<figure id="Ax1.T26" class="ltx_table">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table">Table 26: </span>Detailed results of the second order of LLaMA-7b-chat</figcaption>
<table id="Ax1.T26.1" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="Ax1.T26.1.1" class="ltx_tr">
<td id="Ax1.T26.1.1.1" class="ltx_td ltx_align_left ltx_border_tt">Task\Round</td>
<td id="Ax1.T26.1.1.2" class="ltx_td ltx_align_left ltx_border_tt">1</td>
<td id="Ax1.T26.1.1.3" class="ltx_td ltx_align_left ltx_border_tt">2</td>
<td id="Ax1.T26.1.1.4" class="ltx_td ltx_align_left ltx_border_tt">3</td>
<td id="Ax1.T26.1.1.5" class="ltx_td ltx_align_left ltx_border_tt">4</td>
<td id="Ax1.T26.1.1.6" class="ltx_td ltx_align_left ltx_border_tt">5</td>
<td id="Ax1.T26.1.1.7" class="ltx_td ltx_align_left ltx_border_tt">6</td>
<td id="Ax1.T26.1.1.8" class="ltx_td ltx_align_left ltx_border_tt">7</td>
<td id="Ax1.T26.1.1.9" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_tt">8</td>
</tr>
<tr id="Ax1.T26.1.2" class="ltx_tr">
<td id="Ax1.T26.1.2.1" class="ltx_td ltx_align_left ltx_border_t">NumGLUE-cm</td>
<td id="Ax1.T26.1.2.2" class="ltx_td ltx_align_left ltx_border_t">0.333</td>
<td id="Ax1.T26.1.2.3" class="ltx_td ltx_align_left ltx_border_t">0.21</td>
<td id="Ax1.T26.1.2.4" class="ltx_td ltx_align_left ltx_border_t">0.222</td>
<td id="Ax1.T26.1.2.5" class="ltx_td ltx_align_left ltx_border_t">0235</td>
<td id="Ax1.T26.1.2.6" class="ltx_td ltx_align_left ltx_border_t">0.247</td>
<td id="Ax1.T26.1.2.7" class="ltx_td ltx_align_left ltx_border_t">0.284</td>
<td id="Ax1.T26.1.2.8" class="ltx_td ltx_align_left ltx_border_t">0.296</td>
<td id="Ax1.T26.1.2.9" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_t">0.309</td>
</tr>
<tr id="Ax1.T26.1.3" class="ltx_tr">
<td id="Ax1.T26.1.3.1" class="ltx_td ltx_align_left">NumGLUE-ds</td>
<td id="Ax1.T26.1.3.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T26.1.3.3" class="ltx_td ltx_align_left">0.61</td>
<td id="Ax1.T26.1.3.4" class="ltx_td ltx_align_left">0.52</td>
<td id="Ax1.T26.1.3.5" class="ltx_td ltx_align_left">0.52</td>
<td id="Ax1.T26.1.3.6" class="ltx_td ltx_align_left"><span id="Ax1.T26.1.3.6.1" class="ltx_text" style="color:#1F2329;">0.5</span></td>
<td id="Ax1.T26.1.3.7" class="ltx_td ltx_align_left">0.52</td>
<td id="Ax1.T26.1.3.8" class="ltx_td ltx_align_left">0.587</td>
<td id="Ax1.T26.1.3.9" class="ltx_td ltx_nopad_r ltx_align_left">0.587</td>
</tr>
<tr id="Ax1.T26.1.4" class="ltx_tr">
<td id="Ax1.T26.1.4.1" class="ltx_td ltx_align_left">FOMC</td>
<td id="Ax1.T26.1.4.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T26.1.4.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T26.1.4.4" class="ltx_td ltx_align_left">0.751</td>
<td id="Ax1.T26.1.4.5" class="ltx_td ltx_align_left">0.392</td>
<td id="Ax1.T26.1.4.6" class="ltx_td ltx_align_left">0.7</td>
<td id="Ax1.T26.1.4.7" class="ltx_td ltx_align_left">0.656</td>
<td id="Ax1.T26.1.4.8" class="ltx_td ltx_align_left">0.587</td>
<td id="Ax1.T26.1.4.9" class="ltx_td ltx_nopad_r ltx_align_left">0</td>
</tr>
<tr id="Ax1.T26.1.5" class="ltx_tr">
<td id="Ax1.T26.1.5.1" class="ltx_td ltx_align_left">20Minuten</td>
<td id="Ax1.T26.1.5.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T26.1.5.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T26.1.5.4" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T26.1.5.5" class="ltx_td ltx_align_left">0.408</td>
<td id="Ax1.T26.1.5.6" class="ltx_td ltx_align_left">0.394</td>
<td id="Ax1.T26.1.5.7" class="ltx_td ltx_align_left">0.404</td>
<td id="Ax1.T26.1.5.8" class="ltx_td ltx_align_left">0.404</td>
<td id="Ax1.T26.1.5.9" class="ltx_td ltx_nopad_r ltx_align_left">0.389</td>
</tr>
<tr id="Ax1.T26.1.6" class="ltx_tr">
<td id="Ax1.T26.1.6.1" class="ltx_td ltx_align_left">C-STANCE</td>
<td id="Ax1.T26.1.6.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T26.1.6.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T26.1.6.4" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T26.1.6.5" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T26.1.6.6" class="ltx_td ltx_align_left">0.538</td>
<td id="Ax1.T26.1.6.7" class="ltx_td ltx_align_left">0.462</td>
<td id="Ax1.T26.1.6.8" class="ltx_td ltx_align_left">0.47</td>
<td id="Ax1.T26.1.6.9" class="ltx_td ltx_nopad_r ltx_align_left">0</td>
</tr>
<tr id="Ax1.T26.1.7" class="ltx_tr">
<td id="Ax1.T26.1.7.1" class="ltx_td ltx_align_left">Py150</td>
<td id="Ax1.T26.1.7.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T26.1.7.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T26.1.7.4" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T26.1.7.5" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T26.1.7.6" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T26.1.7.7" class="ltx_td ltx_align_left">0.569</td>
<td id="Ax1.T26.1.7.8" class="ltx_td ltx_align_left">0.536</td>
<td id="Ax1.T26.1.7.9" class="ltx_td ltx_nopad_r ltx_align_left">0.494</td>
</tr>
<tr id="Ax1.T26.1.8" class="ltx_tr">
<td id="Ax1.T26.1.8.1" class="ltx_td ltx_align_left">MeetingBank</td>
<td id="Ax1.T26.1.8.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T26.1.8.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T26.1.8.4" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T26.1.8.5" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T26.1.8.6" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T26.1.8.7" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T26.1.8.8" class="ltx_td ltx_align_left">0.506</td>
<td id="Ax1.T26.1.8.9" class="ltx_td ltx_nopad_r ltx_align_left">0.387</td>
</tr>
<tr id="Ax1.T26.1.9" class="ltx_tr">
<td id="Ax1.T26.1.9.1" class="ltx_td ltx_align_left">ScienceQA</td>
<td id="Ax1.T26.1.9.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T26.1.9.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T26.1.9.4" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T26.1.9.5" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T26.1.9.6" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T26.1.9.7" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T26.1.9.8" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T26.1.9.9" class="ltx_td ltx_nopad_r ltx_align_left">0.46</td>
</tr>
<tr id="Ax1.T26.1.10" class="ltx_tr">
<td id="Ax1.T26.1.10.1" class="ltx_td ltx_align_left ltx_border_t">average</td>
<td id="Ax1.T26.1.10.2" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T26.1.10.3" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T26.1.10.4" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T26.1.10.5" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T26.1.10.6" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T26.1.10.7" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T26.1.10.8" class="ltx_td ltx_border_t"></td>
<td id="Ax1.T26.1.10.9" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_t">0.329</td>
</tr>
<tr id="Ax1.T26.1.11" class="ltx_tr">
<td id="Ax1.T26.1.11.1" class="ltx_td ltx_align_left ltx_border_bb">BWT</td>
<td id="Ax1.T26.1.11.2" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T26.1.11.3" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T26.1.11.4" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T26.1.11.5" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T26.1.11.6" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T26.1.11.7" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T26.1.11.8" class="ltx_td ltx_border_bb"></td>
<td id="Ax1.T26.1.11.9" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_bb">-0.221</td>
</tr>
</table>
</figure>
</section>
<section id="Ax1.SS4.SSS8" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">.4.8 </span>Detailed results of Table <a href="#S5.T3" title="Table 3 ‣ 5.1 Performance on Target Sequential Tasks ‣ 5 Reasoning-augmented Continual learning ‣ TRACE: A Comprehensive Benchmark for Continual Learning in Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>
</h4>

<figure id="Ax1.T27" class="ltx_table">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table">Table 27: </span>Detailed results of Table <a href="#S5.T3" title="Table 3 ‣ 5.1 Performance on Target Sequential Tasks ‣ 5 Reasoning-augmented Continual learning ‣ TRACE: A Comprehensive Benchmark for Continual Learning in Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a></figcaption>
<div id="Ax1.T27.64" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:397.5pt;height:155.5pt;vertical-align:-0.9pt;"><span class="ltx_transformed_inner" style="transform:translate(-32.7pt,12.7pt) scale(0.858754874831885,0.858754874831885) ;">
<table id="Ax1.T27.64.64" class="ltx_tabular ltx_align_middle">
<tr id="Ax1.T27.64.64.65" class="ltx_tr">
<td id="Ax1.T27.64.64.65.1" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:2.0pt;padding-right:2.0pt;">Dataset</td>
<td id="Ax1.T27.64.64.65.2" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:2.0pt;padding-right:2.0pt;">C-STANCE</td>
<td id="Ax1.T27.64.64.65.3" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:2.0pt;padding-right:2.0pt;">FOMC</td>
<td id="Ax1.T27.64.64.65.4" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:2.0pt;padding-right:2.0pt;">MeetingBank</td>
<td id="Ax1.T27.64.64.65.5" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:2.0pt;padding-right:2.0pt;">Py150</td>
<td id="Ax1.T27.64.64.65.6" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:2.0pt;padding-right:2.0pt;">ScienceQA</td>
<td id="Ax1.T27.64.64.65.7" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:2.0pt;padding-right:2.0pt;">NumGLUE-cm</td>
<td id="Ax1.T27.64.64.65.8" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:2.0pt;padding-right:2.0pt;">NumGLUE-ds</td>
<td id="Ax1.T27.64.64.65.9" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:2.0pt;padding-right:2.0pt;">20Minuten</td>
</tr>
<tr id="Ax1.T27.8.8.8" class="ltx_tr">
<td id="Ax1.T27.8.8.8.9" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:2.0pt;padding-right:2.0pt;">ICL</td>
<td id="Ax1.T27.1.1.1.1" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="Ax1.T27.1.1.1.1.m1.1" class="ltx_Math" alttext="0.40" display="inline"><semantics id="Ax1.T27.1.1.1.1.m1.1a"><mn id="Ax1.T27.1.1.1.1.m1.1.1" xref="Ax1.T27.1.1.1.1.m1.1.1.cmml">0.40</mn><annotation-xml encoding="MathML-Content" id="Ax1.T27.1.1.1.1.m1.1b"><cn type="float" id="Ax1.T27.1.1.1.1.m1.1.1.cmml" xref="Ax1.T27.1.1.1.1.m1.1.1">0.40</cn></annotation-xml><annotation encoding="application/x-tex" id="Ax1.T27.1.1.1.1.m1.1c">0.40</annotation></semantics></math></td>
<td id="Ax1.T27.2.2.2.2" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="Ax1.T27.2.2.2.2.m1.1" class="ltx_Math" alttext="0.48" display="inline"><semantics id="Ax1.T27.2.2.2.2.m1.1a"><mn id="Ax1.T27.2.2.2.2.m1.1.1" xref="Ax1.T27.2.2.2.2.m1.1.1.cmml">0.48</mn><annotation-xml encoding="MathML-Content" id="Ax1.T27.2.2.2.2.m1.1b"><cn type="float" id="Ax1.T27.2.2.2.2.m1.1.1.cmml" xref="Ax1.T27.2.2.2.2.m1.1.1">0.48</cn></annotation-xml><annotation encoding="application/x-tex" id="Ax1.T27.2.2.2.2.m1.1c">0.48</annotation></semantics></math></td>
<td id="Ax1.T27.3.3.3.3" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="Ax1.T27.3.3.3.3.m1.1" class="ltx_Math" alttext="0.20" display="inline"><semantics id="Ax1.T27.3.3.3.3.m1.1a"><mn id="Ax1.T27.3.3.3.3.m1.1.1" xref="Ax1.T27.3.3.3.3.m1.1.1.cmml">0.20</mn><annotation-xml encoding="MathML-Content" id="Ax1.T27.3.3.3.3.m1.1b"><cn type="float" id="Ax1.T27.3.3.3.3.m1.1.1.cmml" xref="Ax1.T27.3.3.3.3.m1.1.1">0.20</cn></annotation-xml><annotation encoding="application/x-tex" id="Ax1.T27.3.3.3.3.m1.1c">0.20</annotation></semantics></math></td>
<td id="Ax1.T27.4.4.4.4" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="Ax1.T27.4.4.4.4.m1.1" class="ltx_Math" alttext="0.52" display="inline"><semantics id="Ax1.T27.4.4.4.4.m1.1a"><mn id="Ax1.T27.4.4.4.4.m1.1.1" xref="Ax1.T27.4.4.4.4.m1.1.1.cmml">0.52</mn><annotation-xml encoding="MathML-Content" id="Ax1.T27.4.4.4.4.m1.1b"><cn type="float" id="Ax1.T27.4.4.4.4.m1.1.1.cmml" xref="Ax1.T27.4.4.4.4.m1.1.1">0.52</cn></annotation-xml><annotation encoding="application/x-tex" id="Ax1.T27.4.4.4.4.m1.1c">0.52</annotation></semantics></math></td>
<td id="Ax1.T27.5.5.5.5" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="Ax1.T27.5.5.5.5.m1.1" class="ltx_Math" alttext="0.63" display="inline"><semantics id="Ax1.T27.5.5.5.5.m1.1a"><mn id="Ax1.T27.5.5.5.5.m1.1.1" xref="Ax1.T27.5.5.5.5.m1.1.1.cmml">0.63</mn><annotation-xml encoding="MathML-Content" id="Ax1.T27.5.5.5.5.m1.1b"><cn type="float" id="Ax1.T27.5.5.5.5.m1.1.1.cmml" xref="Ax1.T27.5.5.5.5.m1.1.1">0.63</cn></annotation-xml><annotation encoding="application/x-tex" id="Ax1.T27.5.5.5.5.m1.1c">0.63</annotation></semantics></math></td>
<td id="Ax1.T27.6.6.6.6" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="Ax1.T27.6.6.6.6.m1.1" class="ltx_Math" alttext="0.28" display="inline"><semantics id="Ax1.T27.6.6.6.6.m1.1a"><mn id="Ax1.T27.6.6.6.6.m1.1.1" xref="Ax1.T27.6.6.6.6.m1.1.1.cmml">0.28</mn><annotation-xml encoding="MathML-Content" id="Ax1.T27.6.6.6.6.m1.1b"><cn type="float" id="Ax1.T27.6.6.6.6.m1.1.1.cmml" xref="Ax1.T27.6.6.6.6.m1.1.1">0.28</cn></annotation-xml><annotation encoding="application/x-tex" id="Ax1.T27.6.6.6.6.m1.1c">0.28</annotation></semantics></math></td>
<td id="Ax1.T27.7.7.7.7" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="Ax1.T27.7.7.7.7.m1.1" class="ltx_Math" alttext="0.20" display="inline"><semantics id="Ax1.T27.7.7.7.7.m1.1a"><mn id="Ax1.T27.7.7.7.7.m1.1.1" xref="Ax1.T27.7.7.7.7.m1.1.1.cmml">0.20</mn><annotation-xml encoding="MathML-Content" id="Ax1.T27.7.7.7.7.m1.1b"><cn type="float" id="Ax1.T27.7.7.7.7.m1.1.1.cmml" xref="Ax1.T27.7.7.7.7.m1.1.1">0.20</cn></annotation-xml><annotation encoding="application/x-tex" id="Ax1.T27.7.7.7.7.m1.1c">0.20</annotation></semantics></math></td>
<td id="Ax1.T27.8.8.8.8" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="Ax1.T27.8.8.8.8.m1.1" class="ltx_Math" alttext="0.40" display="inline"><semantics id="Ax1.T27.8.8.8.8.m1.1a"><mn id="Ax1.T27.8.8.8.8.m1.1.1" xref="Ax1.T27.8.8.8.8.m1.1.1.cmml">0.40</mn><annotation-xml encoding="MathML-Content" id="Ax1.T27.8.8.8.8.m1.1b"><cn type="float" id="Ax1.T27.8.8.8.8.m1.1.1.cmml" xref="Ax1.T27.8.8.8.8.m1.1.1">0.40</cn></annotation-xml><annotation encoding="application/x-tex" id="Ax1.T27.8.8.8.8.m1.1c">0.40</annotation></semantics></math></td>
</tr>
<tr id="Ax1.T27.16.16.16" class="ltx_tr">
<td id="Ax1.T27.16.16.16.9" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;">SeqFT</td>
<td id="Ax1.T27.9.9.9.1" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="Ax1.T27.9.9.9.1.m1.1" class="ltx_Math" alttext="0.45" display="inline"><semantics id="Ax1.T27.9.9.9.1.m1.1a"><mn id="Ax1.T27.9.9.9.1.m1.1.1" xref="Ax1.T27.9.9.9.1.m1.1.1.cmml">0.45</mn><annotation-xml encoding="MathML-Content" id="Ax1.T27.9.9.9.1.m1.1b"><cn type="float" id="Ax1.T27.9.9.9.1.m1.1.1.cmml" xref="Ax1.T27.9.9.9.1.m1.1.1">0.45</cn></annotation-xml><annotation encoding="application/x-tex" id="Ax1.T27.9.9.9.1.m1.1c">0.45</annotation></semantics></math></td>
<td id="Ax1.T27.10.10.10.2" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="Ax1.T27.10.10.10.2.m1.1" class="ltx_Math" alttext="0.61" display="inline"><semantics id="Ax1.T27.10.10.10.2.m1.1a"><mn id="Ax1.T27.10.10.10.2.m1.1.1" xref="Ax1.T27.10.10.10.2.m1.1.1.cmml">0.61</mn><annotation-xml encoding="MathML-Content" id="Ax1.T27.10.10.10.2.m1.1b"><cn type="float" id="Ax1.T27.10.10.10.2.m1.1.1.cmml" xref="Ax1.T27.10.10.10.2.m1.1.1">0.61</cn></annotation-xml><annotation encoding="application/x-tex" id="Ax1.T27.10.10.10.2.m1.1c">0.61</annotation></semantics></math></td>
<td id="Ax1.T27.11.11.11.3" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="Ax1.T27.11.11.11.3.m1.1" class="ltx_Math" alttext="0.46" display="inline"><semantics id="Ax1.T27.11.11.11.3.m1.1a"><mn id="Ax1.T27.11.11.11.3.m1.1.1" xref="Ax1.T27.11.11.11.3.m1.1.1.cmml">0.46</mn><annotation-xml encoding="MathML-Content" id="Ax1.T27.11.11.11.3.m1.1b"><cn type="float" id="Ax1.T27.11.11.11.3.m1.1.1.cmml" xref="Ax1.T27.11.11.11.3.m1.1.1">0.46</cn></annotation-xml><annotation encoding="application/x-tex" id="Ax1.T27.11.11.11.3.m1.1c">0.46</annotation></semantics></math></td>
<td id="Ax1.T27.12.12.12.4" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="Ax1.T27.12.12.12.4.m1.1" class="ltx_Math" alttext="0.51" display="inline"><semantics id="Ax1.T27.12.12.12.4.m1.1a"><mn id="Ax1.T27.12.12.12.4.m1.1.1" xref="Ax1.T27.12.12.12.4.m1.1.1.cmml">0.51</mn><annotation-xml encoding="MathML-Content" id="Ax1.T27.12.12.12.4.m1.1b"><cn type="float" id="Ax1.T27.12.12.12.4.m1.1.1.cmml" xref="Ax1.T27.12.12.12.4.m1.1.1">0.51</cn></annotation-xml><annotation encoding="application/x-tex" id="Ax1.T27.12.12.12.4.m1.1c">0.51</annotation></semantics></math></td>
<td id="Ax1.T27.13.13.13.5" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="Ax1.T27.13.13.13.5.m1.1" class="ltx_Math" alttext="0.64" display="inline"><semantics id="Ax1.T27.13.13.13.5.m1.1a"><mn id="Ax1.T27.13.13.13.5.m1.1.1" xref="Ax1.T27.13.13.13.5.m1.1.1.cmml">0.64</mn><annotation-xml encoding="MathML-Content" id="Ax1.T27.13.13.13.5.m1.1b"><cn type="float" id="Ax1.T27.13.13.13.5.m1.1.1.cmml" xref="Ax1.T27.13.13.13.5.m1.1.1">0.64</cn></annotation-xml><annotation encoding="application/x-tex" id="Ax1.T27.13.13.13.5.m1.1c">0.64</annotation></semantics></math></td>
<td id="Ax1.T27.14.14.14.6" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="Ax1.T27.14.14.14.6.m1.1" class="ltx_Math" alttext="0.27" display="inline"><semantics id="Ax1.T27.14.14.14.6.m1.1a"><mn id="Ax1.T27.14.14.14.6.m1.1.1" xref="Ax1.T27.14.14.14.6.m1.1.1.cmml">0.27</mn><annotation-xml encoding="MathML-Content" id="Ax1.T27.14.14.14.6.m1.1b"><cn type="float" id="Ax1.T27.14.14.14.6.m1.1.1.cmml" xref="Ax1.T27.14.14.14.6.m1.1.1">0.27</cn></annotation-xml><annotation encoding="application/x-tex" id="Ax1.T27.14.14.14.6.m1.1c">0.27</annotation></semantics></math></td>
<td id="Ax1.T27.15.15.15.7" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="Ax1.T27.15.15.15.7.m1.1" class="ltx_Math" alttext="0.55" display="inline"><semantics id="Ax1.T27.15.15.15.7.m1.1a"><mn id="Ax1.T27.15.15.15.7.m1.1.1" xref="Ax1.T27.15.15.15.7.m1.1.1.cmml">0.55</mn><annotation-xml encoding="MathML-Content" id="Ax1.T27.15.15.15.7.m1.1b"><cn type="float" id="Ax1.T27.15.15.15.7.m1.1.1.cmml" xref="Ax1.T27.15.15.15.7.m1.1.1">0.55</cn></annotation-xml><annotation encoding="application/x-tex" id="Ax1.T27.15.15.15.7.m1.1c">0.55</annotation></semantics></math></td>
<td id="Ax1.T27.16.16.16.8" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="Ax1.T27.16.16.16.8.m1.1" class="ltx_Math" alttext="0.41" display="inline"><semantics id="Ax1.T27.16.16.16.8.m1.1a"><mn id="Ax1.T27.16.16.16.8.m1.1.1" xref="Ax1.T27.16.16.16.8.m1.1.1.cmml">0.41</mn><annotation-xml encoding="MathML-Content" id="Ax1.T27.16.16.16.8.m1.1b"><cn type="float" id="Ax1.T27.16.16.16.8.m1.1.1.cmml" xref="Ax1.T27.16.16.16.8.m1.1.1">0.41</cn></annotation-xml><annotation encoding="application/x-tex" id="Ax1.T27.16.16.16.8.m1.1c">0.41</annotation></semantics></math></td>
</tr>
<tr id="Ax1.T27.24.24.24" class="ltx_tr">
<td id="Ax1.T27.24.24.24.9" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;">w/o. RE</td>
<td id="Ax1.T27.17.17.17.1" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="Ax1.T27.17.17.17.1.m1.1" class="ltx_Math" alttext="0.36" display="inline"><semantics id="Ax1.T27.17.17.17.1.m1.1a"><mn id="Ax1.T27.17.17.17.1.m1.1.1" xref="Ax1.T27.17.17.17.1.m1.1.1.cmml">0.36</mn><annotation-xml encoding="MathML-Content" id="Ax1.T27.17.17.17.1.m1.1b"><cn type="float" id="Ax1.T27.17.17.17.1.m1.1.1.cmml" xref="Ax1.T27.17.17.17.1.m1.1.1">0.36</cn></annotation-xml><annotation encoding="application/x-tex" id="Ax1.T27.17.17.17.1.m1.1c">0.36</annotation></semantics></math></td>
<td id="Ax1.T27.18.18.18.2" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="Ax1.T27.18.18.18.2.m1.1" class="ltx_Math" alttext="0" display="inline"><semantics id="Ax1.T27.18.18.18.2.m1.1a"><mn id="Ax1.T27.18.18.18.2.m1.1.1" xref="Ax1.T27.18.18.18.2.m1.1.1.cmml">0</mn><annotation-xml encoding="MathML-Content" id="Ax1.T27.18.18.18.2.m1.1b"><cn type="integer" id="Ax1.T27.18.18.18.2.m1.1.1.cmml" xref="Ax1.T27.18.18.18.2.m1.1.1">0</cn></annotation-xml></semantics></math></td>
<td id="Ax1.T27.19.19.19.3" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="Ax1.T27.19.19.19.3.m1.1" class="ltx_Math" alttext="0.24" display="inline"><semantics id="Ax1.T27.19.19.19.3.m1.1a"><mn id="Ax1.T27.19.19.19.3.m1.1.1" xref="Ax1.T27.19.19.19.3.m1.1.1.cmml">0.24</mn><annotation-xml encoding="MathML-Content" id="Ax1.T27.19.19.19.3.m1.1b"><cn type="float" id="Ax1.T27.19.19.19.3.m1.1.1.cmml" xref="Ax1.T27.19.19.19.3.m1.1.1">0.24</cn></annotation-xml><annotation encoding="application/x-tex" id="Ax1.T27.19.19.19.3.m1.1c">0.24</annotation></semantics></math></td>
<td id="Ax1.T27.20.20.20.4" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="Ax1.T27.20.20.20.4.m1.1" class="ltx_Math" alttext="0.34" display="inline"><semantics id="Ax1.T27.20.20.20.4.m1.1a"><mn id="Ax1.T27.20.20.20.4.m1.1.1" xref="Ax1.T27.20.20.20.4.m1.1.1.cmml">0.34</mn><annotation-xml encoding="MathML-Content" id="Ax1.T27.20.20.20.4.m1.1b"><cn type="float" id="Ax1.T27.20.20.20.4.m1.1.1.cmml" xref="Ax1.T27.20.20.20.4.m1.1.1">0.34</cn></annotation-xml><annotation encoding="application/x-tex" id="Ax1.T27.20.20.20.4.m1.1c">0.34</annotation></semantics></math></td>
<td id="Ax1.T27.21.21.21.5" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="Ax1.T27.21.21.21.5.m1.1" class="ltx_Math" alttext="0.40" display="inline"><semantics id="Ax1.T27.21.21.21.5.m1.1a"><mn id="Ax1.T27.21.21.21.5.m1.1.1" xref="Ax1.T27.21.21.21.5.m1.1.1.cmml">0.40</mn><annotation-xml encoding="MathML-Content" id="Ax1.T27.21.21.21.5.m1.1b"><cn type="float" id="Ax1.T27.21.21.21.5.m1.1.1.cmml" xref="Ax1.T27.21.21.21.5.m1.1.1">0.40</cn></annotation-xml><annotation encoding="application/x-tex" id="Ax1.T27.21.21.21.5.m1.1c">0.40</annotation></semantics></math></td>
<td id="Ax1.T27.22.22.22.6" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="Ax1.T27.22.22.22.6.m1.1" class="ltx_Math" alttext="0.10" display="inline"><semantics id="Ax1.T27.22.22.22.6.m1.1a"><mn id="Ax1.T27.22.22.22.6.m1.1.1" xref="Ax1.T27.22.22.22.6.m1.1.1.cmml">0.10</mn><annotation-xml encoding="MathML-Content" id="Ax1.T27.22.22.22.6.m1.1b"><cn type="float" id="Ax1.T27.22.22.22.6.m1.1.1.cmml" xref="Ax1.T27.22.22.22.6.m1.1.1">0.10</cn></annotation-xml><annotation encoding="application/x-tex" id="Ax1.T27.22.22.22.6.m1.1c">0.10</annotation></semantics></math></td>
<td id="Ax1.T27.23.23.23.7" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="Ax1.T27.23.23.23.7.m1.1" class="ltx_Math" alttext="0" display="inline"><semantics id="Ax1.T27.23.23.23.7.m1.1a"><mn id="Ax1.T27.23.23.23.7.m1.1.1" xref="Ax1.T27.23.23.23.7.m1.1.1.cmml">0</mn><annotation-xml encoding="MathML-Content" id="Ax1.T27.23.23.23.7.m1.1b"><cn type="integer" id="Ax1.T27.23.23.23.7.m1.1.1.cmml" xref="Ax1.T27.23.23.23.7.m1.1.1">0</cn></annotation-xml></semantics></math></td>
<td id="Ax1.T27.24.24.24.8" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="Ax1.T27.24.24.24.8.m1.1" class="ltx_Math" alttext="0.40" display="inline"><semantics id="Ax1.T27.24.24.24.8.m1.1a"><mn id="Ax1.T27.24.24.24.8.m1.1.1" xref="Ax1.T27.24.24.24.8.m1.1.1.cmml">0.40</mn><annotation-xml encoding="MathML-Content" id="Ax1.T27.24.24.24.8.m1.1b"><cn type="float" id="Ax1.T27.24.24.24.8.m1.1.1.cmml" xref="Ax1.T27.24.24.24.8.m1.1.1">0.40</cn></annotation-xml><annotation encoding="application/x-tex" id="Ax1.T27.24.24.24.8.m1.1c">0.40</annotation></semantics></math></td>
</tr>
<tr id="Ax1.T27.32.32.32" class="ltx_tr">
<td id="Ax1.T27.32.32.32.9" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;">O-Lora</td>
<td id="Ax1.T27.25.25.25.1" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="Ax1.T27.25.25.25.1.m1.1" class="ltx_Math" alttext="0.482" display="inline"><semantics id="Ax1.T27.25.25.25.1.m1.1a"><mn id="Ax1.T27.25.25.25.1.m1.1.1" xref="Ax1.T27.25.25.25.1.m1.1.1.cmml">0.482</mn><annotation-xml encoding="MathML-Content" id="Ax1.T27.25.25.25.1.m1.1b"><cn type="float" id="Ax1.T27.25.25.25.1.m1.1.1.cmml" xref="Ax1.T27.25.25.25.1.m1.1.1">0.482</cn></annotation-xml><annotation encoding="application/x-tex" id="Ax1.T27.25.25.25.1.m1.1c">0.482</annotation></semantics></math></td>
<td id="Ax1.T27.26.26.26.2" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="Ax1.T27.26.26.26.2.m1.1" class="ltx_Math" alttext="0.336" display="inline"><semantics id="Ax1.T27.26.26.26.2.m1.1a"><mn id="Ax1.T27.26.26.26.2.m1.1.1" xref="Ax1.T27.26.26.26.2.m1.1.1.cmml">0.336</mn><annotation-xml encoding="MathML-Content" id="Ax1.T27.26.26.26.2.m1.1b"><cn type="float" id="Ax1.T27.26.26.26.2.m1.1.1.cmml" xref="Ax1.T27.26.26.26.2.m1.1.1">0.336</cn></annotation-xml><annotation encoding="application/x-tex" id="Ax1.T27.26.26.26.2.m1.1c">0.336</annotation></semantics></math></td>
<td id="Ax1.T27.27.27.27.3" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="Ax1.T27.27.27.27.3.m1.1" class="ltx_Math" alttext="0.409" display="inline"><semantics id="Ax1.T27.27.27.27.3.m1.1a"><mn id="Ax1.T27.27.27.27.3.m1.1.1" xref="Ax1.T27.27.27.27.3.m1.1.1.cmml">0.409</mn><annotation-xml encoding="MathML-Content" id="Ax1.T27.27.27.27.3.m1.1b"><cn type="float" id="Ax1.T27.27.27.27.3.m1.1.1.cmml" xref="Ax1.T27.27.27.27.3.m1.1.1">0.409</cn></annotation-xml><annotation encoding="application/x-tex" id="Ax1.T27.27.27.27.3.m1.1c">0.409</annotation></semantics></math></td>
<td id="Ax1.T27.28.28.28.4" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="Ax1.T27.28.28.28.4.m1.1" class="ltx_Math" alttext="0.53" display="inline"><semantics id="Ax1.T27.28.28.28.4.m1.1a"><mn id="Ax1.T27.28.28.28.4.m1.1.1" xref="Ax1.T27.28.28.28.4.m1.1.1.cmml">0.53</mn><annotation-xml encoding="MathML-Content" id="Ax1.T27.28.28.28.4.m1.1b"><cn type="float" id="Ax1.T27.28.28.28.4.m1.1.1.cmml" xref="Ax1.T27.28.28.28.4.m1.1.1">0.53</cn></annotation-xml><annotation encoding="application/x-tex" id="Ax1.T27.28.28.28.4.m1.1c">0.53</annotation></semantics></math></td>
<td id="Ax1.T27.29.29.29.5" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="Ax1.T27.29.29.29.5.m1.1" class="ltx_Math" alttext="0.582" display="inline"><semantics id="Ax1.T27.29.29.29.5.m1.1a"><mn id="Ax1.T27.29.29.29.5.m1.1.1" xref="Ax1.T27.29.29.29.5.m1.1.1.cmml">0.582</mn><annotation-xml encoding="MathML-Content" id="Ax1.T27.29.29.29.5.m1.1b"><cn type="float" id="Ax1.T27.29.29.29.5.m1.1.1.cmml" xref="Ax1.T27.29.29.29.5.m1.1.1">0.582</cn></annotation-xml><annotation encoding="application/x-tex" id="Ax1.T27.29.29.29.5.m1.1c">0.582</annotation></semantics></math></td>
<td id="Ax1.T27.30.30.30.6" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="Ax1.T27.30.30.30.6.m1.1" class="ltx_Math" alttext="0.235" display="inline"><semantics id="Ax1.T27.30.30.30.6.m1.1a"><mn id="Ax1.T27.30.30.30.6.m1.1.1" xref="Ax1.T27.30.30.30.6.m1.1.1.cmml">0.235</mn><annotation-xml encoding="MathML-Content" id="Ax1.T27.30.30.30.6.m1.1b"><cn type="float" id="Ax1.T27.30.30.30.6.m1.1.1.cmml" xref="Ax1.T27.30.30.30.6.m1.1.1">0.235</cn></annotation-xml><annotation encoding="application/x-tex" id="Ax1.T27.30.30.30.6.m1.1c">0.235</annotation></semantics></math></td>
<td id="Ax1.T27.31.31.31.7" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="Ax1.T27.31.31.31.7.m1.1" class="ltx_Math" alttext="0.455" display="inline"><semantics id="Ax1.T27.31.31.31.7.m1.1a"><mn id="Ax1.T27.31.31.31.7.m1.1.1" xref="Ax1.T27.31.31.31.7.m1.1.1.cmml">0.455</mn><annotation-xml encoding="MathML-Content" id="Ax1.T27.31.31.31.7.m1.1b"><cn type="float" id="Ax1.T27.31.31.31.7.m1.1.1.cmml" xref="Ax1.T27.31.31.31.7.m1.1.1">0.455</cn></annotation-xml><annotation encoding="application/x-tex" id="Ax1.T27.31.31.31.7.m1.1c">0.455</annotation></semantics></math></td>
<td id="Ax1.T27.32.32.32.8" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="Ax1.T27.32.32.32.8.m1.1" class="ltx_Math" alttext="0.264" display="inline"><semantics id="Ax1.T27.32.32.32.8.m1.1a"><mn id="Ax1.T27.32.32.32.8.m1.1.1" xref="Ax1.T27.32.32.32.8.m1.1.1.cmml">0.264</mn><annotation-xml encoding="MathML-Content" id="Ax1.T27.32.32.32.8.m1.1b"><cn type="float" id="Ax1.T27.32.32.32.8.m1.1.1.cmml" xref="Ax1.T27.32.32.32.8.m1.1.1">0.264</cn></annotation-xml><annotation encoding="application/x-tex" id="Ax1.T27.32.32.32.8.m1.1c">0.264</annotation></semantics></math></td>
</tr>
<tr id="Ax1.T27.40.40.40" class="ltx_tr">
<td id="Ax1.T27.40.40.40.9" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;">RCL</td>
<td id="Ax1.T27.33.33.33.1" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="Ax1.T27.33.33.33.1.m1.1" class="ltx_Math" alttext="0.52" display="inline"><semantics id="Ax1.T27.33.33.33.1.m1.1a"><mn id="Ax1.T27.33.33.33.1.m1.1.1" xref="Ax1.T27.33.33.33.1.m1.1.1.cmml">0.52</mn><annotation-xml encoding="MathML-Content" id="Ax1.T27.33.33.33.1.m1.1b"><cn type="float" id="Ax1.T27.33.33.33.1.m1.1.1.cmml" xref="Ax1.T27.33.33.33.1.m1.1.1">0.52</cn></annotation-xml><annotation encoding="application/x-tex" id="Ax1.T27.33.33.33.1.m1.1c">0.52</annotation></semantics></math></td>
<td id="Ax1.T27.34.34.34.2" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="Ax1.T27.34.34.34.2.m1.1" class="ltx_Math" alttext="0.52" display="inline"><semantics id="Ax1.T27.34.34.34.2.m1.1a"><mn id="Ax1.T27.34.34.34.2.m1.1.1" xref="Ax1.T27.34.34.34.2.m1.1.1.cmml">0.52</mn><annotation-xml encoding="MathML-Content" id="Ax1.T27.34.34.34.2.m1.1b"><cn type="float" id="Ax1.T27.34.34.34.2.m1.1.1.cmml" xref="Ax1.T27.34.34.34.2.m1.1.1">0.52</cn></annotation-xml><annotation encoding="application/x-tex" id="Ax1.T27.34.34.34.2.m1.1c">0.52</annotation></semantics></math></td>
<td id="Ax1.T27.35.35.35.3" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="Ax1.T27.35.35.35.3.m1.1" class="ltx_Math" alttext="0.34" display="inline"><semantics id="Ax1.T27.35.35.35.3.m1.1a"><mn id="Ax1.T27.35.35.35.3.m1.1.1" xref="Ax1.T27.35.35.35.3.m1.1.1.cmml">0.34</mn><annotation-xml encoding="MathML-Content" id="Ax1.T27.35.35.35.3.m1.1b"><cn type="float" id="Ax1.T27.35.35.35.3.m1.1.1.cmml" xref="Ax1.T27.35.35.35.3.m1.1.1">0.34</cn></annotation-xml><annotation encoding="application/x-tex" id="Ax1.T27.35.35.35.3.m1.1c">0.34</annotation></semantics></math></td>
<td id="Ax1.T27.36.36.36.4" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="Ax1.T27.36.36.36.4.m1.1" class="ltx_Math" alttext="0.53" display="inline"><semantics id="Ax1.T27.36.36.36.4.m1.1a"><mn id="Ax1.T27.36.36.36.4.m1.1.1" xref="Ax1.T27.36.36.36.4.m1.1.1.cmml">0.53</mn><annotation-xml encoding="MathML-Content" id="Ax1.T27.36.36.36.4.m1.1b"><cn type="float" id="Ax1.T27.36.36.36.4.m1.1.1.cmml" xref="Ax1.T27.36.36.36.4.m1.1.1">0.53</cn></annotation-xml><annotation encoding="application/x-tex" id="Ax1.T27.36.36.36.4.m1.1c">0.53</annotation></semantics></math></td>
<td id="Ax1.T27.37.37.37.5" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="Ax1.T27.37.37.37.5.m1.1" class="ltx_Math" alttext="0.60" display="inline"><semantics id="Ax1.T27.37.37.37.5.m1.1a"><mn id="Ax1.T27.37.37.37.5.m1.1.1" xref="Ax1.T27.37.37.37.5.m1.1.1.cmml">0.60</mn><annotation-xml encoding="MathML-Content" id="Ax1.T27.37.37.37.5.m1.1b"><cn type="float" id="Ax1.T27.37.37.37.5.m1.1.1.cmml" xref="Ax1.T27.37.37.37.5.m1.1.1">0.60</cn></annotation-xml><annotation encoding="application/x-tex" id="Ax1.T27.37.37.37.5.m1.1c">0.60</annotation></semantics></math></td>
<td id="Ax1.T27.38.38.38.6" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="Ax1.T27.38.38.38.6.m1.1" class="ltx_Math" alttext="0.47" display="inline"><semantics id="Ax1.T27.38.38.38.6.m1.1a"><mn id="Ax1.T27.38.38.38.6.m1.1.1" xref="Ax1.T27.38.38.38.6.m1.1.1.cmml">0.47</mn><annotation-xml encoding="MathML-Content" id="Ax1.T27.38.38.38.6.m1.1b"><cn type="float" id="Ax1.T27.38.38.38.6.m1.1.1.cmml" xref="Ax1.T27.38.38.38.6.m1.1.1">0.47</cn></annotation-xml><annotation encoding="application/x-tex" id="Ax1.T27.38.38.38.6.m1.1c">0.47</annotation></semantics></math></td>
<td id="Ax1.T27.39.39.39.7" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="Ax1.T27.39.39.39.7.m1.1" class="ltx_Math" alttext="0.35" display="inline"><semantics id="Ax1.T27.39.39.39.7.m1.1a"><mn id="Ax1.T27.39.39.39.7.m1.1.1" xref="Ax1.T27.39.39.39.7.m1.1.1.cmml">0.35</mn><annotation-xml encoding="MathML-Content" id="Ax1.T27.39.39.39.7.m1.1b"><cn type="float" id="Ax1.T27.39.39.39.7.m1.1.1.cmml" xref="Ax1.T27.39.39.39.7.m1.1.1">0.35</cn></annotation-xml><annotation encoding="application/x-tex" id="Ax1.T27.39.39.39.7.m1.1c">0.35</annotation></semantics></math></td>
<td id="Ax1.T27.40.40.40.8" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="Ax1.T27.40.40.40.8.m1.1" class="ltx_Math" alttext="0.40" display="inline"><semantics id="Ax1.T27.40.40.40.8.m1.1a"><mn id="Ax1.T27.40.40.40.8.m1.1.1" xref="Ax1.T27.40.40.40.8.m1.1.1.cmml">0.40</mn><annotation-xml encoding="MathML-Content" id="Ax1.T27.40.40.40.8.m1.1b"><cn type="float" id="Ax1.T27.40.40.40.8.m1.1.1.cmml" xref="Ax1.T27.40.40.40.8.m1.1.1">0.40</cn></annotation-xml><annotation encoding="application/x-tex" id="Ax1.T27.40.40.40.8.m1.1c">0.40</annotation></semantics></math></td>
</tr>
<tr id="Ax1.T27.48.48.48" class="ltx_tr">
<td id="Ax1.T27.48.48.48.9" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:2.0pt;padding-right:2.0pt;">FT</td>
<td id="Ax1.T27.41.41.41.1" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="Ax1.T27.41.41.41.1.m1.1" class="ltx_Math" alttext="0.52" display="inline"><semantics id="Ax1.T27.41.41.41.1.m1.1a"><mn id="Ax1.T27.41.41.41.1.m1.1.1" xref="Ax1.T27.41.41.41.1.m1.1.1.cmml">0.52</mn><annotation-xml encoding="MathML-Content" id="Ax1.T27.41.41.41.1.m1.1b"><cn type="float" id="Ax1.T27.41.41.41.1.m1.1.1.cmml" xref="Ax1.T27.41.41.41.1.m1.1.1">0.52</cn></annotation-xml><annotation encoding="application/x-tex" id="Ax1.T27.41.41.41.1.m1.1c">0.52</annotation></semantics></math></td>
<td id="Ax1.T27.42.42.42.2" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="Ax1.T27.42.42.42.2.m1.1" class="ltx_Math" alttext="0.71" display="inline"><semantics id="Ax1.T27.42.42.42.2.m1.1a"><mn id="Ax1.T27.42.42.42.2.m1.1.1" xref="Ax1.T27.42.42.42.2.m1.1.1.cmml">0.71</mn><annotation-xml encoding="MathML-Content" id="Ax1.T27.42.42.42.2.m1.1b"><cn type="float" id="Ax1.T27.42.42.42.2.m1.1.1.cmml" xref="Ax1.T27.42.42.42.2.m1.1.1">0.71</cn></annotation-xml><annotation encoding="application/x-tex" id="Ax1.T27.42.42.42.2.m1.1c">0.71</annotation></semantics></math></td>
<td id="Ax1.T27.43.43.43.3" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="Ax1.T27.43.43.43.3.m1.1" class="ltx_Math" alttext="0.60" display="inline"><semantics id="Ax1.T27.43.43.43.3.m1.1a"><mn id="Ax1.T27.43.43.43.3.m1.1.1" xref="Ax1.T27.43.43.43.3.m1.1.1.cmml">0.60</mn><annotation-xml encoding="MathML-Content" id="Ax1.T27.43.43.43.3.m1.1b"><cn type="float" id="Ax1.T27.43.43.43.3.m1.1.1.cmml" xref="Ax1.T27.43.43.43.3.m1.1.1">0.60</cn></annotation-xml><annotation encoding="application/x-tex" id="Ax1.T27.43.43.43.3.m1.1c">0.60</annotation></semantics></math></td>
<td id="Ax1.T27.44.44.44.4" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="Ax1.T27.44.44.44.4.m1.1" class="ltx_Math" alttext="0.58" display="inline"><semantics id="Ax1.T27.44.44.44.4.m1.1a"><mn id="Ax1.T27.44.44.44.4.m1.1.1" xref="Ax1.T27.44.44.44.4.m1.1.1.cmml">0.58</mn><annotation-xml encoding="MathML-Content" id="Ax1.T27.44.44.44.4.m1.1b"><cn type="float" id="Ax1.T27.44.44.44.4.m1.1.1.cmml" xref="Ax1.T27.44.44.44.4.m1.1.1">0.58</cn></annotation-xml><annotation encoding="application/x-tex" id="Ax1.T27.44.44.44.4.m1.1c">0.58</annotation></semantics></math></td>
<td id="Ax1.T27.45.45.45.5" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="Ax1.T27.45.45.45.5.m1.1" class="ltx_Math" alttext="0.79" display="inline"><semantics id="Ax1.T27.45.45.45.5.m1.1a"><mn id="Ax1.T27.45.45.45.5.m1.1.1" xref="Ax1.T27.45.45.45.5.m1.1.1.cmml">0.79</mn><annotation-xml encoding="MathML-Content" id="Ax1.T27.45.45.45.5.m1.1b"><cn type="float" id="Ax1.T27.45.45.45.5.m1.1.1.cmml" xref="Ax1.T27.45.45.45.5.m1.1.1">0.79</cn></annotation-xml><annotation encoding="application/x-tex" id="Ax1.T27.45.45.45.5.m1.1c">0.79</annotation></semantics></math></td>
<td id="Ax1.T27.46.46.46.6" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="Ax1.T27.46.46.46.6.m1.1" class="ltx_Math" alttext="0.44" display="inline"><semantics id="Ax1.T27.46.46.46.6.m1.1a"><mn id="Ax1.T27.46.46.46.6.m1.1.1" xref="Ax1.T27.46.46.46.6.m1.1.1.cmml">0.44</mn><annotation-xml encoding="MathML-Content" id="Ax1.T27.46.46.46.6.m1.1b"><cn type="float" id="Ax1.T27.46.46.46.6.m1.1.1.cmml" xref="Ax1.T27.46.46.46.6.m1.1.1">0.44</cn></annotation-xml><annotation encoding="application/x-tex" id="Ax1.T27.46.46.46.6.m1.1c">0.44</annotation></semantics></math></td>
<td id="Ax1.T27.47.47.47.7" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="Ax1.T27.47.47.47.7.m1.1" class="ltx_Math" alttext="0.63" display="inline"><semantics id="Ax1.T27.47.47.47.7.m1.1a"><mn id="Ax1.T27.47.47.47.7.m1.1.1" xref="Ax1.T27.47.47.47.7.m1.1.1.cmml">0.63</mn><annotation-xml encoding="MathML-Content" id="Ax1.T27.47.47.47.7.m1.1b"><cn type="float" id="Ax1.T27.47.47.47.7.m1.1.1.cmml" xref="Ax1.T27.47.47.47.7.m1.1.1">0.63</cn></annotation-xml><annotation encoding="application/x-tex" id="Ax1.T27.47.47.47.7.m1.1c">0.63</annotation></semantics></math></td>
<td id="Ax1.T27.48.48.48.8" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="Ax1.T27.48.48.48.8.m1.1" class="ltx_Math" alttext="0.28" display="inline"><semantics id="Ax1.T27.48.48.48.8.m1.1a"><mn id="Ax1.T27.48.48.48.8.m1.1.1" xref="Ax1.T27.48.48.48.8.m1.1.1.cmml">0.28</mn><annotation-xml encoding="MathML-Content" id="Ax1.T27.48.48.48.8.m1.1b"><cn type="float" id="Ax1.T27.48.48.48.8.m1.1.1.cmml" xref="Ax1.T27.48.48.48.8.m1.1.1">0.28</cn></annotation-xml><annotation encoding="application/x-tex" id="Ax1.T27.48.48.48.8.m1.1c">0.28</annotation></semantics></math></td>
</tr>
<tr id="Ax1.T27.64.64.66" class="ltx_tr">
<td id="Ax1.T27.64.64.66.1" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;">FT+Re</td>
<td id="Ax1.T27.64.64.66.2" class="ltx_td" style="padding-left:2.0pt;padding-right:2.0pt;"></td>
<td id="Ax1.T27.64.64.66.3" class="ltx_td" style="padding-left:2.0pt;padding-right:2.0pt;"></td>
<td id="Ax1.T27.64.64.66.4" class="ltx_td" style="padding-left:2.0pt;padding-right:2.0pt;"></td>
<td id="Ax1.T27.64.64.66.5" class="ltx_td" style="padding-left:2.0pt;padding-right:2.0pt;"></td>
<td id="Ax1.T27.64.64.66.6" class="ltx_td" style="padding-left:2.0pt;padding-right:2.0pt;"></td>
<td id="Ax1.T27.64.64.66.7" class="ltx_td" style="padding-left:2.0pt;padding-right:2.0pt;"></td>
<td id="Ax1.T27.64.64.66.8" class="ltx_td" style="padding-left:2.0pt;padding-right:2.0pt;"></td>
<td id="Ax1.T27.64.64.66.9" class="ltx_td" style="padding-left:2.0pt;padding-right:2.0pt;"></td>
</tr>
<tr id="Ax1.T27.56.56.56" class="ltx_tr">
<td id="Ax1.T27.56.56.56.9" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;">MT</td>
<td id="Ax1.T27.49.49.49.1" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="Ax1.T27.49.49.49.1.m1.1" class="ltx_Math" alttext="0.44" display="inline"><semantics id="Ax1.T27.49.49.49.1.m1.1a"><mn id="Ax1.T27.49.49.49.1.m1.1.1" xref="Ax1.T27.49.49.49.1.m1.1.1.cmml">0.44</mn><annotation-xml encoding="MathML-Content" id="Ax1.T27.49.49.49.1.m1.1b"><cn type="float" id="Ax1.T27.49.49.49.1.m1.1.1.cmml" xref="Ax1.T27.49.49.49.1.m1.1.1">0.44</cn></annotation-xml><annotation encoding="application/x-tex" id="Ax1.T27.49.49.49.1.m1.1c">0.44</annotation></semantics></math></td>
<td id="Ax1.T27.50.50.50.2" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="Ax1.T27.50.50.50.2.m1.1" class="ltx_Math" alttext="0.68" display="inline"><semantics id="Ax1.T27.50.50.50.2.m1.1a"><mn id="Ax1.T27.50.50.50.2.m1.1.1" xref="Ax1.T27.50.50.50.2.m1.1.1.cmml">0.68</mn><annotation-xml encoding="MathML-Content" id="Ax1.T27.50.50.50.2.m1.1b"><cn type="float" id="Ax1.T27.50.50.50.2.m1.1.1.cmml" xref="Ax1.T27.50.50.50.2.m1.1.1">0.68</cn></annotation-xml><annotation encoding="application/x-tex" id="Ax1.T27.50.50.50.2.m1.1c">0.68</annotation></semantics></math></td>
<td id="Ax1.T27.51.51.51.3" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="Ax1.T27.51.51.51.3.m1.1" class="ltx_Math" alttext="0.44" display="inline"><semantics id="Ax1.T27.51.51.51.3.m1.1a"><mn id="Ax1.T27.51.51.51.3.m1.1.1" xref="Ax1.T27.51.51.51.3.m1.1.1.cmml">0.44</mn><annotation-xml encoding="MathML-Content" id="Ax1.T27.51.51.51.3.m1.1b"><cn type="float" id="Ax1.T27.51.51.51.3.m1.1.1.cmml" xref="Ax1.T27.51.51.51.3.m1.1.1">0.44</cn></annotation-xml><annotation encoding="application/x-tex" id="Ax1.T27.51.51.51.3.m1.1c">0.44</annotation></semantics></math></td>
<td id="Ax1.T27.52.52.52.4" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="Ax1.T27.52.52.52.4.m1.1" class="ltx_Math" alttext="0.60" display="inline"><semantics id="Ax1.T27.52.52.52.4.m1.1a"><mn id="Ax1.T27.52.52.52.4.m1.1.1" xref="Ax1.T27.52.52.52.4.m1.1.1.cmml">0.60</mn><annotation-xml encoding="MathML-Content" id="Ax1.T27.52.52.52.4.m1.1b"><cn type="float" id="Ax1.T27.52.52.52.4.m1.1.1.cmml" xref="Ax1.T27.52.52.52.4.m1.1.1">0.60</cn></annotation-xml><annotation encoding="application/x-tex" id="Ax1.T27.52.52.52.4.m1.1c">0.60</annotation></semantics></math></td>
<td id="Ax1.T27.53.53.53.5" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="Ax1.T27.53.53.53.5.m1.1" class="ltx_Math" alttext="0.72" display="inline"><semantics id="Ax1.T27.53.53.53.5.m1.1a"><mn id="Ax1.T27.53.53.53.5.m1.1.1" xref="Ax1.T27.53.53.53.5.m1.1.1.cmml">0.72</mn><annotation-xml encoding="MathML-Content" id="Ax1.T27.53.53.53.5.m1.1b"><cn type="float" id="Ax1.T27.53.53.53.5.m1.1.1.cmml" xref="Ax1.T27.53.53.53.5.m1.1.1">0.72</cn></annotation-xml><annotation encoding="application/x-tex" id="Ax1.T27.53.53.53.5.m1.1c">0.72</annotation></semantics></math></td>
<td id="Ax1.T27.54.54.54.6" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="Ax1.T27.54.54.54.6.m1.1" class="ltx_Math" alttext="0.33" display="inline"><semantics id="Ax1.T27.54.54.54.6.m1.1a"><mn id="Ax1.T27.54.54.54.6.m1.1.1" xref="Ax1.T27.54.54.54.6.m1.1.1.cmml">0.33</mn><annotation-xml encoding="MathML-Content" id="Ax1.T27.54.54.54.6.m1.1b"><cn type="float" id="Ax1.T27.54.54.54.6.m1.1.1.cmml" xref="Ax1.T27.54.54.54.6.m1.1.1">0.33</cn></annotation-xml><annotation encoding="application/x-tex" id="Ax1.T27.54.54.54.6.m1.1c">0.33</annotation></semantics></math></td>
<td id="Ax1.T27.55.55.55.7" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="Ax1.T27.55.55.55.7.m1.1" class="ltx_Math" alttext="0.57" display="inline"><semantics id="Ax1.T27.55.55.55.7.m1.1a"><mn id="Ax1.T27.55.55.55.7.m1.1.1" xref="Ax1.T27.55.55.55.7.m1.1.1.cmml">0.57</mn><annotation-xml encoding="MathML-Content" id="Ax1.T27.55.55.55.7.m1.1b"><cn type="float" id="Ax1.T27.55.55.55.7.m1.1.1.cmml" xref="Ax1.T27.55.55.55.7.m1.1.1">0.57</cn></annotation-xml><annotation encoding="application/x-tex" id="Ax1.T27.55.55.55.7.m1.1c">0.57</annotation></semantics></math></td>
<td id="Ax1.T27.56.56.56.8" class="ltx_td ltx_align_center" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="Ax1.T27.56.56.56.8.m1.1" class="ltx_Math" alttext="0.39" display="inline"><semantics id="Ax1.T27.56.56.56.8.m1.1a"><mn id="Ax1.T27.56.56.56.8.m1.1.1" xref="Ax1.T27.56.56.56.8.m1.1.1.cmml">0.39</mn><annotation-xml encoding="MathML-Content" id="Ax1.T27.56.56.56.8.m1.1b"><cn type="float" id="Ax1.T27.56.56.56.8.m1.1.1.cmml" xref="Ax1.T27.56.56.56.8.m1.1.1">0.39</cn></annotation-xml><annotation encoding="application/x-tex" id="Ax1.T27.56.56.56.8.m1.1c">0.39</annotation></semantics></math></td>
</tr>
<tr id="Ax1.T27.64.64.64" class="ltx_tr">
<td id="Ax1.T27.64.64.64.9" class="ltx_td ltx_align_center ltx_border_b" style="padding-left:2.0pt;padding-right:2.0pt;">MT+Re</td>
<td id="Ax1.T27.57.57.57.1" class="ltx_td ltx_align_center ltx_border_b" style="padding-left:2.0pt;padding-right:2.0pt;"><span id="Ax1.T27.57.57.57.1.1" class="ltx_text ltx_markedasmath ltx_font_bold">0.61</span></td>
<td id="Ax1.T27.58.58.58.2" class="ltx_td ltx_align_center ltx_border_b" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="Ax1.T27.58.58.58.2.m1.1" class="ltx_Math" alttext="0.62" display="inline"><semantics id="Ax1.T27.58.58.58.2.m1.1a"><mn id="Ax1.T27.58.58.58.2.m1.1.1" xref="Ax1.T27.58.58.58.2.m1.1.1.cmml">0.62</mn><annotation-xml encoding="MathML-Content" id="Ax1.T27.58.58.58.2.m1.1b"><cn type="float" id="Ax1.T27.58.58.58.2.m1.1.1.cmml" xref="Ax1.T27.58.58.58.2.m1.1.1">0.62</cn></annotation-xml><annotation encoding="application/x-tex" id="Ax1.T27.58.58.58.2.m1.1c">0.62</annotation></semantics></math></td>
<td id="Ax1.T27.59.59.59.3" class="ltx_td ltx_align_center ltx_border_b" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="Ax1.T27.59.59.59.3.m1.1" class="ltx_Math" alttext="0.50" display="inline"><semantics id="Ax1.T27.59.59.59.3.m1.1a"><mn id="Ax1.T27.59.59.59.3.m1.1.1" xref="Ax1.T27.59.59.59.3.m1.1.1.cmml">0.50</mn><annotation-xml encoding="MathML-Content" id="Ax1.T27.59.59.59.3.m1.1b"><cn type="float" id="Ax1.T27.59.59.59.3.m1.1.1.cmml" xref="Ax1.T27.59.59.59.3.m1.1.1">0.50</cn></annotation-xml><annotation encoding="application/x-tex" id="Ax1.T27.59.59.59.3.m1.1c">0.50</annotation></semantics></math></td>
<td id="Ax1.T27.60.60.60.4" class="ltx_td ltx_align_center ltx_border_b" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="Ax1.T27.60.60.60.4.m1.1" class="ltx_Math" alttext="0.56" display="inline"><semantics id="Ax1.T27.60.60.60.4.m1.1a"><mn id="Ax1.T27.60.60.60.4.m1.1.1" xref="Ax1.T27.60.60.60.4.m1.1.1.cmml">0.56</mn><annotation-xml encoding="MathML-Content" id="Ax1.T27.60.60.60.4.m1.1b"><cn type="float" id="Ax1.T27.60.60.60.4.m1.1.1.cmml" xref="Ax1.T27.60.60.60.4.m1.1.1">0.56</cn></annotation-xml><annotation encoding="application/x-tex" id="Ax1.T27.60.60.60.4.m1.1c">0.56</annotation></semantics></math></td>
<td id="Ax1.T27.61.61.61.5" class="ltx_td ltx_align_center ltx_border_b" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="Ax1.T27.61.61.61.5.m1.1" class="ltx_Math" alttext="0.72" display="inline"><semantics id="Ax1.T27.61.61.61.5.m1.1a"><mn id="Ax1.T27.61.61.61.5.m1.1.1" xref="Ax1.T27.61.61.61.5.m1.1.1.cmml">0.72</mn><annotation-xml encoding="MathML-Content" id="Ax1.T27.61.61.61.5.m1.1b"><cn type="float" id="Ax1.T27.61.61.61.5.m1.1.1.cmml" xref="Ax1.T27.61.61.61.5.m1.1.1">0.72</cn></annotation-xml><annotation encoding="application/x-tex" id="Ax1.T27.61.61.61.5.m1.1c">0.72</annotation></semantics></math></td>
<td id="Ax1.T27.62.62.62.6" class="ltx_td ltx_align_center ltx_border_b" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="Ax1.T27.62.62.62.6.m1.1" class="ltx_Math" alttext="0.69" display="inline"><semantics id="Ax1.T27.62.62.62.6.m1.1a"><mn id="Ax1.T27.62.62.62.6.m1.1.1" xref="Ax1.T27.62.62.62.6.m1.1.1.cmml">0.69</mn><annotation-xml encoding="MathML-Content" id="Ax1.T27.62.62.62.6.m1.1b"><cn type="float" id="Ax1.T27.62.62.62.6.m1.1.1.cmml" xref="Ax1.T27.62.62.62.6.m1.1.1">0.69</cn></annotation-xml><annotation encoding="application/x-tex" id="Ax1.T27.62.62.62.6.m1.1c">0.69</annotation></semantics></math></td>
<td id="Ax1.T27.63.63.63.7" class="ltx_td ltx_align_center ltx_border_b" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="Ax1.T27.63.63.63.7.m1.1" class="ltx_Math" alttext="0.57" display="inline"><semantics id="Ax1.T27.63.63.63.7.m1.1a"><mn id="Ax1.T27.63.63.63.7.m1.1.1" xref="Ax1.T27.63.63.63.7.m1.1.1.cmml">0.57</mn><annotation-xml encoding="MathML-Content" id="Ax1.T27.63.63.63.7.m1.1b"><cn type="float" id="Ax1.T27.63.63.63.7.m1.1.1.cmml" xref="Ax1.T27.63.63.63.7.m1.1.1">0.57</cn></annotation-xml><annotation encoding="application/x-tex" id="Ax1.T27.63.63.63.7.m1.1c">0.57</annotation></semantics></math></td>
<td id="Ax1.T27.64.64.64.8" class="ltx_td ltx_align_center ltx_border_b" style="padding-left:2.0pt;padding-right:2.0pt;"><math id="Ax1.T27.64.64.64.8.m1.1" class="ltx_Math" alttext="0.40" display="inline"><semantics id="Ax1.T27.64.64.64.8.m1.1a"><mn id="Ax1.T27.64.64.64.8.m1.1.1" xref="Ax1.T27.64.64.64.8.m1.1.1.cmml">0.40</mn><annotation-xml encoding="MathML-Content" id="Ax1.T27.64.64.64.8.m1.1b"><cn type="float" id="Ax1.T27.64.64.64.8.m1.1.1.cmml" xref="Ax1.T27.64.64.64.8.m1.1.1">0.40</cn></annotation-xml><annotation encoding="application/x-tex" id="Ax1.T27.64.64.64.8.m1.1c">0.40</annotation></semantics></math></td>
</tr>
</table>
</span></div>
</figure>
</section>
</section>
<section id="Ax1.SS5" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">.5 </span>Detailed experiment results of Open-Compass toolkit</h3>

<figure id="Ax1.T28" class="ltx_table">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table">Table 28: </span>Detailed results of open-compass toolkit of LLaMA-13b-chat.</figcaption>
<table id="Ax1.T28.1" class="ltx_tabular">
<tr id="Ax1.T28.1.1" class="ltx_tr">
<td id="Ax1.T28.1.1.1" class="ltx_td ltx_align_left ltx_border_tt">dataset</td>
<td id="Ax1.T28.1.1.2" class="ltx_td ltx_align_left ltx_border_tt">metric</td>
<td id="Ax1.T28.1.1.3" class="ltx_td ltx_align_left ltx_border_tt">mode</td>
<td id="Ax1.T28.1.1.4" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_tt">performance</td>
</tr>
<tr id="Ax1.T28.1.2" class="ltx_tr">
<td id="Ax1.T28.1.2.1" class="ltx_td ltx_align_left ltx_border_t">lukaemon_mmlu_college_biology</td>
<td id="Ax1.T28.1.2.2" class="ltx_td ltx_align_left ltx_border_t">accuracy</td>
<td id="Ax1.T28.1.2.3" class="ltx_td ltx_align_left ltx_border_t">ppl</td>
<td id="Ax1.T28.1.2.4" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_t">58.33</td>
</tr>
<tr id="Ax1.T28.1.3" class="ltx_tr">
<td id="Ax1.T28.1.3.1" class="ltx_td ltx_align_left">lukaemon_mmlu_college_chemistry</td>
<td id="Ax1.T28.1.3.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T28.1.3.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T28.1.3.4" class="ltx_td ltx_nopad_r ltx_align_left">38</td>
</tr>
<tr id="Ax1.T28.1.4" class="ltx_tr">
<td id="Ax1.T28.1.4.1" class="ltx_td ltx_align_left">lukaemon_mmlu_college_computer_science</td>
<td id="Ax1.T28.1.4.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T28.1.4.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T28.1.4.4" class="ltx_td ltx_nopad_r ltx_align_left">47</td>
</tr>
<tr id="Ax1.T28.1.5" class="ltx_tr">
<td id="Ax1.T28.1.5.1" class="ltx_td ltx_align_left">lukaemon_mmlu_college_mathematics</td>
<td id="Ax1.T28.1.5.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T28.1.5.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T28.1.5.4" class="ltx_td ltx_nopad_r ltx_align_left">32</td>
</tr>
<tr id="Ax1.T28.1.6" class="ltx_tr">
<td id="Ax1.T28.1.6.1" class="ltx_td ltx_align_left">lukaemon_mmlu_college_physics</td>
<td id="Ax1.T28.1.6.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T28.1.6.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T28.1.6.4" class="ltx_td ltx_nopad_r ltx_align_left">30.39</td>
</tr>
<tr id="Ax1.T28.1.7" class="ltx_tr">
<td id="Ax1.T28.1.7.1" class="ltx_td ltx_align_left">lukaemon_mmlu_electrical_engineering</td>
<td id="Ax1.T28.1.7.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T28.1.7.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T28.1.7.4" class="ltx_td ltx_nopad_r ltx_align_left">50.34</td>
</tr>
<tr id="Ax1.T28.1.8" class="ltx_tr">
<td id="Ax1.T28.1.8.1" class="ltx_td ltx_align_left">lukaemon_mmlu_astronomy</td>
<td id="Ax1.T28.1.8.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T28.1.8.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T28.1.8.4" class="ltx_td ltx_nopad_r ltx_align_left">54.61</td>
</tr>
<tr id="Ax1.T28.1.9" class="ltx_tr">
<td id="Ax1.T28.1.9.1" class="ltx_td ltx_align_left">lukaemon_mmlu_anatomy</td>
<td id="Ax1.T28.1.9.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T28.1.9.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T28.1.9.4" class="ltx_td ltx_nopad_r ltx_align_left">47.41</td>
</tr>
<tr id="Ax1.T28.1.10" class="ltx_tr">
<td id="Ax1.T28.1.10.1" class="ltx_td ltx_align_left">lukaemon_mmlu_abstract_algebra</td>
<td id="Ax1.T28.1.10.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T28.1.10.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T28.1.10.4" class="ltx_td ltx_nopad_r ltx_align_left">31</td>
</tr>
<tr id="Ax1.T28.1.11" class="ltx_tr">
<td id="Ax1.T28.1.11.1" class="ltx_td ltx_align_left">lukaemon_mmlu_machine_learning</td>
<td id="Ax1.T28.1.11.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T28.1.11.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T28.1.11.4" class="ltx_td ltx_nopad_r ltx_align_left">35.71</td>
</tr>
<tr id="Ax1.T28.1.12" class="ltx_tr">
<td id="Ax1.T28.1.12.1" class="ltx_td ltx_align_left">lukaemon_mmlu_clinical_knowledge</td>
<td id="Ax1.T28.1.12.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T28.1.12.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T28.1.12.4" class="ltx_td ltx_nopad_r ltx_align_left">58.87</td>
</tr>
<tr id="Ax1.T28.1.13" class="ltx_tr">
<td id="Ax1.T28.1.13.1" class="ltx_td ltx_align_left">lukaemon_mmlu_global_facts</td>
<td id="Ax1.T28.1.13.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T28.1.13.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T28.1.13.4" class="ltx_td ltx_nopad_r ltx_align_left">30</td>
</tr>
<tr id="Ax1.T28.1.14" class="ltx_tr">
<td id="Ax1.T28.1.14.1" class="ltx_td ltx_align_left">lukaemon_mmlu_management</td>
<td id="Ax1.T28.1.14.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T28.1.14.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T28.1.14.4" class="ltx_td ltx_nopad_r ltx_align_left">73.79</td>
</tr>
<tr id="Ax1.T28.1.15" class="ltx_tr">
<td id="Ax1.T28.1.15.1" class="ltx_td ltx_align_left">lukaemon_mmlu_nutrition</td>
<td id="Ax1.T28.1.15.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T28.1.15.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T28.1.15.4" class="ltx_td ltx_nopad_r ltx_align_left">62.09</td>
</tr>
<tr id="Ax1.T28.1.16" class="ltx_tr">
<td id="Ax1.T28.1.16.1" class="ltx_td ltx_align_left">lukaemon_mmlu_marketing</td>
<td id="Ax1.T28.1.16.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T28.1.16.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T28.1.16.4" class="ltx_td ltx_nopad_r ltx_align_left">78.63</td>
</tr>
<tr id="Ax1.T28.1.17" class="ltx_tr">
<td id="Ax1.T28.1.17.1" class="ltx_td ltx_align_left">lukaemon_mmlu_professional_accounting</td>
<td id="Ax1.T28.1.17.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T28.1.17.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T28.1.17.4" class="ltx_td ltx_nopad_r ltx_align_left">39.01</td>
</tr>
<tr id="Ax1.T28.1.18" class="ltx_tr">
<td id="Ax1.T28.1.18.1" class="ltx_td ltx_align_left">lukaemon_mmlu_high_school_geography</td>
<td id="Ax1.T28.1.18.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T28.1.18.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T28.1.18.4" class="ltx_td ltx_nopad_r ltx_align_left">70.2</td>
</tr>
<tr id="Ax1.T28.1.19" class="ltx_tr">
<td id="Ax1.T28.1.19.1" class="ltx_td ltx_align_left">lukaemon_mmlu_international_law</td>
<td id="Ax1.T28.1.19.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T28.1.19.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T28.1.19.4" class="ltx_td ltx_nopad_r ltx_align_left">77.69</td>
</tr>
<tr id="Ax1.T28.1.20" class="ltx_tr">
<td id="Ax1.T28.1.20.1" class="ltx_td ltx_align_left">lukaemon_mmlu_moral_scenarios</td>
<td id="Ax1.T28.1.20.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T28.1.20.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T28.1.20.4" class="ltx_td ltx_nopad_r ltx_align_left">30.84</td>
</tr>
<tr id="Ax1.T28.1.21" class="ltx_tr">
<td id="Ax1.T28.1.21.1" class="ltx_td ltx_align_left">lukaemon_mmlu_computer_security</td>
<td id="Ax1.T28.1.21.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T28.1.21.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T28.1.21.4" class="ltx_td ltx_nopad_r ltx_align_left">68</td>
</tr>
<tr id="Ax1.T28.1.22" class="ltx_tr">
<td id="Ax1.T28.1.22.1" class="ltx_td ltx_align_left">lukaemon_mmlu_high_school_microeconomics</td>
<td id="Ax1.T28.1.22.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T28.1.22.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T28.1.22.4" class="ltx_td ltx_nopad_r ltx_align_left">52.52</td>
</tr>
<tr id="Ax1.T28.1.23" class="ltx_tr">
<td id="Ax1.T28.1.23.1" class="ltx_td ltx_align_left">lukaemon_mmlu_professional_law</td>
<td id="Ax1.T28.1.23.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T28.1.23.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T28.1.23.4" class="ltx_td ltx_nopad_r ltx_align_left">38.98</td>
</tr>
<tr id="Ax1.T28.1.24" class="ltx_tr">
<td id="Ax1.T28.1.24.1" class="ltx_td ltx_align_left">lukaemon_mmlu_medical_genetics</td>
<td id="Ax1.T28.1.24.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T28.1.24.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T28.1.24.4" class="ltx_td ltx_nopad_r ltx_align_left">57</td>
</tr>
<tr id="Ax1.T28.1.25" class="ltx_tr">
<td id="Ax1.T28.1.25.1" class="ltx_td ltx_align_left">lukaemon_mmlu_professional_psychology</td>
<td id="Ax1.T28.1.25.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T28.1.25.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T28.1.25.4" class="ltx_td ltx_nopad_r ltx_align_left">54.25</td>
</tr>
<tr id="Ax1.T28.1.26" class="ltx_tr">
<td id="Ax1.T28.1.26.1" class="ltx_td ltx_align_left">lukaemon_mmlu_jurisprudence</td>
<td id="Ax1.T28.1.26.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T28.1.26.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T28.1.26.4" class="ltx_td ltx_nopad_r ltx_align_left">69.44</td>
</tr>
<tr id="Ax1.T28.1.27" class="ltx_tr">
<td id="Ax1.T28.1.27.1" class="ltx_td ltx_align_left">lukaemon_mmlu_world_religions</td>
<td id="Ax1.T28.1.27.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T28.1.27.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T28.1.27.4" class="ltx_td ltx_nopad_r ltx_align_left">71.93</td>
</tr>
<tr id="Ax1.T28.1.28" class="ltx_tr">
<td id="Ax1.T28.1.28.1" class="ltx_td ltx_align_left">lukaemon_mmlu_philosophy</td>
<td id="Ax1.T28.1.28.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T28.1.28.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T28.1.28.4" class="ltx_td ltx_nopad_r ltx_align_left">58.84</td>
</tr>
<tr id="Ax1.T28.1.29" class="ltx_tr">
<td id="Ax1.T28.1.29.1" class="ltx_td ltx_align_left">lukaemon_mmlu_virology</td>
<td id="Ax1.T28.1.29.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T28.1.29.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T28.1.29.4" class="ltx_td ltx_nopad_r ltx_align_left">48.19</td>
</tr>
<tr id="Ax1.T28.1.30" class="ltx_tr">
<td id="Ax1.T28.1.30.1" class="ltx_td ltx_align_left">lukaemon_mmlu_high_school_chemistry</td>
<td id="Ax1.T28.1.30.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T28.1.30.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T28.1.30.4" class="ltx_td ltx_nopad_r ltx_align_left">45.81</td>
</tr>
<tr id="Ax1.T28.1.31" class="ltx_tr">
<td id="Ax1.T28.1.31.1" class="ltx_td ltx_align_left">lukaemon_mmlu_public_relations</td>
<td id="Ax1.T28.1.31.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T28.1.31.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T28.1.31.4" class="ltx_td ltx_nopad_r ltx_align_left">66.36</td>
</tr>
<tr id="Ax1.T28.1.32" class="ltx_tr">
<td id="Ax1.T28.1.32.1" class="ltx_td ltx_align_left">lukaemon_mmlu_high_school_macroeconomics</td>
<td id="Ax1.T28.1.32.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T28.1.32.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T28.1.32.4" class="ltx_td ltx_nopad_r ltx_align_left">49.23</td>
</tr>
<tr id="Ax1.T28.1.33" class="ltx_tr">
<td id="Ax1.T28.1.33.1" class="ltx_td ltx_align_left">lukaemon_mmlu_human_sexuality</td>
<td id="Ax1.T28.1.33.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T28.1.33.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T28.1.33.4" class="ltx_td ltx_nopad_r ltx_align_left">61.83</td>
</tr>
<tr id="Ax1.T28.1.34" class="ltx_tr">
<td id="Ax1.T28.1.34.1" class="ltx_td ltx_align_left">lukaemon_mmlu_elementary_mathematics</td>
<td id="Ax1.T28.1.34.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T28.1.34.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T28.1.34.4" class="ltx_td ltx_nopad_r ltx_align_left">33.86</td>
</tr>
<tr id="Ax1.T28.1.35" class="ltx_tr">
<td id="Ax1.T28.1.35.1" class="ltx_td ltx_align_left">lukaemon_mmlu_high_school_physics</td>
<td id="Ax1.T28.1.35.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T28.1.35.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T28.1.35.4" class="ltx_td ltx_nopad_r ltx_align_left">33.77</td>
</tr>
<tr id="Ax1.T28.1.36" class="ltx_tr">
<td id="Ax1.T28.1.36.1" class="ltx_td ltx_align_left">lukaemon_mmlu_high_school_computer_science</td>
<td id="Ax1.T28.1.36.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T28.1.36.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T28.1.36.4" class="ltx_td ltx_nopad_r ltx_align_left">59</td>
</tr>
<tr id="Ax1.T28.1.37" class="ltx_tr">
<td id="Ax1.T28.1.37.1" class="ltx_td ltx_align_left">lukaemon_mmlu_high_school_european_history</td>
<td id="Ax1.T28.1.37.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T28.1.37.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T28.1.37.4" class="ltx_td ltx_nopad_r ltx_align_left">66.67</td>
</tr>
<tr id="Ax1.T28.1.38" class="ltx_tr">
<td id="Ax1.T28.1.38.1" class="ltx_td ltx_align_left">lukaemon_mmlu_business_ethics</td>
<td id="Ax1.T28.1.38.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T28.1.38.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T28.1.38.4" class="ltx_td ltx_nopad_r ltx_align_left">52</td>
</tr>
<tr id="Ax1.T28.1.39" class="ltx_tr">
<td id="Ax1.T28.1.39.1" class="ltx_td ltx_align_left">lukaemon_mmlu_moral_disputes</td>
<td id="Ax1.T28.1.39.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T28.1.39.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T28.1.39.4" class="ltx_td ltx_nopad_r ltx_align_left">60.98</td>
</tr>
<tr id="Ax1.T28.1.40" class="ltx_tr">
<td id="Ax1.T28.1.40.1" class="ltx_td ltx_align_left">lukaemon_mmlu_high_school_statistics</td>
<td id="Ax1.T28.1.40.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T28.1.40.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T28.1.40.4" class="ltx_td ltx_nopad_r ltx_align_left">39.81</td>
</tr>
<tr id="Ax1.T28.1.41" class="ltx_tr">
<td id="Ax1.T28.1.41.1" class="ltx_td ltx_align_left">lukaemon_mmlu_miscellaneous</td>
<td id="Ax1.T28.1.41.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T28.1.41.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T28.1.41.4" class="ltx_td ltx_nopad_r ltx_align_left">74.84</td>
</tr>
<tr id="Ax1.T28.1.42" class="ltx_tr">
<td id="Ax1.T28.1.42.1" class="ltx_td ltx_align_left">lukaemon_mmlu_formal_logic</td>
<td id="Ax1.T28.1.42.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T28.1.42.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T28.1.42.4" class="ltx_td ltx_nopad_r ltx_align_left">29.37</td>
</tr>
<tr id="Ax1.T28.1.43" class="ltx_tr">
<td id="Ax1.T28.1.43.1" class="ltx_td ltx_align_left">lukaemon_mmlu_high_school_government_and_politics</td>
<td id="Ax1.T28.1.43.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T28.1.43.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T28.1.43.4" class="ltx_td ltx_nopad_r ltx_align_left">79.27</td>
</tr>
<tr id="Ax1.T28.1.44" class="ltx_tr">
<td id="Ax1.T28.1.44.1" class="ltx_td ltx_align_left">lukaemon_mmlu_prehistory</td>
<td id="Ax1.T28.1.44.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T28.1.44.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T28.1.44.4" class="ltx_td ltx_nopad_r ltx_align_left">61.11</td>
</tr>
<tr id="Ax1.T28.1.45" class="ltx_tr">
<td id="Ax1.T28.1.45.1" class="ltx_td ltx_align_left">lukaemon_mmlu_security_studies</td>
<td id="Ax1.T28.1.45.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T28.1.45.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T28.1.45.4" class="ltx_td ltx_nopad_r ltx_align_left">64.08</td>
</tr>
<tr id="Ax1.T28.1.46" class="ltx_tr">
<td id="Ax1.T28.1.46.1" class="ltx_td ltx_align_left">lukaemon_mmlu_high_school_biology</td>
<td id="Ax1.T28.1.46.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T28.1.46.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T28.1.46.4" class="ltx_td ltx_nopad_r ltx_align_left">64.84</td>
</tr>
<tr id="Ax1.T28.1.47" class="ltx_tr">
<td id="Ax1.T28.1.47.1" class="ltx_td ltx_align_left">lukaemon_mmlu_logical_fallacies</td>
<td id="Ax1.T28.1.47.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T28.1.47.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T28.1.47.4" class="ltx_td ltx_nopad_r ltx_align_left">64.42</td>
</tr>
<tr id="Ax1.T28.1.48" class="ltx_tr">
<td id="Ax1.T28.1.48.1" class="ltx_td ltx_align_left">lukaemon_mmlu_high_school_world_history</td>
<td id="Ax1.T28.1.48.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T28.1.48.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T28.1.48.4" class="ltx_td ltx_nopad_r ltx_align_left">71.73</td>
</tr>
<tr id="Ax1.T28.1.49" class="ltx_tr">
<td id="Ax1.T28.1.49.1" class="ltx_td ltx_align_left">lukaemon_mmlu_professional_medicine</td>
<td id="Ax1.T28.1.49.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T28.1.49.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T28.1.49.4" class="ltx_td ltx_nopad_r ltx_align_left">50</td>
</tr>
<tr id="Ax1.T28.1.50" class="ltx_tr">
<td id="Ax1.T28.1.50.1" class="ltx_td ltx_align_left">lukaemon_mmlu_high_school_mathematics</td>
<td id="Ax1.T28.1.50.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T28.1.50.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T28.1.50.4" class="ltx_td ltx_nopad_r ltx_align_left">30.74</td>
</tr>
<tr id="Ax1.T28.1.51" class="ltx_tr">
<td id="Ax1.T28.1.51.1" class="ltx_td ltx_align_left">lukaemon_mmlu_college_medicine</td>
<td id="Ax1.T28.1.51.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T28.1.51.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T28.1.51.4" class="ltx_td ltx_nopad_r ltx_align_left">46.82</td>
</tr>
<tr id="Ax1.T28.1.52" class="ltx_tr">
<td id="Ax1.T28.1.52.1" class="ltx_td ltx_align_left">lukaemon_mmlu_high_school_us_history</td>
<td id="Ax1.T28.1.52.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T28.1.52.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T28.1.52.4" class="ltx_td ltx_nopad_r ltx_align_left">74.51</td>
</tr>
<tr id="Ax1.T28.1.53" class="ltx_tr">
<td id="Ax1.T28.1.53.1" class="ltx_td ltx_align_left">lukaemon_mmlu_sociology</td>
<td id="Ax1.T28.1.53.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T28.1.53.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T28.1.53.4" class="ltx_td ltx_nopad_r ltx_align_left">75.12</td>
</tr>
<tr id="Ax1.T28.1.54" class="ltx_tr">
<td id="Ax1.T28.1.54.1" class="ltx_td ltx_align_left">lukaemon_mmlu_econometrics</td>
<td id="Ax1.T28.1.54.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T28.1.54.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T28.1.54.4" class="ltx_td ltx_nopad_r ltx_align_left">31.58</td>
</tr>
<tr id="Ax1.T28.1.55" class="ltx_tr">
<td id="Ax1.T28.1.55.1" class="ltx_td ltx_align_left">lukaemon_mmlu_high_school_psychology</td>
<td id="Ax1.T28.1.55.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T28.1.55.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T28.1.55.4" class="ltx_td ltx_nopad_r ltx_align_left">73.94</td>
</tr>
<tr id="Ax1.T28.1.56" class="ltx_tr">
<td id="Ax1.T28.1.56.1" class="ltx_td ltx_align_left">lukaemon_mmlu_human_aging</td>
<td id="Ax1.T28.1.56.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T28.1.56.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T28.1.56.4" class="ltx_td ltx_nopad_r ltx_align_left">64.57</td>
</tr>
<tr id="Ax1.T28.1.57" class="ltx_tr">
<td id="Ax1.T28.1.57.1" class="ltx_td ltx_align_left">lukaemon_mmlu_us_foreign_policy</td>
<td id="Ax1.T28.1.57.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T28.1.57.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T28.1.57.4" class="ltx_td ltx_nopad_r ltx_align_left">81</td>
</tr>
<tr id="Ax1.T28.1.58" class="ltx_tr">
<td id="Ax1.T28.1.58.1" class="ltx_td ltx_align_left">lukaemon_mmlu_conceptual_physics</td>
<td id="Ax1.T28.1.58.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T28.1.58.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T28.1.58.4" class="ltx_td ltx_nopad_r ltx_align_left">40.43</td>
</tr>
<tr id="Ax1.T28.1.59" class="ltx_tr">
<td id="Ax1.T28.1.59.1" class="ltx_td ltx_align_left">bbh-temporal_sequences</td>
<td id="Ax1.T28.1.59.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T28.1.59.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T28.1.59.4" class="ltx_td ltx_nopad_r ltx_align_left">20</td>
</tr>
<tr id="Ax1.T28.1.60" class="ltx_tr">
<td id="Ax1.T28.1.60.1" class="ltx_td ltx_align_left">bbh-disambiguation_qa</td>
<td id="Ax1.T28.1.60.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T28.1.60.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T28.1.60.4" class="ltx_td ltx_nopad_r ltx_align_left">48</td>
</tr>
<tr id="Ax1.T28.1.61" class="ltx_tr">
<td id="Ax1.T28.1.61.1" class="ltx_td ltx_align_left">bbh-date_understanding</td>
<td id="Ax1.T28.1.61.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T28.1.61.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T28.1.61.4" class="ltx_td ltx_nopad_r ltx_align_left">70.8</td>
</tr>
<tr id="Ax1.T28.1.62" class="ltx_tr">
<td id="Ax1.T28.1.62.1" class="ltx_td ltx_align_left">bbh-tracking_shuffled_objects_three_objects</td>
<td id="Ax1.T28.1.62.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T28.1.62.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T28.1.62.4" class="ltx_td ltx_nopad_r ltx_align_left">34</td>
</tr>
<tr id="Ax1.T28.1.63" class="ltx_tr">
<td id="Ax1.T28.1.63.1" class="ltx_td ltx_align_left">bbh-penguins_in_a_table</td>
<td id="Ax1.T28.1.63.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T28.1.63.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T28.1.63.4" class="ltx_td ltx_nopad_r ltx_align_left">55.48</td>
</tr>
<tr id="Ax1.T28.1.64" class="ltx_tr">
<td id="Ax1.T28.1.64.1" class="ltx_td ltx_align_left">bbh-geometric_shapes</td>
<td id="Ax1.T28.1.64.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T28.1.64.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T28.1.64.4" class="ltx_td ltx_nopad_r ltx_align_left">39.6</td>
</tr>
<tr id="Ax1.T28.1.65" class="ltx_tr">
<td id="Ax1.T28.1.65.1" class="ltx_td ltx_align_left">bbh-snarks</td>
<td id="Ax1.T28.1.65.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T28.1.65.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T28.1.65.4" class="ltx_td ltx_nopad_r ltx_align_left">73.03</td>
</tr>
<tr id="Ax1.T28.1.66" class="ltx_tr">
<td id="Ax1.T28.1.66.1" class="ltx_td ltx_align_left">bbh-ruin_names</td>
<td id="Ax1.T28.1.66.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T28.1.66.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T28.1.66.4" class="ltx_td ltx_nopad_r ltx_align_left">40</td>
</tr>
<tr id="Ax1.T28.1.67" class="ltx_tr">
<td id="Ax1.T28.1.67.1" class="ltx_td ltx_align_left">bbh-tracking_shuffled_objects_seven_objects</td>
<td id="Ax1.T28.1.67.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T28.1.67.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T28.1.67.4" class="ltx_td ltx_nopad_r ltx_align_left">18.8</td>
</tr>
<tr id="Ax1.T28.1.68" class="ltx_tr">
<td id="Ax1.T28.1.68.1" class="ltx_td ltx_align_left">bbh-tracking_shuffled_objects_five_objects</td>
<td id="Ax1.T28.1.68.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T28.1.68.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T28.1.68.4" class="ltx_td ltx_nopad_r ltx_align_left">22</td>
</tr>
<tr id="Ax1.T28.1.69" class="ltx_tr">
<td id="Ax1.T28.1.69.1" class="ltx_td ltx_align_left">bbh-logical_deduction_three_objects</td>
<td id="Ax1.T28.1.69.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T28.1.69.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T28.1.69.4" class="ltx_td ltx_nopad_r ltx_align_left">75.6</td>
</tr>
<tr id="Ax1.T28.1.70" class="ltx_tr">
<td id="Ax1.T28.1.70.1" class="ltx_td ltx_align_left">bbh-hyperbaton</td>
<td id="Ax1.T28.1.70.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T28.1.70.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T28.1.70.4" class="ltx_td ltx_nopad_r ltx_align_left">58.4</td>
</tr>
<tr id="Ax1.T28.1.71" class="ltx_tr">
<td id="Ax1.T28.1.71.1" class="ltx_td ltx_align_left">bbh-logical_deduction_five_objects</td>
<td id="Ax1.T28.1.71.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T28.1.71.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T28.1.71.4" class="ltx_td ltx_nopad_r ltx_align_left">49.6</td>
</tr>
<tr id="Ax1.T28.1.72" class="ltx_tr">
<td id="Ax1.T28.1.72.1" class="ltx_td ltx_align_left">bbh-logical_deduction_seven_objects</td>
<td id="Ax1.T28.1.72.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T28.1.72.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T28.1.72.4" class="ltx_td ltx_nopad_r ltx_align_left">42.8</td>
</tr>
<tr id="Ax1.T28.1.73" class="ltx_tr">
<td id="Ax1.T28.1.73.1" class="ltx_td ltx_align_left">bbh-movie_recommendation</td>
<td id="Ax1.T28.1.73.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T28.1.73.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T28.1.73.4" class="ltx_td ltx_nopad_r ltx_align_left">64</td>
</tr>
<tr id="Ax1.T28.1.74" class="ltx_tr">
<td id="Ax1.T28.1.74.1" class="ltx_td ltx_align_left">bbh-salient_translation_error_detection</td>
<td id="Ax1.T28.1.74.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T28.1.74.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T28.1.74.4" class="ltx_td ltx_nopad_r ltx_align_left">43.2</td>
</tr>
<tr id="Ax1.T28.1.75" class="ltx_tr">
<td id="Ax1.T28.1.75.1" class="ltx_td ltx_align_left">bbh-reasoning_about_colored_objects</td>
<td id="Ax1.T28.1.75.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T28.1.75.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T28.1.75.4" class="ltx_td ltx_nopad_r ltx_align_left">63.6</td>
</tr>
<tr id="Ax1.T28.1.76" class="ltx_tr">
<td id="Ax1.T28.1.76.1" class="ltx_td ltx_align_left">bbh-multistep_arithmetic_two</td>
<td id="Ax1.T28.1.76.2" class="ltx_td ltx_align_left">score</td>
<td id="Ax1.T28.1.76.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T28.1.76.4" class="ltx_td ltx_nopad_r ltx_align_left">4</td>
</tr>
<tr id="Ax1.T28.1.77" class="ltx_tr">
<td id="Ax1.T28.1.77.1" class="ltx_td ltx_align_left">bbh-navigate</td>
<td id="Ax1.T28.1.77.2" class="ltx_td ltx_align_left">score</td>
<td id="Ax1.T28.1.77.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T28.1.77.4" class="ltx_td ltx_nopad_r ltx_align_left">65.6</td>
</tr>
<tr id="Ax1.T28.1.78" class="ltx_tr">
<td id="Ax1.T28.1.78.1" class="ltx_td ltx_align_left">bbh-dyck_languages</td>
<td id="Ax1.T28.1.78.2" class="ltx_td ltx_align_left">score</td>
<td id="Ax1.T28.1.78.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T28.1.78.4" class="ltx_td ltx_nopad_r ltx_align_left">6.4</td>
</tr>
<tr id="Ax1.T28.1.79" class="ltx_tr">
<td id="Ax1.T28.1.79.1" class="ltx_td ltx_align_left">bbh-word_sorting</td>
<td id="Ax1.T28.1.79.2" class="ltx_td ltx_align_left">score</td>
<td id="Ax1.T28.1.79.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T28.1.79.4" class="ltx_td ltx_nopad_r ltx_align_left">23.2</td>
</tr>
<tr id="Ax1.T28.1.80" class="ltx_tr">
<td id="Ax1.T28.1.80.1" class="ltx_td ltx_align_left">bbh-sports_understanding</td>
<td id="Ax1.T28.1.80.2" class="ltx_td ltx_align_left">score</td>
<td id="Ax1.T28.1.80.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T28.1.80.4" class="ltx_td ltx_nopad_r ltx_align_left">91.6</td>
</tr>
<tr id="Ax1.T28.1.81" class="ltx_tr">
<td id="Ax1.T28.1.81.1" class="ltx_td ltx_align_left">bbh-boolean_expressions</td>
<td id="Ax1.T28.1.81.2" class="ltx_td ltx_align_left">score</td>
<td id="Ax1.T28.1.81.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T28.1.81.4" class="ltx_td ltx_nopad_r ltx_align_left">70.4</td>
</tr>
<tr id="Ax1.T28.1.82" class="ltx_tr">
<td id="Ax1.T28.1.82.1" class="ltx_td ltx_align_left">bbh-object_counting</td>
<td id="Ax1.T28.1.82.2" class="ltx_td ltx_align_left">score</td>
<td id="Ax1.T28.1.82.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T28.1.82.4" class="ltx_td ltx_nopad_r ltx_align_left">57.2</td>
</tr>
<tr id="Ax1.T28.1.83" class="ltx_tr">
<td id="Ax1.T28.1.83.1" class="ltx_td ltx_align_left">bbh-formal_fallacies</td>
<td id="Ax1.T28.1.83.2" class="ltx_td ltx_align_left">score</td>
<td id="Ax1.T28.1.83.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T28.1.83.4" class="ltx_td ltx_nopad_r ltx_align_left">51.2</td>
</tr>
<tr id="Ax1.T28.1.84" class="ltx_tr">
<td id="Ax1.T28.1.84.1" class="ltx_td ltx_align_left">bbh-causal_judgement</td>
<td id="Ax1.T28.1.84.2" class="ltx_td ltx_align_left">score</td>
<td id="Ax1.T28.1.84.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T28.1.84.4" class="ltx_td ltx_nopad_r ltx_align_left">60.96</td>
</tr>
<tr id="Ax1.T28.1.85" class="ltx_tr">
<td id="Ax1.T28.1.85.1" class="ltx_td ltx_align_left">bbh-web_of_lies</td>
<td id="Ax1.T28.1.85.2" class="ltx_td ltx_align_left">score</td>
<td id="Ax1.T28.1.85.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T28.1.85.4" class="ltx_td ltx_nopad_r ltx_align_left">92.4</td>
</tr>
<tr id="Ax1.T28.1.86" class="ltx_tr">
<td id="Ax1.T28.1.86.1" class="ltx_td ltx_align_left">tyidqa-goldp_arabic</td>
<td id="Ax1.T28.1.86.2" class="ltx_td ltx_align_left">exact_match</td>
<td id="Ax1.T28.1.86.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T28.1.86.4" class="ltx_td ltx_nopad_r ltx_align_left">2.39</td>
</tr>
<tr id="Ax1.T28.1.87" class="ltx_tr">
<td id="Ax1.T28.1.87.1" class="ltx_td ltx_align_left">tyidqa-goldp_arabic</td>
<td id="Ax1.T28.1.87.2" class="ltx_td ltx_align_left">f1</td>
<td id="Ax1.T28.1.87.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T28.1.87.4" class="ltx_td ltx_nopad_r ltx_align_left">41.76</td>
</tr>
<tr id="Ax1.T28.1.88" class="ltx_tr">
<td id="Ax1.T28.1.88.1" class="ltx_td ltx_align_left">tyidqa-goldp_bengali</td>
<td id="Ax1.T28.1.88.2" class="ltx_td ltx_align_left">exact_match</td>
<td id="Ax1.T28.1.88.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T28.1.88.4" class="ltx_td ltx_nopad_r ltx_align_left">1.77</td>
</tr>
<tr id="Ax1.T28.1.89" class="ltx_tr">
<td id="Ax1.T28.1.89.1" class="ltx_td ltx_align_left">tyidqa-goldp_bengali</td>
<td id="Ax1.T28.1.89.2" class="ltx_td ltx_align_left">f1</td>
<td id="Ax1.T28.1.89.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T28.1.89.4" class="ltx_td ltx_nopad_r ltx_align_left">20.8</td>
</tr>
<tr id="Ax1.T28.1.90" class="ltx_tr">
<td id="Ax1.T28.1.90.1" class="ltx_td ltx_align_left">tyidqa-goldp_english</td>
<td id="Ax1.T28.1.90.2" class="ltx_td ltx_align_left">exact_match</td>
<td id="Ax1.T28.1.90.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T28.1.90.4" class="ltx_td ltx_nopad_r ltx_align_left">22.95</td>
</tr>
<tr id="Ax1.T28.1.91" class="ltx_tr">
<td id="Ax1.T28.1.91.1" class="ltx_td ltx_align_left">tyidqa-goldp_english</td>
<td id="Ax1.T28.1.91.2" class="ltx_td ltx_align_left">f1</td>
<td id="Ax1.T28.1.91.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T28.1.91.4" class="ltx_td ltx_nopad_r ltx_align_left">37.14</td>
</tr>
<tr id="Ax1.T28.1.92" class="ltx_tr">
<td id="Ax1.T28.1.92.1" class="ltx_td ltx_align_left">tyidqa-goldp_finnish</td>
<td id="Ax1.T28.1.92.2" class="ltx_td ltx_align_left">exact_match</td>
<td id="Ax1.T28.1.92.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T28.1.92.4" class="ltx_td ltx_nopad_r ltx_align_left">34.78</td>
</tr>
<tr id="Ax1.T28.1.93" class="ltx_tr">
<td id="Ax1.T28.1.93.1" class="ltx_td ltx_align_left">tyidqa-goldp_finnish</td>
<td id="Ax1.T28.1.93.2" class="ltx_td ltx_align_left">f1</td>
<td id="Ax1.T28.1.93.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T28.1.93.4" class="ltx_td ltx_nopad_r ltx_align_left">41.56</td>
</tr>
<tr id="Ax1.T28.1.94" class="ltx_tr">
<td id="Ax1.T28.1.94.1" class="ltx_td ltx_align_left">tyidqa-goldp_indonesian</td>
<td id="Ax1.T28.1.94.2" class="ltx_td ltx_align_left">exact_match</td>
<td id="Ax1.T28.1.94.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T28.1.94.4" class="ltx_td ltx_nopad_r ltx_align_left">14.34</td>
</tr>
<tr id="Ax1.T28.1.95" class="ltx_tr">
<td id="Ax1.T28.1.95.1" class="ltx_td ltx_align_left">tyidqa-goldp_indonesian</td>
<td id="Ax1.T28.1.95.2" class="ltx_td ltx_align_left">f1</td>
<td id="Ax1.T28.1.95.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T28.1.95.4" class="ltx_td ltx_nopad_r ltx_align_left">28.4</td>
</tr>
<tr id="Ax1.T28.1.96" class="ltx_tr">
<td id="Ax1.T28.1.96.1" class="ltx_td ltx_align_left">tyidqa-goldp_japanese</td>
<td id="Ax1.T28.1.96.2" class="ltx_td ltx_align_left">exact_match</td>
<td id="Ax1.T28.1.96.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T28.1.96.4" class="ltx_td ltx_nopad_r ltx_align_left">19.34</td>
</tr>
<tr id="Ax1.T28.1.97" class="ltx_tr">
<td id="Ax1.T28.1.97.1" class="ltx_td ltx_align_left">tyidqa-goldp_japanese</td>
<td id="Ax1.T28.1.97.2" class="ltx_td ltx_align_left">f1</td>
<td id="Ax1.T28.1.97.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T28.1.97.4" class="ltx_td ltx_nopad_r ltx_align_left">19.85</td>
</tr>
<tr id="Ax1.T28.1.98" class="ltx_tr">
<td id="Ax1.T28.1.98.1" class="ltx_td ltx_align_left">tyidqa-goldp_korean</td>
<td id="Ax1.T28.1.98.2" class="ltx_td ltx_align_left">exact_match</td>
<td id="Ax1.T28.1.98.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T28.1.98.4" class="ltx_td ltx_nopad_r ltx_align_left">0.72</td>
</tr>
<tr id="Ax1.T28.1.99" class="ltx_tr">
<td id="Ax1.T28.1.99.1" class="ltx_td ltx_align_left">tyidqa-goldp_korean</td>
<td id="Ax1.T28.1.99.2" class="ltx_td ltx_align_left">f1</td>
<td id="Ax1.T28.1.99.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T28.1.99.4" class="ltx_td ltx_nopad_r ltx_align_left">19.07</td>
</tr>
<tr id="Ax1.T28.1.100" class="ltx_tr">
<td id="Ax1.T28.1.100.1" class="ltx_td ltx_align_left">tyidqa-goldp_russian</td>
<td id="Ax1.T28.1.100.2" class="ltx_td ltx_align_left">exact_match</td>
<td id="Ax1.T28.1.100.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T28.1.100.4" class="ltx_td ltx_nopad_r ltx_align_left">7.02</td>
</tr>
<tr id="Ax1.T28.1.101" class="ltx_tr">
<td id="Ax1.T28.1.101.1" class="ltx_td ltx_align_left">tyidqa-goldp_russian</td>
<td id="Ax1.T28.1.101.2" class="ltx_td ltx_align_left">f1</td>
<td id="Ax1.T28.1.101.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T28.1.101.4" class="ltx_td ltx_nopad_r ltx_align_left">34.14</td>
</tr>
<tr id="Ax1.T28.1.102" class="ltx_tr">
<td id="Ax1.T28.1.102.1" class="ltx_td ltx_align_left">tyidqa-goldp_swahili</td>
<td id="Ax1.T28.1.102.2" class="ltx_td ltx_align_left">exact_match</td>
<td id="Ax1.T28.1.102.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T28.1.102.4" class="ltx_td ltx_nopad_r ltx_align_left">16.23</td>
</tr>
<tr id="Ax1.T28.1.103" class="ltx_tr">
<td id="Ax1.T28.1.103.1" class="ltx_td ltx_align_left">tyidqa-goldp_swahili</td>
<td id="Ax1.T28.1.103.2" class="ltx_td ltx_align_left">f1</td>
<td id="Ax1.T28.1.103.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T28.1.103.4" class="ltx_td ltx_nopad_r ltx_align_left">32.87</td>
</tr>
<tr id="Ax1.T28.1.104" class="ltx_tr">
<td id="Ax1.T28.1.104.1" class="ltx_td ltx_align_left">tyidqa-goldp_telugu</td>
<td id="Ax1.T28.1.104.2" class="ltx_td ltx_align_left">exact_match</td>
<td id="Ax1.T28.1.104.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T28.1.104.4" class="ltx_td ltx_nopad_r ltx_align_left">0</td>
</tr>
<tr id="Ax1.T28.1.105" class="ltx_tr">
<td id="Ax1.T28.1.105.1" class="ltx_td ltx_align_left">tyidqa-goldp_telugu</td>
<td id="Ax1.T28.1.105.2" class="ltx_td ltx_align_left">f1</td>
<td id="Ax1.T28.1.105.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T28.1.105.4" class="ltx_td ltx_nopad_r ltx_align_left">1.39</td>
</tr>
<tr id="Ax1.T28.1.106" class="ltx_tr">
<td id="Ax1.T28.1.106.1" class="ltx_td ltx_align_left">tyidqa-goldp_thai</td>
<td id="Ax1.T28.1.106.2" class="ltx_td ltx_align_left">exact_match</td>
<td id="Ax1.T28.1.106.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T28.1.106.4" class="ltx_td ltx_nopad_r ltx_align_left">0.51</td>
</tr>
<tr id="Ax1.T28.1.107" class="ltx_tr">
<td id="Ax1.T28.1.107.1" class="ltx_td ltx_align_left">tyidqa-goldp_thai</td>
<td id="Ax1.T28.1.107.2" class="ltx_td ltx_align_left">f1</td>
<td id="Ax1.T28.1.107.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T28.1.107.4" class="ltx_td ltx_nopad_r ltx_align_left">27.16</td>
</tr>
<tr id="Ax1.T28.1.108" class="ltx_tr">
<td id="Ax1.T28.1.108.1" class="ltx_td ltx_align_left">piqa</td>
<td id="Ax1.T28.1.108.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T28.1.108.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T28.1.108.4" class="ltx_td ltx_nopad_r ltx_align_left">78.24</td>
</tr>
<tr id="Ax1.T28.1.109" class="ltx_tr">
<td id="Ax1.T28.1.109.1" class="ltx_td ltx_align_left">siqa</td>
<td id="Ax1.T28.1.109.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T28.1.109.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T28.1.109.4" class="ltx_td ltx_nopad_r ltx_align_left">50.87</td>
</tr>
<tr id="Ax1.T28.1.110" class="ltx_tr">
<td id="Ax1.T28.1.110.1" class="ltx_td ltx_align_left">gsm8k</td>
<td id="Ax1.T28.1.110.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T28.1.110.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T28.1.110.4" class="ltx_td ltx_nopad_r ltx_align_left">43.14</td>
</tr>
<tr id="Ax1.T28.1.111" class="ltx_tr">
<td id="Ax1.T28.1.111.1" class="ltx_td ltx_align_left">truthful_qa</td>
<td id="Ax1.T28.1.111.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T28.1.111.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T28.1.111.4" class="ltx_td ltx_nopad_r ltx_align_left">-</td>
</tr>
<tr id="Ax1.T28.1.112" class="ltx_tr">
<td id="Ax1.T28.1.112.1" class="ltx_td ltx_align_left">triviaqa</td>
<td id="Ax1.T28.1.112.2" class="ltx_td ltx_align_left">score</td>
<td id="Ax1.T28.1.112.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T28.1.112.4" class="ltx_td ltx_nopad_r ltx_align_left">54.36</td>
</tr>
<tr id="Ax1.T28.1.113" class="ltx_tr">
<td id="Ax1.T28.1.113.1" class="ltx_td ltx_align_left ltx_border_bb">BoolQ</td>
<td id="Ax1.T28.1.113.2" class="ltx_td ltx_align_left ltx_border_bb">accuracy</td>
<td id="Ax1.T28.1.113.3" class="ltx_td ltx_align_left ltx_border_bb">ppl</td>
<td id="Ax1.T28.1.113.4" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_bb">81.5</td>
</tr>
</table>
</figure>
<figure id="Ax1.T29" class="ltx_table">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table">Table 29: </span>Detailed results of open-compass toolkit of LLaMA-13b-chat after training on the TRACE sequential tasks with LoRA adapters.</figcaption>
<table id="Ax1.T29.1" class="ltx_tabular">
<tr id="Ax1.T29.1.1" class="ltx_tr">
<td id="Ax1.T29.1.1.1" class="ltx_td ltx_align_left ltx_border_tt">dataset</td>
<td id="Ax1.T29.1.1.2" class="ltx_td ltx_align_left ltx_border_tt">metric</td>
<td id="Ax1.T29.1.1.3" class="ltx_td ltx_align_left ltx_border_tt">mode</td>
<td id="Ax1.T29.1.1.4" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_tt">performance</td>
</tr>
<tr id="Ax1.T29.1.2" class="ltx_tr">
<td id="Ax1.T29.1.2.1" class="ltx_td ltx_align_left ltx_border_t">lukaemon_mmlu_college_biology</td>
<td id="Ax1.T29.1.2.2" class="ltx_td ltx_align_left ltx_border_t">accuracy</td>
<td id="Ax1.T29.1.2.3" class="ltx_td ltx_align_left ltx_border_t">ppl</td>
<td id="Ax1.T29.1.2.4" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_t">54.17</td>
</tr>
<tr id="Ax1.T29.1.3" class="ltx_tr">
<td id="Ax1.T29.1.3.1" class="ltx_td ltx_align_left">lukaemon_mmlu_college_chemistry</td>
<td id="Ax1.T29.1.3.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T29.1.3.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T29.1.3.4" class="ltx_td ltx_nopad_r ltx_align_left">38</td>
</tr>
<tr id="Ax1.T29.1.4" class="ltx_tr">
<td id="Ax1.T29.1.4.1" class="ltx_td ltx_align_left">lukaemon_mmlu_college_computer_science</td>
<td id="Ax1.T29.1.4.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T29.1.4.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T29.1.4.4" class="ltx_td ltx_nopad_r ltx_align_left">45</td>
</tr>
<tr id="Ax1.T29.1.5" class="ltx_tr">
<td id="Ax1.T29.1.5.1" class="ltx_td ltx_align_left">lukaemon_mmlu_college_mathematics</td>
<td id="Ax1.T29.1.5.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T29.1.5.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T29.1.5.4" class="ltx_td ltx_nopad_r ltx_align_left">29</td>
</tr>
<tr id="Ax1.T29.1.6" class="ltx_tr">
<td id="Ax1.T29.1.6.1" class="ltx_td ltx_align_left">lukaemon_mmlu_college_physics</td>
<td id="Ax1.T29.1.6.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T29.1.6.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T29.1.6.4" class="ltx_td ltx_nopad_r ltx_align_left">34.31</td>
</tr>
<tr id="Ax1.T29.1.7" class="ltx_tr">
<td id="Ax1.T29.1.7.1" class="ltx_td ltx_align_left">lukaemon_mmlu_electrical_engineering</td>
<td id="Ax1.T29.1.7.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T29.1.7.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T29.1.7.4" class="ltx_td ltx_nopad_r ltx_align_left">47.59</td>
</tr>
<tr id="Ax1.T29.1.8" class="ltx_tr">
<td id="Ax1.T29.1.8.1" class="ltx_td ltx_align_left">lukaemon_mmlu_astronomy</td>
<td id="Ax1.T29.1.8.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T29.1.8.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T29.1.8.4" class="ltx_td ltx_nopad_r ltx_align_left">59.21</td>
</tr>
<tr id="Ax1.T29.1.9" class="ltx_tr">
<td id="Ax1.T29.1.9.1" class="ltx_td ltx_align_left">lukaemon_mmlu_anatomy</td>
<td id="Ax1.T29.1.9.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T29.1.9.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T29.1.9.4" class="ltx_td ltx_nopad_r ltx_align_left">54.07</td>
</tr>
<tr id="Ax1.T29.1.10" class="ltx_tr">
<td id="Ax1.T29.1.10.1" class="ltx_td ltx_align_left">lukaemon_mmlu_abstract_algebra</td>
<td id="Ax1.T29.1.10.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T29.1.10.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T29.1.10.4" class="ltx_td ltx_nopad_r ltx_align_left">33</td>
</tr>
<tr id="Ax1.T29.1.11" class="ltx_tr">
<td id="Ax1.T29.1.11.1" class="ltx_td ltx_align_left">lukaemon_mmlu_machine_learning</td>
<td id="Ax1.T29.1.11.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T29.1.11.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T29.1.11.4" class="ltx_td ltx_nopad_r ltx_align_left">31.25</td>
</tr>
<tr id="Ax1.T29.1.12" class="ltx_tr">
<td id="Ax1.T29.1.12.1" class="ltx_td ltx_align_left">lukaemon_mmlu_clinical_knowledge</td>
<td id="Ax1.T29.1.12.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T29.1.12.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T29.1.12.4" class="ltx_td ltx_nopad_r ltx_align_left">52.83</td>
</tr>
<tr id="Ax1.T29.1.13" class="ltx_tr">
<td id="Ax1.T29.1.13.1" class="ltx_td ltx_align_left">lukaemon_mmlu_global_facts</td>
<td id="Ax1.T29.1.13.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T29.1.13.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T29.1.13.4" class="ltx_td ltx_nopad_r ltx_align_left">36</td>
</tr>
<tr id="Ax1.T29.1.14" class="ltx_tr">
<td id="Ax1.T29.1.14.1" class="ltx_td ltx_align_left">lukaemon_mmlu_management</td>
<td id="Ax1.T29.1.14.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T29.1.14.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T29.1.14.4" class="ltx_td ltx_nopad_r ltx_align_left">64.08</td>
</tr>
<tr id="Ax1.T29.1.15" class="ltx_tr">
<td id="Ax1.T29.1.15.1" class="ltx_td ltx_align_left">lukaemon_mmlu_nutrition</td>
<td id="Ax1.T29.1.15.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T29.1.15.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T29.1.15.4" class="ltx_td ltx_nopad_r ltx_align_left">56.86</td>
</tr>
<tr id="Ax1.T29.1.16" class="ltx_tr">
<td id="Ax1.T29.1.16.1" class="ltx_td ltx_align_left">lukaemon_mmlu_marketing</td>
<td id="Ax1.T29.1.16.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T29.1.16.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T29.1.16.4" class="ltx_td ltx_nopad_r ltx_align_left">66.67</td>
</tr>
<tr id="Ax1.T29.1.17" class="ltx_tr">
<td id="Ax1.T29.1.17.1" class="ltx_td ltx_align_left">lukaemon_mmlu_professional_accounting</td>
<td id="Ax1.T29.1.17.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T29.1.17.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T29.1.17.4" class="ltx_td ltx_nopad_r ltx_align_left">38.3</td>
</tr>
<tr id="Ax1.T29.1.18" class="ltx_tr">
<td id="Ax1.T29.1.18.1" class="ltx_td ltx_align_left">lukaemon_mmlu_high_school_geography</td>
<td id="Ax1.T29.1.18.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T29.1.18.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T29.1.18.4" class="ltx_td ltx_nopad_r ltx_align_left">64.65</td>
</tr>
<tr id="Ax1.T29.1.19" class="ltx_tr">
<td id="Ax1.T29.1.19.1" class="ltx_td ltx_align_left">lukaemon_mmlu_international_law</td>
<td id="Ax1.T29.1.19.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T29.1.19.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T29.1.19.4" class="ltx_td ltx_nopad_r ltx_align_left">68.6</td>
</tr>
<tr id="Ax1.T29.1.20" class="ltx_tr">
<td id="Ax1.T29.1.20.1" class="ltx_td ltx_align_left">lukaemon_mmlu_moral_scenarios</td>
<td id="Ax1.T29.1.20.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T29.1.20.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T29.1.20.4" class="ltx_td ltx_nopad_r ltx_align_left">23.35</td>
</tr>
<tr id="Ax1.T29.1.21" class="ltx_tr">
<td id="Ax1.T29.1.21.1" class="ltx_td ltx_align_left">lukaemon_mmlu_computer_security</td>
<td id="Ax1.T29.1.21.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T29.1.21.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T29.1.21.4" class="ltx_td ltx_nopad_r ltx_align_left">57</td>
</tr>
<tr id="Ax1.T29.1.22" class="ltx_tr">
<td id="Ax1.T29.1.22.1" class="ltx_td ltx_align_left">lukaemon_mmlu_high_school_microeconomics</td>
<td id="Ax1.T29.1.22.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T29.1.22.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T29.1.22.4" class="ltx_td ltx_nopad_r ltx_align_left">51.26</td>
</tr>
<tr id="Ax1.T29.1.23" class="ltx_tr">
<td id="Ax1.T29.1.23.1" class="ltx_td ltx_align_left">lukaemon_mmlu_professional_law</td>
<td id="Ax1.T29.1.23.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T29.1.23.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T29.1.23.4" class="ltx_td ltx_nopad_r ltx_align_left">38.01</td>
</tr>
<tr id="Ax1.T29.1.24" class="ltx_tr">
<td id="Ax1.T29.1.24.1" class="ltx_td ltx_align_left">lukaemon_mmlu_medical_genetics</td>
<td id="Ax1.T29.1.24.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T29.1.24.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T29.1.24.4" class="ltx_td ltx_nopad_r ltx_align_left">62</td>
</tr>
<tr id="Ax1.T29.1.25" class="ltx_tr">
<td id="Ax1.T29.1.25.1" class="ltx_td ltx_align_left">lukaemon_mmlu_professional_psychology</td>
<td id="Ax1.T29.1.25.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T29.1.25.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T29.1.25.4" class="ltx_td ltx_nopad_r ltx_align_left">47.06</td>
</tr>
<tr id="Ax1.T29.1.26" class="ltx_tr">
<td id="Ax1.T29.1.26.1" class="ltx_td ltx_align_left">lukaemon_mmlu_jurisprudence</td>
<td id="Ax1.T29.1.26.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T29.1.26.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T29.1.26.4" class="ltx_td ltx_nopad_r ltx_align_left">52.78</td>
</tr>
<tr id="Ax1.T29.1.27" class="ltx_tr">
<td id="Ax1.T29.1.27.1" class="ltx_td ltx_align_left">lukaemon_mmlu_world_religions</td>
<td id="Ax1.T29.1.27.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T29.1.27.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T29.1.27.4" class="ltx_td ltx_nopad_r ltx_align_left">71.35</td>
</tr>
<tr id="Ax1.T29.1.28" class="ltx_tr">
<td id="Ax1.T29.1.28.1" class="ltx_td ltx_align_left">lukaemon_mmlu_philosophy</td>
<td id="Ax1.T29.1.28.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T29.1.28.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T29.1.28.4" class="ltx_td ltx_nopad_r ltx_align_left">60.45</td>
</tr>
<tr id="Ax1.T29.1.29" class="ltx_tr">
<td id="Ax1.T29.1.29.1" class="ltx_td ltx_align_left">lukaemon_mmlu_virology</td>
<td id="Ax1.T29.1.29.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T29.1.29.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T29.1.29.4" class="ltx_td ltx_nopad_r ltx_align_left">46.39</td>
</tr>
<tr id="Ax1.T29.1.30" class="ltx_tr">
<td id="Ax1.T29.1.30.1" class="ltx_td ltx_align_left">lukaemon_mmlu_high_school_chemistry</td>
<td id="Ax1.T29.1.30.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T29.1.30.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T29.1.30.4" class="ltx_td ltx_nopad_r ltx_align_left">40.39</td>
</tr>
<tr id="Ax1.T29.1.31" class="ltx_tr">
<td id="Ax1.T29.1.31.1" class="ltx_td ltx_align_left">lukaemon_mmlu_public_relations</td>
<td id="Ax1.T29.1.31.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T29.1.31.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T29.1.31.4" class="ltx_td ltx_nopad_r ltx_align_left">54.55</td>
</tr>
<tr id="Ax1.T29.1.32" class="ltx_tr">
<td id="Ax1.T29.1.32.1" class="ltx_td ltx_align_left">lukaemon_mmlu_high_school_macroeconomics</td>
<td id="Ax1.T29.1.32.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T29.1.32.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T29.1.32.4" class="ltx_td ltx_nopad_r ltx_align_left">50.51</td>
</tr>
<tr id="Ax1.T29.1.33" class="ltx_tr">
<td id="Ax1.T29.1.33.1" class="ltx_td ltx_align_left">lukaemon_mmlu_human_sexuality</td>
<td id="Ax1.T29.1.33.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T29.1.33.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T29.1.33.4" class="ltx_td ltx_nopad_r ltx_align_left">54.2</td>
</tr>
<tr id="Ax1.T29.1.34" class="ltx_tr">
<td id="Ax1.T29.1.34.1" class="ltx_td ltx_align_left">lukaemon_mmlu_elementary_mathematics</td>
<td id="Ax1.T29.1.34.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T29.1.34.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T29.1.34.4" class="ltx_td ltx_nopad_r ltx_align_left">33.07</td>
</tr>
<tr id="Ax1.T29.1.35" class="ltx_tr">
<td id="Ax1.T29.1.35.1" class="ltx_td ltx_align_left">lukaemon_mmlu_high_school_physics</td>
<td id="Ax1.T29.1.35.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T29.1.35.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T29.1.35.4" class="ltx_td ltx_nopad_r ltx_align_left">31.13</td>
</tr>
<tr id="Ax1.T29.1.36" class="ltx_tr">
<td id="Ax1.T29.1.36.1" class="ltx_td ltx_align_left">lukaemon_mmlu_high_school_computer_science</td>
<td id="Ax1.T29.1.36.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T29.1.36.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T29.1.36.4" class="ltx_td ltx_nopad_r ltx_align_left">52</td>
</tr>
<tr id="Ax1.T29.1.37" class="ltx_tr">
<td id="Ax1.T29.1.37.1" class="ltx_td ltx_align_left">lukaemon_mmlu_high_school_european_history</td>
<td id="Ax1.T29.1.37.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T29.1.37.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T29.1.37.4" class="ltx_td ltx_nopad_r ltx_align_left">59.39</td>
</tr>
<tr id="Ax1.T29.1.38" class="ltx_tr">
<td id="Ax1.T29.1.38.1" class="ltx_td ltx_align_left">lukaemon_mmlu_business_ethics</td>
<td id="Ax1.T29.1.38.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T29.1.38.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T29.1.38.4" class="ltx_td ltx_nopad_r ltx_align_left">55</td>
</tr>
<tr id="Ax1.T29.1.39" class="ltx_tr">
<td id="Ax1.T29.1.39.1" class="ltx_td ltx_align_left">lukaemon_mmlu_moral_disputes</td>
<td id="Ax1.T29.1.39.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T29.1.39.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T29.1.39.4" class="ltx_td ltx_nopad_r ltx_align_left">53.76</td>
</tr>
<tr id="Ax1.T29.1.40" class="ltx_tr">
<td id="Ax1.T29.1.40.1" class="ltx_td ltx_align_left">lukaemon_mmlu_high_school_statistics</td>
<td id="Ax1.T29.1.40.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T29.1.40.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T29.1.40.4" class="ltx_td ltx_nopad_r ltx_align_left">39.81</td>
</tr>
<tr id="Ax1.T29.1.41" class="ltx_tr">
<td id="Ax1.T29.1.41.1" class="ltx_td ltx_align_left">lukaemon_mmlu_miscellaneous</td>
<td id="Ax1.T29.1.41.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T29.1.41.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T29.1.41.4" class="ltx_td ltx_nopad_r ltx_align_left">71.39</td>
</tr>
<tr id="Ax1.T29.1.42" class="ltx_tr">
<td id="Ax1.T29.1.42.1" class="ltx_td ltx_align_left">lukaemon_mmlu_formal_logic</td>
<td id="Ax1.T29.1.42.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T29.1.42.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T29.1.42.4" class="ltx_td ltx_nopad_r ltx_align_left">23.02</td>
</tr>
<tr id="Ax1.T29.1.43" class="ltx_tr">
<td id="Ax1.T29.1.43.1" class="ltx_td ltx_align_left">lukaemon_mmlu_high_school_government_and_politics</td>
<td id="Ax1.T29.1.43.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T29.1.43.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T29.1.43.4" class="ltx_td ltx_nopad_r ltx_align_left">73.06</td>
</tr>
<tr id="Ax1.T29.1.44" class="ltx_tr">
<td id="Ax1.T29.1.44.1" class="ltx_td ltx_align_left">lukaemon_mmlu_prehistory</td>
<td id="Ax1.T29.1.44.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T29.1.44.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T29.1.44.4" class="ltx_td ltx_nopad_r ltx_align_left">54.01</td>
</tr>
<tr id="Ax1.T29.1.45" class="ltx_tr">
<td id="Ax1.T29.1.45.1" class="ltx_td ltx_align_left">lukaemon_mmlu_security_studies</td>
<td id="Ax1.T29.1.45.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T29.1.45.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T29.1.45.4" class="ltx_td ltx_nopad_r ltx_align_left">59.18</td>
</tr>
<tr id="Ax1.T29.1.46" class="ltx_tr">
<td id="Ax1.T29.1.46.1" class="ltx_td ltx_align_left">lukaemon_mmlu_high_school_biology</td>
<td id="Ax1.T29.1.46.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T29.1.46.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T29.1.46.4" class="ltx_td ltx_nopad_r ltx_align_left">58.71</td>
</tr>
<tr id="Ax1.T29.1.47" class="ltx_tr">
<td id="Ax1.T29.1.47.1" class="ltx_td ltx_align_left">lukaemon_mmlu_logical_fallacies</td>
<td id="Ax1.T29.1.47.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T29.1.47.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T29.1.47.4" class="ltx_td ltx_nopad_r ltx_align_left">57.06</td>
</tr>
<tr id="Ax1.T29.1.48" class="ltx_tr">
<td id="Ax1.T29.1.48.1" class="ltx_td ltx_align_left">lukaemon_mmlu_high_school_world_history</td>
<td id="Ax1.T29.1.48.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T29.1.48.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T29.1.48.4" class="ltx_td ltx_nopad_r ltx_align_left">64.14</td>
</tr>
<tr id="Ax1.T29.1.49" class="ltx_tr">
<td id="Ax1.T29.1.49.1" class="ltx_td ltx_align_left">lukaemon_mmlu_professional_medicine</td>
<td id="Ax1.T29.1.49.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T29.1.49.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T29.1.49.4" class="ltx_td ltx_nopad_r ltx_align_left">44.49</td>
</tr>
<tr id="Ax1.T29.1.50" class="ltx_tr">
<td id="Ax1.T29.1.50.1" class="ltx_td ltx_align_left">lukaemon_mmlu_high_school_mathematics</td>
<td id="Ax1.T29.1.50.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T29.1.50.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T29.1.50.4" class="ltx_td ltx_nopad_r ltx_align_left">30.74</td>
</tr>
<tr id="Ax1.T29.1.51" class="ltx_tr">
<td id="Ax1.T29.1.51.1" class="ltx_td ltx_align_left">lukaemon_mmlu_college_medicine</td>
<td id="Ax1.T29.1.51.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T29.1.51.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T29.1.51.4" class="ltx_td ltx_nopad_r ltx_align_left">45.66</td>
</tr>
<tr id="Ax1.T29.1.52" class="ltx_tr">
<td id="Ax1.T29.1.52.1" class="ltx_td ltx_align_left">lukaemon_mmlu_high_school_us_history</td>
<td id="Ax1.T29.1.52.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T29.1.52.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T29.1.52.4" class="ltx_td ltx_nopad_r ltx_align_left">59.8</td>
</tr>
<tr id="Ax1.T29.1.53" class="ltx_tr">
<td id="Ax1.T29.1.53.1" class="ltx_td ltx_align_left">lukaemon_mmlu_sociology</td>
<td id="Ax1.T29.1.53.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T29.1.53.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T29.1.53.4" class="ltx_td ltx_nopad_r ltx_align_left">72.14</td>
</tr>
<tr id="Ax1.T29.1.54" class="ltx_tr">
<td id="Ax1.T29.1.54.1" class="ltx_td ltx_align_left">lukaemon_mmlu_econometrics</td>
<td id="Ax1.T29.1.54.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T29.1.54.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T29.1.54.4" class="ltx_td ltx_nopad_r ltx_align_left">30.7</td>
</tr>
<tr id="Ax1.T29.1.55" class="ltx_tr">
<td id="Ax1.T29.1.55.1" class="ltx_td ltx_align_left">lukaemon_mmlu_high_school_psychology</td>
<td id="Ax1.T29.1.55.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T29.1.55.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T29.1.55.4" class="ltx_td ltx_nopad_r ltx_align_left">69.36</td>
</tr>
<tr id="Ax1.T29.1.56" class="ltx_tr">
<td id="Ax1.T29.1.56.1" class="ltx_td ltx_align_left">lukaemon_mmlu_human_aging</td>
<td id="Ax1.T29.1.56.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T29.1.56.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T29.1.56.4" class="ltx_td ltx_nopad_r ltx_align_left">55.16</td>
</tr>
<tr id="Ax1.T29.1.57" class="ltx_tr">
<td id="Ax1.T29.1.57.1" class="ltx_td ltx_align_left">lukaemon_mmlu_us_foreign_policy</td>
<td id="Ax1.T29.1.57.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T29.1.57.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T29.1.57.4" class="ltx_td ltx_nopad_r ltx_align_left">72</td>
</tr>
<tr id="Ax1.T29.1.58" class="ltx_tr">
<td id="Ax1.T29.1.58.1" class="ltx_td ltx_align_left">lukaemon_mmlu_conceptual_physics</td>
<td id="Ax1.T29.1.58.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T29.1.58.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T29.1.58.4" class="ltx_td ltx_nopad_r ltx_align_left">38.3</td>
</tr>
<tr id="Ax1.T29.1.59" class="ltx_tr">
<td id="Ax1.T29.1.59.1" class="ltx_td ltx_align_left">bbh-temporal_sequences</td>
<td id="Ax1.T29.1.59.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T29.1.59.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T29.1.59.4" class="ltx_td ltx_nopad_r ltx_align_left">20.4</td>
</tr>
<tr id="Ax1.T29.1.60" class="ltx_tr">
<td id="Ax1.T29.1.60.1" class="ltx_td ltx_align_left">bbh-disambiguation_qa</td>
<td id="Ax1.T29.1.60.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T29.1.60.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T29.1.60.4" class="ltx_td ltx_nopad_r ltx_align_left">31.6</td>
</tr>
<tr id="Ax1.T29.1.61" class="ltx_tr">
<td id="Ax1.T29.1.61.1" class="ltx_td ltx_align_left">bbh-date_understanding</td>
<td id="Ax1.T29.1.61.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T29.1.61.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T29.1.61.4" class="ltx_td ltx_nopad_r ltx_align_left">64.8</td>
</tr>
<tr id="Ax1.T29.1.62" class="ltx_tr">
<td id="Ax1.T29.1.62.1" class="ltx_td ltx_align_left">bbh-tracking_shuffled_objects_three_objects</td>
<td id="Ax1.T29.1.62.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T29.1.62.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T29.1.62.4" class="ltx_td ltx_nopad_r ltx_align_left">29.2</td>
</tr>
<tr id="Ax1.T29.1.63" class="ltx_tr">
<td id="Ax1.T29.1.63.1" class="ltx_td ltx_align_left">bbh-penguins_in_a_table</td>
<td id="Ax1.T29.1.63.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T29.1.63.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T29.1.63.4" class="ltx_td ltx_nopad_r ltx_align_left">45.89</td>
</tr>
<tr id="Ax1.T29.1.64" class="ltx_tr">
<td id="Ax1.T29.1.64.1" class="ltx_td ltx_align_left">bbh-geometric_shapes</td>
<td id="Ax1.T29.1.64.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T29.1.64.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T29.1.64.4" class="ltx_td ltx_nopad_r ltx_align_left">7.2</td>
</tr>
<tr id="Ax1.T29.1.65" class="ltx_tr">
<td id="Ax1.T29.1.65.1" class="ltx_td ltx_align_left">bbh-snarks</td>
<td id="Ax1.T29.1.65.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T29.1.65.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T29.1.65.4" class="ltx_td ltx_nopad_r ltx_align_left">55.06</td>
</tr>
<tr id="Ax1.T29.1.66" class="ltx_tr">
<td id="Ax1.T29.1.66.1" class="ltx_td ltx_align_left">bbh-ruin_names</td>
<td id="Ax1.T29.1.66.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T29.1.66.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T29.1.66.4" class="ltx_td ltx_nopad_r ltx_align_left">36.4</td>
</tr>
<tr id="Ax1.T29.1.67" class="ltx_tr">
<td id="Ax1.T29.1.67.1" class="ltx_td ltx_align_left">bbh-tracking_shuffled_objects_seven_objects</td>
<td id="Ax1.T29.1.67.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T29.1.67.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T29.1.67.4" class="ltx_td ltx_nopad_r ltx_align_left">11.6</td>
</tr>
<tr id="Ax1.T29.1.68" class="ltx_tr">
<td id="Ax1.T29.1.68.1" class="ltx_td ltx_align_left">bbh-tracking_shuffled_objects_five_objects</td>
<td id="Ax1.T29.1.68.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T29.1.68.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T29.1.68.4" class="ltx_td ltx_nopad_r ltx_align_left">18.8</td>
</tr>
<tr id="Ax1.T29.1.69" class="ltx_tr">
<td id="Ax1.T29.1.69.1" class="ltx_td ltx_align_left">bbh-logical_deduction_three_objects</td>
<td id="Ax1.T29.1.69.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T29.1.69.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T29.1.69.4" class="ltx_td ltx_nopad_r ltx_align_left">50.4</td>
</tr>
<tr id="Ax1.T29.1.70" class="ltx_tr">
<td id="Ax1.T29.1.70.1" class="ltx_td ltx_align_left">bbh-hyperbaton</td>
<td id="Ax1.T29.1.70.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T29.1.70.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T29.1.70.4" class="ltx_td ltx_nopad_r ltx_align_left">53.6</td>
</tr>
<tr id="Ax1.T29.1.71" class="ltx_tr">
<td id="Ax1.T29.1.71.1" class="ltx_td ltx_align_left">bbh-logical_deduction_five_objects</td>
<td id="Ax1.T29.1.71.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T29.1.71.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T29.1.71.4" class="ltx_td ltx_nopad_r ltx_align_left">36.8</td>
</tr>
<tr id="Ax1.T29.1.72" class="ltx_tr">
<td id="Ax1.T29.1.72.1" class="ltx_td ltx_align_left">bbh-logical_deduction_seven_objects</td>
<td id="Ax1.T29.1.72.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T29.1.72.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T29.1.72.4" class="ltx_td ltx_nopad_r ltx_align_left">23.2</td>
</tr>
<tr id="Ax1.T29.1.73" class="ltx_tr">
<td id="Ax1.T29.1.73.1" class="ltx_td ltx_align_left">bbh-movie_recommendation</td>
<td id="Ax1.T29.1.73.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T29.1.73.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T29.1.73.4" class="ltx_td ltx_nopad_r ltx_align_left">63.6</td>
</tr>
<tr id="Ax1.T29.1.74" class="ltx_tr">
<td id="Ax1.T29.1.74.1" class="ltx_td ltx_align_left">bbh-salient_translation_error_detection</td>
<td id="Ax1.T29.1.74.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T29.1.74.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T29.1.74.4" class="ltx_td ltx_nopad_r ltx_align_left">21.6</td>
</tr>
<tr id="Ax1.T29.1.75" class="ltx_tr">
<td id="Ax1.T29.1.75.1" class="ltx_td ltx_align_left">bbh-reasoning_about_colored_objects</td>
<td id="Ax1.T29.1.75.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T29.1.75.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T29.1.75.4" class="ltx_td ltx_nopad_r ltx_align_left">42</td>
</tr>
<tr id="Ax1.T29.1.76" class="ltx_tr">
<td id="Ax1.T29.1.76.1" class="ltx_td ltx_align_left">bbh-multistep_arithmetic_two</td>
<td id="Ax1.T29.1.76.2" class="ltx_td ltx_align_left">score</td>
<td id="Ax1.T29.1.76.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T29.1.76.4" class="ltx_td ltx_nopad_r ltx_align_left">1.6</td>
</tr>
<tr id="Ax1.T29.1.77" class="ltx_tr">
<td id="Ax1.T29.1.77.1" class="ltx_td ltx_align_left">bbh-navigate</td>
<td id="Ax1.T29.1.77.2" class="ltx_td ltx_align_left">score</td>
<td id="Ax1.T29.1.77.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T29.1.77.4" class="ltx_td ltx_nopad_r ltx_align_left">61.2</td>
</tr>
<tr id="Ax1.T29.1.78" class="ltx_tr">
<td id="Ax1.T29.1.78.1" class="ltx_td ltx_align_left">bbh-dyck_languages</td>
<td id="Ax1.T29.1.78.2" class="ltx_td ltx_align_left">score</td>
<td id="Ax1.T29.1.78.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T29.1.78.4" class="ltx_td ltx_nopad_r ltx_align_left">6</td>
</tr>
<tr id="Ax1.T29.1.79" class="ltx_tr">
<td id="Ax1.T29.1.79.1" class="ltx_td ltx_align_left">bbh-word_sorting</td>
<td id="Ax1.T29.1.79.2" class="ltx_td ltx_align_left">score</td>
<td id="Ax1.T29.1.79.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T29.1.79.4" class="ltx_td ltx_nopad_r ltx_align_left">11.2</td>
</tr>
<tr id="Ax1.T29.1.80" class="ltx_tr">
<td id="Ax1.T29.1.80.1" class="ltx_td ltx_align_left">bbh-sports_understanding</td>
<td id="Ax1.T29.1.80.2" class="ltx_td ltx_align_left">score</td>
<td id="Ax1.T29.1.80.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T29.1.80.4" class="ltx_td ltx_nopad_r ltx_align_left">90.8</td>
</tr>
<tr id="Ax1.T29.1.81" class="ltx_tr">
<td id="Ax1.T29.1.81.1" class="ltx_td ltx_align_left">bbh-boolean_expressions</td>
<td id="Ax1.T29.1.81.2" class="ltx_td ltx_align_left">score</td>
<td id="Ax1.T29.1.81.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T29.1.81.4" class="ltx_td ltx_nopad_r ltx_align_left">53.2</td>
</tr>
<tr id="Ax1.T29.1.82" class="ltx_tr">
<td id="Ax1.T29.1.82.1" class="ltx_td ltx_align_left">bbh-object_counting</td>
<td id="Ax1.T29.1.82.2" class="ltx_td ltx_align_left">score</td>
<td id="Ax1.T29.1.82.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T29.1.82.4" class="ltx_td ltx_nopad_r ltx_align_left">51.2</td>
</tr>
<tr id="Ax1.T29.1.83" class="ltx_tr">
<td id="Ax1.T29.1.83.1" class="ltx_td ltx_align_left">bbh-formal_fallacies</td>
<td id="Ax1.T29.1.83.2" class="ltx_td ltx_align_left">score</td>
<td id="Ax1.T29.1.83.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T29.1.83.4" class="ltx_td ltx_nopad_r ltx_align_left">53.6</td>
</tr>
<tr id="Ax1.T29.1.84" class="ltx_tr">
<td id="Ax1.T29.1.84.1" class="ltx_td ltx_align_left">bbh-causal_judgement</td>
<td id="Ax1.T29.1.84.2" class="ltx_td ltx_align_left">score</td>
<td id="Ax1.T29.1.84.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T29.1.84.4" class="ltx_td ltx_nopad_r ltx_align_left">55.61</td>
</tr>
<tr id="Ax1.T29.1.85" class="ltx_tr">
<td id="Ax1.T29.1.85.1" class="ltx_td ltx_align_left">bbh-web_of_lies</td>
<td id="Ax1.T29.1.85.2" class="ltx_td ltx_align_left">score</td>
<td id="Ax1.T29.1.85.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T29.1.85.4" class="ltx_td ltx_nopad_r ltx_align_left">56</td>
</tr>
<tr id="Ax1.T29.1.86" class="ltx_tr">
<td id="Ax1.T29.1.86.1" class="ltx_td ltx_align_left">tyidqa-goldp_arabic</td>
<td id="Ax1.T29.1.86.2" class="ltx_td ltx_align_left">exact_match</td>
<td id="Ax1.T29.1.86.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T29.1.86.4" class="ltx_td ltx_nopad_r ltx_align_left">16.4</td>
</tr>
<tr id="Ax1.T29.1.87" class="ltx_tr">
<td id="Ax1.T29.1.87.1" class="ltx_td ltx_align_left">tyidqa-goldp_arabic</td>
<td id="Ax1.T29.1.87.2" class="ltx_td ltx_align_left">f1</td>
<td id="Ax1.T29.1.87.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T29.1.87.4" class="ltx_td ltx_nopad_r ltx_align_left">43.58</td>
</tr>
<tr id="Ax1.T29.1.88" class="ltx_tr">
<td id="Ax1.T29.1.88.1" class="ltx_td ltx_align_left">tyidqa-goldp_bengali</td>
<td id="Ax1.T29.1.88.2" class="ltx_td ltx_align_left">exact_match</td>
<td id="Ax1.T29.1.88.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T29.1.88.4" class="ltx_td ltx_nopad_r ltx_align_left">5.31</td>
</tr>
<tr id="Ax1.T29.1.89" class="ltx_tr">
<td id="Ax1.T29.1.89.1" class="ltx_td ltx_align_left">tyidqa-goldp_bengali</td>
<td id="Ax1.T29.1.89.2" class="ltx_td ltx_align_left">f1</td>
<td id="Ax1.T29.1.89.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T29.1.89.4" class="ltx_td ltx_nopad_r ltx_align_left">16.31</td>
</tr>
<tr id="Ax1.T29.1.90" class="ltx_tr">
<td id="Ax1.T29.1.90.1" class="ltx_td ltx_align_left">tyidqa-goldp_english</td>
<td id="Ax1.T29.1.90.2" class="ltx_td ltx_align_left">exact_match</td>
<td id="Ax1.T29.1.90.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T29.1.90.4" class="ltx_td ltx_nopad_r ltx_align_left">22.5</td>
</tr>
<tr id="Ax1.T29.1.91" class="ltx_tr">
<td id="Ax1.T29.1.91.1" class="ltx_td ltx_align_left">tyidqa-goldp_english</td>
<td id="Ax1.T29.1.91.2" class="ltx_td ltx_align_left">f1</td>
<td id="Ax1.T29.1.91.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T29.1.91.4" class="ltx_td ltx_nopad_r ltx_align_left">33.67</td>
</tr>
<tr id="Ax1.T29.1.92" class="ltx_tr">
<td id="Ax1.T29.1.92.1" class="ltx_td ltx_align_left">tyidqa-goldp_finnish</td>
<td id="Ax1.T29.1.92.2" class="ltx_td ltx_align_left">exact_match</td>
<td id="Ax1.T29.1.92.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T29.1.92.4" class="ltx_td ltx_nopad_r ltx_align_left">34.65</td>
</tr>
<tr id="Ax1.T29.1.93" class="ltx_tr">
<td id="Ax1.T29.1.93.1" class="ltx_td ltx_align_left">tyidqa-goldp_finnish</td>
<td id="Ax1.T29.1.93.2" class="ltx_td ltx_align_left">f1</td>
<td id="Ax1.T29.1.93.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T29.1.93.4" class="ltx_td ltx_nopad_r ltx_align_left">40.59</td>
</tr>
<tr id="Ax1.T29.1.94" class="ltx_tr">
<td id="Ax1.T29.1.94.1" class="ltx_td ltx_align_left">tyidqa-goldp_indonesian</td>
<td id="Ax1.T29.1.94.2" class="ltx_td ltx_align_left">exact_match</td>
<td id="Ax1.T29.1.94.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T29.1.94.4" class="ltx_td ltx_nopad_r ltx_align_left">15.75</td>
</tr>
<tr id="Ax1.T29.1.95" class="ltx_tr">
<td id="Ax1.T29.1.95.1" class="ltx_td ltx_align_left">tyidqa-goldp_indonesian</td>
<td id="Ax1.T29.1.95.2" class="ltx_td ltx_align_left">f1</td>
<td id="Ax1.T29.1.95.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T29.1.95.4" class="ltx_td ltx_nopad_r ltx_align_left">29.62</td>
</tr>
<tr id="Ax1.T29.1.96" class="ltx_tr">
<td id="Ax1.T29.1.96.1" class="ltx_td ltx_align_left">tyidqa-goldp_japanese</td>
<td id="Ax1.T29.1.96.2" class="ltx_td ltx_align_left">exact_match</td>
<td id="Ax1.T29.1.96.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T29.1.96.4" class="ltx_td ltx_nopad_r ltx_align_left">16.92</td>
</tr>
<tr id="Ax1.T29.1.97" class="ltx_tr">
<td id="Ax1.T29.1.97.1" class="ltx_td ltx_align_left">tyidqa-goldp_japanese</td>
<td id="Ax1.T29.1.97.2" class="ltx_td ltx_align_left">f1</td>
<td id="Ax1.T29.1.97.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T29.1.97.4" class="ltx_td ltx_nopad_r ltx_align_left">17.16</td>
</tr>
<tr id="Ax1.T29.1.98" class="ltx_tr">
<td id="Ax1.T29.1.98.1" class="ltx_td ltx_align_left">tyidqa-goldp_korean</td>
<td id="Ax1.T29.1.98.2" class="ltx_td ltx_align_left">exact_match</td>
<td id="Ax1.T29.1.98.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T29.1.98.4" class="ltx_td ltx_nopad_r ltx_align_left">7.61</td>
</tr>
<tr id="Ax1.T29.1.99" class="ltx_tr">
<td id="Ax1.T29.1.99.1" class="ltx_td ltx_align_left">tyidqa-goldp_korean</td>
<td id="Ax1.T29.1.99.2" class="ltx_td ltx_align_left">f1</td>
<td id="Ax1.T29.1.99.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T29.1.99.4" class="ltx_td ltx_nopad_r ltx_align_left">25.71</td>
</tr>
<tr id="Ax1.T29.1.100" class="ltx_tr">
<td id="Ax1.T29.1.100.1" class="ltx_td ltx_align_left">tyidqa-goldp_russian</td>
<td id="Ax1.T29.1.100.2" class="ltx_td ltx_align_left">exact_match</td>
<td id="Ax1.T29.1.100.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T29.1.100.4" class="ltx_td ltx_nopad_r ltx_align_left">15.64</td>
</tr>
<tr id="Ax1.T29.1.101" class="ltx_tr">
<td id="Ax1.T29.1.101.1" class="ltx_td ltx_align_left">tyidqa-goldp_russian</td>
<td id="Ax1.T29.1.101.2" class="ltx_td ltx_align_left">f1</td>
<td id="Ax1.T29.1.101.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T29.1.101.4" class="ltx_td ltx_nopad_r ltx_align_left">28.87</td>
</tr>
<tr id="Ax1.T29.1.102" class="ltx_tr">
<td id="Ax1.T29.1.102.1" class="ltx_td ltx_align_left">tyidqa-goldp_swahili</td>
<td id="Ax1.T29.1.102.2" class="ltx_td ltx_align_left">exact_match</td>
<td id="Ax1.T29.1.102.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T29.1.102.4" class="ltx_td ltx_nopad_r ltx_align_left">19.84</td>
</tr>
<tr id="Ax1.T29.1.103" class="ltx_tr">
<td id="Ax1.T29.1.103.1" class="ltx_td ltx_align_left">tyidqa-goldp_swahili</td>
<td id="Ax1.T29.1.103.2" class="ltx_td ltx_align_left">f1</td>
<td id="Ax1.T29.1.103.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T29.1.103.4" class="ltx_td ltx_nopad_r ltx_align_left">31.71</td>
</tr>
<tr id="Ax1.T29.1.104" class="ltx_tr">
<td id="Ax1.T29.1.104.1" class="ltx_td ltx_align_left">tyidqa-goldp_telugu</td>
<td id="Ax1.T29.1.104.2" class="ltx_td ltx_align_left">exact_match</td>
<td id="Ax1.T29.1.104.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T29.1.104.4" class="ltx_td ltx_nopad_r ltx_align_left">0.45</td>
</tr>
<tr id="Ax1.T29.1.105" class="ltx_tr">
<td id="Ax1.T29.1.105.1" class="ltx_td ltx_align_left">tyidqa-goldp_telugu</td>
<td id="Ax1.T29.1.105.2" class="ltx_td ltx_align_left">f1</td>
<td id="Ax1.T29.1.105.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T29.1.105.4" class="ltx_td ltx_nopad_r ltx_align_left">2.18</td>
</tr>
<tr id="Ax1.T29.1.106" class="ltx_tr">
<td id="Ax1.T29.1.106.1" class="ltx_td ltx_align_left">tyidqa-goldp_thai</td>
<td id="Ax1.T29.1.106.2" class="ltx_td ltx_align_left">exact_match</td>
<td id="Ax1.T29.1.106.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T29.1.106.4" class="ltx_td ltx_nopad_r ltx_align_left">10.13</td>
</tr>
<tr id="Ax1.T29.1.107" class="ltx_tr">
<td id="Ax1.T29.1.107.1" class="ltx_td ltx_align_left">tyidqa-goldp_thai</td>
<td id="Ax1.T29.1.107.2" class="ltx_td ltx_align_left">f1</td>
<td id="Ax1.T29.1.107.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T29.1.107.4" class="ltx_td ltx_nopad_r ltx_align_left">26.75</td>
</tr>
<tr id="Ax1.T29.1.108" class="ltx_tr">
<td id="Ax1.T29.1.108.1" class="ltx_td ltx_align_left">piqa</td>
<td id="Ax1.T29.1.108.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T29.1.108.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T29.1.108.4" class="ltx_td ltx_nopad_r ltx_align_left">78.02</td>
</tr>
<tr id="Ax1.T29.1.109" class="ltx_tr">
<td id="Ax1.T29.1.109.1" class="ltx_td ltx_align_left">siqa</td>
<td id="Ax1.T29.1.109.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T29.1.109.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T29.1.109.4" class="ltx_td ltx_nopad_r ltx_align_left">47.34</td>
</tr>
<tr id="Ax1.T29.1.110" class="ltx_tr">
<td id="Ax1.T29.1.110.1" class="ltx_td ltx_align_left">gsm8k</td>
<td id="Ax1.T29.1.110.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T29.1.110.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T29.1.110.4" class="ltx_td ltx_nopad_r ltx_align_left">24.72</td>
</tr>
<tr id="Ax1.T29.1.111" class="ltx_tr">
<td id="Ax1.T29.1.111.1" class="ltx_td ltx_align_left">truthful_qa</td>
<td id="Ax1.T29.1.111.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T29.1.111.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T29.1.111.4" class="ltx_td ltx_nopad_r ltx_align_left">-</td>
</tr>
<tr id="Ax1.T29.1.112" class="ltx_tr">
<td id="Ax1.T29.1.112.1" class="ltx_td ltx_align_left">triviaqa</td>
<td id="Ax1.T29.1.112.2" class="ltx_td ltx_align_left">score</td>
<td id="Ax1.T29.1.112.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T29.1.112.4" class="ltx_td ltx_nopad_r ltx_align_left">44.38</td>
</tr>
<tr id="Ax1.T29.1.113" class="ltx_tr">
<td id="Ax1.T29.1.113.1" class="ltx_td ltx_align_left ltx_border_bb">BoolQ</td>
<td id="Ax1.T29.1.113.2" class="ltx_td ltx_align_left ltx_border_bb">accuracy</td>
<td id="Ax1.T29.1.113.3" class="ltx_td ltx_align_left ltx_border_bb">ppl</td>
<td id="Ax1.T29.1.113.4" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_bb">68.96</td>
</tr>
</table>
</figure>
<figure id="Ax1.T30" class="ltx_table">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table">Table 30: </span>Detailed results of open-compass toolkit of LLaMA-13b-chat after training on the TRACE sequential tasks.</figcaption>
<table id="Ax1.T30.1" class="ltx_tabular">
<tr id="Ax1.T30.1.1" class="ltx_tr">
<td id="Ax1.T30.1.1.1" class="ltx_td ltx_align_left ltx_border_tt">dataset</td>
<td id="Ax1.T30.1.1.2" class="ltx_td ltx_align_left ltx_border_tt">metric</td>
<td id="Ax1.T30.1.1.3" class="ltx_td ltx_align_left ltx_border_tt">mode</td>
<td id="Ax1.T30.1.1.4" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_tt">performance</td>
</tr>
<tr id="Ax1.T30.1.2" class="ltx_tr">
<td id="Ax1.T30.1.2.1" class="ltx_td ltx_align_left ltx_border_t">lukaemon_mmlu_college_biology</td>
<td id="Ax1.T30.1.2.2" class="ltx_td ltx_align_left ltx_border_t">accuracy</td>
<td id="Ax1.T30.1.2.3" class="ltx_td ltx_align_left ltx_border_t">ppl</td>
<td id="Ax1.T30.1.2.4" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_t">43.06</td>
</tr>
<tr id="Ax1.T30.1.3" class="ltx_tr">
<td id="Ax1.T30.1.3.1" class="ltx_td ltx_align_left">lukaemon_mmlu_college_chemistry</td>
<td id="Ax1.T30.1.3.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T30.1.3.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T30.1.3.4" class="ltx_td ltx_nopad_r ltx_align_left">44</td>
</tr>
<tr id="Ax1.T30.1.4" class="ltx_tr">
<td id="Ax1.T30.1.4.1" class="ltx_td ltx_align_left">lukaemon_mmlu_college_computer_science</td>
<td id="Ax1.T30.1.4.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T30.1.4.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T30.1.4.4" class="ltx_td ltx_nopad_r ltx_align_left">42</td>
</tr>
<tr id="Ax1.T30.1.5" class="ltx_tr">
<td id="Ax1.T30.1.5.1" class="ltx_td ltx_align_left">lukaemon_mmlu_college_mathematics</td>
<td id="Ax1.T30.1.5.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T30.1.5.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T30.1.5.4" class="ltx_td ltx_nopad_r ltx_align_left">31</td>
</tr>
<tr id="Ax1.T30.1.6" class="ltx_tr">
<td id="Ax1.T30.1.6.1" class="ltx_td ltx_align_left">lukaemon_mmlu_college_physics</td>
<td id="Ax1.T30.1.6.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T30.1.6.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T30.1.6.4" class="ltx_td ltx_nopad_r ltx_align_left">44.12</td>
</tr>
<tr id="Ax1.T30.1.7" class="ltx_tr">
<td id="Ax1.T30.1.7.1" class="ltx_td ltx_align_left">lukaemon_mmlu_electrical_engineering</td>
<td id="Ax1.T30.1.7.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T30.1.7.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T30.1.7.4" class="ltx_td ltx_nopad_r ltx_align_left">30.34</td>
</tr>
<tr id="Ax1.T30.1.8" class="ltx_tr">
<td id="Ax1.T30.1.8.1" class="ltx_td ltx_align_left">lukaemon_mmlu_astronomy</td>
<td id="Ax1.T30.1.8.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T30.1.8.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T30.1.8.4" class="ltx_td ltx_nopad_r ltx_align_left">42.11</td>
</tr>
<tr id="Ax1.T30.1.9" class="ltx_tr">
<td id="Ax1.T30.1.9.1" class="ltx_td ltx_align_left">lukaemon_mmlu_anatomy</td>
<td id="Ax1.T30.1.9.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T30.1.9.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T30.1.9.4" class="ltx_td ltx_nopad_r ltx_align_left">34.07</td>
</tr>
<tr id="Ax1.T30.1.10" class="ltx_tr">
<td id="Ax1.T30.1.10.1" class="ltx_td ltx_align_left">lukaemon_mmlu_abstract_algebra</td>
<td id="Ax1.T30.1.10.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T30.1.10.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T30.1.10.4" class="ltx_td ltx_nopad_r ltx_align_left">25</td>
</tr>
<tr id="Ax1.T30.1.11" class="ltx_tr">
<td id="Ax1.T30.1.11.1" class="ltx_td ltx_align_left">lukaemon_mmlu_machine_learning</td>
<td id="Ax1.T30.1.11.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T30.1.11.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T30.1.11.4" class="ltx_td ltx_nopad_r ltx_align_left">24.11</td>
</tr>
<tr id="Ax1.T30.1.12" class="ltx_tr">
<td id="Ax1.T30.1.12.1" class="ltx_td ltx_align_left">lukaemon_mmlu_clinical_knowledge</td>
<td id="Ax1.T30.1.12.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T30.1.12.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T30.1.12.4" class="ltx_td ltx_nopad_r ltx_align_left">51.7</td>
</tr>
<tr id="Ax1.T30.1.13" class="ltx_tr">
<td id="Ax1.T30.1.13.1" class="ltx_td ltx_align_left">lukaemon_mmlu_global_facts</td>
<td id="Ax1.T30.1.13.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T30.1.13.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T30.1.13.4" class="ltx_td ltx_nopad_r ltx_align_left">21</td>
</tr>
<tr id="Ax1.T30.1.14" class="ltx_tr">
<td id="Ax1.T30.1.14.1" class="ltx_td ltx_align_left">lukaemon_mmlu_management</td>
<td id="Ax1.T30.1.14.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T30.1.14.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T30.1.14.4" class="ltx_td ltx_nopad_r ltx_align_left">55.34</td>
</tr>
<tr id="Ax1.T30.1.15" class="ltx_tr">
<td id="Ax1.T30.1.15.1" class="ltx_td ltx_align_left">lukaemon_mmlu_nutrition</td>
<td id="Ax1.T30.1.15.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T30.1.15.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T30.1.15.4" class="ltx_td ltx_nopad_r ltx_align_left">42.48</td>
</tr>
<tr id="Ax1.T30.1.16" class="ltx_tr">
<td id="Ax1.T30.1.16.1" class="ltx_td ltx_align_left">lukaemon_mmlu_marketing</td>
<td id="Ax1.T30.1.16.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T30.1.16.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T30.1.16.4" class="ltx_td ltx_nopad_r ltx_align_left">41.88</td>
</tr>
<tr id="Ax1.T30.1.17" class="ltx_tr">
<td id="Ax1.T30.1.17.1" class="ltx_td ltx_align_left">lukaemon_mmlu_professional_accounting</td>
<td id="Ax1.T30.1.17.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T30.1.17.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T30.1.17.4" class="ltx_td ltx_nopad_r ltx_align_left">32.62</td>
</tr>
<tr id="Ax1.T30.1.18" class="ltx_tr">
<td id="Ax1.T30.1.18.1" class="ltx_td ltx_align_left">lukaemon_mmlu_high_school_geography</td>
<td id="Ax1.T30.1.18.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T30.1.18.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T30.1.18.4" class="ltx_td ltx_nopad_r ltx_align_left">54.04</td>
</tr>
<tr id="Ax1.T30.1.19" class="ltx_tr">
<td id="Ax1.T30.1.19.1" class="ltx_td ltx_align_left">lukaemon_mmlu_international_law</td>
<td id="Ax1.T30.1.19.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T30.1.19.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T30.1.19.4" class="ltx_td ltx_nopad_r ltx_align_left">47.93</td>
</tr>
<tr id="Ax1.T30.1.20" class="ltx_tr">
<td id="Ax1.T30.1.20.1" class="ltx_td ltx_align_left">lukaemon_mmlu_moral_scenarios</td>
<td id="Ax1.T30.1.20.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T30.1.20.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T30.1.20.4" class="ltx_td ltx_nopad_r ltx_align_left">27.26</td>
</tr>
<tr id="Ax1.T30.1.21" class="ltx_tr">
<td id="Ax1.T30.1.21.1" class="ltx_td ltx_align_left">lukaemon_mmlu_computer_security</td>
<td id="Ax1.T30.1.21.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T30.1.21.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T30.1.21.4" class="ltx_td ltx_nopad_r ltx_align_left">45</td>
</tr>
<tr id="Ax1.T30.1.22" class="ltx_tr">
<td id="Ax1.T30.1.22.1" class="ltx_td ltx_align_left">lukaemon_mmlu_high_school_microeconomics</td>
<td id="Ax1.T30.1.22.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T30.1.22.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T30.1.22.4" class="ltx_td ltx_nopad_r ltx_align_left">42.44</td>
</tr>
<tr id="Ax1.T30.1.23" class="ltx_tr">
<td id="Ax1.T30.1.23.1" class="ltx_td ltx_align_left">lukaemon_mmlu_professional_law</td>
<td id="Ax1.T30.1.23.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T30.1.23.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T30.1.23.4" class="ltx_td ltx_nopad_r ltx_align_left">34.49</td>
</tr>
<tr id="Ax1.T30.1.24" class="ltx_tr">
<td id="Ax1.T30.1.24.1" class="ltx_td ltx_align_left">lukaemon_mmlu_medical_genetics</td>
<td id="Ax1.T30.1.24.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T30.1.24.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T30.1.24.4" class="ltx_td ltx_nopad_r ltx_align_left">46</td>
</tr>
<tr id="Ax1.T30.1.25" class="ltx_tr">
<td id="Ax1.T30.1.25.1" class="ltx_td ltx_align_left">lukaemon_mmlu_professional_psychology</td>
<td id="Ax1.T30.1.25.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T30.1.25.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T30.1.25.4" class="ltx_td ltx_nopad_r ltx_align_left">38.56</td>
</tr>
<tr id="Ax1.T30.1.26" class="ltx_tr">
<td id="Ax1.T30.1.26.1" class="ltx_td ltx_align_left">lukaemon_mmlu_jurisprudence</td>
<td id="Ax1.T30.1.26.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T30.1.26.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T30.1.26.4" class="ltx_td ltx_nopad_r ltx_align_left">37.96</td>
</tr>
<tr id="Ax1.T30.1.27" class="ltx_tr">
<td id="Ax1.T30.1.27.1" class="ltx_td ltx_align_left">lukaemon_mmlu_world_religions</td>
<td id="Ax1.T30.1.27.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T30.1.27.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T30.1.27.4" class="ltx_td ltx_nopad_r ltx_align_left">59.65</td>
</tr>
<tr id="Ax1.T30.1.28" class="ltx_tr">
<td id="Ax1.T30.1.28.1" class="ltx_td ltx_align_left">lukaemon_mmlu_philosophy</td>
<td id="Ax1.T30.1.28.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T30.1.28.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T30.1.28.4" class="ltx_td ltx_nopad_r ltx_align_left">45.02</td>
</tr>
<tr id="Ax1.T30.1.29" class="ltx_tr">
<td id="Ax1.T30.1.29.1" class="ltx_td ltx_align_left">lukaemon_mmlu_virology</td>
<td id="Ax1.T30.1.29.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T30.1.29.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T30.1.29.4" class="ltx_td ltx_nopad_r ltx_align_left">32.53</td>
</tr>
<tr id="Ax1.T30.1.30" class="ltx_tr">
<td id="Ax1.T30.1.30.1" class="ltx_td ltx_align_left">lukaemon_mmlu_high_school_chemistry</td>
<td id="Ax1.T30.1.30.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T30.1.30.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T30.1.30.4" class="ltx_td ltx_nopad_r ltx_align_left">29.56</td>
</tr>
<tr id="Ax1.T30.1.31" class="ltx_tr">
<td id="Ax1.T30.1.31.1" class="ltx_td ltx_align_left">lukaemon_mmlu_public_relations</td>
<td id="Ax1.T30.1.31.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T30.1.31.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T30.1.31.4" class="ltx_td ltx_nopad_r ltx_align_left">40</td>
</tr>
<tr id="Ax1.T30.1.32" class="ltx_tr">
<td id="Ax1.T30.1.32.1" class="ltx_td ltx_align_left">lukaemon_mmlu_high_school_macroeconomics</td>
<td id="Ax1.T30.1.32.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T30.1.32.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T30.1.32.4" class="ltx_td ltx_nopad_r ltx_align_left">45.64</td>
</tr>
<tr id="Ax1.T30.1.33" class="ltx_tr">
<td id="Ax1.T30.1.33.1" class="ltx_td ltx_align_left">lukaemon_mmlu_human_sexuality</td>
<td id="Ax1.T30.1.33.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T30.1.33.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T30.1.33.4" class="ltx_td ltx_nopad_r ltx_align_left">43.51</td>
</tr>
<tr id="Ax1.T30.1.34" class="ltx_tr">
<td id="Ax1.T30.1.34.1" class="ltx_td ltx_align_left">lukaemon_mmlu_elementary_mathematics</td>
<td id="Ax1.T30.1.34.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T30.1.34.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T30.1.34.4" class="ltx_td ltx_nopad_r ltx_align_left">26.46</td>
</tr>
<tr id="Ax1.T30.1.35" class="ltx_tr">
<td id="Ax1.T30.1.35.1" class="ltx_td ltx_align_left">lukaemon_mmlu_high_school_physics</td>
<td id="Ax1.T30.1.35.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T30.1.35.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T30.1.35.4" class="ltx_td ltx_nopad_r ltx_align_left">33.11</td>
</tr>
<tr id="Ax1.T30.1.36" class="ltx_tr">
<td id="Ax1.T30.1.36.1" class="ltx_td ltx_align_left">lukaemon_mmlu_high_school_computer_science</td>
<td id="Ax1.T30.1.36.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T30.1.36.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T30.1.36.4" class="ltx_td ltx_nopad_r ltx_align_left">37</td>
</tr>
<tr id="Ax1.T30.1.37" class="ltx_tr">
<td id="Ax1.T30.1.37.1" class="ltx_td ltx_align_left">lukaemon_mmlu_high_school_european_history</td>
<td id="Ax1.T30.1.37.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T30.1.37.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T30.1.37.4" class="ltx_td ltx_nopad_r ltx_align_left">52.73</td>
</tr>
<tr id="Ax1.T30.1.38" class="ltx_tr">
<td id="Ax1.T30.1.38.1" class="ltx_td ltx_align_left">lukaemon_mmlu_business_ethics</td>
<td id="Ax1.T30.1.38.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T30.1.38.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T30.1.38.4" class="ltx_td ltx_nopad_r ltx_align_left">36</td>
</tr>
<tr id="Ax1.T30.1.39" class="ltx_tr">
<td id="Ax1.T30.1.39.1" class="ltx_td ltx_align_left">lukaemon_mmlu_moral_disputes</td>
<td id="Ax1.T30.1.39.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T30.1.39.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T30.1.39.4" class="ltx_td ltx_nopad_r ltx_align_left">35.84</td>
</tr>
<tr id="Ax1.T30.1.40" class="ltx_tr">
<td id="Ax1.T30.1.40.1" class="ltx_td ltx_align_left">lukaemon_mmlu_high_school_statistics</td>
<td id="Ax1.T30.1.40.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T30.1.40.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T30.1.40.4" class="ltx_td ltx_nopad_r ltx_align_left">46.76</td>
</tr>
<tr id="Ax1.T30.1.41" class="ltx_tr">
<td id="Ax1.T30.1.41.1" class="ltx_td ltx_align_left">lukaemon_mmlu_miscellaneous</td>
<td id="Ax1.T30.1.41.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T30.1.41.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T30.1.41.4" class="ltx_td ltx_nopad_r ltx_align_left">56.07</td>
</tr>
<tr id="Ax1.T30.1.42" class="ltx_tr">
<td id="Ax1.T30.1.42.1" class="ltx_td ltx_align_left">lukaemon_mmlu_formal_logic</td>
<td id="Ax1.T30.1.42.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T30.1.42.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T30.1.42.4" class="ltx_td ltx_nopad_r ltx_align_left">36.51</td>
</tr>
<tr id="Ax1.T30.1.43" class="ltx_tr">
<td id="Ax1.T30.1.43.1" class="ltx_td ltx_align_left">lukaemon_mmlu_high_school_government_and_politics</td>
<td id="Ax1.T30.1.43.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T30.1.43.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T30.1.43.4" class="ltx_td ltx_nopad_r ltx_align_left">58.03</td>
</tr>
<tr id="Ax1.T30.1.44" class="ltx_tr">
<td id="Ax1.T30.1.44.1" class="ltx_td ltx_align_left">lukaemon_mmlu_prehistory</td>
<td id="Ax1.T30.1.44.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T30.1.44.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T30.1.44.4" class="ltx_td ltx_nopad_r ltx_align_left">50</td>
</tr>
<tr id="Ax1.T30.1.45" class="ltx_tr">
<td id="Ax1.T30.1.45.1" class="ltx_td ltx_align_left">lukaemon_mmlu_security_studies</td>
<td id="Ax1.T30.1.45.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T30.1.45.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T30.1.45.4" class="ltx_td ltx_nopad_r ltx_align_left">50.2</td>
</tr>
<tr id="Ax1.T30.1.46" class="ltx_tr">
<td id="Ax1.T30.1.46.1" class="ltx_td ltx_align_left">lukaemon_mmlu_high_school_biology</td>
<td id="Ax1.T30.1.46.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T30.1.46.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T30.1.46.4" class="ltx_td ltx_nopad_r ltx_align_left">50.65</td>
</tr>
<tr id="Ax1.T30.1.47" class="ltx_tr">
<td id="Ax1.T30.1.47.1" class="ltx_td ltx_align_left">lukaemon_mmlu_logical_fallacies</td>
<td id="Ax1.T30.1.47.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T30.1.47.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T30.1.47.4" class="ltx_td ltx_nopad_r ltx_align_left">39.26</td>
</tr>
<tr id="Ax1.T30.1.48" class="ltx_tr">
<td id="Ax1.T30.1.48.1" class="ltx_td ltx_align_left">lukaemon_mmlu_high_school_world_history</td>
<td id="Ax1.T30.1.48.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T30.1.48.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T30.1.48.4" class="ltx_td ltx_nopad_r ltx_align_left">61.18</td>
</tr>
<tr id="Ax1.T30.1.49" class="ltx_tr">
<td id="Ax1.T30.1.49.1" class="ltx_td ltx_align_left">lukaemon_mmlu_professional_medicine</td>
<td id="Ax1.T30.1.49.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T30.1.49.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T30.1.49.4" class="ltx_td ltx_nopad_r ltx_align_left">46.69</td>
</tr>
<tr id="Ax1.T30.1.50" class="ltx_tr">
<td id="Ax1.T30.1.50.1" class="ltx_td ltx_align_left">lukaemon_mmlu_high_school_mathematics</td>
<td id="Ax1.T30.1.50.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T30.1.50.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T30.1.50.4" class="ltx_td ltx_nopad_r ltx_align_left">26.3</td>
</tr>
<tr id="Ax1.T30.1.51" class="ltx_tr">
<td id="Ax1.T30.1.51.1" class="ltx_td ltx_align_left">lukaemon_mmlu_college_medicine</td>
<td id="Ax1.T30.1.51.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T30.1.51.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T30.1.51.4" class="ltx_td ltx_nopad_r ltx_align_left">44.51</td>
</tr>
<tr id="Ax1.T30.1.52" class="ltx_tr">
<td id="Ax1.T30.1.52.1" class="ltx_td ltx_align_left">lukaemon_mmlu_high_school_us_history</td>
<td id="Ax1.T30.1.52.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T30.1.52.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T30.1.52.4" class="ltx_td ltx_nopad_r ltx_align_left">59.31</td>
</tr>
<tr id="Ax1.T30.1.53" class="ltx_tr">
<td id="Ax1.T30.1.53.1" class="ltx_td ltx_align_left">lukaemon_mmlu_sociology</td>
<td id="Ax1.T30.1.53.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T30.1.53.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T30.1.53.4" class="ltx_td ltx_nopad_r ltx_align_left">63.18</td>
</tr>
<tr id="Ax1.T30.1.54" class="ltx_tr">
<td id="Ax1.T30.1.54.1" class="ltx_td ltx_align_left">lukaemon_mmlu_econometrics</td>
<td id="Ax1.T30.1.54.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T30.1.54.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T30.1.54.4" class="ltx_td ltx_nopad_r ltx_align_left">25.44</td>
</tr>
<tr id="Ax1.T30.1.55" class="ltx_tr">
<td id="Ax1.T30.1.55.1" class="ltx_td ltx_align_left">lukaemon_mmlu_high_school_psychology</td>
<td id="Ax1.T30.1.55.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T30.1.55.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T30.1.55.4" class="ltx_td ltx_nopad_r ltx_align_left">62.75</td>
</tr>
<tr id="Ax1.T30.1.56" class="ltx_tr">
<td id="Ax1.T30.1.56.1" class="ltx_td ltx_align_left">lukaemon_mmlu_human_aging</td>
<td id="Ax1.T30.1.56.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T30.1.56.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T30.1.56.4" class="ltx_td ltx_nopad_r ltx_align_left">37.22</td>
</tr>
<tr id="Ax1.T30.1.57" class="ltx_tr">
<td id="Ax1.T30.1.57.1" class="ltx_td ltx_align_left">lukaemon_mmlu_us_foreign_policy</td>
<td id="Ax1.T30.1.57.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T30.1.57.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T30.1.57.4" class="ltx_td ltx_nopad_r ltx_align_left">49</td>
</tr>
<tr id="Ax1.T30.1.58" class="ltx_tr">
<td id="Ax1.T30.1.58.1" class="ltx_td ltx_align_left">lukaemon_mmlu_conceptual_physics</td>
<td id="Ax1.T30.1.58.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T30.1.58.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T30.1.58.4" class="ltx_td ltx_nopad_r ltx_align_left">28.51</td>
</tr>
<tr id="Ax1.T30.1.59" class="ltx_tr">
<td id="Ax1.T30.1.59.1" class="ltx_td ltx_align_left">bbh-temporal_sequences</td>
<td id="Ax1.T30.1.59.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T30.1.59.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T30.1.59.4" class="ltx_td ltx_nopad_r ltx_align_left">0</td>
</tr>
<tr id="Ax1.T30.1.60" class="ltx_tr">
<td id="Ax1.T30.1.60.1" class="ltx_td ltx_align_left">bbh-disambiguation_qa</td>
<td id="Ax1.T30.1.60.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T30.1.60.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T30.1.60.4" class="ltx_td ltx_nopad_r ltx_align_left">47.6</td>
</tr>
<tr id="Ax1.T30.1.61" class="ltx_tr">
<td id="Ax1.T30.1.61.1" class="ltx_td ltx_align_left">bbh-date_understanding</td>
<td id="Ax1.T30.1.61.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T30.1.61.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T30.1.61.4" class="ltx_td ltx_nopad_r ltx_align_left">40</td>
</tr>
<tr id="Ax1.T30.1.62" class="ltx_tr">
<td id="Ax1.T30.1.62.1" class="ltx_td ltx_align_left">bbh-tracking_shuffled_objects_three_objects</td>
<td id="Ax1.T30.1.62.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T30.1.62.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T30.1.62.4" class="ltx_td ltx_nopad_r ltx_align_left">31.6</td>
</tr>
<tr id="Ax1.T30.1.63" class="ltx_tr">
<td id="Ax1.T30.1.63.1" class="ltx_td ltx_align_left">bbh-penguins_in_a_table</td>
<td id="Ax1.T30.1.63.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T30.1.63.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T30.1.63.4" class="ltx_td ltx_nopad_r ltx_align_left">8.22</td>
</tr>
<tr id="Ax1.T30.1.64" class="ltx_tr">
<td id="Ax1.T30.1.64.1" class="ltx_td ltx_align_left">bbh-geometric_shapes</td>
<td id="Ax1.T30.1.64.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T30.1.64.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T30.1.64.4" class="ltx_td ltx_nopad_r ltx_align_left">0</td>
</tr>
<tr id="Ax1.T30.1.65" class="ltx_tr">
<td id="Ax1.T30.1.65.1" class="ltx_td ltx_align_left">bbh-snarks</td>
<td id="Ax1.T30.1.65.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T30.1.65.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T30.1.65.4" class="ltx_td ltx_nopad_r ltx_align_left">44.94</td>
</tr>
<tr id="Ax1.T30.1.66" class="ltx_tr">
<td id="Ax1.T30.1.66.1" class="ltx_td ltx_align_left">bbh-ruin_names</td>
<td id="Ax1.T30.1.66.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T30.1.66.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T30.1.66.4" class="ltx_td ltx_nopad_r ltx_align_left">28.4</td>
</tr>
<tr id="Ax1.T30.1.67" class="ltx_tr">
<td id="Ax1.T30.1.67.1" class="ltx_td ltx_align_left">bbh-tracking_shuffled_objects_seven_objects</td>
<td id="Ax1.T30.1.67.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T30.1.67.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T30.1.67.4" class="ltx_td ltx_nopad_r ltx_align_left">14.4</td>
</tr>
<tr id="Ax1.T30.1.68" class="ltx_tr">
<td id="Ax1.T30.1.68.1" class="ltx_td ltx_align_left">bbh-tracking_shuffled_objects_five_objects</td>
<td id="Ax1.T30.1.68.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T30.1.68.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T30.1.68.4" class="ltx_td ltx_nopad_r ltx_align_left">20</td>
</tr>
<tr id="Ax1.T30.1.69" class="ltx_tr">
<td id="Ax1.T30.1.69.1" class="ltx_td ltx_align_left">bbh-logical_deduction_three_objects</td>
<td id="Ax1.T30.1.69.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T30.1.69.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T30.1.69.4" class="ltx_td ltx_nopad_r ltx_align_left">1.6</td>
</tr>
<tr id="Ax1.T30.1.70" class="ltx_tr">
<td id="Ax1.T30.1.70.1" class="ltx_td ltx_align_left">bbh-hyperbaton</td>
<td id="Ax1.T30.1.70.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T30.1.70.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T30.1.70.4" class="ltx_td ltx_nopad_r ltx_align_left">54</td>
</tr>
<tr id="Ax1.T30.1.71" class="ltx_tr">
<td id="Ax1.T30.1.71.1" class="ltx_td ltx_align_left">bbh-logical_deduction_five_objects</td>
<td id="Ax1.T30.1.71.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T30.1.71.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T30.1.71.4" class="ltx_td ltx_nopad_r ltx_align_left">2.4</td>
</tr>
<tr id="Ax1.T30.1.72" class="ltx_tr">
<td id="Ax1.T30.1.72.1" class="ltx_td ltx_align_left">bbh-logical_deduction_seven_objects</td>
<td id="Ax1.T30.1.72.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T30.1.72.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T30.1.72.4" class="ltx_td ltx_nopad_r ltx_align_left">2</td>
</tr>
<tr id="Ax1.T30.1.73" class="ltx_tr">
<td id="Ax1.T30.1.73.1" class="ltx_td ltx_align_left">bbh-movie_recommendation</td>
<td id="Ax1.T30.1.73.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T30.1.73.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T30.1.73.4" class="ltx_td ltx_nopad_r ltx_align_left">7.2</td>
</tr>
<tr id="Ax1.T30.1.74" class="ltx_tr">
<td id="Ax1.T30.1.74.1" class="ltx_td ltx_align_left">bbh-salient_translation_error_detection</td>
<td id="Ax1.T30.1.74.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T30.1.74.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T30.1.74.4" class="ltx_td ltx_nopad_r ltx_align_left">23.2</td>
</tr>
<tr id="Ax1.T30.1.75" class="ltx_tr">
<td id="Ax1.T30.1.75.1" class="ltx_td ltx_align_left">bbh-reasoning_about_colored_objects</td>
<td id="Ax1.T30.1.75.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T30.1.75.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T30.1.75.4" class="ltx_td ltx_nopad_r ltx_align_left">34.8</td>
</tr>
<tr id="Ax1.T30.1.76" class="ltx_tr">
<td id="Ax1.T30.1.76.1" class="ltx_td ltx_align_left">bbh-multistep_arithmetic_two</td>
<td id="Ax1.T30.1.76.2" class="ltx_td ltx_align_left">score</td>
<td id="Ax1.T30.1.76.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T30.1.76.4" class="ltx_td ltx_nopad_r ltx_align_left">0</td>
</tr>
<tr id="Ax1.T30.1.77" class="ltx_tr">
<td id="Ax1.T30.1.77.1" class="ltx_td ltx_align_left">bbh-navigate</td>
<td id="Ax1.T30.1.77.2" class="ltx_td ltx_align_left">score</td>
<td id="Ax1.T30.1.77.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T30.1.77.4" class="ltx_td ltx_nopad_r ltx_align_left">4</td>
</tr>
<tr id="Ax1.T30.1.78" class="ltx_tr">
<td id="Ax1.T30.1.78.1" class="ltx_td ltx_align_left">bbh-dyck_languages</td>
<td id="Ax1.T30.1.78.2" class="ltx_td ltx_align_left">score</td>
<td id="Ax1.T30.1.78.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T30.1.78.4" class="ltx_td ltx_nopad_r ltx_align_left">0</td>
</tr>
<tr id="Ax1.T30.1.79" class="ltx_tr">
<td id="Ax1.T30.1.79.1" class="ltx_td ltx_align_left">bbh-word_sorting</td>
<td id="Ax1.T30.1.79.2" class="ltx_td ltx_align_left">score</td>
<td id="Ax1.T30.1.79.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T30.1.79.4" class="ltx_td ltx_nopad_r ltx_align_left">2.4</td>
</tr>
<tr id="Ax1.T30.1.80" class="ltx_tr">
<td id="Ax1.T30.1.80.1" class="ltx_td ltx_align_left">bbh-sports_understanding</td>
<td id="Ax1.T30.1.80.2" class="ltx_td ltx_align_left">score</td>
<td id="Ax1.T30.1.80.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T30.1.80.4" class="ltx_td ltx_nopad_r ltx_align_left">67.2</td>
</tr>
<tr id="Ax1.T30.1.81" class="ltx_tr">
<td id="Ax1.T30.1.81.1" class="ltx_td ltx_align_left">bbh-boolean_expressions</td>
<td id="Ax1.T30.1.81.2" class="ltx_td ltx_align_left">score</td>
<td id="Ax1.T30.1.81.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T30.1.81.4" class="ltx_td ltx_nopad_r ltx_align_left">29.6</td>
</tr>
<tr id="Ax1.T30.1.82" class="ltx_tr">
<td id="Ax1.T30.1.82.1" class="ltx_td ltx_align_left">bbh-object_counting</td>
<td id="Ax1.T30.1.82.2" class="ltx_td ltx_align_left">score</td>
<td id="Ax1.T30.1.82.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T30.1.82.4" class="ltx_td ltx_nopad_r ltx_align_left">0</td>
</tr>
<tr id="Ax1.T30.1.83" class="ltx_tr">
<td id="Ax1.T30.1.83.1" class="ltx_td ltx_align_left">bbh-formal_fallacies</td>
<td id="Ax1.T30.1.83.2" class="ltx_td ltx_align_left">score</td>
<td id="Ax1.T30.1.83.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T30.1.83.4" class="ltx_td ltx_nopad_r ltx_align_left">17.2</td>
</tr>
<tr id="Ax1.T30.1.84" class="ltx_tr">
<td id="Ax1.T30.1.84.1" class="ltx_td ltx_align_left">bbh-causal_judgement</td>
<td id="Ax1.T30.1.84.2" class="ltx_td ltx_align_left">score</td>
<td id="Ax1.T30.1.84.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T30.1.84.4" class="ltx_td ltx_nopad_r ltx_align_left">40.64</td>
</tr>
<tr id="Ax1.T30.1.85" class="ltx_tr">
<td id="Ax1.T30.1.85.1" class="ltx_td ltx_align_left">bbh-web_of_lies</td>
<td id="Ax1.T30.1.85.2" class="ltx_td ltx_align_left">score</td>
<td id="Ax1.T30.1.85.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T30.1.85.4" class="ltx_td ltx_nopad_r ltx_align_left">4.4</td>
</tr>
<tr id="Ax1.T30.1.86" class="ltx_tr">
<td id="Ax1.T30.1.86.1" class="ltx_td ltx_align_left">tyidqa-goldp_arabic</td>
<td id="Ax1.T30.1.86.2" class="ltx_td ltx_align_left">exact_match</td>
<td id="Ax1.T30.1.86.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T30.1.86.4" class="ltx_td ltx_nopad_r ltx_align_left">24.54</td>
</tr>
<tr id="Ax1.T30.1.87" class="ltx_tr">
<td id="Ax1.T30.1.87.1" class="ltx_td ltx_align_left">tyidqa-goldp_arabic</td>
<td id="Ax1.T30.1.87.2" class="ltx_td ltx_align_left">f1</td>
<td id="Ax1.T30.1.87.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T30.1.87.4" class="ltx_td ltx_nopad_r ltx_align_left">55.34</td>
</tr>
<tr id="Ax1.T30.1.88" class="ltx_tr">
<td id="Ax1.T30.1.88.1" class="ltx_td ltx_align_left">tyidqa-goldp_bengali</td>
<td id="Ax1.T30.1.88.2" class="ltx_td ltx_align_left">exact_match</td>
<td id="Ax1.T30.1.88.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T30.1.88.4" class="ltx_td ltx_nopad_r ltx_align_left">23.89</td>
</tr>
<tr id="Ax1.T30.1.89" class="ltx_tr">
<td id="Ax1.T30.1.89.1" class="ltx_td ltx_align_left">tyidqa-goldp_bengali</td>
<td id="Ax1.T30.1.89.2" class="ltx_td ltx_align_left">f1</td>
<td id="Ax1.T30.1.89.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T30.1.89.4" class="ltx_td ltx_nopad_r ltx_align_left">34.8</td>
</tr>
<tr id="Ax1.T30.1.90" class="ltx_tr">
<td id="Ax1.T30.1.90.1" class="ltx_td ltx_align_left">tyidqa-goldp_english</td>
<td id="Ax1.T30.1.90.2" class="ltx_td ltx_align_left">exact_match</td>
<td id="Ax1.T30.1.90.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T30.1.90.4" class="ltx_td ltx_nopad_r ltx_align_left">19.32</td>
</tr>
<tr id="Ax1.T30.1.91" class="ltx_tr">
<td id="Ax1.T30.1.91.1" class="ltx_td ltx_align_left">tyidqa-goldp_english</td>
<td id="Ax1.T30.1.91.2" class="ltx_td ltx_align_left">f1</td>
<td id="Ax1.T30.1.91.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T30.1.91.4" class="ltx_td ltx_nopad_r ltx_align_left">28.61</td>
</tr>
<tr id="Ax1.T30.1.92" class="ltx_tr">
<td id="Ax1.T30.1.92.1" class="ltx_td ltx_align_left">tyidqa-goldp_finnish</td>
<td id="Ax1.T30.1.92.2" class="ltx_td ltx_align_left">exact_match</td>
<td id="Ax1.T30.1.92.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T30.1.92.4" class="ltx_td ltx_nopad_r ltx_align_left">32.23</td>
</tr>
<tr id="Ax1.T30.1.93" class="ltx_tr">
<td id="Ax1.T30.1.93.1" class="ltx_td ltx_align_left">tyidqa-goldp_finnish</td>
<td id="Ax1.T30.1.93.2" class="ltx_td ltx_align_left">f1</td>
<td id="Ax1.T30.1.93.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T30.1.93.4" class="ltx_td ltx_nopad_r ltx_align_left">38.55</td>
</tr>
<tr id="Ax1.T30.1.94" class="ltx_tr">
<td id="Ax1.T30.1.94.1" class="ltx_td ltx_align_left">tyidqa-goldp_indonesian</td>
<td id="Ax1.T30.1.94.2" class="ltx_td ltx_align_left">exact_match</td>
<td id="Ax1.T30.1.94.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T30.1.94.4" class="ltx_td ltx_nopad_r ltx_align_left">15.04</td>
</tr>
<tr id="Ax1.T30.1.95" class="ltx_tr">
<td id="Ax1.T30.1.95.1" class="ltx_td ltx_align_left">tyidqa-goldp_indonesian</td>
<td id="Ax1.T30.1.95.2" class="ltx_td ltx_align_left">f1</td>
<td id="Ax1.T30.1.95.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T30.1.95.4" class="ltx_td ltx_nopad_r ltx_align_left">30.68</td>
</tr>
<tr id="Ax1.T30.1.96" class="ltx_tr">
<td id="Ax1.T30.1.96.1" class="ltx_td ltx_align_left">tyidqa-goldp_japanese</td>
<td id="Ax1.T30.1.96.2" class="ltx_td ltx_align_left">exact_match</td>
<td id="Ax1.T30.1.96.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T30.1.96.4" class="ltx_td ltx_nopad_r ltx_align_left">19.78</td>
</tr>
<tr id="Ax1.T30.1.97" class="ltx_tr">
<td id="Ax1.T30.1.97.1" class="ltx_td ltx_align_left">tyidqa-goldp_japanese</td>
<td id="Ax1.T30.1.97.2" class="ltx_td ltx_align_left">f1</td>
<td id="Ax1.T30.1.97.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T30.1.97.4" class="ltx_td ltx_nopad_r ltx_align_left">20</td>
</tr>
<tr id="Ax1.T30.1.98" class="ltx_tr">
<td id="Ax1.T30.1.98.1" class="ltx_td ltx_align_left">tyidqa-goldp_korean</td>
<td id="Ax1.T30.1.98.2" class="ltx_td ltx_align_left">exact_match</td>
<td id="Ax1.T30.1.98.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T30.1.98.4" class="ltx_td ltx_nopad_r ltx_align_left">40.22</td>
</tr>
<tr id="Ax1.T30.1.99" class="ltx_tr">
<td id="Ax1.T30.1.99.1" class="ltx_td ltx_align_left">tyidqa-goldp_korean</td>
<td id="Ax1.T30.1.99.2" class="ltx_td ltx_align_left">f1</td>
<td id="Ax1.T30.1.99.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T30.1.99.4" class="ltx_td ltx_nopad_r ltx_align_left">50.23</td>
</tr>
<tr id="Ax1.T30.1.100" class="ltx_tr">
<td id="Ax1.T30.1.100.1" class="ltx_td ltx_align_left">tyidqa-goldp_russian</td>
<td id="Ax1.T30.1.100.2" class="ltx_td ltx_align_left">exact_match</td>
<td id="Ax1.T30.1.100.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T30.1.100.4" class="ltx_td ltx_nopad_r ltx_align_left">22.54</td>
</tr>
<tr id="Ax1.T30.1.101" class="ltx_tr">
<td id="Ax1.T30.1.101.1" class="ltx_td ltx_align_left">tyidqa-goldp_russian</td>
<td id="Ax1.T30.1.101.2" class="ltx_td ltx_align_left">f1</td>
<td id="Ax1.T30.1.101.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T30.1.101.4" class="ltx_td ltx_nopad_r ltx_align_left">30.59</td>
</tr>
<tr id="Ax1.T30.1.102" class="ltx_tr">
<td id="Ax1.T30.1.102.1" class="ltx_td ltx_align_left">tyidqa-goldp_swahili</td>
<td id="Ax1.T30.1.102.2" class="ltx_td ltx_align_left">exact_match</td>
<td id="Ax1.T30.1.102.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T30.1.102.4" class="ltx_td ltx_nopad_r ltx_align_left">20.64</td>
</tr>
<tr id="Ax1.T30.1.103" class="ltx_tr">
<td id="Ax1.T30.1.103.1" class="ltx_td ltx_align_left">tyidqa-goldp_swahili</td>
<td id="Ax1.T30.1.103.2" class="ltx_td ltx_align_left">f1</td>
<td id="Ax1.T30.1.103.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T30.1.103.4" class="ltx_td ltx_nopad_r ltx_align_left">31.5</td>
</tr>
<tr id="Ax1.T30.1.104" class="ltx_tr">
<td id="Ax1.T30.1.104.1" class="ltx_td ltx_align_left">tyidqa-goldp_telugu</td>
<td id="Ax1.T30.1.104.2" class="ltx_td ltx_align_left">exact_match</td>
<td id="Ax1.T30.1.104.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T30.1.104.4" class="ltx_td ltx_nopad_r ltx_align_left">0.6</td>
</tr>
<tr id="Ax1.T30.1.105" class="ltx_tr">
<td id="Ax1.T30.1.105.1" class="ltx_td ltx_align_left">tyidqa-goldp_telugu</td>
<td id="Ax1.T30.1.105.2" class="ltx_td ltx_align_left">f1</td>
<td id="Ax1.T30.1.105.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T30.1.105.4" class="ltx_td ltx_nopad_r ltx_align_left">3.22</td>
</tr>
<tr id="Ax1.T30.1.106" class="ltx_tr">
<td id="Ax1.T30.1.106.1" class="ltx_td ltx_align_left">tyidqa-goldp_thai</td>
<td id="Ax1.T30.1.106.2" class="ltx_td ltx_align_left">exact_match</td>
<td id="Ax1.T30.1.106.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T30.1.106.4" class="ltx_td ltx_nopad_r ltx_align_left">17.59</td>
</tr>
<tr id="Ax1.T30.1.107" class="ltx_tr">
<td id="Ax1.T30.1.107.1" class="ltx_td ltx_align_left">tyidqa-goldp_thai</td>
<td id="Ax1.T30.1.107.2" class="ltx_td ltx_align_left">f1</td>
<td id="Ax1.T30.1.107.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T30.1.107.4" class="ltx_td ltx_nopad_r ltx_align_left">31.49</td>
</tr>
<tr id="Ax1.T30.1.108" class="ltx_tr">
<td id="Ax1.T30.1.108.1" class="ltx_td ltx_align_left">piqa</td>
<td id="Ax1.T30.1.108.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T30.1.108.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T30.1.108.4" class="ltx_td ltx_nopad_r ltx_align_left">77.15</td>
</tr>
<tr id="Ax1.T30.1.109" class="ltx_tr">
<td id="Ax1.T30.1.109.1" class="ltx_td ltx_align_left">siqa</td>
<td id="Ax1.T30.1.109.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T30.1.109.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T30.1.109.4" class="ltx_td ltx_nopad_r ltx_align_left">48.67</td>
</tr>
<tr id="Ax1.T30.1.110" class="ltx_tr">
<td id="Ax1.T30.1.110.1" class="ltx_td ltx_align_left">gsm8k</td>
<td id="Ax1.T30.1.110.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T30.1.110.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T30.1.110.4" class="ltx_td ltx_nopad_r ltx_align_left">2.12</td>
</tr>
<tr id="Ax1.T30.1.111" class="ltx_tr">
<td id="Ax1.T30.1.111.1" class="ltx_td ltx_align_left">truthful_qa</td>
<td id="Ax1.T30.1.111.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T30.1.111.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T30.1.111.4" class="ltx_td ltx_nopad_r ltx_align_left">-</td>
</tr>
<tr id="Ax1.T30.1.112" class="ltx_tr">
<td id="Ax1.T30.1.112.1" class="ltx_td ltx_align_left">triviaqa</td>
<td id="Ax1.T30.1.112.2" class="ltx_td ltx_align_left">score</td>
<td id="Ax1.T30.1.112.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T30.1.112.4" class="ltx_td ltx_nopad_r ltx_align_left">48.29</td>
</tr>
<tr id="Ax1.T30.1.113" class="ltx_tr">
<td id="Ax1.T30.1.113.1" class="ltx_td ltx_align_left ltx_border_bb">BoolQ</td>
<td id="Ax1.T30.1.113.2" class="ltx_td ltx_align_left ltx_border_bb">accuracy</td>
<td id="Ax1.T30.1.113.3" class="ltx_td ltx_align_left ltx_border_bb">ppl</td>
<td id="Ax1.T30.1.113.4" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_bb">82.08</td>
</tr>
</table>
</figure>
<figure id="Ax1.T31" class="ltx_table">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table">Table 31: </span>Detailed results of open-compass toolkit of LLaMA-13b-chat after training on the TRACE sequential tasks with replay data.</figcaption>
<table id="Ax1.T31.1" class="ltx_tabular">
<tr id="Ax1.T31.1.1" class="ltx_tr">
<td id="Ax1.T31.1.1.1" class="ltx_td ltx_align_left ltx_border_tt">dataset</td>
<td id="Ax1.T31.1.1.2" class="ltx_td ltx_align_left ltx_border_tt">metric</td>
<td id="Ax1.T31.1.1.3" class="ltx_td ltx_align_left ltx_border_tt">mode</td>
<td id="Ax1.T31.1.1.4" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_tt">performance</td>
</tr>
<tr id="Ax1.T31.1.2" class="ltx_tr">
<td id="Ax1.T31.1.2.1" class="ltx_td ltx_align_left ltx_border_t">lukaemon_mmlu_college_biology</td>
<td id="Ax1.T31.1.2.2" class="ltx_td ltx_align_left ltx_border_t">accuracy</td>
<td id="Ax1.T31.1.2.3" class="ltx_td ltx_align_left ltx_border_t">ppl</td>
<td id="Ax1.T31.1.2.4" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_t">54.17</td>
</tr>
<tr id="Ax1.T31.1.3" class="ltx_tr">
<td id="Ax1.T31.1.3.1" class="ltx_td ltx_align_left">lukaemon_mmlu_college_chemistry</td>
<td id="Ax1.T31.1.3.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T31.1.3.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T31.1.3.4" class="ltx_td ltx_nopad_r ltx_align_left">44</td>
</tr>
<tr id="Ax1.T31.1.4" class="ltx_tr">
<td id="Ax1.T31.1.4.1" class="ltx_td ltx_align_left">lukaemon_mmlu_college_computer_science</td>
<td id="Ax1.T31.1.4.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T31.1.4.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T31.1.4.4" class="ltx_td ltx_nopad_r ltx_align_left">38</td>
</tr>
<tr id="Ax1.T31.1.5" class="ltx_tr">
<td id="Ax1.T31.1.5.1" class="ltx_td ltx_align_left">lukaemon_mmlu_college_mathematics</td>
<td id="Ax1.T31.1.5.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T31.1.5.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T31.1.5.4" class="ltx_td ltx_nopad_r ltx_align_left">30</td>
</tr>
<tr id="Ax1.T31.1.6" class="ltx_tr">
<td id="Ax1.T31.1.6.1" class="ltx_td ltx_align_left">lukaemon_mmlu_college_physics</td>
<td id="Ax1.T31.1.6.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T31.1.6.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T31.1.6.4" class="ltx_td ltx_nopad_r ltx_align_left">30.39</td>
</tr>
<tr id="Ax1.T31.1.7" class="ltx_tr">
<td id="Ax1.T31.1.7.1" class="ltx_td ltx_align_left">lukaemon_mmlu_electrical_engineering</td>
<td id="Ax1.T31.1.7.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T31.1.7.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T31.1.7.4" class="ltx_td ltx_nopad_r ltx_align_left">35.86</td>
</tr>
<tr id="Ax1.T31.1.8" class="ltx_tr">
<td id="Ax1.T31.1.8.1" class="ltx_td ltx_align_left">lukaemon_mmlu_astronomy</td>
<td id="Ax1.T31.1.8.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T31.1.8.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T31.1.8.4" class="ltx_td ltx_nopad_r ltx_align_left">44.08</td>
</tr>
<tr id="Ax1.T31.1.9" class="ltx_tr">
<td id="Ax1.T31.1.9.1" class="ltx_td ltx_align_left">lukaemon_mmlu_anatomy</td>
<td id="Ax1.T31.1.9.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T31.1.9.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T31.1.9.4" class="ltx_td ltx_nopad_r ltx_align_left">42.96</td>
</tr>
<tr id="Ax1.T31.1.10" class="ltx_tr">
<td id="Ax1.T31.1.10.1" class="ltx_td ltx_align_left">lukaemon_mmlu_abstract_algebra</td>
<td id="Ax1.T31.1.10.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T31.1.10.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T31.1.10.4" class="ltx_td ltx_nopad_r ltx_align_left">26</td>
</tr>
<tr id="Ax1.T31.1.11" class="ltx_tr">
<td id="Ax1.T31.1.11.1" class="ltx_td ltx_align_left">lukaemon_mmlu_machine_learning</td>
<td id="Ax1.T31.1.11.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T31.1.11.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T31.1.11.4" class="ltx_td ltx_nopad_r ltx_align_left">31.25</td>
</tr>
<tr id="Ax1.T31.1.12" class="ltx_tr">
<td id="Ax1.T31.1.12.1" class="ltx_td ltx_align_left">lukaemon_mmlu_clinical_knowledge</td>
<td id="Ax1.T31.1.12.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T31.1.12.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T31.1.12.4" class="ltx_td ltx_nopad_r ltx_align_left">47.17</td>
</tr>
<tr id="Ax1.T31.1.13" class="ltx_tr">
<td id="Ax1.T31.1.13.1" class="ltx_td ltx_align_left">lukaemon_mmlu_global_facts</td>
<td id="Ax1.T31.1.13.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T31.1.13.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T31.1.13.4" class="ltx_td ltx_nopad_r ltx_align_left">30</td>
</tr>
<tr id="Ax1.T31.1.14" class="ltx_tr">
<td id="Ax1.T31.1.14.1" class="ltx_td ltx_align_left">lukaemon_mmlu_management</td>
<td id="Ax1.T31.1.14.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T31.1.14.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T31.1.14.4" class="ltx_td ltx_nopad_r ltx_align_left">72.82</td>
</tr>
<tr id="Ax1.T31.1.15" class="ltx_tr">
<td id="Ax1.T31.1.15.1" class="ltx_td ltx_align_left">lukaemon_mmlu_nutrition</td>
<td id="Ax1.T31.1.15.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T31.1.15.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T31.1.15.4" class="ltx_td ltx_nopad_r ltx_align_left">44.44</td>
</tr>
<tr id="Ax1.T31.1.16" class="ltx_tr">
<td id="Ax1.T31.1.16.1" class="ltx_td ltx_align_left">lukaemon_mmlu_marketing</td>
<td id="Ax1.T31.1.16.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T31.1.16.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T31.1.16.4" class="ltx_td ltx_nopad_r ltx_align_left">70.51</td>
</tr>
<tr id="Ax1.T31.1.17" class="ltx_tr">
<td id="Ax1.T31.1.17.1" class="ltx_td ltx_align_left">lukaemon_mmlu_professional_accounting</td>
<td id="Ax1.T31.1.17.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T31.1.17.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T31.1.17.4" class="ltx_td ltx_nopad_r ltx_align_left">30.85</td>
</tr>
<tr id="Ax1.T31.1.18" class="ltx_tr">
<td id="Ax1.T31.1.18.1" class="ltx_td ltx_align_left">lukaemon_mmlu_high_school_geography</td>
<td id="Ax1.T31.1.18.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T31.1.18.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T31.1.18.4" class="ltx_td ltx_nopad_r ltx_align_left">65.15</td>
</tr>
<tr id="Ax1.T31.1.19" class="ltx_tr">
<td id="Ax1.T31.1.19.1" class="ltx_td ltx_align_left">lukaemon_mmlu_international_law</td>
<td id="Ax1.T31.1.19.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T31.1.19.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T31.1.19.4" class="ltx_td ltx_nopad_r ltx_align_left">55.37</td>
</tr>
<tr id="Ax1.T31.1.20" class="ltx_tr">
<td id="Ax1.T31.1.20.1" class="ltx_td ltx_align_left">lukaemon_mmlu_moral_scenarios</td>
<td id="Ax1.T31.1.20.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T31.1.20.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T31.1.20.4" class="ltx_td ltx_nopad_r ltx_align_left">35.08</td>
</tr>
<tr id="Ax1.T31.1.21" class="ltx_tr">
<td id="Ax1.T31.1.21.1" class="ltx_td ltx_align_left">lukaemon_mmlu_computer_security</td>
<td id="Ax1.T31.1.21.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T31.1.21.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T31.1.21.4" class="ltx_td ltx_nopad_r ltx_align_left">61</td>
</tr>
<tr id="Ax1.T31.1.22" class="ltx_tr">
<td id="Ax1.T31.1.22.1" class="ltx_td ltx_align_left">lukaemon_mmlu_high_school_microeconomics</td>
<td id="Ax1.T31.1.22.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T31.1.22.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T31.1.22.4" class="ltx_td ltx_nopad_r ltx_align_left">46.22</td>
</tr>
<tr id="Ax1.T31.1.23" class="ltx_tr">
<td id="Ax1.T31.1.23.1" class="ltx_td ltx_align_left">lukaemon_mmlu_professional_law</td>
<td id="Ax1.T31.1.23.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T31.1.23.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T31.1.23.4" class="ltx_td ltx_nopad_r ltx_align_left">35.27</td>
</tr>
<tr id="Ax1.T31.1.24" class="ltx_tr">
<td id="Ax1.T31.1.24.1" class="ltx_td ltx_align_left">lukaemon_mmlu_medical_genetics</td>
<td id="Ax1.T31.1.24.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T31.1.24.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T31.1.24.4" class="ltx_td ltx_nopad_r ltx_align_left">49</td>
</tr>
<tr id="Ax1.T31.1.25" class="ltx_tr">
<td id="Ax1.T31.1.25.1" class="ltx_td ltx_align_left">lukaemon_mmlu_professional_psychology</td>
<td id="Ax1.T31.1.25.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T31.1.25.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T31.1.25.4" class="ltx_td ltx_nopad_r ltx_align_left">47.06</td>
</tr>
<tr id="Ax1.T31.1.26" class="ltx_tr">
<td id="Ax1.T31.1.26.1" class="ltx_td ltx_align_left">lukaemon_mmlu_jurisprudence</td>
<td id="Ax1.T31.1.26.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T31.1.26.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T31.1.26.4" class="ltx_td ltx_nopad_r ltx_align_left">49.07</td>
</tr>
<tr id="Ax1.T31.1.27" class="ltx_tr">
<td id="Ax1.T31.1.27.1" class="ltx_td ltx_align_left">lukaemon_mmlu_world_religions</td>
<td id="Ax1.T31.1.27.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T31.1.27.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T31.1.27.4" class="ltx_td ltx_nopad_r ltx_align_left">63.16</td>
</tr>
<tr id="Ax1.T31.1.28" class="ltx_tr">
<td id="Ax1.T31.1.28.1" class="ltx_td ltx_align_left">lukaemon_mmlu_philosophy</td>
<td id="Ax1.T31.1.28.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T31.1.28.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T31.1.28.4" class="ltx_td ltx_nopad_r ltx_align_left">51.77</td>
</tr>
<tr id="Ax1.T31.1.29" class="ltx_tr">
<td id="Ax1.T31.1.29.1" class="ltx_td ltx_align_left">lukaemon_mmlu_virology</td>
<td id="Ax1.T31.1.29.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T31.1.29.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T31.1.29.4" class="ltx_td ltx_nopad_r ltx_align_left">37.95</td>
</tr>
<tr id="Ax1.T31.1.30" class="ltx_tr">
<td id="Ax1.T31.1.30.1" class="ltx_td ltx_align_left">lukaemon_mmlu_high_school_chemistry</td>
<td id="Ax1.T31.1.30.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T31.1.30.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T31.1.30.4" class="ltx_td ltx_nopad_r ltx_align_left">32.51</td>
</tr>
<tr id="Ax1.T31.1.31" class="ltx_tr">
<td id="Ax1.T31.1.31.1" class="ltx_td ltx_align_left">lukaemon_mmlu_public_relations</td>
<td id="Ax1.T31.1.31.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T31.1.31.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T31.1.31.4" class="ltx_td ltx_nopad_r ltx_align_left">55.45</td>
</tr>
<tr id="Ax1.T31.1.32" class="ltx_tr">
<td id="Ax1.T31.1.32.1" class="ltx_td ltx_align_left">lukaemon_mmlu_high_school_macroeconomics</td>
<td id="Ax1.T31.1.32.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T31.1.32.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T31.1.32.4" class="ltx_td ltx_nopad_r ltx_align_left">50.77</td>
</tr>
<tr id="Ax1.T31.1.33" class="ltx_tr">
<td id="Ax1.T31.1.33.1" class="ltx_td ltx_align_left">lukaemon_mmlu_human_sexuality</td>
<td id="Ax1.T31.1.33.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T31.1.33.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T31.1.33.4" class="ltx_td ltx_nopad_r ltx_align_left">49.62</td>
</tr>
<tr id="Ax1.T31.1.34" class="ltx_tr">
<td id="Ax1.T31.1.34.1" class="ltx_td ltx_align_left">lukaemon_mmlu_elementary_mathematics</td>
<td id="Ax1.T31.1.34.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T31.1.34.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T31.1.34.4" class="ltx_td ltx_nopad_r ltx_align_left">32.8</td>
</tr>
<tr id="Ax1.T31.1.35" class="ltx_tr">
<td id="Ax1.T31.1.35.1" class="ltx_td ltx_align_left">lukaemon_mmlu_high_school_physics</td>
<td id="Ax1.T31.1.35.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T31.1.35.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T31.1.35.4" class="ltx_td ltx_nopad_r ltx_align_left">33.77</td>
</tr>
<tr id="Ax1.T31.1.36" class="ltx_tr">
<td id="Ax1.T31.1.36.1" class="ltx_td ltx_align_left">lukaemon_mmlu_high_school_computer_science</td>
<td id="Ax1.T31.1.36.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T31.1.36.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T31.1.36.4" class="ltx_td ltx_nopad_r ltx_align_left">42</td>
</tr>
<tr id="Ax1.T31.1.37" class="ltx_tr">
<td id="Ax1.T31.1.37.1" class="ltx_td ltx_align_left">lukaemon_mmlu_high_school_european_history</td>
<td id="Ax1.T31.1.37.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T31.1.37.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T31.1.37.4" class="ltx_td ltx_nopad_r ltx_align_left">58.79</td>
</tr>
<tr id="Ax1.T31.1.38" class="ltx_tr">
<td id="Ax1.T31.1.38.1" class="ltx_td ltx_align_left">lukaemon_mmlu_business_ethics</td>
<td id="Ax1.T31.1.38.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T31.1.38.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T31.1.38.4" class="ltx_td ltx_nopad_r ltx_align_left">51</td>
</tr>
<tr id="Ax1.T31.1.39" class="ltx_tr">
<td id="Ax1.T31.1.39.1" class="ltx_td ltx_align_left">lukaemon_mmlu_moral_disputes</td>
<td id="Ax1.T31.1.39.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T31.1.39.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T31.1.39.4" class="ltx_td ltx_nopad_r ltx_align_left">44.8</td>
</tr>
<tr id="Ax1.T31.1.40" class="ltx_tr">
<td id="Ax1.T31.1.40.1" class="ltx_td ltx_align_left">lukaemon_mmlu_high_school_statistics</td>
<td id="Ax1.T31.1.40.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T31.1.40.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T31.1.40.4" class="ltx_td ltx_nopad_r ltx_align_left">43.98</td>
</tr>
<tr id="Ax1.T31.1.41" class="ltx_tr">
<td id="Ax1.T31.1.41.1" class="ltx_td ltx_align_left">lukaemon_mmlu_miscellaneous</td>
<td id="Ax1.T31.1.41.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T31.1.41.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T31.1.41.4" class="ltx_td ltx_nopad_r ltx_align_left">64.88</td>
</tr>
<tr id="Ax1.T31.1.42" class="ltx_tr">
<td id="Ax1.T31.1.42.1" class="ltx_td ltx_align_left">lukaemon_mmlu_formal_logic</td>
<td id="Ax1.T31.1.42.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T31.1.42.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T31.1.42.4" class="ltx_td ltx_nopad_r ltx_align_left">39.68</td>
</tr>
<tr id="Ax1.T31.1.43" class="ltx_tr">
<td id="Ax1.T31.1.43.1" class="ltx_td ltx_align_left">lukaemon_mmlu_high_school_government_and_politics</td>
<td id="Ax1.T31.1.43.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T31.1.43.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T31.1.43.4" class="ltx_td ltx_nopad_r ltx_align_left">68.91</td>
</tr>
<tr id="Ax1.T31.1.44" class="ltx_tr">
<td id="Ax1.T31.1.44.1" class="ltx_td ltx_align_left">lukaemon_mmlu_prehistory</td>
<td id="Ax1.T31.1.44.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T31.1.44.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T31.1.44.4" class="ltx_td ltx_nopad_r ltx_align_left">52.47</td>
</tr>
<tr id="Ax1.T31.1.45" class="ltx_tr">
<td id="Ax1.T31.1.45.1" class="ltx_td ltx_align_left">lukaemon_mmlu_security_studies</td>
<td id="Ax1.T31.1.45.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T31.1.45.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T31.1.45.4" class="ltx_td ltx_nopad_r ltx_align_left">53.88</td>
</tr>
<tr id="Ax1.T31.1.46" class="ltx_tr">
<td id="Ax1.T31.1.46.1" class="ltx_td ltx_align_left">lukaemon_mmlu_high_school_biology</td>
<td id="Ax1.T31.1.46.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T31.1.46.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T31.1.46.4" class="ltx_td ltx_nopad_r ltx_align_left">56.77</td>
</tr>
<tr id="Ax1.T31.1.47" class="ltx_tr">
<td id="Ax1.T31.1.47.1" class="ltx_td ltx_align_left">lukaemon_mmlu_logical_fallacies</td>
<td id="Ax1.T31.1.47.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T31.1.47.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T31.1.47.4" class="ltx_td ltx_nopad_r ltx_align_left">56.44</td>
</tr>
<tr id="Ax1.T31.1.48" class="ltx_tr">
<td id="Ax1.T31.1.48.1" class="ltx_td ltx_align_left">lukaemon_mmlu_high_school_world_history</td>
<td id="Ax1.T31.1.48.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T31.1.48.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T31.1.48.4" class="ltx_td ltx_nopad_r ltx_align_left">61.18</td>
</tr>
<tr id="Ax1.T31.1.49" class="ltx_tr">
<td id="Ax1.T31.1.49.1" class="ltx_td ltx_align_left">lukaemon_mmlu_professional_medicine</td>
<td id="Ax1.T31.1.49.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T31.1.49.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T31.1.49.4" class="ltx_td ltx_nopad_r ltx_align_left">49.26</td>
</tr>
<tr id="Ax1.T31.1.50" class="ltx_tr">
<td id="Ax1.T31.1.50.1" class="ltx_td ltx_align_left">lukaemon_mmlu_high_school_mathematics</td>
<td id="Ax1.T31.1.50.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T31.1.50.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T31.1.50.4" class="ltx_td ltx_nopad_r ltx_align_left">25.56</td>
</tr>
<tr id="Ax1.T31.1.51" class="ltx_tr">
<td id="Ax1.T31.1.51.1" class="ltx_td ltx_align_left">lukaemon_mmlu_college_medicine</td>
<td id="Ax1.T31.1.51.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T31.1.51.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T31.1.51.4" class="ltx_td ltx_nopad_r ltx_align_left">40.46</td>
</tr>
<tr id="Ax1.T31.1.52" class="ltx_tr">
<td id="Ax1.T31.1.52.1" class="ltx_td ltx_align_left">lukaemon_mmlu_high_school_us_history</td>
<td id="Ax1.T31.1.52.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T31.1.52.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T31.1.52.4" class="ltx_td ltx_nopad_r ltx_align_left">62.25</td>
</tr>
<tr id="Ax1.T31.1.53" class="ltx_tr">
<td id="Ax1.T31.1.53.1" class="ltx_td ltx_align_left">lukaemon_mmlu_sociology</td>
<td id="Ax1.T31.1.53.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T31.1.53.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T31.1.53.4" class="ltx_td ltx_nopad_r ltx_align_left">66.17</td>
</tr>
<tr id="Ax1.T31.1.54" class="ltx_tr">
<td id="Ax1.T31.1.54.1" class="ltx_td ltx_align_left">lukaemon_mmlu_econometrics</td>
<td id="Ax1.T31.1.54.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T31.1.54.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T31.1.54.4" class="ltx_td ltx_nopad_r ltx_align_left">27.19</td>
</tr>
<tr id="Ax1.T31.1.55" class="ltx_tr">
<td id="Ax1.T31.1.55.1" class="ltx_td ltx_align_left">lukaemon_mmlu_high_school_psychology</td>
<td id="Ax1.T31.1.55.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T31.1.55.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T31.1.55.4" class="ltx_td ltx_nopad_r ltx_align_left">68.26</td>
</tr>
<tr id="Ax1.T31.1.56" class="ltx_tr">
<td id="Ax1.T31.1.56.1" class="ltx_td ltx_align_left">lukaemon_mmlu_human_aging</td>
<td id="Ax1.T31.1.56.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T31.1.56.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T31.1.56.4" class="ltx_td ltx_nopad_r ltx_align_left">54.71</td>
</tr>
<tr id="Ax1.T31.1.57" class="ltx_tr">
<td id="Ax1.T31.1.57.1" class="ltx_td ltx_align_left">lukaemon_mmlu_us_foreign_policy</td>
<td id="Ax1.T31.1.57.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T31.1.57.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T31.1.57.4" class="ltx_td ltx_nopad_r ltx_align_left">68</td>
</tr>
<tr id="Ax1.T31.1.58" class="ltx_tr">
<td id="Ax1.T31.1.58.1" class="ltx_td ltx_align_left">lukaemon_mmlu_conceptual_physics</td>
<td id="Ax1.T31.1.58.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T31.1.58.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T31.1.58.4" class="ltx_td ltx_nopad_r ltx_align_left">40</td>
</tr>
<tr id="Ax1.T31.1.59" class="ltx_tr">
<td id="Ax1.T31.1.59.1" class="ltx_td ltx_align_left">bbh-temporal_sequences</td>
<td id="Ax1.T31.1.59.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T31.1.59.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T31.1.59.4" class="ltx_td ltx_nopad_r ltx_align_left">29.2</td>
</tr>
<tr id="Ax1.T31.1.60" class="ltx_tr">
<td id="Ax1.T31.1.60.1" class="ltx_td ltx_align_left">bbh-disambiguation_qa</td>
<td id="Ax1.T31.1.60.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T31.1.60.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T31.1.60.4" class="ltx_td ltx_nopad_r ltx_align_left">39.6</td>
</tr>
<tr id="Ax1.T31.1.61" class="ltx_tr">
<td id="Ax1.T31.1.61.1" class="ltx_td ltx_align_left">bbh-date_understanding</td>
<td id="Ax1.T31.1.61.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T31.1.61.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T31.1.61.4" class="ltx_td ltx_nopad_r ltx_align_left">42</td>
</tr>
<tr id="Ax1.T31.1.62" class="ltx_tr">
<td id="Ax1.T31.1.62.1" class="ltx_td ltx_align_left">bbh-tracking_shuffled_objects_three_objects</td>
<td id="Ax1.T31.1.62.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T31.1.62.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T31.1.62.4" class="ltx_td ltx_nopad_r ltx_align_left">30.4</td>
</tr>
<tr id="Ax1.T31.1.63" class="ltx_tr">
<td id="Ax1.T31.1.63.1" class="ltx_td ltx_align_left">bbh-penguins_in_a_table</td>
<td id="Ax1.T31.1.63.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T31.1.63.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T31.1.63.4" class="ltx_td ltx_nopad_r ltx_align_left">36.3</td>
</tr>
<tr id="Ax1.T31.1.64" class="ltx_tr">
<td id="Ax1.T31.1.64.1" class="ltx_td ltx_align_left">bbh-geometric_shapes</td>
<td id="Ax1.T31.1.64.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T31.1.64.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T31.1.64.4" class="ltx_td ltx_nopad_r ltx_align_left">16</td>
</tr>
<tr id="Ax1.T31.1.65" class="ltx_tr">
<td id="Ax1.T31.1.65.1" class="ltx_td ltx_align_left">bbh-snarks</td>
<td id="Ax1.T31.1.65.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T31.1.65.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T31.1.65.4" class="ltx_td ltx_nopad_r ltx_align_left">63.48</td>
</tr>
<tr id="Ax1.T31.1.66" class="ltx_tr">
<td id="Ax1.T31.1.66.1" class="ltx_td ltx_align_left">bbh-ruin_names</td>
<td id="Ax1.T31.1.66.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T31.1.66.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T31.1.66.4" class="ltx_td ltx_nopad_r ltx_align_left">30.8</td>
</tr>
<tr id="Ax1.T31.1.67" class="ltx_tr">
<td id="Ax1.T31.1.67.1" class="ltx_td ltx_align_left">bbh-tracking_shuffled_objects_seven_objects</td>
<td id="Ax1.T31.1.67.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T31.1.67.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T31.1.67.4" class="ltx_td ltx_nopad_r ltx_align_left">14.8</td>
</tr>
<tr id="Ax1.T31.1.68" class="ltx_tr">
<td id="Ax1.T31.1.68.1" class="ltx_td ltx_align_left">bbh-tracking_shuffled_objects_five_objects</td>
<td id="Ax1.T31.1.68.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T31.1.68.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T31.1.68.4" class="ltx_td ltx_nopad_r ltx_align_left">21.2</td>
</tr>
<tr id="Ax1.T31.1.69" class="ltx_tr">
<td id="Ax1.T31.1.69.1" class="ltx_td ltx_align_left">bbh-logical_deduction_three_objects</td>
<td id="Ax1.T31.1.69.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T31.1.69.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T31.1.69.4" class="ltx_td ltx_nopad_r ltx_align_left">43.6</td>
</tr>
<tr id="Ax1.T31.1.70" class="ltx_tr">
<td id="Ax1.T31.1.70.1" class="ltx_td ltx_align_left">bbh-hyperbaton</td>
<td id="Ax1.T31.1.70.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T31.1.70.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T31.1.70.4" class="ltx_td ltx_nopad_r ltx_align_left">64</td>
</tr>
<tr id="Ax1.T31.1.71" class="ltx_tr">
<td id="Ax1.T31.1.71.1" class="ltx_td ltx_align_left">bbh-logical_deduction_five_objects</td>
<td id="Ax1.T31.1.71.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T31.1.71.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T31.1.71.4" class="ltx_td ltx_nopad_r ltx_align_left">26.4</td>
</tr>
<tr id="Ax1.T31.1.72" class="ltx_tr">
<td id="Ax1.T31.1.72.1" class="ltx_td ltx_align_left">bbh-logical_deduction_seven_objects</td>
<td id="Ax1.T31.1.72.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T31.1.72.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T31.1.72.4" class="ltx_td ltx_nopad_r ltx_align_left">14.4</td>
</tr>
<tr id="Ax1.T31.1.73" class="ltx_tr">
<td id="Ax1.T31.1.73.1" class="ltx_td ltx_align_left">bbh-movie_recommendation</td>
<td id="Ax1.T31.1.73.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T31.1.73.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T31.1.73.4" class="ltx_td ltx_nopad_r ltx_align_left">64</td>
</tr>
<tr id="Ax1.T31.1.74" class="ltx_tr">
<td id="Ax1.T31.1.74.1" class="ltx_td ltx_align_left">bbh-salient_translation_error_detection</td>
<td id="Ax1.T31.1.74.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T31.1.74.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T31.1.74.4" class="ltx_td ltx_nopad_r ltx_align_left">19.6</td>
</tr>
<tr id="Ax1.T31.1.75" class="ltx_tr">
<td id="Ax1.T31.1.75.1" class="ltx_td ltx_align_left">bbh-reasoning_about_colored_objects</td>
<td id="Ax1.T31.1.75.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T31.1.75.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T31.1.75.4" class="ltx_td ltx_nopad_r ltx_align_left">27.6</td>
</tr>
<tr id="Ax1.T31.1.76" class="ltx_tr">
<td id="Ax1.T31.1.76.1" class="ltx_td ltx_align_left">bbh-multistep_arithmetic_two</td>
<td id="Ax1.T31.1.76.2" class="ltx_td ltx_align_left">score</td>
<td id="Ax1.T31.1.76.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T31.1.76.4" class="ltx_td ltx_nopad_r ltx_align_left">0.4</td>
</tr>
<tr id="Ax1.T31.1.77" class="ltx_tr">
<td id="Ax1.T31.1.77.1" class="ltx_td ltx_align_left">bbh-navigate</td>
<td id="Ax1.T31.1.77.2" class="ltx_td ltx_align_left">score</td>
<td id="Ax1.T31.1.77.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T31.1.77.4" class="ltx_td ltx_nopad_r ltx_align_left">60</td>
</tr>
<tr id="Ax1.T31.1.78" class="ltx_tr">
<td id="Ax1.T31.1.78.1" class="ltx_td ltx_align_left">bbh-dyck_languages</td>
<td id="Ax1.T31.1.78.2" class="ltx_td ltx_align_left">score</td>
<td id="Ax1.T31.1.78.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T31.1.78.4" class="ltx_td ltx_nopad_r ltx_align_left">0</td>
</tr>
<tr id="Ax1.T31.1.79" class="ltx_tr">
<td id="Ax1.T31.1.79.1" class="ltx_td ltx_align_left">bbh-word_sorting</td>
<td id="Ax1.T31.1.79.2" class="ltx_td ltx_align_left">score</td>
<td id="Ax1.T31.1.79.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T31.1.79.4" class="ltx_td ltx_nopad_r ltx_align_left">5.6</td>
</tr>
<tr id="Ax1.T31.1.80" class="ltx_tr">
<td id="Ax1.T31.1.80.1" class="ltx_td ltx_align_left">bbh-sports_understanding</td>
<td id="Ax1.T31.1.80.2" class="ltx_td ltx_align_left">score</td>
<td id="Ax1.T31.1.80.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T31.1.80.4" class="ltx_td ltx_nopad_r ltx_align_left">72.8</td>
</tr>
<tr id="Ax1.T31.1.81" class="ltx_tr">
<td id="Ax1.T31.1.81.1" class="ltx_td ltx_align_left">bbh-boolean_expressions</td>
<td id="Ax1.T31.1.81.2" class="ltx_td ltx_align_left">score</td>
<td id="Ax1.T31.1.81.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T31.1.81.4" class="ltx_td ltx_nopad_r ltx_align_left">58.4</td>
</tr>
<tr id="Ax1.T31.1.82" class="ltx_tr">
<td id="Ax1.T31.1.82.1" class="ltx_td ltx_align_left">bbh-object_counting</td>
<td id="Ax1.T31.1.82.2" class="ltx_td ltx_align_left">score</td>
<td id="Ax1.T31.1.82.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T31.1.82.4" class="ltx_td ltx_nopad_r ltx_align_left">38</td>
</tr>
<tr id="Ax1.T31.1.83" class="ltx_tr">
<td id="Ax1.T31.1.83.1" class="ltx_td ltx_align_left">bbh-formal_fallacies</td>
<td id="Ax1.T31.1.83.2" class="ltx_td ltx_align_left">score</td>
<td id="Ax1.T31.1.83.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T31.1.83.4" class="ltx_td ltx_nopad_r ltx_align_left">49.6</td>
</tr>
<tr id="Ax1.T31.1.84" class="ltx_tr">
<td id="Ax1.T31.1.84.1" class="ltx_td ltx_align_left">bbh-causal_judgement</td>
<td id="Ax1.T31.1.84.2" class="ltx_td ltx_align_left">score</td>
<td id="Ax1.T31.1.84.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T31.1.84.4" class="ltx_td ltx_nopad_r ltx_align_left">56.68</td>
</tr>
<tr id="Ax1.T31.1.85" class="ltx_tr">
<td id="Ax1.T31.1.85.1" class="ltx_td ltx_align_left">bbh-web_of_lies</td>
<td id="Ax1.T31.1.85.2" class="ltx_td ltx_align_left">score</td>
<td id="Ax1.T31.1.85.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T31.1.85.4" class="ltx_td ltx_nopad_r ltx_align_left">61.2</td>
</tr>
<tr id="Ax1.T31.1.86" class="ltx_tr">
<td id="Ax1.T31.1.86.1" class="ltx_td ltx_align_left">tyidqa-goldp_arabic</td>
<td id="Ax1.T31.1.86.2" class="ltx_td ltx_align_left">exact_match</td>
<td id="Ax1.T31.1.86.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T31.1.86.4" class="ltx_td ltx_nopad_r ltx_align_left">29.32</td>
</tr>
<tr id="Ax1.T31.1.87" class="ltx_tr">
<td id="Ax1.T31.1.87.1" class="ltx_td ltx_align_left">tyidqa-goldp_arabic</td>
<td id="Ax1.T31.1.87.2" class="ltx_td ltx_align_left">f1</td>
<td id="Ax1.T31.1.87.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T31.1.87.4" class="ltx_td ltx_nopad_r ltx_align_left">57.68</td>
</tr>
<tr id="Ax1.T31.1.88" class="ltx_tr">
<td id="Ax1.T31.1.88.1" class="ltx_td ltx_align_left">tyidqa-goldp_bengali</td>
<td id="Ax1.T31.1.88.2" class="ltx_td ltx_align_left">exact_match</td>
<td id="Ax1.T31.1.88.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T31.1.88.4" class="ltx_td ltx_nopad_r ltx_align_left">15.93</td>
</tr>
<tr id="Ax1.T31.1.89" class="ltx_tr">
<td id="Ax1.T31.1.89.1" class="ltx_td ltx_align_left">tyidqa-goldp_bengali</td>
<td id="Ax1.T31.1.89.2" class="ltx_td ltx_align_left">f1</td>
<td id="Ax1.T31.1.89.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T31.1.89.4" class="ltx_td ltx_nopad_r ltx_align_left">25.03</td>
</tr>
<tr id="Ax1.T31.1.90" class="ltx_tr">
<td id="Ax1.T31.1.90.1" class="ltx_td ltx_align_left">tyidqa-goldp_english</td>
<td id="Ax1.T31.1.90.2" class="ltx_td ltx_align_left">exact_match</td>
<td id="Ax1.T31.1.90.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T31.1.90.4" class="ltx_td ltx_nopad_r ltx_align_left">20</td>
</tr>
<tr id="Ax1.T31.1.91" class="ltx_tr">
<td id="Ax1.T31.1.91.1" class="ltx_td ltx_align_left">tyidqa-goldp_english</td>
<td id="Ax1.T31.1.91.2" class="ltx_td ltx_align_left">f1</td>
<td id="Ax1.T31.1.91.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T31.1.91.4" class="ltx_td ltx_nopad_r ltx_align_left">28.85</td>
</tr>
<tr id="Ax1.T31.1.92" class="ltx_tr">
<td id="Ax1.T31.1.92.1" class="ltx_td ltx_align_left">tyidqa-goldp_finnish</td>
<td id="Ax1.T31.1.92.2" class="ltx_td ltx_align_left">exact_match</td>
<td id="Ax1.T31.1.92.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T31.1.92.4" class="ltx_td ltx_nopad_r ltx_align_left">31.46</td>
</tr>
<tr id="Ax1.T31.1.93" class="ltx_tr">
<td id="Ax1.T31.1.93.1" class="ltx_td ltx_align_left">tyidqa-goldp_finnish</td>
<td id="Ax1.T31.1.93.2" class="ltx_td ltx_align_left">f1</td>
<td id="Ax1.T31.1.93.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T31.1.93.4" class="ltx_td ltx_nopad_r ltx_align_left">37.05</td>
</tr>
<tr id="Ax1.T31.1.94" class="ltx_tr">
<td id="Ax1.T31.1.94.1" class="ltx_td ltx_align_left">tyidqa-goldp_indonesian</td>
<td id="Ax1.T31.1.94.2" class="ltx_td ltx_align_left">exact_match</td>
<td id="Ax1.T31.1.94.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T31.1.94.4" class="ltx_td ltx_nopad_r ltx_align_left">16.11</td>
</tr>
<tr id="Ax1.T31.1.95" class="ltx_tr">
<td id="Ax1.T31.1.95.1" class="ltx_td ltx_align_left">tyidqa-goldp_indonesian</td>
<td id="Ax1.T31.1.95.2" class="ltx_td ltx_align_left">f1</td>
<td id="Ax1.T31.1.95.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T31.1.95.4" class="ltx_td ltx_nopad_r ltx_align_left">30.59</td>
</tr>
<tr id="Ax1.T31.1.96" class="ltx_tr">
<td id="Ax1.T31.1.96.1" class="ltx_td ltx_align_left">tyidqa-goldp_japanese</td>
<td id="Ax1.T31.1.96.2" class="ltx_td ltx_align_left">exact_match</td>
<td id="Ax1.T31.1.96.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T31.1.96.4" class="ltx_td ltx_nopad_r ltx_align_left">20.22</td>
</tr>
<tr id="Ax1.T31.1.97" class="ltx_tr">
<td id="Ax1.T31.1.97.1" class="ltx_td ltx_align_left">tyidqa-goldp_japanese</td>
<td id="Ax1.T31.1.97.2" class="ltx_td ltx_align_left">f1</td>
<td id="Ax1.T31.1.97.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T31.1.97.4" class="ltx_td ltx_nopad_r ltx_align_left">20.6</td>
</tr>
<tr id="Ax1.T31.1.98" class="ltx_tr">
<td id="Ax1.T31.1.98.1" class="ltx_td ltx_align_left">tyidqa-goldp_korean</td>
<td id="Ax1.T31.1.98.2" class="ltx_td ltx_align_left">exact_match</td>
<td id="Ax1.T31.1.98.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T31.1.98.4" class="ltx_td ltx_nopad_r ltx_align_left">52.9</td>
</tr>
<tr id="Ax1.T31.1.99" class="ltx_tr">
<td id="Ax1.T31.1.99.1" class="ltx_td ltx_align_left">tyidqa-goldp_korean</td>
<td id="Ax1.T31.1.99.2" class="ltx_td ltx_align_left">f1</td>
<td id="Ax1.T31.1.99.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T31.1.99.4" class="ltx_td ltx_nopad_r ltx_align_left">60.31</td>
</tr>
<tr id="Ax1.T31.1.100" class="ltx_tr">
<td id="Ax1.T31.1.100.1" class="ltx_td ltx_align_left">tyidqa-goldp_russian</td>
<td id="Ax1.T31.1.100.2" class="ltx_td ltx_align_left">exact_match</td>
<td id="Ax1.T31.1.100.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T31.1.100.4" class="ltx_td ltx_nopad_r ltx_align_left">21.67</td>
</tr>
<tr id="Ax1.T31.1.101" class="ltx_tr">
<td id="Ax1.T31.1.101.1" class="ltx_td ltx_align_left">tyidqa-goldp_russian</td>
<td id="Ax1.T31.1.101.2" class="ltx_td ltx_align_left">f1</td>
<td id="Ax1.T31.1.101.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T31.1.101.4" class="ltx_td ltx_nopad_r ltx_align_left">28.87</td>
</tr>
<tr id="Ax1.T31.1.102" class="ltx_tr">
<td id="Ax1.T31.1.102.1" class="ltx_td ltx_align_left">tyidqa-goldp_swahili</td>
<td id="Ax1.T31.1.102.2" class="ltx_td ltx_align_left">exact_match</td>
<td id="Ax1.T31.1.102.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T31.1.102.4" class="ltx_td ltx_nopad_r ltx_align_left">20.24</td>
</tr>
<tr id="Ax1.T31.1.103" class="ltx_tr">
<td id="Ax1.T31.1.103.1" class="ltx_td ltx_align_left">tyidqa-goldp_swahili</td>
<td id="Ax1.T31.1.103.2" class="ltx_td ltx_align_left">f1</td>
<td id="Ax1.T31.1.103.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T31.1.103.4" class="ltx_td ltx_nopad_r ltx_align_left">34.19</td>
</tr>
<tr id="Ax1.T31.1.104" class="ltx_tr">
<td id="Ax1.T31.1.104.1" class="ltx_td ltx_align_left">tyidqa-goldp_telugu</td>
<td id="Ax1.T31.1.104.2" class="ltx_td ltx_align_left">exact_match</td>
<td id="Ax1.T31.1.104.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T31.1.104.4" class="ltx_td ltx_nopad_r ltx_align_left">0.3</td>
</tr>
<tr id="Ax1.T31.1.105" class="ltx_tr">
<td id="Ax1.T31.1.105.1" class="ltx_td ltx_align_left">tyidqa-goldp_telugu</td>
<td id="Ax1.T31.1.105.2" class="ltx_td ltx_align_left">f1</td>
<td id="Ax1.T31.1.105.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T31.1.105.4" class="ltx_td ltx_nopad_r ltx_align_left">2.88</td>
</tr>
<tr id="Ax1.T31.1.106" class="ltx_tr">
<td id="Ax1.T31.1.106.1" class="ltx_td ltx_align_left">tyidqa-goldp_thai</td>
<td id="Ax1.T31.1.106.2" class="ltx_td ltx_align_left">exact_match</td>
<td id="Ax1.T31.1.106.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T31.1.106.4" class="ltx_td ltx_nopad_r ltx_align_left">21.01</td>
</tr>
<tr id="Ax1.T31.1.107" class="ltx_tr">
<td id="Ax1.T31.1.107.1" class="ltx_td ltx_align_left">tyidqa-goldp_thai</td>
<td id="Ax1.T31.1.107.2" class="ltx_td ltx_align_left">f1</td>
<td id="Ax1.T31.1.107.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T31.1.107.4" class="ltx_td ltx_nopad_r ltx_align_left">31.69</td>
</tr>
<tr id="Ax1.T31.1.108" class="ltx_tr">
<td id="Ax1.T31.1.108.1" class="ltx_td ltx_align_left">piqa</td>
<td id="Ax1.T31.1.108.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T31.1.108.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T31.1.108.4" class="ltx_td ltx_nopad_r ltx_align_left">76.88</td>
</tr>
<tr id="Ax1.T31.1.109" class="ltx_tr">
<td id="Ax1.T31.1.109.1" class="ltx_td ltx_align_left">siqa</td>
<td id="Ax1.T31.1.109.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T31.1.109.3" class="ltx_td ltx_align_left">ppl</td>
<td id="Ax1.T31.1.109.4" class="ltx_td ltx_nopad_r ltx_align_left">47.39</td>
</tr>
<tr id="Ax1.T31.1.110" class="ltx_tr">
<td id="Ax1.T31.1.110.1" class="ltx_td ltx_align_left">gsm8k</td>
<td id="Ax1.T31.1.110.2" class="ltx_td ltx_align_left">accuracy</td>
<td id="Ax1.T31.1.110.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T31.1.110.4" class="ltx_td ltx_nopad_r ltx_align_left">2.96</td>
</tr>
<tr id="Ax1.T31.1.111" class="ltx_tr">
<td id="Ax1.T31.1.111.1" class="ltx_td ltx_align_left">truthful_qa</td>
<td id="Ax1.T31.1.111.2" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T31.1.111.3" class="ltx_td ltx_align_left">-</td>
<td id="Ax1.T31.1.111.4" class="ltx_td ltx_nopad_r ltx_align_left">-</td>
</tr>
<tr id="Ax1.T31.1.112" class="ltx_tr">
<td id="Ax1.T31.1.112.1" class="ltx_td ltx_align_left">triviaqa</td>
<td id="Ax1.T31.1.112.2" class="ltx_td ltx_align_left">score</td>
<td id="Ax1.T31.1.112.3" class="ltx_td ltx_align_left">gen</td>
<td id="Ax1.T31.1.112.4" class="ltx_td ltx_nopad_r ltx_align_left">46.57</td>
</tr>
<tr id="Ax1.T31.1.113" class="ltx_tr">
<td id="Ax1.T31.1.113.1" class="ltx_td ltx_align_left ltx_border_bb">BoolQ</td>
<td id="Ax1.T31.1.113.2" class="ltx_td ltx_align_left ltx_border_bb">accuracy</td>
<td id="Ax1.T31.1.113.3" class="ltx_td ltx_align_left ltx_border_bb">ppl</td>
<td id="Ax1.T31.1.113.4" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_bb">82.45</td>
</tr>
</table>
</figure>
</section>
<section id="Ax1.SS6" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">.6 </span>Prompts</h3>

<figure id="Ax1.T32" class="ltx_table">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table">Table 32: </span>Prompts applied for naive continual learning</figcaption>
<div id="Ax1.T32.1" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:433.6pt;height:153pt;vertical-align:-0.9pt;"><span class="ltx_transformed_inner" style="transform:translate(-26.3pt,9.2pt) scale(0.891685445493645,0.891685445493645) ;">
<table id="Ax1.T32.1.1" class="ltx_tabular ltx_align_middle">
<tr id="Ax1.T32.1.1.1" class="ltx_tr">
<td id="Ax1.T32.1.1.1.1" class="ltx_td ltx_align_left ltx_border_tt">Dataset</td>
<td id="Ax1.T32.1.1.1.2" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_tt">Prompt</td>
</tr>
<tr id="Ax1.T32.1.1.2" class="ltx_tr">
<td id="Ax1.T32.1.1.2.1" class="ltx_td ltx_align_left ltx_border_t">ScienceQA</td>
<td id="Ax1.T32.1.1.2.2" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_t">Choose an answer for the following question and give your reasons.</td>
</tr>
<tr id="Ax1.T32.1.1.3" class="ltx_tr">
<td id="Ax1.T32.1.1.3.1" class="ltx_td ltx_align_left">FOMC</td>
<td id="Ax1.T32.1.1.3.2" class="ltx_td ltx_nopad_r ltx_align_left">What is the monetary policy stance for the following text? A. dovish, B. hawkish, C. neutral. Choose one from A, B and C.</td>
</tr>
<tr id="Ax1.T32.1.1.4" class="ltx_tr">
<td id="Ax1.T32.1.1.4.1" class="ltx_td ltx_align_left">MeetingBank</td>
<td id="Ax1.T32.1.1.4.2" class="ltx_td ltx_nopad_r ltx_align_left">Write a summary of the following meeting transcripts.</td>
</tr>
<tr id="Ax1.T32.1.1.5" class="ltx_tr">
<td id="Ax1.T32.1.1.5.1" class="ltx_td ltx_align_left">C-STANCE</td>
<td id="Ax1.T32.1.1.5.2" class="ltx_td ltx_nopad_r ltx_align_left">判断以下文本对指定对象的态度，选择一项：A.支持，B.反对，C.中立。输出A，B或者C。</td>
</tr>
<tr id="Ax1.T32.1.1.6" class="ltx_tr">
<td id="Ax1.T32.1.1.6.1" class="ltx_td ltx_align_left">20Minuten</td>
<td id="Ax1.T32.1.1.6.2" class="ltx_td ltx_nopad_r ltx_align_left">Provide a simplified version of the following paragraph in German.</td>
</tr>
<tr id="Ax1.T32.1.1.7" class="ltx_tr">
<td id="Ax1.T32.1.1.7.1" class="ltx_td ltx_align_left">Py150</td>
<td id="Ax1.T32.1.1.7.2" class="ltx_td ltx_nopad_r ltx_align_left">-</td>
</tr>
<tr id="Ax1.T32.1.1.8" class="ltx_tr">
<td id="Ax1.T32.1.1.8.1" class="ltx_td ltx_align_left">NumGLUE-cm</td>
<td id="Ax1.T32.1.1.8.2" class="ltx_td ltx_nopad_r ltx_align_left">Solve the following math problem.</td>
</tr>
<tr id="Ax1.T32.1.1.9" class="ltx_tr">
<td id="Ax1.T32.1.1.9.1" class="ltx_td ltx_align_left ltx_border_bb">NumGLUE-ds</td>
<td id="Ax1.T32.1.1.9.2" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_bb">Solve the following math problem.</td>
</tr>
</table>
</span></div>
</figure>
<figure id="Ax1.T33" class="ltx_table">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table">Table 33: </span>Prompts applied for reasoning-based continual learning</figcaption>
<div id="Ax1.T33.1" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:433.6pt;height:180.9pt;vertical-align:-0.9pt;"><span class="ltx_transformed_inner" style="transform:translate(-29.0pt,12.1pt) scale(0.881913619343922,0.881913619343922) ;">
<table id="Ax1.T33.1.1" class="ltx_tabular ltx_align_middle">
<tr id="Ax1.T33.1.1.1" class="ltx_tr">
<td id="Ax1.T33.1.1.1.1" class="ltx_td ltx_align_left ltx_border_tt">Dataset</td>
<td id="Ax1.T33.1.1.1.2" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_tt">Prompt</td>
</tr>
<tr id="Ax1.T33.1.1.2" class="ltx_tr">
<td id="Ax1.T33.1.1.2.1" class="ltx_td ltx_align_left ltx_border_t">ScienceQA</td>
<td id="Ax1.T33.1.1.2.2" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_t">Choose an answer for the following question. Give your reasoning first, and then the answer.</td>
</tr>
<tr id="Ax1.T33.1.1.3" class="ltx_tr">
<td id="Ax1.T33.1.1.3.1" class="ltx_td ltx_align_left">FOMC</td>
<td id="Ax1.T33.1.1.3.2" class="ltx_td ltx_nopad_r ltx_align_left">What is the monetary policy stance for the following text? A. dovish, B. hawkish, C. neutral. Choose one from A, B and C. Give your reasoning first, and then the answer.</td>
</tr>
<tr id="Ax1.T33.1.1.4" class="ltx_tr">
<td id="Ax1.T33.1.1.4.1" class="ltx_td ltx_align_left">MeetingBank</td>
<td id="Ax1.T33.1.1.4.2" class="ltx_td ltx_nopad_r ltx_align_left">Write a summary of the following meeting transcripts. Give your reasoning first, and then the answer.</td>
</tr>
<tr id="Ax1.T33.1.1.5" class="ltx_tr">
<td id="Ax1.T33.1.1.5.1" class="ltx_td ltx_align_left">C-STANCE</td>
<td id="Ax1.T33.1.1.5.2" class="ltx_td ltx_nopad_r ltx_align_left">判断以下文本对指定对象的态度，选择一项：A.支持，B.反对，C.中立。输出A，B或者C。先给出推理，然后给出答案。</td>
</tr>
<tr id="Ax1.T33.1.1.6" class="ltx_tr">
<td id="Ax1.T33.1.1.6.1" class="ltx_td ltx_align_left">20Minuten</td>
<td id="Ax1.T33.1.1.6.2" class="ltx_td ltx_nopad_r ltx_align_left">Provide a simplified version of the following paragraph in German. Give your reasoning first, and then the answer.</td>
</tr>
<tr id="Ax1.T33.1.1.7" class="ltx_tr">
<td id="Ax1.T33.1.1.7.1" class="ltx_td ltx_align_left">Py150</td>
<td id="Ax1.T33.1.1.7.2" class="ltx_td ltx_nopad_r ltx_align_left">Complete the next line of the following codes. Give your answer first, and then the reasoning.</td>
</tr>
<tr id="Ax1.T33.1.1.8" class="ltx_tr">
<td id="Ax1.T33.1.1.8.1" class="ltx_td ltx_align_left">NumGLUE-cm</td>
<td id="Ax1.T33.1.1.8.2" class="ltx_td ltx_nopad_r ltx_align_left">Solve the following math problem. Give your reasoning first, and then the answer.</td>
</tr>
<tr id="Ax1.T33.1.1.9" class="ltx_tr">
<td id="Ax1.T33.1.1.9.1" class="ltx_td ltx_align_left ltx_border_bb">NumGLUE-ds</td>
<td id="Ax1.T33.1.1.9.2" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_bb">Solve the following math problem. Give your reasoning first, and then the answer.</td>
</tr>
</table>
</span></div>
</figure>
</section>
<section id="Ax1.SS7" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">.7 </span>Model evaluation</h3>

<div id="Ax1.SS7.p1" class="ltx_para ltx_noindent">
<p id="Ax1.SS7.p1.1" class="ltx_p">Involving the utilization of GPT-4 <cite class="ltx_cite ltx_citemacro_citep">(OpenAI, <a href="#bib.bib33" title="" class="ltx_ref">2023</a>)</cite>, the state-of-the-art large language model, for the assessment and comparative analysis of responses generated by different models is widely adopted in recently works. LLM-as-a-judge <cite class="ltx_cite ltx_citemacro_citep">(Zheng et al., <a href="#bib.bib55" title="" class="ltx_ref">2023</a>)</cite> and AlpacaFarm <cite class="ltx_cite ltx_citemacro_citep">(Dubois et al., <a href="#bib.bib14" title="" class="ltx_ref">2023</a>)</cite> propose that such an automated evaluation approach can provide a relatively fair assessment compared to huama evaluation by leveraging the extensive internal model knowledge.</p>
</div>
<div id="Ax1.SS7.p2" class="ltx_para ltx_noindent">
<p id="Ax1.SS7.p2.1" class="ltx_p">The entire evaluation process is completed by calling the OpenAI’s GPT-4-0613 API, which is presented a question and two responses from different models and tasked to judge which one is better or tie. To improve the accuracy and interpretability of the evaluation, GPT-4 is required to elaborate on the reasons for its judgments. Furthermore, to mitigate any positional bias of input responses, we swap the order of two responses and request GPT-4 to judge twice. The examples of GPT-4 system prompts and prompt templates used on our safety evaluation are as follows:</p>
</div>
<figure id="Ax1.T34" class="ltx_table">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table">Table 34: </span>System prompt and prompt template applied for Harmless evaluation</figcaption>
<table id="Ax1.T34.1" class="ltx_tabular ltx_align_middle">
<tr id="Ax1.T34.1.1" class="ltx_tr">
<td id="Ax1.T34.1.1.1" class="ltx_td ltx_align_left ltx_border_l ltx_border_r ltx_border_t">[system prompt]</td>
</tr>
<tr id="Ax1.T34.1.2" class="ltx_tr">
<td id="Ax1.T34.1.2.1" class="ltx_td ltx_align_left ltx_border_l ltx_border_r">
<table id="Ax1.T34.1.2.1.1" class="ltx_tabular ltx_align_middle">
<tr id="Ax1.T34.1.2.1.1.1" class="ltx_tr">
<td id="Ax1.T34.1.2.1.1.1.1" class="ltx_td ltx_nopad_r ltx_align_left">As a neutral observer, your task is to assess the potential harm of the responses provided by two</td>
</tr>
<tr id="Ax1.T34.1.2.1.1.2" class="ltx_tr">
<td id="Ax1.T34.1.2.1.1.2.1" class="ltx_td ltx_nopad_r ltx_align_left">AI assistants to the user question shown below. Consider any negative impact that the responses</td>
</tr>
<tr id="Ax1.T34.1.2.1.1.3" class="ltx_tr">
<td id="Ax1.T34.1.2.1.1.3.1" class="ltx_td ltx_nopad_r ltx_align_left">may have on the user and society as a whole. Begin by comparing the two responses and provide</td>
</tr>
<tr id="Ax1.T34.1.2.1.1.4" class="ltx_tr">
<td id="Ax1.T34.1.2.1.1.4.1" class="ltx_td ltx_nopad_r ltx_align_left">a brief explanation. Avoid any biases based on position and ensure that the order in which the</td>
</tr>
<tr id="Ax1.T34.1.2.1.1.5" class="ltx_tr">
<td id="Ax1.T34.1.2.1.1.5.1" class="ltx_td ltx_nopad_r ltx_align_left">responses were presented does not influence your decision. Do not let the length of the responses</td>
</tr>
<tr id="Ax1.T34.1.2.1.1.6" class="ltx_tr">
<td id="Ax1.T34.1.2.1.1.6.1" class="ltx_td ltx_nopad_r ltx_align_left">influence your evaluation. Do not favor certain names of the assistants. Strive to be as objective</td>
</tr>
<tr id="Ax1.T34.1.2.1.1.7" class="ltx_tr">
<td id="Ax1.T34.1.2.1.1.7.1" class="ltx_td ltx_nopad_r ltx_align_left">as possible. Output your final verdict using this format: ”[[A]]” if assistant A’s answer is less</td>
</tr>
<tr id="Ax1.T34.1.2.1.1.8" class="ltx_tr">
<td id="Ax1.T34.1.2.1.1.8.1" class="ltx_td ltx_nopad_r ltx_align_left">harmful, ”[[B]]” if assistant B’s answer is less harmful, and ”[[C]]” for an equal level of harm.</td>
</tr>
<tr id="Ax1.T34.1.2.1.1.9" class="ltx_tr">
<td id="Ax1.T34.1.2.1.1.9.1" class="ltx_td ltx_nopad_r ltx_align_left">After output the format, you should give a short explanation.</td>
</tr>
</table>
</td>
</tr>
<tr id="Ax1.T34.1.3" class="ltx_tr">
<td id="Ax1.T34.1.3.1" class="ltx_td ltx_align_left ltx_border_l ltx_border_r">[User Question]</td>
</tr>
<tr id="Ax1.T34.1.4" class="ltx_tr">
<td id="Ax1.T34.1.4.1" class="ltx_td ltx_align_left ltx_border_l ltx_border_r">{question}</td>
</tr>
<tr id="Ax1.T34.1.5" class="ltx_tr">
<td id="Ax1.T34.1.5.1" class="ltx_td ltx_align_left ltx_border_l ltx_border_r">[Assistant A’s Answer]</td>
</tr>
<tr id="Ax1.T34.1.6" class="ltx_tr">
<td id="Ax1.T34.1.6.1" class="ltx_td ltx_align_left ltx_border_l ltx_border_r">{answer_a}</td>
</tr>
<tr id="Ax1.T34.1.7" class="ltx_tr">
<td id="Ax1.T34.1.7.1" class="ltx_td ltx_align_left ltx_border_l ltx_border_r">[Assistant B’s Answer]</td>
</tr>
<tr id="Ax1.T34.1.8" class="ltx_tr">
<td id="Ax1.T34.1.8.1" class="ltx_td ltx_align_left ltx_border_b ltx_border_l ltx_border_r">{answer_b}</td>
</tr>
</table>
</figure>
<figure id="Ax1.T35" class="ltx_table">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table">Table 35: </span>System prompt and prompt template applied for Helpful evaluation</figcaption>
<table id="Ax1.T35.1" class="ltx_tabular ltx_align_middle">
<tr id="Ax1.T35.1.1" class="ltx_tr">
<td id="Ax1.T35.1.1.1" class="ltx_td ltx_align_left ltx_border_l ltx_border_r ltx_border_t">[system prompt]</td>
</tr>
<tr id="Ax1.T35.1.2" class="ltx_tr">
<td id="Ax1.T35.1.2.1" class="ltx_td ltx_align_left ltx_border_l ltx_border_r">
<table id="Ax1.T35.1.2.1.1" class="ltx_tabular ltx_align_middle">
<tr id="Ax1.T35.1.2.1.1.1" class="ltx_tr">
<td id="Ax1.T35.1.2.1.1.1.1" class="ltx_td ltx_nopad_r ltx_align_left">As an impartial judge, your task is to evaluate the helpfulness of the responses provided by two</td>
</tr>
<tr id="Ax1.T35.1.2.1.1.2" class="ltx_tr">
<td id="Ax1.T35.1.2.1.1.2.1" class="ltx_td ltx_nopad_r ltx_align_left">AI assistants to the user question shown below. Consider how well each assistant addresses the</td>
</tr>
<tr id="Ax1.T35.1.2.1.1.3" class="ltx_tr">
<td id="Ax1.T35.1.2.1.1.3.1" class="ltx_td ltx_nopad_r ltx_align_left">user’s query and provides useful information. Begin your evaluation by comparing the two res-</td>
</tr>
<tr id="Ax1.T35.1.2.1.1.4" class="ltx_tr">
<td id="Ax1.T35.1.2.1.1.4.1" class="ltx_td ltx_nopad_r ltx_align_left">ponses and provide a brief explanation. Avoid any positional biases and ensure that the order</td>
</tr>
<tr id="Ax1.T35.1.2.1.1.5" class="ltx_tr">
<td id="Ax1.T35.1.2.1.1.5.1" class="ltx_td ltx_nopad_r ltx_align_left">in which the responses were presented does not influence your decision. Do not allow the length</td>
</tr>
<tr id="Ax1.T35.1.2.1.1.6" class="ltx_tr">
<td id="Ax1.T35.1.2.1.1.6.1" class="ltx_td ltx_nopad_r ltx_align_left">of the responses to influence your evaluation. Do not favor certain names of the assistants.</td>
</tr>
<tr id="Ax1.T35.1.2.1.1.7" class="ltx_tr">
<td id="Ax1.T35.1.2.1.1.7.1" class="ltx_td ltx_nopad_r ltx_align_left">Be as objective as possible. Output your final verdict by strictly following this format: ”[[A]]”</td>
</tr>
<tr id="Ax1.T35.1.2.1.1.8" class="ltx_tr">
<td id="Ax1.T35.1.2.1.1.8.1" class="ltx_td ltx_nopad_r ltx_align_left">if assistant A’s answer is more helpful, ”[[B]]” if assistant B’s answer is more helpful, and ”[[C]]”</td>
</tr>
<tr id="Ax1.T35.1.2.1.1.9" class="ltx_tr">
<td id="Ax1.T35.1.2.1.1.9.1" class="ltx_td ltx_nopad_r ltx_align_left">for a tie. After output the format, you should give a short explanation.</td>
</tr>
</table>
</td>
</tr>
<tr id="Ax1.T35.1.3" class="ltx_tr">
<td id="Ax1.T35.1.3.1" class="ltx_td ltx_align_left ltx_border_l ltx_border_r">[User Question]</td>
</tr>
<tr id="Ax1.T35.1.4" class="ltx_tr">
<td id="Ax1.T35.1.4.1" class="ltx_td ltx_align_left ltx_border_l ltx_border_r">{question}</td>
</tr>
<tr id="Ax1.T35.1.5" class="ltx_tr">
<td id="Ax1.T35.1.5.1" class="ltx_td ltx_align_left ltx_border_l ltx_border_r">[Assistant A’s Answer]</td>
</tr>
<tr id="Ax1.T35.1.6" class="ltx_tr">
<td id="Ax1.T35.1.6.1" class="ltx_td ltx_align_left ltx_border_l ltx_border_r">{answer_a}</td>
</tr>
<tr id="Ax1.T35.1.7" class="ltx_tr">
<td id="Ax1.T35.1.7.1" class="ltx_td ltx_align_left ltx_border_l ltx_border_r">[Assistant B’s Answer]</td>
</tr>
<tr id="Ax1.T35.1.8" class="ltx_tr">
<td id="Ax1.T35.1.8.1" class="ltx_td ltx_align_left ltx_border_b ltx_border_l ltx_border_r">{answer_b}</td>
</tr>
</table>
</figure>
</section>
</section>
</article>
</div>
<div class="ar5iv-footer"><a href="/html/2310.06761" class="ar5iv-nav-button ar5iv-nav-button-prev">◄</a>
    <a class="ar5iv-home-button" href="/"><img height="40" alt="ar5iv homepage" src="/assets/ar5iv.png"></a>
    <a href="/feeling_lucky" class="ar5iv-text-button">Feeling<br>lucky?</a>
    <a href="/log/2310.06762" class="ar5iv-text-button ar5iv-severity-warning">Conversion<br>report</a>
    <a class="ar5iv-text-button" target="_blank" href="https://github.com/dginev/ar5iv/issues/new?template=improve-article--arxiv-id-.md&title=Improve+article+2310.06762">Report<br>an issue</a>
    <a href="https://arxiv.org/abs/2310.06762" class="ar5iv-text-button arxiv-ui-theme">View&nbsp;original<br>on&nbsp;arXiv</a><a href="/html/2310.06763" class="ar5iv-nav-button ar5iv-nav-button-next">►</a>
</div><footer class="ltx_page_footer">
<a class="ar5iv-toggle-color-scheme" href="javascript:toggleColorScheme()" title="Toggle ar5iv color scheme"><span class="color-scheme-icon"></span></a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/license" target="_blank">Copyright</a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/policies/privacy_policy" target="_blank">Privacy Policy</a>

<div class="ltx_page_logo">Generated  on Tue Feb 27 22:57:24 2024 by <a target="_blank" href="http://dlmf.nist.gov/LaTeXML/" class="ltx_LaTeXML_logo"><span style="letter-spacing:-0.2em; margin-right:0.1em;">L<span class="ltx_font_smallcaps" style="position:relative; bottom:2.2pt;">a</span>T<span class="ltx_font_smallcaps" style="font-size:120%;position:relative; bottom:-0.2ex;">e</span></span><span style="font-size:90%; position:relative; bottom:-0.2ex;">XML</span><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAsAAAAOCAYAAAD5YeaVAAAAAXNSR0IArs4c6QAAAAZiS0dEAP8A/wD/oL2nkwAAAAlwSFlzAAALEwAACxMBAJqcGAAAAAd0SU1FB9wKExQZLWTEaOUAAAAddEVYdENvbW1lbnQAQ3JlYXRlZCB3aXRoIFRoZSBHSU1Q72QlbgAAAdpJREFUKM9tkL+L2nAARz9fPZNCKFapUn8kyI0e4iRHSR1Kb8ng0lJw6FYHFwv2LwhOpcWxTjeUunYqOmqd6hEoRDhtDWdA8ApRYsSUCDHNt5ul13vz4w0vWCgUnnEc975arX6ORqN3VqtVZbfbTQC4uEHANM3jSqXymFI6yWazP2KxWAXAL9zCUa1Wy2tXVxheKA9YNoR8Pt+aTqe4FVVVvz05O6MBhqUIBGk8Hn8HAOVy+T+XLJfLS4ZhTiRJgqIoVBRFIoric47jPnmeB1mW/9rr9ZpSSn3Lsmir1fJZlqWlUonKsvwWwD8ymc/nXwVBeLjf7xEKhdBut9Hr9WgmkyGEkJwsy5eHG5vN5g0AKIoCAEgkEkin0wQAfN9/cXPdheu6P33fBwB4ngcAcByHJpPJl+fn54mD3Gg0NrquXxeLRQAAwzAYj8cwTZPwPH9/sVg8PXweDAauqqr2cDjEer1GJBLBZDJBs9mE4zjwfZ85lAGg2+06hmGgXq+j3+/DsixYlgVN03a9Xu8jgCNCyIegIAgx13Vfd7vdu+FweG8YRkjXdWy329+dTgeSJD3ieZ7RNO0VAXAPwDEAO5VKndi2fWrb9jWl9Esul6PZbDY9Go1OZ7PZ9z/lyuD3OozU2wAAAABJRU5ErkJggg==" alt="Mascot Sammy"></a>
</div></footer>
</div>

    <script>
      var canMathML = typeof(MathMLElement) == "function";
      if (!canMathML) {
        var body = document.querySelector("body");
        body.firstElementChild.setAttribute('style', 'opacity: 0;');
        var loading = document.createElement("div");
        loading.setAttribute("id", "mathjax-loading-spinner");
        var message = document.createElement("div");
        message.setAttribute("id", "mathjax-loading-message");
        message.innerText = "Typesetting Equations...";
        body.prepend(loading);
        body.prepend(message);

        var el = document.createElement("script");
        el.src = "https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js";
        document.querySelector("head").appendChild(el);

        window.MathJax = {
          startup: {
            pageReady: () => {
              return MathJax.startup.defaultPageReady().then(() => {
                body.removeChild(loading);
                body.removeChild(message);
                body.firstElementChild.removeAttribute('style');
              }); } } };
      }
    </script>
    <script>
    // Auxiliary function, building the preview feature when
    // an inline citation is clicked
    function clicked_cite(e) {
      e.preventDefault();
      let cite = this.closest('.ltx_cite');
      let next = cite.nextSibling;
      if (next && next.nodeType == Node.ELEMENT_NODE && next.getAttribute('class') == "ar5iv-bibitem-preview") {
        next.remove();
        return; }
      // Before adding a preview modal,
      // cleanup older previews, in case they're still open
      document.querySelectorAll('span.ar5iv-bibitem-preview').forEach(function(node) {
        node.remove();
      })

      // Create the preview
      preview = document.createElement('span');
      preview.setAttribute('class','ar5iv-bibitem-preview');
      let target = document.getElementById(this.getAttribute('href').slice(1));
      target.childNodes.forEach(function (child) {
        preview.append(child.cloneNode(true));
      });
      let close_x = document.createElement('button');
      close_x.setAttribute("aria-label","Close modal for bibliography item preview");
      close_x.textContent = "×";
      close_x.setAttribute('class', 'ar5iv-button-close-preview');
      close_x.setAttribute('onclick','this.parentNode.remove()');
      preview.append(close_x);
      preview.querySelectorAll('.ltx_tag_bibitem').forEach(function(node) {
        node.remove();
      });
      cite.parentNode.insertBefore(preview, cite.nextSibling);
      return;
    }
    // Global Document initialization:
    // - assign the preview feature to all inline citation links
    document.querySelectorAll(".ltx_cite .ltx_ref").forEach(function (link) {
      link.addEventListener("click", clicked_cite);
    });
    </script>
    </body>
</html>

2310.06762
Aligned large language models (LLMs)  demonstrate exceptional capabilities in task-solving, following instructions, and ensuring safety. However, the continual learning aspect of these aligned LLMs has been largely overlooked.
Existing continual learning benchmarks lack sufficient challenge for leading aligned LLMs, owing to both their simplicity and the models’ potential exposure during instruction tuning.
In this paper, we introduce TRACE, a novel benchmark designed to evaluate continual learning in LLMs. TRACE consists of 8 distinct datasets spanning challenging tasks including domain-specific tasks, multilingual capabilities, code generation, and mathematical reasoning.
All datasets are standardized into a unified format, allowing for effortless automatic evaluation of LLMs.
Our experiments show that after training on TRACE, aligned LLMs exhibit significant declines in both general ability and instruction-following capabilities.
For example, the accuracy of llama2-chat 13B on gsm8k dataset declined precipitously from 28.8% to 2% after training on our datasets.
This highlights the challenge of finding a suitable tradeoff between achieving performance on specific tasks while preserving the original prowess of LLMs.
Empirical findings suggest that tasks inherently equipped with reasoning paths contribute significantly to preserving certain capabilities of LLMs against potential declines.
Motivated by this, we introduce the Reasoning-augmented Continual Learning (RCL)  approach. RCL integrates task-specific cues with meta-rationales, effectively reducing catastrophic forgetting in LLMs while expediting convergence on novel tasks.
abstract
TRACE: A Comprehensive Benchmark for Continual Learning in Large Language Models
Abstract
2310.06762
1IntroductionLarge Language Models (LLMs) have revolutionized natural language processing through a two-step process: initial pretraining on extensive corpora, followed by fine-tuning on human-generated instructions and preference data, aligning them with human language and intentions.
Aligned LLMs have showcased impressive capabilities and ensured safer responses. However, as the demands for language models grow, there’s a pressing need to enhance their abilities in areas such as domain-specific knowledge, multilingual proficiency, complex task-solving, and tool usage.
Yet, retraining and realigning them from scratch to meet these demands is impractical due to prohibitive training costs and the challenge of acquiring high-quality data.
Therefore, incrementally training existing Aligned LLMs through continual learning (CL)  is crucial.
This prompts the pressing question:To what degree do Aligned LLMs exhibit catastrophic forgetting when subjected to incremental training?Existing continual learning benchmarksare not suitable for evaluating the state-of-the-art LLMs. Firstly, many of these benchmarks predominantly consist of simplistic natural language understanding datasets. These tasks, due to their inherent simplicity, fail to challenge the capabilities of large-scale models adequately.
Furthermore, a significant drawback lies in the fact that some of these datasets have previously appeared in the instruction tuningsets of these LLMs, suggesting that the models may have already learned these tasks during training.
Secondly, prior benchmarks have primarily focused on metrics that assess the performance of the models on target sequential tasks.
Yet, for aligned models, aspects like generalization to new tasks, the ability to follow human instructions, and safety preservation are of paramount importance. Regrettably, these dimensions have not been extensively studied or incorporated into assessments.To facilitate further research, we presentTRACE, a continual learning benchmark designed for aligned Large Language Models.
Our benchmark consists of 8 distinct datasets spanning challenging tasks including domain-specific tasks, multilingual capabilities, code generation, and mathematical reasoning.
To ensure task balance, we have sampled 5,000 instances for each task, and for classification tasks, we have ensured an equal distribution of samples across all classes.
Additionally, all datasets have been standardized into a unified format, simplifying the evaluation process.
To evaluate continual learning in aligned LLMs, we introduce three metrics: ”General Ability Delta,” ”Instruction Following Delta,” and ”Safety Delta” to assess models’ forgetfulness in such scenarios.We conduct a comprehensive evaluation of 5 aligned LLMs on TRACE.
Evaluation results reveal several key findings: 1)  Nearly all models exhibit a significant decline in general abilities, especially in math and reasoning. For instance, when trained on TRACE, the accuracy of llama2-chat 13B on the gsm8k dataset dropped from 28.8% to a mere 2%.
2)  Unlike other skills, LLMs’ multilingual abilities generally improve. For example, llama2-chat 7B’s performance on the TydiQA dataset surged from an F1 score of 23.47 to 33.23.
3)  Full-parameter training, compared to LoRA training, more easily fits the target tasks, but it also leads to a more pronounced decline in general abilities.
4)  LLMs’ instruction-following capabilities also suffer a significant reduction after continual learning.Figure 1:An overview of TRACE benchmark. TRACE consists of two main components: 1)  A selection of eight datasets constituting a tailored set of tasks for continual learning, covering challenges in domain-specific tasks, multilingual capabilities, code generation, and mathematical reasoning. 2)  A post-training evaluation of LLM capabilities. In addition to traditional continual learning metrics, we introduce General Ability Delta, Instruction Following Delta, and Safety Delta to evaluate shifts in LLM’s inherent abilities.Through experimentation, we observed that tasks augmented with reasoning paths are notably effective in preserving certain capabilities of LLMs, preventing them from substantial declines. Such findings lead us to ponder on leveraging a model’s inherent strengths for rapid transfer on new tasks, rather than starting the learning curve from scratch. This motivation birthed our novel training strategy: Reasoning-augmented Continual Learning (RCL) . RCL prompts the model to generate task analyses and rationales during training. As our results indicate, this approach not only boosts performance on target tasks but also significantly upholds the inherent strengths of LLMs.111The dataset, code can be found athttps://github.com/BeyonderXX/TRACE.
introduction
TRACE: A Comprehensive Benchmark for Continual Learning in Large Language Models
1Introduction
2310.06762
2Related Work2.1Continual LearningContinual learningaims to develop learning algorithms that can accumulate knowledge on non-stationary data. Existing works can be broadly categorized into rehearsal-based, regularization-based, and architecture-based approaches. Rehearsal-based approachesleverage a memory buffer that stores examples from previous tasks, training the model jointly with the current task.
Experience replay (ER) is a common strategy employed in rehearsal-based approaches and serves as a strong baseline.
Regularization-based approachesincorporate additional terms into the loss function to penalize changes in crucial weights.
For example, Orthogonal Gradient Descent (OGD) constrains the parameters to move within the orthogonal space defined by the gradients of previous tasks, Elastic Weight Consolidation (EWC) constraint important parameters to stay close to their previous ones through, Gradient Episodic Memory (GEM) leverages episode memories to avoid forgetting.
Architecture-based approachesfocus on dynamically expanding model capacity or isolating existing model weights to mitigate interference between new and old tasks.
Progressive Promptslearns separate prompts for each incoming task and sequentially concatenates them with previously learned prompts.2.2CL Benchmarks in NLPThe most recognized CL benchmark for NLP encompasses five text classification datasets introduced by, including AG News, Amazon Reviews, Yelp Reviews, DBpedia, and Yahoo Answers. Building upon this,proposed a long CL benchmark which fuses the aforementioned five datasets with an additional four from the GLUE benchmark, five from the SuperGLUE benchmark, and the IMDB dataset.
While these datasets have been incorporated into the Flan collectionand are widely used for current SoTA LLMs, their ubiquity has rendered them less suitable as CL benchmarks for LLMs.
Taking a different approach,focuses on English language generation tasks. In a subsequent study,conducted an analysis of catastrophic forgetting on Bloomzusing’s datasets. However, this benchmark is limited in scope as it solely emphasizes Natural Language Generation (NLG)  tasks and is restricted to English, thus lacking task diversity.
In contrast, TRACE offers a diverse and challenging array of sequential tasks. Additionally, it evaluates aligned models on aspects such as general capability, adherence to instructions, and shifts in safety measures.2.3Chain-of-ThoughtLLMs have shown advanced step-by-step reasoning capabilities, known as Chain-of-Thought (CoT)  reasoning.
Zero-shot-CoTillustrates the profound impact of simply prefacing a reasoning sequence with the sentence, ”Let’s think step by step.” Least-to-most, actively prompts LLMs to segment complex problems into smaller tasks. ScienceQAhighlights the efficacy of CoT in LLMs, particularly beneficial for filling the void in datasets within the scientific realm.
Fine-tune-CoTexploits the prowess of extensive LLMs to generate reasoning samples, further fine-tuning smaller models.
Despite these advancements, the application of CoT in continual learning remains unexplored. Our benchmark, TRACE, showcases that generating explanations can not only accelerate the learning process but also significantly mitigate the forgetting in their foundational capabilities.
body
TRACE: A Comprehensive Benchmark for Continual Learning in Large Language Models
2Related Work
2310.06762
3PreliminariesContinual learningfocuses on developing learning algorithms to accumulate knowledge on non-stationary data. In supervised continual learning, a sequence of tasksarrive in a streaming fashion. Each taskcontains a separate target dataset, where,.
A single model needs to adapt to them sequentially, with only access toat the t-th task.
In general, given a prediction modelparameterized by, continual learning seeks to optimize for the following objective across all tasks:(1) In this paper, we utilize overall performance (OP)  and backward transfer (BWT)  scores as the main metrics. After incrementally learning the t-th task, the model’s score on the i-th task (where)  is denoted as. The overall performance and backward transfer score are calculated using the following formulas:(2) (3) 
body
TRACE: A Comprehensive Benchmark for Continual Learning in Large Language Models
3Preliminaries
2310.06762
4TRACE: A Comprehensive Benchmark for CL in LLMsTRACE is designed to offer a comprehensive continual learning evaluation for LLMs. Illustrated in Figure1, TRACE encompasses two primary components: a curated set of tasks tailored for continual learning, followed by an in-depth evaluation of an LLM’s post-training capabilities.
In this section, we detail TRACE’s sequential tasks and introduce our evaluation metrics. In Section4.4, we evaluate five models using TRACE and present our key findings.4.1Data CreationThere are three principles for the creation of TRACE. First, the datasets should be novel enough that most LLMs have not been trained on them. Second, they should be challenging for large language models. Third, a variety of tasks should be covered in our benchmark.According to these three principles, in this section, we will provide a detailed introduction to the data collection process for each dataset. As these datasets have varying sizes, we create a balanced version by randomly sampling 5000 training examples and 2000 testing examples from the original datasets. As shown in Table4, we get 40,000 training examples and 16,000 testing examples in total.Domain-Specific.These tasks require specific knowledge, so models may perform poorly if they have not appeared frequently enough in the training data. We select three datasets, ScienceQA, FOMCand MeetingBank.
ScienceQA is a multi-hop QA dataset collected from elementary and high school science curricula, with a rich domain diversity from natural science, social science, and language science, requiring the model of reasoning ability and science knowledge.
As we only test the performance of language models, only the examples without multi-modal contexts are included in TRACE. FOMC is a hawkish-dovish classification task, which is novel in the financial domain. The dataset is divided into three subsets: data on meeting minutes, press conference data, and speech data. We use a combination of them. MeetingBank is a new benchmark dataset for city council meeting summarization, an unstudied domain. It demands a global understanding of the whole long context.Multi-lingual.The cross-lingual ability of large language models is limited due to vocabulary and pre-training corpus. For instance, LLaMA’s vocabulary contains few Chinese tokens, affecting its efficiency with Chinese text.expand LLaMA’s Chinese vocabulary and fine-tune with additional Chinese corpus. Yet, capabilities for other languages can be forgotten after training in a specific language, making it vital to evaluate cross-lingual ability in our benchmark.
We select C-STANCEand 20Minutenas multi-lingual datasets. C-STANCE is the first Chinese dataset for zero-shot stance detection collected from Sina Weibo, one of the most popular Chinese social media sites. It includes two challenging subtasks: target-based stance detection and domain-based stance detection. In TRACE, we include the target-based one, which means the targets in testing examples are unseen during training. 20Minuten is a text simplification dataset consisting of full articles paired with shortened, simplified summaries from the Swiss news magazine. We use this dataset to evaluate the ability to generate German text.Code completion.Code completion is another challenging task to evaluate long context modeling ability, and it is one of the most widely used features in software development through IDEs. We select the line-level code completion task of CodeXGLUE, which requires the model to generate the next line given the lengthy code input. The corpus Py150 contains 150,000 Python programs collected from GitHub repositories. Since the golden labels of the testing dataset are not available by, we randomly divide each Python code in Py150 into two parts, taking the first part as inputs and the next line as labels.Mathematical reasoning.Mathematical problems are always used to evaluate the reasoning ability of models. NumGLUEis an 8-task benchmark far from solved including state-of-the-art large-scale language models performing significantly worse than humans. We include the first two tasks of NumGLUE because they are freshly collected data and intelligent modifications of already existing datasets. Both of the two tasks require arithmetic reasoning ability.
It is worth noting that both datasets have original labels consisting only of numbers, without associated inference processes.4.2CL Metrics DesignUnlike traditional continual learning benchmarks focused on sequential target tasks, evaluating aligned LLMs should also account for the preservation of their inherent capabilities. SoTA LLMs, through instruction tuning, exhibit impressive task-solving abilities. Aligning these models with human preferences further boosts their safety and usefulness. Hence, TRACE introduces a broader evaluation, including three unique metrics: ”General Ability Delta,” ”Instruction Following Delta,” and ”Safety Delta.” These measure the changes in LLM’s general capability, instruction adherence, and response safety post-continual learning.General Ability Deltais designed to assess the change in performance of an LLM on generic tasks after training on sequential target tasks.
Let’s consider a set of general tasks denoted as. The baseline performance of the initial LLM on the i-th general task is represented by. After incrementally learning up to the t-th task, the score on the i-th general task becomes. The ”General Ability Delta” after training on the t-th task, represented as, is given by:(4) Instruction Following Deltameasures the change in a model’s ability to follow instructions after training on sequential tasks. Using a set of datasets, represented as, the initial LLM performance on the i-th task is. After incremental learning to the t-th task, its score on the i-th task is. The change, represented by, is computed as:(5) Safety Deltaquantifies the change in a model’s response safety after sequential training. Using a set of datasets designed for safety evaluation, denoted as, the initial safety metric on the i-th dataset is. After training up to the t-th task, its score on the i-th dataset is. The change after the t-th task, represented by, is computed as:(6) 4.3Experimental Setup4.3.1BaselinesWe evaluate the performance of LLMs in a continual learning setting using four approaches—three requiring training and one not:Sequential Full-Parameter Fine-Tuning (SeqFT) : This method involves training all model parameters in sequence.LoRA-based Sequential Fine-Tuning (LoraSeqFT) : Only the low-rank LoRA matrices are fine-tuned, leaving the LLM backbone fixed. This method is chosen based on prior findings of reduced forgetting with ”Efficient Tuning”.Replay-based Sequential Fine-Tuning (Replay) : Replay, a common continual learning strategy, is employed for its simplicity and effectiveness. We incorporate alignment data from LIMA into the replay memory, replaying 10% of historical data.In-Context Learning (ICL) : Task demonstrations are supplied as part of the language prompt, acting as a form of prompt engineering. A 6-shot setting is used for our experiments.To evaluate the resilience of safety alignment models from diverse training backgrounds and strategies, we select five aligned models from three organizations: Meta: LLaMa-2-7B-Chat, LLaMa-2-13B-Chat, BaiChuan: Baichuan 2-7B-Chat, and Large Model Systems Organization: Vicuna-13B-V1.5, Vicuna-7B-V1.5.4.3.2DatasetsTo evaluate a model’sgeneral ability, we assess across five key dimensions: Factual Knowledge, General Reasoning, Multilinguality, Commonsense Reasoning, and Reading Comprehension.Factual Knowledge: We use the Massive Multitask Language Understanding dataset (MMLU)  with questions on 57 subjects, from elementary to professional levels. Following LLaMa-2, we report 5-shot accuracy based on answer perplexity.General Reasoning: Evaluated using Big-Bench-Hard (BBH)  with 23 tasks from Big-Bench. The evaluation uses chain-of-thought prompts with 3-shot in-context examples, and EM scores are reported.Multilinguality: We use TyDiQA, a multilingual QA benchmark with 11 languages. Using the gold-passage setup, 0-shot F1 scores are reported.Commonsense Reasoning: Evaluated using PIQA. Following LLaMa-2, we report 0-shot accuracy based on answer perplexity.Reading Comprehension: Assessed with BoolQ, containing 15942 questions. We report 0-shot accuracy based on answer perplexity.Forinstruction-following capability, we use Self-instruct dataset, which is user-oriented and comprises a diverse set of 175 prompts, encompassing areas like email writing, social media, productivity tools, entertainment, and programming. We also employ LIMA datasetLIMA assembles 300 prompts, primarily sourced from community Q&A forums and supplemented by manually authored examples.For assessing changes insafety, we leverage the CoNa dataset. This corpus encompasses 178 expert-annotated samples, specifically curated to address instructions associated with hateful speech generation.Table 1:OP(BWT)  for all the baseline models and 3 baseline methods.ICLSeqFTLoraSeqFTReplayLLaMA-2-7B-ChatLLaMA-2-13B-ChatVicuna-7B-V1.5Vicuna-13B-V1.5Baichuan2-7B-Instruct4.3.3MetricsAs mentioned in Section3, we measure the performance of LLMs in continual learning tasks using Overall Performance and Backward Transfer Score. To check how well LLMs keep their original abilities, we use General Ability Delta, Instruction Following Delta, and Safety Delta. More details are in Section4.2. For evaluating instruction-following and safety, we score with GPT-4. More details about this scoring can be found in the Appendix.7.4.3.4Implementation DetailsThe detailed settings can be found in Appendix.1.4.4Main Results4.4.1Performance of Target Sequential TasksTable1showcases the performance of five distinct LLMs on TRACE benchmark, after their continual learning phase. From this evaluation, we can draw the following conclusions:In-Context Learning (ICL)  Performance: ICL methods generally perform lower than SeqFT and Replay methods. This suggests that the TRACE benchmark is indeed challenging, and LLMs can’t readily identify solutions just through simple demonstrations.Replay Performance: Among all the baselines, Replay achieved the highest OP score. With its BWT score being positive, it indicates that Replay effectively retains its performance on sequential tasks without significant forgetting. This makes Replay a straightforward and efficient strategy in a continual learning context.Full Parameter Training vs. LoRA: Full parameter training demonstrates better task-specific adaptability compared to LoRA, with a smaller BWT score. For instance, LLaMA-2-7B-Chat’s SeqFT OP(BWT)  is 48.7 (8.3%) , while LoRASeqFT stands at 12.7 (45.7%) . This suggests that when the focus is primarily on sequential tasks, full parameter fine-tuning should be prioritized over parameter-efficient methods like LoRA.Table 2:Comparison of the general language understanding and reasoning abilities.bluemeans increase, whileredmeans decrease.MMLU(factuality) GSM(math) BBH(reasoning) TydiQA(multilinguality) BoolQ(comprehension) PIQA(commonsense) ACC(5-shot) EM(8-shot, CoT) EM(3-shot, CoT) F1(1-shot, GP) ACC(0-shot) ACC(0-shot) LLaMA-2-7B-ChatLLaMA-2-7B-Chat-SeqLLaMA-2-7B-Chat-LoraSeqLLaMA-2-7B-Chat-ReplayLLaMA-2-13B-ChatLLaMA-2-13B-Chat-SeqLLaMA-2-13B-Chat-LoraSeqLLaMA-2-13B-Chat-ReplayBaichuan2-7B-InstructBaichuan2-7B-Instruct-SeqBaichuan2-7B-Instruct-LoraSeqBaichuan2-7B-Instruct-ReplayVicuna-7B-V1.5Vicuna-7B-V1.5-SeqVicuna-7B-V1.5-LoraSeqVicuna-7B-V1.5-ReplayVicuna-13B-V1.5Vicuna-13B-V1.5-SeqVicuna-13B-V1.5-LoraSeqVicuna-13B-V1.5-Replay(a) Helpful evaluation(b) Safety evaluationFigure 2:GPT-4 evaluation with llama-13b-chat, comparing 3 different baselines (Replay, LoRA and Sequential)  to the base model across tasks including helpful and safety.4.4.2Variation of General AbilityTable2presents the evaluations of various LLM models concerning general abilities. The degree of general ability forgetting in LLMs can be analyzed from three perspectives. For a more detailed evaluation, refer to the Appendix.From the Model Perspective:1) Nearly all models display a negative General Ability Delta, indicating a general decline in overall capabilities after continual learning.2) Larger models, in comparison to their smaller counterparts, show a more pronounced forgetting in factual knowledge and reasoning tasks. For instance, the General Ability Delta for llama2-13B-chat-Seq stands at 41.88, whereas the SeqFT version for llama2-7B is 46.43.From the Task Perspective:1) Despite the presence of CoT prompts, there is a noticeable decline in math and reasoning abilities across all models, suggesting that these abilities are highly sensitive to new task learning.2) Excluding the llama2-7b model, most models exhibit a significant drop in performance on MMLU, suggesting a gradual loss of factual knowledge through continual learning.3) TydiQA task sees a general boost post-training, possibly due to the inclusion of Chinese and German datasets in our sequential tasks. Even more intriguing is the observed enhancement (and some declines)  in other languages on TydiQA, suggesting potential cross-linguistic transfer characteristics.4) Performance shifts on PIQA for most models are subtle, indicating the relative robustness of commonsense knowledge during continual learning.From the Methodological Perspective:1) The Replay method proves beneficial in preserving reasoning and factuality skills. Especially for larger models, the mitigation of forgetting through Replay is more pronounced. For instance, for LLaMA-2-7B-Chat, Replay offers a 6.5 EM score boost compared to methods without Replay, while for LLaMA-2-13B-Chat, the increase is 17.1 EM score.4.4.3Instruction Following Ability AnalysisWe evaluated the instruction-following ability of models based on two foundation models: LLaMA-2-7B-Chat and LLaMA-2-13B-Chat. Figure2(a)  illustrates the win rate % for instruction following sequentially trained LLMs and their original versions. Here, the win rate can be approximated as an indicator for the Instruction-following delta. It’s evident that all three training methods exhibit a marked decline in instruction-following capabilities compared to their initial versions, with the decline being most pronounced in the LoRA method. Therefore, be cautious when exploring approaches like LoRA for continual learning in LLMs.4.4.4Safety AnalysisWe tested the safety of answers from models LLaMA-2-7B-Chat and LLaMA-2-13B-Chat. Figure2(b)  shows the win rate % for instruction following between the new LLMs and their starting versions. Here, the win rate can be used as a measure for the Safety Delta. Compared to the original models, most answers were rated as ’Tie’. This suggests that the safety of the model’s answers is largely unaffected by continual learning on general tasks.4.5Influencing Factors of Forgetting in LLMs4.5.1Data Quantity & Training StepsFigure4shows the performance on target tasks for continual learning datasets with different data volumes and training steps. For LLaMA-2-7B-Chat’s SeqFT, we tested with 500, 1000, and 5000 samples from each dataset, training them for 1, 3, 5, 10 epochs. Performance improves as data volume grows, indicating at least 5000 samples from the TRACE-selected datasets are needed for full fitting. Additionally, performance improves with up to 5 training epochs, confirming our baseline epoch setting balances target task optimization and retaining existing capabilities.Figure 3:Performance evaluation of LLaMA-2-7B-Chat’s SeqFT on the TRACE benchmark across varying sample sizes (500, 1000, 5000)  and training epochs (1, 3, 5, 10 (except for 5000) ) .Figure 4:Evolution of LLMs’ reasoning capabilities post-training on different tasks, measured using the BBH performance metric. We report the results of LLaMA-2-7B-chat and LLaMA-2-13B-chat.4.5.2Decoupling the Impact of Different TasksFrom the results in section4.4.2, it’s evident that post-training LLMs on our benchmark, their innate reasoning and mathematical abilities see a significant dip. This brings forth the question: How exactly does the reasoning capability of LLMs transform during the continual learning process?Figure4tracks the reasoning ability (assessed via BBH performance)  after the completion of each training task. Intriguingly, we observed a surge in the model’s reasoning prowess post-training on the ScienceQA task, while it declined for other tasks. Notably, even though the two tasks from NumGLUE are mathematically inclined, their answers don’t provide a clear reasoning path. In contrast, ScienceQA does offer such a pathway in its answers. This observation suggests the potential advantage of incorporating reasoning paths during training to preserve and perhaps even enhance the model’s reasoning capability.
body
TRACE: A Comprehensive Benchmark for Continual Learning in Large Language Models
4TRACE: A Comprehensive Benchmark for CL in LLMs
2310.06762
5Reasoning-augmented Continual learningDrawing from our earlier findings, which underscored the unique capabilities of LLMs, we were inspired to reconsider how we approach their training. Instead of treating LLMs as traditional models and inundating them with large volumes of data to fit a task’s distribution, might we leverage their inherent abilities for rapid task transfer? With these insights as our foundation, we formulated the Reasoning-augmented Continual Learning (RCL)  approach. RCL forms reasoning paths on new datasets, aiming to not only preserve LLMs’ reasoning capabilities but also to enhance their task transfer and output clarity.As depicted in Figure5, RCL has two phases: automated reasoning annotation and sequential training on the augmented dataset. Domain experts created prompts for each task. Three samples per task were manually annotated. GPT-4, using these prompts, generated reasoning paths for every entry. Reasoning was verified against ground truth and underwent human checks. We relied on machine-generated answers due to cost concerns and the consistency of the LM-generated text. To validate reasoning quality, we manually inspected outputs, achieving a 94% approval rate on a 100-sample check, highlighting GPT-4’s reliability. Following this, supervised training was conducted on the target LLM, keeping hyperparameter settings consistent with baselines.5.1Performance on Target Sequential TasksTable3provides a side-by-side comparison of the performance of our RCL method against other techniques, using LLaMA-2-7B-Chat as the foundational model and limiting training samples to 500 for each task. Through an ablation study contrasting single-task training (SingleFT)  with multi-task training, and assessing the impact of reasoning-augmented data, we observed that integrating reasoning pathways into the data consistently boosts performance over the original dataset.Moreover, our approach, even when trained with just 500 samples, achieves comparable results to the SeqFT method trained on 5000 samples. Furthermore, by leveraging fewer datasets and training steps in our method, we would mitigate the decline in LLMs’ inherent capabilities.Figure 5:An overview of Reasoning-augmented continual learning method. Our method unfolds in two stages: 1)  Automatic annotation of sample reasoning paths using GPT-4. We guide GPT-4 through in-context learning and validate the generated paths via answer verification. 2)  Continual learning on reasoning-augmented dataset.Table 3:OP and BWT for different baselines.O-LoraandPPrefer to two SoTA continual learning methods. All methods exceptSeq(5k) andICLare training with 0.5k samples.Rerefers to reasoning-augmented,Single FTrefers to fine-tuning the model on single task andMTrefers to Multi-task training.Overall avg.BWTICL-ICL+Re-O-LoraPP2.3%SeqFTSeqFT(5k) 48.7RCLSingleFT-SingleFT+Re-MT w/o. Re-MT w. Re-5.2Impacts on General AbilityWe report the performance of general abilities in Figure6. We can conclude that RCL reaches comparable performance with SeqFT and Replay method on MMLU, TydiQA, BoolQA and PIQA though only. However, RCL stands out in reasoning tasks such as GSM and BBH. For instance, for the GSM task, RCL outperforms SeqFT and Replay by 12.7 and 13.2 points respectively, showing the advantages of providing reasoning paths in maintaining the reasoning abilities of models. Besides, combining RCL with replay further improves its performance on reasoning tasks.5.3Impacts on Instruction-Following and SafetyThe impact of incorporating RCL on instruction-following capabilities is presented in Table5. It’s evident that RCL enhances the model’s ability to follow instructions by 8% and 5% compared to SeqFT and Replay, respectively.
body
TRACE: A Comprehensive Benchmark for Continual Learning in Large Language Models
5Reasoning-augmented Continual learning
2310.06762
6DiscussionCan traditional continual learning methods be effectively applied to LLMs?In section2, we introduce various traditional continual learning methods, including replay-based, regularization-based, and architecture-based approaches.
Unfortunately, several characteristics of LLMs challenge the straightforward adoption of these approaches:1.High Training Cost: LLMs require significant data for both pre-training and alignment, leading to a high training cost. Using simple replay to maintain past capabilities can be very expensive. Therefore, selecting key data from past training to keep LLMs’ diverse predictive abilities is essential.2.Large Number of Parameters: The huge parameter size of LLMs demands advanced hardware for training. Many regularization techniquesneed to store gradients from past tasks, which is a big challenge for both CPU and GPU memory.3.One-for-All Deployment of LLMs: LLMs are designed for a wide range of tasks, meaning tailoring parameters for specific tasks might limit their ability to generalize to new tasks. Additionally, methods that adjust the network dynamically can complicate deployment, as it becomes tricky to handle multiple task queries at once.Figure 6:OpenCompass Evaluation Results.RCL+Replayrefers to combining our RCL method with replay method.How should LLMs approach continual learning?Our experiments with TRACE show that direct end-to-end training might compel LLMs to focus myopically on specific patterns of the target task, such as shortcuts, thereby undermining their capacities in more universal scenarios.
Intrinsically, LLMs are trained on large and varied datasets. So, they already have the skills to handle many tasks, and can even learn with very few examples.
Based on the ideas from LIMA’s Superficial Alignment Hypothesis, our focus should perhaps pivot more towards adeptly aligning LLMs’ existing capabilities to novel tasks rather than embarking on learning from scratch.
Consequently, strategies like our RCL approach, which capitalize on the LLMs’ inherent abilities for quick transfer to new tasks, might also serve as potent tools in mitigating catastrophic forgetting.
body
TRACE: A Comprehensive Benchmark for Continual Learning in Large Language Models
6Discussion
2310.06762
7ConclusionExisting benchmarks often fall short in thoroughly evaluating LLMs, either due to their simplistic nature or neglect of critical capabilities like instruction following and safety.
To tackle this, we introduced TRACE, a comprehensive benchmark with diverse challenging tasks and well-rounded metrics. Our experiments showed the real challenges LLMs face, especially a clear drop in their general abilities during continual learning. At the same time, our Reasoning-augmented Continual Learning (RCL)  method highlights the importance of using reasoning in training, even though it’s not a complete solution. We believe this area is very important and hope our work lays a solid foundation for future studies.
conclusion
TRACE: A Comprehensive Benchmark for Continual Learning in Large Language Models
7Conclusion