<!DOCTYPE html><html lang="en">
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<title>[2401.10491] Knowledge Fusion of Large Language Models</title><meta property="og:description" content="While training large language models (LLMs) from scratch can generate models with distinct functionalities and strengths, it comes at significant costs and may result in redundant capabilities. Alternatively, a cost-ef…">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="twitter:card" content="summary">
<meta name="twitter:title" content="Knowledge Fusion of Large Language Models">
<meta name="twitter:image:src" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta name="twitter:image:alt" content="ar5iv logo">
<meta property="og:title" content="Knowledge Fusion of Large Language Models">
<meta property="og:site_name" content="ar5iv">
<meta property="og:image" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta property="og:type" content="article">
<meta property="og:url" content="https://ar5iv.labs.arxiv.org/html/2401.10491">

<!--Generated on Tue Feb 27 08:19:55 2024 by LaTeXML (version 0.8.8) http://dlmf.nist.gov/LaTeXML/.-->
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">

<script>
  function detectColorScheme(){
    var theme="light";
    var current_theme = localStorage.getItem("ar5iv_theme");
    if(current_theme){
      if(current_theme == "dark"){
        theme = "dark";
      } }
    else if(!window.matchMedia) { return false; }
    else if(window.matchMedia("(prefers-color-scheme: dark)").matches) {
      theme = "dark"; }
    if (theme=="dark") {
      document.documentElement.setAttribute("data-theme", "dark");
    } else {
      document.documentElement.setAttribute("data-theme", "light"); } }

  detectColorScheme();

  function toggleColorScheme(){
    var current_theme = localStorage.getItem("ar5iv_theme");
    if (current_theme) {
      if (current_theme == "light") {
        localStorage.setItem("ar5iv_theme", "dark"); }
      else {
        localStorage.setItem("ar5iv_theme", "light"); } }
    else {
        localStorage.setItem("ar5iv_theme", "dark"); }
    detectColorScheme(); }
</script>
<link media="all" rel="stylesheet" href="/assets/ar5iv-fonts.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv-site.0.2.2.css">
</head>
<body>
<div class="ltx_page_main">
<div class="ltx_page_content">
<article class="ltx_document ltx_authors_1line">
<h1 class="ltx_title ltx_title_document">Knowledge Fusion of Large Language Models</h1>
<div class="ltx_authors">
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Fanqi Wan<sup id="id4.4.id1" class="ltx_sup">1</sup>, Xinting Huang<sup id="id5.5.id2" class="ltx_sup">2</sup>, Deng Cai<sup id="id6.6.id3" class="ltx_sup">2</sup>, Xiaojun Quan<sup id="id7.7.id4" class="ltx_sup">1</sup><sup id="id3.3.1" class="ltx_sup"><math id="id3.3.1.m1.1" class="ltx_Math" alttext="\dagger" display="inline"><semantics id="id3.3.1.m1.1a"><mo id="id3.3.1.m1.1.1" xref="id3.3.1.m1.1.1.cmml">†</mo><annotation-xml encoding="MathML-Content" id="id3.3.1.m1.1b"><ci id="id3.3.1.m1.1.1.cmml" xref="id3.3.1.m1.1.1">†</ci></annotation-xml><annotation encoding="application/x-tex" id="id3.3.1.m1.1c">\dagger</annotation></semantics></math></sup>, Wei Bi<sup id="id8.8.id5" class="ltx_sup">2</sup>, Shuming Shi<sup id="id9.9.id6" class="ltx_sup">2</sup> 
<br class="ltx_break"><sup id="id10.10.id7" class="ltx_sup">1</sup>School of Computer Science and Engineering, Sun Yat-sen University, China 
<br class="ltx_break"><sup id="id11.11.id8" class="ltx_sup">2</sup>Tencent AI Lab 
<br class="ltx_break"><span id="id12.12.id9" class="ltx_text ltx_font_typewriter">wanfq@mail2.sysu.edu.cn,quanxj3@mail.sysu.edu.cn</span> 
<br class="ltx_break"><span id="id13.13.id10" class="ltx_text ltx_font_typewriter">{timxthuang,jcykcai,victoriabi,shumingshi}@tencent.com</span> 
<br class="ltx_break">
</span><span class="ltx_author_notes">  Work was done during the internship at Tencent AI lab.  Corresponding authors.</span></span>
</div>

<div class="ltx_abstract">
<h6 class="ltx_title ltx_title_abstract">Abstract</h6>
<p id="id14.id1" class="ltx_p">While training large language models (LLMs) from scratch can generate models with distinct functionalities and strengths, it comes at significant costs and may result in redundant capabilities. Alternatively, a cost-effective and compelling approach is to merge existing pre-trained LLMs into a more potent model. However, due to the varying architectures of these LLMs, directly blending their weights is impractical. In this paper, we introduce the notion of knowledge fusion for LLMs, aimed at combining the capabilities of existing LLMs and transferring them into a single LLM. By leveraging the generative distributions of source LLMs, we externalize their collective knowledge and unique strengths, thereby potentially elevating the capabilities of the target model beyond those of any individual source LLM. We validate our approach using three popular LLMs with different architectures—Llama-2, MPT, and OpenLLaMA—across various benchmarks and tasks. Our findings confirm that the fusion of LLMs can improve the performance of the target model across a range of capabilities such as reasoning, commonsense, and code generation. Our code, model weights, and data are public at <a target="_blank" href="https://github.com/fanqiwan/FuseLLM" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/fanqiwan/FuseLLM</a>.</p>
</div>
<section id="S1" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">1 </span>Introduction</h2>

<div id="S1.p1" class="ltx_para ltx_noindent">
<p id="S1.p1.1" class="ltx_p">With the continuous success of large language models (LLMs) such as GPT <cite class="ltx_cite ltx_citemacro_citep">(Brown et al., <a href="#bib.bib5" title="" class="ltx_ref">2020</a>)</cite> and LLaMA <cite class="ltx_cite ltx_citemacro_citep">(Touvron et al., <a href="#bib.bib53" title="" class="ltx_ref">2023</a>)</cite> series across a wide range of natural language processing (NLP) tasks, it has become a strategic imperative for corporations to create their own LLMs. However, the costs associated with LLM development are astronomical. In addition to requiring vast amounts of training data, advanced techniques, substantial computational resources, and skilled labor, the development process also exerts significant pressure on energy consumption and the environment <cite class="ltx_cite ltx_citemacro_citep">(Rillig et al., <a href="#bib.bib44" title="" class="ltx_ref">2023</a>)</cite>. While these LLMs exhibit structural and functional differences, they share similar capabilities across a spectrum of NLP tasks. Consequently, beyond the traditional approach of training an LLM from scratch, an alternative option is to combine existing LLMs into a new, more powerful one, which is termed <em id="S1.p1.1.1" class="ltx_emph ltx_font_italic">knowledge fusion of LLMs</em> in this paper. If successful, this fusion not only cuts the cost of initial training but also allows the integrated model to benefit from the strengths of all the LLMs. This new model can also be fine-tuned and adapted for various downstream tasks. Moreover, the fusion can also happen among fine-tuned LLMs that specialize in a specific task.</p>
</div>
<div id="S1.p2" class="ltx_para ltx_noindent">
<p id="S1.p2.1" class="ltx_p">The endeavor to integrate the capabilities of multiple models has been a long-standing pursuit. For example, ensemble methods <cite class="ltx_cite ltx_citemacro_citep">(Littlestone &amp; Warmuth, <a href="#bib.bib35" title="" class="ltx_ref">1994</a>; Jiang et al., <a href="#bib.bib26" title="" class="ltx_ref">2023</a>)</cite> directly aggregate the outputs of different models to enhance prediction performance and robustness. However, this approach requires maintaining multiple trained models and executing each during inference, which is impractical for LLMs due to their substantial memory and inference time requirements. Likewise, this approach doesn’t facilitate fine-tuning, which is essential for many LLMs. Another approach is to directly merge several neural networks into a single network through parameter-wise arithmetic operations <cite class="ltx_cite ltx_citemacro_citep">(Wortsman et al., <a href="#bib.bib61" title="" class="ltx_ref">2022</a>; Jin et al., <a href="#bib.bib28" title="" class="ltx_ref">2022</a>)</cite>. This approach typically assumes uniform network architectures and attempts to establish mappings between the weights of distinct neural networks, which is often unattainable in the context of LLMs. Moreover, weight merging may lead to suboptimal results when substantial differences exist in the parameter space <cite class="ltx_cite ltx_citemacro_citep">(Li et al., <a href="#bib.bib33" title="" class="ltx_ref">2022</a>)</cite>.</p>
</div>
<div id="S1.p3" class="ltx_para ltx_noindent">
<p id="S1.p3.1" class="ltx_p">In this paper, we explore the fusion of LLMs from a probabilistic distribution perspective. For an input text, we argue that the probabilistic distributions generated by different source LLMs can reflect their inherent knowledge in understanding this text.
Therefore, the proposed <span id="S1.p3.1.1" class="ltx_text ltx_font_smallcaps">FuseLLM</span> leverages the generative distributions of source LLMs to externalize both their collective knowledge and individual strengths and transfer them to the target LLM through lightweight continual training. To achieve this, we develop a new strategy for aligning tokenizations originating from different LLMs and explore two methods for fusing the probability distributions generated by these diverse LLMs. During the continual training, <span id="S1.p3.1.2" class="ltx_text ltx_font_smallcaps">FuseLLM</span> places significant emphasis on minimizing the divergence between the target LLM’s probabilistic distributions and those of the source LLMs.</p>
</div>
<div id="S1.p4" class="ltx_para ltx_noindent">
<p id="S1.p4.1" class="ltx_p">To empirically demonstrate the effectiveness of <span id="S1.p4.1.1" class="ltx_text ltx_font_smallcaps">FuseLLM</span>, we examine a challenging yet general scenario of LLMs fusion, where the source models share minimal commonalities. Specifically, we focus on three popular open-source LLMs that possess distinct architectures and functionalities: Llama-2 <cite class="ltx_cite ltx_citemacro_citep">(Touvron et al., <a href="#bib.bib53" title="" class="ltx_ref">2023</a>)</cite>, OpenLLaMA <cite class="ltx_cite ltx_citemacro_citep">(Geng &amp; Liu, <a href="#bib.bib20" title="" class="ltx_ref">2023</a>)</cite>, and MPT <cite class="ltx_cite ltx_citemacro_citep">(Team, <a href="#bib.bib52" title="" class="ltx_ref">2023</a>)</cite>.
Evaluations across three benchmarks, which consist of a total of 42 tasks spanning reasoning, commonsense, and code generation, confirm that the target model trained by our method outperforms each source LLM and the baseline in most tasks. Moreover, we simulate the existence of functionally distinct LLMs with identical architecture by continually training a single base model on several domain-specific corpora. When evaluated based on perplexity, our method demonstrates superior potential in combining the capabilities of these structurally identical LLMs compared to traditional ensemble and weight merging methods.</p>
</div>
<div id="S1.p5" class="ltx_para ltx_noindent">
<p id="S1.p5.1" class="ltx_p">To sum up, this paper explores a novel challenge called LLMs fusion, with the goal of creating a unified model that effectively utilizes the collective capabilities and unique strengths of diverse LLMs. Illustrated in Figure <a href="#S1.F1" title="Figure 1 ‣ 1 Introduction ‣ Knowledge Fusion of Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>, our proposed approach distinguishes itself from traditional ensemble and weight merging techniques by prioritizing the fusion of multiple LLMs through knowledge externalization and transfer. This study yields several findings that may spark future research. Firstly, while we demonstrate the effectiveness of our method through lightweight continual training on a compact, high-quality corpus, the thoughtful selection of the training corpus can be a crucial consideration, particularly with regard to its relevance to downstream tasks. Secondly, in scenarios where the capabilities of source LLMs vary significantly, the fusion function appears to be crucial in effectively combining their respective strengths. Lastly, when compared to traditional model ensemble and merging techniques, the field of LLMs fusion appears to be a more promising avenue for exploration, especially in light of the diverse structures and substantial model sizes of LLMs.</p>
</div>
<figure id="S1.F1" class="ltx_figure"><img src="/html/2401.10491/assets/x1.png" id="S1.F1.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="401" height="117" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 1: </span>Illustration of conventional model fusion techniques (ensemble and weight merging) and our knowledge fusion approach for LLMs (<span id="S1.F1.3.1" class="ltx_text ltx_font_smallcaps">FuseLLM</span>). Different animal icons represent different LLMs, with various species denoting LLMs possessing differing architectures. <span id="S1.F1.4.2" class="ltx_text ltx_font_smallcaps">FuseLLM</span> externalizes the knowledge from multiple LLMs and transfers their capabilities to a target LLM.</figcaption>
</figure>
</section>
<section id="S2" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">2 </span>Related Work</h2>

<section id="S2.SS0.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Model Fusion</h4>

<div id="S2.SS0.SSS0.Px1.p1" class="ltx_para ltx_noindent">
<p id="S2.SS0.SSS0.Px1.p1.1" class="ltx_p">The integration of capabilities from diverse models has been a long-standing objective, with existing approaches mainly falling into two categories. Firstly, the traditional technique of model <em id="S2.SS0.SSS0.Px1.p1.1.1" class="ltx_emph ltx_font_italic">ensemble</em> combines the outputs of multiple models to enhance overall system performance <cite class="ltx_cite ltx_citemacro_citep">(Littlestone &amp; Warmuth, <a href="#bib.bib35" title="" class="ltx_ref">1994</a>; Sagi &amp; Rokach, <a href="#bib.bib45" title="" class="ltx_ref">2018</a>)</cite>. Note that this technique doesn’t involve the explicit merging of multiple models into a new one. Common methods for model ensemble typically employ weighted averaging <cite class="ltx_cite ltx_citemacro_citep">(Littlestone &amp; Warmuth, <a href="#bib.bib35" title="" class="ltx_ref">1994</a>)</cite> or majority voting <cite class="ltx_cite ltx_citemacro_citep">(Monteith et al., <a href="#bib.bib38" title="" class="ltx_ref">2011</a>)</cite> to consolidate predictions from various models. Recently, <cite class="ltx_cite ltx_citemacro_cite">Jiang et al. (<a href="#bib.bib26" title="" class="ltx_ref">2023</a>)</cite> introduced an ensemble framework designed to leverage the diverse strengths of multiple open-source LLMs. This framework first employs a pairwise comparison method to detect subtle distinctions among candidate outputs. Then, it combines the top-ranked candidates to produce an enhanced output, capitalizing on their strengths while mitigating their weaknesses.</p>
</div>
<div id="S2.SS0.SSS0.Px1.p2" class="ltx_para ltx_noindent">
<p id="S2.SS0.SSS0.Px1.p2.1" class="ltx_p">Secondly, <em id="S2.SS0.SSS0.Px1.p2.1.1" class="ltx_emph ltx_font_italic">weight merging</em> presents another approach that facilitates model fusion at the parameter level. <cite class="ltx_cite ltx_citemacro_cite">Gupta et al. (<a href="#bib.bib22" title="" class="ltx_ref">2020</a>)</cite> and <cite class="ltx_cite ltx_citemacro_cite">Wortsman et al. (<a href="#bib.bib61" title="" class="ltx_ref">2022</a>)</cite> merged weights from models with identical structures, obtained through different strategies or configurations, to achieve improved overall performance. Similarly, <cite class="ltx_cite ltx_citemacro_cite">Cha et al. (<a href="#bib.bib8" title="" class="ltx_ref">2021</a>)</cite>, <cite class="ltx_cite ltx_citemacro_cite">Rame et al. (<a href="#bib.bib43" title="" class="ltx_ref">2022</a>)</cite>, and <cite class="ltx_cite ltx_citemacro_cite">Arpit et al. (<a href="#bib.bib2" title="" class="ltx_ref">2022</a>)</cite> explored weighted averaging of models derived from different configurations to enhance out-of-distribution generalization. Furthermore, <cite class="ltx_cite ltx_citemacro_cite">Jin et al. (<a href="#bib.bib28" title="" class="ltx_ref">2022</a>)</cite> merged models designed for specific domains or tasks to create a generalist capable of addressing all domains or tasks. Going beyond parameter merging of entire models, <cite class="ltx_cite ltx_citemacro_cite">Wang et al. (<a href="#bib.bib58" title="" class="ltx_ref">2022b</a>)</cite>, <cite class="ltx_cite ltx_citemacro_cite">Huang et al. (<a href="#bib.bib25" title="" class="ltx_ref">2023</a>)</cite>, and <cite class="ltx_cite ltx_citemacro_cite">Zhang et al. (<a href="#bib.bib64" title="" class="ltx_ref">2023</a>)</cite> applied linear mathematical operations to adapter parameters to achieve superior generalization performance.</p>
</div>
<div id="S2.SS0.SSS0.Px1.p3" class="ltx_para ltx_noindent">
<p id="S2.SS0.SSS0.Px1.p3.1" class="ltx_p">In a nutshell, while model ensemble requires the parallel deployment of multiple models, weight merging is generally limited to models with identical architectures. In contrast, the approach proposed in this paper supports the fusion of multiple LLMs with diverse architectures by explicitly transferring their knowledge and capabilities to a target LLM.</p>
</div>
</section>
<section id="S2.SS0.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Knowledge Distillation</h4>

<div id="S2.SS0.SSS0.Px2.p1" class="ltx_para ltx_noindent">
<p id="S2.SS0.SSS0.Px2.p1.1" class="ltx_p">Knowledge distillation <cite class="ltx_cite ltx_citemacro_citep">(Hinton et al., <a href="#bib.bib23" title="" class="ltx_ref">2015</a>)</cite>, initially proposed for model compression, involves training a student model under the guidance of one or more teacher models. In the NLP community, knowledge distillation has been widely applied to text classification tasks. These applications include training the student model to replicate the teacher’s output distribution <cite class="ltx_cite ltx_citemacro_citep">(Sanh et al., <a href="#bib.bib46" title="" class="ltx_ref">2019</a>; Turc et al., <a href="#bib.bib54" title="" class="ltx_ref">2019</a>)</cite>, as well as features <cite class="ltx_cite ltx_citemacro_citep">(Sun et al., <a href="#bib.bib49" title="" class="ltx_ref">2019</a>; Jiao et al., <a href="#bib.bib27" title="" class="ltx_ref">2020</a>)</cite> and relations <cite class="ltx_cite ltx_citemacro_citep">(Wang et al., <a href="#bib.bib56" title="" class="ltx_ref">2020</a>)</cite> derived from intermediate layers of the teacher model. In the realm of text generation, the conventional approach focuses on minimizing the KL divergence between the student and teacher generation distributions.
This is achieved by using the teacher’s probability distributions at each time step as supervision <cite class="ltx_cite ltx_citemacro_citep">(Khanuja et al., <a href="#bib.bib31" title="" class="ltx_ref">2021</a>; Gu et al., <a href="#bib.bib21" title="" class="ltx_ref">2023</a>; Agarwal et al., <a href="#bib.bib1" title="" class="ltx_ref">2023</a>)</cite> or by directly training on the teacher’s generated texts <cite class="ltx_cite ltx_citemacro_citep">(Peng et al., <a href="#bib.bib41" title="" class="ltx_ref">2023</a>; Xu et al., <a href="#bib.bib62" title="" class="ltx_ref">2023</a>)</cite>.</p>
</div>
<div id="S2.SS0.SSS0.Px2.p2" class="ltx_para ltx_noindent">
<p id="S2.SS0.SSS0.Px2.p2.1" class="ltx_p">While our method shares a framework similar to multi-teacher knowledge distillation, there are two significant distinctions. First, in traditional knowledge distillation, the student models are typically constrained to be smaller in size than the teachers. In our scenario, however, there are no limitations on the size of the target model. Second, traditional knowledge distillation often results in the student models lagging behind the teachers in performance after distillation. In contrast, we anticipate that after the fusion, the target model will surpass any of the source models in performance.</p>
</div>
</section>
</section>
<section id="S3" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">3 </span>Knowledge Fusion of LLMs</h2>

<div id="S3.p1" class="ltx_para ltx_noindent">
<p id="S3.p1.4" class="ltx_p">The primary objective of LLMs fusion is to externalize the collective knowledge embedded within multiple source LLMs and integrate their capabilities into a target LLM. Given <math id="S3.p1.1.m1.1" class="ltx_Math" alttext="K" display="inline"><semantics id="S3.p1.1.m1.1a"><mi id="S3.p1.1.m1.1.1" xref="S3.p1.1.m1.1.1.cmml">K</mi><annotation-xml encoding="MathML-Content" id="S3.p1.1.m1.1b"><ci id="S3.p1.1.m1.1.1.cmml" xref="S3.p1.1.m1.1.1">𝐾</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.p1.1.m1.1c">K</annotation></semantics></math> source LLMs <math id="S3.p1.2.m2.1" class="ltx_Math" alttext="\{\mathcal{M}^{s}_{j}\}_{j=1}^{K}" display="inline"><semantics id="S3.p1.2.m2.1a"><msubsup id="S3.p1.2.m2.1.1" xref="S3.p1.2.m2.1.1.cmml"><mrow id="S3.p1.2.m2.1.1.1.1.1" xref="S3.p1.2.m2.1.1.1.1.2.cmml"><mo stretchy="false" id="S3.p1.2.m2.1.1.1.1.1.2" xref="S3.p1.2.m2.1.1.1.1.2.cmml">{</mo><msubsup id="S3.p1.2.m2.1.1.1.1.1.1" xref="S3.p1.2.m2.1.1.1.1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.p1.2.m2.1.1.1.1.1.1.2.2" xref="S3.p1.2.m2.1.1.1.1.1.1.2.2.cmml">ℳ</mi><mi id="S3.p1.2.m2.1.1.1.1.1.1.3" xref="S3.p1.2.m2.1.1.1.1.1.1.3.cmml">j</mi><mi id="S3.p1.2.m2.1.1.1.1.1.1.2.3" xref="S3.p1.2.m2.1.1.1.1.1.1.2.3.cmml">s</mi></msubsup><mo stretchy="false" id="S3.p1.2.m2.1.1.1.1.1.3" xref="S3.p1.2.m2.1.1.1.1.2.cmml">}</mo></mrow><mrow id="S3.p1.2.m2.1.1.1.3" xref="S3.p1.2.m2.1.1.1.3.cmml"><mi id="S3.p1.2.m2.1.1.1.3.2" xref="S3.p1.2.m2.1.1.1.3.2.cmml">j</mi><mo id="S3.p1.2.m2.1.1.1.3.1" xref="S3.p1.2.m2.1.1.1.3.1.cmml">=</mo><mn id="S3.p1.2.m2.1.1.1.3.3" xref="S3.p1.2.m2.1.1.1.3.3.cmml">1</mn></mrow><mi id="S3.p1.2.m2.1.1.3" xref="S3.p1.2.m2.1.1.3.cmml">K</mi></msubsup><annotation-xml encoding="MathML-Content" id="S3.p1.2.m2.1b"><apply id="S3.p1.2.m2.1.1.cmml" xref="S3.p1.2.m2.1.1"><csymbol cd="ambiguous" id="S3.p1.2.m2.1.1.2.cmml" xref="S3.p1.2.m2.1.1">superscript</csymbol><apply id="S3.p1.2.m2.1.1.1.cmml" xref="S3.p1.2.m2.1.1"><csymbol cd="ambiguous" id="S3.p1.2.m2.1.1.1.2.cmml" xref="S3.p1.2.m2.1.1">subscript</csymbol><set id="S3.p1.2.m2.1.1.1.1.2.cmml" xref="S3.p1.2.m2.1.1.1.1.1"><apply id="S3.p1.2.m2.1.1.1.1.1.1.cmml" xref="S3.p1.2.m2.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.p1.2.m2.1.1.1.1.1.1.1.cmml" xref="S3.p1.2.m2.1.1.1.1.1.1">subscript</csymbol><apply id="S3.p1.2.m2.1.1.1.1.1.1.2.cmml" xref="S3.p1.2.m2.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.p1.2.m2.1.1.1.1.1.1.2.1.cmml" xref="S3.p1.2.m2.1.1.1.1.1.1">superscript</csymbol><ci id="S3.p1.2.m2.1.1.1.1.1.1.2.2.cmml" xref="S3.p1.2.m2.1.1.1.1.1.1.2.2">ℳ</ci><ci id="S3.p1.2.m2.1.1.1.1.1.1.2.3.cmml" xref="S3.p1.2.m2.1.1.1.1.1.1.2.3">𝑠</ci></apply><ci id="S3.p1.2.m2.1.1.1.1.1.1.3.cmml" xref="S3.p1.2.m2.1.1.1.1.1.1.3">𝑗</ci></apply></set><apply id="S3.p1.2.m2.1.1.1.3.cmml" xref="S3.p1.2.m2.1.1.1.3"><eq id="S3.p1.2.m2.1.1.1.3.1.cmml" xref="S3.p1.2.m2.1.1.1.3.1"></eq><ci id="S3.p1.2.m2.1.1.1.3.2.cmml" xref="S3.p1.2.m2.1.1.1.3.2">𝑗</ci><cn type="integer" id="S3.p1.2.m2.1.1.1.3.3.cmml" xref="S3.p1.2.m2.1.1.1.3.3">1</cn></apply></apply><ci id="S3.p1.2.m2.1.1.3.cmml" xref="S3.p1.2.m2.1.1.3">𝐾</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.p1.2.m2.1c">\{\mathcal{M}^{s}_{j}\}_{j=1}^{K}</annotation></semantics></math> with varying architectures, each having undergone individual pre-training or fine-tuning on distinct datasets, the key idea behind our approach is to initially stimulate LLMs to manifest their inherent knowledge by challenging them to predict the next token. The probabilistic distributions of these predictions are thoroughly assessed, and the most accurate predictions are utilized to continually train the target LLM <math id="S3.p1.3.m3.1" class="ltx_Math" alttext="\mathcal{M}^{t}" display="inline"><semantics id="S3.p1.3.m3.1a"><msup id="S3.p1.3.m3.1.1" xref="S3.p1.3.m3.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.p1.3.m3.1.1.2" xref="S3.p1.3.m3.1.1.2.cmml">ℳ</mi><mi id="S3.p1.3.m3.1.1.3" xref="S3.p1.3.m3.1.1.3.cmml">t</mi></msup><annotation-xml encoding="MathML-Content" id="S3.p1.3.m3.1b"><apply id="S3.p1.3.m3.1.1.cmml" xref="S3.p1.3.m3.1.1"><csymbol cd="ambiguous" id="S3.p1.3.m3.1.1.1.cmml" xref="S3.p1.3.m3.1.1">superscript</csymbol><ci id="S3.p1.3.m3.1.1.2.cmml" xref="S3.p1.3.m3.1.1.2">ℳ</ci><ci id="S3.p1.3.m3.1.1.3.cmml" xref="S3.p1.3.m3.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.p1.3.m3.1c">\mathcal{M}^{t}</annotation></semantics></math> on a corpus <math id="S3.p1.4.m4.1" class="ltx_Math" alttext="\mathcal{C}" display="inline"><semantics id="S3.p1.4.m4.1a"><mi class="ltx_font_mathcaligraphic" id="S3.p1.4.m4.1.1" xref="S3.p1.4.m4.1.1.cmml">𝒞</mi><annotation-xml encoding="MathML-Content" id="S3.p1.4.m4.1b"><ci id="S3.p1.4.m4.1.1.cmml" xref="S3.p1.4.m4.1.1">𝒞</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.p1.4.m4.1c">\mathcal{C}</annotation></semantics></math> using the causal language modeling objective. In the following sections, we start with a brief introduction to the preliminaries, followed by a detailed explanation of our LLMs fusion framework. Finally, we delve into the implementation details.</p>
</div>
<section id="S3.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.1 </span>Preliminaries</h3>

<div id="S3.SS1.p1" class="ltx_para ltx_noindent">
<p id="S3.SS1.p1.6" class="ltx_p">Let <math id="S3.SS1.p1.1.m1.1" class="ltx_Math" alttext="t" display="inline"><semantics id="S3.SS1.p1.1.m1.1a"><mi id="S3.SS1.p1.1.m1.1.1" xref="S3.SS1.p1.1.m1.1.1.cmml">t</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.1.m1.1b"><ci id="S3.SS1.p1.1.m1.1.1.cmml" xref="S3.SS1.p1.1.m1.1.1">𝑡</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.1.m1.1c">t</annotation></semantics></math> denote a text sequence of length <math id="S3.SS1.p1.2.m2.1" class="ltx_Math" alttext="N" display="inline"><semantics id="S3.SS1.p1.2.m2.1a"><mi id="S3.SS1.p1.2.m2.1.1" xref="S3.SS1.p1.2.m2.1.1.cmml">N</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.2.m2.1b"><ci id="S3.SS1.p1.2.m2.1.1.cmml" xref="S3.SS1.p1.2.m2.1.1">𝑁</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.2.m2.1c">N</annotation></semantics></math> sampled from the corpus <math id="S3.SS1.p1.3.m3.1" class="ltx_Math" alttext="\mathcal{C}" display="inline"><semantics id="S3.SS1.p1.3.m3.1a"><mi class="ltx_font_mathcaligraphic" id="S3.SS1.p1.3.m3.1.1" xref="S3.SS1.p1.3.m3.1.1.cmml">𝒞</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.3.m3.1b"><ci id="S3.SS1.p1.3.m3.1.1.cmml" xref="S3.SS1.p1.3.m3.1.1">𝒞</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.3.m3.1c">\mathcal{C}</annotation></semantics></math> and <math id="S3.SS1.p1.4.m4.4" class="ltx_Math" alttext="t_{&lt;i}=(t_{1},t_{2},\ldots,t_{i-1})" display="inline"><semantics id="S3.SS1.p1.4.m4.4a"><mrow id="S3.SS1.p1.4.m4.4.4" xref="S3.SS1.p1.4.m4.4.4.cmml"><msub id="S3.SS1.p1.4.m4.4.4.5" xref="S3.SS1.p1.4.m4.4.4.5.cmml"><mi id="S3.SS1.p1.4.m4.4.4.5.2" xref="S3.SS1.p1.4.m4.4.4.5.2.cmml">t</mi><mrow id="S3.SS1.p1.4.m4.4.4.5.3" xref="S3.SS1.p1.4.m4.4.4.5.3.cmml"><mi id="S3.SS1.p1.4.m4.4.4.5.3.2" xref="S3.SS1.p1.4.m4.4.4.5.3.2.cmml"></mi><mo id="S3.SS1.p1.4.m4.4.4.5.3.1" xref="S3.SS1.p1.4.m4.4.4.5.3.1.cmml">&lt;</mo><mi id="S3.SS1.p1.4.m4.4.4.5.3.3" xref="S3.SS1.p1.4.m4.4.4.5.3.3.cmml">i</mi></mrow></msub><mo id="S3.SS1.p1.4.m4.4.4.4" xref="S3.SS1.p1.4.m4.4.4.4.cmml">=</mo><mrow id="S3.SS1.p1.4.m4.4.4.3.3" xref="S3.SS1.p1.4.m4.4.4.3.4.cmml"><mo stretchy="false" id="S3.SS1.p1.4.m4.4.4.3.3.4" xref="S3.SS1.p1.4.m4.4.4.3.4.cmml">(</mo><msub id="S3.SS1.p1.4.m4.2.2.1.1.1" xref="S3.SS1.p1.4.m4.2.2.1.1.1.cmml"><mi id="S3.SS1.p1.4.m4.2.2.1.1.1.2" xref="S3.SS1.p1.4.m4.2.2.1.1.1.2.cmml">t</mi><mn id="S3.SS1.p1.4.m4.2.2.1.1.1.3" xref="S3.SS1.p1.4.m4.2.2.1.1.1.3.cmml">1</mn></msub><mo id="S3.SS1.p1.4.m4.4.4.3.3.5" xref="S3.SS1.p1.4.m4.4.4.3.4.cmml">,</mo><msub id="S3.SS1.p1.4.m4.3.3.2.2.2" xref="S3.SS1.p1.4.m4.3.3.2.2.2.cmml"><mi id="S3.SS1.p1.4.m4.3.3.2.2.2.2" xref="S3.SS1.p1.4.m4.3.3.2.2.2.2.cmml">t</mi><mn id="S3.SS1.p1.4.m4.3.3.2.2.2.3" xref="S3.SS1.p1.4.m4.3.3.2.2.2.3.cmml">2</mn></msub><mo id="S3.SS1.p1.4.m4.4.4.3.3.6" xref="S3.SS1.p1.4.m4.4.4.3.4.cmml">,</mo><mi mathvariant="normal" id="S3.SS1.p1.4.m4.1.1" xref="S3.SS1.p1.4.m4.1.1.cmml">…</mi><mo id="S3.SS1.p1.4.m4.4.4.3.3.7" xref="S3.SS1.p1.4.m4.4.4.3.4.cmml">,</mo><msub id="S3.SS1.p1.4.m4.4.4.3.3.3" xref="S3.SS1.p1.4.m4.4.4.3.3.3.cmml"><mi id="S3.SS1.p1.4.m4.4.4.3.3.3.2" xref="S3.SS1.p1.4.m4.4.4.3.3.3.2.cmml">t</mi><mrow id="S3.SS1.p1.4.m4.4.4.3.3.3.3" xref="S3.SS1.p1.4.m4.4.4.3.3.3.3.cmml"><mi id="S3.SS1.p1.4.m4.4.4.3.3.3.3.2" xref="S3.SS1.p1.4.m4.4.4.3.3.3.3.2.cmml">i</mi><mo id="S3.SS1.p1.4.m4.4.4.3.3.3.3.1" xref="S3.SS1.p1.4.m4.4.4.3.3.3.3.1.cmml">−</mo><mn id="S3.SS1.p1.4.m4.4.4.3.3.3.3.3" xref="S3.SS1.p1.4.m4.4.4.3.3.3.3.3.cmml">1</mn></mrow></msub><mo stretchy="false" id="S3.SS1.p1.4.m4.4.4.3.3.8" xref="S3.SS1.p1.4.m4.4.4.3.4.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.4.m4.4b"><apply id="S3.SS1.p1.4.m4.4.4.cmml" xref="S3.SS1.p1.4.m4.4.4"><eq id="S3.SS1.p1.4.m4.4.4.4.cmml" xref="S3.SS1.p1.4.m4.4.4.4"></eq><apply id="S3.SS1.p1.4.m4.4.4.5.cmml" xref="S3.SS1.p1.4.m4.4.4.5"><csymbol cd="ambiguous" id="S3.SS1.p1.4.m4.4.4.5.1.cmml" xref="S3.SS1.p1.4.m4.4.4.5">subscript</csymbol><ci id="S3.SS1.p1.4.m4.4.4.5.2.cmml" xref="S3.SS1.p1.4.m4.4.4.5.2">𝑡</ci><apply id="S3.SS1.p1.4.m4.4.4.5.3.cmml" xref="S3.SS1.p1.4.m4.4.4.5.3"><lt id="S3.SS1.p1.4.m4.4.4.5.3.1.cmml" xref="S3.SS1.p1.4.m4.4.4.5.3.1"></lt><csymbol cd="latexml" id="S3.SS1.p1.4.m4.4.4.5.3.2.cmml" xref="S3.SS1.p1.4.m4.4.4.5.3.2">absent</csymbol><ci id="S3.SS1.p1.4.m4.4.4.5.3.3.cmml" xref="S3.SS1.p1.4.m4.4.4.5.3.3">𝑖</ci></apply></apply><vector id="S3.SS1.p1.4.m4.4.4.3.4.cmml" xref="S3.SS1.p1.4.m4.4.4.3.3"><apply id="S3.SS1.p1.4.m4.2.2.1.1.1.cmml" xref="S3.SS1.p1.4.m4.2.2.1.1.1"><csymbol cd="ambiguous" id="S3.SS1.p1.4.m4.2.2.1.1.1.1.cmml" xref="S3.SS1.p1.4.m4.2.2.1.1.1">subscript</csymbol><ci id="S3.SS1.p1.4.m4.2.2.1.1.1.2.cmml" xref="S3.SS1.p1.4.m4.2.2.1.1.1.2">𝑡</ci><cn type="integer" id="S3.SS1.p1.4.m4.2.2.1.1.1.3.cmml" xref="S3.SS1.p1.4.m4.2.2.1.1.1.3">1</cn></apply><apply id="S3.SS1.p1.4.m4.3.3.2.2.2.cmml" xref="S3.SS1.p1.4.m4.3.3.2.2.2"><csymbol cd="ambiguous" id="S3.SS1.p1.4.m4.3.3.2.2.2.1.cmml" xref="S3.SS1.p1.4.m4.3.3.2.2.2">subscript</csymbol><ci id="S3.SS1.p1.4.m4.3.3.2.2.2.2.cmml" xref="S3.SS1.p1.4.m4.3.3.2.2.2.2">𝑡</ci><cn type="integer" id="S3.SS1.p1.4.m4.3.3.2.2.2.3.cmml" xref="S3.SS1.p1.4.m4.3.3.2.2.2.3">2</cn></apply><ci id="S3.SS1.p1.4.m4.1.1.cmml" xref="S3.SS1.p1.4.m4.1.1">…</ci><apply id="S3.SS1.p1.4.m4.4.4.3.3.3.cmml" xref="S3.SS1.p1.4.m4.4.4.3.3.3"><csymbol cd="ambiguous" id="S3.SS1.p1.4.m4.4.4.3.3.3.1.cmml" xref="S3.SS1.p1.4.m4.4.4.3.3.3">subscript</csymbol><ci id="S3.SS1.p1.4.m4.4.4.3.3.3.2.cmml" xref="S3.SS1.p1.4.m4.4.4.3.3.3.2">𝑡</ci><apply id="S3.SS1.p1.4.m4.4.4.3.3.3.3.cmml" xref="S3.SS1.p1.4.m4.4.4.3.3.3.3"><minus id="S3.SS1.p1.4.m4.4.4.3.3.3.3.1.cmml" xref="S3.SS1.p1.4.m4.4.4.3.3.3.3.1"></minus><ci id="S3.SS1.p1.4.m4.4.4.3.3.3.3.2.cmml" xref="S3.SS1.p1.4.m4.4.4.3.3.3.3.2">𝑖</ci><cn type="integer" id="S3.SS1.p1.4.m4.4.4.3.3.3.3.3.cmml" xref="S3.SS1.p1.4.m4.4.4.3.3.3.3.3">1</cn></apply></apply></vector></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.4.m4.4c">t_{&lt;i}=(t_{1},t_{2},\ldots,t_{i-1})</annotation></semantics></math> denote the sequence preceding the <math id="S3.SS1.p1.5.m5.1" class="ltx_Math" alttext="i" display="inline"><semantics id="S3.SS1.p1.5.m5.1a"><mi id="S3.SS1.p1.5.m5.1.1" xref="S3.SS1.p1.5.m5.1.1.cmml">i</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.5.m5.1b"><ci id="S3.SS1.p1.5.m5.1.1.cmml" xref="S3.SS1.p1.5.m5.1.1">𝑖</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.5.m5.1c">i</annotation></semantics></math>th token.
The causal language modeling (CLM) objective for training a language model parameterized by <math id="S3.SS1.p1.6.m6.1" class="ltx_Math" alttext="\theta" display="inline"><semantics id="S3.SS1.p1.6.m6.1a"><mi id="S3.SS1.p1.6.m6.1.1" xref="S3.SS1.p1.6.m6.1.1.cmml">θ</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.6.m6.1b"><ci id="S3.SS1.p1.6.m6.1.1.cmml" xref="S3.SS1.p1.6.m6.1.1">𝜃</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.6.m6.1c">\theta</annotation></semantics></math> is defined as minimizing the negative log-likelihood:</p>
<table id="S3.E1" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S3.E1.m1.1" class="ltx_Math" alttext="\mathcal{L}_{\text{CLM}}=-\mathbb{E}_{t\sim\mathcal{C}}\left[\sum_{i}\log p_{\theta}(t_{i}|t_{&lt;i})\right]," display="block"><semantics id="S3.E1.m1.1a"><mrow id="S3.E1.m1.1.1.1" xref="S3.E1.m1.1.1.1.1.cmml"><mrow id="S3.E1.m1.1.1.1.1" xref="S3.E1.m1.1.1.1.1.cmml"><msub id="S3.E1.m1.1.1.1.1.3" xref="S3.E1.m1.1.1.1.1.3.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.E1.m1.1.1.1.1.3.2" xref="S3.E1.m1.1.1.1.1.3.2.cmml">ℒ</mi><mtext id="S3.E1.m1.1.1.1.1.3.3" xref="S3.E1.m1.1.1.1.1.3.3a.cmml">CLM</mtext></msub><mo id="S3.E1.m1.1.1.1.1.2" xref="S3.E1.m1.1.1.1.1.2.cmml">=</mo><mrow id="S3.E1.m1.1.1.1.1.1" xref="S3.E1.m1.1.1.1.1.1.cmml"><mo id="S3.E1.m1.1.1.1.1.1a" xref="S3.E1.m1.1.1.1.1.1.cmml">−</mo><mrow id="S3.E1.m1.1.1.1.1.1.1" xref="S3.E1.m1.1.1.1.1.1.1.cmml"><msub id="S3.E1.m1.1.1.1.1.1.1.3" xref="S3.E1.m1.1.1.1.1.1.1.3.cmml"><mi id="S3.E1.m1.1.1.1.1.1.1.3.2" xref="S3.E1.m1.1.1.1.1.1.1.3.2.cmml">𝔼</mi><mrow id="S3.E1.m1.1.1.1.1.1.1.3.3" xref="S3.E1.m1.1.1.1.1.1.1.3.3.cmml"><mi id="S3.E1.m1.1.1.1.1.1.1.3.3.2" xref="S3.E1.m1.1.1.1.1.1.1.3.3.2.cmml">t</mi><mo id="S3.E1.m1.1.1.1.1.1.1.3.3.1" xref="S3.E1.m1.1.1.1.1.1.1.3.3.1.cmml">∼</mo><mi class="ltx_font_mathcaligraphic" id="S3.E1.m1.1.1.1.1.1.1.3.3.3" xref="S3.E1.m1.1.1.1.1.1.1.3.3.3.cmml">𝒞</mi></mrow></msub><mo lspace="0em" rspace="0em" id="S3.E1.m1.1.1.1.1.1.1.2" xref="S3.E1.m1.1.1.1.1.1.1.2.cmml">​</mo><mrow id="S3.E1.m1.1.1.1.1.1.1.1.1" xref="S3.E1.m1.1.1.1.1.1.1.1.2.cmml"><mo id="S3.E1.m1.1.1.1.1.1.1.1.1.2" xref="S3.E1.m1.1.1.1.1.1.1.1.2.1.cmml">[</mo><mrow id="S3.E1.m1.1.1.1.1.1.1.1.1.1" xref="S3.E1.m1.1.1.1.1.1.1.1.1.1.cmml"><munder id="S3.E1.m1.1.1.1.1.1.1.1.1.1.2" xref="S3.E1.m1.1.1.1.1.1.1.1.1.1.2.cmml"><mo lspace="0em" movablelimits="false" id="S3.E1.m1.1.1.1.1.1.1.1.1.1.2.2" xref="S3.E1.m1.1.1.1.1.1.1.1.1.1.2.2.cmml">∑</mo><mi id="S3.E1.m1.1.1.1.1.1.1.1.1.1.2.3" xref="S3.E1.m1.1.1.1.1.1.1.1.1.1.2.3.cmml">i</mi></munder><mrow id="S3.E1.m1.1.1.1.1.1.1.1.1.1.1" xref="S3.E1.m1.1.1.1.1.1.1.1.1.1.1.cmml"><mrow id="S3.E1.m1.1.1.1.1.1.1.1.1.1.1.3" xref="S3.E1.m1.1.1.1.1.1.1.1.1.1.1.3.cmml"><mi id="S3.E1.m1.1.1.1.1.1.1.1.1.1.1.3.1" xref="S3.E1.m1.1.1.1.1.1.1.1.1.1.1.3.1.cmml">log</mi><mo lspace="0.167em" id="S3.E1.m1.1.1.1.1.1.1.1.1.1.1.3a" xref="S3.E1.m1.1.1.1.1.1.1.1.1.1.1.3.cmml">⁡</mo><msub id="S3.E1.m1.1.1.1.1.1.1.1.1.1.1.3.2" xref="S3.E1.m1.1.1.1.1.1.1.1.1.1.1.3.2.cmml"><mi id="S3.E1.m1.1.1.1.1.1.1.1.1.1.1.3.2.2" xref="S3.E1.m1.1.1.1.1.1.1.1.1.1.1.3.2.2.cmml">p</mi><mi id="S3.E1.m1.1.1.1.1.1.1.1.1.1.1.3.2.3" xref="S3.E1.m1.1.1.1.1.1.1.1.1.1.1.3.2.3.cmml">θ</mi></msub></mrow><mo lspace="0em" rspace="0em" id="S3.E1.m1.1.1.1.1.1.1.1.1.1.1.2" xref="S3.E1.m1.1.1.1.1.1.1.1.1.1.1.2.cmml">​</mo><mrow id="S3.E1.m1.1.1.1.1.1.1.1.1.1.1.1.1" xref="S3.E1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.E1.m1.1.1.1.1.1.1.1.1.1.1.1.1.2" xref="S3.E1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml">(</mo><mrow id="S3.E1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1" xref="S3.E1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml"><msub id="S3.E1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.2" xref="S3.E1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.cmml"><mi id="S3.E1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.2" xref="S3.E1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.2.cmml">t</mi><mi id="S3.E1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.3" xref="S3.E1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.3.cmml">i</mi></msub><mo fence="false" id="S3.E1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1" xref="S3.E1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml">|</mo><msub id="S3.E1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3" xref="S3.E1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.cmml"><mi id="S3.E1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.2" xref="S3.E1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.2.cmml">t</mi><mrow id="S3.E1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.3" xref="S3.E1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.3.cmml"><mi id="S3.E1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.3.2" xref="S3.E1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.3.2.cmml"></mi><mo id="S3.E1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.3.1" xref="S3.E1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.3.1.cmml">&lt;</mo><mi id="S3.E1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.3.3" xref="S3.E1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.3.3.cmml">i</mi></mrow></msub></mrow><mo stretchy="false" id="S3.E1.m1.1.1.1.1.1.1.1.1.1.1.1.1.3" xref="S3.E1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><mo id="S3.E1.m1.1.1.1.1.1.1.1.1.3" xref="S3.E1.m1.1.1.1.1.1.1.1.2.1.cmml">]</mo></mrow></mrow></mrow></mrow><mo id="S3.E1.m1.1.1.1.2" xref="S3.E1.m1.1.1.1.1.cmml">,</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.E1.m1.1b"><apply id="S3.E1.m1.1.1.1.1.cmml" xref="S3.E1.m1.1.1.1"><eq id="S3.E1.m1.1.1.1.1.2.cmml" xref="S3.E1.m1.1.1.1.1.2"></eq><apply id="S3.E1.m1.1.1.1.1.3.cmml" xref="S3.E1.m1.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.E1.m1.1.1.1.1.3.1.cmml" xref="S3.E1.m1.1.1.1.1.3">subscript</csymbol><ci id="S3.E1.m1.1.1.1.1.3.2.cmml" xref="S3.E1.m1.1.1.1.1.3.2">ℒ</ci><ci id="S3.E1.m1.1.1.1.1.3.3a.cmml" xref="S3.E1.m1.1.1.1.1.3.3"><mtext mathsize="70%" id="S3.E1.m1.1.1.1.1.3.3.cmml" xref="S3.E1.m1.1.1.1.1.3.3">CLM</mtext></ci></apply><apply id="S3.E1.m1.1.1.1.1.1.cmml" xref="S3.E1.m1.1.1.1.1.1"><minus id="S3.E1.m1.1.1.1.1.1.2.cmml" xref="S3.E1.m1.1.1.1.1.1"></minus><apply id="S3.E1.m1.1.1.1.1.1.1.cmml" xref="S3.E1.m1.1.1.1.1.1.1"><times id="S3.E1.m1.1.1.1.1.1.1.2.cmml" xref="S3.E1.m1.1.1.1.1.1.1.2"></times><apply id="S3.E1.m1.1.1.1.1.1.1.3.cmml" xref="S3.E1.m1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.E1.m1.1.1.1.1.1.1.3.1.cmml" xref="S3.E1.m1.1.1.1.1.1.1.3">subscript</csymbol><ci id="S3.E1.m1.1.1.1.1.1.1.3.2.cmml" xref="S3.E1.m1.1.1.1.1.1.1.3.2">𝔼</ci><apply id="S3.E1.m1.1.1.1.1.1.1.3.3.cmml" xref="S3.E1.m1.1.1.1.1.1.1.3.3"><csymbol cd="latexml" id="S3.E1.m1.1.1.1.1.1.1.3.3.1.cmml" xref="S3.E1.m1.1.1.1.1.1.1.3.3.1">similar-to</csymbol><ci id="S3.E1.m1.1.1.1.1.1.1.3.3.2.cmml" xref="S3.E1.m1.1.1.1.1.1.1.3.3.2">𝑡</ci><ci id="S3.E1.m1.1.1.1.1.1.1.3.3.3.cmml" xref="S3.E1.m1.1.1.1.1.1.1.3.3.3">𝒞</ci></apply></apply><apply id="S3.E1.m1.1.1.1.1.1.1.1.2.cmml" xref="S3.E1.m1.1.1.1.1.1.1.1.1"><csymbol cd="latexml" id="S3.E1.m1.1.1.1.1.1.1.1.2.1.cmml" xref="S3.E1.m1.1.1.1.1.1.1.1.1.2">delimited-[]</csymbol><apply id="S3.E1.m1.1.1.1.1.1.1.1.1.1.cmml" xref="S3.E1.m1.1.1.1.1.1.1.1.1.1"><apply id="S3.E1.m1.1.1.1.1.1.1.1.1.1.2.cmml" xref="S3.E1.m1.1.1.1.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S3.E1.m1.1.1.1.1.1.1.1.1.1.2.1.cmml" xref="S3.E1.m1.1.1.1.1.1.1.1.1.1.2">subscript</csymbol><sum id="S3.E1.m1.1.1.1.1.1.1.1.1.1.2.2.cmml" xref="S3.E1.m1.1.1.1.1.1.1.1.1.1.2.2"></sum><ci id="S3.E1.m1.1.1.1.1.1.1.1.1.1.2.3.cmml" xref="S3.E1.m1.1.1.1.1.1.1.1.1.1.2.3">𝑖</ci></apply><apply id="S3.E1.m1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S3.E1.m1.1.1.1.1.1.1.1.1.1.1"><times id="S3.E1.m1.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="S3.E1.m1.1.1.1.1.1.1.1.1.1.1.2"></times><apply id="S3.E1.m1.1.1.1.1.1.1.1.1.1.1.3.cmml" xref="S3.E1.m1.1.1.1.1.1.1.1.1.1.1.3"><log id="S3.E1.m1.1.1.1.1.1.1.1.1.1.1.3.1.cmml" xref="S3.E1.m1.1.1.1.1.1.1.1.1.1.1.3.1"></log><apply id="S3.E1.m1.1.1.1.1.1.1.1.1.1.1.3.2.cmml" xref="S3.E1.m1.1.1.1.1.1.1.1.1.1.1.3.2"><csymbol cd="ambiguous" id="S3.E1.m1.1.1.1.1.1.1.1.1.1.1.3.2.1.cmml" xref="S3.E1.m1.1.1.1.1.1.1.1.1.1.1.3.2">subscript</csymbol><ci id="S3.E1.m1.1.1.1.1.1.1.1.1.1.1.3.2.2.cmml" xref="S3.E1.m1.1.1.1.1.1.1.1.1.1.1.3.2.2">𝑝</ci><ci id="S3.E1.m1.1.1.1.1.1.1.1.1.1.1.3.2.3.cmml" xref="S3.E1.m1.1.1.1.1.1.1.1.1.1.1.3.2.3">𝜃</ci></apply></apply><apply id="S3.E1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S3.E1.m1.1.1.1.1.1.1.1.1.1.1.1.1"><csymbol cd="latexml" id="S3.E1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S3.E1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1">conditional</csymbol><apply id="S3.E1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="S3.E1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S3.E1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.1.cmml" xref="S3.E1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.2">subscript</csymbol><ci id="S3.E1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.2.cmml" xref="S3.E1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.2">𝑡</ci><ci id="S3.E1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.3.cmml" xref="S3.E1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.3">𝑖</ci></apply><apply id="S3.E1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.cmml" xref="S3.E1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.E1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.1.cmml" xref="S3.E1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3">subscript</csymbol><ci id="S3.E1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.2.cmml" xref="S3.E1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.2">𝑡</ci><apply id="S3.E1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.3.cmml" xref="S3.E1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.3"><lt id="S3.E1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.3.1.cmml" xref="S3.E1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.3.1"></lt><csymbol cd="latexml" id="S3.E1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.3.2.cmml" xref="S3.E1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.3.2">absent</csymbol><ci id="S3.E1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.3.3.cmml" xref="S3.E1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.3.3">𝑖</ci></apply></apply></apply></apply></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E1.m1.1c">\mathcal{L}_{\text{CLM}}=-\mathbb{E}_{t\sim\mathcal{C}}\left[\sum_{i}\log p_{\theta}(t_{i}|t_{&lt;i})\right],</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(1)</span></td>
</tr></tbody>
</table>
</div>
<div id="S3.SS1.p2" class="ltx_para ltx_noindent">
<p id="S3.SS1.p2.2" class="ltx_p">where <math id="S3.SS1.p2.1.m1.1" class="ltx_Math" alttext="p_{\theta}(t_{i}|t_{&lt;i})" display="inline"><semantics id="S3.SS1.p2.1.m1.1a"><mrow id="S3.SS1.p2.1.m1.1.1" xref="S3.SS1.p2.1.m1.1.1.cmml"><msub id="S3.SS1.p2.1.m1.1.1.3" xref="S3.SS1.p2.1.m1.1.1.3.cmml"><mi id="S3.SS1.p2.1.m1.1.1.3.2" xref="S3.SS1.p2.1.m1.1.1.3.2.cmml">p</mi><mi id="S3.SS1.p2.1.m1.1.1.3.3" xref="S3.SS1.p2.1.m1.1.1.3.3.cmml">θ</mi></msub><mo lspace="0em" rspace="0em" id="S3.SS1.p2.1.m1.1.1.2" xref="S3.SS1.p2.1.m1.1.1.2.cmml">​</mo><mrow id="S3.SS1.p2.1.m1.1.1.1.1" xref="S3.SS1.p2.1.m1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS1.p2.1.m1.1.1.1.1.2" xref="S3.SS1.p2.1.m1.1.1.1.1.1.cmml">(</mo><mrow id="S3.SS1.p2.1.m1.1.1.1.1.1" xref="S3.SS1.p2.1.m1.1.1.1.1.1.cmml"><msub id="S3.SS1.p2.1.m1.1.1.1.1.1.2" xref="S3.SS1.p2.1.m1.1.1.1.1.1.2.cmml"><mi id="S3.SS1.p2.1.m1.1.1.1.1.1.2.2" xref="S3.SS1.p2.1.m1.1.1.1.1.1.2.2.cmml">t</mi><mi id="S3.SS1.p2.1.m1.1.1.1.1.1.2.3" xref="S3.SS1.p2.1.m1.1.1.1.1.1.2.3.cmml">i</mi></msub><mo fence="false" id="S3.SS1.p2.1.m1.1.1.1.1.1.1" xref="S3.SS1.p2.1.m1.1.1.1.1.1.1.cmml">|</mo><msub id="S3.SS1.p2.1.m1.1.1.1.1.1.3" xref="S3.SS1.p2.1.m1.1.1.1.1.1.3.cmml"><mi id="S3.SS1.p2.1.m1.1.1.1.1.1.3.2" xref="S3.SS1.p2.1.m1.1.1.1.1.1.3.2.cmml">t</mi><mrow id="S3.SS1.p2.1.m1.1.1.1.1.1.3.3" xref="S3.SS1.p2.1.m1.1.1.1.1.1.3.3.cmml"><mi id="S3.SS1.p2.1.m1.1.1.1.1.1.3.3.2" xref="S3.SS1.p2.1.m1.1.1.1.1.1.3.3.2.cmml"></mi><mo id="S3.SS1.p2.1.m1.1.1.1.1.1.3.3.1" xref="S3.SS1.p2.1.m1.1.1.1.1.1.3.3.1.cmml">&lt;</mo><mi id="S3.SS1.p2.1.m1.1.1.1.1.1.3.3.3" xref="S3.SS1.p2.1.m1.1.1.1.1.1.3.3.3.cmml">i</mi></mrow></msub></mrow><mo stretchy="false" id="S3.SS1.p2.1.m1.1.1.1.1.3" xref="S3.SS1.p2.1.m1.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p2.1.m1.1b"><apply id="S3.SS1.p2.1.m1.1.1.cmml" xref="S3.SS1.p2.1.m1.1.1"><times id="S3.SS1.p2.1.m1.1.1.2.cmml" xref="S3.SS1.p2.1.m1.1.1.2"></times><apply id="S3.SS1.p2.1.m1.1.1.3.cmml" xref="S3.SS1.p2.1.m1.1.1.3"><csymbol cd="ambiguous" id="S3.SS1.p2.1.m1.1.1.3.1.cmml" xref="S3.SS1.p2.1.m1.1.1.3">subscript</csymbol><ci id="S3.SS1.p2.1.m1.1.1.3.2.cmml" xref="S3.SS1.p2.1.m1.1.1.3.2">𝑝</ci><ci id="S3.SS1.p2.1.m1.1.1.3.3.cmml" xref="S3.SS1.p2.1.m1.1.1.3.3">𝜃</ci></apply><apply id="S3.SS1.p2.1.m1.1.1.1.1.1.cmml" xref="S3.SS1.p2.1.m1.1.1.1.1"><csymbol cd="latexml" id="S3.SS1.p2.1.m1.1.1.1.1.1.1.cmml" xref="S3.SS1.p2.1.m1.1.1.1.1.1.1">conditional</csymbol><apply id="S3.SS1.p2.1.m1.1.1.1.1.1.2.cmml" xref="S3.SS1.p2.1.m1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S3.SS1.p2.1.m1.1.1.1.1.1.2.1.cmml" xref="S3.SS1.p2.1.m1.1.1.1.1.1.2">subscript</csymbol><ci id="S3.SS1.p2.1.m1.1.1.1.1.1.2.2.cmml" xref="S3.SS1.p2.1.m1.1.1.1.1.1.2.2">𝑡</ci><ci id="S3.SS1.p2.1.m1.1.1.1.1.1.2.3.cmml" xref="S3.SS1.p2.1.m1.1.1.1.1.1.2.3">𝑖</ci></apply><apply id="S3.SS1.p2.1.m1.1.1.1.1.1.3.cmml" xref="S3.SS1.p2.1.m1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.SS1.p2.1.m1.1.1.1.1.1.3.1.cmml" xref="S3.SS1.p2.1.m1.1.1.1.1.1.3">subscript</csymbol><ci id="S3.SS1.p2.1.m1.1.1.1.1.1.3.2.cmml" xref="S3.SS1.p2.1.m1.1.1.1.1.1.3.2">𝑡</ci><apply id="S3.SS1.p2.1.m1.1.1.1.1.1.3.3.cmml" xref="S3.SS1.p2.1.m1.1.1.1.1.1.3.3"><lt id="S3.SS1.p2.1.m1.1.1.1.1.1.3.3.1.cmml" xref="S3.SS1.p2.1.m1.1.1.1.1.1.3.3.1"></lt><csymbol cd="latexml" id="S3.SS1.p2.1.m1.1.1.1.1.1.3.3.2.cmml" xref="S3.SS1.p2.1.m1.1.1.1.1.1.3.3.2">absent</csymbol><ci id="S3.SS1.p2.1.m1.1.1.1.1.1.3.3.3.cmml" xref="S3.SS1.p2.1.m1.1.1.1.1.1.3.3.3">𝑖</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p2.1.m1.1c">p_{\theta}(t_{i}|t_{&lt;i})</annotation></semantics></math> is the model’s predicted probability for token <math id="S3.SS1.p2.2.m2.1" class="ltx_Math" alttext="t_{i}" display="inline"><semantics id="S3.SS1.p2.2.m2.1a"><msub id="S3.SS1.p2.2.m2.1.1" xref="S3.SS1.p2.2.m2.1.1.cmml"><mi id="S3.SS1.p2.2.m2.1.1.2" xref="S3.SS1.p2.2.m2.1.1.2.cmml">t</mi><mi id="S3.SS1.p2.2.m2.1.1.3" xref="S3.SS1.p2.2.m2.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS1.p2.2.m2.1b"><apply id="S3.SS1.p2.2.m2.1.1.cmml" xref="S3.SS1.p2.2.m2.1.1"><csymbol cd="ambiguous" id="S3.SS1.p2.2.m2.1.1.1.cmml" xref="S3.SS1.p2.2.m2.1.1">subscript</csymbol><ci id="S3.SS1.p2.2.m2.1.1.2.cmml" xref="S3.SS1.p2.2.m2.1.1.2">𝑡</ci><ci id="S3.SS1.p2.2.m2.1.1.3.cmml" xref="S3.SS1.p2.2.m2.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p2.2.m2.1c">t_{i}</annotation></semantics></math> given the preceding tokens.</p>
</div>
<div id="S3.SS1.p3" class="ltx_para ltx_noindent">
<p id="S3.SS1.p3.7" class="ltx_p">The above objective decomposes sequence likelihood into token-level cross-entropy losses, comparing each token’s predicted distribution to its one-hot representation. To provide a more generalized perspective, we reframe this token-level view into a sequential distribution format.
Specifically, for the text sequence <math id="S3.SS1.p3.1.m1.1" class="ltx_Math" alttext="t" display="inline"><semantics id="S3.SS1.p3.1.m1.1a"><mi id="S3.SS1.p3.1.m1.1.1" xref="S3.SS1.p3.1.m1.1.1.cmml">t</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p3.1.m1.1b"><ci id="S3.SS1.p3.1.m1.1.1.cmml" xref="S3.SS1.p3.1.m1.1.1">𝑡</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p3.1.m1.1c">t</annotation></semantics></math>, we aggregate token-level predictions and create a probabilistic distribution matrix, <math id="S3.SS1.p3.2.m2.1" class="ltx_Math" alttext="\mathbf{P}_{t}^{\theta}\in\mathbb{R}^{N\times V}" display="inline"><semantics id="S3.SS1.p3.2.m2.1a"><mrow id="S3.SS1.p3.2.m2.1.1" xref="S3.SS1.p3.2.m2.1.1.cmml"><msubsup id="S3.SS1.p3.2.m2.1.1.2" xref="S3.SS1.p3.2.m2.1.1.2.cmml"><mi id="S3.SS1.p3.2.m2.1.1.2.2.2" xref="S3.SS1.p3.2.m2.1.1.2.2.2.cmml">𝐏</mi><mi id="S3.SS1.p3.2.m2.1.1.2.2.3" xref="S3.SS1.p3.2.m2.1.1.2.2.3.cmml">t</mi><mi id="S3.SS1.p3.2.m2.1.1.2.3" xref="S3.SS1.p3.2.m2.1.1.2.3.cmml">θ</mi></msubsup><mo id="S3.SS1.p3.2.m2.1.1.1" xref="S3.SS1.p3.2.m2.1.1.1.cmml">∈</mo><msup id="S3.SS1.p3.2.m2.1.1.3" xref="S3.SS1.p3.2.m2.1.1.3.cmml"><mi id="S3.SS1.p3.2.m2.1.1.3.2" xref="S3.SS1.p3.2.m2.1.1.3.2.cmml">ℝ</mi><mrow id="S3.SS1.p3.2.m2.1.1.3.3" xref="S3.SS1.p3.2.m2.1.1.3.3.cmml"><mi id="S3.SS1.p3.2.m2.1.1.3.3.2" xref="S3.SS1.p3.2.m2.1.1.3.3.2.cmml">N</mi><mo lspace="0.222em" rspace="0.222em" id="S3.SS1.p3.2.m2.1.1.3.3.1" xref="S3.SS1.p3.2.m2.1.1.3.3.1.cmml">×</mo><mi id="S3.SS1.p3.2.m2.1.1.3.3.3" xref="S3.SS1.p3.2.m2.1.1.3.3.3.cmml">V</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p3.2.m2.1b"><apply id="S3.SS1.p3.2.m2.1.1.cmml" xref="S3.SS1.p3.2.m2.1.1"><in id="S3.SS1.p3.2.m2.1.1.1.cmml" xref="S3.SS1.p3.2.m2.1.1.1"></in><apply id="S3.SS1.p3.2.m2.1.1.2.cmml" xref="S3.SS1.p3.2.m2.1.1.2"><csymbol cd="ambiguous" id="S3.SS1.p3.2.m2.1.1.2.1.cmml" xref="S3.SS1.p3.2.m2.1.1.2">superscript</csymbol><apply id="S3.SS1.p3.2.m2.1.1.2.2.cmml" xref="S3.SS1.p3.2.m2.1.1.2"><csymbol cd="ambiguous" id="S3.SS1.p3.2.m2.1.1.2.2.1.cmml" xref="S3.SS1.p3.2.m2.1.1.2">subscript</csymbol><ci id="S3.SS1.p3.2.m2.1.1.2.2.2.cmml" xref="S3.SS1.p3.2.m2.1.1.2.2.2">𝐏</ci><ci id="S3.SS1.p3.2.m2.1.1.2.2.3.cmml" xref="S3.SS1.p3.2.m2.1.1.2.2.3">𝑡</ci></apply><ci id="S3.SS1.p3.2.m2.1.1.2.3.cmml" xref="S3.SS1.p3.2.m2.1.1.2.3">𝜃</ci></apply><apply id="S3.SS1.p3.2.m2.1.1.3.cmml" xref="S3.SS1.p3.2.m2.1.1.3"><csymbol cd="ambiguous" id="S3.SS1.p3.2.m2.1.1.3.1.cmml" xref="S3.SS1.p3.2.m2.1.1.3">superscript</csymbol><ci id="S3.SS1.p3.2.m2.1.1.3.2.cmml" xref="S3.SS1.p3.2.m2.1.1.3.2">ℝ</ci><apply id="S3.SS1.p3.2.m2.1.1.3.3.cmml" xref="S3.SS1.p3.2.m2.1.1.3.3"><times id="S3.SS1.p3.2.m2.1.1.3.3.1.cmml" xref="S3.SS1.p3.2.m2.1.1.3.3.1"></times><ci id="S3.SS1.p3.2.m2.1.1.3.3.2.cmml" xref="S3.SS1.p3.2.m2.1.1.3.3.2">𝑁</ci><ci id="S3.SS1.p3.2.m2.1.1.3.3.3.cmml" xref="S3.SS1.p3.2.m2.1.1.3.3.3">𝑉</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p3.2.m2.1c">\mathbf{P}_{t}^{\theta}\in\mathbb{R}^{N\times V}</annotation></semantics></math>, where the <math id="S3.SS1.p3.3.m3.1" class="ltx_Math" alttext="i" display="inline"><semantics id="S3.SS1.p3.3.m3.1a"><mi id="S3.SS1.p3.3.m3.1.1" xref="S3.SS1.p3.3.m3.1.1.cmml">i</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p3.3.m3.1b"><ci id="S3.SS1.p3.3.m3.1.1.cmml" xref="S3.SS1.p3.3.m3.1.1">𝑖</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p3.3.m3.1c">i</annotation></semantics></math>-th row represents the distribution predicted by the model for the <math id="S3.SS1.p3.4.m4.1" class="ltx_Math" alttext="i" display="inline"><semantics id="S3.SS1.p3.4.m4.1a"><mi id="S3.SS1.p3.4.m4.1.1" xref="S3.SS1.p3.4.m4.1.1.cmml">i</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p3.4.m4.1b"><ci id="S3.SS1.p3.4.m4.1.1.cmml" xref="S3.SS1.p3.4.m4.1.1">𝑖</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p3.4.m4.1c">i</annotation></semantics></math>th token over the vocabulary of size <math id="S3.SS1.p3.5.m5.1" class="ltx_Math" alttext="V" display="inline"><semantics id="S3.SS1.p3.5.m5.1a"><mi id="S3.SS1.p3.5.m5.1.1" xref="S3.SS1.p3.5.m5.1.1.cmml">V</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p3.5.m5.1b"><ci id="S3.SS1.p3.5.m5.1.1.cmml" xref="S3.SS1.p3.5.m5.1.1">𝑉</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p3.5.m5.1c">V</annotation></semantics></math>. The CLM objective can then be interpreted as reducing the discrepancy between <math id="S3.SS1.p3.6.m6.1" class="ltx_Math" alttext="\mathbf{P}_{t}^{\theta}" display="inline"><semantics id="S3.SS1.p3.6.m6.1a"><msubsup id="S3.SS1.p3.6.m6.1.1" xref="S3.SS1.p3.6.m6.1.1.cmml"><mi id="S3.SS1.p3.6.m6.1.1.2.2" xref="S3.SS1.p3.6.m6.1.1.2.2.cmml">𝐏</mi><mi id="S3.SS1.p3.6.m6.1.1.2.3" xref="S3.SS1.p3.6.m6.1.1.2.3.cmml">t</mi><mi id="S3.SS1.p3.6.m6.1.1.3" xref="S3.SS1.p3.6.m6.1.1.3.cmml">θ</mi></msubsup><annotation-xml encoding="MathML-Content" id="S3.SS1.p3.6.m6.1b"><apply id="S3.SS1.p3.6.m6.1.1.cmml" xref="S3.SS1.p3.6.m6.1.1"><csymbol cd="ambiguous" id="S3.SS1.p3.6.m6.1.1.1.cmml" xref="S3.SS1.p3.6.m6.1.1">superscript</csymbol><apply id="S3.SS1.p3.6.m6.1.1.2.cmml" xref="S3.SS1.p3.6.m6.1.1"><csymbol cd="ambiguous" id="S3.SS1.p3.6.m6.1.1.2.1.cmml" xref="S3.SS1.p3.6.m6.1.1">subscript</csymbol><ci id="S3.SS1.p3.6.m6.1.1.2.2.cmml" xref="S3.SS1.p3.6.m6.1.1.2.2">𝐏</ci><ci id="S3.SS1.p3.6.m6.1.1.2.3.cmml" xref="S3.SS1.p3.6.m6.1.1.2.3">𝑡</ci></apply><ci id="S3.SS1.p3.6.m6.1.1.3.cmml" xref="S3.SS1.p3.6.m6.1.1.3">𝜃</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p3.6.m6.1c">\mathbf{P}_{t}^{\theta}</annotation></semantics></math> and the one-hot label matrix, <math id="S3.SS1.p3.7.m7.2" class="ltx_Math" alttext="\mathbf{O}_{t}\in\{0,1\}^{N\times V}" display="inline"><semantics id="S3.SS1.p3.7.m7.2a"><mrow id="S3.SS1.p3.7.m7.2.3" xref="S3.SS1.p3.7.m7.2.3.cmml"><msub id="S3.SS1.p3.7.m7.2.3.2" xref="S3.SS1.p3.7.m7.2.3.2.cmml"><mi id="S3.SS1.p3.7.m7.2.3.2.2" xref="S3.SS1.p3.7.m7.2.3.2.2.cmml">𝐎</mi><mi id="S3.SS1.p3.7.m7.2.3.2.3" xref="S3.SS1.p3.7.m7.2.3.2.3.cmml">t</mi></msub><mo id="S3.SS1.p3.7.m7.2.3.1" xref="S3.SS1.p3.7.m7.2.3.1.cmml">∈</mo><msup id="S3.SS1.p3.7.m7.2.3.3" xref="S3.SS1.p3.7.m7.2.3.3.cmml"><mrow id="S3.SS1.p3.7.m7.2.3.3.2.2" xref="S3.SS1.p3.7.m7.2.3.3.2.1.cmml"><mo stretchy="false" id="S3.SS1.p3.7.m7.2.3.3.2.2.1" xref="S3.SS1.p3.7.m7.2.3.3.2.1.cmml">{</mo><mn id="S3.SS1.p3.7.m7.1.1" xref="S3.SS1.p3.7.m7.1.1.cmml">0</mn><mo id="S3.SS1.p3.7.m7.2.3.3.2.2.2" xref="S3.SS1.p3.7.m7.2.3.3.2.1.cmml">,</mo><mn id="S3.SS1.p3.7.m7.2.2" xref="S3.SS1.p3.7.m7.2.2.cmml">1</mn><mo stretchy="false" id="S3.SS1.p3.7.m7.2.3.3.2.2.3" xref="S3.SS1.p3.7.m7.2.3.3.2.1.cmml">}</mo></mrow><mrow id="S3.SS1.p3.7.m7.2.3.3.3" xref="S3.SS1.p3.7.m7.2.3.3.3.cmml"><mi id="S3.SS1.p3.7.m7.2.3.3.3.2" xref="S3.SS1.p3.7.m7.2.3.3.3.2.cmml">N</mi><mo lspace="0.222em" rspace="0.222em" id="S3.SS1.p3.7.m7.2.3.3.3.1" xref="S3.SS1.p3.7.m7.2.3.3.3.1.cmml">×</mo><mi id="S3.SS1.p3.7.m7.2.3.3.3.3" xref="S3.SS1.p3.7.m7.2.3.3.3.3.cmml">V</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p3.7.m7.2b"><apply id="S3.SS1.p3.7.m7.2.3.cmml" xref="S3.SS1.p3.7.m7.2.3"><in id="S3.SS1.p3.7.m7.2.3.1.cmml" xref="S3.SS1.p3.7.m7.2.3.1"></in><apply id="S3.SS1.p3.7.m7.2.3.2.cmml" xref="S3.SS1.p3.7.m7.2.3.2"><csymbol cd="ambiguous" id="S3.SS1.p3.7.m7.2.3.2.1.cmml" xref="S3.SS1.p3.7.m7.2.3.2">subscript</csymbol><ci id="S3.SS1.p3.7.m7.2.3.2.2.cmml" xref="S3.SS1.p3.7.m7.2.3.2.2">𝐎</ci><ci id="S3.SS1.p3.7.m7.2.3.2.3.cmml" xref="S3.SS1.p3.7.m7.2.3.2.3">𝑡</ci></apply><apply id="S3.SS1.p3.7.m7.2.3.3.cmml" xref="S3.SS1.p3.7.m7.2.3.3"><csymbol cd="ambiguous" id="S3.SS1.p3.7.m7.2.3.3.1.cmml" xref="S3.SS1.p3.7.m7.2.3.3">superscript</csymbol><set id="S3.SS1.p3.7.m7.2.3.3.2.1.cmml" xref="S3.SS1.p3.7.m7.2.3.3.2.2"><cn type="integer" id="S3.SS1.p3.7.m7.1.1.cmml" xref="S3.SS1.p3.7.m7.1.1">0</cn><cn type="integer" id="S3.SS1.p3.7.m7.2.2.cmml" xref="S3.SS1.p3.7.m7.2.2">1</cn></set><apply id="S3.SS1.p3.7.m7.2.3.3.3.cmml" xref="S3.SS1.p3.7.m7.2.3.3.3"><times id="S3.SS1.p3.7.m7.2.3.3.3.1.cmml" xref="S3.SS1.p3.7.m7.2.3.3.3.1"></times><ci id="S3.SS1.p3.7.m7.2.3.3.3.2.cmml" xref="S3.SS1.p3.7.m7.2.3.3.3.2">𝑁</ci><ci id="S3.SS1.p3.7.m7.2.3.3.3.3.cmml" xref="S3.SS1.p3.7.m7.2.3.3.3.3">𝑉</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p3.7.m7.2c">\mathbf{O}_{t}\in\{0,1\}^{N\times V}</annotation></semantics></math>, where each row is a one-hot representation of the corresponding gold token.
Formally, the CLM objective is transformed into the following representation:</p>
<table id="S3.E2" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S3.E2.m1.1" class="ltx_Math" alttext="\mathcal{L}_{\text{CLM}}=-\mathbb{E}_{t\sim\mathcal{C}}\left[\mathbb{D}(\mathbf{P}_{t}^{\theta},\mathbf{O}_{t})\right]," display="block"><semantics id="S3.E2.m1.1a"><mrow id="S3.E2.m1.1.1.1" xref="S3.E2.m1.1.1.1.1.cmml"><mrow id="S3.E2.m1.1.1.1.1" xref="S3.E2.m1.1.1.1.1.cmml"><msub id="S3.E2.m1.1.1.1.1.3" xref="S3.E2.m1.1.1.1.1.3.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.E2.m1.1.1.1.1.3.2" xref="S3.E2.m1.1.1.1.1.3.2.cmml">ℒ</mi><mtext id="S3.E2.m1.1.1.1.1.3.3" xref="S3.E2.m1.1.1.1.1.3.3a.cmml">CLM</mtext></msub><mo id="S3.E2.m1.1.1.1.1.2" xref="S3.E2.m1.1.1.1.1.2.cmml">=</mo><mrow id="S3.E2.m1.1.1.1.1.1" xref="S3.E2.m1.1.1.1.1.1.cmml"><mo id="S3.E2.m1.1.1.1.1.1a" xref="S3.E2.m1.1.1.1.1.1.cmml">−</mo><mrow id="S3.E2.m1.1.1.1.1.1.1" xref="S3.E2.m1.1.1.1.1.1.1.cmml"><msub id="S3.E2.m1.1.1.1.1.1.1.3" xref="S3.E2.m1.1.1.1.1.1.1.3.cmml"><mi id="S3.E2.m1.1.1.1.1.1.1.3.2" xref="S3.E2.m1.1.1.1.1.1.1.3.2.cmml">𝔼</mi><mrow id="S3.E2.m1.1.1.1.1.1.1.3.3" xref="S3.E2.m1.1.1.1.1.1.1.3.3.cmml"><mi id="S3.E2.m1.1.1.1.1.1.1.3.3.2" xref="S3.E2.m1.1.1.1.1.1.1.3.3.2.cmml">t</mi><mo id="S3.E2.m1.1.1.1.1.1.1.3.3.1" xref="S3.E2.m1.1.1.1.1.1.1.3.3.1.cmml">∼</mo><mi class="ltx_font_mathcaligraphic" id="S3.E2.m1.1.1.1.1.1.1.3.3.3" xref="S3.E2.m1.1.1.1.1.1.1.3.3.3.cmml">𝒞</mi></mrow></msub><mo lspace="0em" rspace="0em" id="S3.E2.m1.1.1.1.1.1.1.2" xref="S3.E2.m1.1.1.1.1.1.1.2.cmml">​</mo><mrow id="S3.E2.m1.1.1.1.1.1.1.1.1" xref="S3.E2.m1.1.1.1.1.1.1.1.2.cmml"><mo id="S3.E2.m1.1.1.1.1.1.1.1.1.2" xref="S3.E2.m1.1.1.1.1.1.1.1.2.1.cmml">[</mo><mrow id="S3.E2.m1.1.1.1.1.1.1.1.1.1" xref="S3.E2.m1.1.1.1.1.1.1.1.1.1.cmml"><mi id="S3.E2.m1.1.1.1.1.1.1.1.1.1.4" xref="S3.E2.m1.1.1.1.1.1.1.1.1.1.4.cmml">𝔻</mi><mo lspace="0em" rspace="0em" id="S3.E2.m1.1.1.1.1.1.1.1.1.1.3" xref="S3.E2.m1.1.1.1.1.1.1.1.1.1.3.cmml">​</mo><mrow id="S3.E2.m1.1.1.1.1.1.1.1.1.1.2.2" xref="S3.E2.m1.1.1.1.1.1.1.1.1.1.2.3.cmml"><mo stretchy="false" id="S3.E2.m1.1.1.1.1.1.1.1.1.1.2.2.3" xref="S3.E2.m1.1.1.1.1.1.1.1.1.1.2.3.cmml">(</mo><msubsup id="S3.E2.m1.1.1.1.1.1.1.1.1.1.1.1.1" xref="S3.E2.m1.1.1.1.1.1.1.1.1.1.1.1.1.cmml"><mi id="S3.E2.m1.1.1.1.1.1.1.1.1.1.1.1.1.2.2" xref="S3.E2.m1.1.1.1.1.1.1.1.1.1.1.1.1.2.2.cmml">𝐏</mi><mi id="S3.E2.m1.1.1.1.1.1.1.1.1.1.1.1.1.2.3" xref="S3.E2.m1.1.1.1.1.1.1.1.1.1.1.1.1.2.3.cmml">t</mi><mi id="S3.E2.m1.1.1.1.1.1.1.1.1.1.1.1.1.3" xref="S3.E2.m1.1.1.1.1.1.1.1.1.1.1.1.1.3.cmml">θ</mi></msubsup><mo id="S3.E2.m1.1.1.1.1.1.1.1.1.1.2.2.4" xref="S3.E2.m1.1.1.1.1.1.1.1.1.1.2.3.cmml">,</mo><msub id="S3.E2.m1.1.1.1.1.1.1.1.1.1.2.2.2" xref="S3.E2.m1.1.1.1.1.1.1.1.1.1.2.2.2.cmml"><mi id="S3.E2.m1.1.1.1.1.1.1.1.1.1.2.2.2.2" xref="S3.E2.m1.1.1.1.1.1.1.1.1.1.2.2.2.2.cmml">𝐎</mi><mi id="S3.E2.m1.1.1.1.1.1.1.1.1.1.2.2.2.3" xref="S3.E2.m1.1.1.1.1.1.1.1.1.1.2.2.2.3.cmml">t</mi></msub><mo stretchy="false" id="S3.E2.m1.1.1.1.1.1.1.1.1.1.2.2.5" xref="S3.E2.m1.1.1.1.1.1.1.1.1.1.2.3.cmml">)</mo></mrow></mrow><mo id="S3.E2.m1.1.1.1.1.1.1.1.1.3" xref="S3.E2.m1.1.1.1.1.1.1.1.2.1.cmml">]</mo></mrow></mrow></mrow></mrow><mo id="S3.E2.m1.1.1.1.2" xref="S3.E2.m1.1.1.1.1.cmml">,</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.E2.m1.1b"><apply id="S3.E2.m1.1.1.1.1.cmml" xref="S3.E2.m1.1.1.1"><eq id="S3.E2.m1.1.1.1.1.2.cmml" xref="S3.E2.m1.1.1.1.1.2"></eq><apply id="S3.E2.m1.1.1.1.1.3.cmml" xref="S3.E2.m1.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.E2.m1.1.1.1.1.3.1.cmml" xref="S3.E2.m1.1.1.1.1.3">subscript</csymbol><ci id="S3.E2.m1.1.1.1.1.3.2.cmml" xref="S3.E2.m1.1.1.1.1.3.2">ℒ</ci><ci id="S3.E2.m1.1.1.1.1.3.3a.cmml" xref="S3.E2.m1.1.1.1.1.3.3"><mtext mathsize="70%" id="S3.E2.m1.1.1.1.1.3.3.cmml" xref="S3.E2.m1.1.1.1.1.3.3">CLM</mtext></ci></apply><apply id="S3.E2.m1.1.1.1.1.1.cmml" xref="S3.E2.m1.1.1.1.1.1"><minus id="S3.E2.m1.1.1.1.1.1.2.cmml" xref="S3.E2.m1.1.1.1.1.1"></minus><apply id="S3.E2.m1.1.1.1.1.1.1.cmml" xref="S3.E2.m1.1.1.1.1.1.1"><times id="S3.E2.m1.1.1.1.1.1.1.2.cmml" xref="S3.E2.m1.1.1.1.1.1.1.2"></times><apply id="S3.E2.m1.1.1.1.1.1.1.3.cmml" xref="S3.E2.m1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.E2.m1.1.1.1.1.1.1.3.1.cmml" xref="S3.E2.m1.1.1.1.1.1.1.3">subscript</csymbol><ci id="S3.E2.m1.1.1.1.1.1.1.3.2.cmml" xref="S3.E2.m1.1.1.1.1.1.1.3.2">𝔼</ci><apply id="S3.E2.m1.1.1.1.1.1.1.3.3.cmml" xref="S3.E2.m1.1.1.1.1.1.1.3.3"><csymbol cd="latexml" id="S3.E2.m1.1.1.1.1.1.1.3.3.1.cmml" xref="S3.E2.m1.1.1.1.1.1.1.3.3.1">similar-to</csymbol><ci id="S3.E2.m1.1.1.1.1.1.1.3.3.2.cmml" xref="S3.E2.m1.1.1.1.1.1.1.3.3.2">𝑡</ci><ci id="S3.E2.m1.1.1.1.1.1.1.3.3.3.cmml" xref="S3.E2.m1.1.1.1.1.1.1.3.3.3">𝒞</ci></apply></apply><apply id="S3.E2.m1.1.1.1.1.1.1.1.2.cmml" xref="S3.E2.m1.1.1.1.1.1.1.1.1"><csymbol cd="latexml" id="S3.E2.m1.1.1.1.1.1.1.1.2.1.cmml" xref="S3.E2.m1.1.1.1.1.1.1.1.1.2">delimited-[]</csymbol><apply id="S3.E2.m1.1.1.1.1.1.1.1.1.1.cmml" xref="S3.E2.m1.1.1.1.1.1.1.1.1.1"><times id="S3.E2.m1.1.1.1.1.1.1.1.1.1.3.cmml" xref="S3.E2.m1.1.1.1.1.1.1.1.1.1.3"></times><ci id="S3.E2.m1.1.1.1.1.1.1.1.1.1.4.cmml" xref="S3.E2.m1.1.1.1.1.1.1.1.1.1.4">𝔻</ci><interval closure="open" id="S3.E2.m1.1.1.1.1.1.1.1.1.1.2.3.cmml" xref="S3.E2.m1.1.1.1.1.1.1.1.1.1.2.2"><apply id="S3.E2.m1.1.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S3.E2.m1.1.1.1.1.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.E2.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S3.E2.m1.1.1.1.1.1.1.1.1.1.1.1.1">superscript</csymbol><apply id="S3.E2.m1.1.1.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="S3.E2.m1.1.1.1.1.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.E2.m1.1.1.1.1.1.1.1.1.1.1.1.1.2.1.cmml" xref="S3.E2.m1.1.1.1.1.1.1.1.1.1.1.1.1">subscript</csymbol><ci id="S3.E2.m1.1.1.1.1.1.1.1.1.1.1.1.1.2.2.cmml" xref="S3.E2.m1.1.1.1.1.1.1.1.1.1.1.1.1.2.2">𝐏</ci><ci id="S3.E2.m1.1.1.1.1.1.1.1.1.1.1.1.1.2.3.cmml" xref="S3.E2.m1.1.1.1.1.1.1.1.1.1.1.1.1.2.3">𝑡</ci></apply><ci id="S3.E2.m1.1.1.1.1.1.1.1.1.1.1.1.1.3.cmml" xref="S3.E2.m1.1.1.1.1.1.1.1.1.1.1.1.1.3">𝜃</ci></apply><apply id="S3.E2.m1.1.1.1.1.1.1.1.1.1.2.2.2.cmml" xref="S3.E2.m1.1.1.1.1.1.1.1.1.1.2.2.2"><csymbol cd="ambiguous" id="S3.E2.m1.1.1.1.1.1.1.1.1.1.2.2.2.1.cmml" xref="S3.E2.m1.1.1.1.1.1.1.1.1.1.2.2.2">subscript</csymbol><ci id="S3.E2.m1.1.1.1.1.1.1.1.1.1.2.2.2.2.cmml" xref="S3.E2.m1.1.1.1.1.1.1.1.1.1.2.2.2.2">𝐎</ci><ci id="S3.E2.m1.1.1.1.1.1.1.1.1.1.2.2.2.3.cmml" xref="S3.E2.m1.1.1.1.1.1.1.1.1.1.2.2.2.3">𝑡</ci></apply></interval></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E2.m1.1c">\mathcal{L}_{\text{CLM}}=-\mathbb{E}_{t\sim\mathcal{C}}\left[\mathbb{D}(\mathbf{P}_{t}^{\theta},\mathbf{O}_{t})\right],</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(2)</span></td>
</tr></tbody>
</table>
<p id="S3.SS1.p3.8" class="ltx_p">where <math id="S3.SS1.p3.8.m1.2" class="ltx_Math" alttext="\mathbb{D}(\cdot,\cdot)" display="inline"><semantics id="S3.SS1.p3.8.m1.2a"><mrow id="S3.SS1.p3.8.m1.2.3" xref="S3.SS1.p3.8.m1.2.3.cmml"><mi id="S3.SS1.p3.8.m1.2.3.2" xref="S3.SS1.p3.8.m1.2.3.2.cmml">𝔻</mi><mo lspace="0em" rspace="0em" id="S3.SS1.p3.8.m1.2.3.1" xref="S3.SS1.p3.8.m1.2.3.1.cmml">​</mo><mrow id="S3.SS1.p3.8.m1.2.3.3.2" xref="S3.SS1.p3.8.m1.2.3.3.1.cmml"><mo stretchy="false" id="S3.SS1.p3.8.m1.2.3.3.2.1" xref="S3.SS1.p3.8.m1.2.3.3.1.cmml">(</mo><mo lspace="0em" rspace="0em" id="S3.SS1.p3.8.m1.1.1" xref="S3.SS1.p3.8.m1.1.1.cmml">⋅</mo><mo rspace="0em" id="S3.SS1.p3.8.m1.2.3.3.2.2" xref="S3.SS1.p3.8.m1.2.3.3.1.cmml">,</mo><mo lspace="0em" rspace="0em" id="S3.SS1.p3.8.m1.2.2" xref="S3.SS1.p3.8.m1.2.2.cmml">⋅</mo><mo stretchy="false" id="S3.SS1.p3.8.m1.2.3.3.2.3" xref="S3.SS1.p3.8.m1.2.3.3.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p3.8.m1.2b"><apply id="S3.SS1.p3.8.m1.2.3.cmml" xref="S3.SS1.p3.8.m1.2.3"><times id="S3.SS1.p3.8.m1.2.3.1.cmml" xref="S3.SS1.p3.8.m1.2.3.1"></times><ci id="S3.SS1.p3.8.m1.2.3.2.cmml" xref="S3.SS1.p3.8.m1.2.3.2">𝔻</ci><interval closure="open" id="S3.SS1.p3.8.m1.2.3.3.1.cmml" xref="S3.SS1.p3.8.m1.2.3.3.2"><ci id="S3.SS1.p3.8.m1.1.1.cmml" xref="S3.SS1.p3.8.m1.1.1">⋅</ci><ci id="S3.SS1.p3.8.m1.2.2.cmml" xref="S3.SS1.p3.8.m1.2.2">⋅</ci></interval></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p3.8.m1.2c">\mathbb{D}(\cdot,\cdot)</annotation></semantics></math> represents the discrepancy function between two matrices, and it is equivalent to Eq. <a href="#S3.E1" title="In 3.1 Preliminaries ‣ 3 Knowledge Fusion of LLMs ‣ Knowledge Fusion of Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a> when implemented as the KL divergence.</p>
</div>
</section>
<section id="S3.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.2 </span>LLMs Fusion</h3>

<div id="S3.SS2.p1" class="ltx_para ltx_noindent">
<p id="S3.SS2.p1.1" class="ltx_p">Taking this perspective on a language model, we argue that the probabilistic distribution matrix can reflect its certain inherent knowledge in understanding the text. Consequently, different probabilistic distribution matrices for the same text, originating from various LLMs, can be used to represent the diverse knowledge embedded within these models. Acknowledging this, the proposed <span id="S3.SS2.p1.1.1" class="ltx_text ltx_font_smallcaps">FuseLLM</span> approach tackles LLMs fusion through probabilistic modeling, aiming to create a unified LLM by merging the probabilistic distributions of the source LLMs. To achieve this, when starting with a set of LLMs to fuse, <span id="S3.SS2.p1.1.2" class="ltx_text ltx_font_smallcaps">FuseLLM</span> undergoes lightweight continual training of the target LLM on a raw text corpus that mirrors the pre-training dataset. Instead of relying solely on the CLM objective, <span id="S3.SS2.p1.1.3" class="ltx_text ltx_font_smallcaps">FuseLLM</span> places significant emphasis on minimizing the divergence between the target LLM’s probabilistic distributions and those of the source LLMs.</p>
</div>
<div id="S3.SS2.p2" class="ltx_para ltx_noindent">
<p id="S3.SS2.p2.6" class="ltx_p">For each text in the corpus <math id="S3.SS2.p2.1.m1.1" class="ltx_Math" alttext="\mathcal{C}" display="inline"><semantics id="S3.SS2.p2.1.m1.1a"><mi class="ltx_font_mathcaligraphic" id="S3.SS2.p2.1.m1.1.1" xref="S3.SS2.p2.1.m1.1.1.cmml">𝒞</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.p2.1.m1.1b"><ci id="S3.SS2.p2.1.m1.1.1.cmml" xref="S3.SS2.p2.1.m1.1.1">𝒞</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p2.1.m1.1c">\mathcal{C}</annotation></semantics></math>, we apply the provided <math id="S3.SS2.p2.2.m2.1" class="ltx_Math" alttext="K" display="inline"><semantics id="S3.SS2.p2.2.m2.1a"><mi id="S3.SS2.p2.2.m2.1.1" xref="S3.SS2.p2.2.m2.1.1.cmml">K</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.p2.2.m2.1b"><ci id="S3.SS2.p2.2.m2.1.1.cmml" xref="S3.SS2.p2.2.m2.1.1">𝐾</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p2.2.m2.1c">K</annotation></semantics></math> source LLMs and obtain a set of probabilistic distribution matrices, denoted as <math id="S3.SS2.p2.3.m3.1" class="ltx_Math" alttext="\{\mathbf{P}_{t}^{\theta_{j}}\}_{j=1}^{K}" display="inline"><semantics id="S3.SS2.p2.3.m3.1a"><msubsup id="S3.SS2.p2.3.m3.1.1" xref="S3.SS2.p2.3.m3.1.1.cmml"><mrow id="S3.SS2.p2.3.m3.1.1.1.1.1" xref="S3.SS2.p2.3.m3.1.1.1.1.2.cmml"><mo stretchy="false" id="S3.SS2.p2.3.m3.1.1.1.1.1.2" xref="S3.SS2.p2.3.m3.1.1.1.1.2.cmml">{</mo><msubsup id="S3.SS2.p2.3.m3.1.1.1.1.1.1" xref="S3.SS2.p2.3.m3.1.1.1.1.1.1.cmml"><mi id="S3.SS2.p2.3.m3.1.1.1.1.1.1.2.2" xref="S3.SS2.p2.3.m3.1.1.1.1.1.1.2.2.cmml">𝐏</mi><mi id="S3.SS2.p2.3.m3.1.1.1.1.1.1.2.3" xref="S3.SS2.p2.3.m3.1.1.1.1.1.1.2.3.cmml">t</mi><msub id="S3.SS2.p2.3.m3.1.1.1.1.1.1.3" xref="S3.SS2.p2.3.m3.1.1.1.1.1.1.3.cmml"><mi id="S3.SS2.p2.3.m3.1.1.1.1.1.1.3.2" xref="S3.SS2.p2.3.m3.1.1.1.1.1.1.3.2.cmml">θ</mi><mi id="S3.SS2.p2.3.m3.1.1.1.1.1.1.3.3" xref="S3.SS2.p2.3.m3.1.1.1.1.1.1.3.3.cmml">j</mi></msub></msubsup><mo stretchy="false" id="S3.SS2.p2.3.m3.1.1.1.1.1.3" xref="S3.SS2.p2.3.m3.1.1.1.1.2.cmml">}</mo></mrow><mrow id="S3.SS2.p2.3.m3.1.1.1.3" xref="S3.SS2.p2.3.m3.1.1.1.3.cmml"><mi id="S3.SS2.p2.3.m3.1.1.1.3.2" xref="S3.SS2.p2.3.m3.1.1.1.3.2.cmml">j</mi><mo id="S3.SS2.p2.3.m3.1.1.1.3.1" xref="S3.SS2.p2.3.m3.1.1.1.3.1.cmml">=</mo><mn id="S3.SS2.p2.3.m3.1.1.1.3.3" xref="S3.SS2.p2.3.m3.1.1.1.3.3.cmml">1</mn></mrow><mi id="S3.SS2.p2.3.m3.1.1.3" xref="S3.SS2.p2.3.m3.1.1.3.cmml">K</mi></msubsup><annotation-xml encoding="MathML-Content" id="S3.SS2.p2.3.m3.1b"><apply id="S3.SS2.p2.3.m3.1.1.cmml" xref="S3.SS2.p2.3.m3.1.1"><csymbol cd="ambiguous" id="S3.SS2.p2.3.m3.1.1.2.cmml" xref="S3.SS2.p2.3.m3.1.1">superscript</csymbol><apply id="S3.SS2.p2.3.m3.1.1.1.cmml" xref="S3.SS2.p2.3.m3.1.1"><csymbol cd="ambiguous" id="S3.SS2.p2.3.m3.1.1.1.2.cmml" xref="S3.SS2.p2.3.m3.1.1">subscript</csymbol><set id="S3.SS2.p2.3.m3.1.1.1.1.2.cmml" xref="S3.SS2.p2.3.m3.1.1.1.1.1"><apply id="S3.SS2.p2.3.m3.1.1.1.1.1.1.cmml" xref="S3.SS2.p2.3.m3.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS2.p2.3.m3.1.1.1.1.1.1.1.cmml" xref="S3.SS2.p2.3.m3.1.1.1.1.1.1">superscript</csymbol><apply id="S3.SS2.p2.3.m3.1.1.1.1.1.1.2.cmml" xref="S3.SS2.p2.3.m3.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS2.p2.3.m3.1.1.1.1.1.1.2.1.cmml" xref="S3.SS2.p2.3.m3.1.1.1.1.1.1">subscript</csymbol><ci id="S3.SS2.p2.3.m3.1.1.1.1.1.1.2.2.cmml" xref="S3.SS2.p2.3.m3.1.1.1.1.1.1.2.2">𝐏</ci><ci id="S3.SS2.p2.3.m3.1.1.1.1.1.1.2.3.cmml" xref="S3.SS2.p2.3.m3.1.1.1.1.1.1.2.3">𝑡</ci></apply><apply id="S3.SS2.p2.3.m3.1.1.1.1.1.1.3.cmml" xref="S3.SS2.p2.3.m3.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.SS2.p2.3.m3.1.1.1.1.1.1.3.1.cmml" xref="S3.SS2.p2.3.m3.1.1.1.1.1.1.3">subscript</csymbol><ci id="S3.SS2.p2.3.m3.1.1.1.1.1.1.3.2.cmml" xref="S3.SS2.p2.3.m3.1.1.1.1.1.1.3.2">𝜃</ci><ci id="S3.SS2.p2.3.m3.1.1.1.1.1.1.3.3.cmml" xref="S3.SS2.p2.3.m3.1.1.1.1.1.1.3.3">𝑗</ci></apply></apply></set><apply id="S3.SS2.p2.3.m3.1.1.1.3.cmml" xref="S3.SS2.p2.3.m3.1.1.1.3"><eq id="S3.SS2.p2.3.m3.1.1.1.3.1.cmml" xref="S3.SS2.p2.3.m3.1.1.1.3.1"></eq><ci id="S3.SS2.p2.3.m3.1.1.1.3.2.cmml" xref="S3.SS2.p2.3.m3.1.1.1.3.2">𝑗</ci><cn type="integer" id="S3.SS2.p2.3.m3.1.1.1.3.3.cmml" xref="S3.SS2.p2.3.m3.1.1.1.3.3">1</cn></apply></apply><ci id="S3.SS2.p2.3.m3.1.1.3.cmml" xref="S3.SS2.p2.3.m3.1.1.3">𝐾</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p2.3.m3.1c">\{\mathbf{P}_{t}^{\theta_{j}}\}_{j=1}^{K}</annotation></semantics></math>, where <math id="S3.SS2.p2.4.m4.1" class="ltx_Math" alttext="\theta_{j}" display="inline"><semantics id="S3.SS2.p2.4.m4.1a"><msub id="S3.SS2.p2.4.m4.1.1" xref="S3.SS2.p2.4.m4.1.1.cmml"><mi id="S3.SS2.p2.4.m4.1.1.2" xref="S3.SS2.p2.4.m4.1.1.2.cmml">θ</mi><mi id="S3.SS2.p2.4.m4.1.1.3" xref="S3.SS2.p2.4.m4.1.1.3.cmml">j</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.p2.4.m4.1b"><apply id="S3.SS2.p2.4.m4.1.1.cmml" xref="S3.SS2.p2.4.m4.1.1"><csymbol cd="ambiguous" id="S3.SS2.p2.4.m4.1.1.1.cmml" xref="S3.SS2.p2.4.m4.1.1">subscript</csymbol><ci id="S3.SS2.p2.4.m4.1.1.2.cmml" xref="S3.SS2.p2.4.m4.1.1.2">𝜃</ci><ci id="S3.SS2.p2.4.m4.1.1.3.cmml" xref="S3.SS2.p2.4.m4.1.1.3">𝑗</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p2.4.m4.1c">\theta_{j}</annotation></semantics></math> represents the parameters of the <math id="S3.SS2.p2.5.m5.1" class="ltx_Math" alttext="j" display="inline"><semantics id="S3.SS2.p2.5.m5.1a"><mi id="S3.SS2.p2.5.m5.1.1" xref="S3.SS2.p2.5.m5.1.1.cmml">j</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.p2.5.m5.1b"><ci id="S3.SS2.p2.5.m5.1.1.cmml" xref="S3.SS2.p2.5.m5.1.1">𝑗</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p2.5.m5.1c">j</annotation></semantics></math>th LLM. Utilizing these matrices, we externalize the knowledge from individual models into a unified space, essentially creating unified probabilistic representations over the text.
We acknowledge that variances in vocabulary among the source LLMs can lead to misaligned matrices <math id="S3.SS2.p2.6.m6.1" class="ltx_Math" alttext="\{\mathbf{P}_{t}^{\theta_{j}}\}_{j=1}^{K}" display="inline"><semantics id="S3.SS2.p2.6.m6.1a"><msubsup id="S3.SS2.p2.6.m6.1.1" xref="S3.SS2.p2.6.m6.1.1.cmml"><mrow id="S3.SS2.p2.6.m6.1.1.1.1.1" xref="S3.SS2.p2.6.m6.1.1.1.1.2.cmml"><mo stretchy="false" id="S3.SS2.p2.6.m6.1.1.1.1.1.2" xref="S3.SS2.p2.6.m6.1.1.1.1.2.cmml">{</mo><msubsup id="S3.SS2.p2.6.m6.1.1.1.1.1.1" xref="S3.SS2.p2.6.m6.1.1.1.1.1.1.cmml"><mi id="S3.SS2.p2.6.m6.1.1.1.1.1.1.2.2" xref="S3.SS2.p2.6.m6.1.1.1.1.1.1.2.2.cmml">𝐏</mi><mi id="S3.SS2.p2.6.m6.1.1.1.1.1.1.2.3" xref="S3.SS2.p2.6.m6.1.1.1.1.1.1.2.3.cmml">t</mi><msub id="S3.SS2.p2.6.m6.1.1.1.1.1.1.3" xref="S3.SS2.p2.6.m6.1.1.1.1.1.1.3.cmml"><mi id="S3.SS2.p2.6.m6.1.1.1.1.1.1.3.2" xref="S3.SS2.p2.6.m6.1.1.1.1.1.1.3.2.cmml">θ</mi><mi id="S3.SS2.p2.6.m6.1.1.1.1.1.1.3.3" xref="S3.SS2.p2.6.m6.1.1.1.1.1.1.3.3.cmml">j</mi></msub></msubsup><mo stretchy="false" id="S3.SS2.p2.6.m6.1.1.1.1.1.3" xref="S3.SS2.p2.6.m6.1.1.1.1.2.cmml">}</mo></mrow><mrow id="S3.SS2.p2.6.m6.1.1.1.3" xref="S3.SS2.p2.6.m6.1.1.1.3.cmml"><mi id="S3.SS2.p2.6.m6.1.1.1.3.2" xref="S3.SS2.p2.6.m6.1.1.1.3.2.cmml">j</mi><mo id="S3.SS2.p2.6.m6.1.1.1.3.1" xref="S3.SS2.p2.6.m6.1.1.1.3.1.cmml">=</mo><mn id="S3.SS2.p2.6.m6.1.1.1.3.3" xref="S3.SS2.p2.6.m6.1.1.1.3.3.cmml">1</mn></mrow><mi id="S3.SS2.p2.6.m6.1.1.3" xref="S3.SS2.p2.6.m6.1.1.3.cmml">K</mi></msubsup><annotation-xml encoding="MathML-Content" id="S3.SS2.p2.6.m6.1b"><apply id="S3.SS2.p2.6.m6.1.1.cmml" xref="S3.SS2.p2.6.m6.1.1"><csymbol cd="ambiguous" id="S3.SS2.p2.6.m6.1.1.2.cmml" xref="S3.SS2.p2.6.m6.1.1">superscript</csymbol><apply id="S3.SS2.p2.6.m6.1.1.1.cmml" xref="S3.SS2.p2.6.m6.1.1"><csymbol cd="ambiguous" id="S3.SS2.p2.6.m6.1.1.1.2.cmml" xref="S3.SS2.p2.6.m6.1.1">subscript</csymbol><set id="S3.SS2.p2.6.m6.1.1.1.1.2.cmml" xref="S3.SS2.p2.6.m6.1.1.1.1.1"><apply id="S3.SS2.p2.6.m6.1.1.1.1.1.1.cmml" xref="S3.SS2.p2.6.m6.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS2.p2.6.m6.1.1.1.1.1.1.1.cmml" xref="S3.SS2.p2.6.m6.1.1.1.1.1.1">superscript</csymbol><apply id="S3.SS2.p2.6.m6.1.1.1.1.1.1.2.cmml" xref="S3.SS2.p2.6.m6.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS2.p2.6.m6.1.1.1.1.1.1.2.1.cmml" xref="S3.SS2.p2.6.m6.1.1.1.1.1.1">subscript</csymbol><ci id="S3.SS2.p2.6.m6.1.1.1.1.1.1.2.2.cmml" xref="S3.SS2.p2.6.m6.1.1.1.1.1.1.2.2">𝐏</ci><ci id="S3.SS2.p2.6.m6.1.1.1.1.1.1.2.3.cmml" xref="S3.SS2.p2.6.m6.1.1.1.1.1.1.2.3">𝑡</ci></apply><apply id="S3.SS2.p2.6.m6.1.1.1.1.1.1.3.cmml" xref="S3.SS2.p2.6.m6.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.SS2.p2.6.m6.1.1.1.1.1.1.3.1.cmml" xref="S3.SS2.p2.6.m6.1.1.1.1.1.1.3">subscript</csymbol><ci id="S3.SS2.p2.6.m6.1.1.1.1.1.1.3.2.cmml" xref="S3.SS2.p2.6.m6.1.1.1.1.1.1.3.2">𝜃</ci><ci id="S3.SS2.p2.6.m6.1.1.1.1.1.1.3.3.cmml" xref="S3.SS2.p2.6.m6.1.1.1.1.1.1.3.3">𝑗</ci></apply></apply></set><apply id="S3.SS2.p2.6.m6.1.1.1.3.cmml" xref="S3.SS2.p2.6.m6.1.1.1.3"><eq id="S3.SS2.p2.6.m6.1.1.1.3.1.cmml" xref="S3.SS2.p2.6.m6.1.1.1.3.1"></eq><ci id="S3.SS2.p2.6.m6.1.1.1.3.2.cmml" xref="S3.SS2.p2.6.m6.1.1.1.3.2">𝑗</ci><cn type="integer" id="S3.SS2.p2.6.m6.1.1.1.3.3.cmml" xref="S3.SS2.p2.6.m6.1.1.1.3.3">1</cn></apply></apply><ci id="S3.SS2.p2.6.m6.1.1.3.cmml" xref="S3.SS2.p2.6.m6.1.1.3">𝐾</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p2.6.m6.1c">\{\mathbf{P}_{t}^{\theta_{j}}\}_{j=1}^{K}</annotation></semantics></math>. To address this, we employ a token alignment strategy, which is explained in Section <a href="#S3.SS3" title="3.3 Implementation of FuseLLM ‣ 3 Knowledge Fusion of LLMs ‣ Knowledge Fusion of Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.3</span></a>, to foster more coherent probabilistic interpretations across models.</p>
</div>
<div id="S3.SS2.p3" class="ltx_para ltx_noindent">
<p id="S3.SS2.p3.1" class="ltx_p">Having aligned the probabilistic matrices, we proceed to fuse them into a single compact representation. Various fusion strategies can be applied for this purpose, as detailed in Section <a href="#S3.SS3" title="3.3 Implementation of FuseLLM ‣ 3 Knowledge Fusion of LLMs ‣ Knowledge Fusion of Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.3</span></a>. We use <math id="S3.SS2.p3.1.m1.1" class="ltx_Math" alttext="\mathbf{P}_{t}" display="inline"><semantics id="S3.SS2.p3.1.m1.1a"><msub id="S3.SS2.p3.1.m1.1.1" xref="S3.SS2.p3.1.m1.1.1.cmml"><mi id="S3.SS2.p3.1.m1.1.1.2" xref="S3.SS2.p3.1.m1.1.1.2.cmml">𝐏</mi><mi id="S3.SS2.p3.1.m1.1.1.3" xref="S3.SS2.p3.1.m1.1.1.3.cmml">t</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.p3.1.m1.1b"><apply id="S3.SS2.p3.1.m1.1.1.cmml" xref="S3.SS2.p3.1.m1.1.1"><csymbol cd="ambiguous" id="S3.SS2.p3.1.m1.1.1.1.cmml" xref="S3.SS2.p3.1.m1.1.1">subscript</csymbol><ci id="S3.SS2.p3.1.m1.1.1.2.cmml" xref="S3.SS2.p3.1.m1.1.1.2">𝐏</ci><ci id="S3.SS2.p3.1.m1.1.1.3.cmml" xref="S3.SS2.p3.1.m1.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p3.1.m1.1c">\mathbf{P}_{t}</annotation></semantics></math> to represent the fused representation matrix as follows:</p>
<table id="S3.E3" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S3.E3.m1.2" class="ltx_Math" alttext="\mathbf{P}_{t}=\mathbb{F}\text{usion}(\mathbf{P}_{t}^{\theta_{1}},\mathbf{P}_{t}^{\theta_{2}},\ldots,\mathbf{P}_{t}^{\theta_{K}})," display="block"><semantics id="S3.E3.m1.2a"><mrow id="S3.E3.m1.2.2.1" xref="S3.E3.m1.2.2.1.1.cmml"><mrow id="S3.E3.m1.2.2.1.1" xref="S3.E3.m1.2.2.1.1.cmml"><msub id="S3.E3.m1.2.2.1.1.5" xref="S3.E3.m1.2.2.1.1.5.cmml"><mi id="S3.E3.m1.2.2.1.1.5.2" xref="S3.E3.m1.2.2.1.1.5.2.cmml">𝐏</mi><mi id="S3.E3.m1.2.2.1.1.5.3" xref="S3.E3.m1.2.2.1.1.5.3.cmml">t</mi></msub><mo id="S3.E3.m1.2.2.1.1.4" xref="S3.E3.m1.2.2.1.1.4.cmml">=</mo><mrow id="S3.E3.m1.2.2.1.1.3" xref="S3.E3.m1.2.2.1.1.3.cmml"><mi id="S3.E3.m1.2.2.1.1.3.5" xref="S3.E3.m1.2.2.1.1.3.5.cmml">𝔽</mi><mo lspace="0em" rspace="0em" id="S3.E3.m1.2.2.1.1.3.4" xref="S3.E3.m1.2.2.1.1.3.4.cmml">​</mo><mtext id="S3.E3.m1.2.2.1.1.3.6" xref="S3.E3.m1.2.2.1.1.3.6a.cmml">usion</mtext><mo lspace="0em" rspace="0em" id="S3.E3.m1.2.2.1.1.3.4a" xref="S3.E3.m1.2.2.1.1.3.4.cmml">​</mo><mrow id="S3.E3.m1.2.2.1.1.3.3.3" xref="S3.E3.m1.2.2.1.1.3.3.4.cmml"><mo stretchy="false" id="S3.E3.m1.2.2.1.1.3.3.3.4" xref="S3.E3.m1.2.2.1.1.3.3.4.cmml">(</mo><msubsup id="S3.E3.m1.2.2.1.1.1.1.1.1" xref="S3.E3.m1.2.2.1.1.1.1.1.1.cmml"><mi id="S3.E3.m1.2.2.1.1.1.1.1.1.2.2" xref="S3.E3.m1.2.2.1.1.1.1.1.1.2.2.cmml">𝐏</mi><mi id="S3.E3.m1.2.2.1.1.1.1.1.1.2.3" xref="S3.E3.m1.2.2.1.1.1.1.1.1.2.3.cmml">t</mi><msub id="S3.E3.m1.2.2.1.1.1.1.1.1.3" xref="S3.E3.m1.2.2.1.1.1.1.1.1.3.cmml"><mi id="S3.E3.m1.2.2.1.1.1.1.1.1.3.2" xref="S3.E3.m1.2.2.1.1.1.1.1.1.3.2.cmml">θ</mi><mn id="S3.E3.m1.2.2.1.1.1.1.1.1.3.3" xref="S3.E3.m1.2.2.1.1.1.1.1.1.3.3.cmml">1</mn></msub></msubsup><mo id="S3.E3.m1.2.2.1.1.3.3.3.5" xref="S3.E3.m1.2.2.1.1.3.3.4.cmml">,</mo><msubsup id="S3.E3.m1.2.2.1.1.2.2.2.2" xref="S3.E3.m1.2.2.1.1.2.2.2.2.cmml"><mi id="S3.E3.m1.2.2.1.1.2.2.2.2.2.2" xref="S3.E3.m1.2.2.1.1.2.2.2.2.2.2.cmml">𝐏</mi><mi id="S3.E3.m1.2.2.1.1.2.2.2.2.2.3" xref="S3.E3.m1.2.2.1.1.2.2.2.2.2.3.cmml">t</mi><msub id="S3.E3.m1.2.2.1.1.2.2.2.2.3" xref="S3.E3.m1.2.2.1.1.2.2.2.2.3.cmml"><mi id="S3.E3.m1.2.2.1.1.2.2.2.2.3.2" xref="S3.E3.m1.2.2.1.1.2.2.2.2.3.2.cmml">θ</mi><mn id="S3.E3.m1.2.2.1.1.2.2.2.2.3.3" xref="S3.E3.m1.2.2.1.1.2.2.2.2.3.3.cmml">2</mn></msub></msubsup><mo id="S3.E3.m1.2.2.1.1.3.3.3.6" xref="S3.E3.m1.2.2.1.1.3.3.4.cmml">,</mo><mi mathvariant="normal" id="S3.E3.m1.1.1" xref="S3.E3.m1.1.1.cmml">…</mi><mo id="S3.E3.m1.2.2.1.1.3.3.3.7" xref="S3.E3.m1.2.2.1.1.3.3.4.cmml">,</mo><msubsup id="S3.E3.m1.2.2.1.1.3.3.3.3" xref="S3.E3.m1.2.2.1.1.3.3.3.3.cmml"><mi id="S3.E3.m1.2.2.1.1.3.3.3.3.2.2" xref="S3.E3.m1.2.2.1.1.3.3.3.3.2.2.cmml">𝐏</mi><mi id="S3.E3.m1.2.2.1.1.3.3.3.3.2.3" xref="S3.E3.m1.2.2.1.1.3.3.3.3.2.3.cmml">t</mi><msub id="S3.E3.m1.2.2.1.1.3.3.3.3.3" xref="S3.E3.m1.2.2.1.1.3.3.3.3.3.cmml"><mi id="S3.E3.m1.2.2.1.1.3.3.3.3.3.2" xref="S3.E3.m1.2.2.1.1.3.3.3.3.3.2.cmml">θ</mi><mi id="S3.E3.m1.2.2.1.1.3.3.3.3.3.3" xref="S3.E3.m1.2.2.1.1.3.3.3.3.3.3.cmml">K</mi></msub></msubsup><mo stretchy="false" id="S3.E3.m1.2.2.1.1.3.3.3.8" xref="S3.E3.m1.2.2.1.1.3.3.4.cmml">)</mo></mrow></mrow></mrow><mo id="S3.E3.m1.2.2.1.2" xref="S3.E3.m1.2.2.1.1.cmml">,</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.E3.m1.2b"><apply id="S3.E3.m1.2.2.1.1.cmml" xref="S3.E3.m1.2.2.1"><eq id="S3.E3.m1.2.2.1.1.4.cmml" xref="S3.E3.m1.2.2.1.1.4"></eq><apply id="S3.E3.m1.2.2.1.1.5.cmml" xref="S3.E3.m1.2.2.1.1.5"><csymbol cd="ambiguous" id="S3.E3.m1.2.2.1.1.5.1.cmml" xref="S3.E3.m1.2.2.1.1.5">subscript</csymbol><ci id="S3.E3.m1.2.2.1.1.5.2.cmml" xref="S3.E3.m1.2.2.1.1.5.2">𝐏</ci><ci id="S3.E3.m1.2.2.1.1.5.3.cmml" xref="S3.E3.m1.2.2.1.1.5.3">𝑡</ci></apply><apply id="S3.E3.m1.2.2.1.1.3.cmml" xref="S3.E3.m1.2.2.1.1.3"><times id="S3.E3.m1.2.2.1.1.3.4.cmml" xref="S3.E3.m1.2.2.1.1.3.4"></times><ci id="S3.E3.m1.2.2.1.1.3.5.cmml" xref="S3.E3.m1.2.2.1.1.3.5">𝔽</ci><ci id="S3.E3.m1.2.2.1.1.3.6a.cmml" xref="S3.E3.m1.2.2.1.1.3.6"><mtext id="S3.E3.m1.2.2.1.1.3.6.cmml" xref="S3.E3.m1.2.2.1.1.3.6">usion</mtext></ci><vector id="S3.E3.m1.2.2.1.1.3.3.4.cmml" xref="S3.E3.m1.2.2.1.1.3.3.3"><apply id="S3.E3.m1.2.2.1.1.1.1.1.1.cmml" xref="S3.E3.m1.2.2.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.E3.m1.2.2.1.1.1.1.1.1.1.cmml" xref="S3.E3.m1.2.2.1.1.1.1.1.1">superscript</csymbol><apply id="S3.E3.m1.2.2.1.1.1.1.1.1.2.cmml" xref="S3.E3.m1.2.2.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.E3.m1.2.2.1.1.1.1.1.1.2.1.cmml" xref="S3.E3.m1.2.2.1.1.1.1.1.1">subscript</csymbol><ci id="S3.E3.m1.2.2.1.1.1.1.1.1.2.2.cmml" xref="S3.E3.m1.2.2.1.1.1.1.1.1.2.2">𝐏</ci><ci id="S3.E3.m1.2.2.1.1.1.1.1.1.2.3.cmml" xref="S3.E3.m1.2.2.1.1.1.1.1.1.2.3">𝑡</ci></apply><apply id="S3.E3.m1.2.2.1.1.1.1.1.1.3.cmml" xref="S3.E3.m1.2.2.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.E3.m1.2.2.1.1.1.1.1.1.3.1.cmml" xref="S3.E3.m1.2.2.1.1.1.1.1.1.3">subscript</csymbol><ci id="S3.E3.m1.2.2.1.1.1.1.1.1.3.2.cmml" xref="S3.E3.m1.2.2.1.1.1.1.1.1.3.2">𝜃</ci><cn type="integer" id="S3.E3.m1.2.2.1.1.1.1.1.1.3.3.cmml" xref="S3.E3.m1.2.2.1.1.1.1.1.1.3.3">1</cn></apply></apply><apply id="S3.E3.m1.2.2.1.1.2.2.2.2.cmml" xref="S3.E3.m1.2.2.1.1.2.2.2.2"><csymbol cd="ambiguous" id="S3.E3.m1.2.2.1.1.2.2.2.2.1.cmml" xref="S3.E3.m1.2.2.1.1.2.2.2.2">superscript</csymbol><apply id="S3.E3.m1.2.2.1.1.2.2.2.2.2.cmml" xref="S3.E3.m1.2.2.1.1.2.2.2.2"><csymbol cd="ambiguous" id="S3.E3.m1.2.2.1.1.2.2.2.2.2.1.cmml" xref="S3.E3.m1.2.2.1.1.2.2.2.2">subscript</csymbol><ci id="S3.E3.m1.2.2.1.1.2.2.2.2.2.2.cmml" xref="S3.E3.m1.2.2.1.1.2.2.2.2.2.2">𝐏</ci><ci id="S3.E3.m1.2.2.1.1.2.2.2.2.2.3.cmml" xref="S3.E3.m1.2.2.1.1.2.2.2.2.2.3">𝑡</ci></apply><apply id="S3.E3.m1.2.2.1.1.2.2.2.2.3.cmml" xref="S3.E3.m1.2.2.1.1.2.2.2.2.3"><csymbol cd="ambiguous" id="S3.E3.m1.2.2.1.1.2.2.2.2.3.1.cmml" xref="S3.E3.m1.2.2.1.1.2.2.2.2.3">subscript</csymbol><ci id="S3.E3.m1.2.2.1.1.2.2.2.2.3.2.cmml" xref="S3.E3.m1.2.2.1.1.2.2.2.2.3.2">𝜃</ci><cn type="integer" id="S3.E3.m1.2.2.1.1.2.2.2.2.3.3.cmml" xref="S3.E3.m1.2.2.1.1.2.2.2.2.3.3">2</cn></apply></apply><ci id="S3.E3.m1.1.1.cmml" xref="S3.E3.m1.1.1">…</ci><apply id="S3.E3.m1.2.2.1.1.3.3.3.3.cmml" xref="S3.E3.m1.2.2.1.1.3.3.3.3"><csymbol cd="ambiguous" id="S3.E3.m1.2.2.1.1.3.3.3.3.1.cmml" xref="S3.E3.m1.2.2.1.1.3.3.3.3">superscript</csymbol><apply id="S3.E3.m1.2.2.1.1.3.3.3.3.2.cmml" xref="S3.E3.m1.2.2.1.1.3.3.3.3"><csymbol cd="ambiguous" id="S3.E3.m1.2.2.1.1.3.3.3.3.2.1.cmml" xref="S3.E3.m1.2.2.1.1.3.3.3.3">subscript</csymbol><ci id="S3.E3.m1.2.2.1.1.3.3.3.3.2.2.cmml" xref="S3.E3.m1.2.2.1.1.3.3.3.3.2.2">𝐏</ci><ci id="S3.E3.m1.2.2.1.1.3.3.3.3.2.3.cmml" xref="S3.E3.m1.2.2.1.1.3.3.3.3.2.3">𝑡</ci></apply><apply id="S3.E3.m1.2.2.1.1.3.3.3.3.3.cmml" xref="S3.E3.m1.2.2.1.1.3.3.3.3.3"><csymbol cd="ambiguous" id="S3.E3.m1.2.2.1.1.3.3.3.3.3.1.cmml" xref="S3.E3.m1.2.2.1.1.3.3.3.3.3">subscript</csymbol><ci id="S3.E3.m1.2.2.1.1.3.3.3.3.3.2.cmml" xref="S3.E3.m1.2.2.1.1.3.3.3.3.3.2">𝜃</ci><ci id="S3.E3.m1.2.2.1.1.3.3.3.3.3.3.cmml" xref="S3.E3.m1.2.2.1.1.3.3.3.3.3.3">𝐾</ci></apply></apply></vector></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E3.m1.2c">\mathbf{P}_{t}=\mathbb{F}\text{usion}(\mathbf{P}_{t}^{\theta_{1}},\mathbf{P}_{t}^{\theta_{2}},\ldots,\mathbf{P}_{t}^{\theta_{K}}),</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(3)</span></td>
</tr></tbody>
</table>
<p id="S3.SS2.p3.3" class="ltx_p">where <math id="S3.SS2.p3.2.m1.1" class="ltx_Math" alttext="\mathbb{F}\text{usion}(\cdot)" display="inline"><semantics id="S3.SS2.p3.2.m1.1a"><mrow id="S3.SS2.p3.2.m1.1.2" xref="S3.SS2.p3.2.m1.1.2.cmml"><mi id="S3.SS2.p3.2.m1.1.2.2" xref="S3.SS2.p3.2.m1.1.2.2.cmml">𝔽</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p3.2.m1.1.2.1" xref="S3.SS2.p3.2.m1.1.2.1.cmml">​</mo><mtext id="S3.SS2.p3.2.m1.1.2.3" xref="S3.SS2.p3.2.m1.1.2.3a.cmml">usion</mtext><mo lspace="0em" rspace="0em" id="S3.SS2.p3.2.m1.1.2.1a" xref="S3.SS2.p3.2.m1.1.2.1.cmml">​</mo><mrow id="S3.SS2.p3.2.m1.1.2.4.2" xref="S3.SS2.p3.2.m1.1.2.cmml"><mo stretchy="false" id="S3.SS2.p3.2.m1.1.2.4.2.1" xref="S3.SS2.p3.2.m1.1.2.cmml">(</mo><mo lspace="0em" rspace="0em" id="S3.SS2.p3.2.m1.1.1" xref="S3.SS2.p3.2.m1.1.1.cmml">⋅</mo><mo stretchy="false" id="S3.SS2.p3.2.m1.1.2.4.2.2" xref="S3.SS2.p3.2.m1.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p3.2.m1.1b"><apply id="S3.SS2.p3.2.m1.1.2.cmml" xref="S3.SS2.p3.2.m1.1.2"><times id="S3.SS2.p3.2.m1.1.2.1.cmml" xref="S3.SS2.p3.2.m1.1.2.1"></times><ci id="S3.SS2.p3.2.m1.1.2.2.cmml" xref="S3.SS2.p3.2.m1.1.2.2">𝔽</ci><ci id="S3.SS2.p3.2.m1.1.2.3a.cmml" xref="S3.SS2.p3.2.m1.1.2.3"><mtext id="S3.SS2.p3.2.m1.1.2.3.cmml" xref="S3.SS2.p3.2.m1.1.2.3">usion</mtext></ci><ci id="S3.SS2.p3.2.m1.1.1.cmml" xref="S3.SS2.p3.2.m1.1.1">⋅</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p3.2.m1.1c">\mathbb{F}\text{usion}(\cdot)</annotation></semantics></math> denotes the function that combines multiple matrices, and the resulting matrix <math id="S3.SS2.p3.3.m2.1" class="ltx_Math" alttext="\mathbf{P}_{t}" display="inline"><semantics id="S3.SS2.p3.3.m2.1a"><msub id="S3.SS2.p3.3.m2.1.1" xref="S3.SS2.p3.3.m2.1.1.cmml"><mi id="S3.SS2.p3.3.m2.1.1.2" xref="S3.SS2.p3.3.m2.1.1.2.cmml">𝐏</mi><mi id="S3.SS2.p3.3.m2.1.1.3" xref="S3.SS2.p3.3.m2.1.1.3.cmml">t</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.p3.3.m2.1b"><apply id="S3.SS2.p3.3.m2.1.1.cmml" xref="S3.SS2.p3.3.m2.1.1"><csymbol cd="ambiguous" id="S3.SS2.p3.3.m2.1.1.1.cmml" xref="S3.SS2.p3.3.m2.1.1">subscript</csymbol><ci id="S3.SS2.p3.3.m2.1.1.2.cmml" xref="S3.SS2.p3.3.m2.1.1.2">𝐏</ci><ci id="S3.SS2.p3.3.m2.1.1.3.cmml" xref="S3.SS2.p3.3.m2.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p3.3.m2.1c">\mathbf{P}_{t}</annotation></semantics></math> is seen as a representation of the collective knowledge and distinctive strengths of the source LLMs.</p>
</div>
<div id="S3.SS2.p4" class="ltx_para ltx_noindent">
<p id="S3.SS2.p4.3" class="ltx_p">To transfer the capabilities of source LLMs to the target LLM, we enforce alignment between the target LLM’s predictions and the fused representation matrix <math id="S3.SS2.p4.1.m1.1" class="ltx_Math" alttext="\mathbf{P}_{t}" display="inline"><semantics id="S3.SS2.p4.1.m1.1a"><msub id="S3.SS2.p4.1.m1.1.1" xref="S3.SS2.p4.1.m1.1.1.cmml"><mi id="S3.SS2.p4.1.m1.1.1.2" xref="S3.SS2.p4.1.m1.1.1.2.cmml">𝐏</mi><mi id="S3.SS2.p4.1.m1.1.1.3" xref="S3.SS2.p4.1.m1.1.1.3.cmml">t</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.p4.1.m1.1b"><apply id="S3.SS2.p4.1.m1.1.1.cmml" xref="S3.SS2.p4.1.m1.1.1"><csymbol cd="ambiguous" id="S3.SS2.p4.1.m1.1.1.1.cmml" xref="S3.SS2.p4.1.m1.1.1">subscript</csymbol><ci id="S3.SS2.p4.1.m1.1.1.2.cmml" xref="S3.SS2.p4.1.m1.1.1.2">𝐏</ci><ci id="S3.SS2.p4.1.m1.1.1.3.cmml" xref="S3.SS2.p4.1.m1.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p4.1.m1.1c">\mathbf{P}_{t}</annotation></semantics></math>. We use <math id="S3.SS2.p4.2.m2.1" class="ltx_Math" alttext="\mathbf{Q}_{t}" display="inline"><semantics id="S3.SS2.p4.2.m2.1a"><msub id="S3.SS2.p4.2.m2.1.1" xref="S3.SS2.p4.2.m2.1.1.cmml"><mi id="S3.SS2.p4.2.m2.1.1.2" xref="S3.SS2.p4.2.m2.1.1.2.cmml">𝐐</mi><mi id="S3.SS2.p4.2.m2.1.1.3" xref="S3.SS2.p4.2.m2.1.1.3.cmml">t</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.p4.2.m2.1b"><apply id="S3.SS2.p4.2.m2.1.1.cmml" xref="S3.SS2.p4.2.m2.1.1"><csymbol cd="ambiguous" id="S3.SS2.p4.2.m2.1.1.1.cmml" xref="S3.SS2.p4.2.m2.1.1">subscript</csymbol><ci id="S3.SS2.p4.2.m2.1.1.2.cmml" xref="S3.SS2.p4.2.m2.1.1.2">𝐐</ci><ci id="S3.SS2.p4.2.m2.1.1.3.cmml" xref="S3.SS2.p4.2.m2.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p4.2.m2.1c">\mathbf{Q}_{t}</annotation></semantics></math> to represent the output distribution matrix of the target LLM for text <math id="S3.SS2.p4.3.m3.1" class="ltx_Math" alttext="t" display="inline"><semantics id="S3.SS2.p4.3.m3.1a"><mi id="S3.SS2.p4.3.m3.1.1" xref="S3.SS2.p4.3.m3.1.1.cmml">t</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.p4.3.m3.1b"><ci id="S3.SS2.p4.3.m3.1.1.cmml" xref="S3.SS2.p4.3.m3.1.1">𝑡</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p4.3.m3.1c">t</annotation></semantics></math>, and then define our fusion objective as follows:</p>
<table id="S3.E4" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S3.E4.m1.1" class="ltx_Math" alttext="\mathcal{L}_{\text{Fusion}}=-\mathbb{E}_{t\sim\mathcal{C}}\left[\mathbb{D}(\mathbf{Q}_{t},\mathbf{P}_{t})\right]." display="block"><semantics id="S3.E4.m1.1a"><mrow id="S3.E4.m1.1.1.1" xref="S3.E4.m1.1.1.1.1.cmml"><mrow id="S3.E4.m1.1.1.1.1" xref="S3.E4.m1.1.1.1.1.cmml"><msub id="S3.E4.m1.1.1.1.1.3" xref="S3.E4.m1.1.1.1.1.3.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.E4.m1.1.1.1.1.3.2" xref="S3.E4.m1.1.1.1.1.3.2.cmml">ℒ</mi><mtext id="S3.E4.m1.1.1.1.1.3.3" xref="S3.E4.m1.1.1.1.1.3.3a.cmml">Fusion</mtext></msub><mo id="S3.E4.m1.1.1.1.1.2" xref="S3.E4.m1.1.1.1.1.2.cmml">=</mo><mrow id="S3.E4.m1.1.1.1.1.1" xref="S3.E4.m1.1.1.1.1.1.cmml"><mo id="S3.E4.m1.1.1.1.1.1a" xref="S3.E4.m1.1.1.1.1.1.cmml">−</mo><mrow id="S3.E4.m1.1.1.1.1.1.1" xref="S3.E4.m1.1.1.1.1.1.1.cmml"><msub id="S3.E4.m1.1.1.1.1.1.1.3" xref="S3.E4.m1.1.1.1.1.1.1.3.cmml"><mi id="S3.E4.m1.1.1.1.1.1.1.3.2" xref="S3.E4.m1.1.1.1.1.1.1.3.2.cmml">𝔼</mi><mrow id="S3.E4.m1.1.1.1.1.1.1.3.3" xref="S3.E4.m1.1.1.1.1.1.1.3.3.cmml"><mi id="S3.E4.m1.1.1.1.1.1.1.3.3.2" xref="S3.E4.m1.1.1.1.1.1.1.3.3.2.cmml">t</mi><mo id="S3.E4.m1.1.1.1.1.1.1.3.3.1" xref="S3.E4.m1.1.1.1.1.1.1.3.3.1.cmml">∼</mo><mi class="ltx_font_mathcaligraphic" id="S3.E4.m1.1.1.1.1.1.1.3.3.3" xref="S3.E4.m1.1.1.1.1.1.1.3.3.3.cmml">𝒞</mi></mrow></msub><mo lspace="0em" rspace="0em" id="S3.E4.m1.1.1.1.1.1.1.2" xref="S3.E4.m1.1.1.1.1.1.1.2.cmml">​</mo><mrow id="S3.E4.m1.1.1.1.1.1.1.1.1" xref="S3.E4.m1.1.1.1.1.1.1.1.2.cmml"><mo id="S3.E4.m1.1.1.1.1.1.1.1.1.2" xref="S3.E4.m1.1.1.1.1.1.1.1.2.1.cmml">[</mo><mrow id="S3.E4.m1.1.1.1.1.1.1.1.1.1" xref="S3.E4.m1.1.1.1.1.1.1.1.1.1.cmml"><mi id="S3.E4.m1.1.1.1.1.1.1.1.1.1.4" xref="S3.E4.m1.1.1.1.1.1.1.1.1.1.4.cmml">𝔻</mi><mo lspace="0em" rspace="0em" id="S3.E4.m1.1.1.1.1.1.1.1.1.1.3" xref="S3.E4.m1.1.1.1.1.1.1.1.1.1.3.cmml">​</mo><mrow id="S3.E4.m1.1.1.1.1.1.1.1.1.1.2.2" xref="S3.E4.m1.1.1.1.1.1.1.1.1.1.2.3.cmml"><mo stretchy="false" id="S3.E4.m1.1.1.1.1.1.1.1.1.1.2.2.3" xref="S3.E4.m1.1.1.1.1.1.1.1.1.1.2.3.cmml">(</mo><msub id="S3.E4.m1.1.1.1.1.1.1.1.1.1.1.1.1" xref="S3.E4.m1.1.1.1.1.1.1.1.1.1.1.1.1.cmml"><mi id="S3.E4.m1.1.1.1.1.1.1.1.1.1.1.1.1.2" xref="S3.E4.m1.1.1.1.1.1.1.1.1.1.1.1.1.2.cmml">𝐐</mi><mi id="S3.E4.m1.1.1.1.1.1.1.1.1.1.1.1.1.3" xref="S3.E4.m1.1.1.1.1.1.1.1.1.1.1.1.1.3.cmml">t</mi></msub><mo id="S3.E4.m1.1.1.1.1.1.1.1.1.1.2.2.4" xref="S3.E4.m1.1.1.1.1.1.1.1.1.1.2.3.cmml">,</mo><msub id="S3.E4.m1.1.1.1.1.1.1.1.1.1.2.2.2" xref="S3.E4.m1.1.1.1.1.1.1.1.1.1.2.2.2.cmml"><mi id="S3.E4.m1.1.1.1.1.1.1.1.1.1.2.2.2.2" xref="S3.E4.m1.1.1.1.1.1.1.1.1.1.2.2.2.2.cmml">𝐏</mi><mi id="S3.E4.m1.1.1.1.1.1.1.1.1.1.2.2.2.3" xref="S3.E4.m1.1.1.1.1.1.1.1.1.1.2.2.2.3.cmml">t</mi></msub><mo stretchy="false" id="S3.E4.m1.1.1.1.1.1.1.1.1.1.2.2.5" xref="S3.E4.m1.1.1.1.1.1.1.1.1.1.2.3.cmml">)</mo></mrow></mrow><mo id="S3.E4.m1.1.1.1.1.1.1.1.1.3" xref="S3.E4.m1.1.1.1.1.1.1.1.2.1.cmml">]</mo></mrow></mrow></mrow></mrow><mo lspace="0em" id="S3.E4.m1.1.1.1.2" xref="S3.E4.m1.1.1.1.1.cmml">.</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.E4.m1.1b"><apply id="S3.E4.m1.1.1.1.1.cmml" xref="S3.E4.m1.1.1.1"><eq id="S3.E4.m1.1.1.1.1.2.cmml" xref="S3.E4.m1.1.1.1.1.2"></eq><apply id="S3.E4.m1.1.1.1.1.3.cmml" xref="S3.E4.m1.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.E4.m1.1.1.1.1.3.1.cmml" xref="S3.E4.m1.1.1.1.1.3">subscript</csymbol><ci id="S3.E4.m1.1.1.1.1.3.2.cmml" xref="S3.E4.m1.1.1.1.1.3.2">ℒ</ci><ci id="S3.E4.m1.1.1.1.1.3.3a.cmml" xref="S3.E4.m1.1.1.1.1.3.3"><mtext mathsize="70%" id="S3.E4.m1.1.1.1.1.3.3.cmml" xref="S3.E4.m1.1.1.1.1.3.3">Fusion</mtext></ci></apply><apply id="S3.E4.m1.1.1.1.1.1.cmml" xref="S3.E4.m1.1.1.1.1.1"><minus id="S3.E4.m1.1.1.1.1.1.2.cmml" xref="S3.E4.m1.1.1.1.1.1"></minus><apply id="S3.E4.m1.1.1.1.1.1.1.cmml" xref="S3.E4.m1.1.1.1.1.1.1"><times id="S3.E4.m1.1.1.1.1.1.1.2.cmml" xref="S3.E4.m1.1.1.1.1.1.1.2"></times><apply id="S3.E4.m1.1.1.1.1.1.1.3.cmml" xref="S3.E4.m1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.E4.m1.1.1.1.1.1.1.3.1.cmml" xref="S3.E4.m1.1.1.1.1.1.1.3">subscript</csymbol><ci id="S3.E4.m1.1.1.1.1.1.1.3.2.cmml" xref="S3.E4.m1.1.1.1.1.1.1.3.2">𝔼</ci><apply id="S3.E4.m1.1.1.1.1.1.1.3.3.cmml" xref="S3.E4.m1.1.1.1.1.1.1.3.3"><csymbol cd="latexml" id="S3.E4.m1.1.1.1.1.1.1.3.3.1.cmml" xref="S3.E4.m1.1.1.1.1.1.1.3.3.1">similar-to</csymbol><ci id="S3.E4.m1.1.1.1.1.1.1.3.3.2.cmml" xref="S3.E4.m1.1.1.1.1.1.1.3.3.2">𝑡</ci><ci id="S3.E4.m1.1.1.1.1.1.1.3.3.3.cmml" xref="S3.E4.m1.1.1.1.1.1.1.3.3.3">𝒞</ci></apply></apply><apply id="S3.E4.m1.1.1.1.1.1.1.1.2.cmml" xref="S3.E4.m1.1.1.1.1.1.1.1.1"><csymbol cd="latexml" id="S3.E4.m1.1.1.1.1.1.1.1.2.1.cmml" xref="S3.E4.m1.1.1.1.1.1.1.1.1.2">delimited-[]</csymbol><apply id="S3.E4.m1.1.1.1.1.1.1.1.1.1.cmml" xref="S3.E4.m1.1.1.1.1.1.1.1.1.1"><times id="S3.E4.m1.1.1.1.1.1.1.1.1.1.3.cmml" xref="S3.E4.m1.1.1.1.1.1.1.1.1.1.3"></times><ci id="S3.E4.m1.1.1.1.1.1.1.1.1.1.4.cmml" xref="S3.E4.m1.1.1.1.1.1.1.1.1.1.4">𝔻</ci><interval closure="open" id="S3.E4.m1.1.1.1.1.1.1.1.1.1.2.3.cmml" xref="S3.E4.m1.1.1.1.1.1.1.1.1.1.2.2"><apply id="S3.E4.m1.1.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S3.E4.m1.1.1.1.1.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.E4.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S3.E4.m1.1.1.1.1.1.1.1.1.1.1.1.1">subscript</csymbol><ci id="S3.E4.m1.1.1.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="S3.E4.m1.1.1.1.1.1.1.1.1.1.1.1.1.2">𝐐</ci><ci id="S3.E4.m1.1.1.1.1.1.1.1.1.1.1.1.1.3.cmml" xref="S3.E4.m1.1.1.1.1.1.1.1.1.1.1.1.1.3">𝑡</ci></apply><apply id="S3.E4.m1.1.1.1.1.1.1.1.1.1.2.2.2.cmml" xref="S3.E4.m1.1.1.1.1.1.1.1.1.1.2.2.2"><csymbol cd="ambiguous" id="S3.E4.m1.1.1.1.1.1.1.1.1.1.2.2.2.1.cmml" xref="S3.E4.m1.1.1.1.1.1.1.1.1.1.2.2.2">subscript</csymbol><ci id="S3.E4.m1.1.1.1.1.1.1.1.1.1.2.2.2.2.cmml" xref="S3.E4.m1.1.1.1.1.1.1.1.1.1.2.2.2.2">𝐏</ci><ci id="S3.E4.m1.1.1.1.1.1.1.1.1.1.2.2.2.3.cmml" xref="S3.E4.m1.1.1.1.1.1.1.1.1.1.2.2.2.3">𝑡</ci></apply></interval></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E4.m1.1c">\mathcal{L}_{\text{Fusion}}=-\mathbb{E}_{t\sim\mathcal{C}}\left[\mathbb{D}(\mathbf{Q}_{t},\mathbf{P}_{t})\right].</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(4)</span></td>
</tr></tbody>
</table>
</div>
<div id="S3.SS2.p5" class="ltx_para ltx_noindent">
<p id="S3.SS2.p5.2" class="ltx_p">The overall objective for our continual training consists of a weighted combination of the causal language modeling objective <math id="S3.SS2.p5.1.m1.1" class="ltx_Math" alttext="\mathcal{L}_{\text{CLM}}" display="inline"><semantics id="S3.SS2.p5.1.m1.1a"><msub id="S3.SS2.p5.1.m1.1.1" xref="S3.SS2.p5.1.m1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS2.p5.1.m1.1.1.2" xref="S3.SS2.p5.1.m1.1.1.2.cmml">ℒ</mi><mtext id="S3.SS2.p5.1.m1.1.1.3" xref="S3.SS2.p5.1.m1.1.1.3a.cmml">CLM</mtext></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.p5.1.m1.1b"><apply id="S3.SS2.p5.1.m1.1.1.cmml" xref="S3.SS2.p5.1.m1.1.1"><csymbol cd="ambiguous" id="S3.SS2.p5.1.m1.1.1.1.cmml" xref="S3.SS2.p5.1.m1.1.1">subscript</csymbol><ci id="S3.SS2.p5.1.m1.1.1.2.cmml" xref="S3.SS2.p5.1.m1.1.1.2">ℒ</ci><ci id="S3.SS2.p5.1.m1.1.1.3a.cmml" xref="S3.SS2.p5.1.m1.1.1.3"><mtext mathsize="70%" id="S3.SS2.p5.1.m1.1.1.3.cmml" xref="S3.SS2.p5.1.m1.1.1.3">CLM</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p5.1.m1.1c">\mathcal{L}_{\text{CLM}}</annotation></semantics></math> and the fusion objective <math id="S3.SS2.p5.2.m2.1" class="ltx_Math" alttext="\mathcal{L}_{\text{Fusion}}" display="inline"><semantics id="S3.SS2.p5.2.m2.1a"><msub id="S3.SS2.p5.2.m2.1.1" xref="S3.SS2.p5.2.m2.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS2.p5.2.m2.1.1.2" xref="S3.SS2.p5.2.m2.1.1.2.cmml">ℒ</mi><mtext id="S3.SS2.p5.2.m2.1.1.3" xref="S3.SS2.p5.2.m2.1.1.3a.cmml">Fusion</mtext></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.p5.2.m2.1b"><apply id="S3.SS2.p5.2.m2.1.1.cmml" xref="S3.SS2.p5.2.m2.1.1"><csymbol cd="ambiguous" id="S3.SS2.p5.2.m2.1.1.1.cmml" xref="S3.SS2.p5.2.m2.1.1">subscript</csymbol><ci id="S3.SS2.p5.2.m2.1.1.2.cmml" xref="S3.SS2.p5.2.m2.1.1.2">ℒ</ci><ci id="S3.SS2.p5.2.m2.1.1.3a.cmml" xref="S3.SS2.p5.2.m2.1.1.3"><mtext mathsize="70%" id="S3.SS2.p5.2.m2.1.1.3.cmml" xref="S3.SS2.p5.2.m2.1.1.3">Fusion</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p5.2.m2.1c">\mathcal{L}_{\text{Fusion}}</annotation></semantics></math> as follows:</p>
<table id="S3.E5" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S3.E5.m1.1" class="ltx_Math" alttext="\mathcal{L}=\lambda\mathcal{L}_{\text{CLM}}+(1-\lambda)\mathcal{L}_{\text{Fusion}}." display="block"><semantics id="S3.E5.m1.1a"><mrow id="S3.E5.m1.1.1.1" xref="S3.E5.m1.1.1.1.1.cmml"><mrow id="S3.E5.m1.1.1.1.1" xref="S3.E5.m1.1.1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.E5.m1.1.1.1.1.3" xref="S3.E5.m1.1.1.1.1.3.cmml">ℒ</mi><mo id="S3.E5.m1.1.1.1.1.2" xref="S3.E5.m1.1.1.1.1.2.cmml">=</mo><mrow id="S3.E5.m1.1.1.1.1.1" xref="S3.E5.m1.1.1.1.1.1.cmml"><mrow id="S3.E5.m1.1.1.1.1.1.3" xref="S3.E5.m1.1.1.1.1.1.3.cmml"><mi id="S3.E5.m1.1.1.1.1.1.3.2" xref="S3.E5.m1.1.1.1.1.1.3.2.cmml">λ</mi><mo lspace="0em" rspace="0em" id="S3.E5.m1.1.1.1.1.1.3.1" xref="S3.E5.m1.1.1.1.1.1.3.1.cmml">​</mo><msub id="S3.E5.m1.1.1.1.1.1.3.3" xref="S3.E5.m1.1.1.1.1.1.3.3.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.E5.m1.1.1.1.1.1.3.3.2" xref="S3.E5.m1.1.1.1.1.1.3.3.2.cmml">ℒ</mi><mtext id="S3.E5.m1.1.1.1.1.1.3.3.3" xref="S3.E5.m1.1.1.1.1.1.3.3.3a.cmml">CLM</mtext></msub></mrow><mo id="S3.E5.m1.1.1.1.1.1.2" xref="S3.E5.m1.1.1.1.1.1.2.cmml">+</mo><mrow id="S3.E5.m1.1.1.1.1.1.1" xref="S3.E5.m1.1.1.1.1.1.1.cmml"><mrow id="S3.E5.m1.1.1.1.1.1.1.1.1" xref="S3.E5.m1.1.1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.E5.m1.1.1.1.1.1.1.1.1.2" xref="S3.E5.m1.1.1.1.1.1.1.1.1.1.cmml">(</mo><mrow id="S3.E5.m1.1.1.1.1.1.1.1.1.1" xref="S3.E5.m1.1.1.1.1.1.1.1.1.1.cmml"><mn id="S3.E5.m1.1.1.1.1.1.1.1.1.1.2" xref="S3.E5.m1.1.1.1.1.1.1.1.1.1.2.cmml">1</mn><mo id="S3.E5.m1.1.1.1.1.1.1.1.1.1.1" xref="S3.E5.m1.1.1.1.1.1.1.1.1.1.1.cmml">−</mo><mi id="S3.E5.m1.1.1.1.1.1.1.1.1.1.3" xref="S3.E5.m1.1.1.1.1.1.1.1.1.1.3.cmml">λ</mi></mrow><mo stretchy="false" id="S3.E5.m1.1.1.1.1.1.1.1.1.3" xref="S3.E5.m1.1.1.1.1.1.1.1.1.1.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="S3.E5.m1.1.1.1.1.1.1.2" xref="S3.E5.m1.1.1.1.1.1.1.2.cmml">​</mo><msub id="S3.E5.m1.1.1.1.1.1.1.3" xref="S3.E5.m1.1.1.1.1.1.1.3.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.E5.m1.1.1.1.1.1.1.3.2" xref="S3.E5.m1.1.1.1.1.1.1.3.2.cmml">ℒ</mi><mtext id="S3.E5.m1.1.1.1.1.1.1.3.3" xref="S3.E5.m1.1.1.1.1.1.1.3.3a.cmml">Fusion</mtext></msub></mrow></mrow></mrow><mo lspace="0em" id="S3.E5.m1.1.1.1.2" xref="S3.E5.m1.1.1.1.1.cmml">.</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.E5.m1.1b"><apply id="S3.E5.m1.1.1.1.1.cmml" xref="S3.E5.m1.1.1.1"><eq id="S3.E5.m1.1.1.1.1.2.cmml" xref="S3.E5.m1.1.1.1.1.2"></eq><ci id="S3.E5.m1.1.1.1.1.3.cmml" xref="S3.E5.m1.1.1.1.1.3">ℒ</ci><apply id="S3.E5.m1.1.1.1.1.1.cmml" xref="S3.E5.m1.1.1.1.1.1"><plus id="S3.E5.m1.1.1.1.1.1.2.cmml" xref="S3.E5.m1.1.1.1.1.1.2"></plus><apply id="S3.E5.m1.1.1.1.1.1.3.cmml" xref="S3.E5.m1.1.1.1.1.1.3"><times id="S3.E5.m1.1.1.1.1.1.3.1.cmml" xref="S3.E5.m1.1.1.1.1.1.3.1"></times><ci id="S3.E5.m1.1.1.1.1.1.3.2.cmml" xref="S3.E5.m1.1.1.1.1.1.3.2">𝜆</ci><apply id="S3.E5.m1.1.1.1.1.1.3.3.cmml" xref="S3.E5.m1.1.1.1.1.1.3.3"><csymbol cd="ambiguous" id="S3.E5.m1.1.1.1.1.1.3.3.1.cmml" xref="S3.E5.m1.1.1.1.1.1.3.3">subscript</csymbol><ci id="S3.E5.m1.1.1.1.1.1.3.3.2.cmml" xref="S3.E5.m1.1.1.1.1.1.3.3.2">ℒ</ci><ci id="S3.E5.m1.1.1.1.1.1.3.3.3a.cmml" xref="S3.E5.m1.1.1.1.1.1.3.3.3"><mtext mathsize="70%" id="S3.E5.m1.1.1.1.1.1.3.3.3.cmml" xref="S3.E5.m1.1.1.1.1.1.3.3.3">CLM</mtext></ci></apply></apply><apply id="S3.E5.m1.1.1.1.1.1.1.cmml" xref="S3.E5.m1.1.1.1.1.1.1"><times id="S3.E5.m1.1.1.1.1.1.1.2.cmml" xref="S3.E5.m1.1.1.1.1.1.1.2"></times><apply id="S3.E5.m1.1.1.1.1.1.1.1.1.1.cmml" xref="S3.E5.m1.1.1.1.1.1.1.1.1"><minus id="S3.E5.m1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S3.E5.m1.1.1.1.1.1.1.1.1.1.1"></minus><cn type="integer" id="S3.E5.m1.1.1.1.1.1.1.1.1.1.2.cmml" xref="S3.E5.m1.1.1.1.1.1.1.1.1.1.2">1</cn><ci id="S3.E5.m1.1.1.1.1.1.1.1.1.1.3.cmml" xref="S3.E5.m1.1.1.1.1.1.1.1.1.1.3">𝜆</ci></apply><apply id="S3.E5.m1.1.1.1.1.1.1.3.cmml" xref="S3.E5.m1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.E5.m1.1.1.1.1.1.1.3.1.cmml" xref="S3.E5.m1.1.1.1.1.1.1.3">subscript</csymbol><ci id="S3.E5.m1.1.1.1.1.1.1.3.2.cmml" xref="S3.E5.m1.1.1.1.1.1.1.3.2">ℒ</ci><ci id="S3.E5.m1.1.1.1.1.1.1.3.3a.cmml" xref="S3.E5.m1.1.1.1.1.1.1.3.3"><mtext mathsize="70%" id="S3.E5.m1.1.1.1.1.1.1.3.3.cmml" xref="S3.E5.m1.1.1.1.1.1.1.3.3">Fusion</mtext></ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E5.m1.1c">\mathcal{L}=\lambda\mathcal{L}_{\text{CLM}}+(1-\lambda)\mathcal{L}_{\text{Fusion}}.</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(5)</span></td>
</tr></tbody>
</table>
</div>
</section>
<section id="S3.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.3 </span>Implementation of <span id="S3.SS3.1.1" class="ltx_text ltx_font_smallcaps">FuseLLM</span>
</h3>

<div id="S3.SS3.p1" class="ltx_para ltx_noindent">
<p id="S3.SS3.p1.1" class="ltx_p">In this section, we present the implementation details of token alignment and the fusion function for fusing different LLMs in our <span id="S3.SS3.p1.1.1" class="ltx_text ltx_font_smallcaps">FuseLLM</span> method.</p>
</div>
<section id="S3.SS3.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Token Alignment</h4>

<div id="S3.SS3.SSS0.Px1.p1" class="ltx_para ltx_noindent">
<p id="S3.SS3.SSS0.Px1.p1.1" class="ltx_p">Ensuring token alignment across multiple LLMs is crucial for effective knowledge fusion, as it guarantees proper mapping of probabilistic distribution matrices.
<cite class="ltx_cite ltx_citemacro_cite">Fu et al. (<a href="#bib.bib17" title="" class="ltx_ref">2023</a>)</cite> employed dynamic programming to recursively minimize the total cost of editing one sequence of tokens to match the other.
If a one-to-one mapping exists between two tokens, the corresponding distributions are perfectly mapped. Otherwise, the mapped distribution degenerates into a one-hot vector.
Since tokens generated by different tokenizers for the same sequence typically exhibit limited differences, we propose to enhance the success rate of token alignment by replacing the exact match (EM) constraint in <cite class="ltx_cite ltx_citemacro_cite">Fu et al. (<a href="#bib.bib17" title="" class="ltx_ref">2023</a>)</cite> with a minimum edit distance (MinED) strategy, which maps tokens from different tokenizers based on MinED.
This relaxation of token alignment helps preserve substantial information in the distribution matrices while introducing minor errors. For more details of the token alignment, please refer to Appendix <a href="#A1" title="Appendix A Details of Token Alignment ‣ Knowledge Fusion of Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">A</span></a>.</p>
</div>
</section>
<section id="S3.SS3.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Fusion Strategies</h4>

<div id="S3.SS3.SSS0.Px2.p1" class="ltx_para ltx_noindent">
<p id="S3.SS3.SSS0.Px2.p1.1" class="ltx_p">To combine the collective knowledge of source LLMs while preserving their unique strengths, it is essential to evaluate the quality of different LLMs and assign varying levels of importance to their respective distribution matrices. For this purpose, when dealing with text <math id="S3.SS3.SSS0.Px2.p1.1.m1.1" class="ltx_Math" alttext="t" display="inline"><semantics id="S3.SS3.SSS0.Px2.p1.1.m1.1a"><mi id="S3.SS3.SSS0.Px2.p1.1.m1.1.1" xref="S3.SS3.SSS0.Px2.p1.1.m1.1.1.cmml">t</mi><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px2.p1.1.m1.1b"><ci id="S3.SS3.SSS0.Px2.p1.1.m1.1.1.cmml" xref="S3.SS3.SSS0.Px2.p1.1.m1.1.1">𝑡</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px2.p1.1.m1.1c">t</annotation></semantics></math>, we utilize the cross-entropy loss between the distribution matrices and the gold labels as an indicator of the prediction quality of the LLMs <cite class="ltx_cite ltx_citemacro_citep">(Marion et al., <a href="#bib.bib36" title="" class="ltx_ref">2023</a>)</cite>. A lower cross-entropy score for a source LLM signifies a more accurate understanding of the text, and its prediction should be accorded greater significance. Based on this criterion, we introduce two fusion functions: (1) MinCE: This function outputs the distribution matrix with the minimum cross-entropy score; (2) AvgCE: This function produces a weighted average of the distribution matrices based on cross-entropy scores.</p>
</div>
<div id="S3.SS3.SSS0.Px2.p2" class="ltx_para ltx_noindent">
<p id="S3.SS3.SSS0.Px2.p2.1" class="ltx_p">The complete process of the <span id="S3.SS3.SSS0.Px2.p2.1.1" class="ltx_text ltx_font_smallcaps">FuseLLM</span> method is described in Algorithm <a href="#alg1" title="Algorithm 1 ‣ Fusion Strategies ‣ 3.3 Implementation of FuseLLM ‣ 3 Knowledge Fusion of LLMs ‣ Knowledge Fusion of Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>.</p>
</div>
<figure id="alg1" class="ltx_float ltx_float_algorithm ltx_framed ltx_framed_top">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_float"><span id="alg1.3.1.1" class="ltx_text ltx_font_bold">Algorithm 1</span> </span> <span id="alg1.4.2" class="ltx_text ltx_font_smallcaps">FuseLLM</span> for LLMs Fusion</figcaption>
<div id="alg1.5" class="ltx_listing ltx_listing">
<div id="alg1.l0" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">0:</span>  Source LLMs <math id="alg1.l0.m1.1" class="ltx_Math" alttext="\{\mathcal{M}^{s}_{j}\}_{j=1}^{K}" display="inline"><semantics id="alg1.l0.m1.1a"><msubsup id="alg1.l0.m1.1.1" xref="alg1.l0.m1.1.1.cmml"><mrow id="alg1.l0.m1.1.1.1.1.1" xref="alg1.l0.m1.1.1.1.1.2.cmml"><mo stretchy="false" id="alg1.l0.m1.1.1.1.1.1.2" xref="alg1.l0.m1.1.1.1.1.2.cmml">{</mo><msubsup id="alg1.l0.m1.1.1.1.1.1.1" xref="alg1.l0.m1.1.1.1.1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="alg1.l0.m1.1.1.1.1.1.1.2.2" xref="alg1.l0.m1.1.1.1.1.1.1.2.2.cmml">ℳ</mi><mi id="alg1.l0.m1.1.1.1.1.1.1.3" xref="alg1.l0.m1.1.1.1.1.1.1.3.cmml">j</mi><mi id="alg1.l0.m1.1.1.1.1.1.1.2.3" xref="alg1.l0.m1.1.1.1.1.1.1.2.3.cmml">s</mi></msubsup><mo stretchy="false" id="alg1.l0.m1.1.1.1.1.1.3" xref="alg1.l0.m1.1.1.1.1.2.cmml">}</mo></mrow><mrow id="alg1.l0.m1.1.1.1.3" xref="alg1.l0.m1.1.1.1.3.cmml"><mi id="alg1.l0.m1.1.1.1.3.2" xref="alg1.l0.m1.1.1.1.3.2.cmml">j</mi><mo id="alg1.l0.m1.1.1.1.3.1" xref="alg1.l0.m1.1.1.1.3.1.cmml">=</mo><mn id="alg1.l0.m1.1.1.1.3.3" xref="alg1.l0.m1.1.1.1.3.3.cmml">1</mn></mrow><mi id="alg1.l0.m1.1.1.3" xref="alg1.l0.m1.1.1.3.cmml">K</mi></msubsup><annotation-xml encoding="MathML-Content" id="alg1.l0.m1.1b"><apply id="alg1.l0.m1.1.1.cmml" xref="alg1.l0.m1.1.1"><csymbol cd="ambiguous" id="alg1.l0.m1.1.1.2.cmml" xref="alg1.l0.m1.1.1">superscript</csymbol><apply id="alg1.l0.m1.1.1.1.cmml" xref="alg1.l0.m1.1.1"><csymbol cd="ambiguous" id="alg1.l0.m1.1.1.1.2.cmml" xref="alg1.l0.m1.1.1">subscript</csymbol><set id="alg1.l0.m1.1.1.1.1.2.cmml" xref="alg1.l0.m1.1.1.1.1.1"><apply id="alg1.l0.m1.1.1.1.1.1.1.cmml" xref="alg1.l0.m1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="alg1.l0.m1.1.1.1.1.1.1.1.cmml" xref="alg1.l0.m1.1.1.1.1.1.1">subscript</csymbol><apply id="alg1.l0.m1.1.1.1.1.1.1.2.cmml" xref="alg1.l0.m1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="alg1.l0.m1.1.1.1.1.1.1.2.1.cmml" xref="alg1.l0.m1.1.1.1.1.1.1">superscript</csymbol><ci id="alg1.l0.m1.1.1.1.1.1.1.2.2.cmml" xref="alg1.l0.m1.1.1.1.1.1.1.2.2">ℳ</ci><ci id="alg1.l0.m1.1.1.1.1.1.1.2.3.cmml" xref="alg1.l0.m1.1.1.1.1.1.1.2.3">𝑠</ci></apply><ci id="alg1.l0.m1.1.1.1.1.1.1.3.cmml" xref="alg1.l0.m1.1.1.1.1.1.1.3">𝑗</ci></apply></set><apply id="alg1.l0.m1.1.1.1.3.cmml" xref="alg1.l0.m1.1.1.1.3"><eq id="alg1.l0.m1.1.1.1.3.1.cmml" xref="alg1.l0.m1.1.1.1.3.1"></eq><ci id="alg1.l0.m1.1.1.1.3.2.cmml" xref="alg1.l0.m1.1.1.1.3.2">𝑗</ci><cn type="integer" id="alg1.l0.m1.1.1.1.3.3.cmml" xref="alg1.l0.m1.1.1.1.3.3">1</cn></apply></apply><ci id="alg1.l0.m1.1.1.3.cmml" xref="alg1.l0.m1.1.1.3">𝐾</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l0.m1.1c">\{\mathcal{M}^{s}_{j}\}_{j=1}^{K}</annotation></semantics></math>, training corpus <math id="alg1.l0.m2.1" class="ltx_Math" alttext="\mathcal{C}" display="inline"><semantics id="alg1.l0.m2.1a"><mi class="ltx_font_mathcaligraphic" id="alg1.l0.m2.1.1" xref="alg1.l0.m2.1.1.cmml">𝒞</mi><annotation-xml encoding="MathML-Content" id="alg1.l0.m2.1b"><ci id="alg1.l0.m2.1.1.cmml" xref="alg1.l0.m2.1.1">𝒞</ci></annotation-xml><annotation encoding="application/x-tex" id="alg1.l0.m2.1c">\mathcal{C}</annotation></semantics></math>.

</div>
<div id="alg1.l1" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">1:</span>  Initialize the target LLM <math id="alg1.l1.m1.1" class="ltx_Math" alttext="\mathcal{M}^{t}" display="inline"><semantics id="alg1.l1.m1.1a"><msup id="alg1.l1.m1.1.1" xref="alg1.l1.m1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="alg1.l1.m1.1.1.2" xref="alg1.l1.m1.1.1.2.cmml">ℳ</mi><mi id="alg1.l1.m1.1.1.3" xref="alg1.l1.m1.1.1.3.cmml">t</mi></msup><annotation-xml encoding="MathML-Content" id="alg1.l1.m1.1b"><apply id="alg1.l1.m1.1.1.cmml" xref="alg1.l1.m1.1.1"><csymbol cd="ambiguous" id="alg1.l1.m1.1.1.1.cmml" xref="alg1.l1.m1.1.1">superscript</csymbol><ci id="alg1.l1.m1.1.1.2.cmml" xref="alg1.l1.m1.1.1.2">ℳ</ci><ci id="alg1.l1.m1.1.1.3.cmml" xref="alg1.l1.m1.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l1.m1.1c">\mathcal{M}^{t}</annotation></semantics></math> with one of the source LLMs.

</div>
<div id="alg1.l2" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">2:</span>  <span id="alg1.l2.1" class="ltx_text ltx_font_bold">for</span> text <math id="alg1.l2.m1.1" class="ltx_Math" alttext="t" display="inline"><semantics id="alg1.l2.m1.1a"><mi id="alg1.l2.m1.1.1" xref="alg1.l2.m1.1.1.cmml">t</mi><annotation-xml encoding="MathML-Content" id="alg1.l2.m1.1b"><ci id="alg1.l2.m1.1.1.cmml" xref="alg1.l2.m1.1.1">𝑡</ci></annotation-xml><annotation encoding="application/x-tex" id="alg1.l2.m1.1c">t</annotation></semantics></math> in <math id="alg1.l2.m2.1" class="ltx_Math" alttext="\mathcal{C}" display="inline"><semantics id="alg1.l2.m2.1a"><mi class="ltx_font_mathcaligraphic" id="alg1.l2.m2.1.1" xref="alg1.l2.m2.1.1.cmml">𝒞</mi><annotation-xml encoding="MathML-Content" id="alg1.l2.m2.1b"><ci id="alg1.l2.m2.1.1.cmml" xref="alg1.l2.m2.1.1">𝒞</ci></annotation-xml><annotation encoding="application/x-tex" id="alg1.l2.m2.1c">\mathcal{C}</annotation></semantics></math> <span id="alg1.l2.2" class="ltx_text ltx_font_bold">do</span>



</div>
<div id="alg1.l3" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">3:</span>     Apply the <math id="alg1.l3.m1.1" class="ltx_Math" alttext="K" display="inline"><semantics id="alg1.l3.m1.1a"><mi id="alg1.l3.m1.1.1" xref="alg1.l3.m1.1.1.cmml">K</mi><annotation-xml encoding="MathML-Content" id="alg1.l3.m1.1b"><ci id="alg1.l3.m1.1.1.cmml" xref="alg1.l3.m1.1.1">𝐾</ci></annotation-xml><annotation encoding="application/x-tex" id="alg1.l3.m1.1c">K</annotation></semantics></math> source LLMs to compute probabilistic distribution matrices <math id="alg1.l3.m2.1" class="ltx_Math" alttext="\{\mathbf{P}_{t}^{\theta_{j}}\}_{j=1}^{K}" display="inline"><semantics id="alg1.l3.m2.1a"><msubsup id="alg1.l3.m2.1.1" xref="alg1.l3.m2.1.1.cmml"><mrow id="alg1.l3.m2.1.1.1.1.1" xref="alg1.l3.m2.1.1.1.1.2.cmml"><mo stretchy="false" id="alg1.l3.m2.1.1.1.1.1.2" xref="alg1.l3.m2.1.1.1.1.2.cmml">{</mo><msubsup id="alg1.l3.m2.1.1.1.1.1.1" xref="alg1.l3.m2.1.1.1.1.1.1.cmml"><mi id="alg1.l3.m2.1.1.1.1.1.1.2.2" xref="alg1.l3.m2.1.1.1.1.1.1.2.2.cmml">𝐏</mi><mi id="alg1.l3.m2.1.1.1.1.1.1.2.3" xref="alg1.l3.m2.1.1.1.1.1.1.2.3.cmml">t</mi><msub id="alg1.l3.m2.1.1.1.1.1.1.3" xref="alg1.l3.m2.1.1.1.1.1.1.3.cmml"><mi id="alg1.l3.m2.1.1.1.1.1.1.3.2" xref="alg1.l3.m2.1.1.1.1.1.1.3.2.cmml">θ</mi><mi id="alg1.l3.m2.1.1.1.1.1.1.3.3" xref="alg1.l3.m2.1.1.1.1.1.1.3.3.cmml">j</mi></msub></msubsup><mo stretchy="false" id="alg1.l3.m2.1.1.1.1.1.3" xref="alg1.l3.m2.1.1.1.1.2.cmml">}</mo></mrow><mrow id="alg1.l3.m2.1.1.1.3" xref="alg1.l3.m2.1.1.1.3.cmml"><mi id="alg1.l3.m2.1.1.1.3.2" xref="alg1.l3.m2.1.1.1.3.2.cmml">j</mi><mo id="alg1.l3.m2.1.1.1.3.1" xref="alg1.l3.m2.1.1.1.3.1.cmml">=</mo><mn id="alg1.l3.m2.1.1.1.3.3" xref="alg1.l3.m2.1.1.1.3.3.cmml">1</mn></mrow><mi id="alg1.l3.m2.1.1.3" xref="alg1.l3.m2.1.1.3.cmml">K</mi></msubsup><annotation-xml encoding="MathML-Content" id="alg1.l3.m2.1b"><apply id="alg1.l3.m2.1.1.cmml" xref="alg1.l3.m2.1.1"><csymbol cd="ambiguous" id="alg1.l3.m2.1.1.2.cmml" xref="alg1.l3.m2.1.1">superscript</csymbol><apply id="alg1.l3.m2.1.1.1.cmml" xref="alg1.l3.m2.1.1"><csymbol cd="ambiguous" id="alg1.l3.m2.1.1.1.2.cmml" xref="alg1.l3.m2.1.1">subscript</csymbol><set id="alg1.l3.m2.1.1.1.1.2.cmml" xref="alg1.l3.m2.1.1.1.1.1"><apply id="alg1.l3.m2.1.1.1.1.1.1.cmml" xref="alg1.l3.m2.1.1.1.1.1.1"><csymbol cd="ambiguous" id="alg1.l3.m2.1.1.1.1.1.1.1.cmml" xref="alg1.l3.m2.1.1.1.1.1.1">superscript</csymbol><apply id="alg1.l3.m2.1.1.1.1.1.1.2.cmml" xref="alg1.l3.m2.1.1.1.1.1.1"><csymbol cd="ambiguous" id="alg1.l3.m2.1.1.1.1.1.1.2.1.cmml" xref="alg1.l3.m2.1.1.1.1.1.1">subscript</csymbol><ci id="alg1.l3.m2.1.1.1.1.1.1.2.2.cmml" xref="alg1.l3.m2.1.1.1.1.1.1.2.2">𝐏</ci><ci id="alg1.l3.m2.1.1.1.1.1.1.2.3.cmml" xref="alg1.l3.m2.1.1.1.1.1.1.2.3">𝑡</ci></apply><apply id="alg1.l3.m2.1.1.1.1.1.1.3.cmml" xref="alg1.l3.m2.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="alg1.l3.m2.1.1.1.1.1.1.3.1.cmml" xref="alg1.l3.m2.1.1.1.1.1.1.3">subscript</csymbol><ci id="alg1.l3.m2.1.1.1.1.1.1.3.2.cmml" xref="alg1.l3.m2.1.1.1.1.1.1.3.2">𝜃</ci><ci id="alg1.l3.m2.1.1.1.1.1.1.3.3.cmml" xref="alg1.l3.m2.1.1.1.1.1.1.3.3">𝑗</ci></apply></apply></set><apply id="alg1.l3.m2.1.1.1.3.cmml" xref="alg1.l3.m2.1.1.1.3"><eq id="alg1.l3.m2.1.1.1.3.1.cmml" xref="alg1.l3.m2.1.1.1.3.1"></eq><ci id="alg1.l3.m2.1.1.1.3.2.cmml" xref="alg1.l3.m2.1.1.1.3.2">𝑗</ci><cn type="integer" id="alg1.l3.m2.1.1.1.3.3.cmml" xref="alg1.l3.m2.1.1.1.3.3">1</cn></apply></apply><ci id="alg1.l3.m2.1.1.3.cmml" xref="alg1.l3.m2.1.1.3">𝐾</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l3.m2.1c">\{\mathbf{P}_{t}^{\theta_{j}}\}_{j=1}^{K}</annotation></semantics></math>.

</div>
<div id="alg1.l4" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">4:</span>     Align <math id="alg1.l4.m1.1" class="ltx_Math" alttext="\{\mathbf{P}_{t}^{\theta_{j}}\}_{j=1}^{K}" display="inline"><semantics id="alg1.l4.m1.1a"><msubsup id="alg1.l4.m1.1.1" xref="alg1.l4.m1.1.1.cmml"><mrow id="alg1.l4.m1.1.1.1.1.1" xref="alg1.l4.m1.1.1.1.1.2.cmml"><mo stretchy="false" id="alg1.l4.m1.1.1.1.1.1.2" xref="alg1.l4.m1.1.1.1.1.2.cmml">{</mo><msubsup id="alg1.l4.m1.1.1.1.1.1.1" xref="alg1.l4.m1.1.1.1.1.1.1.cmml"><mi id="alg1.l4.m1.1.1.1.1.1.1.2.2" xref="alg1.l4.m1.1.1.1.1.1.1.2.2.cmml">𝐏</mi><mi id="alg1.l4.m1.1.1.1.1.1.1.2.3" xref="alg1.l4.m1.1.1.1.1.1.1.2.3.cmml">t</mi><msub id="alg1.l4.m1.1.1.1.1.1.1.3" xref="alg1.l4.m1.1.1.1.1.1.1.3.cmml"><mi id="alg1.l4.m1.1.1.1.1.1.1.3.2" xref="alg1.l4.m1.1.1.1.1.1.1.3.2.cmml">θ</mi><mi id="alg1.l4.m1.1.1.1.1.1.1.3.3" xref="alg1.l4.m1.1.1.1.1.1.1.3.3.cmml">j</mi></msub></msubsup><mo stretchy="false" id="alg1.l4.m1.1.1.1.1.1.3" xref="alg1.l4.m1.1.1.1.1.2.cmml">}</mo></mrow><mrow id="alg1.l4.m1.1.1.1.3" xref="alg1.l4.m1.1.1.1.3.cmml"><mi id="alg1.l4.m1.1.1.1.3.2" xref="alg1.l4.m1.1.1.1.3.2.cmml">j</mi><mo id="alg1.l4.m1.1.1.1.3.1" xref="alg1.l4.m1.1.1.1.3.1.cmml">=</mo><mn id="alg1.l4.m1.1.1.1.3.3" xref="alg1.l4.m1.1.1.1.3.3.cmml">1</mn></mrow><mi id="alg1.l4.m1.1.1.3" xref="alg1.l4.m1.1.1.3.cmml">K</mi></msubsup><annotation-xml encoding="MathML-Content" id="alg1.l4.m1.1b"><apply id="alg1.l4.m1.1.1.cmml" xref="alg1.l4.m1.1.1"><csymbol cd="ambiguous" id="alg1.l4.m1.1.1.2.cmml" xref="alg1.l4.m1.1.1">superscript</csymbol><apply id="alg1.l4.m1.1.1.1.cmml" xref="alg1.l4.m1.1.1"><csymbol cd="ambiguous" id="alg1.l4.m1.1.1.1.2.cmml" xref="alg1.l4.m1.1.1">subscript</csymbol><set id="alg1.l4.m1.1.1.1.1.2.cmml" xref="alg1.l4.m1.1.1.1.1.1"><apply id="alg1.l4.m1.1.1.1.1.1.1.cmml" xref="alg1.l4.m1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="alg1.l4.m1.1.1.1.1.1.1.1.cmml" xref="alg1.l4.m1.1.1.1.1.1.1">superscript</csymbol><apply id="alg1.l4.m1.1.1.1.1.1.1.2.cmml" xref="alg1.l4.m1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="alg1.l4.m1.1.1.1.1.1.1.2.1.cmml" xref="alg1.l4.m1.1.1.1.1.1.1">subscript</csymbol><ci id="alg1.l4.m1.1.1.1.1.1.1.2.2.cmml" xref="alg1.l4.m1.1.1.1.1.1.1.2.2">𝐏</ci><ci id="alg1.l4.m1.1.1.1.1.1.1.2.3.cmml" xref="alg1.l4.m1.1.1.1.1.1.1.2.3">𝑡</ci></apply><apply id="alg1.l4.m1.1.1.1.1.1.1.3.cmml" xref="alg1.l4.m1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="alg1.l4.m1.1.1.1.1.1.1.3.1.cmml" xref="alg1.l4.m1.1.1.1.1.1.1.3">subscript</csymbol><ci id="alg1.l4.m1.1.1.1.1.1.1.3.2.cmml" xref="alg1.l4.m1.1.1.1.1.1.1.3.2">𝜃</ci><ci id="alg1.l4.m1.1.1.1.1.1.1.3.3.cmml" xref="alg1.l4.m1.1.1.1.1.1.1.3.3">𝑗</ci></apply></apply></set><apply id="alg1.l4.m1.1.1.1.3.cmml" xref="alg1.l4.m1.1.1.1.3"><eq id="alg1.l4.m1.1.1.1.3.1.cmml" xref="alg1.l4.m1.1.1.1.3.1"></eq><ci id="alg1.l4.m1.1.1.1.3.2.cmml" xref="alg1.l4.m1.1.1.1.3.2">𝑗</ci><cn type="integer" id="alg1.l4.m1.1.1.1.3.3.cmml" xref="alg1.l4.m1.1.1.1.3.3">1</cn></apply></apply><ci id="alg1.l4.m1.1.1.3.cmml" xref="alg1.l4.m1.1.1.3">𝐾</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l4.m1.1c">\{\mathbf{P}_{t}^{\theta_{j}}\}_{j=1}^{K}</annotation></semantics></math> using the MinED alignment method.

</div>
<div id="alg1.l5" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">5:</span>     Fuse <math id="alg1.l5.m1.1" class="ltx_Math" alttext="\{\mathbf{P}_{t}^{\theta_{j}}\}_{j=1}^{K}" display="inline"><semantics id="alg1.l5.m1.1a"><msubsup id="alg1.l5.m1.1.1" xref="alg1.l5.m1.1.1.cmml"><mrow id="alg1.l5.m1.1.1.1.1.1" xref="alg1.l5.m1.1.1.1.1.2.cmml"><mo stretchy="false" id="alg1.l5.m1.1.1.1.1.1.2" xref="alg1.l5.m1.1.1.1.1.2.cmml">{</mo><msubsup id="alg1.l5.m1.1.1.1.1.1.1" xref="alg1.l5.m1.1.1.1.1.1.1.cmml"><mi id="alg1.l5.m1.1.1.1.1.1.1.2.2" xref="alg1.l5.m1.1.1.1.1.1.1.2.2.cmml">𝐏</mi><mi id="alg1.l5.m1.1.1.1.1.1.1.2.3" xref="alg1.l5.m1.1.1.1.1.1.1.2.3.cmml">t</mi><msub id="alg1.l5.m1.1.1.1.1.1.1.3" xref="alg1.l5.m1.1.1.1.1.1.1.3.cmml"><mi id="alg1.l5.m1.1.1.1.1.1.1.3.2" xref="alg1.l5.m1.1.1.1.1.1.1.3.2.cmml">θ</mi><mi id="alg1.l5.m1.1.1.1.1.1.1.3.3" xref="alg1.l5.m1.1.1.1.1.1.1.3.3.cmml">j</mi></msub></msubsup><mo stretchy="false" id="alg1.l5.m1.1.1.1.1.1.3" xref="alg1.l5.m1.1.1.1.1.2.cmml">}</mo></mrow><mrow id="alg1.l5.m1.1.1.1.3" xref="alg1.l5.m1.1.1.1.3.cmml"><mi id="alg1.l5.m1.1.1.1.3.2" xref="alg1.l5.m1.1.1.1.3.2.cmml">j</mi><mo id="alg1.l5.m1.1.1.1.3.1" xref="alg1.l5.m1.1.1.1.3.1.cmml">=</mo><mn id="alg1.l5.m1.1.1.1.3.3" xref="alg1.l5.m1.1.1.1.3.3.cmml">1</mn></mrow><mi id="alg1.l5.m1.1.1.3" xref="alg1.l5.m1.1.1.3.cmml">K</mi></msubsup><annotation-xml encoding="MathML-Content" id="alg1.l5.m1.1b"><apply id="alg1.l5.m1.1.1.cmml" xref="alg1.l5.m1.1.1"><csymbol cd="ambiguous" id="alg1.l5.m1.1.1.2.cmml" xref="alg1.l5.m1.1.1">superscript</csymbol><apply id="alg1.l5.m1.1.1.1.cmml" xref="alg1.l5.m1.1.1"><csymbol cd="ambiguous" id="alg1.l5.m1.1.1.1.2.cmml" xref="alg1.l5.m1.1.1">subscript</csymbol><set id="alg1.l5.m1.1.1.1.1.2.cmml" xref="alg1.l5.m1.1.1.1.1.1"><apply id="alg1.l5.m1.1.1.1.1.1.1.cmml" xref="alg1.l5.m1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="alg1.l5.m1.1.1.1.1.1.1.1.cmml" xref="alg1.l5.m1.1.1.1.1.1.1">superscript</csymbol><apply id="alg1.l5.m1.1.1.1.1.1.1.2.cmml" xref="alg1.l5.m1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="alg1.l5.m1.1.1.1.1.1.1.2.1.cmml" xref="alg1.l5.m1.1.1.1.1.1.1">subscript</csymbol><ci id="alg1.l5.m1.1.1.1.1.1.1.2.2.cmml" xref="alg1.l5.m1.1.1.1.1.1.1.2.2">𝐏</ci><ci id="alg1.l5.m1.1.1.1.1.1.1.2.3.cmml" xref="alg1.l5.m1.1.1.1.1.1.1.2.3">𝑡</ci></apply><apply id="alg1.l5.m1.1.1.1.1.1.1.3.cmml" xref="alg1.l5.m1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="alg1.l5.m1.1.1.1.1.1.1.3.1.cmml" xref="alg1.l5.m1.1.1.1.1.1.1.3">subscript</csymbol><ci id="alg1.l5.m1.1.1.1.1.1.1.3.2.cmml" xref="alg1.l5.m1.1.1.1.1.1.1.3.2">𝜃</ci><ci id="alg1.l5.m1.1.1.1.1.1.1.3.3.cmml" xref="alg1.l5.m1.1.1.1.1.1.1.3.3">𝑗</ci></apply></apply></set><apply id="alg1.l5.m1.1.1.1.3.cmml" xref="alg1.l5.m1.1.1.1.3"><eq id="alg1.l5.m1.1.1.1.3.1.cmml" xref="alg1.l5.m1.1.1.1.3.1"></eq><ci id="alg1.l5.m1.1.1.1.3.2.cmml" xref="alg1.l5.m1.1.1.1.3.2">𝑗</ci><cn type="integer" id="alg1.l5.m1.1.1.1.3.3.cmml" xref="alg1.l5.m1.1.1.1.3.3">1</cn></apply></apply><ci id="alg1.l5.m1.1.1.3.cmml" xref="alg1.l5.m1.1.1.3">𝐾</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l5.m1.1c">\{\mathbf{P}_{t}^{\theta_{j}}\}_{j=1}^{K}</annotation></semantics></math> to obtain <math id="alg1.l5.m2.1" class="ltx_Math" alttext="\mathbf{P}_{t}" display="inline"><semantics id="alg1.l5.m2.1a"><msub id="alg1.l5.m2.1.1" xref="alg1.l5.m2.1.1.cmml"><mi id="alg1.l5.m2.1.1.2" xref="alg1.l5.m2.1.1.2.cmml">𝐏</mi><mi id="alg1.l5.m2.1.1.3" xref="alg1.l5.m2.1.1.3.cmml">t</mi></msub><annotation-xml encoding="MathML-Content" id="alg1.l5.m2.1b"><apply id="alg1.l5.m2.1.1.cmml" xref="alg1.l5.m2.1.1"><csymbol cd="ambiguous" id="alg1.l5.m2.1.1.1.cmml" xref="alg1.l5.m2.1.1">subscript</csymbol><ci id="alg1.l5.m2.1.1.2.cmml" xref="alg1.l5.m2.1.1.2">𝐏</ci><ci id="alg1.l5.m2.1.1.3.cmml" xref="alg1.l5.m2.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l5.m2.1c">\mathbf{P}_{t}</annotation></semantics></math> with the MinCE or AvgCE fusion function.

</div>
<div id="alg1.l6" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">6:</span>     Update parameters of <math id="alg1.l6.m1.1" class="ltx_Math" alttext="\mathcal{M}^{t}" display="inline"><semantics id="alg1.l6.m1.1a"><msup id="alg1.l6.m1.1.1" xref="alg1.l6.m1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="alg1.l6.m1.1.1.2" xref="alg1.l6.m1.1.1.2.cmml">ℳ</mi><mi id="alg1.l6.m1.1.1.3" xref="alg1.l6.m1.1.1.3.cmml">t</mi></msup><annotation-xml encoding="MathML-Content" id="alg1.l6.m1.1b"><apply id="alg1.l6.m1.1.1.cmml" xref="alg1.l6.m1.1.1"><csymbol cd="ambiguous" id="alg1.l6.m1.1.1.1.cmml" xref="alg1.l6.m1.1.1">superscript</csymbol><ci id="alg1.l6.m1.1.1.2.cmml" xref="alg1.l6.m1.1.1.2">ℳ</ci><ci id="alg1.l6.m1.1.1.3.cmml" xref="alg1.l6.m1.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l6.m1.1c">\mathcal{M}^{t}</annotation></semantics></math> by minimizing the overall loss fuction in Eq. <a href="#S3.E5" title="In 3.2 LLMs Fusion ‣ 3 Knowledge Fusion of LLMs ‣ Knowledge Fusion of Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a>.

</div>
<div id="alg1.l7" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">7:</span>  <span id="alg1.l7.1" class="ltx_text ltx_font_bold">end</span> <span id="alg1.l7.2" class="ltx_text ltx_font_bold">for</span>
</div>
<div id="alg1.l8" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">8:</span>  <span id="alg1.l8.1" class="ltx_text ltx_font_bold">return</span>  <math id="alg1.l8.m1.1" class="ltx_Math" alttext="\mathcal{M}^{t}" display="inline"><semantics id="alg1.l8.m1.1a"><msup id="alg1.l8.m1.1.1" xref="alg1.l8.m1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="alg1.l8.m1.1.1.2" xref="alg1.l8.m1.1.1.2.cmml">ℳ</mi><mi id="alg1.l8.m1.1.1.3" xref="alg1.l8.m1.1.1.3.cmml">t</mi></msup><annotation-xml encoding="MathML-Content" id="alg1.l8.m1.1b"><apply id="alg1.l8.m1.1.1.cmml" xref="alg1.l8.m1.1.1"><csymbol cd="ambiguous" id="alg1.l8.m1.1.1.1.cmml" xref="alg1.l8.m1.1.1">superscript</csymbol><ci id="alg1.l8.m1.1.1.2.cmml" xref="alg1.l8.m1.1.1.2">ℳ</ci><ci id="alg1.l8.m1.1.1.3.cmml" xref="alg1.l8.m1.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l8.m1.1c">\mathcal{M}^{t}</annotation></semantics></math>.

</div>
</div>
</figure>
</section>
</section>
</section>
<section id="S4" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">4 </span>Experiments</h2>

<div id="S4.p1" class="ltx_para ltx_noindent">
<p id="S4.p1.1" class="ltx_p">In our experiments, we consider a general but challenging scenario of LLMs fusion where the source models share minimal commonalities in architectures or functionalities. Specifically, we conduct experiments on the 7B scale and select three representative open-source models: Llama-2, OpenLLaMA, and MPT as the source LLMs for fusion. Regarding the target LLM, we opt for another Llama-2 7B, which is generally the most robust one among the three source LLMs. The target LLM starts with the same pre-trained weights as its source counterpart but differs in that it updates parameters during training. To evaluate the performance of <span id="S4.p1.1.1" class="ltx_text ltx_font_smallcaps">FuseLLM</span>, we conduct experiments on benchmarks assessing the capabilities of LLMs in reasoning, commonsense, and code generation.</p>
</div>
<section id="S4.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.1 </span>Experimental Setup</h3>

<section id="S4.SS1.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Dataset for continual training</h4>

<div id="S4.SS1.SSS0.Px1.p1" class="ltx_para ltx_noindent">
<p id="S4.SS1.SSS0.Px1.p1.1" class="ltx_p">To continually train the target LLM for LLMs fusion, it is essential to
have a compact yet diverse training dataset. We have chosen MiniPile, a meticulously curated dataset resulting from a thorough clustering and filtering process.
MiniPile comprises approximately 1 million documents across 22
domains and 1.8 billion tokens, constituting less than 0.1% of the 2 trillion training tokens of Llama-2.
More dataset details can be found in Appendix <a href="#A2" title="Appendix B Details of MiniPile ‣ Knowledge Fusion of Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">B</span></a>.</p>
</div>
</section>
<section id="S4.SS1.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Fusion function</h4>

<div id="S4.SS1.SSS0.Px2.p1" class="ltx_para ltx_noindent">
<p id="S4.SS1.SSS0.Px2.p1.1" class="ltx_p">For the fusion function, we use the minimum cross-entropy (MinCE). However, the impact of employing alternative fusion functions will be examined in Section <a href="#S4.SS4" title="4.4 Analysis of Implementation Process ‣ 4 Experiments ‣ Knowledge Fusion of Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4.4</span></a>.</p>
</div>
</section>
<section id="S4.SS1.SSS0.Px3" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Training details</h4>

<div id="S4.SS1.SSS0.Px3.p1" class="ltx_para ltx_noindent">
<p id="S4.SS1.SSS0.Px3.p1.1" class="ltx_p">We train the target LLM of Llama-2 7B using a batch size of 128 and a maximum length of 2048 on a single node equipped with 8 NVIDIA A100 GPUs, each with 40GB of memory. Our training framework is implemented based on the Huggingface Transformers <cite class="ltx_cite ltx_citemacro_citep">(Wolf et al., <a href="#bib.bib60" title="" class="ltx_ref">2020</a>)</cite> and accelerated with FlashAttention <cite class="ltx_cite ltx_citemacro_citep">(Dao et al., <a href="#bib.bib15" title="" class="ltx_ref">2022</a>)</cite>. We empirically set the combination weight <math id="S4.SS1.SSS0.Px3.p1.1.m1.1" class="ltx_Math" alttext="\lambda" display="inline"><semantics id="S4.SS1.SSS0.Px3.p1.1.m1.1a"><mi id="S4.SS1.SSS0.Px3.p1.1.m1.1.1" xref="S4.SS1.SSS0.Px3.p1.1.m1.1.1.cmml">λ</mi><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS0.Px3.p1.1.m1.1b"><ci id="S4.SS1.SSS0.Px3.p1.1.m1.1.1.cmml" xref="S4.SS1.SSS0.Px3.p1.1.m1.1.1">𝜆</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS0.Px3.p1.1.m1.1c">\lambda</annotation></semantics></math> in Eq. <a href="#S3.E5" title="In 3.2 LLMs Fusion ‣ 3 Knowledge Fusion of LLMs ‣ Knowledge Fusion of Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a> to 0.9. The training consists of only a single epoch, which takes approximately 33 hours. For further hyper-parameter details, please refer to Appendix <a href="#A3" title="Appendix C Training Details ‣ Knowledge Fusion of Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">C</span></a>.</p>
</div>
</section>
<section id="S4.SS1.SSS0.Px4" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Evaluation</h4>

<div id="S4.SS1.SSS0.Px4.p1" class="ltx_para ltx_noindent">
<p id="S4.SS1.SSS0.Px4.p1.1" class="ltx_p">We evaluate <span id="S4.SS1.SSS0.Px4.p1.1.1" class="ltx_text ltx_font_smallcaps">FuseLLM</span> on three benchmarks that represent different core capabilities of LLMs, spanning <span id="S4.SS1.SSS0.Px4.p1.1.2" class="ltx_text ltx_font_italic">reasoning</span>, <span id="S4.SS1.SSS0.Px4.p1.1.3" class="ltx_text ltx_font_italic">commonsense</span>, and <span id="S4.SS1.SSS0.Px4.p1.1.4" class="ltx_text ltx_font_italic">code generation</span>.</p>
<ul id="S4.I1" class="ltx_itemize">
<li id="S4.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S4.I1.i1.p1" class="ltx_para ltx_noindent">
<p id="S4.I1.i1.p1.1" class="ltx_p">Big-Bench Hard (BBH) <cite class="ltx_cite ltx_citemacro_citep">(Suzgun et al., <a href="#bib.bib50" title="" class="ltx_ref">2022</a>)</cite> is a benchmark to evaluate the general <span id="S4.I1.i1.p1.1.1" class="ltx_text ltx_font_italic">reasoning</span> ability of LLMs. It contains 23 multiple-choice tasks and 4 free-form generation tasks from the Big-Bench <cite class="ltx_cite ltx_citemacro_citep">(Srivastava et al., <a href="#bib.bib48" title="" class="ltx_ref">2022</a>)</cite>, which can be classified into four categories: algorithmic and arithmetic reasoning, natural language understanding, world knowledge, and multilingual knowledge and reasoning. We follow previous work <cite class="ltx_cite ltx_citemacro_citep">(Wang et al., <a href="#bib.bib59" title="" class="ltx_ref">2023b</a>)</cite> to generate the predictions based on few-shot chain-of-thought (CoT) prompts and then calculate the exact match (EM) accuracy.</p>
</div>
</li>
<li id="S4.I1.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S4.I1.i2.p1" class="ltx_para ltx_noindent">
<p id="S4.I1.i2.p1.1" class="ltx_p">Common Sense (CS) is a benchmark to evaluate the <span id="S4.I1.i2.p1.1.1" class="ltx_text ltx_font_italic">commonsense</span> capability of LLMs. We consider 5 standard multiple-choice tasks: ARC easy and challenge <cite class="ltx_cite ltx_citemacro_citep">(Clark et al., <a href="#bib.bib13" title="" class="ltx_ref">2018</a>)</cite>, BoolQ <cite class="ltx_cite ltx_citemacro_citep">(Clark et al., <a href="#bib.bib11" title="" class="ltx_ref">2019a</a>)</cite>, HellaSwag <cite class="ltx_cite ltx_citemacro_citep">(Zellers et al., <a href="#bib.bib63" title="" class="ltx_ref">2019</a>)</cite>, and OpenBookQA <cite class="ltx_cite ltx_citemacro_citep">(Mihaylov et al., <a href="#bib.bib37" title="" class="ltx_ref">2018</a>)</cite>. We employ lm-eval-hardness <cite class="ltx_cite ltx_citemacro_citep">(Gao et al., <a href="#bib.bib19" title="" class="ltx_ref">2021</a>)</cite> to conduct a likelihood-based zero-shot evaluation. Specifically, we select the option with the highest likelihood given the context and report the accuracy.</p>
</div>
</li>
<li id="S4.I1.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S4.I1.i3.p1" class="ltx_para ltx_noindent">
<p id="S4.I1.i3.p1.1" class="ltx_p">MultiPL-E (ME) <cite class="ltx_cite ltx_citemacro_citep">(Cassano et al., <a href="#bib.bib6" title="" class="ltx_ref">2022</a>)</cite> is a multilingual programming benchmark to assess the <em id="S4.I1.i3.p1.1.1" class="ltx_emph ltx_font_italic">coding</em> ability of LLMs. It is translated from the Python benchmark <cite class="ltx_cite ltx_citemacro_citep">(Chen et al., <a href="#bib.bib9" title="" class="ltx_ref">2021</a>)</cite> into parallel datasets in 18 programming languages. We use the bigcode-evaluation-hardness <cite class="ltx_cite ltx_citemacro_citep">(Ben Allal et al., <a href="#bib.bib3" title="" class="ltx_ref">2022</a>)</cite> to perform zero-shot code generation in 10 popular programming languages in the HumanEval category and report the pass@1 <cite class="ltx_cite ltx_citemacro_citep">(Chen et al., <a href="#bib.bib9" title="" class="ltx_ref">2021</a>)</cite> based on 20 generated samples for each question.</p>
</div>
</li>
</ul>
</div>
</section>
<section id="S4.SS1.SSS0.Px5" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Baselines</h4>

<div id="S4.SS1.SSS0.Px5.p1" class="ltx_para ltx_noindent">
<p id="S4.SS1.SSS0.Px5.p1.1" class="ltx_p">In our experiments, we compare our <span id="S4.SS1.SSS0.Px5.p1.1.1" class="ltx_text ltx_font_smallcaps">FuseLLM</span> with two sets of baselines: (1) <span id="S4.SS1.SSS0.Px5.p1.1.2" class="ltx_text ltx_font_bold">original LLMs</span>, including Llama-2 7B, OpenLLaMA 7B, and MPT 7B; and (2) <span id="S4.SS1.SSS0.Px5.p1.1.3" class="ltx_text ltx_font_bold">Llama-2 CLM</span>: continually trained Llama-2 7B on MiniPile using only the casual language modeling objective.</p>
</div>
</section>
</section>
<section id="S4.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.2 </span>Overall Results</h3>

<div id="S4.SS2.p1" class="ltx_para ltx_noindent">
<p id="S4.SS2.p1.1" class="ltx_p">Table <a href="#S4.T1" title="Table 1 ‣ 4.2 Overall Results ‣ 4 Experiments ‣ Knowledge Fusion of Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a> presents the overall results of <span id="S4.SS2.p1.1.1" class="ltx_text ltx_font_smallcaps">FuseLLM</span> in comparison to the baseline methods on BBH. We can observe that the three source LLMs exhibit varying performance across the 27 BBH tasks, with Llama-2 generally outperforming the others. After continual training with a compact and diverse corpus, Llama-2 CLM shows a relative improvement of 1.86% compared to Llama-2, although this improvement is relatively modest and inconsistent across tasks. On average, <span id="S4.SS2.p1.1.2" class="ltx_text ltx_font_smallcaps">FuseLLM</span> demonstrates an average relative performance gain of 5.16% over the original Llama-2 across all 27 tasks. In specific tasks, the enhancements achieved by <span id="S4.SS2.p1.1.3" class="ltx_text ltx_font_smallcaps">FuseLLM</span> are substantial (e.g., from 54.40 to 65.20 in the Hyperbaton task). In tasks such as Dick Languages where simple continual pre-training leads to a decline in performance, <span id="S4.SS2.p1.1.4" class="ltx_text ltx_font_smallcaps">FuseLLM</span> leverages the combined strengths of individual source LLMs to recover performance improvements. Note that <span id="S4.SS2.p1.1.5" class="ltx_text ltx_font_smallcaps">FuseLLM</span> occasionally exhibits degraded performance on tasks such as Geometric Shapes and Word Sorting, which could be attributed to two reasons. First, the other source LLMs, apart from Llama-2, perform poorly on these tasks, affecting the fusion results. Second, the relevance between the continual training dataset and downstream tasks also contributes to the performance degradation.</p>
</div>
<div id="S4.SS2.p2" class="ltx_para ltx_noindent">
<p id="S4.SS2.p2.1" class="ltx_p">Table <a href="#S4.T2" title="Table 2 ‣ 4.2 Overall Results ‣ 4 Experiments ‣ Knowledge Fusion of Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a> shows the zero-shot performance of <span id="S4.SS2.p2.1.1" class="ltx_text ltx_font_smallcaps">FuseLLM</span> and the baseline methods on the Common Sense (CS) benchmark. The results demonstrate that <span id="S4.SS2.p2.1.2" class="ltx_text ltx_font_smallcaps">FuseLLM</span> consistently surpasses the baselines across all five tasks, achieving a relative performance improvement of 1.25% over Llama-2. In contrast, Llama-2 CLM exhibits a marginal improvement, with only a 0.16% relative enhancement compared to Llama-2. Notably, substantial improvements from Llama-2 to <span id="S4.SS2.p2.1.3" class="ltx_text ltx_font_smallcaps">FuseLLM</span> are observed in the challenging ARC-challenge (2.40%) and OpenBookQA (2.71%) tasks, highlighting the effectiveness of <span id="S4.SS2.p2.1.4" class="ltx_text ltx_font_smallcaps">FuseLLM</span> in leveraging collective knowledge to address intricate problems.</p>
</div>
<figure id="S4.T1" class="ltx_table">
<div id="S4.T1.1" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:377.6pt;height:393.8pt;vertical-align:-0.8pt;"><span class="ltx_transformed_inner" style="transform:translate(-62.0pt,64.5pt) scale(0.752833543102604,0.752833543102604) ;">
<table id="S4.T1.1.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S4.T1.1.1.1.1" class="ltx_tr">
<th id="S4.T1.1.1.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_border_tt"><span id="S4.T1.1.1.1.1.1.1" class="ltx_text ltx_font_bold">Task</span></th>
<th id="S4.T1.1.1.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S4.T1.1.1.1.1.2.1" class="ltx_text ltx_font_bold">OpenLLaMA</span></th>
<th id="S4.T1.1.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S4.T1.1.1.1.1.3.1" class="ltx_text ltx_font_bold">MPT</span></th>
<th id="S4.T1.1.1.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S4.T1.1.1.1.1.4.1" class="ltx_text ltx_font_bold">Llama-2</span></th>
<th id="S4.T1.1.1.1.1.5" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_border_tt" style="background-color:#ECF4FF;"><span id="S4.T1.1.1.1.1.5.1" class="ltx_text ltx_font_bold" style="background-color:#ECF4FF;">Llama-2 CLM</span></th>
<th id="S4.T1.1.1.1.1.6" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_border_tt" style="background-color:#ECF4FF;"><span id="S4.T1.1.1.1.1.6.1" class="ltx_text ltx_font_bold ltx_font_smallcaps" style="background-color:#ECF4FF;">FuseLLM</span></th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S4.T1.1.1.2.1" class="ltx_tr">
<td id="S4.T1.1.1.2.1.1" class="ltx_td ltx_align_left ltx_border_tt">Boolean Expressions</td>
<td id="S4.T1.1.1.2.1.2" class="ltx_td ltx_align_center ltx_border_tt">74.40</td>
<td id="S4.T1.1.1.2.1.3" class="ltx_td ltx_align_center ltx_border_tt">66.00</td>
<td id="S4.T1.1.1.2.1.4" class="ltx_td ltx_align_center ltx_border_tt">68.80</td>
<td id="S4.T1.1.1.2.1.5" class="ltx_td ltx_align_left ltx_border_tt" style="background-color:#ECF4FF;"><span id="S4.T1.1.1.2.1.5.1" class="ltx_text ltx_font_bold" style="background-color:#ECF4FF;">76.00<span id="S4.T1.1.1.2.1.5.1.1" class="ltx_text ltx_font_medium"> <span id="S4.T1.1.1.2.1.5.1.1.1" class="ltx_text" style="color:#66B2FF;">(+10.47%)</span></span></span></td>
<td id="S4.T1.1.1.2.1.6" class="ltx_td ltx_align_left ltx_border_tt" style="background-color:#ECF4FF;"><span id="S4.T1.1.1.2.1.6.1" class="ltx_text" style="background-color:#ECF4FF;">71.60 <span id="S4.T1.1.1.2.1.6.1.1" class="ltx_text" style="color:#0000FF;">(+4.07%)</span></span></td>
</tr>
<tr id="S4.T1.1.1.3.2" class="ltx_tr">
<td id="S4.T1.1.1.3.2.1" class="ltx_td ltx_align_left">Causal Judgement</td>
<td id="S4.T1.1.1.3.2.2" class="ltx_td ltx_align_center">45.45</td>
<td id="S4.T1.1.1.3.2.3" class="ltx_td ltx_align_center"><span id="S4.T1.1.1.3.2.3.1" class="ltx_text ltx_font_bold">50.80</span></td>
<td id="S4.T1.1.1.3.2.4" class="ltx_td ltx_align_center"><span id="S4.T1.1.1.3.2.4.1" class="ltx_text ltx_font_bold">50.80</span></td>
<td id="S4.T1.1.1.3.2.5" class="ltx_td ltx_align_left" style="background-color:#ECF4FF;"><span id="S4.T1.1.1.3.2.5.1" class="ltx_text" style="background-color:#ECF4FF;">46.52 <span id="S4.T1.1.1.3.2.5.1.1" class="ltx_text" style="color:#FF7171;">(-8.43%)</span></span></td>
<td id="S4.T1.1.1.3.2.6" class="ltx_td ltx_align_left" style="background-color:#ECF4FF;"><span id="S4.T1.1.1.3.2.6.1" class="ltx_text" style="background-color:#ECF4FF;">46.52 <span id="S4.T1.1.1.3.2.6.1.1" class="ltx_text" style="color:#FF0000;">(-8.43%)</span></span></td>
</tr>
<tr id="S4.T1.1.1.4.3" class="ltx_tr">
<td id="S4.T1.1.1.4.3.1" class="ltx_td ltx_align_left">Date Understanding</td>
<td id="S4.T1.1.1.4.3.2" class="ltx_td ltx_align_center">43.60</td>
<td id="S4.T1.1.1.4.3.3" class="ltx_td ltx_align_center">43.60</td>
<td id="S4.T1.1.1.4.3.4" class="ltx_td ltx_align_center">59.60</td>
<td id="S4.T1.1.1.4.3.5" class="ltx_td ltx_align_left" style="background-color:#ECF4FF;"><span id="S4.T1.1.1.4.3.5.1" class="ltx_text" style="background-color:#ECF4FF;">59.20 <span id="S4.T1.1.1.4.3.5.1.1" class="ltx_text" style="color:#FF7171;">(-0.67%)</span></span></td>
<td id="S4.T1.1.1.4.3.6" class="ltx_td ltx_align_left" style="background-color:#ECF4FF;"><span id="S4.T1.1.1.4.3.6.1" class="ltx_text ltx_font_bold" style="background-color:#ECF4FF;">62.40<span id="S4.T1.1.1.4.3.6.1.1" class="ltx_text ltx_font_medium"> <span id="S4.T1.1.1.4.3.6.1.1.1" class="ltx_text" style="color:#0000FF;">(+4.70%)</span></span></span></td>
</tr>
<tr id="S4.T1.1.1.5.4" class="ltx_tr">
<td id="S4.T1.1.1.5.4.1" class="ltx_td ltx_align_left">Disambiguation QA</td>
<td id="S4.T1.1.1.5.4.2" class="ltx_td ltx_align_center">36.00</td>
<td id="S4.T1.1.1.5.4.3" class="ltx_td ltx_align_center">47.60</td>
<td id="S4.T1.1.1.5.4.4" class="ltx_td ltx_align_center">46.80</td>
<td id="S4.T1.1.1.5.4.5" class="ltx_td ltx_align_left" style="background-color:#ECF4FF;"><span id="S4.T1.1.1.5.4.5.1" class="ltx_text" style="background-color:#ECF4FF;">48.00 <span id="S4.T1.1.1.5.4.5.1.1" class="ltx_text" style="color:#66B2FF;">(+2.56%)</span></span></td>
<td id="S4.T1.1.1.5.4.6" class="ltx_td ltx_align_left" style="background-color:#ECF4FF;"><span id="S4.T1.1.1.5.4.6.1" class="ltx_text ltx_font_bold" style="background-color:#ECF4FF;">50.00<span id="S4.T1.1.1.5.4.6.1.1" class="ltx_text ltx_font_medium"> <span id="S4.T1.1.1.5.4.6.1.1.1" class="ltx_text" style="color:#0000FF;">(+6.84%)</span></span></span></td>
</tr>
<tr id="S4.T1.1.1.6.5" class="ltx_tr">
<td id="S4.T1.1.1.6.5.1" class="ltx_td ltx_align_left">Dyck Languages</td>
<td id="S4.T1.1.1.6.5.2" class="ltx_td ltx_align_center">5.20</td>
<td id="S4.T1.1.1.6.5.3" class="ltx_td ltx_align_center">5.20</td>
<td id="S4.T1.1.1.6.5.4" class="ltx_td ltx_align_center">7.20</td>
<td id="S4.T1.1.1.6.5.5" class="ltx_td ltx_align_left" style="background-color:#ECF4FF;"><span id="S4.T1.1.1.6.5.5.1" class="ltx_text" style="background-color:#ECF4FF;">6.40 <span id="S4.T1.1.1.6.5.5.1.1" class="ltx_text" style="color:#FF7171;">(-11.11%)</span></span></td>
<td id="S4.T1.1.1.6.5.6" class="ltx_td ltx_align_left" style="background-color:#ECF4FF;"><span id="S4.T1.1.1.6.5.6.1" class="ltx_text ltx_font_bold" style="background-color:#ECF4FF;">8.80<span id="S4.T1.1.1.6.5.6.1.1" class="ltx_text ltx_font_medium"> <span id="S4.T1.1.1.6.5.6.1.1.1" class="ltx_text" style="color:#0000FF;">(+22.22%)</span></span></span></td>
</tr>
<tr id="S4.T1.1.1.7.6" class="ltx_tr">
<td id="S4.T1.1.1.7.6.1" class="ltx_td ltx_align_left">Formal Fallacies</td>
<td id="S4.T1.1.1.7.6.2" class="ltx_td ltx_align_center">50.80</td>
<td id="S4.T1.1.1.7.6.3" class="ltx_td ltx_align_center"><span id="S4.T1.1.1.7.6.3.1" class="ltx_text ltx_font_bold">52.80</span></td>
<td id="S4.T1.1.1.7.6.4" class="ltx_td ltx_align_center">49.20</td>
<td id="S4.T1.1.1.7.6.5" class="ltx_td ltx_align_left" style="background-color:#ECF4FF;"><span id="S4.T1.1.1.7.6.5.1" class="ltx_text" style="background-color:#ECF4FF;">48.80 <span id="S4.T1.1.1.7.6.5.1.1" class="ltx_text" style="color:#FF7171;">(-0.81%)</span></span></td>
<td id="S4.T1.1.1.7.6.6" class="ltx_td ltx_align_left" style="background-color:#ECF4FF;"><span id="S4.T1.1.1.7.6.6.1" class="ltx_text" style="background-color:#ECF4FF;">49.20 <span id="S4.T1.1.1.7.6.6.1.1" class="ltx_text" style="color:#0000FF;">(+0.00%)</span></span></td>
</tr>
<tr id="S4.T1.1.1.8.7" class="ltx_tr">
<td id="S4.T1.1.1.8.7.1" class="ltx_td ltx_align_left">Geometric Shapes</td>
<td id="S4.T1.1.1.8.7.2" class="ltx_td ltx_align_center">0.00</td>
<td id="S4.T1.1.1.8.7.3" class="ltx_td ltx_align_center">0.00</td>
<td id="S4.T1.1.1.8.7.4" class="ltx_td ltx_align_center"><span id="S4.T1.1.1.8.7.4.1" class="ltx_text ltx_font_bold">34.40</span></td>
<td id="S4.T1.1.1.8.7.5" class="ltx_td ltx_align_left" style="background-color:#ECF4FF;"><span id="S4.T1.1.1.8.7.5.1" class="ltx_text" style="background-color:#ECF4FF;">19.20 <span id="S4.T1.1.1.8.7.5.1.1" class="ltx_text" style="color:#FF7171;">(-44.17%)</span></span></td>
<td id="S4.T1.1.1.8.7.6" class="ltx_td ltx_align_left" style="background-color:#ECF4FF;"><span id="S4.T1.1.1.8.7.6.1" class="ltx_text" style="background-color:#ECF4FF;">22.80 <span id="S4.T1.1.1.8.7.6.1.1" class="ltx_text" style="color:#FF0000;">(-33.72%)</span></span></td>
</tr>
<tr id="S4.T1.1.1.9.8" class="ltx_tr">
<td id="S4.T1.1.1.9.8.1" class="ltx_td ltx_align_left">Hyperbaton</td>
<td id="S4.T1.1.1.9.8.2" class="ltx_td ltx_align_center">62.80</td>
<td id="S4.T1.1.1.9.8.3" class="ltx_td ltx_align_center">53.60</td>
<td id="S4.T1.1.1.9.8.4" class="ltx_td ltx_align_center">54.40</td>
<td id="S4.T1.1.1.9.8.5" class="ltx_td ltx_align_left" style="background-color:#ECF4FF;"><span id="S4.T1.1.1.9.8.5.1" class="ltx_text" style="background-color:#ECF4FF;">56.40 <span id="S4.T1.1.1.9.8.5.1.1" class="ltx_text" style="color:#66B2FF;">(+3.68%)</span></span></td>
<td id="S4.T1.1.1.9.8.6" class="ltx_td ltx_align_left" style="background-color:#ECF4FF;"><span id="S4.T1.1.1.9.8.6.1" class="ltx_text ltx_font_bold" style="background-color:#ECF4FF;">65.20<span id="S4.T1.1.1.9.8.6.1.1" class="ltx_text ltx_font_medium"> <span id="S4.T1.1.1.9.8.6.1.1.1" class="ltx_text" style="color:#0000FF;">(+19.85%)</span></span></span></td>
</tr>
<tr id="S4.T1.1.1.10.9" class="ltx_tr">
<td id="S4.T1.1.1.10.9.1" class="ltx_td ltx_align_left">Logical Deduction (3 objects)</td>
<td id="S4.T1.1.1.10.9.2" class="ltx_td ltx_align_center">43.60</td>
<td id="S4.T1.1.1.10.9.3" class="ltx_td ltx_align_center">40.80</td>
<td id="S4.T1.1.1.10.9.4" class="ltx_td ltx_align_center">54.00</td>
<td id="S4.T1.1.1.10.9.5" class="ltx_td ltx_align_left" style="background-color:#ECF4FF;"><span id="S4.T1.1.1.10.9.5.1" class="ltx_text" style="background-color:#ECF4FF;">57.20 <span id="S4.T1.1.1.10.9.5.1.1" class="ltx_text" style="color:#66B2FF;">(+5.93%)</span></span></td>
<td id="S4.T1.1.1.10.9.6" class="ltx_td ltx_align_left" style="background-color:#ECF4FF;"><span id="S4.T1.1.1.10.9.6.1" class="ltx_text ltx_font_bold" style="background-color:#ECF4FF;">60.40<span id="S4.T1.1.1.10.9.6.1.1" class="ltx_text ltx_font_medium"> <span id="S4.T1.1.1.10.9.6.1.1.1" class="ltx_text" style="color:#0000FF;">(+11.85%)</span></span></span></td>
</tr>
<tr id="S4.T1.1.1.11.10" class="ltx_tr">
<td id="S4.T1.1.1.11.10.1" class="ltx_td ltx_align_left">Logical Deduction (5 objects)</td>
<td id="S4.T1.1.1.11.10.2" class="ltx_td ltx_align_center">24.80</td>
<td id="S4.T1.1.1.11.10.3" class="ltx_td ltx_align_center">31.60</td>
<td id="S4.T1.1.1.11.10.4" class="ltx_td ltx_align_center">31.20</td>
<td id="S4.T1.1.1.11.10.5" class="ltx_td ltx_align_left" style="background-color:#ECF4FF;"><span id="S4.T1.1.1.11.10.5.1" class="ltx_text ltx_font_bold" style="background-color:#ECF4FF;">35.60<span id="S4.T1.1.1.11.10.5.1.1" class="ltx_text ltx_font_medium"> <span id="S4.T1.1.1.11.10.5.1.1.1" class="ltx_text" style="color:#66B2FF;">(+14.10%)</span></span></span></td>
<td id="S4.T1.1.1.11.10.6" class="ltx_td ltx_align_left" style="background-color:#ECF4FF;"><span id="S4.T1.1.1.11.10.6.1" class="ltx_text" style="background-color:#ECF4FF;">33.20 <span id="S4.T1.1.1.11.10.6.1.1" class="ltx_text" style="color:#0000FF;">(+6.41%)</span></span></td>
</tr>
<tr id="S4.T1.1.1.12.11" class="ltx_tr">
<td id="S4.T1.1.1.12.11.1" class="ltx_td ltx_align_left">Logical Deduction (7 objects)</td>
<td id="S4.T1.1.1.12.11.2" class="ltx_td ltx_align_center">16.80</td>
<td id="S4.T1.1.1.12.11.3" class="ltx_td ltx_align_center">18.40</td>
<td id="S4.T1.1.1.12.11.4" class="ltx_td ltx_align_center">24.80</td>
<td id="S4.T1.1.1.12.11.5" class="ltx_td ltx_align_left" style="background-color:#ECF4FF;"><span id="S4.T1.1.1.12.11.5.1" class="ltx_text ltx_font_bold" style="background-color:#ECF4FF;">29.60<span id="S4.T1.1.1.12.11.5.1.1" class="ltx_text ltx_font_medium"> <span id="S4.T1.1.1.12.11.5.1.1.1" class="ltx_text" style="color:#66B2FF;">(+19.35%)</span></span></span></td>
<td id="S4.T1.1.1.12.11.6" class="ltx_td ltx_align_left" style="background-color:#ECF4FF;"><span id="S4.T1.1.1.12.11.6.1" class="ltx_text" style="background-color:#ECF4FF;">25.60 <span id="S4.T1.1.1.12.11.6.1.1" class="ltx_text" style="color:#0000FF;">(+3.23%)</span></span></td>
</tr>
<tr id="S4.T1.1.1.13.12" class="ltx_tr">
<td id="S4.T1.1.1.13.12.1" class="ltx_td ltx_align_left">Movie Recommendation</td>
<td id="S4.T1.1.1.13.12.2" class="ltx_td ltx_align_center">39.60</td>
<td id="S4.T1.1.1.13.12.3" class="ltx_td ltx_align_center">52.00</td>
<td id="S4.T1.1.1.13.12.4" class="ltx_td ltx_align_center">72.80</td>
<td id="S4.T1.1.1.13.12.5" class="ltx_td ltx_align_left" style="background-color:#ECF4FF;"><span id="S4.T1.1.1.13.12.5.1" class="ltx_text" style="background-color:#ECF4FF;">71.60 <span id="S4.T1.1.1.13.12.5.1.1" class="ltx_text" style="color:#FF7171;">(-1.65%)</span></span></td>
<td id="S4.T1.1.1.13.12.6" class="ltx_td ltx_align_left" style="background-color:#ECF4FF;"><span id="S4.T1.1.1.13.12.6.1" class="ltx_text ltx_font_bold" style="background-color:#ECF4FF;">73.60<span id="S4.T1.1.1.13.12.6.1.1" class="ltx_text ltx_font_medium"> <span id="S4.T1.1.1.13.12.6.1.1.1" class="ltx_text" style="color:#0000FF;">(+1.10%)</span></span></span></td>
</tr>
<tr id="S4.T1.1.1.14.13" class="ltx_tr">
<td id="S4.T1.1.1.14.13.1" class="ltx_td ltx_align_left">Multistep Arithmetic Two</td>
<td id="S4.T1.1.1.14.13.2" class="ltx_td ltx_align_center">0.80</td>
<td id="S4.T1.1.1.14.13.3" class="ltx_td ltx_align_center">0.40</td>
<td id="S4.T1.1.1.14.13.4" class="ltx_td ltx_align_center">0.80</td>
<td id="S4.T1.1.1.14.13.5" class="ltx_td ltx_align_left" style="background-color:#ECF4FF;"><span id="S4.T1.1.1.14.13.5.1" class="ltx_text" style="background-color:#ECF4FF;">4.40 <span id="S4.T1.1.1.14.13.5.1.1" class="ltx_text" style="color:#66B2FF;">(+450.00%)</span></span></td>
<td id="S4.T1.1.1.14.13.6" class="ltx_td ltx_align_left" style="background-color:#ECF4FF;"><span id="S4.T1.1.1.14.13.6.1" class="ltx_text ltx_font_bold" style="background-color:#ECF4FF;">4.80<span id="S4.T1.1.1.14.13.6.1.1" class="ltx_text ltx_font_medium"> <span id="S4.T1.1.1.14.13.6.1.1.1" class="ltx_text" style="color:#0000FF;">(+500.00%)</span></span></span></td>
</tr>
<tr id="S4.T1.1.1.15.14" class="ltx_tr">
<td id="S4.T1.1.1.15.14.1" class="ltx_td ltx_align_left">Navigate</td>
<td id="S4.T1.1.1.15.14.2" class="ltx_td ltx_align_center">54.00</td>
<td id="S4.T1.1.1.15.14.3" class="ltx_td ltx_align_center">48.80</td>
<td id="S4.T1.1.1.15.14.4" class="ltx_td ltx_align_center">56.00</td>
<td id="S4.T1.1.1.15.14.5" class="ltx_td ltx_align_left" style="background-color:#ECF4FF;"><span id="S4.T1.1.1.15.14.5.1" class="ltx_text" style="background-color:#ECF4FF;">61.20 <span id="S4.T1.1.1.15.14.5.1.1" class="ltx_text" style="color:#66B2FF;">(+9.29%)</span></span></td>
<td id="S4.T1.1.1.15.14.6" class="ltx_td ltx_align_left" style="background-color:#ECF4FF;"><span id="S4.T1.1.1.15.14.6.1" class="ltx_text ltx_font_bold" style="background-color:#ECF4FF;">64.40<span id="S4.T1.1.1.15.14.6.1.1" class="ltx_text ltx_font_medium"> <span id="S4.T1.1.1.15.14.6.1.1.1" class="ltx_text" style="color:#0000FF;">(+15.00%)</span></span></span></td>
</tr>
<tr id="S4.T1.1.1.16.15" class="ltx_tr">
<td id="S4.T1.1.1.16.15.1" class="ltx_td ltx_align_left">Object Counting</td>
<td id="S4.T1.1.1.16.15.2" class="ltx_td ltx_align_center">49.60</td>
<td id="S4.T1.1.1.16.15.3" class="ltx_td ltx_align_center">40.40</td>
<td id="S4.T1.1.1.16.15.4" class="ltx_td ltx_align_center">49.60</td>
<td id="S4.T1.1.1.16.15.5" class="ltx_td ltx_align_left" style="background-color:#ECF4FF;"><span id="S4.T1.1.1.16.15.5.1" class="ltx_text" style="background-color:#ECF4FF;">51.60 <span id="S4.T1.1.1.16.15.5.1.1" class="ltx_text" style="color:#66B2FF;">(+4.03%)</span></span></td>
<td id="S4.T1.1.1.16.15.6" class="ltx_td ltx_align_left" style="background-color:#ECF4FF;"><span id="S4.T1.1.1.16.15.6.1" class="ltx_text ltx_font_bold" style="background-color:#ECF4FF;">55.20<span id="S4.T1.1.1.16.15.6.1.1" class="ltx_text ltx_font_medium"> <span id="S4.T1.1.1.16.15.6.1.1.1" class="ltx_text" style="color:#0000FF;">(+11.29%)</span></span></span></td>
</tr>
<tr id="S4.T1.1.1.17.16" class="ltx_tr">
<td id="S4.T1.1.1.17.16.1" class="ltx_td ltx_align_left">Penguins in a Table</td>
<td id="S4.T1.1.1.17.16.2" class="ltx_td ltx_align_center">28.08</td>
<td id="S4.T1.1.1.17.16.3" class="ltx_td ltx_align_center">28.08</td>
<td id="S4.T1.1.1.17.16.4" class="ltx_td ltx_align_center">32.19</td>
<td id="S4.T1.1.1.17.16.5" class="ltx_td ltx_align_left" style="background-color:#ECF4FF;"><span id="S4.T1.1.1.17.16.5.1" class="ltx_text" style="background-color:#ECF4FF;">31.51 <span id="S4.T1.1.1.17.16.5.1.1" class="ltx_text" style="color:#FF7171;">(-2.11%)</span></span></td>
<td id="S4.T1.1.1.17.16.6" class="ltx_td ltx_align_left" style="background-color:#ECF4FF;"><span id="S4.T1.1.1.17.16.6.1" class="ltx_text ltx_font_bold" style="background-color:#ECF4FF;">32.88<span id="S4.T1.1.1.17.16.6.1.1" class="ltx_text ltx_font_medium"> <span id="S4.T1.1.1.17.16.6.1.1.1" class="ltx_text" style="color:#0000FF;">(+2.14%)</span></span></span></td>
</tr>
<tr id="S4.T1.1.1.18.17" class="ltx_tr">
<td id="S4.T1.1.1.18.17.1" class="ltx_td ltx_align_left">Reasoning about Colored Objects</td>
<td id="S4.T1.1.1.18.17.2" class="ltx_td ltx_align_center">28.00</td>
<td id="S4.T1.1.1.18.17.3" class="ltx_td ltx_align_center">31.60</td>
<td id="S4.T1.1.1.18.17.4" class="ltx_td ltx_align_center">46.40</td>
<td id="S4.T1.1.1.18.17.5" class="ltx_td ltx_align_left" style="background-color:#ECF4FF;"><span id="S4.T1.1.1.18.17.5.1" class="ltx_text" style="background-color:#ECF4FF;">47.20 <span id="S4.T1.1.1.18.17.5.1.1" class="ltx_text" style="color:#66B2FF;">(+1.72%)</span></span></td>
<td id="S4.T1.1.1.18.17.6" class="ltx_td ltx_align_left" style="background-color:#ECF4FF;"><span id="S4.T1.1.1.18.17.6.1" class="ltx_text ltx_font_bold" style="background-color:#ECF4FF;">48.40<span id="S4.T1.1.1.18.17.6.1.1" class="ltx_text ltx_font_medium"> <span id="S4.T1.1.1.18.17.6.1.1.1" class="ltx_text" style="color:#0000FF;">(+4.31%)</span></span></span></td>
</tr>
<tr id="S4.T1.1.1.19.18" class="ltx_tr">
<td id="S4.T1.1.1.19.18.1" class="ltx_td ltx_align_left">Ruin Names</td>
<td id="S4.T1.1.1.19.18.2" class="ltx_td ltx_align_center">31.20</td>
<td id="S4.T1.1.1.19.18.3" class="ltx_td ltx_align_center">23.20</td>
<td id="S4.T1.1.1.19.18.4" class="ltx_td ltx_align_center"><span id="S4.T1.1.1.19.18.4.1" class="ltx_text ltx_font_bold">34.00</span></td>
<td id="S4.T1.1.1.19.18.5" class="ltx_td ltx_align_left" style="background-color:#ECF4FF;"><span id="S4.T1.1.1.19.18.5.1" class="ltx_text" style="background-color:#ECF4FF;">30.80 <span id="S4.T1.1.1.19.18.5.1.1" class="ltx_text" style="color:#FF7171;">(-9.41%)</span></span></td>
<td id="S4.T1.1.1.19.18.6" class="ltx_td ltx_align_left" style="background-color:#ECF4FF;"><span id="S4.T1.1.1.19.18.6.1" class="ltx_text" style="background-color:#ECF4FF;">32.40 <span id="S4.T1.1.1.19.18.6.1.1" class="ltx_text" style="color:#FF0000;">(-4.71%)</span></span></td>
</tr>
<tr id="S4.T1.1.1.20.19" class="ltx_tr">
<td id="S4.T1.1.1.20.19.1" class="ltx_td ltx_align_left">Salient Translation Error Detection</td>
<td id="S4.T1.1.1.20.19.2" class="ltx_td ltx_align_center">14.80</td>
<td id="S4.T1.1.1.20.19.3" class="ltx_td ltx_align_center">0.00</td>
<td id="S4.T1.1.1.20.19.4" class="ltx_td ltx_align_center">24.80</td>
<td id="S4.T1.1.1.20.19.5" class="ltx_td ltx_align_left" style="background-color:#ECF4FF;"><span id="S4.T1.1.1.20.19.5.1" class="ltx_text" style="background-color:#ECF4FF;">27.60 <span id="S4.T1.1.1.20.19.5.1.1" class="ltx_text" style="color:#66B2FF;">(+11.29%)</span></span></td>
<td id="S4.T1.1.1.20.19.6" class="ltx_td ltx_align_left" style="background-color:#ECF4FF;"><span id="S4.T1.1.1.20.19.6.1" class="ltx_text ltx_font_bold" style="background-color:#ECF4FF;">29.20<span id="S4.T1.1.1.20.19.6.1.1" class="ltx_text ltx_font_medium"> <span id="S4.T1.1.1.20.19.6.1.1.1" class="ltx_text" style="color:#0000FF;">(+17.74%)</span></span></span></td>
</tr>
<tr id="S4.T1.1.1.21.20" class="ltx_tr">
<td id="S4.T1.1.1.21.20.1" class="ltx_td ltx_align_left">Snarks</td>
<td id="S4.T1.1.1.21.20.2" class="ltx_td ltx_align_center">44.94</td>
<td id="S4.T1.1.1.21.20.3" class="ltx_td ltx_align_center">45.51</td>
<td id="S4.T1.1.1.21.20.4" class="ltx_td ltx_align_center">47.75</td>
<td id="S4.T1.1.1.21.20.5" class="ltx_td ltx_align_left" style="background-color:#ECF4FF;"><span id="S4.T1.1.1.21.20.5.1" class="ltx_text ltx_font_bold" style="background-color:#ECF4FF;">49.44<span id="S4.T1.1.1.21.20.5.1.1" class="ltx_text ltx_font_medium"> <span id="S4.T1.1.1.21.20.5.1.1.1" class="ltx_text" style="color:#66B2FF;">(+3.54%)</span></span></span></td>
<td id="S4.T1.1.1.21.20.6" class="ltx_td ltx_align_left" style="background-color:#ECF4FF;"><span id="S4.T1.1.1.21.20.6.1" class="ltx_text ltx_font_bold" style="background-color:#ECF4FF;">49.44<span id="S4.T1.1.1.21.20.6.1.1" class="ltx_text ltx_font_medium"> <span id="S4.T1.1.1.21.20.6.1.1.1" class="ltx_text" style="color:#0000FF;">(+3.54%)</span></span></span></td>
</tr>
<tr id="S4.T1.1.1.22.21" class="ltx_tr">
<td id="S4.T1.1.1.22.21.1" class="ltx_td ltx_align_left">Sports Understanding</td>
<td id="S4.T1.1.1.22.21.2" class="ltx_td ltx_align_center">64.40</td>
<td id="S4.T1.1.1.22.21.3" class="ltx_td ltx_align_center">82.40</td>
<td id="S4.T1.1.1.22.21.4" class="ltx_td ltx_align_center">90.00</td>
<td id="S4.T1.1.1.22.21.5" class="ltx_td ltx_align_left" style="background-color:#ECF4FF;"><span id="S4.T1.1.1.22.21.5.1" class="ltx_text" style="background-color:#ECF4FF;">90.00 <span id="S4.T1.1.1.22.21.5.1.1" class="ltx_text" style="color:#66B2FF;">(+0.00%)</span></span></td>
<td id="S4.T1.1.1.22.21.6" class="ltx_td ltx_align_left" style="background-color:#ECF4FF;"><span id="S4.T1.1.1.22.21.6.1" class="ltx_text ltx_font_bold" style="background-color:#ECF4FF;">91.20<span id="S4.T1.1.1.22.21.6.1.1" class="ltx_text ltx_font_medium"> <span id="S4.T1.1.1.22.21.6.1.1.1" class="ltx_text" style="color:#0000FF;">(+1.33%)</span></span></span></td>
</tr>
<tr id="S4.T1.1.1.23.22" class="ltx_tr">
<td id="S4.T1.1.1.23.22.1" class="ltx_td ltx_align_left">Temporal Sequences</td>
<td id="S4.T1.1.1.23.22.2" class="ltx_td ltx_align_center"><span id="S4.T1.1.1.23.22.2.1" class="ltx_text ltx_font_bold">32.00</span></td>
<td id="S4.T1.1.1.23.22.3" class="ltx_td ltx_align_center">21.20</td>
<td id="S4.T1.1.1.23.22.4" class="ltx_td ltx_align_center">12.80</td>
<td id="S4.T1.1.1.23.22.5" class="ltx_td ltx_align_left" style="background-color:#ECF4FF;"><span id="S4.T1.1.1.23.22.5.1" class="ltx_text" style="background-color:#ECF4FF;">16.40 <span id="S4.T1.1.1.23.22.5.1.1" class="ltx_text" style="color:#66B2FF;">(+28.13%)</span></span></td>
<td id="S4.T1.1.1.23.22.6" class="ltx_td ltx_align_left" style="background-color:#ECF4FF;"><span id="S4.T1.1.1.23.22.6.1" class="ltx_text" style="background-color:#ECF4FF;">16.40 <span id="S4.T1.1.1.23.22.6.1.1" class="ltx_text" style="color:#0000FF;">(+28.13%)</span></span></td>
</tr>
<tr id="S4.T1.1.1.24.23" class="ltx_tr">
<td id="S4.T1.1.1.24.23.1" class="ltx_td ltx_align_left">Tracking Shuffled Objects (3 objects)</td>
<td id="S4.T1.1.1.24.23.2" class="ltx_td ltx_align_center"><span id="S4.T1.1.1.24.23.2.1" class="ltx_text ltx_font_bold">36.40</span></td>
<td id="S4.T1.1.1.24.23.3" class="ltx_td ltx_align_center">30.40</td>
<td id="S4.T1.1.1.24.23.4" class="ltx_td ltx_align_center">33.20</td>
<td id="S4.T1.1.1.24.23.5" class="ltx_td ltx_align_left" style="background-color:#ECF4FF;"><span id="S4.T1.1.1.24.23.5.1" class="ltx_text" style="background-color:#ECF4FF;">33.20 <span id="S4.T1.1.1.24.23.5.1.1" class="ltx_text" style="color:#66B2FF;">(+3.61%)</span></span></td>
<td id="S4.T1.1.1.24.23.6" class="ltx_td ltx_align_left" style="background-color:#ECF4FF;"><span id="S4.T1.1.1.24.23.6.1" class="ltx_text" style="background-color:#ECF4FF;">34.40 <span id="S4.T1.1.1.24.23.6.1.1" class="ltx_text" style="color:#0000FF;">(+3.61%)</span></span></td>
</tr>
<tr id="S4.T1.1.1.25.24" class="ltx_tr">
<td id="S4.T1.1.1.25.24.1" class="ltx_td ltx_align_left">Tracking Shuffled Objects (5 objects)</td>
<td id="S4.T1.1.1.25.24.2" class="ltx_td ltx_align_center"><span id="S4.T1.1.1.25.24.2.1" class="ltx_text ltx_font_bold">19.20</span></td>
<td id="S4.T1.1.1.25.24.3" class="ltx_td ltx_align_center">14.40</td>
<td id="S4.T1.1.1.25.24.4" class="ltx_td ltx_align_center">15.60</td>
<td id="S4.T1.1.1.25.24.5" class="ltx_td ltx_align_left" style="background-color:#ECF4FF;"><span id="S4.T1.1.1.25.24.5.1" class="ltx_text" style="background-color:#ECF4FF;">15.20 <span id="S4.T1.1.1.25.24.5.1.1" class="ltx_text" style="color:#FF7171;">(-2.56%)</span></span></td>
<td id="S4.T1.1.1.25.24.6" class="ltx_td ltx_align_left" style="background-color:#ECF4FF;"><span id="S4.T1.1.1.25.24.6.1" class="ltx_text" style="background-color:#ECF4FF;">15.60 <span id="S4.T1.1.1.25.24.6.1.1" class="ltx_text" style="color:#0000FF;">(+0.00%)</span></span></td>
</tr>
<tr id="S4.T1.1.1.26.25" class="ltx_tr">
<td id="S4.T1.1.1.26.25.1" class="ltx_td ltx_align_left">Tracking Shuffled Objects (7 objects)</td>
<td id="S4.T1.1.1.26.25.2" class="ltx_td ltx_align_center">10.80</td>
<td id="S4.T1.1.1.26.25.3" class="ltx_td ltx_align_center">2.00</td>
<td id="S4.T1.1.1.26.25.4" class="ltx_td ltx_align_center"><span id="S4.T1.1.1.26.25.4.1" class="ltx_text ltx_font_bold">11.20</span></td>
<td id="S4.T1.1.1.26.25.5" class="ltx_td ltx_align_left" style="background-color:#ECF4FF;"><span id="S4.T1.1.1.26.25.5.1" class="ltx_text" style="background-color:#ECF4FF;">9.60 <span id="S4.T1.1.1.26.25.5.1.1" class="ltx_text" style="color:#FF7171;">(-14.29%)</span></span></td>
<td id="S4.T1.1.1.26.25.6" class="ltx_td ltx_align_left" style="background-color:#ECF4FF;"><span id="S4.T1.1.1.26.25.6.1" class="ltx_text" style="background-color:#ECF4FF;">10.40 <span id="S4.T1.1.1.26.25.6.1.1" class="ltx_text" style="color:#FF0000;">(-7.14%)</span></span></td>
</tr>
<tr id="S4.T1.1.1.27.26" class="ltx_tr">
<td id="S4.T1.1.1.27.26.1" class="ltx_td ltx_align_left">Web of Lies</td>
<td id="S4.T1.1.1.27.26.2" class="ltx_td ltx_align_center">51.60</td>
<td id="S4.T1.1.1.27.26.3" class="ltx_td ltx_align_center">63.60</td>
<td id="S4.T1.1.1.27.26.4" class="ltx_td ltx_align_center">50.80</td>
<td id="S4.T1.1.1.27.26.5" class="ltx_td ltx_align_left" style="background-color:#ECF4FF;"><span id="S4.T1.1.1.27.26.5.1" class="ltx_text" style="background-color:#ECF4FF;">61.60 <span id="S4.T1.1.1.27.26.5.1.1" class="ltx_text" style="color:#66B2FF;">(+21.26%)</span></span></td>
<td id="S4.T1.1.1.27.26.6" class="ltx_td ltx_align_left" style="background-color:#ECF4FF;"><span id="S4.T1.1.1.27.26.6.1" class="ltx_text ltx_font_bold" style="background-color:#ECF4FF;">65.60<span id="S4.T1.1.1.27.26.6.1.1" class="ltx_text ltx_font_medium"> <span id="S4.T1.1.1.27.26.6.1.1.1" class="ltx_text" style="color:#0000FF;">(+29.13%)</span></span></span></td>
</tr>
<tr id="S4.T1.1.1.28.27" class="ltx_tr">
<td id="S4.T1.1.1.28.27.1" class="ltx_td ltx_align_left">Word Sorting</td>
<td id="S4.T1.1.1.28.27.2" class="ltx_td ltx_align_center">5.60</td>
<td id="S4.T1.1.1.28.27.3" class="ltx_td ltx_align_center">6.80</td>
<td id="S4.T1.1.1.28.27.4" class="ltx_td ltx_align_center"><span id="S4.T1.1.1.28.27.4.1" class="ltx_text ltx_font_bold">12.80</span></td>
<td id="S4.T1.1.1.28.27.5" class="ltx_td ltx_align_left" style="background-color:#ECF4FF;"><span id="S4.T1.1.1.28.27.5.1" class="ltx_text" style="background-color:#ECF4FF;">7.60 <span id="S4.T1.1.1.28.27.5.1.1" class="ltx_text" style="color:#FF7171;">(-40.63%)</span></span></td>
<td id="S4.T1.1.1.28.27.6" class="ltx_td ltx_align_left" style="background-color:#ECF4FF;"><span id="S4.T1.1.1.28.27.6.1" class="ltx_text" style="background-color:#ECF4FF;">7.60 <span id="S4.T1.1.1.28.27.6.1.1" class="ltx_text" style="color:#FF0000;">(-40.63%)</span></span></td>
</tr>
<tr id="S4.T1.1.1.29.28" class="ltx_tr">
<td id="S4.T1.1.1.29.28.1" class="ltx_td ltx_align_left ltx_border_bb ltx_border_t">Avg. 27 Tasks</td>
<td id="S4.T1.1.1.29.28.2" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t">33.87</td>
<td id="S4.T1.1.1.29.28.3" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t">33.38</td>
<td id="S4.T1.1.1.29.28.4" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t">39.70</td>
<td id="S4.T1.1.1.29.28.5" class="ltx_td ltx_align_left ltx_border_bb ltx_border_t" style="background-color:#ECF4FF;"><span id="S4.T1.1.1.29.28.5.1" class="ltx_text" style="background-color:#ECF4FF;">40.44 <span id="S4.T1.1.1.29.28.5.1.1" class="ltx_text" style="color:#66B2FF;">(+1.86%)</span></span></td>
<td id="S4.T1.1.1.29.28.6" class="ltx_td ltx_align_left ltx_border_bb ltx_border_t" style="background-color:#ECF4FF;"><span id="S4.T1.1.1.29.28.6.1" class="ltx_text ltx_font_bold" style="background-color:#ECF4FF;">41.75<span id="S4.T1.1.1.29.28.6.1.1" class="ltx_text ltx_font_medium"> <span id="S4.T1.1.1.29.28.6.1.1.1" class="ltx_text" style="color:#0000FF;">(+5.16%)</span></span></span></td>
</tr>
</tbody>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 1: </span>Overall results of <span id="S4.T1.3.1" class="ltx_text ltx_font_smallcaps">FuseLLM</span> and baselines in reasoning evaluations on Big-Bench Hard (BBH), where percentages indicate the rate of improvement/decrease compared to Llama-2.</figcaption>
</figure>
<figure id="S4.T2" class="ltx_table">
<div id="S4.T2.1" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:318.0pt;height:101.2pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(-38.9pt,12.4pt) scale(0.803508801692787,0.803508801692787) ;">
<table id="S4.T2.1.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S4.T2.1.1.1.1" class="ltx_tr">
<th id="S4.T2.1.1.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_tt"><span id="S4.T2.1.1.1.1.1.1" class="ltx_text ltx_font_bold">Task</span></th>
<th id="S4.T2.1.1.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S4.T2.1.1.1.1.2.1" class="ltx_text ltx_font_bold">OpenLLaMA</span></th>
<th id="S4.T2.1.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S4.T2.1.1.1.1.3.1" class="ltx_text ltx_font_bold">MPT</span></th>
<th id="S4.T2.1.1.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S4.T2.1.1.1.1.4.1" class="ltx_text ltx_font_bold">Llama-2</span></th>
<th id="S4.T2.1.1.1.1.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="background-color:#ECF4FF;"><span id="S4.T2.1.1.1.1.5.1" class="ltx_text ltx_font_bold" style="background-color:#ECF4FF;">Llama-2 CLM</span></th>
<th id="S4.T2.1.1.1.1.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="background-color:#ECF4FF;"><span id="S4.T2.1.1.1.1.6.1" class="ltx_text ltx_font_bold ltx_font_smallcaps" style="background-color:#ECF4FF;">FuseLLM</span></th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S4.T2.1.1.2.1" class="ltx_tr">
<th id="S4.T2.1.1.2.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_tt">ARC-easy</th>
<td id="S4.T2.1.1.2.1.2" class="ltx_td ltx_align_center ltx_border_tt">69.70</td>
<td id="S4.T2.1.1.2.1.3" class="ltx_td ltx_align_center ltx_border_tt">70.12</td>
<td id="S4.T2.1.1.2.1.4" class="ltx_td ltx_align_center ltx_border_tt">74.58</td>
<td id="S4.T2.1.1.2.1.5" class="ltx_td ltx_align_center ltx_border_tt" style="background-color:#ECF4FF;"><span id="S4.T2.1.1.2.1.5.1" class="ltx_text" style="background-color:#ECF4FF;">74.54 <span id="S4.T2.1.1.2.1.5.1.1" class="ltx_text" style="color:#FF7171;">(-0.05%)</span></span></td>
<td id="S4.T2.1.1.2.1.6" class="ltx_td ltx_align_center ltx_border_tt" style="background-color:#ECF4FF;"><span id="S4.T2.1.1.2.1.6.1" class="ltx_text ltx_font_bold" style="background-color:#ECF4FF;">75.04<span id="S4.T2.1.1.2.1.6.1.1" class="ltx_text ltx_font_medium"> <span id="S4.T2.1.1.2.1.6.1.1.1" class="ltx_text" style="color:#0000FF;">(+0.62%)</span></span></span></td>
</tr>
<tr id="S4.T2.1.1.3.2" class="ltx_tr">
<th id="S4.T2.1.1.3.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">ARC-challenge</th>
<td id="S4.T2.1.1.3.2.2" class="ltx_td ltx_align_center">41.38</td>
<td id="S4.T2.1.1.3.2.3" class="ltx_td ltx_align_center">42.15</td>
<td id="S4.T2.1.1.3.2.4" class="ltx_td ltx_align_center">46.33</td>
<td id="S4.T2.1.1.3.2.5" class="ltx_td ltx_align_center" style="background-color:#ECF4FF;"><span id="S4.T2.1.1.3.2.5.1" class="ltx_text" style="background-color:#ECF4FF;">46.50 <span id="S4.T2.1.1.3.2.5.1.1" class="ltx_text" style="color:#66B2FF;">(+0.37%)</span></span></td>
<td id="S4.T2.1.1.3.2.6" class="ltx_td ltx_align_center" style="background-color:#ECF4FF;"><span id="S4.T2.1.1.3.2.6.1" class="ltx_text ltx_font_bold" style="background-color:#ECF4FF;">47.44<span id="S4.T2.1.1.3.2.6.1.1" class="ltx_text ltx_font_medium"> <span id="S4.T2.1.1.3.2.6.1.1.1" class="ltx_text" style="color:#0000FF;">(+2.40%)</span></span></span></td>
</tr>
<tr id="S4.T2.1.1.4.3" class="ltx_tr">
<th id="S4.T2.1.1.4.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">BoolQ</th>
<td id="S4.T2.1.1.4.3.2" class="ltx_td ltx_align_center">72.29</td>
<td id="S4.T2.1.1.4.3.3" class="ltx_td ltx_align_center">74.74</td>
<td id="S4.T2.1.1.4.3.4" class="ltx_td ltx_align_center">77.71</td>
<td id="S4.T2.1.1.4.3.5" class="ltx_td ltx_align_center" style="background-color:#ECF4FF;"><span id="S4.T2.1.1.4.3.5.1" class="ltx_text" style="background-color:#ECF4FF;">76.88 <span id="S4.T2.1.1.4.3.5.1.1" class="ltx_text" style="color:#FF7171;">(-1.07%)</span></span></td>
<td id="S4.T2.1.1.4.3.6" class="ltx_td ltx_align_center" style="background-color:#ECF4FF;"><span id="S4.T2.1.1.4.3.6.1" class="ltx_text ltx_font_bold" style="background-color:#ECF4FF;">78.13<span id="S4.T2.1.1.4.3.6.1.1" class="ltx_text ltx_font_medium"> <span id="S4.T2.1.1.4.3.6.1.1.1" class="ltx_text" style="color:#0000FF;">(+0.54%)</span></span></span></td>
</tr>
<tr id="S4.T2.1.1.5.4" class="ltx_tr">
<th id="S4.T2.1.1.5.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">HellaSwag</th>
<td id="S4.T2.1.1.5.4.2" class="ltx_td ltx_align_center">74.53</td>
<td id="S4.T2.1.1.5.4.3" class="ltx_td ltx_align_center">76.25</td>
<td id="S4.T2.1.1.5.4.4" class="ltx_td ltx_align_center">76.00</td>
<td id="S4.T2.1.1.5.4.5" class="ltx_td ltx_align_center" style="background-color:#ECF4FF;"><span id="S4.T2.1.1.5.4.5.1" class="ltx_text" style="background-color:#ECF4FF;">76.57 <span id="S4.T2.1.1.5.4.5.1.1" class="ltx_text" style="color:#66B2FF;">(+0.75%)</span></span></td>
<td id="S4.T2.1.1.5.4.6" class="ltx_td ltx_align_center" style="background-color:#ECF4FF;"><span id="S4.T2.1.1.5.4.6.1" class="ltx_text ltx_font_bold" style="background-color:#ECF4FF;">76.78<span id="S4.T2.1.1.5.4.6.1.1" class="ltx_text ltx_font_medium"> <span id="S4.T2.1.1.5.4.6.1.1.1" class="ltx_text" style="color:#0000FF;">(+1.03%)</span></span></span></td>
</tr>
<tr id="S4.T2.1.1.6.5" class="ltx_tr">
<th id="S4.T2.1.1.6.5.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">OpenBookQA</th>
<td id="S4.T2.1.1.6.5.2" class="ltx_td ltx_align_center">41.00</td>
<td id="S4.T2.1.1.6.5.3" class="ltx_td ltx_align_center">42.40</td>
<td id="S4.T2.1.1.6.5.4" class="ltx_td ltx_align_center">44.20</td>
<td id="S4.T2.1.1.6.5.5" class="ltx_td ltx_align_center" style="background-color:#ECF4FF;"><span id="S4.T2.1.1.6.5.5.1" class="ltx_text" style="background-color:#ECF4FF;">44.80 <span id="S4.T2.1.1.6.5.5.1.1" class="ltx_text" style="color:#66B2FF;">(+1.36%)</span></span></td>
<td id="S4.T2.1.1.6.5.6" class="ltx_td ltx_align_center" style="background-color:#ECF4FF;"><span id="S4.T2.1.1.6.5.6.1" class="ltx_text ltx_font_bold" style="background-color:#ECF4FF;">45.40<span id="S4.T2.1.1.6.5.6.1.1" class="ltx_text ltx_font_medium"> <span id="S4.T2.1.1.6.5.6.1.1.1" class="ltx_text" style="color:#0000FF;">(+2.71%)</span></span></span></td>
</tr>
<tr id="S4.T2.1.1.7.6" class="ltx_tr">
<th id="S4.T2.1.1.7.6.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb ltx_border_t">Avg. 5 Tasks</th>
<td id="S4.T2.1.1.7.6.2" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t">59.78</td>
<td id="S4.T2.1.1.7.6.3" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t">61.13</td>
<td id="S4.T2.1.1.7.6.4" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t">63.76</td>
<td id="S4.T2.1.1.7.6.5" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t" style="background-color:#ECF4FF;"><span id="S4.T2.1.1.7.6.5.1" class="ltx_text" style="background-color:#ECF4FF;">63.86 <span id="S4.T2.1.1.7.6.5.1.1" class="ltx_text" style="color:#66B2FF;">(+0.16%)</span></span></td>
<td id="S4.T2.1.1.7.6.6" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t" style="background-color:#ECF4FF;"><span id="S4.T2.1.1.7.6.6.1" class="ltx_text ltx_font_bold" style="background-color:#ECF4FF;">64.56<span id="S4.T2.1.1.7.6.6.1.1" class="ltx_text ltx_font_medium"> <span id="S4.T2.1.1.7.6.6.1.1.1" class="ltx_text" style="color:#0000FF;">(+1.25%)</span></span></span></td>
</tr>
</tbody>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 2: </span>Overall results of <span id="S4.T2.3.1" class="ltx_text ltx_font_smallcaps">FuseLLM</span> and baselines in commonsense evaluations on CommenSense (CS), where percentages indicate the rate of improvement/decrease compared to Llama-2.</figcaption>
</figure>
<figure id="S4.T3" class="ltx_table">
<div id="S4.T3.1" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:318.0pt;height:172.8pt;vertical-align:-0.8pt;"><span class="ltx_transformed_inner" style="transform:translate(-40.7pt,22.0pt) scale(0.796298556381354,0.796298556381354) ;">
<table id="S4.T3.1.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S4.T3.1.1.1.1" class="ltx_tr">
<th id="S4.T3.1.1.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_tt"><span id="S4.T3.1.1.1.1.1.1" class="ltx_text ltx_font_bold">Task</span></th>
<th id="S4.T3.1.1.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S4.T3.1.1.1.1.2.1" class="ltx_text ltx_font_bold">OpenLLaMA</span></th>
<th id="S4.T3.1.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S4.T3.1.1.1.1.3.1" class="ltx_text ltx_font_bold">MPT</span></th>
<th id="S4.T3.1.1.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S4.T3.1.1.1.1.4.1" class="ltx_text ltx_font_bold">Llama-2</span></th>
<th id="S4.T3.1.1.1.1.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="background-color:#ECF4FF;"><span id="S4.T3.1.1.1.1.5.1" class="ltx_text ltx_font_bold" style="background-color:#ECF4FF;">Llama-2 CLM</span></th>
<th id="S4.T3.1.1.1.1.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="background-color:#ECF4FF;"><span id="S4.T3.1.1.1.1.6.1" class="ltx_text ltx_font_bold ltx_font_smallcaps" style="background-color:#ECF4FF;">FuseLLM</span></th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S4.T3.1.1.2.1" class="ltx_tr">
<th id="S4.T3.1.1.2.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_tt">C++</th>
<td id="S4.T3.1.1.2.1.2" class="ltx_td ltx_align_center ltx_border_tt"><span id="S4.T3.1.1.2.1.2.1" class="ltx_text ltx_font_bold">14.47</span></td>
<td id="S4.T3.1.1.2.1.3" class="ltx_td ltx_align_center ltx_border_tt">13.11</td>
<td id="S4.T3.1.1.2.1.4" class="ltx_td ltx_align_center ltx_border_tt">7.45</td>
<td id="S4.T3.1.1.2.1.5" class="ltx_td ltx_align_center ltx_border_tt" style="background-color:#ECF4FF;"><span id="S4.T3.1.1.2.1.5.1" class="ltx_text" style="background-color:#ECF4FF;">9.88 <span id="S4.T3.1.1.2.1.5.1.1" class="ltx_text" style="color:#66B2FF;">(+32.62%)</span></span></td>
<td id="S4.T3.1.1.2.1.6" class="ltx_td ltx_align_center ltx_border_tt" style="background-color:#ECF4FF;"><span id="S4.T3.1.1.2.1.6.1" class="ltx_text" style="background-color:#ECF4FF;">9.25 <span id="S4.T3.1.1.2.1.6.1.1" class="ltx_text" style="color:#0000FF;">(+24.16%)</span></span></td>
</tr>
<tr id="S4.T3.1.1.3.2" class="ltx_tr">
<th id="S4.T3.1.1.3.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">Go</th>
<td id="S4.T3.1.1.3.2.2" class="ltx_td ltx_align_center"><span id="S4.T3.1.1.3.2.2.1" class="ltx_text ltx_font_bold">68.20</span></td>
<td id="S4.T3.1.1.3.2.3" class="ltx_td ltx_align_center">66.96</td>
<td id="S4.T3.1.1.3.2.4" class="ltx_td ltx_align_center">57.02</td>
<td id="S4.T3.1.1.3.2.5" class="ltx_td ltx_align_center" style="background-color:#ECF4FF;"><span id="S4.T3.1.1.3.2.5.1" class="ltx_text" style="background-color:#ECF4FF;">54.44 <span id="S4.T3.1.1.3.2.5.1.1" class="ltx_text" style="color:#FF7171;">(-4.52%)</span></span></td>
<td id="S4.T3.1.1.3.2.6" class="ltx_td ltx_align_center" style="background-color:#ECF4FF;"><span id="S4.T3.1.1.3.2.6.1" class="ltx_text" style="background-color:#ECF4FF;">59.78 <span id="S4.T3.1.1.3.2.6.1.1" class="ltx_text" style="color:#0000FF;">(+4.84%)</span></span></td>
</tr>
<tr id="S4.T3.1.1.4.3" class="ltx_tr">
<th id="S4.T3.1.1.4.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">Java</th>
<td id="S4.T3.1.1.4.3.2" class="ltx_td ltx_align_center"><span id="S4.T3.1.1.4.3.2.1" class="ltx_text ltx_font_bold">14.28</span></td>
<td id="S4.T3.1.1.4.3.3" class="ltx_td ltx_align_center">13.42</td>
<td id="S4.T3.1.1.4.3.4" class="ltx_td ltx_align_center">10.31</td>
<td id="S4.T3.1.1.4.3.5" class="ltx_td ltx_align_center" style="background-color:#ECF4FF;"><span id="S4.T3.1.1.4.3.5.1" class="ltx_text" style="background-color:#ECF4FF;">10.50 <span id="S4.T3.1.1.4.3.5.1.1" class="ltx_text" style="color:#66B2FF;">(+1.84%)</span></span></td>
<td id="S4.T3.1.1.4.3.6" class="ltx_td ltx_align_center" style="background-color:#ECF4FF;"><span id="S4.T3.1.1.4.3.6.1" class="ltx_text" style="background-color:#ECF4FF;">10.34 <span id="S4.T3.1.1.4.3.6.1.1" class="ltx_text" style="color:#0000FF;">(+0.29%)</span></span></td>
</tr>
<tr id="S4.T3.1.1.5.4" class="ltx_tr">
<th id="S4.T3.1.1.5.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">JavaScript</th>
<td id="S4.T3.1.1.5.4.2" class="ltx_td ltx_align_center"><span id="S4.T3.1.1.5.4.2.1" class="ltx_text ltx_font_bold">17.61</span></td>
<td id="S4.T3.1.1.5.4.3" class="ltx_td ltx_align_center">13.01</td>
<td id="S4.T3.1.1.5.4.4" class="ltx_td ltx_align_center">13.17</td>
<td id="S4.T3.1.1.5.4.5" class="ltx_td ltx_align_center" style="background-color:#ECF4FF;"><span id="S4.T3.1.1.5.4.5.1" class="ltx_text" style="background-color:#ECF4FF;">14.25 <span id="S4.T3.1.1.5.4.5.1.1" class="ltx_text" style="color:#66B2FF;">(+8.20%)</span></span></td>
<td id="S4.T3.1.1.5.4.6" class="ltx_td ltx_align_center" style="background-color:#ECF4FF;"><span id="S4.T3.1.1.5.4.6.1" class="ltx_text" style="background-color:#ECF4FF;">14.32 <span id="S4.T3.1.1.5.4.6.1.1" class="ltx_text" style="color:#0000FF;">(+8.73%)</span></span></td>
</tr>
<tr id="S4.T3.1.1.6.5" class="ltx_tr">
<th id="S4.T3.1.1.6.5.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">PHP</th>
<td id="S4.T3.1.1.6.5.2" class="ltx_td ltx_align_center"><span id="S4.T3.1.1.6.5.2.1" class="ltx_text ltx_font_bold">11.24</span></td>
<td id="S4.T3.1.1.6.5.3" class="ltx_td ltx_align_center">9.53</td>
<td id="S4.T3.1.1.6.5.4" class="ltx_td ltx_align_center">9.75</td>
<td id="S4.T3.1.1.6.5.5" class="ltx_td ltx_align_center" style="background-color:#ECF4FF;"><span id="S4.T3.1.1.6.5.5.1" class="ltx_text" style="background-color:#ECF4FF;">9.04 <span id="S4.T3.1.1.6.5.5.1.1" class="ltx_text" style="color:#FF7171;">(-7.28%)</span></span></td>
<td id="S4.T3.1.1.6.5.6" class="ltx_td ltx_align_center" style="background-color:#ECF4FF;"><span id="S4.T3.1.1.6.5.6.1" class="ltx_text" style="background-color:#ECF4FF;">9.41 <span id="S4.T3.1.1.6.5.6.1.1" class="ltx_text" style="color:#FF0000;">(-3.49%)</span></span></td>
</tr>
<tr id="S4.T3.1.1.7.6" class="ltx_tr">
<th id="S4.T3.1.1.7.6.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">Python</th>
<td id="S4.T3.1.1.7.6.2" class="ltx_td ltx_align_center">15.96</td>
<td id="S4.T3.1.1.7.6.3" class="ltx_td ltx_align_center"><span id="S4.T3.1.1.7.6.3.1" class="ltx_text ltx_font_bold">17.24</span></td>
<td id="S4.T3.1.1.7.6.4" class="ltx_td ltx_align_center">13.85</td>
<td id="S4.T3.1.1.7.6.5" class="ltx_td ltx_align_center" style="background-color:#ECF4FF;"><span id="S4.T3.1.1.7.6.5.1" class="ltx_text" style="background-color:#ECF4FF;">13.07 <span id="S4.T3.1.1.7.6.5.1.1" class="ltx_text" style="color:#FF7171;">(-5.63%)</span></span></td>
<td id="S4.T3.1.1.7.6.6" class="ltx_td ltx_align_center" style="background-color:#ECF4FF;"><span id="S4.T3.1.1.7.6.6.1" class="ltx_text" style="background-color:#ECF4FF;">13.91 <span id="S4.T3.1.1.7.6.6.1.1" class="ltx_text" style="color:#0000FF;">(+0.43%)</span></span></td>
</tr>
<tr id="S4.T3.1.1.8.7" class="ltx_tr">
<th id="S4.T3.1.1.8.7.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">R</th>
<td id="S4.T3.1.1.8.7.2" class="ltx_td ltx_align_center"><span id="S4.T3.1.1.8.7.2.1" class="ltx_text ltx_font_bold">7.52</span></td>
<td id="S4.T3.1.1.8.7.3" class="ltx_td ltx_align_center">4.53</td>
<td id="S4.T3.1.1.8.7.4" class="ltx_td ltx_align_center">4.97</td>
<td id="S4.T3.1.1.8.7.5" class="ltx_td ltx_align_center" style="background-color:#ECF4FF;"><span id="S4.T3.1.1.8.7.5.1" class="ltx_text" style="background-color:#ECF4FF;">5.25 <span id="S4.T3.1.1.8.7.5.1.1" class="ltx_text" style="color:#66B2FF;">(+5.63%)</span></span></td>
<td id="S4.T3.1.1.8.7.6" class="ltx_td ltx_align_center" style="background-color:#ECF4FF;"><span id="S4.T3.1.1.8.7.6.1" class="ltx_text" style="background-color:#ECF4FF;">5.84 <span id="S4.T3.1.1.8.7.6.1.1" class="ltx_text" style="color:#0000FF;">(+17.51%)</span></span></td>
</tr>
<tr id="S4.T3.1.1.9.8" class="ltx_tr">
<th id="S4.T3.1.1.9.8.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">Ruby</th>
<td id="S4.T3.1.1.9.8.2" class="ltx_td ltx_align_center">10.34</td>
<td id="S4.T3.1.1.9.8.3" class="ltx_td ltx_align_center"><span id="S4.T3.1.1.9.8.3.1" class="ltx_text ltx_font_bold">12.33</span></td>
<td id="S4.T3.1.1.9.8.4" class="ltx_td ltx_align_center">10.37</td>
<td id="S4.T3.1.1.9.8.5" class="ltx_td ltx_align_center" style="background-color:#ECF4FF;"><span id="S4.T3.1.1.9.8.5.1" class="ltx_text" style="background-color:#ECF4FF;">10.68 <span id="S4.T3.1.1.9.8.5.1.1" class="ltx_text" style="color:#66B2FF;">(+2.99%)</span></span></td>
<td id="S4.T3.1.1.9.8.6" class="ltx_td ltx_align_center" style="background-color:#ECF4FF;"><span id="S4.T3.1.1.9.8.6.1" class="ltx_text" style="background-color:#ECF4FF;">11.24 <span id="S4.T3.1.1.9.8.6.1.1" class="ltx_text" style="color:#0000FF;">(+8.39%)</span></span></td>
</tr>
<tr id="S4.T3.1.1.10.9" class="ltx_tr">
<th id="S4.T3.1.1.10.9.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">Rust</th>
<td id="S4.T3.1.1.10.9.2" class="ltx_td ltx_align_center">6.18</td>
<td id="S4.T3.1.1.10.9.3" class="ltx_td ltx_align_center"><span id="S4.T3.1.1.10.9.3.1" class="ltx_text ltx_font_bold">8.29</span></td>
<td id="S4.T3.1.1.10.9.4" class="ltx_td ltx_align_center">6.77</td>
<td id="S4.T3.1.1.10.9.5" class="ltx_td ltx_align_center" style="background-color:#ECF4FF;"><span id="S4.T3.1.1.10.9.5.1" class="ltx_text" style="background-color:#ECF4FF;">6.96 <span id="S4.T3.1.1.10.9.5.1.1" class="ltx_text" style="color:#66B2FF;">(+2.81%)</span></span></td>
<td id="S4.T3.1.1.10.9.6" class="ltx_td ltx_align_center" style="background-color:#ECF4FF;"><span id="S4.T3.1.1.10.9.6.1" class="ltx_text" style="background-color:#ECF4FF;">7.05 <span id="S4.T3.1.1.10.9.6.1.1" class="ltx_text" style="color:#0000FF;">(+4.14%)</span></span></td>
</tr>
<tr id="S4.T3.1.1.11.10" class="ltx_tr">
<th id="S4.T3.1.1.11.10.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">TypeScript</th>
<td id="S4.T3.1.1.11.10.2" class="ltx_td ltx_align_center"><span id="S4.T3.1.1.11.10.2.1" class="ltx_text ltx_font_bold">15.31</span></td>
<td id="S4.T3.1.1.11.10.3" class="ltx_td ltx_align_center">14.13</td>
<td id="S4.T3.1.1.11.10.4" class="ltx_td ltx_align_center">12.61</td>
<td id="S4.T3.1.1.11.10.5" class="ltx_td ltx_align_center" style="background-color:#ECF4FF;"><span id="S4.T3.1.1.11.10.5.1" class="ltx_text" style="background-color:#ECF4FF;">14.19 <span id="S4.T3.1.1.11.10.5.1.1" class="ltx_text" style="color:#66B2FF;">(+12.53%)</span></span></td>
<td id="S4.T3.1.1.11.10.6" class="ltx_td ltx_align_center" style="background-color:#ECF4FF;"><span id="S4.T3.1.1.11.10.6.1" class="ltx_text" style="background-color:#ECF4FF;">14.50 <span id="S4.T3.1.1.11.10.6.1.1" class="ltx_text" style="color:#0000FF;">(+14.99%)</span></span></td>
</tr>
<tr id="S4.T3.1.1.12.11" class="ltx_tr">
<th id="S4.T3.1.1.12.11.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb ltx_border_t">Avg. 10 Tasks</th>
<td id="S4.T3.1.1.12.11.2" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t"><span id="S4.T3.1.1.12.11.2.1" class="ltx_text ltx_font_bold">18.11</span></td>
<td id="S4.T3.1.1.12.11.3" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t">17.26</td>
<td id="S4.T3.1.1.12.11.4" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t">14.63</td>
<td id="S4.T3.1.1.12.11.5" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t" style="background-color:#ECF4FF;"><span id="S4.T3.1.1.12.11.5.1" class="ltx_text" style="background-color:#ECF4FF;">14.83 <span id="S4.T3.1.1.12.11.5.1.1" class="ltx_text" style="color:#66B2FF;">(+1.37%)</span></span></td>
<td id="S4.T3.1.1.12.11.6" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t" style="background-color:#ECF4FF;"><span id="S4.T3.1.1.12.11.6.1" class="ltx_text" style="background-color:#ECF4FF;">15.56 <span id="S4.T3.1.1.12.11.6.1.1" class="ltx_text" style="color:#0000FF;">(+6.36%)</span></span></td>
</tr>
</tbody>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 3: </span>Overall results of <span id="S4.T3.3.1" class="ltx_text ltx_font_smallcaps">FuseLLM</span> and baselines in code generation evaluations on MultiPL-E (ME), where percentages indicate the rate of improvement/decrease compared to Llama-2.</figcaption>
</figure>
<div id="S4.SS2.p3" class="ltx_para ltx_noindent">
<p id="S4.SS2.p3.1" class="ltx_p">For the code generation evaluation, the zero-shot performance of <span id="S4.SS2.p3.1.1" class="ltx_text ltx_font_smallcaps">FuseLLM</span> on the MultiPL-E (ME) benchmark is reported in Table <a href="#S4.T3" title="Table 3 ‣ 4.2 Overall Results ‣ 4 Experiments ‣ Knowledge Fusion of Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>. We observe that <span id="S4.SS2.p3.1.2" class="ltx_text ltx_font_smallcaps">FuseLLM</span> outperforms Llama-2 in 9 out of the 10 tasks, with a notable enhancement in the pass@1 score for specific programming languages such as R, increasing from 4.97 to 5.84. Given that both OpenLLaMA and MPT demonstrate remarkable performances in code generation tasks compared to Llama-2, the fusion result via <span id="S4.SS2.p3.1.3" class="ltx_text ltx_font_smallcaps">FuseLLM</span> achieves an average performance gain of 6.36%, which is considerably higher than the 1.37% improvement observed in Llama-2 CLM. However, it’s important to note that <span id="S4.SS2.p3.1.4" class="ltx_text ltx_font_smallcaps">FuseLLM</span> still exhibits a performance gap compared to OpenLLaMA or MPT in this evaluation. This discrepancy can be attributed to two primary reasons: the inferior performances of Llama-2 as the target model compared to other source LLMs and an insufficient proportion of coding-related texts in the continual training corpus, estimated at approximately 7.59%<span id="footnote1" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">1</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">1</sup><span class="ltx_tag ltx_tag_note">1</span>Since MiniPile lacks specific data percentages for individual domains, we approximate this by considering the percentage of the Github domain in The Pile.</span></span></span>.</p>
</div>
</section>
<section id="S4.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.3 </span>The Fused Probabilistic Distributions</h3>

<figure id="S4.F2" class="ltx_figure ltx_align_floatright"><img src="/html/2401.10491/assets/x2.png" id="S4.F2.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="438" height="267" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 2: </span>Effect of the fused distributions in accelerating the optimization process on BBH, where the x-axis denotes the number of training tokens and the y-axis denotes the exact match accuracy.</figcaption>
</figure>
<div id="S4.SS3.p1" class="ltx_para ltx_noindent">
<p id="S4.SS3.p1.1" class="ltx_p">We investigate the effectiveness of the fused probabilistic distributions obtained from multiple LLMs and track the trend of performance improvement during the training process. Figure <a href="#S4.F2" title="Figure 2 ‣ 4.3 The Fused Probabilistic Distributions ‣ 4 Experiments ‣ Knowledge Fusion of Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a> illustrates the comparison of few-shot CoT performance between Llama-2 CLM and <span id="S4.SS3.p1.1.1" class="ltx_text ltx_font_smallcaps">FuseLLM</span> with varying scales of training data on BBH. Our observations reveal that <span id="S4.SS3.p1.1.2" class="ltx_text ltx_font_smallcaps">FuseLLM</span> enhances the exact match (EM) accuracy by 2.5% compared to Llama-2 CLM and achieves the best performance of Llama-2 CLM within 0.52 billion tokens. Notably, this represents a <math id="S4.SS3.p1.1.m1.1" class="ltx_math_unparsed" alttext="3.9\times" display="inline"><semantics id="S4.SS3.p1.1.m1.1a"><mrow id="S4.SS3.p1.1.m1.1b"><mn id="S4.SS3.p1.1.m1.1.1">3.9</mn><mo lspace="0.222em" id="S4.SS3.p1.1.m1.1.2">×</mo></mrow><annotation encoding="application/x-tex" id="S4.SS3.p1.1.m1.1c">3.9\times</annotation></semantics></math> reduction in token requirements compared to the 1.57 billion tokens needed by Llama-2 CLM. These results suggest that the probabilistic distributions derived from LLMs contain knowledge that is more readily learnable than the original text sequences, which accelerates the optimization process. This finding aligns with the observations in <cite class="ltx_cite ltx_citemacro_cite">Hsieh et al. (<a href="#bib.bib24" title="" class="ltx_ref">2023</a>)</cite>. We further conduct an experiment to show that our performance improvement stems from the integration of knowledge from multiple LLMs rather than solely from continual training. The results and analysis are shown in Appendix <a href="#A7" title="Appendix G Cause of performance improvement ‣ Knowledge Fusion of Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">G</span></a>.</p>
</div>
</section>
<section id="S4.SS4" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.4 </span>Analysis of Implementation Process</h3>

<div id="S4.SS4.p1" class="ltx_para ltx_noindent">
<p id="S4.SS4.p1.1" class="ltx_p">In this section, we delve into the crucial elements of <span id="S4.SS4.p1.1.1" class="ltx_text ltx_font_smallcaps">FuseLLM</span>’s implementation, including the number of source LLMs, the criteria for token alignment, and the choice of the fusion function.</p>
</div>
<figure id="S4.T4" class="ltx_table ltx_align_floatright">
<div id="S4.T4.1" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:429.3pt;height:168.9pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(31.7pt,-12.5pt) scale(1.1731775229217,1.1731775229217) ;">
<table id="S4.T4.1.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S4.T4.1.1.1.1" class="ltx_tr">
<th id="S4.T4.1.1.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_tt"><span id="S4.T4.1.1.1.1.1.1" class="ltx_text ltx_font_bold">Model</span></th>
<th id="S4.T4.1.1.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S4.T4.1.1.1.1.2.1" class="ltx_text ltx_font_bold">BBH</span></th>
<th id="S4.T4.1.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S4.T4.1.1.1.1.3.1" class="ltx_text ltx_font_bold">CS</span></th>
<th id="S4.T4.1.1.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S4.T4.1.1.1.1.4.1" class="ltx_text ltx_font_bold">ME</span></th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S4.T4.1.1.2.1" class="ltx_tr">
<th id="S4.T4.1.1.2.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_tt">OpenLLaMA</th>
<td id="S4.T4.1.1.2.1.2" class="ltx_td ltx_align_center ltx_border_tt">33.87</td>
<td id="S4.T4.1.1.2.1.3" class="ltx_td ltx_align_center ltx_border_tt">59.78</td>
<td id="S4.T4.1.1.2.1.4" class="ltx_td ltx_align_center ltx_border_tt">18.11</td>
</tr>
<tr id="S4.T4.1.1.3.2" class="ltx_tr">
<th id="S4.T4.1.1.3.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">MPT</th>
<td id="S4.T4.1.1.3.2.2" class="ltx_td ltx_align_center">33.38</td>
<td id="S4.T4.1.1.3.2.3" class="ltx_td ltx_align_center">61.13</td>
<td id="S4.T4.1.1.3.2.4" class="ltx_td ltx_align_center">17.26</td>
</tr>
<tr id="S4.T4.1.1.4.3" class="ltx_tr">
<th id="S4.T4.1.1.4.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">Llama-2</th>
<td id="S4.T4.1.1.4.3.2" class="ltx_td ltx_align_center">39.70</td>
<td id="S4.T4.1.1.4.3.3" class="ltx_td ltx_align_center">63.76</td>
<td id="S4.T4.1.1.4.3.4" class="ltx_td ltx_align_center">14.63</td>
</tr>
<tr id="S4.T4.1.1.5.4" class="ltx_tr" style="background-color:#ECF4FF;">
<th id="S4.T4.1.1.5.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="S4.T4.1.1.5.4.1.1" class="ltx_text" style="background-color:#ECF4FF;">Llama-2 CLM</span></th>
<td id="S4.T4.1.1.5.4.2" class="ltx_td ltx_align_center"><span id="S4.T4.1.1.5.4.2.1" class="ltx_text" style="background-color:#ECF4FF;">40.44 <span id="S4.T4.1.1.5.4.2.1.1" class="ltx_text" style="color:#0000FF;">(+1.86%)</span></span></td>
<td id="S4.T4.1.1.5.4.3" class="ltx_td ltx_align_center"><span id="S4.T4.1.1.5.4.3.1" class="ltx_text" style="background-color:#ECF4FF;">63.86 <span id="S4.T4.1.1.5.4.3.1.1" class="ltx_text" style="color:#0000FF;">(+0.16%)</span></span></td>
<td id="S4.T4.1.1.5.4.4" class="ltx_td ltx_align_center"><span id="S4.T4.1.1.5.4.4.1" class="ltx_text" style="background-color:#ECF4FF;">14.83 <span id="S4.T4.1.1.5.4.4.1.1" class="ltx_text" style="color:#0000FF;">(+1.37%)</span></span></td>
</tr>
<tr id="S4.T4.1.1.6.5" class="ltx_tr" style="background-color:#ECF4FF;">
<th id="S4.T4.1.1.6.5.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="S4.T4.1.1.6.5.1.1" class="ltx_text" style="background-color:#ECF4FF;">Llama-2 + OpenLLaMA</span></th>
<td id="S4.T4.1.1.6.5.2" class="ltx_td ltx_align_center"><span id="S4.T4.1.1.6.5.2.1" class="ltx_text" style="background-color:#ECF4FF;">41.00 <span id="S4.T4.1.1.6.5.2.1.1" class="ltx_text" style="color:#0000FF;">(+3.27%)</span></span></td>
<td id="S4.T4.1.1.6.5.3" class="ltx_td ltx_align_center"><span id="S4.T4.1.1.6.5.3.1" class="ltx_text" style="background-color:#ECF4FF;">64.50 <span id="S4.T4.1.1.6.5.3.1.1" class="ltx_text" style="color:#0000FF;">(+1.16%)</span></span></td>
<td id="S4.T4.1.1.6.5.4" class="ltx_td ltx_align_center"><span id="S4.T4.1.1.6.5.4.1" class="ltx_text" style="background-color:#ECF4FF;">15.51 <span id="S4.T4.1.1.6.5.4.1.1" class="ltx_text" style="color:#0000FF;">(+6.02%)</span></span></td>
</tr>
<tr id="S4.T4.1.1.7.6" class="ltx_tr" style="background-color:#ECF4FF;">
<th id="S4.T4.1.1.7.6.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="S4.T4.1.1.7.6.1.1" class="ltx_text" style="background-color:#ECF4FF;">Llama-2 + MPT</span></th>
<td id="S4.T4.1.1.7.6.2" class="ltx_td ltx_align_center"><span id="S4.T4.1.1.7.6.2.1" class="ltx_text" style="background-color:#ECF4FF;">41.16 <span id="S4.T4.1.1.7.6.2.1.1" class="ltx_text" style="color:#0000FF;">(+3.68%)</span></span></td>
<td id="S4.T4.1.1.7.6.3" class="ltx_td ltx_align_center"><span id="S4.T4.1.1.7.6.3.1" class="ltx_text" style="background-color:#ECF4FF;">64.51 <span id="S4.T4.1.1.7.6.3.1.1" class="ltx_text" style="color:#0000FF;">(+1.18%)</span></span></td>
<td id="S4.T4.1.1.7.6.4" class="ltx_td ltx_align_center"><span id="S4.T4.1.1.7.6.4.1" class="ltx_text" style="background-color:#ECF4FF;">15.47 <span id="S4.T4.1.1.7.6.4.1.1" class="ltx_text" style="color:#0000FF;">(+5.74%)</span></span></td>
</tr>
<tr id="S4.T4.1.1.8.7" class="ltx_tr" style="background-color:#ECF4FF;">
<th id="S4.T4.1.1.8.7.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb"><span id="S4.T4.1.1.8.7.1.1" class="ltx_text ltx_font_smallcaps" style="background-color:#ECF4FF;">FuseLLM</span></th>
<td id="S4.T4.1.1.8.7.2" class="ltx_td ltx_align_center ltx_border_bb"><span id="S4.T4.1.1.8.7.2.1" class="ltx_text" style="background-color:#ECF4FF;">41.75 <span id="S4.T4.1.1.8.7.2.1.1" class="ltx_text" style="color:#0000FF;">(+5.16%)</span></span></td>
<td id="S4.T4.1.1.8.7.3" class="ltx_td ltx_align_center ltx_border_bb"><span id="S4.T4.1.1.8.7.3.1" class="ltx_text" style="background-color:#ECF4FF;">64.56 <span id="S4.T4.1.1.8.7.3.1.1" class="ltx_text" style="color:#0000FF;">(+1.25%)</span></span></td>
<td id="S4.T4.1.1.8.7.4" class="ltx_td ltx_align_center ltx_border_bb"><span id="S4.T4.1.1.8.7.4.1" class="ltx_text" style="background-color:#ECF4FF;">15.56 <span id="S4.T4.1.1.8.7.4.1.1" class="ltx_text" style="color:#0000FF;">(+6.36%)</span></span></td>
</tr>
</tbody>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 4: </span>Results of <span id="S4.T4.3.1" class="ltx_text ltx_font_smallcaps">FuseLLM</span> by incorporating varying numbers of models.</figcaption>
</figure>
<section id="S4.SS4.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Number of source LLMs.</h4>

<div id="S4.SS4.SSS0.Px1.p1" class="ltx_para ltx_noindent">
<p id="S4.SS4.SSS0.Px1.p1.1" class="ltx_p">In Table <a href="#S4.T4" title="Table 4 ‣ 4.4 Analysis of Implementation Process ‣ 4 Experiments ‣ Knowledge Fusion of Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>, we present the results of fusing different numbers of LLMs. We note that the performance of <span id="S4.SS4.SSS0.Px1.p1.1.1" class="ltx_text ltx_font_smallcaps">FuseLLM</span> demonstrates apparent improvement as the number of models increases from 1 to 3. Nevertheless, the benefits of integrating additional models exhibit variations across benchmarks. Remarkably, a consistent performance improvement is observed in BBH. Whereas in CS or ME, the advantages are more prominent when fusing two models. This phenomenon may be attributed to the considerable performance differences among the three models on various tasks in BBH, while the performance differences in tasks of CS or ME are relatively smaller.</p>
</div>
</section>
<section id="S4.SS4.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Criteria for token alignment.</h4>

<div id="S4.SS4.SSS0.Px2.p1" class="ltx_para ltx_noindent">
<p id="S4.SS4.SSS0.Px2.p1.1" class="ltx_p">During the fusion of LLMs, ensuring the proper alignment of tokens and vocabularies from multiple models is of paramount importance. In Table <a href="#S4.T5" title="Table 5 ‣ Criteria for token alignment. ‣ 4.4 Analysis of Implementation Process ‣ 4 Experiments ‣ Knowledge Fusion of Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a> (upper), we present a comparison of two alignment criteria. It is evident that the proposed MinED method, which is based on minimum edit distance, consistently outperforms the EM method introduced by <cite class="ltx_cite ltx_citemacro_cite">Fu et al. (<a href="#bib.bib17" title="" class="ltx_ref">2023</a>)</cite>, which relies on exact matching. We suggest that this performance enhancement results from MinED’s ability to relax the constraints of EM, as tokens separated by distinct tokenizers within the same sequence often exhibit minor discrepancies. Consequently, MinED effectively supplements a considerable amount of useful token information while introducing negligible errors.</p>
</div>
<figure id="S4.T5" class="ltx_table ltx_align_floatright">
<div id="S4.T5.1" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:429.3pt;height:186.6pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(69.7pt,-30.3pt) scale(1.48057748720242,1.48057748720242) ;">
<table id="S4.T5.1.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="S4.T5.1.1.1.1" class="ltx_tr">
<th id="S4.T5.1.1.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_tt"><span id="S4.T5.1.1.1.1.1.1" class="ltx_text ltx_font_bold">Choice</span></th>
<td id="S4.T5.1.1.1.1.2" class="ltx_td ltx_align_center ltx_border_tt"><span id="S4.T5.1.1.1.1.2.1" class="ltx_text ltx_font_bold">BBH</span></td>
<td id="S4.T5.1.1.1.1.3" class="ltx_td ltx_align_center ltx_border_tt"><span id="S4.T5.1.1.1.1.3.1" class="ltx_text ltx_font_bold">ME</span></td>
<td id="S4.T5.1.1.1.1.4" class="ltx_td ltx_align_center ltx_border_tt"><span id="S4.T5.1.1.1.1.4.1" class="ltx_text ltx_font_bold">CS</span></td>
</tr>
<tr id="S4.T5.1.1.2.2" class="ltx_tr">
<th id="S4.T5.1.1.2.2.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_tt" colspan="4"><em id="S4.T5.1.1.2.2.1.1" class="ltx_emph ltx_font_italic">Alignment Criteria</em></th>
</tr>
<tr id="S4.T5.1.1.3.3" class="ltx_tr">
<th id="S4.T5.1.1.3.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">EM</th>
<td id="S4.T5.1.1.3.3.2" class="ltx_td ltx_align_center ltx_border_t">41.57</td>
<td id="S4.T5.1.1.3.3.3" class="ltx_td ltx_align_center ltx_border_t">15.49</td>
<td id="S4.T5.1.1.3.3.4" class="ltx_td ltx_align_center ltx_border_t">64.24</td>
</tr>
<tr id="S4.T5.1.1.4.4" class="ltx_tr">
<th id="S4.T5.1.1.4.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">MinED</th>
<td id="S4.T5.1.1.4.4.2" class="ltx_td ltx_align_center">41.75 <span id="S4.T5.1.1.4.4.2.1" class="ltx_text" style="color:#0000FF;">(+0.43%)</span>
</td>
<td id="S4.T5.1.1.4.4.3" class="ltx_td ltx_align_center">15.56 <span id="S4.T5.1.1.4.4.3.1" class="ltx_text" style="color:#0000FF;">(+0.45%)</span>
</td>
<td id="S4.T5.1.1.4.4.4" class="ltx_td ltx_align_center">64.56 <span id="S4.T5.1.1.4.4.4.1" class="ltx_text" style="color:#0000FF;">(+0.50%)</span>
</td>
</tr>
<tr id="S4.T5.1.1.5.5" class="ltx_tr">
<th id="S4.T5.1.1.5.5.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_tt" colspan="4"><em id="S4.T5.1.1.5.5.1.1" class="ltx_emph ltx_font_italic">Fusion Function</em></th>
</tr>
<tr id="S4.T5.1.1.6.6" class="ltx_tr">
<th id="S4.T5.1.1.6.6.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">AvgCE</th>
<td id="S4.T5.1.1.6.6.2" class="ltx_td ltx_align_center ltx_border_t">41.04</td>
<td id="S4.T5.1.1.6.6.3" class="ltx_td ltx_align_center ltx_border_t">15.39</td>
<td id="S4.T5.1.1.6.6.4" class="ltx_td ltx_align_center ltx_border_t">63.98</td>
</tr>
<tr id="S4.T5.1.1.7.7" class="ltx_tr">
<th id="S4.T5.1.1.7.7.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb">MinCE</th>
<td id="S4.T5.1.1.7.7.2" class="ltx_td ltx_align_center ltx_border_bb">41.75 <span id="S4.T5.1.1.7.7.2.1" class="ltx_text" style="color:#0000FF;">(+1.73%)</span>
</td>
<td id="S4.T5.1.1.7.7.3" class="ltx_td ltx_align_center ltx_border_bb">15.56 <span id="S4.T5.1.1.7.7.3.1" class="ltx_text" style="color:#0000FF;">(+1.10%)</span>
</td>
<td id="S4.T5.1.1.7.7.4" class="ltx_td ltx_align_center ltx_border_bb">64.56 <span id="S4.T5.1.1.7.7.4.1" class="ltx_text" style="color:#0000FF;">(+0.91%)</span>
</td>
</tr>
</tbody>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 5: </span>Comparison of different token alignment criteria (upper) and fusion functions (down).</figcaption>
</figure>
</section>
<section id="S4.SS4.SSS0.Px3" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Fusion function.</h4>

<div id="S4.SS4.SSS0.Px3.p1" class="ltx_para ltx_noindent">
<p id="S4.SS4.SSS0.Px3.p1.1" class="ltx_p">In Section <a href="#S3.SS3" title="3.3 Implementation of FuseLLM ‣ 3 Knowledge Fusion of LLMs ‣ Knowledge Fusion of Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.3</span></a>, we introduce two variations of the fusion function for <span id="S4.SS4.SSS0.Px3.p1.1.1" class="ltx_text ltx_font_smallcaps">FuseLLM</span>: one utilizing a distribution matrix with minimum cross entropy (MinCE) and the other adopting a weighted average of distribution matrices based on cross entropy (AvgCE). A comparison of the two functions is presented in Table <a href="#S4.T5" title="Table 5 ‣ Criteria for token alignment. ‣ 4.4 Analysis of Implementation Process ‣ 4 Experiments ‣ Knowledge Fusion of Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a> (down). The findings demonstrate that <span id="S4.SS4.SSS0.Px3.p1.1.2" class="ltx_text ltx_font_smallcaps">FuseLLM</span> with MinCE consistently outperforms AvgCE across all benchmarks. This can be attributed to the distortions introduced by the straightforward weighted summation used in AvgCE, which may diminish the distinct advantages of individual LLMs.</p>
</div>
</section>
</section>
<section id="S4.SS5" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.5 </span><span id="S4.SS5.1.1" class="ltx_text ltx_font_smallcaps">FuseLLM</span> vs. Knowledge Distillation</h3>

<figure id="S4.T6" class="ltx_table ltx_align_floatright">
<div id="S4.T6.1" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:429.3pt;height:193.4pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(54.8pt,-24.7pt) scale(1.34291522750662,1.34291522750662) ;">
<table id="S4.T6.1.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S4.T6.1.1.1.1" class="ltx_tr">
<th id="S4.T6.1.1.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_tt"><span id="S4.T6.1.1.1.1.1.1" class="ltx_text ltx_font_bold">Model</span></th>
<th id="S4.T6.1.1.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S4.T6.1.1.1.1.2.1" class="ltx_text ltx_font_bold">BBH</span></th>
<th id="S4.T6.1.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S4.T6.1.1.1.1.3.1" class="ltx_text ltx_font_bold">CS</span></th>
<th id="S4.T6.1.1.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S4.T6.1.1.1.1.4.1" class="ltx_text ltx_font_bold">ME</span></th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S4.T6.1.1.2.1" class="ltx_tr">
<th id="S4.T6.1.1.2.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_tt">Llama-2 13B</th>
<td id="S4.T6.1.1.2.1.2" class="ltx_td ltx_align_center ltx_border_tt">47.92</td>
<td id="S4.T6.1.1.2.1.3" class="ltx_td ltx_align_center ltx_border_tt">66.33</td>
<td id="S4.T6.1.1.2.1.4" class="ltx_td ltx_align_center ltx_border_tt">18.76</td>
</tr>
<tr id="S4.T6.1.1.3.2" class="ltx_tr">
<th id="S4.T6.1.1.3.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">OpenLLaMA</th>
<td id="S4.T6.1.1.3.2.2" class="ltx_td ltx_align_center">33.87</td>
<td id="S4.T6.1.1.3.2.3" class="ltx_td ltx_align_center">59.78</td>
<td id="S4.T6.1.1.3.2.4" class="ltx_td ltx_align_center">18.11</td>
</tr>
<tr id="S4.T6.1.1.4.3" class="ltx_tr">
<th id="S4.T6.1.1.4.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">MPT</th>
<td id="S4.T6.1.1.4.3.2" class="ltx_td ltx_align_center">33.38</td>
<td id="S4.T6.1.1.4.3.3" class="ltx_td ltx_align_center">61.13</td>
<td id="S4.T6.1.1.4.3.4" class="ltx_td ltx_align_center">17.26</td>
</tr>
<tr id="S4.T6.1.1.5.4" class="ltx_tr">
<th id="S4.T6.1.1.5.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">Llama-2</th>
<td id="S4.T6.1.1.5.4.2" class="ltx_td ltx_align_center">39.70</td>
<td id="S4.T6.1.1.5.4.3" class="ltx_td ltx_align_center">63.76</td>
<td id="S4.T6.1.1.5.4.4" class="ltx_td ltx_align_center">14.63</td>
</tr>
<tr id="S4.T6.1.1.6.5" class="ltx_tr" style="background-color:#ECF4FF;">
<th id="S4.T6.1.1.6.5.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="S4.T6.1.1.6.5.1.1" class="ltx_text" style="background-color:#ECF4FF;">Llama-2 CLM</span></th>
<td id="S4.T6.1.1.6.5.2" class="ltx_td ltx_align_center"><span id="S4.T6.1.1.6.5.2.1" class="ltx_text" style="background-color:#ECF4FF;">40.44 <span id="S4.T6.1.1.6.5.2.1.1" class="ltx_text" style="color:#0000FF;">(+1.86%)</span></span></td>
<td id="S4.T6.1.1.6.5.3" class="ltx_td ltx_align_center"><span id="S4.T6.1.1.6.5.3.1" class="ltx_text" style="background-color:#ECF4FF;">63.86 <span id="S4.T6.1.1.6.5.3.1.1" class="ltx_text" style="color:#0000FF;">(+0.16%)</span></span></td>
<td id="S4.T6.1.1.6.5.4" class="ltx_td ltx_align_center"><span id="S4.T6.1.1.6.5.4.1" class="ltx_text" style="background-color:#ECF4FF;">14.83 <span id="S4.T6.1.1.6.5.4.1.1" class="ltx_text" style="color:#0000FF;">(+1.37%)</span></span></td>
</tr>
<tr id="S4.T6.1.1.7.6" class="ltx_tr" style="background-color:#ECF4FF;">
<th id="S4.T6.1.1.7.6.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="S4.T6.1.1.7.6.1.1" class="ltx_text" style="background-color:#ECF4FF;">Llama-2 KD</span></th>
<td id="S4.T6.1.1.7.6.2" class="ltx_td ltx_align_center"><span id="S4.T6.1.1.7.6.2.1" class="ltx_text" style="background-color:#ECF4FF;">40.88 <span id="S4.T6.1.1.7.6.2.1.1" class="ltx_text" style="color:#0000FF;">(+2.97%)</span></span></td>
<td id="S4.T6.1.1.7.6.3" class="ltx_td ltx_align_center"><span id="S4.T6.1.1.7.6.3.1" class="ltx_text" style="background-color:#ECF4FF;">64.41 <span id="S4.T6.1.1.7.6.3.1.1" class="ltx_text" style="color:#0000FF;">(+1.02%)</span></span></td>
<td id="S4.T6.1.1.7.6.4" class="ltx_td ltx_align_center"><span id="S4.T6.1.1.7.6.4.1" class="ltx_text" style="background-color:#ECF4FF;">15.45 <span id="S4.T6.1.1.7.6.4.1.1" class="ltx_text" style="color:#0000FF;">(+5.60%)</span></span></td>
</tr>
<tr id="S4.T6.1.1.8.7" class="ltx_tr" style="background-color:#ECF4FF;">
<th id="S4.T6.1.1.8.7.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb"><span id="S4.T6.1.1.8.7.1.1" class="ltx_text ltx_font_smallcaps" style="background-color:#ECF4FF;">FuseLLM</span></th>
<td id="S4.T6.1.1.8.7.2" class="ltx_td ltx_align_center ltx_border_bb"><span id="S4.T6.1.1.8.7.2.1" class="ltx_text" style="background-color:#ECF4FF;">41.75 <span id="S4.T6.1.1.8.7.2.1.1" class="ltx_text" style="color:#0000FF;">(+5.16%)</span></span></td>
<td id="S4.T6.1.1.8.7.3" class="ltx_td ltx_align_center ltx_border_bb"><span id="S4.T6.1.1.8.7.3.1" class="ltx_text" style="background-color:#ECF4FF;">64.56 <span id="S4.T6.1.1.8.7.3.1.1" class="ltx_text" style="color:#0000FF;">(+1.25%)</span></span></td>
<td id="S4.T6.1.1.8.7.4" class="ltx_td ltx_align_center ltx_border_bb"><span id="S4.T6.1.1.8.7.4.1" class="ltx_text" style="background-color:#ECF4FF;">15.56 <span id="S4.T6.1.1.8.7.4.1.1" class="ltx_text" style="color:#0000FF;">(+6.36%)</span></span></td>
</tr>
</tbody>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 6: </span>Comparison of <span id="S4.T6.3.1" class="ltx_text ltx_font_smallcaps">FuseLLM</span> and knowledge distillation. Llama-2 KD denotes the enhanced Llama-2 7B achieved via knowledge distillation from Llama-2 13B. Percentages indicate the rate of improvement compared to Llama-2.</figcaption>
</figure>
<div id="S4.SS5.p1" class="ltx_para ltx_noindent">
<p id="S4.SS5.p1.1" class="ltx_p">While knowledge distillation techniques can also be utilized to enhance a LLM’s capabilities, <span id="S4.SS5.p1.1.1" class="ltx_text ltx_font_smallcaps">FuseLLM</span> stands out due to two distinct aspects, as previously outlined. In this section, we compare <span id="S4.SS5.p1.1.2" class="ltx_text ltx_font_smallcaps">FuseLLM</span> with traditional knowledge distillation. Specifically, we extract probabilistic distributions from Llama-2 13B and apply the conventional knowledge distillation method to transfer its abilities into Llama-2 7B. As illustrated in Table <a href="#S4.T6" title="Table 6 ‣ 4.5 FuseLLM vs. Knowledge Distillation ‣ 4 Experiments ‣ Knowledge Fusion of Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a>, the distilled model (Llama-2 KD) outperforms the original Llama-2 7B across all benchmarks, demonstrating the effectiveness of knowledge distillation. However, when compared to <span id="S4.SS5.p1.1.3" class="ltx_text ltx_font_smallcaps">FuseLLM</span>, the improvement achieved by Llama-2 KD is relatively modest, especially in the case of BBH (2.97% vs. 5.16%). This suggests that the superior results achieved by <span id="S4.SS5.p1.1.4" class="ltx_text ltx_font_smallcaps">FuseLLM</span> through the integration of three 7B models with diverse architectures via continual training outweigh the benefits of simply distilling knowledge from a single 13B model. This observation highlights the idea that “More is different, but different can also be more” <cite class="ltx_cite ltx_citemacro_citep">(Tay et al., <a href="#bib.bib51" title="" class="ltx_ref">2022</a>)</cite>.</p>
</div>
</section>
<section id="S4.SS6" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.6 </span><span id="S4.SS6.1.1" class="ltx_text ltx_font_smallcaps">FuseLLM</span> vs. Ensemble/Merging</h3>

<figure id="S4.T7" class="ltx_table ltx_align_floatright">
<div id="S4.T7.1" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:429.3pt;height:235.6pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(83.4pt,-45.8pt) scale(1.63589249697359,1.63589249697359) ;">
<table id="S4.T7.1.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S4.T7.1.1.1.1" class="ltx_tr">
<th id="S4.T7.1.1.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_tt"><span id="S4.T7.1.1.1.1.1.1" class="ltx_text ltx_font_bold">Model</span></th>
<th id="S4.T7.1.1.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S4.T7.1.1.1.1.2.1" class="ltx_text ltx_font_bold">Phil</span></th>
<th id="S4.T7.1.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S4.T7.1.1.1.1.3.1" class="ltx_text ltx_font_bold">NIH</span></th>
<th id="S4.T7.1.1.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S4.T7.1.1.1.1.4.1" class="ltx_text ltx_font_bold">USPTO</span></th>
<th id="S4.T7.1.1.1.1.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S4.T7.1.1.1.1.5.1" class="ltx_text ltx_font_bold">Average</span></th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S4.T7.1.1.2.1" class="ltx_tr">
<th id="S4.T7.1.1.2.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_tt">Pythia</th>
<td id="S4.T7.1.1.2.1.2" class="ltx_td ltx_align_center ltx_border_tt">0.9008</td>
<td id="S4.T7.1.1.2.1.3" class="ltx_td ltx_align_center ltx_border_tt">0.6740</td>
<td id="S4.T7.1.1.2.1.4" class="ltx_td ltx_align_center ltx_border_tt">0.6077</td>
<td id="S4.T7.1.1.2.1.5" class="ltx_td ltx_align_center ltx_border_tt">0.7275</td>
</tr>
<tr id="S4.T7.1.1.3.2" class="ltx_tr">
<th id="S4.T7.1.1.3.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">Phil</th>
<td id="S4.T7.1.1.3.2.2" class="ltx_td ltx_align_center"><span id="S4.T7.1.1.3.2.2.1" class="ltx_text ltx_font_bold">0.8397</span></td>
<td id="S4.T7.1.1.3.2.3" class="ltx_td ltx_align_center">0.6861</td>
<td id="S4.T7.1.1.3.2.4" class="ltx_td ltx_align_center">0.6228</td>
<td id="S4.T7.1.1.3.2.5" class="ltx_td ltx_align_center">0.7162</td>
</tr>
<tr id="S4.T7.1.1.4.3" class="ltx_tr">
<th id="S4.T7.1.1.4.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">NIH</th>
<td id="S4.T7.1.1.4.3.2" class="ltx_td ltx_align_center">0.9248</td>
<td id="S4.T7.1.1.4.3.3" class="ltx_td ltx_align_center"><span id="S4.T7.1.1.4.3.3.1" class="ltx_text ltx_font_bold">0.6215</span></td>
<td id="S4.T7.1.1.4.3.4" class="ltx_td ltx_align_center">0.6278</td>
<td id="S4.T7.1.1.4.3.5" class="ltx_td ltx_align_center">0.7247</td>
</tr>
<tr id="S4.T7.1.1.5.4" class="ltx_tr">
<th id="S4.T7.1.1.5.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">USPTO</th>
<td id="S4.T7.1.1.5.4.2" class="ltx_td ltx_align_center">0.9296</td>
<td id="S4.T7.1.1.5.4.3" class="ltx_td ltx_align_center">0.6872</td>
<td id="S4.T7.1.1.5.4.4" class="ltx_td ltx_align_center"><span id="S4.T7.1.1.5.4.4.1" class="ltx_text ltx_font_bold">0.6017</span></td>
<td id="S4.T7.1.1.5.4.5" class="ltx_td ltx_align_center">0.7395</td>
</tr>
<tr id="S4.T7.1.1.6.5" class="ltx_tr" style="background-color:#ECF4FF;">
<th id="S4.T7.1.1.6.5.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="S4.T7.1.1.6.5.1.1" class="ltx_text" style="background-color:#ECF4FF;">Ensemble</span></th>
<td id="S4.T7.1.1.6.5.2" class="ltx_td ltx_align_center"><span id="S4.T7.1.1.6.5.2.1" class="ltx_text" style="background-color:#ECF4FF;">0.8960</span></td>
<td id="S4.T7.1.1.6.5.3" class="ltx_td ltx_align_center"><span id="S4.T7.1.1.6.5.3.1" class="ltx_text" style="background-color:#ECF4FF;">0.6647</span></td>
<td id="S4.T7.1.1.6.5.4" class="ltx_td ltx_align_center"><span id="S4.T7.1.1.6.5.4.1" class="ltx_text" style="background-color:#ECF4FF;">0.6180</span></td>
<td id="S4.T7.1.1.6.5.5" class="ltx_td ltx_align_center"><span id="S4.T7.1.1.6.5.5.1" class="ltx_text" style="background-color:#ECF4FF;">0.7262</span></td>
</tr>
<tr id="S4.T7.1.1.7.6" class="ltx_tr" style="background-color:#ECF4FF;">
<th id="S4.T7.1.1.7.6.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="S4.T7.1.1.7.6.1.1" class="ltx_text" style="background-color:#ECF4FF;">Weight Merging</span></th>
<td id="S4.T7.1.1.7.6.2" class="ltx_td ltx_align_center"><span id="S4.T7.1.1.7.6.2.1" class="ltx_text" style="background-color:#ECF4FF;">0.8786</span></td>
<td id="S4.T7.1.1.7.6.3" class="ltx_td ltx_align_center"><span id="S4.T7.1.1.7.6.3.1" class="ltx_text" style="background-color:#ECF4FF;">0.6496</span></td>
<td id="S4.T7.1.1.7.6.4" class="ltx_td ltx_align_center"><span id="S4.T7.1.1.7.6.4.1" class="ltx_text" style="background-color:#ECF4FF;">0.6054</span></td>
<td id="S4.T7.1.1.7.6.5" class="ltx_td ltx_align_center"><span id="S4.T7.1.1.7.6.5.1" class="ltx_text" style="background-color:#ECF4FF;">0.7112</span></td>
</tr>
<tr id="S4.T7.1.1.8.7" class="ltx_tr" style="background-color:#ECF4FF;">
<th id="S4.T7.1.1.8.7.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb"><span id="S4.T7.1.1.8.7.1.1" class="ltx_text ltx_font_smallcaps" style="background-color:#ECF4FF;">FuseLLM</span></th>
<td id="S4.T7.1.1.8.7.2" class="ltx_td ltx_align_center ltx_border_bb"><span id="S4.T7.1.1.8.7.2.1" class="ltx_text" style="background-color:#ECF4FF;">0.8463</span></td>
<td id="S4.T7.1.1.8.7.3" class="ltx_td ltx_align_center ltx_border_bb"><span id="S4.T7.1.1.8.7.3.1" class="ltx_text" style="background-color:#ECF4FF;">0.6569</span></td>
<td id="S4.T7.1.1.8.7.4" class="ltx_td ltx_align_center ltx_border_bb"><span id="S4.T7.1.1.8.7.4.1" class="ltx_text" style="background-color:#ECF4FF;">0.6068</span></td>
<td id="S4.T7.1.1.8.7.5" class="ltx_td ltx_align_center ltx_border_bb"><span id="S4.T7.1.1.8.7.5.1" class="ltx_text ltx_font_bold" style="background-color:#ECF4FF;">0.7034</span></td>
</tr>
</tbody>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 7: </span>Comparison of perplexity between <span id="S4.T7.3.1" class="ltx_text ltx_font_smallcaps">FuseLLM</span> and ensemble&amp;weight merging.</figcaption>
</figure>
<div id="S4.SS6.p1" class="ltx_para ltx_noindent">
<p id="S4.SS6.p1.1" class="ltx_p">As previously mentioned, conventional techniques such as model ensemble and weight merging are commonly employed to fuse multiple LLMs. To compare the efficacy of our <span id="S4.SS6.p1.1.1" class="ltx_text ltx_font_smallcaps">FuseLLM</span> with these existing fusion methods, we conduct experiments simulating scenarios where multiple LLMs originated from the same base model but were trained on distinct corpora. We first select three relevant domains (PhilPapers, NIH ExPorter, and USPTO Backgrounds) from The Pile and use 1 billion tokens from each domain to continually train Pythia 1B <cite class="ltx_cite ltx_citemacro_citep">(Biderman et al., <a href="#bib.bib4" title="" class="ltx_ref">2023</a>)</cite>, resulting in three distinct LLMs with identical structures.
Then, we apply different fusion techniques to these LLMs: (1) The ensemble method calculates a weighted average of the probabilities generated by all LLMs, considering the performance of each model; (2) The weight merging method merges multiple LLMs into a single one within the parameter space, with the merging weights determined by model performance; (3)
<span id="S4.SS6.p1.1.2" class="ltx_text ltx_font_smallcaps">FuseLLM</span> undergoes continual training on 0.1 billion tokens sampled from the three domains. The results of perplexity for <span id="S4.SS6.p1.1.3" class="ltx_text ltx_font_smallcaps">FuseLLM</span> and the other fusion methods on the test sets are presented in Table <a href="#S4.T7" title="Table 7 ‣ 4.6 FuseLLM vs. Ensemble/Merging ‣ 4 Experiments ‣ Knowledge Fusion of Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">7</span></a>. We measure perplexity in bits per UTF-8 encoded byte (BPB) following the implementation in The Pile. We observe that after training with 1 billion tokens, the capabilities of the original LLM are transferred to each domain-specific LLM, resulting in decreased performance in other domains. While all fusion techniques can integrate the strengths of diverse models, <span id="S4.SS6.p1.1.4" class="ltx_text ltx_font_smallcaps">FuseLLM</span> consistently achieves the lowest average perplexity across the three domains. This underscores its potential for harnessing collective knowledge more effectively than ensemble and weight merging methods.</p>
</div>
</section>
</section>
<section id="S5" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">5 </span>Conclusion</h2>

<div id="S5.p1" class="ltx_para ltx_noindent">
<p id="S5.p1.1" class="ltx_p">In this study, we have explored the realm of knowledge fusion for LLMs to create a unified model that combines the capabilities and distinctive strengths of multiple structurally diverse LLMs. We introduced a novel method, <span id="S5.p1.1.1" class="ltx_text ltx_font_smallcaps">FuseLLM</span>, which leverages the generative distributions of these source LLMs to externalize their knowledge and employs them in the continual training of the target LLM. Through a series of experiments, we have demonstrated the superiority of <span id="S5.p1.1.2" class="ltx_text ltx_font_smallcaps">FuseLLM</span> over individual source LLMs and established baselines. Notably, in a simulated experiment featuring multiple structurally identical LLMs, <span id="S5.p1.1.3" class="ltx_text ltx_font_smallcaps">FuseLLM</span> has showcased its competitive effectiveness compared to ensemble and weight merging methods. Hence, the domain of LLMs fusion emerges as a more promising avenue for exploration, particularly given the diverse structures and substantial model sizes of LLMs. We believe that these findings will inspire future research endeavors.</p>
</div>
</section>
<section id="Sx1" class="ltx_section">
<h2 class="ltx_title ltx_title_section">Acknowledgements</h2>

<div id="Sx1.p1" class="ltx_para ltx_noindent">
<p id="Sx1.p1.1" class="ltx_p">This work was supported by the National Natural Science Foundation of China (No. 62176270), the Guangdong Basic and Applied Basic Research Foundation (No. 2023A1515012832), and the Tencent AI Lab Rhino-Bird Focused Research Program.</p>
</div>
</section>
<section id="bib" class="ltx_bibliography">
<h2 class="ltx_title ltx_title_bibliography">References</h2>

<ul class="ltx_biblist">
<li id="bib.bib1" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Agarwal et al. (2023)</span>
<span class="ltx_bibblock">
Rishabh Agarwal, Nino Vieillard, Piotr Stanczyk, Sabela Ramos, Matthieu Geist,
and Olivier Bachem.

</span>
<span class="ltx_bibblock">Gkd: Generalized knowledge distillation for auto-regressive sequence
models.

</span>
<span class="ltx_bibblock"><em id="bib.bib1.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2306.13649</em>, 2023.

</span>
</li>
<li id="bib.bib2" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Arpit et al. (2022)</span>
<span class="ltx_bibblock">
Devansh Arpit, Huan Wang, Yingbo Zhou, and Caiming Xiong.

</span>
<span class="ltx_bibblock">Ensemble of averages: Improving model selection and boosting
performance in domain generalization.

</span>
<span class="ltx_bibblock"><em id="bib.bib2.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>,
35:8265–8277, 2022.

</span>
</li>
<li id="bib.bib3" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ben Allal et al. (2022)</span>
<span class="ltx_bibblock">
Loubna Ben Allal, Niklas Muennighoff, Logesh Kumar Umapathi, Ben Lipkin, and
Leandro von Werra.

</span>
<span class="ltx_bibblock">A framework for the evaluation of code generation models, 2022.

</span>
</li>
<li id="bib.bib4" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Biderman et al. (2023)</span>
<span class="ltx_bibblock">
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley,
Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit,
USVSN Sai Prashanth, Edward Raff, et al.

</span>
<span class="ltx_bibblock">Pythia: A suite for analyzing large language models across training
and scaling.

</span>
<span class="ltx_bibblock">In <em id="bib.bib4.1.1" class="ltx_emph ltx_font_italic">International Conference on Machine Learning</em>, pp. 2397–2430. PMLR, 2023.

</span>
</li>
<li id="bib.bib5" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Brown et al. (2020)</span>
<span class="ltx_bibblock">
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla
Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell,
et al.

</span>
<span class="ltx_bibblock">Language models are few-shot learners.

</span>
<span class="ltx_bibblock"><em id="bib.bib5.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>,
33:1877–1901, 2020.

</span>
</li>
<li id="bib.bib6" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Cassano et al. (2022)</span>
<span class="ltx_bibblock">
Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna
Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane
Anderson, Molly Q Feldman, et al.

</span>
<span class="ltx_bibblock">Multipl-e: A scalable and extensible approach to benchmarking neural
code generation.

</span>
<span class="ltx_bibblock"><em id="bib.bib6.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2208.08227</em>, 2022.

</span>
</li>
<li id="bib.bib7" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Cettolo et al. (2017)</span>
<span class="ltx_bibblock">
Mauro Cettolo, Marcello Federico, Luisa Bentivogli, Niehues Jan, Stüker
Sebastian, Sudoh Katsuitho, Yoshino Koichiro, and Federmann Christian.

</span>
<span class="ltx_bibblock">Overview of the iwslt 2017 evaluation campaign.

</span>
<span class="ltx_bibblock">In <em id="bib.bib7.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 14th International Workshop on Spoken
Language Translation</em>, pp.  2–14, 2017.

</span>
</li>
<li id="bib.bib8" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Cha et al. (2021)</span>
<span class="ltx_bibblock">
Junbum Cha, Sanghyuk Chun, Kyungjae Lee, Han-Cheol Cho, Seunghyun Park, Yunsung
Lee, and Sungrae Park.

</span>
<span class="ltx_bibblock">Swad: Domain generalization by seeking flat minima.

</span>
<span class="ltx_bibblock"><em id="bib.bib8.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>,
34:22405–22418, 2021.

</span>
</li>
<li id="bib.bib9" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Chen et al. (2021)</span>
<span class="ltx_bibblock">
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira
Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg
Brockman, et al.

</span>
<span class="ltx_bibblock">Evaluating large language models trained on code.

</span>
<span class="ltx_bibblock"><em id="bib.bib9.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2107.03374</em>, 2021.

</span>
</li>
<li id="bib.bib10" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Chiang et al. (2023)</span>
<span class="ltx_bibblock">
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin
Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and
Eric P. Xing.

</span>
<span class="ltx_bibblock">Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt
quality, March 2023.

</span>
</li>
<li id="bib.bib11" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Clark et al. (2019a)</span>
<span class="ltx_bibblock">
Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael
Collins, and Kristina Toutanova.

</span>
<span class="ltx_bibblock">Boolq: Exploring the surprising difficulty of natural yes/no
questions.

</span>
<span class="ltx_bibblock">In <em id="bib.bib11.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2019 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language
Technologies, Volume 1 (Long and Short Papers)</em>, pp.  2924–2936,
2019a.

</span>
</li>
<li id="bib.bib12" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Clark et al. (2019b)</span>
<span class="ltx_bibblock">
Kevin Clark, Minh-Thang Luong, Urvashi Khandelwal, Christopher D Manning, and
Quoc Le.

</span>
<span class="ltx_bibblock">Bam! born-again multi-task networks for natural language
understanding.

</span>
<span class="ltx_bibblock">In <em id="bib.bib12.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 57th Annual Meeting of the Association
for Computational Linguistics</em>, pp.  5931–5937, 2019b.

</span>
</li>
<li id="bib.bib13" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Clark et al. (2018)</span>
<span class="ltx_bibblock">
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa
Schoenick, and Oyvind Tafjord.

</span>
<span class="ltx_bibblock">Think you have solved question answering? try arc, the ai2 reasoning
challenge.

</span>
<span class="ltx_bibblock"><em id="bib.bib13.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:1803.05457</em>, 2018.

</span>
</li>
<li id="bib.bib14" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Computer (2023)</span>
<span class="ltx_bibblock">
Together Computer.

</span>
<span class="ltx_bibblock">Redpajama: An open source recipe to reproduce llama training dataset,
April 2023.

</span>
</li>
<li id="bib.bib15" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Dao et al. (2022)</span>
<span class="ltx_bibblock">
Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré.

</span>
<span class="ltx_bibblock">Flashattention: Fast and memory-efficient exact attention with
io-awareness.

</span>
<span class="ltx_bibblock"><em id="bib.bib15.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>,
35:16344–16359, 2022.

</span>
</li>
<li id="bib.bib16" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Dua et al. (2019)</span>
<span class="ltx_bibblock">
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and
Matt Gardner.

</span>
<span class="ltx_bibblock">Drop: A reading comprehension benchmark requiring discrete reasoning
over paragraphs.

</span>
<span class="ltx_bibblock">In <em id="bib.bib16.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2019 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language
Technologies, Volume 1 (Long and Short Papers)</em>, pp.  2368–2378, 2019.

</span>
</li>
<li id="bib.bib17" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Fu et al. (2023)</span>
<span class="ltx_bibblock">
Yao Fu, Hao Peng, Litu Ou, Ashish Sabharwal, and Tushar Khot.

</span>
<span class="ltx_bibblock">Specializing smaller language models towards multi-step reasoning.

</span>
<span class="ltx_bibblock"><em id="bib.bib17.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2301.12726</em>, 2023.

</span>
</li>
<li id="bib.bib18" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Gao et al. (2020)</span>
<span class="ltx_bibblock">
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles
Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al.

</span>
<span class="ltx_bibblock">The pile: An 800gb dataset of diverse text for language modeling.

</span>
<span class="ltx_bibblock"><em id="bib.bib18.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2101.00027</em>, 2020.

</span>
</li>
<li id="bib.bib19" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Gao et al. (2021)</span>
<span class="ltx_bibblock">
Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles
Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff,
Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang,
and Andy Zou.

</span>
<span class="ltx_bibblock">A framework for few-shot language model evaluation, sep 2021.

</span>
</li>
<li id="bib.bib20" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Geng &amp; Liu (2023)</span>
<span class="ltx_bibblock">
Xinyang Geng and Hao Liu.

</span>
<span class="ltx_bibblock">Openllama: An open reproduction of llama, May 2023.

</span>
</li>
<li id="bib.bib21" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Gu et al. (2023)</span>
<span class="ltx_bibblock">
Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang.

</span>
<span class="ltx_bibblock">Knowledge distillation of large language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib21.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2306.08543</em>, 2023.

</span>
</li>
<li id="bib.bib22" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Gupta et al. (2020)</span>
<span class="ltx_bibblock">
Vipul Gupta, Santiago Akle Serrano, and Dennis DeCoste.

</span>
<span class="ltx_bibblock">Stochastic weight averaging in parallel: Large-batch training that
generalizes well.

</span>
<span class="ltx_bibblock"><em id="bib.bib22.1.1" class="ltx_emph ltx_font_italic">International Conference on Learning Representations</em>, 2020.

</span>
</li>
<li id="bib.bib23" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hinton et al. (2015)</span>
<span class="ltx_bibblock">
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean.

</span>
<span class="ltx_bibblock">Distilling the knowledge in a neural network.

</span>
<span class="ltx_bibblock"><em id="bib.bib23.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:1503.02531</em>, 2015.

</span>
</li>
<li id="bib.bib24" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hsieh et al. (2023)</span>
<span class="ltx_bibblock">
Cheng-Yu Hsieh, Chun-Liang Li, Chih-Kuan Yeh, Hootan Nakhost, Yasuhisa Fujii,
Alexander Ratner, Ranjay Krishna, Chen-Yu Lee, and Tomas Pfister.

</span>
<span class="ltx_bibblock">Distilling step-by-step! outperforming larger language models with
less training data and smaller model sizes.

</span>
<span class="ltx_bibblock"><em id="bib.bib24.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2305.02301</em>, 2023.

</span>
</li>
<li id="bib.bib25" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Huang et al. (2023)</span>
<span class="ltx_bibblock">
Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang, Chao Du, and Min Lin.

</span>
<span class="ltx_bibblock">Lorahub: Efficient cross-task generalization via dynamic lora
composition.

</span>
<span class="ltx_bibblock"><em id="bib.bib25.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2307.13269</em>, 2023.

</span>
</li>
<li id="bib.bib26" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Jiang et al. (2023)</span>
<span class="ltx_bibblock">
Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin.

</span>
<span class="ltx_bibblock">Llm-blender: Ensembling large language models with pairwise ranking
and generative fusion.

</span>
<span class="ltx_bibblock"><em id="bib.bib26.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2306.02561</em>, 2023.

</span>
</li>
<li id="bib.bib27" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Jiao et al. (2020)</span>
<span class="ltx_bibblock">
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang
Wang, and Qun Liu.

</span>
<span class="ltx_bibblock">Tinybert: Distilling bert for natural language understanding.

</span>
<span class="ltx_bibblock">In <em id="bib.bib27.1.1" class="ltx_emph ltx_font_italic">Findings of the Association for Computational Linguistics:
EMNLP 2020</em>, pp.  4163–4174, 2020.

</span>
</li>
<li id="bib.bib28" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Jin et al. (2022)</span>
<span class="ltx_bibblock">
Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietro, and Pengxiang Cheng.

</span>
<span class="ltx_bibblock">Dataless knowledge fusion by merging weights of language models.

</span>
<span class="ltx_bibblock">In <em id="bib.bib28.1.1" class="ltx_emph ltx_font_italic">The Eleventh International Conference on Learning
Representations</em>, 2022.

</span>
</li>
<li id="bib.bib29" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Joshi et al. (2017)</span>
<span class="ltx_bibblock">
Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer.

</span>
<span class="ltx_bibblock">Triviaqa: A large scale distantly supervised challenge dataset for
reading comprehension.

</span>
<span class="ltx_bibblock">In <em id="bib.bib29.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 55th Annual Meeting of the Association
for Computational Linguistics (Volume 1: Long Papers)</em>, pp.  1601–1611,
2017.

</span>
</li>
<li id="bib.bib30" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Kaddour et al. (2023)</span>
<span class="ltx_bibblock">
Jean Kaddour, Oscar Key, Piotr Nawrot, Pasquale Minervini, and Matt J Kusner.

</span>
<span class="ltx_bibblock">No train no gain: Revisiting efficient training algorithms for
transformer-based language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib30.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2307.06440</em>, 2023.

</span>
</li>
<li id="bib.bib31" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Khanuja et al. (2021)</span>
<span class="ltx_bibblock">
Simran Khanuja, Melvin Johnson, and Partha Talukdar.

</span>
<span class="ltx_bibblock">Mergedistill: Merging language models using pre-trained distillation.

</span>
<span class="ltx_bibblock">In <em id="bib.bib31.1.1" class="ltx_emph ltx_font_italic">Findings of the Association for Computational Linguistics:
ACL-IJCNLP 2021</em>, pp.  2874–2887, 2021.

</span>
</li>
<li id="bib.bib32" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lee et al. (2023)</span>
<span class="ltx_bibblock">
Ariel N Lee, Cole J Hunter, and Nataniel Ruiz.

</span>
<span class="ltx_bibblock">Platypus: Quick, cheap, and powerful refinement of llms.

</span>
<span class="ltx_bibblock"><em id="bib.bib32.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2308.07317</em>, 2023.

</span>
</li>
<li id="bib.bib33" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Li et al. (2022)</span>
<span class="ltx_bibblock">
Margaret Li, Suchin Gururangan, Tim Dettmers, Mike Lewis, Tim Althoff, Noah A
Smith, and Luke Zettlemoyer.

</span>
<span class="ltx_bibblock">Branch-train-merge: Embarrassingly parallel training of expert
language models.

</span>
<span class="ltx_bibblock">In <em id="bib.bib33.1.1" class="ltx_emph ltx_font_italic">First Workshop on Interpolation Regularizers and Beyond at
NeurIPS 2022</em>, 2022.

</span>
</li>
<li id="bib.bib34" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Li et al. (2023)</span>
<span class="ltx_bibblock">
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov,
Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al.

</span>
<span class="ltx_bibblock">Starcoder: may the source be with you!

</span>
<span class="ltx_bibblock"><em id="bib.bib34.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2305.06161</em>, 2023.

</span>
</li>
<li id="bib.bib35" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Littlestone &amp; Warmuth (1994)</span>
<span class="ltx_bibblock">
Nick Littlestone and Manfred K Warmuth.

</span>
<span class="ltx_bibblock">The weighted majority algorithm.

</span>
<span class="ltx_bibblock"><em id="bib.bib35.1.1" class="ltx_emph ltx_font_italic">Information and Computation</em>, 108(2):212–261, 1994.

</span>
</li>
<li id="bib.bib36" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Marion et al. (2023)</span>
<span class="ltx_bibblock">
Max Marion, Ahmet Üstün, Luiza Pozzobon, Alex Wang, Marzieh Fadaee, and
Sara Hooker.

</span>
<span class="ltx_bibblock">When less is more: Investigating data pruning for pretraining llms at
scale.

</span>
<span class="ltx_bibblock"><em id="bib.bib36.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2309.04564</em>, 2023.

</span>
</li>
<li id="bib.bib37" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Mihaylov et al. (2018)</span>
<span class="ltx_bibblock">
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal.

</span>
<span class="ltx_bibblock">Can a suit of armor conduct electricity? a new dataset for open book
question answering.

</span>
<span class="ltx_bibblock">In <em id="bib.bib37.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2018 Conference on Empirical Methods in
Natural Language Processing</em>, pp.  2381–2391, 2018.

</span>
</li>
<li id="bib.bib38" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Monteith et al. (2011)</span>
<span class="ltx_bibblock">
Kristine Monteith, James L Carroll, Kevin Seppi, and Tony Martinez.

</span>
<span class="ltx_bibblock">Turning bayesian model averaging into bayesian model combination.

</span>
<span class="ltx_bibblock">In <em id="bib.bib38.1.1" class="ltx_emph ltx_font_italic">The 2011 International Joint Conference on Neural Networks</em>,
pp.  2657–2663. IEEE, 2011.

</span>
</li>
<li id="bib.bib39" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Paperno et al. (2016)</span>
<span class="ltx_bibblock">
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc-Quan Pham,
Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel
Fernández.

</span>
<span class="ltx_bibblock">The lambada dataset: Word prediction requiring a broad discourse
context.

</span>
<span class="ltx_bibblock">In <em id="bib.bib39.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 54th Annual Meeting of the Association
for Computational Linguistics (Volume 1: Long Papers)</em>, pp.  1525–1534,
2016.

</span>
</li>
<li id="bib.bib40" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Papineni et al. (2002)</span>
<span class="ltx_bibblock">
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu.

</span>
<span class="ltx_bibblock">Bleu: a method for automatic evaluation of machine translation.

</span>
<span class="ltx_bibblock">In <em id="bib.bib40.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 40th annual meeting of the Association
for Computational Linguistics</em>, pp.  311–318, 2002.

</span>
</li>
<li id="bib.bib41" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Peng et al. (2023)</span>
<span class="ltx_bibblock">
Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao.

</span>
<span class="ltx_bibblock">Instruction tuning with gpt-4.

</span>
<span class="ltx_bibblock"><em id="bib.bib41.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2304.03277</em>, 2023.

</span>
</li>
<li id="bib.bib42" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Raffel et al. (2020)</span>
<span class="ltx_bibblock">
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael
Matena, Yanqi Zhou, Wei Li, and Peter J Liu.

</span>
<span class="ltx_bibblock">Exploring the limits of transfer learning with a unified text-to-text
transformer.

</span>
<span class="ltx_bibblock"><em id="bib.bib42.1.1" class="ltx_emph ltx_font_italic">The Journal of Machine Learning Research</em>, 21(1):5485–5551, 2020.

</span>
</li>
<li id="bib.bib43" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Rame et al. (2022)</span>
<span class="ltx_bibblock">
Alexandre Rame, Matthieu Kirchmeyer, Thibaud Rahier, Alain Rakotomamonjy,
Patrick Gallinari, and Matthieu Cord.

</span>
<span class="ltx_bibblock">Diverse weight averaging for out-of-distribution generalization.

</span>
<span class="ltx_bibblock"><em id="bib.bib43.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>,
35:10821–10836, 2022.

</span>
</li>
<li id="bib.bib44" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Rillig et al. (2023)</span>
<span class="ltx_bibblock">
Matthias C Rillig, Marlene Ågerstrand, Mohan Bi, Kenneth A Gould, and Uli
Sauerland.

</span>
<span class="ltx_bibblock">Risks and benefits of large language models for the environment.

</span>
<span class="ltx_bibblock"><em id="bib.bib44.1.1" class="ltx_emph ltx_font_italic">Environmental Science &amp; Technology</em>, 57(9):3464–3466, 2023.

</span>
</li>
<li id="bib.bib45" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Sagi &amp; Rokach (2018)</span>
<span class="ltx_bibblock">
Omer Sagi and Lior Rokach.

</span>
<span class="ltx_bibblock">Ensemble learning: A survey.

</span>
<span class="ltx_bibblock"><em id="bib.bib45.1.1" class="ltx_emph ltx_font_italic">Wiley Interdisciplinary Reviews: Data Mining and Knowledge
Discovery</em>, 8(4):e1249, 2018.

</span>
</li>
<li id="bib.bib46" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Sanh et al. (2019)</span>
<span class="ltx_bibblock">
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf.

</span>
<span class="ltx_bibblock">Distilbert, a distilled version of bert: smaller, faster, cheaper and
lighter.

</span>
<span class="ltx_bibblock"><em id="bib.bib46.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:1910.01108</em>, 2019.

</span>
</li>
<li id="bib.bib47" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Sanyal et al. (2023)</span>
<span class="ltx_bibblock">
Sunny Sanyal, Jean Kaddour, Abhishek Kumar, and Sujay Sanghavi.

</span>
<span class="ltx_bibblock">Understanding the effectiveness of early weight averaging for
training large language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib47.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2306.03241</em>, 2023.

</span>
</li>
<li id="bib.bib48" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Srivastava et al. (2022)</span>
<span class="ltx_bibblock">
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar
Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià
Garriga-Alonso, et al.

</span>
<span class="ltx_bibblock">Beyond the imitation game: Quantifying and extrapolating the
capabilities of language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib48.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2206.04615</em>, 2022.

</span>
</li>
<li id="bib.bib49" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Sun et al. (2019)</span>
<span class="ltx_bibblock">
Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu.

</span>
<span class="ltx_bibblock">Patient knowledge distillation for bert model compression.

</span>
<span class="ltx_bibblock">In <em id="bib.bib49.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2019 Conference on Empirical Methods in
Natural Language Processing and the 9th International Joint Conference on
Natural Language Processing (EMNLP-IJCNLP)</em>, pp.  4323–4332, 2019.

</span>
</li>
<li id="bib.bib50" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Suzgun et al. (2022)</span>
<span class="ltx_bibblock">
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay,
Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al.

</span>
<span class="ltx_bibblock">Challenging big-bench tasks and whether chain-of-thought can solve
them.

</span>
<span class="ltx_bibblock"><em id="bib.bib50.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2210.09261</em>, 2022.

</span>
</li>
<li id="bib.bib51" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Tay et al. (2022)</span>
<span class="ltx_bibblock">
Yi Tay, Jason Wei, Hyung Won Chung, Vinh Q Tran, David R So, Siamak Shakeri,
Xavier Garcia, Huaixiu Steven Zheng, Jinfeng Rao, Aakanksha Chowdhery, et al.

</span>
<span class="ltx_bibblock">Transcending scaling laws with 0.1% extra compute.

</span>
<span class="ltx_bibblock"><em id="bib.bib51.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2210.11399</em>, 2022.

</span>
</li>
<li id="bib.bib52" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Team (2023)</span>
<span class="ltx_bibblock">
MosaicML NLP Team.

</span>
<span class="ltx_bibblock">Introducing mpt-7b: A new standard for open-source, commercially
usable llms, 2023.

</span>
<span class="ltx_bibblock">Accessed: 2023-05-05.

</span>
</li>
<li id="bib.bib53" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Touvron et al. (2023)</span>
<span class="ltx_bibblock">
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine
Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale,
et al.

</span>
<span class="ltx_bibblock">Llama 2: Open foundation and fine-tuned chat models.

</span>
<span class="ltx_bibblock"><em id="bib.bib53.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2307.09288</em>, 2023.

</span>
</li>
<li id="bib.bib54" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Turc et al. (2019)</span>
<span class="ltx_bibblock">
Iulia Turc, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.

</span>
<span class="ltx_bibblock">Well-read students learn better: On the importance of pre-training
compact models.

</span>
<span class="ltx_bibblock"><em id="bib.bib54.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:1908.08962</em>, 2019.

</span>
</li>
<li id="bib.bib55" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wang et al. (2022a)</span>
<span class="ltx_bibblock">
Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang,
Rangan Majumder, and Furu Wei.

</span>
<span class="ltx_bibblock">Text embeddings by weakly-supervised contrastive pre-training.

</span>
<span class="ltx_bibblock"><em id="bib.bib55.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2212.03533</em>, 2022a.

</span>
</li>
<li id="bib.bib56" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wang et al. (2020)</span>
<span class="ltx_bibblock">
Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou.

</span>
<span class="ltx_bibblock">Minilm: Deep self-attention distillation for task-agnostic
compression of pre-trained transformers.

</span>
<span class="ltx_bibblock"><em id="bib.bib56.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>,
33:5776–5788, 2020.

</span>
</li>
<li id="bib.bib57" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wang et al. (2023a)</span>
<span class="ltx_bibblock">
Xiaoxuan Wang, Ziniu Hu, Pan Lu, Yanqiao Zhu, Jieyu Zhang, Satyen Subramaniam,
Arjun Loomba, Shichang Zhang, Yizhou Sun, and Wei Wang.

</span>
<span class="ltx_bibblock">Scibench: Evaluating college-level scientific problem-solving
abilities of large language models.

</span>
<span class="ltx_bibblock">In <em id="bib.bib57.1.1" class="ltx_emph ltx_font_italic">The 3rd Workshop on Mathematical Reasoning and AI at
NeurIPS’23</em>, 2023a.

</span>
</li>
<li id="bib.bib58" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wang et al. (2022b)</span>
<span class="ltx_bibblock">
Yaqing Wang, Subhabrata Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Hassan
Awadallah, and Jianfeng Gao.

</span>
<span class="ltx_bibblock">Adamix: Mixture-of-adapter for parameter-efficient tuning of large
language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib58.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2205.12410</em>, 2022b.

</span>
</li>
<li id="bib.bib59" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wang et al. (2023b)</span>
<span class="ltx_bibblock">
Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot,
Khyathi Raghavi Chandu, David Wadden, Kelsey MacMillan, Noah A Smith,
Iz Beltagy, et al.

</span>
<span class="ltx_bibblock">How far can camels go? exploring the state of instruction tuning on
open resources.

</span>
<span class="ltx_bibblock"><em id="bib.bib59.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2306.04751</em>, 2023b.

</span>
</li>
<li id="bib.bib60" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wolf et al. (2020)</span>
<span class="ltx_bibblock">
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue,
Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz,
et al.

</span>
<span class="ltx_bibblock">Transformers: State-of-the-art natural language processing.

</span>
<span class="ltx_bibblock">In <em id="bib.bib60.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2020 Conference on Empirical Methods in
Natural Language Processing: System Dyiemonstrations</em>, pp.  38–45, 2020.

</span>
</li>
<li id="bib.bib61" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wortsman et al. (2022)</span>
<span class="ltx_bibblock">
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael
Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon,
Simon Kornblith, et al.

</span>
<span class="ltx_bibblock">Model soups: averaging weights of multiple fine-tuned models improves
accuracy without increasing inference time.

</span>
<span class="ltx_bibblock">In <em id="bib.bib61.1.1" class="ltx_emph ltx_font_italic">International Conference on Machine Learning</em>, pp. 23965–23998. PMLR, 2022.

</span>
</li>
<li id="bib.bib62" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Xu et al. (2023)</span>
<span class="ltx_bibblock">
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang
Tao, and Daxin Jiang.

</span>
<span class="ltx_bibblock">Wizardlm: Empowering large language models to follow complex
instructions.

</span>
<span class="ltx_bibblock"><em id="bib.bib62.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2304.12244</em>, 2023.

</span>
</li>
<li id="bib.bib63" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zellers et al. (2019)</span>
<span class="ltx_bibblock">
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi.

</span>
<span class="ltx_bibblock">Hellaswag: Can a machine really finish your sentence?

</span>
<span class="ltx_bibblock">In <em id="bib.bib63.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 57th Annual Meeting of the Association
for Computational Linguistics</em>, pp.  4791–4800, 2019.

</span>
</li>
<li id="bib.bib64" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhang et al. (2023)</span>
<span class="ltx_bibblock">
Jinghan Zhang, Shiqi Chen, Junteng Liu, and Junxian He.

</span>
<span class="ltx_bibblock">Composing parameter-efficient modules with arithmetic operations.

</span>
<span class="ltx_bibblock"><em id="bib.bib64.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2306.14870</em>, 2023.

</span>
</li>
</ul>
</section>
<section id="A1" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix A </span>Details of Token Alignment</h2>

<div id="A1.p1" class="ltx_para ltx_noindent">
<p id="A1.p1.1" class="ltx_p">For an input text, token alignment involves aligning two distribution matrices from two source LLMs. Therefore, the alignment comprises two dimensions: token-wise with respect to the text and distribution-wise with respect to the vocabulary. To provide a clear explanation, we show an example of different methods for token alignment in Figure <a href="#A1.F3" title="Figure 3 ‣ Appendix A Details of Token Alignment ‣ Knowledge Fusion of Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>.</p>
</div>
<div id="A1.p2" class="ltx_para ltx_noindent">
<p id="A1.p2.1" class="ltx_p">In the token dimension, we utilize the dynamic programming approach to recursively minimize the total cost of editing one sequence of tokens to align with another. When the mapped tokens are identical, such as the token “now” in the given example, these tokens are successfully aligned, allowing for the corresponding distributions to align subsequently. However, when the mapped tokens exhibit differences, such as the “get” and “gets” tokens in the example, the previous EM method proposed by <cite class="ltx_cite ltx_citemacro_cite">Fu et al. (<a href="#bib.bib17" title="" class="ltx_ref">2023</a>)</cite> does not align these tokens, resulting in the distributions degenerating into one-hot vectors. In contrast, our proposed MinED method successfully aligns the “gets” token with the “get” token, as they exhibit the minimal edit distance in the vocabularies from the two source LLMs.</p>
</div>
<div id="A1.p3" class="ltx_para ltx_noindent">
<p id="A1.p3.1" class="ltx_p">Concerning the distribution dimension, the alignment is performed between two vocabularies from different tokenizers of two source LLMs. Therefore, for distribution values with identical tokens, such as “current 0.05” and “current 0.04”, they will be aligned effectively. For distribution values involving different tokens, such as “immediate 0.04” and “immediately 0.03”, the EM method disregards this value. However, the proposed MinED method maps “immediately” to “immediate” due to their minimal edit distance, resulting in the successful alignment of these distribution values.</p>
</div>
<figure id="A1.F3" class="ltx_figure"><img src="/html/2401.10491/assets/x3.png" id="A1.F3.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="418" height="121" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 3: </span>An example of different methods for token alignment.</figcaption>
</figure>
</section>
<section id="A2" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix B </span>Details of MiniPile</h2>

<div id="A2.p1" class="ltx_para ltx_noindent">
<p id="A2.p1.1" class="ltx_p">MiniPile is curated from The Pile <cite class="ltx_cite ltx_citemacro_citep">(Gao et al., <a href="#bib.bib18" title="" class="ltx_ref">2020</a>)</cite> through a three-stage pruning process: (1) extracting embeddings for all documents with E5-Large <cite class="ltx_cite ltx_citemacro_citep">(Wang et al., <a href="#bib.bib55" title="" class="ltx_ref">2022a</a>)</cite>, which is a sentence embedding model, (2) clustering the embeddings using K-means, and (3) filtering out low-quality clusters. Therefore, MiniPile retains a compact scale while exhibiting extensive diversity, making it a prevalent choice for efficient training of LLMs <cite class="ltx_cite ltx_citemacro_citep">(Kaddour et al., <a href="#bib.bib30" title="" class="ltx_ref">2023</a>; Sanyal et al., <a href="#bib.bib47" title="" class="ltx_ref">2023</a>)</cite>.</p>
</div>
</section>
<section id="A3" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix C </span>Training Details</h2>

<div id="A3.p1" class="ltx_para ltx_noindent">
<p id="A3.p1.2" class="ltx_p">Our model is optimized using the AdamW optimizer with <math id="A3.p1.1.m1.1" class="ltx_Math" alttext="\beta_{1}=0.9" display="inline"><semantics id="A3.p1.1.m1.1a"><mrow id="A3.p1.1.m1.1.1" xref="A3.p1.1.m1.1.1.cmml"><msub id="A3.p1.1.m1.1.1.2" xref="A3.p1.1.m1.1.1.2.cmml"><mi id="A3.p1.1.m1.1.1.2.2" xref="A3.p1.1.m1.1.1.2.2.cmml">β</mi><mn id="A3.p1.1.m1.1.1.2.3" xref="A3.p1.1.m1.1.1.2.3.cmml">1</mn></msub><mo id="A3.p1.1.m1.1.1.1" xref="A3.p1.1.m1.1.1.1.cmml">=</mo><mn id="A3.p1.1.m1.1.1.3" xref="A3.p1.1.m1.1.1.3.cmml">0.9</mn></mrow><annotation-xml encoding="MathML-Content" id="A3.p1.1.m1.1b"><apply id="A3.p1.1.m1.1.1.cmml" xref="A3.p1.1.m1.1.1"><eq id="A3.p1.1.m1.1.1.1.cmml" xref="A3.p1.1.m1.1.1.1"></eq><apply id="A3.p1.1.m1.1.1.2.cmml" xref="A3.p1.1.m1.1.1.2"><csymbol cd="ambiguous" id="A3.p1.1.m1.1.1.2.1.cmml" xref="A3.p1.1.m1.1.1.2">subscript</csymbol><ci id="A3.p1.1.m1.1.1.2.2.cmml" xref="A3.p1.1.m1.1.1.2.2">𝛽</ci><cn type="integer" id="A3.p1.1.m1.1.1.2.3.cmml" xref="A3.p1.1.m1.1.1.2.3">1</cn></apply><cn type="float" id="A3.p1.1.m1.1.1.3.cmml" xref="A3.p1.1.m1.1.1.3">0.9</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.p1.1.m1.1c">\beta_{1}=0.9</annotation></semantics></math> and <math id="A3.p1.2.m2.1" class="ltx_Math" alttext="\beta_{2}=0.95" display="inline"><semantics id="A3.p1.2.m2.1a"><mrow id="A3.p1.2.m2.1.1" xref="A3.p1.2.m2.1.1.cmml"><msub id="A3.p1.2.m2.1.1.2" xref="A3.p1.2.m2.1.1.2.cmml"><mi id="A3.p1.2.m2.1.1.2.2" xref="A3.p1.2.m2.1.1.2.2.cmml">β</mi><mn id="A3.p1.2.m2.1.1.2.3" xref="A3.p1.2.m2.1.1.2.3.cmml">2</mn></msub><mo id="A3.p1.2.m2.1.1.1" xref="A3.p1.2.m2.1.1.1.cmml">=</mo><mn id="A3.p1.2.m2.1.1.3" xref="A3.p1.2.m2.1.1.3.cmml">0.95</mn></mrow><annotation-xml encoding="MathML-Content" id="A3.p1.2.m2.1b"><apply id="A3.p1.2.m2.1.1.cmml" xref="A3.p1.2.m2.1.1"><eq id="A3.p1.2.m2.1.1.1.cmml" xref="A3.p1.2.m2.1.1.1"></eq><apply id="A3.p1.2.m2.1.1.2.cmml" xref="A3.p1.2.m2.1.1.2"><csymbol cd="ambiguous" id="A3.p1.2.m2.1.1.2.1.cmml" xref="A3.p1.2.m2.1.1.2">subscript</csymbol><ci id="A3.p1.2.m2.1.1.2.2.cmml" xref="A3.p1.2.m2.1.1.2.2">𝛽</ci><cn type="integer" id="A3.p1.2.m2.1.1.2.3.cmml" xref="A3.p1.2.m2.1.1.2.3">2</cn></apply><cn type="float" id="A3.p1.2.m2.1.1.3.cmml" xref="A3.p1.2.m2.1.1.3">0.95</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.p1.2.m2.1c">\beta_{2}=0.95</annotation></semantics></math>, with gradient clipping set to 1.0 and weight decay to 0.1. A cosine learning rate schedule is employed, with a maximum learning rate of 1e-5 and a warmup ratio of 0.008. To accelerate the training, we employ packing <cite class="ltx_cite ltx_citemacro_citep">(Raffel et al., <a href="#bib.bib42" title="" class="ltx_ref">2020</a>)</cite>, where multiple training instances are grouped into a single sequence separated by end-of-sequence tokens, allowing for training on more tokens in each batch.</p>
</div>
</section>
<section id="A4" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix D </span>Additional Evaluation Results</h2>

<div id="A4.p1" class="ltx_para ltx_noindent">
<p id="A4.p1.1" class="ltx_p">To further illustrate the effectiveness of <span id="A4.p1.1.1" class="ltx_text ltx_font_smallcaps">FuseLLM</span>, we incorporate additional generative benchmarks related to <span id="A4.p1.1.2" class="ltx_text ltx_font_italic">knowledge-based question-answering</span>, <span id="A4.p1.1.3" class="ltx_text ltx_font_italic">reading comprehension</span>, <span id="A4.p1.1.4" class="ltx_text ltx_font_italic">content analysis</span>, <span id="A4.p1.1.5" class="ltx_text ltx_font_italic">machine translation</span>, and <span id="A4.p1.1.6" class="ltx_text ltx_font_italic">theorem application</span>. The results presented in Table <a href="#A4.T8" title="Table 8 ‣ Appendix D Additional Evaluation Results ‣ Knowledge Fusion of Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">8</span></a> highlight FuseLLM’s superiority over all source LLMs across all tasks.</p>
</div>
<div id="A4.p2" class="ltx_para ltx_noindent">
<ul id="A4.I1" class="ltx_itemize">
<li id="A4.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A4.I1.i1.p1" class="ltx_para ltx_noindent">
<p id="A4.I1.i1.p1.1" class="ltx_p">TriviaQA <cite class="ltx_cite ltx_citemacro_citep">(Joshi et al., <a href="#bib.bib29" title="" class="ltx_ref">2017</a>)</cite> is a benchmark to evaluate the <span id="A4.I1.i1.p1.1.1" class="ltx_text ltx_font_italic">knowledge-based question-answering</span> ability. We conduct a zero-shot evaluation and report the EM accuracy.</p>
</div>
</li>
<li id="A4.I1.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A4.I1.i2.p1" class="ltx_para ltx_noindent">
<p id="A4.I1.i2.p1.1" class="ltx_p">DROP <cite class="ltx_cite ltx_citemacro_citep">(Dua et al., <a href="#bib.bib16" title="" class="ltx_ref">2019</a>)</cite> is a benchmark to evaluate the <span id="A4.I1.i2.p1.1.1" class="ltx_text ltx_font_italic">reading comprehension</span> ability. We conduct a few-shot evaluation with CoT prompts and report the EM accuracy.</p>
</div>
</li>
<li id="A4.I1.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A4.I1.i3.p1" class="ltx_para ltx_noindent">
<p id="A4.I1.i3.p1.1" class="ltx_p">LAMBADA <cite class="ltx_cite ltx_citemacro_citep">(Paperno et al., <a href="#bib.bib39" title="" class="ltx_ref">2016</a>)</cite> is a benchmark to evaluate the <span id="A4.I1.i3.p1.1.1" class="ltx_text ltx_font_italic">content analysis</span> ability. We conduct a zero-shot evaluation and report the EM accuracy.</p>
</div>
</li>
<li id="A4.I1.i4" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A4.I1.i4.p1" class="ltx_para ltx_noindent">
<p id="A4.I1.i4.p1.1" class="ltx_p">IWSLT2017 <cite class="ltx_cite ltx_citemacro_citep">(Cettolo et al., <a href="#bib.bib7" title="" class="ltx_ref">2017</a>)</cite> is a benchmark to evaluate the <span id="A4.I1.i4.p1.1.1" class="ltx_text ltx_font_italic">machine translation</span> ability. We conduct a zero-shot evaluation and report the BLEU <cite class="ltx_cite ltx_citemacro_citep">(Papineni et al., <a href="#bib.bib40" title="" class="ltx_ref">2002</a>)</cite> score.</p>
</div>
</li>
<li id="A4.I1.i5" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A4.I1.i5.p1" class="ltx_para ltx_noindent">
<p id="A4.I1.i5.p1.1" class="ltx_p">SciBench <cite class="ltx_cite ltx_citemacro_citep">(Wang et al., <a href="#bib.bib57" title="" class="ltx_ref">2023a</a>)</cite> is a benchmark to evaluate the <span id="A4.I1.i5.p1.1.1" class="ltx_text ltx_font_italic">theorem application</span> ability. We conduct a few-shot evaluation with CoT prompts and report the EM accuracy.</p>
</div>
</li>
</ul>
</div>
<figure id="A4.T8" class="ltx_table">
<div id="A4.T8.1" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:318.0pt;height:86.2pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(-40.1pt,10.9pt) scale(0.798464660505564,0.798464660505564) ;">
<table id="A4.T8.1.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A4.T8.1.1.1.1" class="ltx_tr">
<th id="A4.T8.1.1.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_tt"><span id="A4.T8.1.1.1.1.1.1" class="ltx_text ltx_font_bold">Task</span></th>
<th id="A4.T8.1.1.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A4.T8.1.1.1.1.2.1" class="ltx_text ltx_font_bold">OpenLLaMA</span></th>
<th id="A4.T8.1.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A4.T8.1.1.1.1.3.1" class="ltx_text ltx_font_bold">MPT</span></th>
<th id="A4.T8.1.1.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A4.T8.1.1.1.1.4.1" class="ltx_text ltx_font_bold">Llama-2</span></th>
<th id="A4.T8.1.1.1.1.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="background-color:#ECF4FF;"><span id="A4.T8.1.1.1.1.5.1" class="ltx_text ltx_font_bold" style="background-color:#ECF4FF;">Llama-2 CLM</span></th>
<th id="A4.T8.1.1.1.1.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="background-color:#ECF4FF;"><span id="A4.T8.1.1.1.1.6.1" class="ltx_text ltx_font_bold ltx_font_smallcaps" style="background-color:#ECF4FF;">FuseLLM</span></th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A4.T8.1.1.2.1" class="ltx_tr">
<th id="A4.T8.1.1.2.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_tt">TrivialQA</th>
<td id="A4.T8.1.1.2.1.2" class="ltx_td ltx_align_center ltx_border_tt">39.96</td>
<td id="A4.T8.1.1.2.1.3" class="ltx_td ltx_align_center ltx_border_tt">28.89</td>
<td id="A4.T8.1.1.2.1.4" class="ltx_td ltx_align_center ltx_border_tt">52.46</td>
<td id="A4.T8.1.1.2.1.5" class="ltx_td ltx_align_center ltx_border_tt" style="background-color:#ECF4FF;"><span id="A4.T8.1.1.2.1.5.1" class="ltx_text" style="background-color:#ECF4FF;">53.14 <span id="A4.T8.1.1.2.1.5.1.1" class="ltx_text" style="color:#66B2FF;">(+1.30%)</span></span></td>
<td id="A4.T8.1.1.2.1.6" class="ltx_td ltx_align_center ltx_border_tt" style="background-color:#ECF4FF;"><span id="A4.T8.1.1.2.1.6.1" class="ltx_text ltx_font_bold" style="background-color:#ECF4FF;">54.49<span id="A4.T8.1.1.2.1.6.1.1" class="ltx_text ltx_font_medium"> <span id="A4.T8.1.1.2.1.6.1.1.1" class="ltx_text" style="color:#0000FF;">(+3.87%)</span></span></span></td>
</tr>
<tr id="A4.T8.1.1.3.2" class="ltx_tr">
<th id="A4.T8.1.1.3.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">DROP</th>
<td id="A4.T8.1.1.3.2.2" class="ltx_td ltx_align_center">22.31</td>
<td id="A4.T8.1.1.3.2.3" class="ltx_td ltx_align_center">23.54</td>
<td id="A4.T8.1.1.3.2.4" class="ltx_td ltx_align_center">27.25</td>
<td id="A4.T8.1.1.3.2.5" class="ltx_td ltx_align_center" style="background-color:#ECF4FF;"><span id="A4.T8.1.1.3.2.5.1" class="ltx_text" style="background-color:#ECF4FF;">28.51 <span id="A4.T8.1.1.3.2.5.1.1" class="ltx_text" style="color:#66B2FF;">(+4.62%)</span></span></td>
<td id="A4.T8.1.1.3.2.6" class="ltx_td ltx_align_center" style="background-color:#ECF4FF;"><span id="A4.T8.1.1.3.2.6.1" class="ltx_text ltx_font_bold" style="background-color:#ECF4FF;">28.97<span id="A4.T8.1.1.3.2.6.1.1" class="ltx_text ltx_font_medium"> <span id="A4.T8.1.1.3.2.6.1.1.1" class="ltx_text" style="color:#0000FF;">(+6.31%)</span></span></span></td>
</tr>
<tr id="A4.T8.1.1.4.3" class="ltx_tr">
<th id="A4.T8.1.1.4.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">LAMBADA</th>
<td id="A4.T8.1.1.4.3.2" class="ltx_td ltx_align_center">70.31</td>
<td id="A4.T8.1.1.4.3.3" class="ltx_td ltx_align_center">70.08</td>
<td id="A4.T8.1.1.4.3.4" class="ltx_td ltx_align_center">73.28</td>
<td id="A4.T8.1.1.4.3.5" class="ltx_td ltx_align_center" style="background-color:#ECF4FF;"><span id="A4.T8.1.1.4.3.5.1" class="ltx_text" style="background-color:#ECF4FF;">73.45 <span id="A4.T8.1.1.4.3.5.1.1" class="ltx_text" style="color:#66B2FF;">(+0.23%)</span></span></td>
<td id="A4.T8.1.1.4.3.6" class="ltx_td ltx_align_center" style="background-color:#ECF4FF;"><span id="A4.T8.1.1.4.3.6.1" class="ltx_text ltx_font_bold" style="background-color:#ECF4FF;">73.72<span id="A4.T8.1.1.4.3.6.1.1" class="ltx_text ltx_font_medium"> <span id="A4.T8.1.1.4.3.6.1.1.1" class="ltx_text" style="color:#0000FF;">(+0.60%)</span></span></span></td>
</tr>
<tr id="A4.T8.1.1.5.4" class="ltx_tr">
<th id="A4.T8.1.1.5.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">IWSLT2017</th>
<td id="A4.T8.1.1.5.4.2" class="ltx_td ltx_align_center">5.51</td>
<td id="A4.T8.1.1.5.4.3" class="ltx_td ltx_align_center">5.49</td>
<td id="A4.T8.1.1.5.4.4" class="ltx_td ltx_align_center">6.48</td>
<td id="A4.T8.1.1.5.4.5" class="ltx_td ltx_align_center" style="background-color:#ECF4FF;"><span id="A4.T8.1.1.5.4.5.1" class="ltx_text ltx_font_bold" style="background-color:#ECF4FF;">6.91<span id="A4.T8.1.1.5.4.5.1.1" class="ltx_text ltx_font_medium"> <span id="A4.T8.1.1.5.4.5.1.1.1" class="ltx_text" style="color:#66B2FF;">(+6.64%)</span></span></span></td>
<td id="A4.T8.1.1.5.4.6" class="ltx_td ltx_align_center" style="background-color:#ECF4FF;"><span id="A4.T8.1.1.5.4.6.1" class="ltx_text" style="background-color:#ECF4FF;">6.75 <span id="A4.T8.1.1.5.4.6.1.1" class="ltx_text" style="color:#0000FF;">(+4.17%)</span></span></td>
</tr>
<tr id="A4.T8.1.1.6.5" class="ltx_tr">
<th id="A4.T8.1.1.6.5.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb">SciBench</th>
<td id="A4.T8.1.1.6.5.2" class="ltx_td ltx_align_center ltx_border_bb">0.68</td>
<td id="A4.T8.1.1.6.5.3" class="ltx_td ltx_align_center ltx_border_bb">0.88</td>
<td id="A4.T8.1.1.6.5.4" class="ltx_td ltx_align_center ltx_border_bb">0.14</td>
<td id="A4.T8.1.1.6.5.5" class="ltx_td ltx_align_center ltx_border_bb" style="background-color:#ECF4FF;"><span id="A4.T8.1.1.6.5.5.1" class="ltx_text" style="background-color:#ECF4FF;">0.94 <span id="A4.T8.1.1.6.5.5.1.1" class="ltx_text" style="color:#66B2FF;">(+571.43%)</span></span></td>
<td id="A4.T8.1.1.6.5.6" class="ltx_td ltx_align_center ltx_border_bb" style="background-color:#ECF4FF;"><span id="A4.T8.1.1.6.5.6.1" class="ltx_text ltx_font_bold" style="background-color:#ECF4FF;">1.65<span id="A4.T8.1.1.6.5.6.1.1" class="ltx_text ltx_font_medium"> <span id="A4.T8.1.1.6.5.6.1.1.1" class="ltx_text" style="color:#0000FF;">(+1078.57%)</span></span></span></td>
</tr>
</tbody>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 8: </span>Overall results of <span id="A4.T8.3.1" class="ltx_text ltx_font_smallcaps">FuseLLM</span> and baselines in additional generative benchmarks, where percentages indicate the rate of improvement/decrease compared to Llama-2.</figcaption>
</figure>
</section>
<section id="A5" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix E </span><span id="A5.1.1" class="ltx_text ltx_font_smallcaps">FuseLLM</span> vs. Previous Model Fusion Methods</h2>

<figure id="A5.T9" class="ltx_table ltx_align_floatright">
<div id="A5.T9.1" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:429.3pt;height:296.2pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(110.3pt,-76.1pt) scale(2.05673067110057,2.05673067110057) ;">
<table id="A5.T9.1.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A5.T9.1.1.1.1" class="ltx_tr">
<th id="A5.T9.1.1.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_tt"><span id="A5.T9.1.1.1.1.1.1" class="ltx_text ltx_font_bold">Model</span></th>
<th id="A5.T9.1.1.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A5.T9.1.1.1.1.2.1" class="ltx_text ltx_font_bold">BBH</span></th>
<th id="A5.T9.1.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A5.T9.1.1.1.1.3.1" class="ltx_text ltx_font_bold">ME</span></th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A5.T9.1.1.2.1" class="ltx_tr">
<th id="A5.T9.1.1.2.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_tt">OpenLLaMA</th>
<td id="A5.T9.1.1.2.1.2" class="ltx_td ltx_align_center ltx_border_tt">33.87</td>
<td id="A5.T9.1.1.2.1.3" class="ltx_td ltx_align_center ltx_border_tt"><span id="A5.T9.1.1.2.1.3.1" class="ltx_text ltx_font_bold">18.11</span></td>
</tr>
<tr id="A5.T9.1.1.3.2" class="ltx_tr">
<th id="A5.T9.1.1.3.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">MPT</th>
<td id="A5.T9.1.1.3.2.2" class="ltx_td ltx_align_center">33.38</td>
<td id="A5.T9.1.1.3.2.3" class="ltx_td ltx_align_center">17.26</td>
</tr>
<tr id="A5.T9.1.1.4.3" class="ltx_tr">
<th id="A5.T9.1.1.4.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">Llama-2</th>
<td id="A5.T9.1.1.4.3.2" class="ltx_td ltx_align_center">39.70</td>
<td id="A5.T9.1.1.4.3.3" class="ltx_td ltx_align_center">14.63</td>
</tr>
<tr id="A5.T9.1.1.5.4" class="ltx_tr" style="background-color:#ECF4FF;">
<th id="A5.T9.1.1.5.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="A5.T9.1.1.5.4.1.1" class="ltx_text" style="background-color:#ECF4FF;">LLM-Blender (Rank&amp;Fuse)</span></th>
<td id="A5.T9.1.1.5.4.2" class="ltx_td ltx_align_center"><span id="A5.T9.1.1.5.4.2.1" class="ltx_text" style="background-color:#ECF4FF;">24.48</span></td>
<td id="A5.T9.1.1.5.4.3" class="ltx_td ltx_align_center"><span id="A5.T9.1.1.5.4.3.1" class="ltx_text" style="background-color:#ECF4FF;">0.06</span></td>
</tr>
<tr id="A5.T9.1.1.6.5" class="ltx_tr" style="background-color:#ECF4FF;">
<th id="A5.T9.1.1.6.5.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="A5.T9.1.1.6.5.1.1" class="ltx_text" style="background-color:#ECF4FF;">LLM-Blender (Rank)</span></th>
<td id="A5.T9.1.1.6.5.2" class="ltx_td ltx_align_center"><span id="A5.T9.1.1.6.5.2.1" class="ltx_text" style="background-color:#ECF4FF;">37.17</span></td>
<td id="A5.T9.1.1.6.5.3" class="ltx_td ltx_align_center"><span id="A5.T9.1.1.6.5.3.1" class="ltx_text" style="background-color:#ECF4FF;">17.85</span></td>
</tr>
<tr id="A5.T9.1.1.7.6" class="ltx_tr" style="background-color:#ECF4FF;">
<th id="A5.T9.1.1.7.6.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="A5.T9.1.1.7.6.1.1" class="ltx_text" style="background-color:#ECF4FF;">Llama-2 CLM</span></th>
<td id="A5.T9.1.1.7.6.2" class="ltx_td ltx_align_center"><span id="A5.T9.1.1.7.6.2.1" class="ltx_text" style="background-color:#ECF4FF;">40.44</span></td>
<td id="A5.T9.1.1.7.6.3" class="ltx_td ltx_align_center"><span id="A5.T9.1.1.7.6.3.1" class="ltx_text" style="background-color:#ECF4FF;">14.83</span></td>
</tr>
<tr id="A5.T9.1.1.8.7" class="ltx_tr" style="background-color:#ECF4FF;">
<th id="A5.T9.1.1.8.7.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb"><span id="A5.T9.1.1.8.7.1.1" class="ltx_text ltx_font_smallcaps" style="background-color:#ECF4FF;">FuseLLM</span></th>
<td id="A5.T9.1.1.8.7.2" class="ltx_td ltx_align_center ltx_border_bb"><span id="A5.T9.1.1.8.7.2.1" class="ltx_text ltx_font_bold" style="background-color:#ECF4FF;">41.75</span></td>
<td id="A5.T9.1.1.8.7.3" class="ltx_td ltx_align_center ltx_border_bb"><span id="A5.T9.1.1.8.7.3.1" class="ltx_text" style="background-color:#ECF4FF;">15.56</span></td>
</tr>
</tbody>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 9: </span>Comparison of <span id="A5.T9.3.1" class="ltx_text ltx_font_smallcaps">FuseLLM</span> and LLM-Blender.</figcaption>
</figure>
<div id="A5.p1" class="ltx_para ltx_noindent">
<p id="A5.p1.1" class="ltx_p">The motivation behind FuseLLM is to integrate the collective knowledge of multiple LLMs with diverse architectures and pre-training corpora. Consequently, the traditional fusion method of model merging, which demands identical model architectures, is not directly applicable in this context. While the model ensemble technique aggregates predictions from multiple LLMs, the drawback lies in the substantial memory and time costs when maintaining multiple source LLMs during inference. We further compare <span id="A5.p1.1.1" class="ltx_text ltx_font_smallcaps">FuseLLM</span> with an ensemble method for LLMs, LLM-Blender <cite class="ltx_cite ltx_citemacro_citep">(Jiang et al., <a href="#bib.bib26" title="" class="ltx_ref">2023</a>)</cite>, which ranks and combines the output texts from multiple LLMs with ranker and fuser models. Specifically, we conduct experiments on the Big-Bench Hard and MultiPL-E benchmarks using the open-source ranker and fuser models. Notably, the CommonSense benchmark, which utilizes perplexity-based evaluation, cannot adapt the LLM-Blender method. The experimental results are shown in Table <a href="#A5.T9" title="Table 9 ‣ Appendix E FuseLLM vs. Previous Model Fusion Methods ‣ Knowledge Fusion of Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">9</span></a>, where LLM-Blender (Rank&amp;Fuse) refers to using the ranker to obtain the top three results and then using the fuser to combine them, and LLM-Blender (Rank) represents simply using the ranker to obtain the top one result. We observed a notable performance deterioration after fusion when employing both the ranker and fuser. This could be attributed to the fuser model’s training within the instruction-tuning context, potentially leading to inadequate generalization to the test tasks. Furthermore, while LLM-Blender (Rank) outperforms the LLM-Blender (Rank&amp;Fuse), it remains inferior to the best-performing source LLM. This suggests the ranker model’s inability to discriminate the optimal responses when combining different LLMs efficiently.</p>
</div>
</section>
<section id="A6" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix F </span>Incorporating Instruction-Tuning Models with <span id="A6.1.1" class="ltx_text ltx_font_smallcaps">FuseLLM</span>
</h2>

<div id="A6.p1" class="ltx_para ltx_noindent">
<p id="A6.p1.1" class="ltx_p">Recall that the proposed <span id="A6.p1.1.1" class="ltx_text ltx_font_smallcaps">FuseLLM</span> involves extracting distribution matrices from multiple distinct source LLMs and continually training the target LLM. Therefore, <span id="A6.p1.1.2" class="ltx_text ltx_font_smallcaps">FuseLLM</span> is also applicable to instruction-tuning models, provided that all corresponding continual-training samples adhere to the instruction-tuning format and mask the instruction part when calculating the training loss.</p>
</div>
<figure id="A6.T10" class="ltx_table ltx_align_floatright">
<div id="A6.T10.1" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:429.3pt;height:195.7pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(96.2pt,-43.9pt) scale(1.81218049057967,1.81218049057967) ;">
<table id="A6.T10.1.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A6.T10.1.1.1.1" class="ltx_tr">
<th id="A6.T10.1.1.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_tt"><span id="A6.T10.1.1.1.1.1.1" class="ltx_text ltx_font_bold">Model</span></th>
<th id="A6.T10.1.1.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A6.T10.1.1.1.1.2.1" class="ltx_text ltx_font_bold">Vicuna Benchmark</span></th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A6.T10.1.1.2.1" class="ltx_tr">
<th id="A6.T10.1.1.2.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_tt">OpenLLaMA ShareGPT</th>
<td id="A6.T10.1.1.2.1.2" class="ltx_td ltx_align_center ltx_border_tt">7.23</td>
</tr>
<tr id="A6.T10.1.1.3.2" class="ltx_tr">
<th id="A6.T10.1.1.3.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">MPT Open-Platypus</th>
<td id="A6.T10.1.1.3.2.2" class="ltx_td ltx_align_center">6.46</td>
</tr>
<tr id="A6.T10.1.1.4.3" class="ltx_tr">
<th id="A6.T10.1.1.4.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">Llama-2 Evol-Instruct</th>
<td id="A6.T10.1.1.4.3.2" class="ltx_td ltx_align_center">7.88</td>
</tr>
<tr id="A6.T10.1.1.5.4" class="ltx_tr" style="background-color:#ECF4FF;">
<th id="A6.T10.1.1.5.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="A6.T10.1.1.5.4.1.1" class="ltx_text" style="background-color:#ECF4FF;">Llama-2 Evol-Instruct CLM</span></th>
<td id="A6.T10.1.1.5.4.2" class="ltx_td ltx_align_center"><span id="A6.T10.1.1.5.4.2.1" class="ltx_text" style="background-color:#ECF4FF;">8.03 <span id="A6.T10.1.1.5.4.2.1.1" class="ltx_text" style="color:#0000FF;">(+1.90%)</span></span></td>
</tr>
<tr id="A6.T10.1.1.6.5" class="ltx_tr" style="background-color:#ECF4FF;">
<th id="A6.T10.1.1.6.5.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb"><span id="A6.T10.1.1.6.5.1.1" class="ltx_text ltx_font_smallcaps" style="background-color:#ECF4FF;">FuseLLM</span></th>
<td id="A6.T10.1.1.6.5.2" class="ltx_td ltx_align_center ltx_border_bb"><span id="A6.T10.1.1.6.5.2.1" class="ltx_text ltx_font_bold" style="background-color:#ECF4FF;">8.16<span id="A6.T10.1.1.6.5.2.1.1" class="ltx_text ltx_font_medium"> <span id="A6.T10.1.1.6.5.2.1.1.1" class="ltx_text" style="color:#0000FF;">(+3.55%)</span></span></span></td>
</tr>
</tbody>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 10: </span>Results of fusing instruction-tuning models with <span id="A6.T10.3.1" class="ltx_text ltx_font_smallcaps">FuseLLM</span>.</figcaption>
</figure>
<div id="A6.p2" class="ltx_para ltx_noindent">
<p id="A6.p2.1" class="ltx_p">To confirm this, we conduct new experiments on the fusion of instruction-tuning LLMs. Specifically, we initially fine-tune Llama-2, OpenLLaMA, and MPT using 20k samples from Evol-Instruct <cite class="ltx_cite ltx_citemacro_citep">(Xu et al., <a href="#bib.bib62" title="" class="ltx_ref">2023</a>)</cite>, ShareGPT <cite class="ltx_cite ltx_citemacro_citep">(Chiang et al., <a href="#bib.bib10" title="" class="ltx_ref">2023</a>)</cite>, and Open-Platypus <cite class="ltx_cite ltx_citemacro_citep">(Lee et al., <a href="#bib.bib32" title="" class="ltx_ref">2023</a>)</cite> datasets, respectively. Consequently, the three source LLMs transitioned into instruction-tuning LLMs. Then, we sample another 5k samples from each of the aforementioned datasets to create a corpus for continual training, specifying Llama-2 Evol-Instruct as the target LLM for knowledge fusion. We assess the instruction-following performance on the Vicuna Benchmark using GPT-4 as an evaluator following <cite class="ltx_cite ltx_citemacro_cite">Chiang et al. (<a href="#bib.bib10" title="" class="ltx_ref">2023</a>)</cite>, which gives a score from 1 to 10 for each answer. The results shown in Table <a href="#A6.T10" title="Table 10 ‣ Appendix F Incorporating Instruction-Tuning Models with FuseLLM ‣ Knowledge Fusion of Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">10</span></a> demonstrate that FuseLLM surpasses each individual source instruction-tuning LLM, achieving the best performance with GPT-4 judgment.</p>
</div>
</section>
<section id="A7" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix G </span>Cause of performance improvement</h2>

<figure id="A7.F4" class="ltx_figure ltx_align_floatright"><img src="/html/2401.10491/assets/x4.png" id="A7.F4.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="438" height="248" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 4: </span>Perplexity comparison on RedPajama. The bars denote the percentage of examples with reduced perplexity when transitioning from CLM to <span id="A7.F4.2.1" class="ltx_text ltx_font_smallcaps">FuseLLM</span> (solid) and from Llama-2 to OpenLLaMA or MPT (slashed).</figcaption>
</figure>
<div id="A7.p1" class="ltx_para ltx_noindent">
<p id="A7.p1.1" class="ltx_p">To further demonstrate that our performance improvement stems from the integration of knowledge from multiple LLMs rather than solely from continual training, we conduct an evaluation on an alternative corpus, RedPajama <cite class="ltx_cite ltx_citemacro_citep">(Computer, <a href="#bib.bib14" title="" class="ltx_ref">2023</a>)</cite>. To mitigate the impact of different tokenizers, we employ the bits per UTF-8 encoded
byte (BPB) metric proposed by <cite class="ltx_cite ltx_citemacro_cite">Gao et al. (<a href="#bib.bib18" title="" class="ltx_ref">2020</a>)</cite>, where a smaller value indicates lower perplexity. Then, we compute the percentage of test samples in each domain exhibiting decreased BPB from Llama-2 CLM to <span id="A7.p1.1.1" class="ltx_text ltx_font_smallcaps">FuseLLM</span> and from Llama-2 to OpenLLaMA or MPT. The results in Figure <a href="#A7.F4" title="Figure 4 ‣ Appendix G Cause of performance improvement ‣ Knowledge Fusion of Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a> suggest that when <span id="A7.p1.1.2" class="ltx_text ltx_font_smallcaps">FuseLLM</span> outperforms Llama-2 CLM, the performance of OpenLLaMA or MPT typically surpasses that of Llama-2, as evidenced by the slashed bars in each domain. This phenomenon is particularly pronounced in domains such as Arxiv, StackExchange, Wikipedia, and Github, where it exceeds <math id="A7.p1.1.m1.1" class="ltx_Math" alttext="95\%" display="inline"><semantics id="A7.p1.1.m1.1a"><mrow id="A7.p1.1.m1.1.1" xref="A7.p1.1.m1.1.1.cmml"><mn id="A7.p1.1.m1.1.1.2" xref="A7.p1.1.m1.1.1.2.cmml">95</mn><mo id="A7.p1.1.m1.1.1.1" xref="A7.p1.1.m1.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="A7.p1.1.m1.1b"><apply id="A7.p1.1.m1.1.1.cmml" xref="A7.p1.1.m1.1.1"><csymbol cd="latexml" id="A7.p1.1.m1.1.1.1.cmml" xref="A7.p1.1.m1.1.1.1">percent</csymbol><cn type="integer" id="A7.p1.1.m1.1.1.2.cmml" xref="A7.p1.1.m1.1.1.2">95</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A7.p1.1.m1.1c">95\%</annotation></semantics></math>. This compelling evidence suggests that the performance enhancements achieved by <span id="A7.p1.1.3" class="ltx_text ltx_font_smallcaps">FuseLLM</span> are indeed attributed to the integration of knowledge from multiple LLMs.</p>
</div>
</section>
<section id="A8" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix H </span>Weight for loss combination</h2>

<figure id="A8.T11" class="ltx_table ltx_align_floatright">
<div id="A8.T11.3" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:429.3pt;height:78pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(66.1pt,-12.0pt) scale(1.44526955023658,1.44526955023658) ;">
<table id="A8.T11.3.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A8.T11.3.1.1.1" class="ltx_tr">
<th id="A8.T11.3.1.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_tt"><span id="A8.T11.3.1.1.1.1.1" class="ltx_text ltx_font_bold">Design</span></th>
<th id="A8.T11.3.1.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A8.T11.3.1.1.1.2.1" class="ltx_text ltx_font_bold">BBH</span></th>
<th id="A8.T11.3.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A8.T11.3.1.1.1.3.1" class="ltx_text ltx_font_bold">ME</span></th>
<th id="A8.T11.3.1.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A8.T11.3.1.1.1.4.1" class="ltx_text ltx_font_bold">CS</span></th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A8.T11.3.1.2.1" class="ltx_tr">
<th id="A8.T11.3.1.2.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_tt">Dynamic</th>
<td id="A8.T11.3.1.2.1.2" class="ltx_td ltx_align_center ltx_border_tt">41.75</td>
<td id="A8.T11.3.1.2.1.3" class="ltx_td ltx_align_center ltx_border_tt">15.52</td>
<td id="A8.T11.3.1.2.1.4" class="ltx_td ltx_align_center ltx_border_tt">64.50</td>
</tr>
<tr id="A8.T11.3.1.3.2" class="ltx_tr">
<th id="A8.T11.3.1.3.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb">Static</th>
<td id="A8.T11.3.1.3.2.2" class="ltx_td ltx_align_center ltx_border_bb">41.75 <span id="A8.T11.3.1.3.2.2.1" class="ltx_text" style="color:#0000FF;">(+0.00%)</span>
</td>
<td id="A8.T11.3.1.3.2.3" class="ltx_td ltx_align_center ltx_border_bb">15.56 <span id="A8.T11.3.1.3.2.3.1" class="ltx_text" style="color:#0000FF;">(+0.25%)</span>
</td>
<td id="A8.T11.3.1.3.2.4" class="ltx_td ltx_align_center ltx_border_bb">64.56 <span id="A8.T11.3.1.3.2.4.1" class="ltx_text" style="color:#0000FF;">(+0.09%)</span>
</td>
</tr>
</tbody>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 11: </span>Comparison of different designs for <math id="A8.T11.2.m1.1" class="ltx_Math" alttext="\lambda" display="inline"><semantics id="A8.T11.2.m1.1b"><mi id="A8.T11.2.m1.1.1" xref="A8.T11.2.m1.1.1.cmml">λ</mi><annotation-xml encoding="MathML-Content" id="A8.T11.2.m1.1c"><ci id="A8.T11.2.m1.1.1.cmml" xref="A8.T11.2.m1.1.1">𝜆</ci></annotation-xml><annotation encoding="application/x-tex" id="A8.T11.2.m1.1d">\lambda</annotation></semantics></math>.</figcaption>
</figure>
<div id="A8.p1" class="ltx_para ltx_noindent">
<p id="A8.p1.6" class="ltx_p">As an alternative to the static weight for <math id="A8.p1.1.m1.1" class="ltx_Math" alttext="\lambda" display="inline"><semantics id="A8.p1.1.m1.1a"><mi id="A8.p1.1.m1.1.1" xref="A8.p1.1.m1.1.1.cmml">λ</mi><annotation-xml encoding="MathML-Content" id="A8.p1.1.m1.1b"><ci id="A8.p1.1.m1.1.1.cmml" xref="A8.p1.1.m1.1.1">𝜆</ci></annotation-xml><annotation encoding="application/x-tex" id="A8.p1.1.m1.1c">\lambda</annotation></semantics></math> in Eq. <a href="#S3.E5" title="In 3.2 LLMs Fusion ‣ 3 Knowledge Fusion of LLMs ‣ Knowledge Fusion of Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a>, we consider a dynamic design, which is referred to as teacher annealing <cite class="ltx_cite ltx_citemacro_citep">(Clark et al., <a href="#bib.bib12" title="" class="ltx_ref">2019b</a>)</cite>. This technique gradually increases <math id="A8.p1.2.m2.1" class="ltx_Math" alttext="\lambda" display="inline"><semantics id="A8.p1.2.m2.1a"><mi id="A8.p1.2.m2.1.1" xref="A8.p1.2.m2.1.1.cmml">λ</mi><annotation-xml encoding="MathML-Content" id="A8.p1.2.m2.1b"><ci id="A8.p1.2.m2.1.1.cmml" xref="A8.p1.2.m2.1.1">𝜆</ci></annotation-xml><annotation encoding="application/x-tex" id="A8.p1.2.m2.1c">\lambda</annotation></semantics></math> during the training process, giving preference to <math id="A8.p1.3.m3.1" class="ltx_Math" alttext="\mathcal{L}_{\text{Fusion}}" display="inline"><semantics id="A8.p1.3.m3.1a"><msub id="A8.p1.3.m3.1.1" xref="A8.p1.3.m3.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="A8.p1.3.m3.1.1.2" xref="A8.p1.3.m3.1.1.2.cmml">ℒ</mi><mtext id="A8.p1.3.m3.1.1.3" xref="A8.p1.3.m3.1.1.3a.cmml">Fusion</mtext></msub><annotation-xml encoding="MathML-Content" id="A8.p1.3.m3.1b"><apply id="A8.p1.3.m3.1.1.cmml" xref="A8.p1.3.m3.1.1"><csymbol cd="ambiguous" id="A8.p1.3.m3.1.1.1.cmml" xref="A8.p1.3.m3.1.1">subscript</csymbol><ci id="A8.p1.3.m3.1.1.2.cmml" xref="A8.p1.3.m3.1.1.2">ℒ</ci><ci id="A8.p1.3.m3.1.1.3a.cmml" xref="A8.p1.3.m3.1.1.3"><mtext mathsize="70%" id="A8.p1.3.m3.1.1.3.cmml" xref="A8.p1.3.m3.1.1.3">Fusion</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A8.p1.3.m3.1c">\mathcal{L}_{\text{Fusion}}</annotation></semantics></math> initially and subsequently shifting the focus to <math id="A8.p1.4.m4.1" class="ltx_Math" alttext="\mathcal{L}_{\text{CLM}}" display="inline"><semantics id="A8.p1.4.m4.1a"><msub id="A8.p1.4.m4.1.1" xref="A8.p1.4.m4.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="A8.p1.4.m4.1.1.2" xref="A8.p1.4.m4.1.1.2.cmml">ℒ</mi><mtext id="A8.p1.4.m4.1.1.3" xref="A8.p1.4.m4.1.1.3a.cmml">CLM</mtext></msub><annotation-xml encoding="MathML-Content" id="A8.p1.4.m4.1b"><apply id="A8.p1.4.m4.1.1.cmml" xref="A8.p1.4.m4.1.1"><csymbol cd="ambiguous" id="A8.p1.4.m4.1.1.1.cmml" xref="A8.p1.4.m4.1.1">subscript</csymbol><ci id="A8.p1.4.m4.1.1.2.cmml" xref="A8.p1.4.m4.1.1.2">ℒ</ci><ci id="A8.p1.4.m4.1.1.3a.cmml" xref="A8.p1.4.m4.1.1.3"><mtext mathsize="70%" id="A8.p1.4.m4.1.1.3.cmml" xref="A8.p1.4.m4.1.1.3">CLM</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A8.p1.4.m4.1c">\mathcal{L}_{\text{CLM}}</annotation></semantics></math>. In this experiment, we set the <math id="A8.p1.5.m5.1" class="ltx_Math" alttext="\lambda" display="inline"><semantics id="A8.p1.5.m5.1a"><mi id="A8.p1.5.m5.1.1" xref="A8.p1.5.m5.1.1.cmml">λ</mi><annotation-xml encoding="MathML-Content" id="A8.p1.5.m5.1b"><ci id="A8.p1.5.m5.1.1.cmml" xref="A8.p1.5.m5.1.1">𝜆</ci></annotation-xml><annotation encoding="application/x-tex" id="A8.p1.5.m5.1c">\lambda</annotation></semantics></math> to increase linearly from 0.7 to 1.0. Table <a href="#A8.T11" title="Table 11 ‣ Appendix H Weight for loss combination ‣ Knowledge Fusion of Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">11</span></a> presents a comparison between the static and dynamic designs of <math id="A8.p1.6.m6.1" class="ltx_Math" alttext="\lambda" display="inline"><semantics id="A8.p1.6.m6.1a"><mi id="A8.p1.6.m6.1.1" xref="A8.p1.6.m6.1.1.cmml">λ</mi><annotation-xml encoding="MathML-Content" id="A8.p1.6.m6.1b"><ci id="A8.p1.6.m6.1.1.cmml" xref="A8.p1.6.m6.1.1">𝜆</ci></annotation-xml><annotation encoding="application/x-tex" id="A8.p1.6.m6.1c">\lambda</annotation></semantics></math>, demonstrating that the two approaches yield comparable performance. Therefore, we opt for the static design for simplicity.</p>
</div>
</section>
<section id="A9" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix I </span>Using Strongest LLM as Target</h2>

<div id="A9.p1" class="ltx_para ltx_noindent">
<p id="A9.p1.1" class="ltx_p">In the experiments, we consistently employ Llama-2 as the target LLM to maintain a fixed setup. We further conduct supplementary experiments on code generation tasks using the strongest LLM, OpenLLaMA, as the target LLM. The results are shown in Table <a href="#A9.T12" title="Table 12 ‣ Appendix I Using Strongest LLM as Target ‣ Knowledge Fusion of Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">12</span></a>. We observe a decrease in the performance of OpenLLaMA CLM compared to the source OpenLLaMA and MPT. This decline is attributed to the inclusion of the StarCoder <cite class="ltx_cite ltx_citemacro_citep">(Li et al., <a href="#bib.bib34" title="" class="ltx_ref">2023</a>)</cite> data in OpenLLaMA’s pre-training corpus, whereas there is a limited amount of code data in our continual training corpus. For the same reason, even though FuseLLM showcases improved performance compared to OpenLLaMA CLM, it still lags behind the original OpenLLaMA.</p>
</div>
<figure id="A9.T12" class="ltx_table">
<div id="A9.T12.1" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:318.0pt;height:179.7pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(-32.1pt,18.1pt) scale(0.832068013864406,0.832068013864406) ;">
<table id="A9.T12.1.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A9.T12.1.1.1.1" class="ltx_tr">
<th id="A9.T12.1.1.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_tt"><span id="A9.T12.1.1.1.1.1.1" class="ltx_text ltx_font_bold">Task</span></th>
<th id="A9.T12.1.1.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A9.T12.1.1.1.1.2.1" class="ltx_text ltx_font_bold">OpenLLaMA</span></th>
<th id="A9.T12.1.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A9.T12.1.1.1.1.3.1" class="ltx_text ltx_font_bold">MPT</span></th>
<th id="A9.T12.1.1.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A9.T12.1.1.1.1.4.1" class="ltx_text ltx_font_bold">Llama-2</span></th>
<th id="A9.T12.1.1.1.1.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="background-color:#ECF4FF;"><span id="A9.T12.1.1.1.1.5.1" class="ltx_text ltx_font_bold" style="background-color:#ECF4FF;">OpenLLaMA CLM</span></th>
<th id="A9.T12.1.1.1.1.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="background-color:#ECF4FF;"><span id="A9.T12.1.1.1.1.6.1" class="ltx_text ltx_font_bold ltx_font_smallcaps" style="background-color:#ECF4FF;">FuseLLM</span></th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A9.T12.1.1.2.1" class="ltx_tr">
<th id="A9.T12.1.1.2.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_tt">C++</th>
<td id="A9.T12.1.1.2.1.2" class="ltx_td ltx_align_center ltx_border_tt"><span id="A9.T12.1.1.2.1.2.1" class="ltx_text ltx_font_bold">14.47</span></td>
<td id="A9.T12.1.1.2.1.3" class="ltx_td ltx_align_center ltx_border_tt">13.11</td>
<td id="A9.T12.1.1.2.1.4" class="ltx_td ltx_align_center ltx_border_tt">7.45</td>
<td id="A9.T12.1.1.2.1.5" class="ltx_td ltx_align_center ltx_border_tt" style="background-color:#ECF4FF;"><span id="A9.T12.1.1.2.1.5.1" class="ltx_text" style="background-color:#ECF4FF;">12.89</span></td>
<td id="A9.T12.1.1.2.1.6" class="ltx_td ltx_align_center ltx_border_tt" style="background-color:#ECF4FF;"><span id="A9.T12.1.1.2.1.6.1" class="ltx_text" style="background-color:#ECF4FF;">14.01</span></td>
</tr>
<tr id="A9.T12.1.1.3.2" class="ltx_tr">
<th id="A9.T12.1.1.3.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">Go</th>
<td id="A9.T12.1.1.3.2.2" class="ltx_td ltx_align_center">68.20</td>
<td id="A9.T12.1.1.3.2.3" class="ltx_td ltx_align_center">66.96</td>
<td id="A9.T12.1.1.3.2.4" class="ltx_td ltx_align_center">57.02</td>
<td id="A9.T12.1.1.3.2.5" class="ltx_td ltx_align_center" style="background-color:#ECF4FF;"><span id="A9.T12.1.1.3.2.5.1" class="ltx_text" style="background-color:#ECF4FF;">68.48</span></td>
<td id="A9.T12.1.1.3.2.6" class="ltx_td ltx_align_center" style="background-color:#ECF4FF;"><span id="A9.T12.1.1.3.2.6.1" class="ltx_text ltx_font_bold" style="background-color:#ECF4FF;">69.32</span></td>
</tr>
<tr id="A9.T12.1.1.4.3" class="ltx_tr">
<th id="A9.T12.1.1.4.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">Java</th>
<td id="A9.T12.1.1.4.3.2" class="ltx_td ltx_align_center"><span id="A9.T12.1.1.4.3.2.1" class="ltx_text ltx_font_bold">14.28</span></td>
<td id="A9.T12.1.1.4.3.3" class="ltx_td ltx_align_center">13.42</td>
<td id="A9.T12.1.1.4.3.4" class="ltx_td ltx_align_center">10.31</td>
<td id="A9.T12.1.1.4.3.5" class="ltx_td ltx_align_center" style="background-color:#ECF4FF;"><span id="A9.T12.1.1.4.3.5.1" class="ltx_text" style="background-color:#ECF4FF;">12.80</span></td>
<td id="A9.T12.1.1.4.3.6" class="ltx_td ltx_align_center" style="background-color:#ECF4FF;"><span id="A9.T12.1.1.4.3.6.1" class="ltx_text" style="background-color:#ECF4FF;">13.26</span></td>
</tr>
<tr id="A9.T12.1.1.5.4" class="ltx_tr">
<th id="A9.T12.1.1.5.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">JavaScript</th>
<td id="A9.T12.1.1.5.4.2" class="ltx_td ltx_align_center"><span id="A9.T12.1.1.5.4.2.1" class="ltx_text ltx_font_bold">17.61</span></td>
<td id="A9.T12.1.1.5.4.3" class="ltx_td ltx_align_center">13.01</td>
<td id="A9.T12.1.1.5.4.4" class="ltx_td ltx_align_center">13.17</td>
<td id="A9.T12.1.1.5.4.5" class="ltx_td ltx_align_center" style="background-color:#ECF4FF;"><span id="A9.T12.1.1.5.4.5.1" class="ltx_text" style="background-color:#ECF4FF;">15.50</span></td>
<td id="A9.T12.1.1.5.4.6" class="ltx_td ltx_align_center" style="background-color:#ECF4FF;"><span id="A9.T12.1.1.5.4.6.1" class="ltx_text" style="background-color:#ECF4FF;">17.02</span></td>
</tr>
<tr id="A9.T12.1.1.6.5" class="ltx_tr">
<th id="A9.T12.1.1.6.5.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">PHP</th>
<td id="A9.T12.1.1.6.5.2" class="ltx_td ltx_align_center"><span id="A9.T12.1.1.6.5.2.1" class="ltx_text ltx_font_bold">11.24</span></td>
<td id="A9.T12.1.1.6.5.3" class="ltx_td ltx_align_center">9.53</td>
<td id="A9.T12.1.1.6.5.4" class="ltx_td ltx_align_center">9.75</td>
<td id="A9.T12.1.1.6.5.5" class="ltx_td ltx_align_center" style="background-color:#ECF4FF;"><span id="A9.T12.1.1.6.5.5.1" class="ltx_text" style="background-color:#ECF4FF;">10.06</span></td>
<td id="A9.T12.1.1.6.5.6" class="ltx_td ltx_align_center" style="background-color:#ECF4FF;"><span id="A9.T12.1.1.6.5.6.1" class="ltx_text" style="background-color:#ECF4FF;">10.09</span></td>
</tr>
<tr id="A9.T12.1.1.7.6" class="ltx_tr">
<th id="A9.T12.1.1.7.6.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">Python</th>
<td id="A9.T12.1.1.7.6.2" class="ltx_td ltx_align_center">15.96</td>
<td id="A9.T12.1.1.7.6.3" class="ltx_td ltx_align_center"><span id="A9.T12.1.1.7.6.3.1" class="ltx_text ltx_font_bold">17.24</span></td>
<td id="A9.T12.1.1.7.6.4" class="ltx_td ltx_align_center">13.85</td>
<td id="A9.T12.1.1.7.6.5" class="ltx_td ltx_align_center" style="background-color:#ECF4FF;"><span id="A9.T12.1.1.7.6.5.1" class="ltx_text" style="background-color:#ECF4FF;">15.16</span></td>
<td id="A9.T12.1.1.7.6.6" class="ltx_td ltx_align_center" style="background-color:#ECF4FF;"><span id="A9.T12.1.1.7.6.6.1" class="ltx_text" style="background-color:#ECF4FF;">16.06</span></td>
</tr>
<tr id="A9.T12.1.1.8.7" class="ltx_tr">
<th id="A9.T12.1.1.8.7.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">R</th>
<td id="A9.T12.1.1.8.7.2" class="ltx_td ltx_align_center"><span id="A9.T12.1.1.8.7.2.1" class="ltx_text ltx_font_bold">7.52</span></td>
<td id="A9.T12.1.1.8.7.3" class="ltx_td ltx_align_center">4.53</td>
<td id="A9.T12.1.1.8.7.4" class="ltx_td ltx_align_center">4.97</td>
<td id="A9.T12.1.1.8.7.5" class="ltx_td ltx_align_center" style="background-color:#ECF4FF;"><span id="A9.T12.1.1.8.7.5.1" class="ltx_text" style="background-color:#ECF4FF;">6.89</span></td>
<td id="A9.T12.1.1.8.7.6" class="ltx_td ltx_align_center" style="background-color:#ECF4FF;"><span id="A9.T12.1.1.8.7.6.1" class="ltx_text" style="background-color:#ECF4FF;">6.68</span></td>
</tr>
<tr id="A9.T12.1.1.9.8" class="ltx_tr">
<th id="A9.T12.1.1.9.8.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">Ruby</th>
<td id="A9.T12.1.1.9.8.2" class="ltx_td ltx_align_center">10.34</td>
<td id="A9.T12.1.1.9.8.3" class="ltx_td ltx_align_center"><span id="A9.T12.1.1.9.8.3.1" class="ltx_text ltx_font_bold">12.33</span></td>
<td id="A9.T12.1.1.9.8.4" class="ltx_td ltx_align_center">10.37</td>
<td id="A9.T12.1.1.9.8.5" class="ltx_td ltx_align_center" style="background-color:#ECF4FF;"><span id="A9.T12.1.1.9.8.5.1" class="ltx_text" style="background-color:#ECF4FF;">9.44</span></td>
<td id="A9.T12.1.1.9.8.6" class="ltx_td ltx_align_center" style="background-color:#ECF4FF;"><span id="A9.T12.1.1.9.8.6.1" class="ltx_text" style="background-color:#ECF4FF;">10.03</span></td>
</tr>
<tr id="A9.T12.1.1.10.9" class="ltx_tr">
<th id="A9.T12.1.1.10.9.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">Rust</th>
<td id="A9.T12.1.1.10.9.2" class="ltx_td ltx_align_center">6.18</td>
<td id="A9.T12.1.1.10.9.3" class="ltx_td ltx_align_center"><span id="A9.T12.1.1.10.9.3.1" class="ltx_text ltx_font_bold">8.29</span></td>
<td id="A9.T12.1.1.10.9.4" class="ltx_td ltx_align_center">6.77</td>
<td id="A9.T12.1.1.10.9.5" class="ltx_td ltx_align_center" style="background-color:#ECF4FF;"><span id="A9.T12.1.1.10.9.5.1" class="ltx_text" style="background-color:#ECF4FF;">5.99</span></td>
<td id="A9.T12.1.1.10.9.6" class="ltx_td ltx_align_center" style="background-color:#ECF4FF;"><span id="A9.T12.1.1.10.9.6.1" class="ltx_text" style="background-color:#ECF4FF;">5.65</span></td>
</tr>
<tr id="A9.T12.1.1.11.10" class="ltx_tr">
<th id="A9.T12.1.1.11.10.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">TypeScript</th>
<td id="A9.T12.1.1.11.10.2" class="ltx_td ltx_align_center">15.31</td>
<td id="A9.T12.1.1.11.10.3" class="ltx_td ltx_align_center">14.13</td>
<td id="A9.T12.1.1.11.10.4" class="ltx_td ltx_align_center">12.61</td>
<td id="A9.T12.1.1.11.10.5" class="ltx_td ltx_align_center" style="background-color:#ECF4FF;"><span id="A9.T12.1.1.11.10.5.1" class="ltx_text" style="background-color:#ECF4FF;">14.10</span></td>
<td id="A9.T12.1.1.11.10.6" class="ltx_td ltx_align_center" style="background-color:#ECF4FF;"><span id="A9.T12.1.1.11.10.6.1" class="ltx_text ltx_font_bold" style="background-color:#ECF4FF;">15.37</span></td>
</tr>
<tr id="A9.T12.1.1.12.11" class="ltx_tr">
<th id="A9.T12.1.1.12.11.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb ltx_border_t">Avg. 10 Tasks</th>
<td id="A9.T12.1.1.12.11.2" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t"><span id="A9.T12.1.1.12.11.2.1" class="ltx_text ltx_font_bold">18.11</span></td>
<td id="A9.T12.1.1.12.11.3" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t">17.26</td>
<td id="A9.T12.1.1.12.11.4" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t">14.63</td>
<td id="A9.T12.1.1.12.11.5" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t" style="background-color:#ECF4FF;"><span id="A9.T12.1.1.12.11.5.1" class="ltx_text" style="background-color:#ECF4FF;">17.13</span></td>
<td id="A9.T12.1.1.12.11.6" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t" style="background-color:#ECF4FF;"><span id="A9.T12.1.1.12.11.6.1" class="ltx_text" style="background-color:#ECF4FF;">17.75</span></td>
</tr>
</tbody>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 12: </span>Overall results of <span id="A9.T12.3.1" class="ltx_text ltx_font_smallcaps">FuseLLM</span> using OpenLLaMA as the target LLM in code generation evaluations on MultiPL-E (ME).</figcaption>
</figure>
</section>
<section id="A10" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix J </span>Case Studies</h2>

<div id="A10.p1" class="ltx_para ltx_noindent">
<p id="A10.p1.1" class="ltx_p">In Table <a href="#A10.T13" title="Table 13 ‣ Appendix J Case Studies ‣ Knowledge Fusion of Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">13</span></a>, Table <a href="#A10.T14" title="Table 14 ‣ Appendix J Case Studies ‣ Knowledge Fusion of Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">14</span></a>, and Table <a href="#A10.T15" title="Table 15 ‣ Appendix J Case Studies ‣ Knowledge Fusion of Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">15</span></a>, we present case studies to demonstrate how <span id="A10.p1.1.1" class="ltx_text ltx_font_smallcaps">FuseLLM</span> combines the strengths of multiple source LLMs to produce accurate results in different tasks.</p>
</div>
<figure id="A10.T13" class="ltx_table">
<div id="A10.T13.1" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:411.9pt;height:434.7pt;vertical-align:-0.9pt;"><span class="ltx_transformed_inner" style="transform:translate(-15.1pt,15.9pt) scale(0.931722885201843,0.931722885201843) ;">
<table id="A10.T13.1.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A10.T13.1.1.1.1" class="ltx_tr">
<th id="A10.T13.1.1.1.1.1" class="ltx_td ltx_align_justify ltx_align_top ltx_th ltx_th_column ltx_border_t" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="A10.T13.1.1.1.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A10.T13.1.1.1.1.1.1.1" class="ltx_p" style="width:426.8pt;"><span id="A10.T13.1.1.1.1.1.1.1.1" class="ltx_text ltx_font_bold">BBH; Logical Deduction (3 objects);</span></span>
</span>
</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A10.T13.1.1.2.1" class="ltx_tr">
<td id="A10.T13.1.1.2.1.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="A10.T13.1.1.2.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A10.T13.1.1.2.1.1.1.1" class="ltx_p" style="width:426.8pt;"><span id="A10.T13.1.1.2.1.1.1.1.1" class="ltx_text ltx_font_bold">Question:</span> The following paragraphs each describe a set of three objects arranged in a fixed order. The statements are logically consistent within each paragraph. A fruit stand sells three fruits: oranges, watermelons, and pears. The oranges are the second-most expensive. The pears are more expensive than the watermelons.
Options:
(A) The oranges are the second-most expensive
(B) The watermelons are the second-most expensive
(C) The pears are the second-most expensive</span>
</span>
</td>
</tr>
<tr id="A10.T13.1.1.3.2" class="ltx_tr">
<td id="A10.T13.1.1.3.2.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="A10.T13.1.1.3.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="A10.T13.1.1.3.2.1.1.1" class="ltx_p" style="width:426.8pt;"><span id="A10.T13.1.1.3.2.1.1.1.1" class="ltx_text ltx_font_bold">Target:</span> (A)</span>
</span>
</td>
</tr>
<tr id="A10.T13.1.1.4.3" class="ltx_tr">
<td id="A10.T13.1.1.4.3.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="A10.T13.1.1.4.3.1.1" class="ltx_inline-block ltx_align_top">
<span id="A10.T13.1.1.4.3.1.1.1" class="ltx_p" style="width:426.8pt;"><span id="A10.T13.1.1.4.3.1.1.1.1" class="ltx_text ltx_font_bold">OpenLLaMA:</span> Let’s think step by step.
(1) The oranges are the second-most expensive: ”(expensive) ? oranges ? watermelons ? (expensive)”.
(2) The pears are more expensive than the watermelons: ”(expensive) ? oranges pears watermelons (expensive)”.
(3) Combining (1) and (2) we get the following ordering: ”(expensive) oranges pears watermelons (expensive)”.
According to this ordering, the second-most expensive fruit is the oranges.
The oranges are the second-most expensive. So the answer is <span id="A10.T13.1.1.4.3.1.1.1.2" class="ltx_text" style="color:#0000FF;">(A)</span>.</span>
</span>
</td>
</tr>
<tr id="A10.T13.1.1.5.4" class="ltx_tr">
<td id="A10.T13.1.1.5.4.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="A10.T13.1.1.5.4.1.1" class="ltx_inline-block ltx_align_top">
<span id="A10.T13.1.1.5.4.1.1.1" class="ltx_p" style="width:426.8pt;"><span id="A10.T13.1.1.5.4.1.1.1.1" class="ltx_text ltx_font_bold">MPT:</span> Let’s think step by step.
(1) The oranges are the second-most expensive: ”(most expensive)? oranges? (least expensive)”.
(2) The pears are more expensive than the watermelons: ”(most expensive)? oranges? pears? (least expensive)”.
(3) Combining (1) and (2) we get the following ordering: ”(most expensive) oranges pears (least expensive)”.
According to this ordering, the second-most expensive fruit is the oranges.
The oranges are the second-most expensive. So the answer is <span id="A10.T13.1.1.5.4.1.1.1.2" class="ltx_text" style="color:#0000FF;">(A)</span>.</span>
</span>
</td>
</tr>
<tr id="A10.T13.1.1.6.5" class="ltx_tr">
<td id="A10.T13.1.1.6.5.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="A10.T13.1.1.6.5.1.1" class="ltx_inline-block ltx_align_top">
<span id="A10.T13.1.1.6.5.1.1.1" class="ltx_p" style="width:426.8pt;"><span id="A10.T13.1.1.6.5.1.1.1.1" class="ltx_text ltx_font_bold">Llama-2:</span> Let’s think step by step.
(1) The oranges are the second-most expensive: ”(second-most-expensive) ? oranges ? (most-expensive)”.
(2) The pears are more expensive than the watermelons: ”(second-most-expensive) ? oranges pears (most-expensive)”.
(3) Combining (1) and (2) we get the following ordering: ”(second-most-expensive) oranges pears (most-expensive)”.
According to this ordering, the second-most expensive fruit is the pears.
The pears are the second-most expensive. So the answer is <span id="A10.T13.1.1.6.5.1.1.1.2" class="ltx_text" style="color:#FF0000;">(C)</span>.</span>
</span>
</td>
</tr>
<tr id="A10.T13.1.1.7.6" class="ltx_tr">
<td id="A10.T13.1.1.7.6.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="A10.T13.1.1.7.6.1.1" class="ltx_inline-block ltx_align_top">
<span id="A10.T13.1.1.7.6.1.1.1" class="ltx_p" style="width:426.8pt;"><span id="A10.T13.1.1.7.6.1.1.1.1" class="ltx_text ltx_font_bold">Llama-2 CLM:</span> Let’s think step by step.
(1) The oranges are the second-most expensive: ”(second-most) ? oranges ? (most)”.
(2) The pears are more expensive than the watermelons: ”(more) ? pears watermelons (less)”.
(3) Combining (1) and (2) we get the following ordering: ”(second-most) oranges pears (most) watermelons”.
According to this ordering, the second-most expensive fruit is the pears.
The pears are the second-most expensive. So the answer is <span id="A10.T13.1.1.7.6.1.1.1.2" class="ltx_text" style="color:#FF0000;">(C)</span>.</span>
</span>
</td>
</tr>
<tr id="A10.T13.1.1.8.7" class="ltx_tr">
<td id="A10.T13.1.1.8.7.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_b ltx_border_t" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="A10.T13.1.1.8.7.1.1" class="ltx_inline-block ltx_align_top">
<span id="A10.T13.1.1.8.7.1.1.1" class="ltx_p" style="width:426.8pt;"><span id="A10.T13.1.1.8.7.1.1.1.1" class="ltx_text ltx_font_bold ltx_font_smallcaps">FuseLLM<span id="A10.T13.1.1.8.7.1.1.1.1.1" class="ltx_text ltx_font_upright">:</span></span> Let’s think step by step.
(1) The oranges are the second-most expensive: ”(second-most) ? oranges ? (most) ? (least) ? (least) ? (most) ? (second-most)”.
(2) The pears are more expensive than the watermelons: ”(more) ? pears ? (less) ? watermelons ? (most) ? (least) ? (least) ? (most)”.
(3) Combining (1) and (2) we get the following ordering: ”(second-most) oranges pears watermelons (most)”.
According to this ordering, the second-most expensive fruit is the oranges.
The oranges are the second-most expensive. So the answer is <span id="A10.T13.1.1.8.7.1.1.1.2" class="ltx_text" style="color:#0000FF;">(A)</span>.</span>
</span>
</td>
</tr>
</tbody>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 13: </span>Case studies on the Logical Deduction (3 objects) task.</figcaption>
</figure>
<figure id="A10.T14" class="ltx_table">
<div id="A10.T14.1" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:411.9pt;height:613.6pt;vertical-align:-0.9pt;"><span class="ltx_transformed_inner" style="transform:translate(-15.1pt,22.4pt) scale(0.931722885201843,0.931722885201843) ;">
<table id="A10.T14.1.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A10.T14.1.1.1.1" class="ltx_tr">
<th id="A10.T14.1.1.1.1.1" class="ltx_td ltx_align_justify ltx_align_top ltx_th ltx_th_column ltx_border_t" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="A10.T14.1.1.1.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A10.T14.1.1.1.1.1.1.1" class="ltx_p" style="width:426.8pt;"><span id="A10.T14.1.1.1.1.1.1.1.1" class="ltx_text ltx_font_bold">BBH; Hyperbaton;</span></span>
</span>
</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A10.T14.1.1.2.1" class="ltx_tr">
<td id="A10.T14.1.1.2.1.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="A10.T14.1.1.2.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A10.T14.1.1.2.1.1.1.1" class="ltx_p" style="width:426.8pt;"><span id="A10.T14.1.1.2.1.1.1.1.1" class="ltx_text ltx_font_bold">Question:</span> Which sentence has the correct adjective order:
Options:
(A) repulsive prismlike glass shoe
(B) glass repulsive prismlike shoe</span>
</span>
</td>
</tr>
<tr id="A10.T14.1.1.3.2" class="ltx_tr">
<td id="A10.T14.1.1.3.2.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="A10.T14.1.1.3.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="A10.T14.1.1.3.2.1.1.1" class="ltx_p" style="width:426.8pt;"><span id="A10.T14.1.1.3.2.1.1.1.1" class="ltx_text ltx_font_bold">Target:</span> (A)</span>
</span>
</td>
</tr>
<tr id="A10.T14.1.1.4.3" class="ltx_tr">
<td id="A10.T14.1.1.4.3.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="A10.T14.1.1.4.3.1.1" class="ltx_inline-block ltx_align_top">
<span id="A10.T14.1.1.4.3.1.1.1" class="ltx_p" style="width:426.8pt;"><span id="A10.T14.1.1.4.3.1.1.1.1" class="ltx_text ltx_font_bold">OpenLLaMA:</span> Let’s think step by step.
When there is more than one adjective before a noun, the adjectives need to respect the following order before a noun: ”[1. opinion] [2. size] [3. age] [4. shape] [5. color] [6. origin] [7. material] [8. purpose] noun”.
Option (A): ”repulsive prismlike glass shoe”. (1) ”repulsive” falls into the opinion category. (2) ”prismlike” falls into the shape category. (3) ”glass” falls into the material category. Option (A) has the following adjective order: [1. opinion] [3. shape] [6. material] (or, in numeric terms, 1 3 6). Because 1 ¡ 3 ¡ 6 is correct, (A) has the correct ordering.
Option (B): ”glass repulsive prismlike shoe”. Option (B) has the following adjective order: [6. material] [1. opinion] [3. shape] (or, in numeric terms, 6 1 3). Because 6 ¡ 1 ¡ 3 is not correct, (B) does not have the correct ordering. So the answer is <span id="A10.T14.1.1.4.3.1.1.1.2" class="ltx_text" style="color:#0000FF;">(A)</span>.</span>
</span>
</td>
</tr>
<tr id="A10.T14.1.1.5.4" class="ltx_tr">
<td id="A10.T14.1.1.5.4.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="A10.T14.1.1.5.4.1.1" class="ltx_inline-block ltx_align_top">
<span id="A10.T14.1.1.5.4.1.1.1" class="ltx_p" style="width:426.8pt;"><span id="A10.T14.1.1.5.4.1.1.1.1" class="ltx_text ltx_font_bold">MPT:</span> Let’s think step by step.
When there is more than one adjective before a noun, the adjectives need to respect the following order before a noun: ”[1. opinion] [2. size] [3. age] [4. shape] [5. color] [6. origin] [7. material] [8. purpose] noun”.
Option (A): ”repulsive prismlike glass shoe”. (1) ”repulsive” falls into the opinion category. (2) ”prismlike” falls into the shape category. (3) ”glass” falls into the material category. Option (A) has the following adjective order: [1. opinion] [3. material] [2. shape] (or, in numeric terms, 1 3 2 ). Because 1 ¡ 3 ¡ 2 is not correct, (A) does not have the correct ordering.
Option (B): ”glass repulsive prismlike shoe”. Option (B) has the following adjective order: [1. opinion] [3. material] [2. shape] (or, in numeric terms, 1 3 2 ). Because 1 ¡ 3 ¡ 2 is correct, (B) has the correct ordering. So the answer is <span id="A10.T14.1.1.5.4.1.1.1.2" class="ltx_text" style="color:#FF0000;">(B)</span>.</span>
</span>
</td>
</tr>
<tr id="A10.T14.1.1.6.5" class="ltx_tr">
<td id="A10.T14.1.1.6.5.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="A10.T14.1.1.6.5.1.1" class="ltx_inline-block ltx_align_top">
<span id="A10.T14.1.1.6.5.1.1.1" class="ltx_p" style="width:426.8pt;"><span id="A10.T14.1.1.6.5.1.1.1.1" class="ltx_text ltx_font_bold">Llama-2:</span> Let’s think step by step.
When there is more than one adjective before a noun, the adjectives need to respect the following order before a noun: ”[1. opinion] [2. size] [3. age] [4. shape] [5. color] [6. origin] [7. material] [8. purpose] noun”.
Option (A): ”repulsive prismlike glass shoe”. (1) ”repulsive” falls into the opinion category. (2) ”prismlike” falls into the shape category. (3) ”glass” falls into the material category. Option (A) has the following adjective order: [1. opinion] [2. size] [3. age] [6. origin] [7. material] (or, in numeric terms, 1 2 3 6 7). Because 1 ¡ 2 ¡ 3 ¡ 6 ¡ 7 is not correct, (A) does not have the correct ordering.
Option (B): ”glass repulsive prismlike shoe”. Option (B) has the following adjective order: [1. opinion] [2. size] [3. age] [4. shape] [7. material] (or, in numeric terms, 1 2 3 4 7). Because 1 ¡ 2 ¡ 3 ¡ 4 ¡ 7 is correct, (B) has the correct ordering. So the answer is <span id="A10.T14.1.1.6.5.1.1.1.2" class="ltx_text" style="color:#FF0000;">(B)</span>.</span>
</span>
</td>
</tr>
<tr id="A10.T14.1.1.7.6" class="ltx_tr">
<td id="A10.T14.1.1.7.6.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="A10.T14.1.1.7.6.1.1" class="ltx_inline-block ltx_align_top">
<span id="A10.T14.1.1.7.6.1.1.1" class="ltx_p" style="width:426.8pt;"><span id="A10.T14.1.1.7.6.1.1.1.1" class="ltx_text ltx_font_bold">Llama-2 CLM:</span> Let’s think step by step.
When there is more than one adjective before a noun, the adjectives need to respect the following order before a noun: ”[1. opinion] [2. size] [3. age] [4. shape] [5. color] [6. origin] [7. material] [8. purpose] noun”.
Option (A): ”repulsive prismlike glass shoe”. (1) ”repulsive” falls into the opinion category. (2) ”prismlike” falls into the shape category. (3) ”glass” falls into the material category. Option (A) has the following adjective order: [1. opinion] [2. size] [3. age] [4. shape] [5. color] [7. material] (or, in numeric terms, 1 2 3 4 5 7). Because 1 ¡ 2 ¡ 3 ¡ 4 ¡ 5 ¡ 7 is correct, (A) has the correct ordering.
Option (B): ”glass repulsive prismlike shoe”. Option (B) has the following adjective order: [7. material] [1. opinion] [2. size] [3. age] [4. shape] (or, in numeric terms, 7 1 2 3 4). Because 7 ¡ 1 ¡ 2 ¡ 3 ¡ 4 is correct, (B) has the correct ordering. So the answer is <span id="A10.T14.1.1.7.6.1.1.1.2" class="ltx_text" style="color:#FF0000;">(B)</span>.</span>
</span>
</td>
</tr>
<tr id="A10.T14.1.1.8.7" class="ltx_tr">
<td id="A10.T14.1.1.8.7.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_b ltx_border_t" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="A10.T14.1.1.8.7.1.1" class="ltx_inline-block ltx_align_top">
<span id="A10.T14.1.1.8.7.1.1.1" class="ltx_p" style="width:426.8pt;"><span id="A10.T14.1.1.8.7.1.1.1.1" class="ltx_text ltx_font_bold ltx_font_smallcaps">FuseLLM<span id="A10.T14.1.1.8.7.1.1.1.1.1" class="ltx_text ltx_font_upright">:</span></span> Let’s think step by step.
When there is more than one adjective before a noun, the adjectives need to respect the following order before a noun: ”[1. opinion] [2. size] [3. age] [4. shape] [5. color] [6. origin] [7. material] [8. purpose] noun”.
Option (A): ”repulsive prismlike glass shoe”. (1) ”repulsive” falls into the opinion category. (2) ”prismlike” falls into the shape category. (3) ”glass” falls into the material category. Option (A) has the following adjective order: [1. opinion] [2. size] [3. age] [4. shape] [5. color] [6. origin] [7. material] (or, in numeric terms, 1 2 3 4 5 6 7). Because 1 ¡ 2 ¡ 3 ¡ 4 ¡ 5 ¡ 6 ¡ 7 is correct, (A) has the correct ordering.
Option (B): ”glass repulsive prismlike shoe”. Option (B) has the following adjective order: [1. opinion] [7. material] [2. size] [3. age] [4. shape] [5. color] (or, in numeric terms, 1 7 2 3 4 5). Because 1 ¡ 7 ¡ 2 ¡ 3 ¡ 4 ¡ 5 is not correct, (B) does not have the correct ordering. So the answer is <span id="A10.T14.1.1.8.7.1.1.1.2" class="ltx_text" style="color:#0000FF;">(A)</span>.</span>
</span>
</td>
</tr>
</tbody>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 14: </span>Case studies on the Hyperbaton task.</figcaption>
</figure>
<figure id="A10.T15" class="ltx_table">
<div id="A10.T15.1" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:411.9pt;height:601.1pt;vertical-align:-0.9pt;"><span class="ltx_transformed_inner" style="transform:translate(-15.1pt,22.0pt) scale(0.931722885201843,0.931722885201843) ;">
<table id="A10.T15.1.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A10.T15.1.1.1.1" class="ltx_tr">
<th id="A10.T15.1.1.1.1.1" class="ltx_td ltx_align_justify ltx_align_top ltx_th ltx_th_column ltx_border_t" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="A10.T15.1.1.1.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A10.T15.1.1.1.1.1.1.1" class="ltx_p" style="width:426.8pt;"><span id="A10.T15.1.1.1.1.1.1.1.1" class="ltx_text ltx_font_bold">BBH; Disambiguation QA;</span></span>
</span>
</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A10.T15.1.1.2.1" class="ltx_tr">
<td id="A10.T15.1.1.2.1.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="A10.T15.1.1.2.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A10.T15.1.1.2.1.1.1.1" class="ltx_p" style="width:426.8pt;"><span id="A10.T15.1.1.2.1.1.1.1.1" class="ltx_text ltx_font_bold">Question:</span> In the following sentences, explain the antecedent of the pronoun (which thing the pronoun refers to), or state that it is ambiguous.
Sentence: The technician told the customer that he could pay with cash.
Options:
(A) The technician could pay
(B) The customer could pay
(C) Ambiguous</span>
</span>
</td>
</tr>
<tr id="A10.T15.1.1.3.2" class="ltx_tr">
<td id="A10.T15.1.1.3.2.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="A10.T15.1.1.3.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="A10.T15.1.1.3.2.1.1.1" class="ltx_p" style="width:426.8pt;"><span id="A10.T15.1.1.3.2.1.1.1.1" class="ltx_text ltx_font_bold">Target:</span> (B)</span>
</span>
</td>
</tr>
<tr id="A10.T15.1.1.4.3" class="ltx_tr">
<td id="A10.T15.1.1.4.3.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="A10.T15.1.1.4.3.1.1" class="ltx_inline-block ltx_align_top">
<span id="A10.T15.1.1.4.3.1.1.1" class="ltx_p" style="width:426.8pt;"><span id="A10.T15.1.1.4.3.1.1.1.1" class="ltx_text ltx_font_bold">OpenLLaMA:</span> Let’s think step by step.
Here we need to determine who the pronoun ”he” might be referring to. There are two possible referents for ”he”, namely the technician and the customer. The verb ”told” might be able to help us determine which one is more likely (if either). Let X be the technician and Y the customer. The sentence is then of the form ”X told Y that (X or Y) could pay with cash.”
Let’s consider Y first: ”X told Y that Y could pay with cash.” This case makes sense, because X might want to inform Y that Y could pay with cash.
Now, consider X: ”X told Y that X could pay with cash.” This case also makes sense, because X might want to inform Y that X could pay with cash.
Because both X and Y are possible at the same time, we conclude that the antecedent of the pronoun is ambiguous. So the answer is <span id="A10.T15.1.1.4.3.1.1.1.2" class="ltx_text" style="color:#FF0000;">(C)</span>.</span>
</span>
</td>
</tr>
<tr id="A10.T15.1.1.5.4" class="ltx_tr">
<td id="A10.T15.1.1.5.4.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="A10.T15.1.1.5.4.1.1" class="ltx_inline-block ltx_align_top">
<span id="A10.T15.1.1.5.4.1.1.1" class="ltx_p" style="width:426.8pt;"><span id="A10.T15.1.1.5.4.1.1.1.1" class="ltx_text ltx_font_bold">MPT:</span> Let’s think step by step.
Here we need to determine who the pronoun ”he” might be referring to. There are two possible referents for ”he”, namely the technician and the customer. The verb ”told” might be able to help us determine which one is more likely (if either). Let X be the technician and Y the customer. The sentence is then of the form ”X told Y that (X or Y) could pay.”
Let’s consider Y first: ”X told Y that Y could pay.” This case makes sense, because Y might want to pay with cash.
Now, consider X: ”X told Y that X could pay.” This case doesn’t make sense, because X would already have the information that X could pay, because it is information about themself.
Because in this context, X is the technician and Y is the customer, the answer should be the customer. So the answer is <span id="A10.T15.1.1.5.4.1.1.1.2" class="ltx_text" style="color:#0000FF;">(B)</span>.</span>
</span>
</td>
</tr>
<tr id="A10.T15.1.1.6.5" class="ltx_tr">
<td id="A10.T15.1.1.6.5.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="A10.T15.1.1.6.5.1.1" class="ltx_inline-block ltx_align_top">
<span id="A10.T15.1.1.6.5.1.1.1" class="ltx_p" style="width:426.8pt;"><span id="A10.T15.1.1.6.5.1.1.1.1" class="ltx_text ltx_font_bold">Llama-2:</span> Let’s think step by step.
Here we need to determine who the pronoun ”he” might be referring to. There are two possible referents for ”he”, namely the technician and the customer. The verb ”told” might be able to help us determine which one is more likely (if either). Let X be the technician and Y the customer. The sentence is then of the form ”X told Y that (X or Y) could do something.”
Let’s consider Y first: ”X told Y that Y could do something.” This case makes sense, because Y is the one who is being told that they can do something.
Now, consider X: ”X told Y that X could do something.” This case doesn’t make sense, because X is the one who is telling Y that they can do something.
Because in this context, X is the technician and Y is the customer, the answer should be the technician. So the answer is <span id="A10.T15.1.1.6.5.1.1.1.2" class="ltx_text" style="color:#FF0000;">(A)</span>.</span>
</span>
</td>
</tr>
<tr id="A10.T15.1.1.7.6" class="ltx_tr">
<td id="A10.T15.1.1.7.6.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="A10.T15.1.1.7.6.1.1" class="ltx_inline-block ltx_align_top">
<span id="A10.T15.1.1.7.6.1.1.1" class="ltx_p" style="width:426.8pt;"><span id="A10.T15.1.1.7.6.1.1.1.1" class="ltx_text ltx_font_bold">Llama-2 CLM:</span> Let’s think step by step.
Here we need to determine who the pronoun ”he” might be referring to. There are two possible referents for ”he”, namely the technician and the customer. The verb ”told” might be able to help us determine which one is more likely (if either). Let X be the technician and Y the customer. The sentence is then of the form ”X told Y that (X or Y) could pay.”
Let’s consider Y first: ”X told Y that Y could pay.” This case doesn’t make sense, because Y is the receiver of the information, so it is now X’s turn to pay.
Now, consider X: ”X told Y that X could pay.” This case makes sense, because X is the one who is telling Y that X can pay.
Because in this context, X is the technician and Y is the customer, the answer should be the technician. So the answer is <span id="A10.T15.1.1.7.6.1.1.1.2" class="ltx_text" style="color:#FF0000;">(A)</span>.</span>
</span>
</td>
</tr>
<tr id="A10.T15.1.1.8.7" class="ltx_tr">
<td id="A10.T15.1.1.8.7.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_b ltx_border_t" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="A10.T15.1.1.8.7.1.1" class="ltx_inline-block ltx_align_top">
<span id="A10.T15.1.1.8.7.1.1.1" class="ltx_p" style="width:426.8pt;"><span id="A10.T15.1.1.8.7.1.1.1.1" class="ltx_text ltx_font_bold ltx_font_smallcaps">FuseLLM<span id="A10.T15.1.1.8.7.1.1.1.1.1" class="ltx_text ltx_font_upright">:</span></span> Let’s think step by step.
Here we need to determine who the pronoun ”he” might be referring to. There are two possible referents for ”he”, namely the technician and the customer. The verb ”told” might be able to help us determine which one is more likely (if either). Let X be the technician and Y the customer. The sentence is then of the form ”X told Y that (X or Y) could pay.”
Let’s consider Y first: ”X told Y that Y could pay.” This case makes sense, because Y is the customer, and the customer is the one who is paying.
Now, consider X: ”X told Y that X could pay.” This case doesn’t make sense, because X is the technician, and the technician is not the one who is paying.
Because in this context, X is the technician and Y is the customer, the answer should be the customer. So the answer is <span id="A10.T15.1.1.8.7.1.1.1.2" class="ltx_text" style="color:#0000FF;">(B)</span>.</span>
</span>
</td>
</tr>
</tbody>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 15: </span>Case studies on the Disambiguation QA task.</figcaption>
</figure>
</section>
</article>
</div>
<div class="ar5iv-footer"><a href="/html/2401.10490" class="ar5iv-nav-button ar5iv-nav-button-prev">◄</a>
    <a class="ar5iv-home-button" href="/"><img height="40" alt="ar5iv homepage" src="/assets/ar5iv.png"></a>
    <a href="/feeling_lucky" class="ar5iv-text-button">Feeling<br>lucky?</a>
    <a href="/log/2401.10491" class="ar5iv-text-button ar5iv-severity-warning">Conversion<br>report</a>
    <a class="ar5iv-text-button" target="_blank" href="https://github.com/dginev/ar5iv/issues/new?template=improve-article--arxiv-id-.md&title=Improve+article+2401.10491">Report<br>an issue</a>
    <a href="https://arxiv.org/abs/2401.10491" class="ar5iv-text-button arxiv-ui-theme">View&nbsp;original<br>on&nbsp;arXiv</a><a href="/html/2401.10492" class="ar5iv-nav-button ar5iv-nav-button-next">►</a>
</div><footer class="ltx_page_footer">
<a class="ar5iv-toggle-color-scheme" href="javascript:toggleColorScheme()" title="Toggle ar5iv color scheme"><span class="color-scheme-icon"></span></a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/license" target="_blank">Copyright</a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/policies/privacy_policy" target="_blank">Privacy Policy</a>

<div class="ltx_page_logo">Generated  on Tue Feb 27 08:19:55 2024 by <a target="_blank" href="http://dlmf.nist.gov/LaTeXML/" class="ltx_LaTeXML_logo"><span style="letter-spacing:-0.2em; margin-right:0.1em;">L<span class="ltx_font_smallcaps" style="position:relative; bottom:2.2pt;">a</span>T<span class="ltx_font_smallcaps" style="font-size:120%;position:relative; bottom:-0.2ex;">e</span></span><span style="font-size:90%; position:relative; bottom:-0.2ex;">XML</span><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAsAAAAOCAYAAAD5YeaVAAAAAXNSR0IArs4c6QAAAAZiS0dEAP8A/wD/oL2nkwAAAAlwSFlzAAALEwAACxMBAJqcGAAAAAd0SU1FB9wKExQZLWTEaOUAAAAddEVYdENvbW1lbnQAQ3JlYXRlZCB3aXRoIFRoZSBHSU1Q72QlbgAAAdpJREFUKM9tkL+L2nAARz9fPZNCKFapUn8kyI0e4iRHSR1Kb8ng0lJw6FYHFwv2LwhOpcWxTjeUunYqOmqd6hEoRDhtDWdA8ApRYsSUCDHNt5ul13vz4w0vWCgUnnEc975arX6ORqN3VqtVZbfbTQC4uEHANM3jSqXymFI6yWazP2KxWAXAL9zCUa1Wy2tXVxheKA9YNoR8Pt+aTqe4FVVVvz05O6MBhqUIBGk8Hn8HAOVy+T+XLJfLS4ZhTiRJgqIoVBRFIoric47jPnmeB1mW/9rr9ZpSSn3Lsmir1fJZlqWlUonKsvwWwD8ymc/nXwVBeLjf7xEKhdBut9Hr9WgmkyGEkJwsy5eHG5vN5g0AKIoCAEgkEkin0wQAfN9/cXPdheu6P33fBwB4ngcAcByHJpPJl+fn54mD3Gg0NrquXxeLRQAAwzAYj8cwTZPwPH9/sVg8PXweDAauqqr2cDjEer1GJBLBZDJBs9mE4zjwfZ85lAGg2+06hmGgXq+j3+/DsixYlgVN03a9Xu8jgCNCyIegIAgx13Vfd7vdu+FweG8YRkjXdWy329+dTgeSJD3ieZ7RNO0VAXAPwDEAO5VKndi2fWrb9jWl9Esul6PZbDY9Go1OZ7PZ9z/lyuD3OozU2wAAAABJRU5ErkJggg==" alt="Mascot Sammy"></a>
</div></footer>
</div>

    <script>
      var canMathML = typeof(MathMLElement) == "function";
      if (!canMathML) {
        var body = document.querySelector("body");
        body.firstElementChild.setAttribute('style', 'opacity: 0;');
        var loading = document.createElement("div");
        loading.setAttribute("id", "mathjax-loading-spinner");
        var message = document.createElement("div");
        message.setAttribute("id", "mathjax-loading-message");
        message.innerText = "Typesetting Equations...";
        body.prepend(loading);
        body.prepend(message);

        var el = document.createElement("script");
        el.src = "https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js";
        document.querySelector("head").appendChild(el);

        window.MathJax = {
          startup: {
            pageReady: () => {
              return MathJax.startup.defaultPageReady().then(() => {
                body.removeChild(loading);
                body.removeChild(message);
                body.firstElementChild.removeAttribute('style');
              }); } } };
      }
    </script>
    <script>
    // Auxiliary function, building the preview feature when
    // an inline citation is clicked
    function clicked_cite(e) {
      e.preventDefault();
      let cite = this.closest('.ltx_cite');
      let next = cite.nextSibling;
      if (next && next.nodeType == Node.ELEMENT_NODE && next.getAttribute('class') == "ar5iv-bibitem-preview") {
        next.remove();
        return; }
      // Before adding a preview modal,
      // cleanup older previews, in case they're still open
      document.querySelectorAll('span.ar5iv-bibitem-preview').forEach(function(node) {
        node.remove();
      })

      // Create the preview
      preview = document.createElement('span');
      preview.setAttribute('class','ar5iv-bibitem-preview');
      let target = document.getElementById(this.getAttribute('href').slice(1));
      target.childNodes.forEach(function (child) {
        preview.append(child.cloneNode(true));
      });
      let close_x = document.createElement('button');
      close_x.setAttribute("aria-label","Close modal for bibliography item preview");
      close_x.textContent = "×";
      close_x.setAttribute('class', 'ar5iv-button-close-preview');
      close_x.setAttribute('onclick','this.parentNode.remove()');
      preview.append(close_x);
      preview.querySelectorAll('.ltx_tag_bibitem').forEach(function(node) {
        node.remove();
      });
      cite.parentNode.insertBefore(preview, cite.nextSibling);
      return;
    }
    // Global Document initialization:
    // - assign the preview feature to all inline citation links
    document.querySelectorAll(".ltx_cite .ltx_ref").forEach(function (link) {
      link.addEventListener("click", clicked_cite);
    });
    </script>
    </body>
</html>

2401.10491
While training large language models (LLMs)  from scratch can generate models with distinct functionalities and strengths, it comes at significant costs and may result in redundant capabilities. Alternatively, a cost-effective and compelling approach is to merge existing pre-trained LLMs into a more potent model. However, due to the varying architectures of these LLMs, directly blending their weights is impractical. In this paper, we introduce the notion of knowledge fusion for LLMs, aimed at combining the capabilities of existing LLMs and transferring them into a single LLM. By leveraging the generative distributions of source LLMs, we externalize their collective knowledge and unique strengths, thereby potentially elevating the capabilities of the target model beyond those of any individual source LLM. We validate our approach using three popular LLMs with different architectures—Llama-2, MPT, and OpenLLaMA—across various benchmarks and tasks. Our findings confirm that the fusion of LLMs can improve the performance of the target model across a range of capabilities such as reasoning, commonsense, and code generation. Our code, model weights, and data are public athttps://github.com/fanqiwan/FuseLLM.
abstract
Knowledge Fusion of Large Language Models
Abstract
2401.10491
1IntroductionWith the continuous success of large language models (LLMs)  such as GPTand LLaMAseries across a wide range of natural language processing (NLP)  tasks, it has become a strategic imperative for corporations to create their own LLMs. However, the costs associated with LLM development are astronomical. In addition to requiring vast amounts of training data, advanced techniques, substantial computational resources, and skilled labor, the development process also exerts significant pressure on energy consumption and the environment. While these LLMs exhibit structural and functional differences, they share similar capabilities across a spectrum of NLP tasks. Consequently, beyond the traditional approach of training an LLM from scratch, an alternative option is to combine existing LLMs into a new, more powerful one, which is termedknowledge fusion of LLMsin this paper. If successful, this fusion not only cuts the cost of initial training but also allows the integrated model to benefit from the strengths of all the LLMs. This new model can also be fine-tuned and adapted for various downstream tasks. Moreover, the fusion can also happen among fine-tuned LLMs that specialize in a specific task.The endeavor to integrate the capabilities of multiple models has been a long-standing pursuit. For example, ensemble methodsdirectly aggregate the outputs of different models to enhance prediction performance and robustness. However, this approach requires maintaining multiple trained models and executing each during inference, which is impractical for LLMs due to their substantial memory and inference time requirements. Likewise, this approach doesn’t facilitate fine-tuning, which is essential for many LLMs. Another approach is to directly merge several neural networks into a single network through parameter-wise arithmetic operations. This approach typically assumes uniform network architectures and attempts to establish mappings between the weights of distinct neural networks, which is often unattainable in the context of LLMs. Moreover, weight merging may lead to suboptimal results when substantial differences exist in the parameter space.In this paper, we explore the fusion of LLMs from a probabilistic distribution perspective. For an input text, we argue that the probabilistic distributions generated by different source LLMs can reflect their inherent knowledge in understanding this text.
Therefore, the proposedFuseLLMleverages the generative distributions of source LLMs to externalize both their collective knowledge and individual strengths and transfer them to the target LLM through lightweight continual training. To achieve this, we develop a new strategy for aligning tokenizations originating from different LLMs and explore two methods for fusing the probability distributions generated by these diverse LLMs. During the continual training,FuseLLMplaces significant emphasis on minimizing the divergence between the target LLM’s probabilistic distributions and those of the source LLMs.To empirically demonstrate the effectiveness ofFuseLLM, we examine a challenging yet general scenario of LLMs fusion, where the source models share minimal commonalities. Specifically, we focus on three popular open-source LLMs that possess distinct architectures and functionalities: Llama-2, OpenLLaMA, and MPT.
Evaluations across three benchmarks, which consist of a total of 42 tasks spanning reasoning, commonsense, and code generation, confirm that the target model trained by our method outperforms each source LLM and the baseline in most tasks. Moreover, we simulate the existence of functionally distinct LLMs with identical architecture by continually training a single base model on several domain-specific corpora. When evaluated based on perplexity, our method demonstrates superior potential in combining the capabilities of these structurally identical LLMs compared to traditional ensemble and weight merging methods.To sum up, this paper explores a novel challenge called LLMs fusion, with the goal of creating a unified model that effectively utilizes the collective capabilities and unique strengths of diverse LLMs. Illustrated in Figure1, our proposed approach distinguishes itself from traditional ensemble and weight merging techniques by prioritizing the fusion of multiple LLMs through knowledge externalization and transfer. This study yields several findings that may spark future research. Firstly, while we demonstrate the effectiveness of our method through lightweight continual training on a compact, high-quality corpus, the thoughtful selection of the training corpus can be a crucial consideration, particularly with regard to its relevance to downstream tasks. Secondly, in scenarios where the capabilities of source LLMs vary significantly, the fusion function appears to be crucial in effectively combining their respective strengths. Lastly, when compared to traditional model ensemble and merging techniques, the field of LLMs fusion appears to be a more promising avenue for exploration, especially in light of the diverse structures and substantial model sizes of LLMs.Figure 1:Illustration of conventional model fusion techniques (ensemble and weight merging)  and our knowledge fusion approach for LLMs (FuseLLM) . Different animal icons represent different LLMs, with various species denoting LLMs possessing differing architectures.FuseLLMexternalizes the knowledge from multiple LLMs and transfers their capabilities to a target LLM.
introduction
Knowledge Fusion of Large Language Models
1Introduction
2401.10491
2Related WorkModel FusionThe integration of capabilities from diverse models has been a long-standing objective, with existing approaches mainly falling into two categories. Firstly, the traditional technique of modelensemblecombines the outputs of multiple models to enhance overall system performance. Note that this technique doesn’t involve the explicit merging of multiple models into a new one. Common methods for model ensemble typically employ weighted averagingor majority votingto consolidate predictions from various models. Recently,introduced an ensemble framework designed to leverage the diverse strengths of multiple open-source LLMs. This framework first employs a pairwise comparison method to detect subtle distinctions among candidate outputs. Then, it combines the top-ranked candidates to produce an enhanced output, capitalizing on their strengths while mitigating their weaknesses.Secondly,weight mergingpresents another approach that facilitates model fusion at the parameter level.andmerged weights from models with identical structures, obtained through different strategies or configurations, to achieve improved overall performance. Similarly,,, andexplored weighted averaging of models derived from different configurations to enhance out-of-distribution generalization. Furthermore,merged models designed for specific domains or tasks to create a generalist capable of addressing all domains or tasks. Going beyond parameter merging of entire models,,, andapplied linear mathematical operations to adapter parameters to achieve superior generalization performance.In a nutshell, while model ensemble requires the parallel deployment of multiple models, weight merging is generally limited to models with identical architectures. In contrast, the approach proposed in this paper supports the fusion of multiple LLMs with diverse architectures by explicitly transferring their knowledge and capabilities to a target LLM.Knowledge DistillationKnowledge distillation, initially proposed for model compression, involves training a student model under the guidance of one or more teacher models. In the NLP community, knowledge distillation has been widely applied to text classification tasks. These applications include training the student model to replicate the teacher’s output distribution, as well as featuresand relationsderived from intermediate layers of the teacher model. In the realm of text generation, the conventional approach focuses on minimizing the KL divergence between the student and teacher generation distributions.
This is achieved by using the teacher’s probability distributions at each time step as supervisionor by directly training on the teacher’s generated texts.While our method shares a framework similar to multi-teacher knowledge distillation, there are two significant distinctions. First, in traditional knowledge distillation, the student models are typically constrained to be smaller in size than the teachers. In our scenario, however, there are no limitations on the size of the target model. Second, traditional knowledge distillation often results in the student models lagging behind the teachers in performance after distillation. In contrast, we anticipate that after the fusion, the target model will surpass any of the source models in performance.
body
Knowledge Fusion of Large Language Models
2Related Work
2401.10491
3Knowledge Fusion of LLMsThe primary objective of LLMs fusion is to externalize the collective knowledge embedded within multiple source LLMs and integrate their capabilities into a target LLM. Givensource LLMswith varying architectures, each having undergone individual pre-training or fine-tuning on distinct datasets, the key idea behind our approach is to initially stimulate LLMs to manifest their inherent knowledge by challenging them to predict the next token. The probabilistic distributions of these predictions are thoroughly assessed, and the most accurate predictions are utilized to continually train the target LLMon a corpususing the causal language modeling objective. In the following sections, we start with a brief introduction to the preliminaries, followed by a detailed explanation of our LLMs fusion framework. Finally, we delve into the implementation details.3.1PreliminariesLetdenote a text sequence of lengthsampled from the corpusanddenote the sequence preceding theth token.
The causal language modeling (CLM)  objective for training a language model parameterized byis defined as minimizing the negative log-likelihood:(1) whereis the model’s predicted probability for tokengiven the preceding tokens.The above objective decomposes sequence likelihood into token-level cross-entropy losses, comparing each token’s predicted distribution to its one-hot representation. To provide a more generalized perspective, we reframe this token-level view into a sequential distribution format.
Specifically, for the text sequence, we aggregate token-level predictions and create a probabilistic distribution matrix,, where the-th row represents the distribution predicted by the model for theth token over the vocabulary of size. The CLM objective can then be interpreted as reducing the discrepancy betweenand the one-hot label matrix,, where each row is a one-hot representation of the corresponding gold token.
Formally, the CLM objective is transformed into the following representation:(2) whererepresents the discrepancy function between two matrices, and it is equivalent to Eq.1when implemented as the KL divergence.3.2LLMs FusionTaking this perspective on a language model, we argue that the probabilistic distribution matrix can reflect its certain inherent knowledge in understanding the text. Consequently, different probabilistic distribution matrices for the same text, originating from various LLMs, can be used to represent the diverse knowledge embedded within these models. Acknowledging this, the proposedFuseLLMapproach tackles LLMs fusion through probabilistic modeling, aiming to create a unified LLM by merging the probabilistic distributions of the source LLMs. To achieve this, when starting with a set of LLMs to fuse,FuseLLMundergoes lightweight continual training of the target LLM on a raw text corpus that mirrors the pre-training dataset. Instead of relying solely on the CLM objective,FuseLLMplaces significant emphasis on minimizing the divergence between the target LLM’s probabilistic distributions and those of the source LLMs.For each text in the corpus, we apply the providedsource LLMs and obtain a set of probabilistic distribution matrices, denoted as, whererepresents the parameters of theth LLM. Utilizing these matrices, we externalize the knowledge from individual models into a unified space, essentially creating unified probabilistic representations over the text.
We acknowledge that variances in vocabulary among the source LLMs can lead to misaligned matrices. To address this, we employ a token alignment strategy, which is explained in Section3.3, to foster more coherent probabilistic interpretations across models.Having aligned the probabilistic matrices, we proceed to fuse them into a single compact representation. Various fusion strategies can be applied for this purpose, as detailed in Section3.3. We useto represent the fused representation matrix as follows:(3) wheredenotes the function that combines multiple matrices, and the resulting matrixis seen as a representation of the collective knowledge and distinctive strengths of the source LLMs.To transfer the capabilities of source LLMs to the target LLM, we enforce alignment between the target LLM’s predictions and the fused representation matrix. We useto represent the output distribution matrix of the target LLM for text, and then define our fusion objective as follows:(4) The overall objective for our continual training consists of a weighted combination of the causal language modeling objectiveand the fusion objectiveas follows:(5) 3.3Implementation ofFuseLLMIn this section, we present the implementation details of token alignment and the fusion function for fusing different LLMs in ourFuseLLMmethod.Token AlignmentEnsuring token alignment across multiple LLMs is crucial for effective knowledge fusion, as it guarantees proper mapping of probabilistic distribution matrices.employed dynamic programming to recursively minimize the total cost of editing one sequence of tokens to match the other.
If a one-to-one mapping exists between two tokens, the corresponding distributions are perfectly mapped. Otherwise, the mapped distribution degenerates into a one-hot vector.
Since tokens generated by different tokenizers for the same sequence typically exhibit limited differences, we propose to enhance the success rate of token alignment by replacing the exact match (EM)  constraint inwith a minimum edit distance (MinED)  strategy, which maps tokens from different tokenizers based on MinED.
This relaxation of token alignment helps preserve substantial information in the distribution matrices while introducing minor errors. For more details of the token alignment, please refer to AppendixA.Fusion StrategiesTo combine the collective knowledge of source LLMs while preserving their unique strengths, it is essential to evaluate the quality of different LLMs and assign varying levels of importance to their respective distribution matrices. For this purpose, when dealing with text, we utilize the cross-entropy loss between the distribution matrices and the gold labels as an indicator of the prediction quality of the LLMs. A lower cross-entropy score for a source LLM signifies a more accurate understanding of the text, and its prediction should be accorded greater significance. Based on this criterion, we introduce two fusion functions: (1)  MinCE: This function outputs the distribution matrix with the minimum cross-entropy score; (2)  AvgCE: This function produces a weighted average of the distribution matrices based on cross-entropy scores.The complete process of theFuseLLMmethod is described in Algorithm1.Algorithm 1FuseLLMfor LLMs Fusion0:Source LLMs, training corpus.1:Initialize the target LLMwith one of the source LLMs.2:fortextindo3:Apply thesource LLMs to compute probabilistic distribution matrices.4:Alignusing the MinED alignment method.5:Fuseto obtainwith the MinCE or AvgCE fusion function.6:Update parameters ofby minimizing the overall loss fuction in Eq.5.7:endfor8:return.
body
Knowledge Fusion of Large Language Models
3Knowledge Fusion of LLMs
2401.10491
4ExperimentsIn our experiments, we consider a general but challenging scenario of LLMs fusion where the source models share minimal commonalities in architectures or functionalities. Specifically, we conduct experiments on the 7B scale and select three representative open-source models: Llama-2, OpenLLaMA, and MPT as the source LLMs for fusion. Regarding the target LLM, we opt for another Llama-2 7B, which is generally the most robust one among the three source LLMs. The target LLM starts with the same pre-trained weights as its source counterpart but differs in that it updates parameters during training. To evaluate the performance ofFuseLLM, we conduct experiments on benchmarks assessing the capabilities of LLMs in reasoning, commonsense, and code generation.4.1Experimental SetupDataset for continual trainingTo continually train the target LLM for LLMs fusion, it is essential to
have a compact yet diverse training dataset. We have chosen MiniPile, a meticulously curated dataset resulting from a thorough clustering and filtering process.
MiniPile comprises approximately 1 million documents across 22
domains and 1.8 billion tokens, constituting less than 0.1% of the 2 trillion training tokens of Llama-2.
More dataset details can be found in AppendixB.Fusion functionFor the fusion function, we use the minimum cross-entropy (MinCE) . However, the impact of employing alternative fusion functions will be examined in Section4.4.Training detailsWe train the target LLM of Llama-2 7B using a batch size of 128 and a maximum length of 2048 on a single node equipped with 8 NVIDIA A100 GPUs, each with 40GB of memory. Our training framework is implemented based on the Huggingface Transformersand accelerated with FlashAttention. We empirically set the combination weightin Eq.5to 0.9. The training consists of only a single epoch, which takes approximately 33 hours. For further hyper-parameter details, please refer to AppendixC.EvaluationWe evaluateFuseLLMon three benchmarks that represent different core capabilities of LLMs, spanningreasoning,commonsense, andcode generation.•Big-Bench Hard (BBH) is a benchmark to evaluate the generalreasoningability of LLMs. It contains 23 multiple-choice tasks and 4 free-form generation tasks from the Big-Bench, which can be classified into four categories: algorithmic and arithmetic reasoning, natural language understanding, world knowledge, and multilingual knowledge and reasoning. We follow previous workto generate the predictions based on few-shot chain-of-thought (CoT)  prompts and then calculate the exact match (EM)  accuracy.•Common Sense (CS)  is a benchmark to evaluate thecommonsensecapability of LLMs. We consider 5 standard multiple-choice tasks: ARC easy and challenge, BoolQ, HellaSwag, and OpenBookQA. We employ lm-eval-hardnessto conduct a likelihood-based zero-shot evaluation. Specifically, we select the option with the highest likelihood given the context and report the accuracy.•MultiPL-E (ME) is a multilingual programming benchmark to assess thecodingability of LLMs. It is translated from the Python benchmarkinto parallel datasets in 18 programming languages. We use the bigcode-evaluation-hardnessto perform zero-shot code generation in 10 popular programming languages in the HumanEval category and report the pass@1based on 20 generated samples for each question.BaselinesIn our experiments, we compare ourFuseLLMwith two sets of baselines: (1) original LLMs, including Llama-2 7B, OpenLLaMA 7B, and MPT 7B; and (2) Llama-2 CLM: continually trained Llama-2 7B on MiniPile using only the casual language modeling objective.4.2Overall ResultsTable1presents the overall results ofFuseLLMin comparison to the baseline methods on BBH. We can observe that the three source LLMs exhibit varying performance across the 27 BBH tasks, with Llama-2 generally outperforming the others. After continual training with a compact and diverse corpus, Llama-2 CLM shows a relative improvement of 1.86% compared to Llama-2, although this improvement is relatively modest and inconsistent across tasks. On average,FuseLLMdemonstrates an average relative performance gain of 5.16% over the original Llama-2 across all 27 tasks. In specific tasks, the enhancements achieved byFuseLLMare substantial (e.g., from 54.40 to 65.20 in the Hyperbaton task) . In tasks such as Dick Languages where simple continual pre-training leads to a decline in performance,FuseLLMleverages the combined strengths of individual source LLMs to recover performance improvements. Note thatFuseLLMoccasionally exhibits degraded performance on tasks such as Geometric Shapes and Word Sorting, which could be attributed to two reasons. First, the other source LLMs, apart from Llama-2, perform poorly on these tasks, affecting the fusion results. Second, the relevance between the continual training dataset and downstream tasks also contributes to the performance degradation.Table2shows the zero-shot performance ofFuseLLMand the baseline methods on the Common Sense (CS)  benchmark. The results demonstrate thatFuseLLMconsistently surpasses the baselines across all five tasks, achieving a relative performance improvement of 1.25% over Llama-2. In contrast, Llama-2 CLM exhibits a marginal improvement, with only a 0.16% relative enhancement compared to Llama-2. Notably, substantial improvements from Llama-2 toFuseLLMare observed in the challenging ARC-challenge (2.40%)  and OpenBookQA (2.71%)  tasks, highlighting the effectiveness ofFuseLLMin leveraging collective knowledge to address intricate problems.TaskOpenLLaMAMPTLlama-2Llama-2 CLMFuseLLMBoolean Expressions74.4066.0068.8076.00(+10.47%) 71.60(+4.07%) Causal Judgement45.4550.8050.8046.52(-8.43%) 46.52(-8.43%) Date Understanding43.6043.6059.6059.20(-0.67%) 62.40(+4.70%) Disambiguation QA36.0047.6046.8048.00(+2.56%) 50.00(+6.84%) Dyck Languages5.205.207.206.40(-11.11%) 8.80(+22.22%) Formal Fallacies50.8052.8049.2048.80(-0.81%) 49.20(+0.00%) Geometric Shapes0.000.0034.4019.20(-44.17%) 22.80(-33.72%) Hyperbaton62.8053.6054.4056.40(+3.68%) 65.20(+19.85%) Logical Deduction (3 objects) 43.6040.8054.0057.20(+5.93%) 60.40(+11.85%) Logical Deduction (5 objects) 24.8031.6031.2035.60(+14.10%) 33.20(+6.41%) Logical Deduction (7 objects) 16.8018.4024.8029.60(+19.35%) 25.60(+3.23%) Movie Recommendation39.6052.0072.8071.60(-1.65%) 73.60(+1.10%) Multistep Arithmetic Two0.800.400.804.40(+450.00%) 4.80(+500.00%) Navigate54.0048.8056.0061.20(+9.29%) 64.40(+15.00%) Object Counting49.6040.4049.6051.60(+4.03%) 55.20(+11.29%) Penguins in a Table28.0828.0832.1931.51(-2.11%) 32.88(+2.14%) Reasoning about Colored Objects28.0031.6046.4047.20(+1.72%) 48.40(+4.31%) Ruin Names31.2023.2034.0030.80(-9.41%) 32.40(-4.71%) Salient Translation Error Detection14.800.0024.8027.60(+11.29%) 29.20(+17.74%) Snarks44.9445.5147.7549.44(+3.54%) 49.44(+3.54%) Sports Understanding64.4082.4090.0090.00(+0.00%) 91.20(+1.33%) Temporal Sequences32.0021.2012.8016.40(+28.13%) 16.40(+28.13%) Tracking Shuffled Objects (3 objects) 36.4030.4033.2033.20(+3.61%) 34.40(+3.61%) Tracking Shuffled Objects (5 objects) 19.2014.4015.6015.20(-2.56%) 15.60(+0.00%) Tracking Shuffled Objects (7 objects) 10.802.0011.209.60(-14.29%) 10.40(-7.14%) Web of Lies51.6063.6050.8061.60(+21.26%) 65.60(+29.13%) Word Sorting5.606.8012.807.60(-40.63%) 7.60(-40.63%) Avg. 27 Tasks33.8733.3839.7040.44(+1.86%) 41.75(+5.16%) Table 1:Overall results ofFuseLLMand baselines in reasoning evaluations on Big-Bench Hard (BBH) , where percentages indicate the rate of improvement/decrease compared to Llama-2.TaskOpenLLaMAMPTLlama-2Llama-2 CLMFuseLLMARC-easy69.7070.1274.5874.54(-0.05%) 75.04(+0.62%) ARC-challenge41.3842.1546.3346.50(+0.37%) 47.44(+2.40%) BoolQ72.2974.7477.7176.88(-1.07%) 78.13(+0.54%) HellaSwag74.5376.2576.0076.57(+0.75%) 76.78(+1.03%) OpenBookQA41.0042.4044.2044.80(+1.36%) 45.40(+2.71%) Avg. 5 Tasks59.7861.1363.7663.86(+0.16%) 64.56(+1.25%) Table 2:Overall results ofFuseLLMand baselines in commonsense evaluations on CommenSense (CS) , where percentages indicate the rate of improvement/decrease compared to Llama-2.TaskOpenLLaMAMPTLlama-2Llama-2 CLMFuseLLMC++14.4713.117.459.88(+32.62%) 9.25(+24.16%) Go68.2066.9657.0254.44(-4.52%) 59.78(+4.84%) Java14.2813.4210.3110.50(+1.84%) 10.34(+0.29%) JavaScript17.6113.0113.1714.25(+8.20%) 14.32(+8.73%) PHP11.249.539.759.04(-7.28%) 9.41(-3.49%) Python15.9617.2413.8513.07(-5.63%) 13.91(+0.43%) R7.524.534.975.25(+5.63%) 5.84(+17.51%) Ruby10.3412.3310.3710.68(+2.99%) 11.24(+8.39%) Rust6.188.296.776.96(+2.81%) 7.05(+4.14%) TypeScript15.3114.1312.6114.19(+12.53%) 14.50(+14.99%) Avg. 10 Tasks18.1117.2614.6314.83(+1.37%) 15.56(+6.36%) Table 3:Overall results ofFuseLLMand baselines in code generation evaluations on MultiPL-E (ME) , where percentages indicate the rate of improvement/decrease compared to Llama-2.For the code generation evaluation, the zero-shot performance ofFuseLLMon the MultiPL-E (ME)  benchmark is reported in Table3. We observe thatFuseLLMoutperforms Llama-2 in 9 out of the 10 tasks, with a notable enhancement in the pass@1 score for specific programming languages such as R, increasing from 4.97 to 5.84. Given that both OpenLLaMA and MPT demonstrate remarkable performances in code generation tasks compared to Llama-2, the fusion result viaFuseLLMachieves an average performance gain of 6.36%, which is considerably higher than the 1.37% improvement observed in Llama-2 CLM. However, it’s important to note thatFuseLLMstill exhibits a performance gap compared to OpenLLaMA or MPT in this evaluation. This discrepancy can be attributed to two primary reasons: the inferior performances of Llama-2 as the target model compared to other source LLMs and an insufficient proportion of coding-related texts in the continual training corpus, estimated at approximately 7.59%111Since MiniPile lacks specific data percentages for individual domains, we approximate this by considering the percentage of the Github domain in The Pile..4.3The Fused Probabilistic DistributionsFigure 2:Effect of the fused distributions in accelerating the optimization process on BBH, where the x-axis denotes the number of training tokens and the y-axis denotes the exact match accuracy.We investigate the effectiveness of the fused probabilistic distributions obtained from multiple LLMs and track the trend of performance improvement during the training process. Figure2illustrates the comparison of few-shot CoT performance between Llama-2 CLM andFuseLLMwith varying scales of training data on BBH. Our observations reveal thatFuseLLMenhances the exact match (EM)  accuracy by 2.5% compared to Llama-2 CLM and achieves the best performance of Llama-2 CLM within 0.52 billion tokens. Notably, this represents areduction in token requirements compared to the 1.57 billion tokens needed by Llama-2 CLM. These results suggest that the probabilistic distributions derived from LLMs contain knowledge that is more readily learnable than the original text sequences, which accelerates the optimization process. This finding aligns with the observations in. We further conduct an experiment to show that our performance improvement stems from the integration of knowledge from multiple LLMs rather than solely from continual training. The results and analysis are shown in AppendixG.4.4Analysis of Implementation ProcessIn this section, we delve into the crucial elements ofFuseLLM’s implementation, including the number of source LLMs, the criteria for token alignment, and the choice of the fusion function.ModelBBHCSMEOpenLLaMA33.8759.7818.11MPT33.3861.1317.26Llama-239.7063.7614.63Llama-2 CLM40.44(+1.86%) 63.86(+0.16%) 14.83(+1.37%) Llama-2 + OpenLLaMA41.00(+3.27%) 64.50(+1.16%) 15.51(+6.02%) Llama-2 + MPT41.16(+3.68%) 64.51(+1.18%) 15.47(+5.74%) FuseLLM41.75(+5.16%) 64.56(+1.25%) 15.56(+6.36%) Table 4:Results ofFuseLLMby incorporating varying numbers of models.Number of source LLMs.In Table4, we present the results of fusing different numbers of LLMs. We note that the performance ofFuseLLMdemonstrates apparent improvement as the number of models increases from 1 to 3. Nevertheless, the benefits of integrating additional models exhibit variations across benchmarks. Remarkably, a consistent performance improvement is observed in BBH. Whereas in CS or ME, the advantages are more prominent when fusing two models. This phenomenon may be attributed to the considerable performance differences among the three models on various tasks in BBH, while the performance differences in tasks of CS or ME are relatively smaller.Criteria for token alignment.During the fusion of LLMs, ensuring the proper alignment of tokens and vocabularies from multiple models is of paramount importance. In Table5(upper) , we present a comparison of two alignment criteria. It is evident that the proposed MinED method, which is based on minimum edit distance, consistently outperforms the EM method introduced by, which relies on exact matching. We suggest that this performance enhancement results from MinED’s ability to relax the constraints of EM, as tokens separated by distinct tokenizers within the same sequence often exhibit minor discrepancies. Consequently, MinED effectively supplements a considerable amount of useful token information while introducing negligible errors.ChoiceBBHMECSAlignment CriteriaEM41.5715.4964.24MinED41.75(+0.43%) 15.56(+0.45%) 64.56(+0.50%) Fusion FunctionAvgCE41.0415.3963.98MinCE41.75(+1.73%) 15.56(+1.10%) 64.56(+0.91%) Table 5:Comparison of different token alignment criteria (upper)  and fusion functions (down) .Fusion function.In Section3.3, we introduce two variations of the fusion function forFuseLLM: one utilizing a distribution matrix with minimum cross entropy (MinCE)  and the other adopting a weighted average of distribution matrices based on cross entropy (AvgCE) . A comparison of the two functions is presented in Table5(down) . The findings demonstrate thatFuseLLMwith MinCE consistently outperforms AvgCE across all benchmarks. This can be attributed to the distortions introduced by the straightforward weighted summation used in AvgCE, which may diminish the distinct advantages of individual LLMs.4.5FuseLLMvs. Knowledge DistillationModelBBHCSMELlama-2 13B47.9266.3318.76OpenLLaMA33.8759.7818.11MPT33.3861.1317.26Llama-239.7063.7614.63Llama-2 CLM40.44(+1.86%) 63.86(+0.16%) 14.83(+1.37%) Llama-2 KD40.88(+2.97%) 64.41(+1.02%) 15.45(+5.60%) FuseLLM41.75(+5.16%) 64.56(+1.25%) 15.56(+6.36%) Table 6:Comparison ofFuseLLMand knowledge distillation. Llama-2 KD denotes the enhanced Llama-2 7B achieved via knowledge distillation from Llama-2 13B. Percentages indicate the rate of improvement compared to Llama-2.While knowledge distillation techniques can also be utilized to enhance a LLM’s capabilities,FuseLLMstands out due to two distinct aspects, as previously outlined. In this section, we compareFuseLLMwith traditional knowledge distillation. Specifically, we extract probabilistic distributions from Llama-2 13B and apply the conventional knowledge distillation method to transfer its abilities into Llama-2 7B. As illustrated in Table6, the distilled model (Llama-2 KD)  outperforms the original Llama-2 7B across all benchmarks, demonstrating the effectiveness of knowledge distillation. However, when compared toFuseLLM, the improvement achieved by Llama-2 KD is relatively modest, especially in the case of BBH (2.97% vs. 5.16%) . This suggests that the superior results achieved byFuseLLMthrough the integration of three 7B models with diverse architectures via continual training outweigh the benefits of simply distilling knowledge from a single 13B model. This observation highlights the idea that “More is different, but different can also be more”.4.6FuseLLMvs. Ensemble/MergingModelPhilNIHUSPTOAveragePythia0.90080.67400.60770.7275Phil0.83970.68610.62280.7162NIH0.92480.62150.62780.7247USPTO0.92960.68720.60170.7395Ensemble0.89600.66470.61800.7262Weight Merging0.87860.64960.60540.7112FuseLLM0.84630.65690.60680.7034Table 7:Comparison of perplexity betweenFuseLLMand ensemble&weight merging.As previously mentioned, conventional techniques such as model ensemble and weight merging are commonly employed to fuse multiple LLMs. To compare the efficacy of ourFuseLLMwith these existing fusion methods, we conduct experiments simulating scenarios where multiple LLMs originated from the same base model but were trained on distinct corpora. We first select three relevant domains (PhilPapers, NIH ExPorter, and USPTO Backgrounds)  from The Pile and use 1 billion tokens from each domain to continually train Pythia 1B, resulting in three distinct LLMs with identical structures.
Then, we apply different fusion techniques to these LLMs: (1)  The ensemble method calculates a weighted average of the probabilities generated by all LLMs, considering the performance of each model; (2)  The weight merging method merges multiple LLMs into a single one within the parameter space, with the merging weights determined by model performance; (3) FuseLLMundergoes continual training on 0.1 billion tokens sampled from the three domains. The results of perplexity forFuseLLMand the other fusion methods on the test sets are presented in Table7. We measure perplexity in bits per UTF-8 encoded byte (BPB)  following the implementation in The Pile. We observe that after training with 1 billion tokens, the capabilities of the original LLM are transferred to each domain-specific LLM, resulting in decreased performance in other domains. While all fusion techniques can integrate the strengths of diverse models,FuseLLMconsistently achieves the lowest average perplexity across the three domains. This underscores its potential for harnessing collective knowledge more effectively than ensemble and weight merging methods.
body
Knowledge Fusion of Large Language Models
4Experiments
2401.10491
5ConclusionIn this study, we have explored the realm of knowledge fusion for LLMs to create a unified model that combines the capabilities and distinctive strengths of multiple structurally diverse LLMs. We introduced a novel method,FuseLLM, which leverages the generative distributions of these source LLMs to externalize their knowledge and employs them in the continual training of the target LLM. Through a series of experiments, we have demonstrated the superiority ofFuseLLMover individual source LLMs and established baselines. Notably, in a simulated experiment featuring multiple structurally identical LLMs,FuseLLMhas showcased its competitive effectiveness compared to ensemble and weight merging methods. Hence, the domain of LLMs fusion emerges as a more promising avenue for exploration, particularly given the diverse structures and substantial model sizes of LLMs. We believe that these findings will inspire future research endeavors.
body
Knowledge Fusion of Large Language Models
5Conclusion
2401.10491
AcknowledgementsThis work was supported by the National Natural Science Foundation of China (No. 62176270) , the Guangdong Basic and Applied Basic Research Foundation (No. 2023A1515012832) , and the Tencent AI Lab Rhino-Bird Focused Research Program.
conclusion
Knowledge Fusion of Large Language Models
Acknowledgements