<!DOCTYPE html><html lang="en">
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<title>[2407.04528] GPT vs RETRO: Exploring the Intersection of Retrieval and Parameter-Efficient Fine-Tuning</title><meta property="og:description" content="Parameter-Efficient Fine-Tuning (PEFT) and Retrieval-Augmented Generation (RAG) have become popular methods for adapting large language models while minimizing compute requirements. In this paper, we apply PEFT methods…">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="twitter:card" content="summary">
<meta name="twitter:title" content="GPT vs RETRO: Exploring the Intersection of Retrieval and Parameter-Efficient Fine-Tuning">
<meta name="twitter:image:src" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta name="twitter:image:alt" content="ar5iv logo">
<meta property="og:title" content="GPT vs RETRO: Exploring the Intersection of Retrieval and Parameter-Efficient Fine-Tuning">
<meta property="og:site_name" content="ar5iv">
<meta property="og:image" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta property="og:type" content="article">
<meta property="og:url" content="https://ar5iv.labs.arxiv.org/html/2407.04528">

<!--Generated on Mon Aug  5 16:30:32 2024 by LaTeXML (version 0.8.8) http://dlmf.nist.gov/LaTeXML/.-->
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">

<script>
  function detectColorScheme(){
    var theme="light";
    var current_theme = localStorage.getItem("ar5iv_theme");
    if(current_theme){
      if(current_theme == "dark"){
        theme = "dark";
      } }
    else if(!window.matchMedia) { return false; }
    else if(window.matchMedia("(prefers-color-scheme: dark)").matches) {
      theme = "dark"; }
    if (theme=="dark") {
      document.documentElement.setAttribute("data-theme", "dark");
    } else {
      document.documentElement.setAttribute("data-theme", "light"); } }

  detectColorScheme();

  function toggleColorScheme(){
    var current_theme = localStorage.getItem("ar5iv_theme");
    if (current_theme) {
      if (current_theme == "light") {
        localStorage.setItem("ar5iv_theme", "dark"); }
      else {
        localStorage.setItem("ar5iv_theme", "light"); } }
    else {
        localStorage.setItem("ar5iv_theme", "dark"); }
    detectColorScheme(); }
</script>
<link media="all" rel="stylesheet" href="/assets/ar5iv-fonts.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv-site.0.2.2.css">
</head>
<body>
<div class="ltx_page_main">
<div class="ltx_page_content">
<article class="ltx_document">
<h1 class="ltx_title ltx_title_document">GPT vs RETRO: Exploring the Intersection of Retrieval and Parameter-Efficient Fine-Tuning</h1>
<div class="ltx_authors">
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Aleksander Ficek, Jiaqi Zeng<span id="footnotex1" class="ltx_note ltx_role_footnotemark"><sup class="ltx_note_mark">1</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">1</sup><span class="ltx_note_type">footnotemark: </span><span class="ltx_tag ltx_tag_note">1</span></span></span></span>, Oleksii Kuchaiev 
<br class="ltx_break">NVIDIA
<br class="ltx_break"><span id="id1.1.id1" class="ltx_text ltx_font_typewriter">{aficek,jiaqiz,okuchaiev}@nvidia.com</span> 
<br class="ltx_break">
</span><span class="ltx_author_notes">Equal contribution.</span></span>
</div>

<div class="ltx_abstract">
<h6 class="ltx_title ltx_title_abstract">Abstract</h6>
<p id="id2.id1" class="ltx_p">Parameter-Efficient Fine-Tuning (PEFT) and Retrieval-Augmented Generation (RAG) have become popular methods for adapting large language models while minimizing compute requirements. In this paper, we apply PEFT methods (P-tuning, Adapters, and LoRA) to a modified Retrieval-Enhanced Transformer (RETRO) and a baseline GPT model across several sizes, ranging from 823 million to 48 billion parameters. We show that RETRO models outperform GPT models in zero-shot settings due to their unique pre-training process but GPT models have higher performance potential with PEFT. Additionally, our study indicates that 8B parameter models strike an optimal balance between cost and performance and P-tuning lags behind other PEFT techniques. We further provide a comparative analysis of between applying PEFT to an Instruction-tuned RETRO model and base RETRO model. This work presents the first comprehensive comparison of various PEFT methods integrated with RAG, applied to both GPT and RETRO models, highlighting their relative performance.</p>
</div>
<div id="p1" class="ltx_para ltx_noindent">
<div id="p1.1" class="ltx_block ltx_align_bottom">
<p id="p1.1.1" class="ltx_p"><span id="p1.1.1.1" class="ltx_text ltx_font_bold">GPT vs RETRO: Exploring the Intersection of Retrieval and Parameter-Efficient Fine-Tuning</span></p>
<br class="ltx_break ltx_centering">
<p id="p1.1.2" class="ltx_p ltx_align_center" style="width:433.6pt;"><span id="p1.1.2.1" class="ltx_text ltx_inline-block" style="width:0.0pt;">
<span id="p1.1.2.1.1" class="ltx_tabular ltx_align_top">
<span class="ltx_tbody">
<span id="p1.1.2.1.1.1.1" class="ltx_tr">
<span id="p1.1.2.1.1.1.1.1" class="ltx_td ltx_align_center"><span id="p1.1.2.1.1.1.1.1.1" class="ltx_text ltx_font_bold">Aleksander Ficek<span id="p1.1.2.1.1.1.1.1.1.1" class="ltx_note ltx_role_thanks"><sup class="ltx_note_mark">†</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">†</sup><span class="ltx_note_type">thanks: </span>Equal contribution.</span></span></span>, Jiaqi Zeng<span id="footnotex2" class="ltx_note ltx_role_footnotemark"><sup class="ltx_note_mark">1</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">1</sup><span class="ltx_note_type">footnotemark: </span><span class="ltx_tag ltx_tag_note"><span id="footnotex2.1.1.1" class="ltx_text ltx_font_medium">1</span></span></span></span></span>, Oleksii Kuchaiev</span></span></span>
<span id="p1.1.2.1.1.2.2" class="ltx_tr">
<span id="p1.1.2.1.1.2.2.1" class="ltx_td ltx_align_center">NVIDIA</span></span>
<span id="p1.1.2.1.1.3.3" class="ltx_tr">
<span id="p1.1.2.1.1.3.3.1" class="ltx_td ltx_align_center"><span id="p1.1.2.1.1.3.3.1.1" class="ltx_text ltx_font_typewriter">{aficek,jiaqiz,okuchaiev}@nvidia.com</span></span></span>
</span>
</span></span></p>
<br class="ltx_break ltx_centering">
</div>
</div>
<section id="S1" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">1 </span>Introduction</h2>

<div id="S1.p1" class="ltx_para">
<p id="S1.p1.1" class="ltx_p">Pre-trained large language models have made a demonstrable impact across applications in academia and industry. Many use cases, however, require LLMs adapted to specific tasks and unique information but lack the resources for extensive retraining. To address this, Parameter-Efficient Fine-Tuning (PEFT) <cite class="ltx_cite ltx_citemacro_citep">(Han et al., <a href="#bib.bib7" title="" class="ltx_ref">2024</a>)</cite> and Retrieval-Augmented Generation (RAG) <cite class="ltx_cite ltx_citemacro_citep">(Gao et al., <a href="#bib.bib4" title="" class="ltx_ref">2023</a>)</cite> have become popular methods due to their effectiveness and efficiency, inspiring new lines of research.</p>
</div>
<figure id="S1.F1" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_2"><img src="/html/2407.04528/assets/x1.png" id="S1.F1.g1" class="ltx_graphics ltx_figure_panel ltx_img_landscape" width="178" height="103" alt="Refer to caption"></div>
<div class="ltx_flex_cell ltx_flex_size_2"><img src="/html/2407.04528/assets/x2.png" id="S1.F1.g2" class="ltx_graphics ltx_figure_panel ltx_img_landscape" width="178" height="103" alt="Refer to caption"></div>
</div>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">Figure 1: </span>Average GPT vs RETRO scores of six datasets across model sizes of 823M to 48B parameters.</figcaption>
</figure>
<div id="S1.p2" class="ltx_para">
<p id="S1.p2.1" class="ltx_p">PEFT has been proven to be a comparable substitute to Supervised Fine-Tuning (SFT) by achieving competitive performance at a fraction of the number of updated parameters <cite class="ltx_cite ltx_citemacro_citep">(Han et al., <a href="#bib.bib7" title="" class="ltx_ref">2024</a>)</cite>. In this paper we select P-tuning <cite class="ltx_cite ltx_citemacro_citep">(Liu et al., <a href="#bib.bib19" title="" class="ltx_ref">2023</a>)</cite>, Adapter modules <cite class="ltx_cite ltx_citemacro_citep">(Houlsby et al., <a href="#bib.bib8" title="" class="ltx_ref">2019</a>)</cite> and Low-Rank Adaptation (LoRA) <cite class="ltx_cite ltx_citemacro_citep">(Hu et al., <a href="#bib.bib9" title="" class="ltx_ref">2021</a>)</cite> as representative PEFT methods. P-tuning involves training continuous prompt embeddings to guide output for specific tasks without modifying base model parameters. Adapters operate by training fully connected layers inserted throughout the base model while keeping the remaining parameters frozen. LoRA employs a similar strategy but further decomposes the inserted layers into low-rank matrices, enhancing efficiency.</p>
</div>
<div id="S1.p3" class="ltx_para">
<p id="S1.p3.1" class="ltx_p">Retrieval-augmented generation (RAG) improves model quality by incorporating external knowledge through mechanisms like BM-25 or TF-IDF <cite class="ltx_cite ltx_citemacro_citep">(Robertson et al., <a href="#bib.bib25" title="" class="ltx_ref">2009</a>)</cite>, online web search <cite class="ltx_cite ltx_citemacro_citep">(Page et al., <a href="#bib.bib22" title="" class="ltx_ref">1999</a>)</cite>, or trained dense retriever models <cite class="ltx_cite ltx_citemacro_citep">(Karpukhin et al., <a href="#bib.bib12" title="" class="ltx_ref">2020</a>)</cite>. Any LLM can be transformed into a retrieval-augmented model by concatenating retrieved sources with the input query, provided it fits within the model’s context window. <cite class="ltx_cite ltx_citemacro_citet">Xu et al. (<a href="#bib.bib29" title="" class="ltx_ref">2023</a>)</cite> found that retrieval significantly improves GPT model quality on long context tasks, reducing the "lost in the middle" effect <cite class="ltx_cite ltx_citemacro_citep">(Liu et al., <a href="#bib.bib18" title="" class="ltx_ref">2024</a>)</cite> and offering inherent efficiency benefits.</p>
</div>
<div id="S1.p4" class="ltx_para">
<p id="S1.p4.1" class="ltx_p">Alternatively, there exist multiple works <cite class="ltx_cite ltx_citemacro_citep">(Borgeaud et al., <a href="#bib.bib1" title="" class="ltx_ref">2022</a>; Guu et al., <a href="#bib.bib6" title="" class="ltx_ref">2020</a>; Izacard et al., <a href="#bib.bib10" title="" class="ltx_ref">2023</a>; Nakano et al., <a href="#bib.bib20" title="" class="ltx_ref">2021</a>)</cite> that have integrated retrieval as part of model pretraining or fine-tuning to notable success when compared to typical GPT models despite being a much lesser explored domain. RETRO <cite class="ltx_cite ltx_citemacro_citep">(Borgeaud et al., <a href="#bib.bib1" title="" class="ltx_ref">2022</a>)</cite> is of particular interest due to its unique approach of incorporating a retrieval module directly into the transformer architecture via a chunked-cross attention mechanism and ability to scale to trillions of tokens resulting in reduced perplexity. Subsequently, <cite class="ltx_cite ltx_citemacro_citet">Wang et al. (<a href="#bib.bib28" title="" class="ltx_ref">2023b</a>)</cite> showed that RETRO at sizes up to 9.5 billion parameters largely outperforms GPT on specific knowledge-intensive tasks. Furthermore, <cite class="ltx_cite ltx_citemacro_citet">Wang et al. (<a href="#bib.bib27" title="" class="ltx_ref">2023a</a>)</cite> illustrated that when scaled up to 48 billion parameters and instruction-tuned, RETRO performed better than equivalent GPT models on several question answering, reading comprehension and summarization tasks.</p>
</div>
<div id="S1.p5" class="ltx_para">
<p id="S1.p5.1" class="ltx_p">In this paper we continue the exploration of RETRO versus GPT through the lens of parameter efficient finetuning. We apply P-tuning, Adapter modules and LoRA to multiple tasks with retrieval for both RETRO and GPT models. To our knowledge, this paper provides the first in-depth comparison of various Parameter Efficient Fine-Tuning integrated with Retrieval-Augmented Generation, uniquely applied to both GPT and RETRO models.</p>
</div>
</section>
<section id="S2" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">2 </span>Related Work</h2>

<div id="S2.p1" class="ltx_para">
<p id="S2.p1.1" class="ltx_p">In this section we focus on recent work that combine finetuning with retrieval. A comprehensive survey <cite class="ltx_cite ltx_citemacro_cite">Gao et al. (<a href="#bib.bib4" title="" class="ltx_ref">2023</a>)</cite> synthetized multiple comparative studies on PEFT and RAG, underscoring the potential benefits of combining these approaches as a promising direction for future investigation. There are multiple works that provide methods to combine RAG with fine-tuning to improve accuracy <cite class="ltx_cite ltx_citemacro_citep">(Zhang et al., <a href="#bib.bib30" title="" class="ltx_ref">2024a</a>, <a href="#bib.bib31" title="" class="ltx_ref">b</a>; Rangan and Yin, <a href="#bib.bib24" title="" class="ltx_ref">2024</a>)</cite>. Multiple studies have explored the comparison between fine-tuning and retrieval. <cite class="ltx_cite ltx_citemacro_citet">Lakatos et al. (<a href="#bib.bib15" title="" class="ltx_ref">2024</a>)</cite> and <cite class="ltx_cite ltx_citemacro_citet">Ovadia et al. (<a href="#bib.bib21" title="" class="ltx_ref">2023</a>)</cite> reported improved accuracy using RAG over fine-tuning GPT models, while also noting suboptimal results when combining the two methods. <cite class="ltx_cite ltx_citemacro_citet">Gupta et al. (<a href="#bib.bib5" title="" class="ltx_ref">2024</a>)</cite> demonstrated improved outcomes by integrating both approaches for specific agriculture and geography tasks. Additionally, <cite class="ltx_cite ltx_citemacro_citet">Soudani et al. (<a href="#bib.bib26" title="" class="ltx_ref">2024</a>)</cite> compared the efficacy of these methods, including full and QLoRA fine-tuning <cite class="ltx_cite ltx_citemacro_citep">(Dettmers et al., <a href="#bib.bib3" title="" class="ltx_ref">2024</a>)</cite>, in low-frequency entity question-answering tasks.
These studies collectively suggest the need for comprehensive investigation into multiple PEFT techniques combined with RAG and maintain retrieval pretrained LLMs with PEFT to be unexplored, thereby motivating our research.</p>
</div>
<figure id="S2.T1" class="ltx_table">
<div id="S2.T1.1" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:433.6pt;height:252.6pt;vertical-align:-0.6pt;"><span class="ltx_transformed_inner" style="transform:translate(-139.4pt,81.0pt) scale(0.60873096486715,0.60873096486715) ;">
<table id="S2.T1.1.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="S2.T1.1.1.1.1" class="ltx_tr">
<th id="S2.T1.1.1.1.1.1" class="ltx_td ltx_th ltx_th_row ltx_border_tt" style="padding-top:1pt;padding-bottom:1pt;"></th>
<th id="S2.T1.1.1.1.1.2" class="ltx_td ltx_th ltx_th_row ltx_border_tt" style="padding-top:1pt;padding-bottom:1pt;"></th>
<td id="S2.T1.1.1.1.1.3" class="ltx_td ltx_align_center ltx_border_tt" style="padding-top:1pt;padding-bottom:1pt;" colspan="2"><span id="S2.T1.1.1.1.1.3.1" class="ltx_text ltx_font_bold">NQ</span></td>
<td id="S2.T1.1.1.1.1.4" class="ltx_td ltx_align_center ltx_border_tt" style="padding-top:1pt;padding-bottom:1pt;" colspan="2"><span id="S2.T1.1.1.1.1.4.1" class="ltx_text ltx_font_bold">TQA</span></td>
<td id="S2.T1.1.1.1.1.5" class="ltx_td ltx_align_center ltx_border_tt" style="padding-top:1pt;padding-bottom:1pt;" colspan="2"><span id="S2.T1.1.1.1.1.5.1" class="ltx_text ltx_font_bold">NQA</span></td>
<td id="S2.T1.1.1.1.1.6" class="ltx_td ltx_align_center ltx_border_tt" style="padding-top:1pt;padding-bottom:1pt;" colspan="2"><span id="S2.T1.1.1.1.1.6.1" class="ltx_text ltx_font_bold">QASPER</span></td>
<td id="S2.T1.1.1.1.1.7" class="ltx_td ltx_align_center ltx_border_tt" style="padding-top:1pt;padding-bottom:1pt;" colspan="2"><span id="S2.T1.1.1.1.1.7.1" class="ltx_text ltx_font_bold">QUALITY</span></td>
<td id="S2.T1.1.1.1.1.8" class="ltx_td ltx_align_center ltx_border_tt" style="padding-top:1pt;padding-bottom:1pt;" colspan="2"><span id="S2.T1.1.1.1.1.8.1" class="ltx_text ltx_font_bold">QMSUM</span></td>
<td id="S2.T1.1.1.1.1.9" class="ltx_td ltx_align_center ltx_border_tt" style="padding-top:1pt;padding-bottom:1pt;" colspan="2"><span id="S2.T1.1.1.1.1.9.1" class="ltx_text ltx_font_bold">AVERAGE</span></td>
</tr>
<tr id="S2.T1.1.1.2.2" class="ltx_tr">
<th id="S2.T1.1.1.2.2.1" class="ltx_td ltx_th ltx_th_row" style="padding-top:1pt;padding-bottom:1pt;"></th>
<th id="S2.T1.1.1.2.2.2" class="ltx_td ltx_th ltx_th_row" style="padding-top:1pt;padding-bottom:1pt;"></th>
<td id="S2.T1.1.1.2.2.3" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">GPT</td>
<td id="S2.T1.1.1.2.2.4" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">RETRO</td>
<td id="S2.T1.1.1.2.2.5" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">GPT</td>
<td id="S2.T1.1.1.2.2.6" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">RETRO</td>
<td id="S2.T1.1.1.2.2.7" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">GPT</td>
<td id="S2.T1.1.1.2.2.8" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">RETRO</td>
<td id="S2.T1.1.1.2.2.9" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">GPT</td>
<td id="S2.T1.1.1.2.2.10" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">RETRO</td>
<td id="S2.T1.1.1.2.2.11" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">GPT</td>
<td id="S2.T1.1.1.2.2.12" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">RETRO</td>
<td id="S2.T1.1.1.2.2.13" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">GPT</td>
<td id="S2.T1.1.1.2.2.14" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">RETRO</td>
<td id="S2.T1.1.1.2.2.15" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">GPT</td>
<td id="S2.T1.1.1.2.2.16" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">RETRO</td>
</tr>
<tr id="S2.T1.1.1.3.3" class="ltx_tr">
<th id="S2.T1.1.1.3.3.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;" rowspan="4"><span id="S2.T1.1.1.3.3.1.1" class="ltx_text ltx_font_bold">Extra Small</span></th>
<th id="S2.T1.1.1.3.3.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;">Zero-shot</th>
<td id="S2.T1.1.1.3.3.3" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;">2.95</td>
<td id="S2.T1.1.1.3.3.4" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.3.3.4.1" class="ltx_text ltx_font_bold">8.28</span></td>
<td id="S2.T1.1.1.3.3.5" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;">9.99</td>
<td id="S2.T1.1.1.3.3.6" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.3.3.6.1" class="ltx_text ltx_font_bold">19.26</span></td>
<td id="S2.T1.1.1.3.3.7" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.3.3.7.1" class="ltx_text ltx_font_bold">7.07</span></td>
<td id="S2.T1.1.1.3.3.8" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;">4.87</td>
<td id="S2.T1.1.1.3.3.9" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;">9.00</td>
<td id="S2.T1.1.1.3.3.10" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.3.3.10.1" class="ltx_text ltx_font_bold">10.79</span></td>
<td id="S2.T1.1.1.3.3.11" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;">0.38</td>
<td id="S2.T1.1.1.3.3.12" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;">0.48</td>
<td id="S2.T1.1.1.3.3.13" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.3.3.13.1" class="ltx_text ltx_font_bold">9.83</span></td>
<td id="S2.T1.1.1.3.3.14" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;">7.78</td>
<td id="S2.T1.1.1.3.3.15" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;">6.54</td>
<td id="S2.T1.1.1.3.3.16" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.3.3.16.1" class="ltx_text ltx_font_bold">8.58</span></td>
</tr>
<tr id="S2.T1.1.1.4.4" class="ltx_tr">
<th id="S2.T1.1.1.4.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row" style="padding-top:1pt;padding-bottom:1pt;">P-tuning</th>
<td id="S2.T1.1.1.4.4.2" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.4.4.2.1" class="ltx_text ltx_font_bold">24.74</span></td>
<td id="S2.T1.1.1.4.4.3" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">7.60</td>
<td id="S2.T1.1.1.4.4.4" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.4.4.4.1" class="ltx_text ltx_font_bold">63.63</span></td>
<td id="S2.T1.1.1.4.4.5" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">24.61</td>
<td id="S2.T1.1.1.4.4.6" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.4.4.6.1" class="ltx_text ltx_font_bold">16.74</span></td>
<td id="S2.T1.1.1.4.4.7" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">6.69</td>
<td id="S2.T1.1.1.4.4.8" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.4.4.8.1" class="ltx_text ltx_font_bold">24.54</span></td>
<td id="S2.T1.1.1.4.4.9" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">11.94</td>
<td id="S2.T1.1.1.4.4.10" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.4.4.10.1" class="ltx_text ltx_font_bold">24.59</span></td>
<td id="S2.T1.1.1.4.4.11" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">17.45</td>
<td id="S2.T1.1.1.4.4.12" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.4.4.12.1" class="ltx_text ltx_font_bold">18.25</span></td>
<td id="S2.T1.1.1.4.4.13" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">13.63</td>
<td id="S2.T1.1.1.4.4.14" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.4.4.14.1" class="ltx_text ltx_font_bold">28.75</span></td>
<td id="S2.T1.1.1.4.4.15" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">13.65</td>
</tr>
<tr id="S2.T1.1.1.5.5" class="ltx_tr">
<th id="S2.T1.1.1.5.5.1" class="ltx_td ltx_align_left ltx_th ltx_th_row" style="padding-top:1pt;padding-bottom:1pt;">Adapter</th>
<td id="S2.T1.1.1.5.5.2" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.5.5.2.1" class="ltx_text ltx_font_bold">38.48</span></td>
<td id="S2.T1.1.1.5.5.3" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">23.69</td>
<td id="S2.T1.1.1.5.5.4" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.5.5.4.1" class="ltx_text ltx_font_bold">67.99</span></td>
<td id="S2.T1.1.1.5.5.5" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">59.60</td>
<td id="S2.T1.1.1.5.5.6" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.5.5.6.1" class="ltx_text ltx_font_bold">17.76</span></td>
<td id="S2.T1.1.1.5.5.7" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">15.42</td>
<td id="S2.T1.1.1.5.5.8" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.5.5.8.1" class="ltx_text ltx_font_bold">23.52</span></td>
<td id="S2.T1.1.1.5.5.9" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">20.96</td>
<td id="S2.T1.1.1.5.5.10" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">24.26</td>
<td id="S2.T1.1.1.5.5.11" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.5.5.11.1" class="ltx_text ltx_font_bold">25.93</span></td>
<td id="S2.T1.1.1.5.5.12" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.5.5.12.1" class="ltx_text ltx_font_bold">19.74</span></td>
<td id="S2.T1.1.1.5.5.13" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">14.42</td>
<td id="S2.T1.1.1.5.5.14" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.5.5.14.1" class="ltx_text ltx_font_bold">31.96</span></td>
<td id="S2.T1.1.1.5.5.15" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">26.67</td>
</tr>
<tr id="S2.T1.1.1.6.6" class="ltx_tr">
<th id="S2.T1.1.1.6.6.1" class="ltx_td ltx_align_left ltx_th ltx_th_row" style="padding-top:1pt;padding-bottom:1pt;">LoRA</th>
<td id="S2.T1.1.1.6.6.2" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.6.6.2.1" class="ltx_text ltx_font_bold">37.09</span></td>
<td id="S2.T1.1.1.6.6.3" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">22.13</td>
<td id="S2.T1.1.1.6.6.4" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.6.6.4.1" class="ltx_text ltx_font_bold">67.31</span></td>
<td id="S2.T1.1.1.6.6.5" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">59.02</td>
<td id="S2.T1.1.1.6.6.6" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.6.6.6.1" class="ltx_text ltx_font_bold">18.08</span></td>
<td id="S2.T1.1.1.6.6.7" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">15.81</td>
<td id="S2.T1.1.1.6.6.8" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.6.6.8.1" class="ltx_text ltx_font_bold">23.54</span></td>
<td id="S2.T1.1.1.6.6.9" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">19.85</td>
<td id="S2.T1.1.1.6.6.10" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">24.93</td>
<td id="S2.T1.1.1.6.6.11" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.6.6.11.1" class="ltx_text ltx_font_bold">25.65</span></td>
<td id="S2.T1.1.1.6.6.12" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.6.6.12.1" class="ltx_text ltx_font_bold">19.27</span></td>
<td id="S2.T1.1.1.6.6.13" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">13.79</td>
<td id="S2.T1.1.1.6.6.14" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.6.6.14.1" class="ltx_text ltx_font_bold">31.70</span></td>
<td id="S2.T1.1.1.6.6.15" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">26.04</td>
</tr>
<tr id="S2.T1.1.1.7.7" class="ltx_tr">
<th id="S2.T1.1.1.7.7.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;" rowspan="4"><span id="S2.T1.1.1.7.7.1.1" class="ltx_text ltx_font_bold">Small</span></th>
<th id="S2.T1.1.1.7.7.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;">Zero-shot</th>
<td id="S2.T1.1.1.7.7.3" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;">11.65</td>
<td id="S2.T1.1.1.7.7.4" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.7.7.4.1" class="ltx_text ltx_font_bold">18.77</span></td>
<td id="S2.T1.1.1.7.7.5" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;">29.88</td>
<td id="S2.T1.1.1.7.7.6" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.7.7.6.1" class="ltx_text ltx_font_bold">38.42</span></td>
<td id="S2.T1.1.1.7.7.7" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;">7.07</td>
<td id="S2.T1.1.1.7.7.8" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.7.7.8.1" class="ltx_text ltx_font_bold">7.12</span></td>
<td id="S2.T1.1.1.7.7.9" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;">12.31</td>
<td id="S2.T1.1.1.7.7.10" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.7.7.10.1" class="ltx_text ltx_font_bold">12.42</span></td>
<td id="S2.T1.1.1.7.7.11" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;">0.00</td>
<td id="S2.T1.1.1.7.7.12" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.7.7.12.1" class="ltx_text ltx_font_bold">1.01</span></td>
<td id="S2.T1.1.1.7.7.13" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.7.7.13.1" class="ltx_text ltx_font_bold">12.35</span></td>
<td id="S2.T1.1.1.7.7.14" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;">9.25</td>
<td id="S2.T1.1.1.7.7.15" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;">12.21</td>
<td id="S2.T1.1.1.7.7.16" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.7.7.16.1" class="ltx_text ltx_font_bold">14.50</span></td>
</tr>
<tr id="S2.T1.1.1.8.8" class="ltx_tr">
<th id="S2.T1.1.1.8.8.1" class="ltx_td ltx_align_left ltx_th ltx_th_row" style="padding-top:1pt;padding-bottom:1pt;">P-tuning</th>
<td id="S2.T1.1.1.8.8.2" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.8.8.2.1" class="ltx_text ltx_font_bold">39.27</span></td>
<td id="S2.T1.1.1.8.8.3" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">18.58</td>
<td id="S2.T1.1.1.8.8.4" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.8.8.4.1" class="ltx_text ltx_font_bold">70.31</span></td>
<td id="S2.T1.1.1.8.8.5" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">61.13</td>
<td id="S2.T1.1.1.8.8.6" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.8.8.6.1" class="ltx_text ltx_font_bold">19.98</span></td>
<td id="S2.T1.1.1.8.8.7" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">15.13</td>
<td id="S2.T1.1.1.8.8.8" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.8.8.8.1" class="ltx_text ltx_font_bold">24.75</span></td>
<td id="S2.T1.1.1.8.8.9" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">20.34</td>
<td id="S2.T1.1.1.8.8.10" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">22.77</td>
<td id="S2.T1.1.1.8.8.11" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.8.8.11.1" class="ltx_text ltx_font_bold">24.11</span></td>
<td id="S2.T1.1.1.8.8.12" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.8.8.12.1" class="ltx_text ltx_font_bold">18.76</span></td>
<td id="S2.T1.1.1.8.8.13" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">14.61</td>
<td id="S2.T1.1.1.8.8.14" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.8.8.14.1" class="ltx_text ltx_font_bold">32.64</span></td>
<td id="S2.T1.1.1.8.8.15" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">25.65</td>
</tr>
<tr id="S2.T1.1.1.9.9" class="ltx_tr">
<th id="S2.T1.1.1.9.9.1" class="ltx_td ltx_align_left ltx_th ltx_th_row" style="padding-top:1pt;padding-bottom:1pt;">Adapter</th>
<td id="S2.T1.1.1.9.9.2" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.9.9.2.1" class="ltx_text ltx_font_bold">42.29</span></td>
<td id="S2.T1.1.1.9.9.3" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">23.68</td>
<td id="S2.T1.1.1.9.9.4" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.9.9.4.1" class="ltx_text ltx_font_bold">73.21</span></td>
<td id="S2.T1.1.1.9.9.5" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">64.91</td>
<td id="S2.T1.1.1.9.9.6" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.9.9.6.1" class="ltx_text ltx_font_bold">21.40</span></td>
<td id="S2.T1.1.1.9.9.7" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">18.10</td>
<td id="S2.T1.1.1.9.9.8" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.9.9.8.1" class="ltx_text ltx_font_bold">27.29</span></td>
<td id="S2.T1.1.1.9.9.9" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">20.55</td>
<td id="S2.T1.1.1.9.9.10" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">24.93</td>
<td id="S2.T1.1.1.9.9.11" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.9.9.11.1" class="ltx_text ltx_font_bold">25.07</span></td>
<td id="S2.T1.1.1.9.9.12" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">20.17</td>
<td id="S2.T1.1.1.9.9.13" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">15.03</td>
<td id="S2.T1.1.1.9.9.14" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.9.9.14.1" class="ltx_text ltx_font_bold">34.88</span></td>
<td id="S2.T1.1.1.9.9.15" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">27.89</td>
</tr>
<tr id="S2.T1.1.1.10.10" class="ltx_tr">
<th id="S2.T1.1.1.10.10.1" class="ltx_td ltx_align_left ltx_th ltx_th_row" style="padding-top:1pt;padding-bottom:1pt;">LoRA</th>
<td id="S2.T1.1.1.10.10.2" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.10.10.2.1" class="ltx_text ltx_font_bold">39.27</span></td>
<td id="S2.T1.1.1.10.10.3" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">28.06</td>
<td id="S2.T1.1.1.10.10.4" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.10.10.4.1" class="ltx_text ltx_font_bold">72.34</span></td>
<td id="S2.T1.1.1.10.10.5" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">64.59</td>
<td id="S2.T1.1.1.10.10.6" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.10.10.6.1" class="ltx_text ltx_font_bold">20.98</span></td>
<td id="S2.T1.1.1.10.10.7" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">17.90</td>
<td id="S2.T1.1.1.10.10.8" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.10.10.8.1" class="ltx_text ltx_font_bold">24.83</span></td>
<td id="S2.T1.1.1.10.10.9" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">21.28</td>
<td id="S2.T1.1.1.10.10.10" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.10.10.10.1" class="ltx_text ltx_font_bold">25.79</span></td>
<td id="S2.T1.1.1.10.10.11" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">24.69</td>
<td id="S2.T1.1.1.10.10.12" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.10.10.12.1" class="ltx_text ltx_font_bold">20.31</span></td>
<td id="S2.T1.1.1.10.10.13" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">14.46</td>
<td id="S2.T1.1.1.10.10.14" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.10.10.14.1" class="ltx_text ltx_font_bold">33.92</span></td>
<td id="S2.T1.1.1.10.10.15" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">28.50</td>
</tr>
<tr id="S2.T1.1.1.11.11" class="ltx_tr">
<th id="S2.T1.1.1.11.11.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;" rowspan="5"><span id="S2.T1.1.1.11.11.1.1" class="ltx_text ltx_font_bold">Medium</span></th>
<th id="S2.T1.1.1.11.11.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;">Zero-shot</th>
<td id="S2.T1.1.1.11.11.3" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;">23.67</td>
<td id="S2.T1.1.1.11.11.4" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.11.11.4.1" class="ltx_text ltx_font_bold">24.11</span></td>
<td id="S2.T1.1.1.11.11.5" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;">51.00</td>
<td id="S2.T1.1.1.11.11.6" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.11.11.6.1" class="ltx_text ltx_font_bold">52.17</span></td>
<td id="S2.T1.1.1.11.11.7" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.11.11.7.1" class="ltx_text ltx_font_bold">8.90</span></td>
<td id="S2.T1.1.1.11.11.8" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;">6.39</td>
<td id="S2.T1.1.1.11.11.9" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;">9.01</td>
<td id="S2.T1.1.1.11.11.10" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.11.11.10.1" class="ltx_text ltx_font_bold">10.04</span></td>
<td id="S2.T1.1.1.11.11.11" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.11.11.11.1" class="ltx_text ltx_font_bold">1.44</span></td>
<td id="S2.T1.1.1.11.11.12" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;">0.14</td>
<td id="S2.T1.1.1.11.11.13" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.11.11.13.1" class="ltx_text ltx_font_bold">11.28</span></td>
<td id="S2.T1.1.1.11.11.14" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;">9.15</td>
<td id="S2.T1.1.1.11.11.15" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.11.11.15.1" class="ltx_text ltx_font_bold">17.55</span></td>
<td id="S2.T1.1.1.11.11.16" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;">17.00</td>
</tr>
<tr id="S2.T1.1.1.12.12" class="ltx_tr">
<th id="S2.T1.1.1.12.12.1" class="ltx_td ltx_align_left ltx_th ltx_th_row" style="padding-top:1pt;padding-bottom:1pt;">P-tuning</th>
<td id="S2.T1.1.1.12.12.2" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.12.12.2.1" class="ltx_text ltx_font_bold">45.52</span></td>
<td id="S2.T1.1.1.12.12.3" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">24.18</td>
<td id="S2.T1.1.1.12.12.4" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.12.12.4.1" class="ltx_text ltx_font_bold">77.00</span></td>
<td id="S2.T1.1.1.12.12.5" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">67.94</td>
<td id="S2.T1.1.1.12.12.6" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.12.12.6.1" class="ltx_text ltx_font_bold">24.50</span></td>
<td id="S2.T1.1.1.12.12.7" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">19.02</td>
<td id="S2.T1.1.1.12.12.8" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.12.12.8.1" class="ltx_text ltx_font_bold">33.31</span></td>
<td id="S2.T1.1.1.12.12.9" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">24.20</td>
<td id="S2.T1.1.1.12.12.10" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.12.12.10.1" class="ltx_text ltx_font_bold">32.74</span></td>
<td id="S2.T1.1.1.12.12.11" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">31.93</td>
<td id="S2.T1.1.1.12.12.12" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.12.12.12.1" class="ltx_text ltx_font_bold">20.37</span></td>
<td id="S2.T1.1.1.12.12.13" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">15.40</td>
<td id="S2.T1.1.1.12.12.14" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.12.12.14.1" class="ltx_text ltx_font_bold">38.91</span></td>
<td id="S2.T1.1.1.12.12.15" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">30.44</td>
</tr>
<tr id="S2.T1.1.1.13.13" class="ltx_tr">
<th id="S2.T1.1.1.13.13.1" class="ltx_td ltx_align_left ltx_th ltx_th_row" style="padding-top:1pt;padding-bottom:1pt;">Adapter</th>
<td id="S2.T1.1.1.13.13.2" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.13.13.2.1" class="ltx_text ltx_font_bold">46.71</span></td>
<td id="S2.T1.1.1.13.13.3" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">43.01</td>
<td id="S2.T1.1.1.13.13.4" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.13.13.4.1" class="ltx_text ltx_font_bold">78.05</span></td>
<td id="S2.T1.1.1.13.13.5" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">71.35</td>
<td id="S2.T1.1.1.13.13.6" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.13.13.6.1" class="ltx_text ltx_font_bold">24.30</span></td>
<td id="S2.T1.1.1.13.13.7" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">20.51</td>
<td id="S2.T1.1.1.13.13.8" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.13.13.8.1" class="ltx_text ltx_font_bold">32.53</span></td>
<td id="S2.T1.1.1.13.13.9" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">25.90</td>
<td id="S2.T1.1.1.13.13.10" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.13.13.10.1" class="ltx_text ltx_font_bold">40.84</span></td>
<td id="S2.T1.1.1.13.13.11" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">31.98</td>
<td id="S2.T1.1.1.13.13.12" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.13.13.12.1" class="ltx_text ltx_font_bold">20.03</span></td>
<td id="S2.T1.1.1.13.13.13" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">15.61</td>
<td id="S2.T1.1.1.13.13.14" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.13.13.14.1" class="ltx_text ltx_font_bold">40.41</span></td>
<td id="S2.T1.1.1.13.13.15" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">34.65</td>
</tr>
<tr id="S2.T1.1.1.14.14" class="ltx_tr">
<th id="S2.T1.1.1.14.14.1" class="ltx_td ltx_align_left ltx_th ltx_th_row" style="padding-top:1pt;padding-bottom:1pt;">LoRA</th>
<td id="S2.T1.1.1.14.14.2" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.14.14.2.1" class="ltx_text ltx_font_bold">46.81</span></td>
<td id="S2.T1.1.1.14.14.3" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">42.11</td>
<td id="S2.T1.1.1.14.14.4" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.14.14.4.1" class="ltx_text ltx_font_bold">78.26</span></td>
<td id="S2.T1.1.1.14.14.5" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">70.75</td>
<td id="S2.T1.1.1.14.14.6" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.14.14.6.1" class="ltx_text ltx_font_bold">25.17</span></td>
<td id="S2.T1.1.1.14.14.7" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">20.42</td>
<td id="S2.T1.1.1.14.14.8" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.14.14.8.1" class="ltx_text ltx_font_bold">31.84</span></td>
<td id="S2.T1.1.1.14.14.9" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">24.48</td>
<td id="S2.T1.1.1.14.14.10" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.14.14.10.1" class="ltx_text ltx_font_bold">41.56</span></td>
<td id="S2.T1.1.1.14.14.11" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">32.41</td>
<td id="S2.T1.1.1.14.14.12" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.14.14.12.1" class="ltx_text ltx_font_bold">21.47</span></td>
<td id="S2.T1.1.1.14.14.13" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">15.30</td>
<td id="S2.T1.1.1.14.14.14" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.14.14.14.1" class="ltx_text ltx_font_bold">40.85</span></td>
<td id="S2.T1.1.1.14.14.15" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">34.24</td>
</tr>
<tr id="S2.T1.1.1.15.15" class="ltx_tr">
<th id="S2.T1.1.1.15.15.1" class="ltx_td ltx_align_left ltx_th ltx_th_row" style="padding-top:1pt;padding-bottom:1pt;">Fine-tuning</th>
<td id="S2.T1.1.1.15.15.2" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.15.15.2.1" class="ltx_text ltx_font_bold">41.34</span></td>
<td id="S2.T1.1.1.15.15.3" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">29.79</td>
<td id="S2.T1.1.1.15.15.4" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.15.15.4.1" class="ltx_text ltx_font_bold">79.82</span></td>
<td id="S2.T1.1.1.15.15.5" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">68.84</td>
<td id="S2.T1.1.1.15.15.6" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.15.15.6.1" class="ltx_text ltx_font_bold">22.33</span></td>
<td id="S2.T1.1.1.15.15.7" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">19.37</td>
<td id="S2.T1.1.1.15.15.8" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.15.15.8.1" class="ltx_text ltx_font_bold">49.67</span></td>
<td id="S2.T1.1.1.15.15.9" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">23.53</td>
<td id="S2.T1.1.1.15.15.10" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.15.15.10.1" class="ltx_text ltx_font_bold">37.01</span></td>
<td id="S2.T1.1.1.15.15.11" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">33.56</td>
<td id="S2.T1.1.1.15.15.12" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.15.15.12.1" class="ltx_text ltx_font_bold">21.95</span></td>
<td id="S2.T1.1.1.15.15.13" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">15.29</td>
<td id="S2.T1.1.1.15.15.14" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.15.15.14.1" class="ltx_text ltx_font_bold">42.02</span></td>
<td id="S2.T1.1.1.15.15.15" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">31.73</td>
</tr>
<tr id="S2.T1.1.1.16.16" class="ltx_tr">
<th id="S2.T1.1.1.16.16.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;" rowspan="4"><span id="S2.T1.1.1.16.16.1.1" class="ltx_text ltx_font_bold">Large</span></th>
<th id="S2.T1.1.1.16.16.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;">Zero-shot</th>
<td id="S2.T1.1.1.16.16.3" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;">25.37</td>
<td id="S2.T1.1.1.16.16.4" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.16.16.4.1" class="ltx_text ltx_font_bold">31.43</span></td>
<td id="S2.T1.1.1.16.16.5" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;">48.68</td>
<td id="S2.T1.1.1.16.16.6" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.16.16.6.1" class="ltx_text ltx_font_bold">60.30</span></td>
<td id="S2.T1.1.1.16.16.7" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.16.16.7.1" class="ltx_text ltx_font_bold">13.92</span></td>
<td id="S2.T1.1.1.16.16.8" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;">7.98</td>
<td id="S2.T1.1.1.16.16.9" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;">8.73</td>
<td id="S2.T1.1.1.16.16.10" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.16.16.10.1" class="ltx_text ltx_font_bold">10.52</span></td>
<td id="S2.T1.1.1.16.16.11" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.16.16.11.1" class="ltx_text ltx_font_bold">2.97</span></td>
<td id="S2.T1.1.1.16.16.12" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;">1.87</td>
<td id="S2.T1.1.1.16.16.13" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;">6.30</td>
<td id="S2.T1.1.1.16.16.14" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.16.16.14.1" class="ltx_text ltx_font_bold">9.33</span></td>
<td id="S2.T1.1.1.16.16.15" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;">17.66</td>
<td id="S2.T1.1.1.16.16.16" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.16.16.16.1" class="ltx_text ltx_font_bold">20.24</span></td>
</tr>
<tr id="S2.T1.1.1.17.17" class="ltx_tr">
<th id="S2.T1.1.1.17.17.1" class="ltx_td ltx_align_left ltx_th ltx_th_row" style="padding-top:1pt;padding-bottom:1pt;">P-tuning</th>
<td id="S2.T1.1.1.17.17.2" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.17.17.2.1" class="ltx_text ltx_font_bold">45.20</span></td>
<td id="S2.T1.1.1.17.17.3" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">15.78</td>
<td id="S2.T1.1.1.17.17.4" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.17.17.4.1" class="ltx_text ltx_font_bold">78.33</span></td>
<td id="S2.T1.1.1.17.17.5" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">73.22</td>
<td id="S2.T1.1.1.17.17.6" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.17.17.6.1" class="ltx_text ltx_font_bold">25.21</span></td>
<td id="S2.T1.1.1.17.17.7" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">21.58</td>
<td id="S2.T1.1.1.17.17.8" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.17.17.8.1" class="ltx_text ltx_font_bold">34.24</span></td>
<td id="S2.T1.1.1.17.17.9" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">24.50</td>
<td id="S2.T1.1.1.17.17.10" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.17.17.10.1" class="ltx_text ltx_font_bold">47.65</span></td>
<td id="S2.T1.1.1.17.17.11" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">39.93</td>
<td id="S2.T1.1.1.17.17.12" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.17.17.12.1" class="ltx_text ltx_font_bold">20.07</span></td>
<td id="S2.T1.1.1.17.17.13" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">15.00</td>
<td id="S2.T1.1.1.17.17.14" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.17.17.14.1" class="ltx_text ltx_font_bold">41.78</span></td>
<td id="S2.T1.1.1.17.17.15" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">31.67</td>
</tr>
<tr id="S2.T1.1.1.18.18" class="ltx_tr">
<th id="S2.T1.1.1.18.18.1" class="ltx_td ltx_align_left ltx_th ltx_th_row" style="padding-top:1pt;padding-bottom:1pt;">Adapter</th>
<td id="S2.T1.1.1.18.18.2" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.18.18.2.1" class="ltx_text ltx_font_bold">47.48</span></td>
<td id="S2.T1.1.1.18.18.3" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">44.43</td>
<td id="S2.T1.1.1.18.18.4" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.18.18.4.1" class="ltx_text ltx_font_bold">79.68</span></td>
<td id="S2.T1.1.1.18.18.5" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">73.57</td>
<td id="S2.T1.1.1.18.18.6" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.18.18.6.1" class="ltx_text ltx_font_bold">26.37</span></td>
<td id="S2.T1.1.1.18.18.7" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">22.03</td>
<td id="S2.T1.1.1.18.18.8" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.18.18.8.1" class="ltx_text ltx_font_bold">32.12</span></td>
<td id="S2.T1.1.1.18.18.9" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">26.09</td>
<td id="S2.T1.1.1.18.18.10" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.18.18.10.1" class="ltx_text ltx_font_bold">46.74</span></td>
<td id="S2.T1.1.1.18.18.11" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">38.06</td>
<td id="S2.T1.1.1.18.18.12" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.18.18.12.1" class="ltx_text ltx_font_bold">20.81</span></td>
<td id="S2.T1.1.1.18.18.13" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">15.22</td>
<td id="S2.T1.1.1.18.18.14" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.18.18.14.1" class="ltx_text ltx_font_bold">42.20</span></td>
<td id="S2.T1.1.1.18.18.15" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">36.57</td>
</tr>
<tr id="S2.T1.1.1.19.19" class="ltx_tr">
<th id="S2.T1.1.1.19.19.1" class="ltx_td ltx_align_left ltx_th ltx_th_row" style="padding-top:1pt;padding-bottom:1pt;">LoRA</th>
<td id="S2.T1.1.1.19.19.2" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.19.19.2.1" class="ltx_text ltx_font_bold">47.33</span></td>
<td id="S2.T1.1.1.19.19.3" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">44.48</td>
<td id="S2.T1.1.1.19.19.4" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.19.19.4.1" class="ltx_text ltx_font_bold">79.79</span></td>
<td id="S2.T1.1.1.19.19.5" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">73.63</td>
<td id="S2.T1.1.1.19.19.6" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.19.19.6.1" class="ltx_text ltx_font_bold">25.85</span></td>
<td id="S2.T1.1.1.19.19.7" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">21.49</td>
<td id="S2.T1.1.1.19.19.8" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.19.19.8.1" class="ltx_text ltx_font_bold">32.25</span></td>
<td id="S2.T1.1.1.19.19.9" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">25.21</td>
<td id="S2.T1.1.1.19.19.10" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.19.19.10.1" class="ltx_text ltx_font_bold">42.62</span></td>
<td id="S2.T1.1.1.19.19.11" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">39.31</td>
<td id="S2.T1.1.1.19.19.12" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.19.19.12.1" class="ltx_text ltx_font_bold">21.67</span></td>
<td id="S2.T1.1.1.19.19.13" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">15.02</td>
<td id="S2.T1.1.1.19.19.14" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.19.19.14.1" class="ltx_text ltx_font_bold">41.58</span></td>
<td id="S2.T1.1.1.19.19.15" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">36.53</td>
</tr>
<tr id="S2.T1.1.1.20.20" class="ltx_tr">
<th id="S2.T1.1.1.20.20.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_bb ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;" rowspan="4"><span id="S2.T1.1.1.20.20.1.1" class="ltx_text ltx_font_bold">Extra Large</span></th>
<th id="S2.T1.1.1.20.20.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;">Zero-shot</th>
<td id="S2.T1.1.1.20.20.3" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;">26.97</td>
<td id="S2.T1.1.1.20.20.4" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.20.20.4.1" class="ltx_text ltx_font_bold">33.49</span></td>
<td id="S2.T1.1.1.20.20.5" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;">44.71</td>
<td id="S2.T1.1.1.20.20.6" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.20.20.6.1" class="ltx_text ltx_font_bold">62.87</span></td>
<td id="S2.T1.1.1.20.20.7" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.20.20.7.1" class="ltx_text ltx_font_bold">11.89</span></td>
<td id="S2.T1.1.1.20.20.8" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;">10.07</td>
<td id="S2.T1.1.1.20.20.9" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;">11.58</td>
<td id="S2.T1.1.1.20.20.10" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.20.20.10.1" class="ltx_text ltx_font_bold">13.38</span></td>
<td id="S2.T1.1.1.20.20.11" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.20.20.11.1" class="ltx_text ltx_font_bold">3.07</span></td>
<td id="S2.T1.1.1.20.20.12" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;">0.96</td>
<td id="S2.T1.1.1.20.20.13" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;">7.65</td>
<td id="S2.T1.1.1.20.20.14" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.20.20.14.1" class="ltx_text ltx_font_bold">9.99</span></td>
<td id="S2.T1.1.1.20.20.15" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;">17.65</td>
<td id="S2.T1.1.1.20.20.16" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.20.20.16.1" class="ltx_text ltx_font_bold">21.79</span></td>
</tr>
<tr id="S2.T1.1.1.21.21" class="ltx_tr">
<th id="S2.T1.1.1.21.21.1" class="ltx_td ltx_align_left ltx_th ltx_th_row" style="padding-top:1pt;padding-bottom:1pt;">P-tuning</th>
<td id="S2.T1.1.1.21.21.2" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.21.21.2.1" class="ltx_text ltx_font_bold">47.27</span></td>
<td id="S2.T1.1.1.21.21.3" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">24.53</td>
<td id="S2.T1.1.1.21.21.4" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.21.21.4.1" class="ltx_text ltx_font_bold">80.27</span></td>
<td id="S2.T1.1.1.21.21.5" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">74.38</td>
<td id="S2.T1.1.1.21.21.6" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.21.21.6.1" class="ltx_text ltx_font_bold">27.09</span></td>
<td id="S2.T1.1.1.21.21.7" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">22.48</td>
<td id="S2.T1.1.1.21.21.8" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.21.21.8.1" class="ltx_text ltx_font_bold">34.08</span></td>
<td id="S2.T1.1.1.21.21.9" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">24.93</td>
<td id="S2.T1.1.1.21.21.10" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.21.21.10.1" class="ltx_text ltx_font_bold">57.19</span></td>
<td id="S2.T1.1.1.21.21.11" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">38.06</td>
<td id="S2.T1.1.1.21.21.12" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.21.21.12.1" class="ltx_text ltx_font_bold">21.17</span></td>
<td id="S2.T1.1.1.21.21.13" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">15.53</td>
<td id="S2.T1.1.1.21.21.14" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.21.21.14.1" class="ltx_text ltx_font_bold">44.51</span></td>
<td id="S2.T1.1.1.21.21.15" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">33.32</td>
</tr>
<tr id="S2.T1.1.1.22.22" class="ltx_tr">
<th id="S2.T1.1.1.22.22.1" class="ltx_td ltx_align_left ltx_th ltx_th_row" style="padding-top:1pt;padding-bottom:1pt;">Adapter</th>
<td id="S2.T1.1.1.22.22.2" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.22.22.2.1" class="ltx_text ltx_font_bold">49.68</span></td>
<td id="S2.T1.1.1.22.22.3" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">46.41</td>
<td id="S2.T1.1.1.22.22.4" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.22.22.4.1" class="ltx_text ltx_font_bold">81.64</span></td>
<td id="S2.T1.1.1.22.22.5" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">75.10</td>
<td id="S2.T1.1.1.22.22.6" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.22.22.6.1" class="ltx_text ltx_font_bold">26.94</span></td>
<td id="S2.T1.1.1.22.22.7" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">22.24</td>
<td id="S2.T1.1.1.22.22.8" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.22.22.8.1" class="ltx_text ltx_font_bold">33.94</span></td>
<td id="S2.T1.1.1.22.22.9" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">26.38</td>
<td id="S2.T1.1.1.22.22.10" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.22.22.10.1" class="ltx_text ltx_font_bold">54.65</span></td>
<td id="S2.T1.1.1.22.22.11" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">42.62</td>
<td id="S2.T1.1.1.22.22.12" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.22.22.12.1" class="ltx_text ltx_font_bold">21.19</span></td>
<td id="S2.T1.1.1.22.22.13" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">15.71</td>
<td id="S2.T1.1.1.22.22.14" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.22.22.14.1" class="ltx_text ltx_font_bold">46.82</span></td>
<td id="S2.T1.1.1.22.22.15" class="ltx_td ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">38.08</td>
</tr>
<tr id="S2.T1.1.1.23.23" class="ltx_tr">
<th id="S2.T1.1.1.23.23.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb" style="padding-top:1pt;padding-bottom:1pt;">LoRA</th>
<td id="S2.T1.1.1.23.23.2" class="ltx_td ltx_align_center ltx_border_bb" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.23.23.2.1" class="ltx_text ltx_font_bold">49.21</span></td>
<td id="S2.T1.1.1.23.23.3" class="ltx_td ltx_align_center ltx_border_bb" style="padding-top:1pt;padding-bottom:1pt;">44.53</td>
<td id="S2.T1.1.1.23.23.4" class="ltx_td ltx_align_center ltx_border_bb" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.23.23.4.1" class="ltx_text ltx_font_bold">81.87</span></td>
<td id="S2.T1.1.1.23.23.5" class="ltx_td ltx_align_center ltx_border_bb" style="padding-top:1pt;padding-bottom:1pt;">74.92</td>
<td id="S2.T1.1.1.23.23.6" class="ltx_td ltx_align_center ltx_border_bb" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.23.23.6.1" class="ltx_text ltx_font_bold">27.31</span></td>
<td id="S2.T1.1.1.23.23.7" class="ltx_td ltx_align_center ltx_border_bb" style="padding-top:1pt;padding-bottom:1pt;">22.16</td>
<td id="S2.T1.1.1.23.23.8" class="ltx_td ltx_align_center ltx_border_bb" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.23.23.8.1" class="ltx_text ltx_font_bold">31.98</span></td>
<td id="S2.T1.1.1.23.23.9" class="ltx_td ltx_align_center ltx_border_bb" style="padding-top:1pt;padding-bottom:1pt;">27.49</td>
<td id="S2.T1.1.1.23.23.10" class="ltx_td ltx_align_center ltx_border_bb" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.23.23.10.1" class="ltx_text ltx_font_bold">49.19</span></td>
<td id="S2.T1.1.1.23.23.11" class="ltx_td ltx_align_center ltx_border_bb" style="padding-top:1pt;padding-bottom:1pt;">39.65</td>
<td id="S2.T1.1.1.23.23.12" class="ltx_td ltx_align_center ltx_border_bb" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.23.23.12.1" class="ltx_text ltx_font_bold">22.77</span></td>
<td id="S2.T1.1.1.23.23.13" class="ltx_td ltx_align_center ltx_border_bb" style="padding-top:1pt;padding-bottom:1pt;">15.73</td>
<td id="S2.T1.1.1.23.23.14" class="ltx_td ltx_align_center ltx_border_bb" style="padding-top:1pt;padding-bottom:1pt;"><span id="S2.T1.1.1.23.23.14.1" class="ltx_text ltx_font_bold">43.72</span></td>
<td id="S2.T1.1.1.23.23.15" class="ltx_td ltx_align_center ltx_border_bb" style="padding-top:1pt;padding-bottom:1pt;">37.41</td>
</tr>
</tbody>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 1: </span>A comprehensive comparison between GPT vs RETRO on six datasets. <span id="S2.T1.3.1" class="ltx_text ltx_font_bold">Bold</span> indicates the better result in each head-to-head comparison.</figcaption>
</figure>
</section>
<section id="S3" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">3 </span>Experimental Setup</h2>

<section id="S3.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.1 </span>Datasets</h3>

<div id="S3.SS1.p1" class="ltx_para">
<p id="S3.SS1.p1.1" class="ltx_p">To cover several task categories, we use six datasets suited to benefit from retrieval and finetuning. We select <span id="S3.SS1.p1.1.1" class="ltx_text ltx_font_bold">Natural Questions</span> (NQ) <cite class="ltx_cite ltx_citemacro_citep">(Kwiatkowski et al., <a href="#bib.bib14" title="" class="ltx_ref">2019</a>)</cite>, <span id="S3.SS1.p1.1.2" class="ltx_text ltx_font_bold">TriviaQA</span> (TQA) <cite class="ltx_cite ltx_citemacro_citep">(Joshi et al., <a href="#bib.bib11" title="" class="ltx_ref">2017</a>)</cite>, <span id="S3.SS1.p1.1.3" class="ltx_text ltx_font_bold">NarrativeQA</span> (NQA) <cite class="ltx_cite ltx_citemacro_citep">(Kočiskỳ et al., <a href="#bib.bib13" title="" class="ltx_ref">2018</a>)</cite> and <span id="S3.SS1.p1.1.4" class="ltx_text ltx_font_bold">Qasper</span> <cite class="ltx_cite ltx_citemacro_citep">(Dasigi et al., <a href="#bib.bib2" title="" class="ltx_ref">2021</a>)</cite> for document question answering, <span id="S3.SS1.p1.1.5" class="ltx_text ltx_font_bold">QuALITY</span> <cite class="ltx_cite ltx_citemacro_citep">(Pang et al., <a href="#bib.bib23" title="" class="ltx_ref">2021</a>)</cite> for multiple-choice question answering, and <span id="S3.SS1.p1.1.6" class="ltx_text ltx_font_bold">QMSum</span> <cite class="ltx_cite ltx_citemacro_citep">(Zhong et al., <a href="#bib.bib32" title="" class="ltx_ref">2021</a>)</cite> for query-based summarization (see detail statistics in Appendix <a href="#A2" title="Appendix B Dataset Statistics ‣ GPT vs RETRO: Exploring the Intersection of Retrieval and Parameter-Efficient Fine-Tuning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">B</span></a>). Each of these datasets contain necessary external knowledge that must be filtered via retrieval and response behaviour that encourages finetuning. Following the official metrics, we use F1 score for evaluating document QA, exact match for mutliple-choice QA and the geometric mean of ROUGE-1/2/L <cite class="ltx_cite ltx_citemacro_citep">(Lin, <a href="#bib.bib16" title="" class="ltx_ref">2004</a>)</cite> for summarization.</p>
</div>
</section>
<section id="S3.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.2 </span>Models</h3>

<div id="S3.SS2.p1" class="ltx_para">
<p id="S3.SS2.p1.1" class="ltx_p">In order to undertand the effect of model scales, we use base GPT models of sizes 823M (Extra Small), 2.25B (Small), 8.5B (Medium), 22B (Large), and 43B (Extra Large), as introduced in <cite class="ltx_cite ltx_citemacro_citet">Wang et al. (<a href="#bib.bib27" title="" class="ltx_ref">2023a</a>)</cite>, which were pretrained on a massive dataset of 1.2 trillion tokens. We employ the corresponding RETRO models from the same work as the foundation for our retrieval pretrained LLM experiments. Notably, the RETRO architecture features an encoder that extracts neighbors from an external database, which increases the total model size to 877M, 2.47B, 9.5B, 24B, and 48B, respectively.
<cite class="ltx_cite ltx_citemacro_citet">Wang et al. (<a href="#bib.bib27" title="" class="ltx_ref">2023a</a>)</cite> found ablating the encoder after pretraining led to comparable results. In our paper we include it so that adapter modules and LoRA
layers are added throughout decoder and encoder components. For more on the base models, we refer readers to the original work.</p>
</div>
</section>
<section id="S3.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.3 </span>Retrieval</h3>

<div id="S3.SS3.p1" class="ltx_para">
<p id="S3.SS3.p1.1" class="ltx_p">We follow <cite class="ltx_cite ltx_citemacro_citet">Wang et al. (<a href="#bib.bib27" title="" class="ltx_ref">2023a</a>); Xu et al. (<a href="#bib.bib29" title="" class="ltx_ref">2023</a>)</cite> to use Dragon+ <cite class="ltx_cite ltx_citemacro_citep">(Lin et al., <a href="#bib.bib17" title="" class="ltx_ref">2023</a>)</cite> as a retriever. Dragon+ is a dual encoder model that consists of a query encoder and a context encoder.
We first chunk each context document with 100 words, and then encode
both the questions and all chunks independently with corresponding encoders. The most relevant
5 chunks, ranked by the dot product of the question embedding and chunk embedding, are retrieved as neighbors.
For GPT models, they are concatenated together (following the left to right order from the most relevant to least relevant) as the
context of the prompt for generation. For RETRO models, they interact with the question during generation through chunked cross-attention.</p>
</div>
</section>
<section id="S3.SS4" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.4 </span>Parameter Efficient Fine-Tuning</h3>

<div id="S3.SS4.p1" class="ltx_para">
<p id="S3.SS4.p1.1" class="ltx_p">We implement P-tuning in RETRO akin to GPT. Virtual tokens are added to the beginning of the decoder. Based on the design of chunked cross-attention <cite class="ltx_cite ltx_citemacro_citep">(Wang et al., <a href="#bib.bib27" title="" class="ltx_ref">2023a</a>)</cite>, left padding is added to ensure the length of input (virtual tokens + context + question) is a multiple of chunk size. Adapter and LoRA layers are inserted within the transformer blocks in both RETRO and GPT at the feedforward and attention layers. In RETRO they are also inserted in the transformer encoder ingesting the retrieved neighbors. We provide additional hyperparameter tuning, resource utilization and prompt template details in Appendix <a href="#A1" title="Appendix A Details on Experimental Setup ‣ GPT vs RETRO: Exploring the Intersection of Retrieval and Parameter-Efficient Fine-Tuning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">A</span></a>.</p>
</div>
</section>
</section>
<section id="S4" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">4 </span>Results</h2>

<section id="S4.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.1 </span>Main Results</h3>

<div id="S4.SS1.p1" class="ltx_para">
<p id="S4.SS1.p1.1" class="ltx_p">Table <a href="#S2.T1" title="Table 1 ‣ 2 Related Work ‣ GPT vs RETRO: Exploring the Intersection of Retrieval and Parameter-Efficient Fine-Tuning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a> shows the comprehensive comparison between GPT and RETRO models across five sizes and six datasets. From this table we observe:</p>
</div>
<div id="S4.SS1.p2" class="ltx_para">
<p id="S4.SS1.p2.1" class="ltx_p">(1) RETRO is generally better than GPT at zero-shot settings. This superiority stems from its unique pre-training approach. By learning to extract salient information from retrieved text and integrate it into its generation process, RETRO develops the capability to harness relevant contextual knowledge, ultimately leading to its good zero-shot performance. In contrast, GPT relies on an auto-regressive loss during pre-training, focusing on accurately predicting next tokens without the benefit of external retrievals. As a result, GPT’s ability to learn context-aware question-answering is limited to the presence of relevant data within the pre-training corpus, resulting in significantly less targeted training compared to RETRO.</p>
</div>
<div id="S4.SS1.p3" class="ltx_para">
<p id="S4.SS1.p3.1" class="ltx_p">(2) Both RETRO and GPT models exhibit saturation points on these datasets around the 8B mark, with a similar pattern emerging between the two models, albeit with RETRO performing less well. This can be seen in Figure <a href="#S1.F1" title="Figure 1 ‣ 1 Introduction ‣ GPT vs RETRO: Exploring the Intersection of Retrieval and Parameter-Efficient Fine-Tuning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a> and suggests that, for a specific task, a medium-sized PEFT model strikes the optimal balance between cost and performance, making it a sweet spot for many applications.</p>
</div>
<div id="S4.SS1.p4" class="ltx_para">
<p id="S4.SS1.p4.1" class="ltx_p">(3) For GPT models, P-tuning performs worse at smaller models (&lt;=8B), but it performs as good or better than LoRA and Adapter at the large model sizes. This difference is visualized in Figure <a href="#S4.F2" title="Figure 2 ‣ 4.1 Main Results ‣ 4 Results ‣ GPT vs RETRO: Exploring the Intersection of Retrieval and Parameter-Efficient Fine-Tuning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a> and Figure <a href="#A3.F3" title="Figure 3 ‣ Appendix C Supplementary Figures and Tables ‣ GPT vs RETRO: Exploring the Intersection of Retrieval and Parameter-Efficient Fine-Tuning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a> (Appendix <a href="#A3" title="Appendix C Supplementary Figures and Tables ‣ GPT vs RETRO: Exploring the Intersection of Retrieval and Parameter-Efficient Fine-Tuning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">C</span></a>). However, for RETRO models, P-tuning generally under performs the other PEFT methods across all model sizes.</p>
</div>
<div id="S4.SS1.p5" class="ltx_para">
<p id="S4.SS1.p5.1" class="ltx_p">(4) The performance ceiling for PEFT-tuned models is significantly higher for GPT compared to RETRO as seen in Figure <a href="#A3.F4" title="Figure 4 ‣ Appendix C Supplementary Figures and Tables ‣ GPT vs RETRO: Exploring the Intersection of Retrieval and Parameter-Efficient Fine-Tuning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a> (Appendix <a href="#A3" title="Appendix C Supplementary Figures and Tables ‣ GPT vs RETRO: Exploring the Intersection of Retrieval and Parameter-Efficient Fine-Tuning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">C</span></a>). For example, using medium-sized models, the average score of LoRA with GPT is 40.85, while with RETRO it is 34.24. This disparity suggests that GPT has more room for improvement with PEFT tuning. This phenomenon can also be possibly explained by the two different pre-training strategies. Since GPT has been less targeted on retrieval-augmented generation during pre-training, it opens a larger room for improvement during fine-tuning.</p>
</div>
<div id="S4.SS1.p6" class="ltx_para">
<p id="S4.SS1.p6.1" class="ltx_p">(5) We also conduct full fine-tuning on medium-sized GPT and RETRO models. For GPT model, we find that full fine-tuning achieves slightly better performance than PEFT on average, with a notable exception on the QASPER dataset, where the difference is more pronounced and full fine-tuning yields significantly better results.
In contrast, for RETRO model, full fine-tuning generally performs worse than PEFT, demonstrating the effectiveness of PEFT. These findings are consistent with the observations of <cite class="ltx_cite ltx_citemacro_citet">Hu et al. (<a href="#bib.bib9" title="" class="ltx_ref">2021</a>)</cite>.</p>
</div>
<figure id="S4.F2" class="ltx_figure"><img src="/html/2407.04528/assets/x3.png" id="S4.F2.g1" class="ltx_graphics ltx_centering ltx_img_square" width="165" height="143" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 2: </span>Comparison of Extra Large GPT and RETRO results averaged across 6 datasets.</figcaption>
</figure>
</section>
<section id="S4.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.2 </span>Comparing to Instruction-tuned RETRO</h3>

<div id="S4.SS2.p1" class="ltx_para">
<p id="S4.SS2.p1.1" class="ltx_p">Instruction tuning post retrieval-augmented pre-training <cite class="ltx_cite ltx_citemacro_citep">(Wang et al., <a href="#bib.bib27" title="" class="ltx_ref">2023a</a>)</cite> has been demonstrated to improve zero-shot performance on RETRO models. A natural thought is that whether Instruction-tuned RETRO (I-RETRO) serve as a better foundation for applying PEFT compared to the base RETRO. To investigate this, we additionally apply PEFT to a medium-sized I-RETRO model and show overall results in Table <a href="#S4.T2" title="Table 2 ‣ 4.2 Comparing to Instruction-tuned RETRO ‣ 4 Results ‣ GPT vs RETRO: Exploring the Intersection of Retrieval and Parameter-Efficient Fine-Tuning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a> and more granular results in Table <a href="#A3.T4" title="Table 4 ‣ Appendix C Supplementary Figures and Tables ‣ GPT vs RETRO: Exploring the Intersection of Retrieval and Parameter-Efficient Fine-Tuning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a> (Appendix <a href="#A3" title="Appendix C Supplementary Figures and Tables ‣ GPT vs RETRO: Exploring the Intersection of Retrieval and Parameter-Efficient Fine-Tuning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">C</span></a>). Our findings reveal that while I-RETRO exhibits improved performance in the zero-shot setting, it has limited scope for further improvement using PEFT. Even with substantial hyperparameter tuning, the average scores across six datasets, using each of the three PEFT methods, demonstrate an approximately 10% gap between I-RETRO and base RETRO. We hypothesize that conceptually both models should be tunable to similar performance but will leave that exploration to future work.</p>
</div>
<figure id="S4.T2" class="ltx_table">
<div id="S4.T2.1" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:433.6pt;height:100.5pt;vertical-align:-0.9pt;"><span class="ltx_transformed_inner" style="transform:translate(-18.3pt,4.2pt) scale(0.921995516704772,0.921995516704772) ;">
<table id="S4.T2.1.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S4.T2.1.1.1.1" class="ltx_tr">
<th id="S4.T2.1.1.1.1.1" class="ltx_td ltx_th ltx_th_row ltx_border_tt" style="padding-top:2.5pt;padding-bottom:2.5pt;"></th>
<th id="S4.T2.1.1.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding-top:2.5pt;padding-bottom:2.5pt;" colspan="2"><span id="S4.T2.1.1.1.1.2.1" class="ltx_text ltx_font_bold">Average QA</span></th>
<th id="S4.T2.1.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding-top:2.5pt;padding-bottom:2.5pt;" colspan="2"><span id="S4.T2.1.1.1.1.3.1" class="ltx_text ltx_font_bold">QUALITY</span></th>
<th id="S4.T2.1.1.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding-top:2.5pt;padding-bottom:2.5pt;" colspan="2"><span id="S4.T2.1.1.1.1.4.1" class="ltx_text ltx_font_bold">QMSUM</span></th>
<th id="S4.T2.1.1.1.1.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding-top:2.5pt;padding-bottom:2.5pt;" colspan="2"><span id="S4.T2.1.1.1.1.5.1" class="ltx_text ltx_font_bold">Average</span></th>
</tr>
<tr id="S4.T2.1.1.2.2" class="ltx_tr">
<th id="S4.T2.1.1.2.2.1" class="ltx_td ltx_th ltx_th_row" style="padding-top:2.5pt;padding-bottom:2.5pt;"></th>
<th id="S4.T2.1.1.2.2.2" class="ltx_td ltx_align_center ltx_th ltx_th_column" style="padding-top:2.5pt;padding-bottom:2.5pt;">I-RETRO</th>
<th id="S4.T2.1.1.2.2.3" class="ltx_td ltx_align_center ltx_th ltx_th_column" style="padding-top:2.5pt;padding-bottom:2.5pt;">RETRO</th>
<th id="S4.T2.1.1.2.2.4" class="ltx_td ltx_align_center ltx_th ltx_th_column" style="padding-top:2.5pt;padding-bottom:2.5pt;">I-RETRO</th>
<th id="S4.T2.1.1.2.2.5" class="ltx_td ltx_align_center ltx_th ltx_th_column" style="padding-top:2.5pt;padding-bottom:2.5pt;">RETRO</th>
<th id="S4.T2.1.1.2.2.6" class="ltx_td ltx_align_center ltx_th ltx_th_column" style="padding-top:2.5pt;padding-bottom:2.5pt;">I-RETRO</th>
<th id="S4.T2.1.1.2.2.7" class="ltx_td ltx_align_center ltx_th ltx_th_column" style="padding-top:2.5pt;padding-bottom:2.5pt;">RETRO</th>
<th id="S4.T2.1.1.2.2.8" class="ltx_td ltx_align_center ltx_th ltx_th_column" style="padding-top:2.5pt;padding-bottom:2.5pt;">I-RETRO</th>
<th id="S4.T2.1.1.2.2.9" class="ltx_td ltx_align_center ltx_th ltx_th_column" style="padding-top:2.5pt;padding-bottom:2.5pt;">RETRO</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S4.T2.1.1.3.1" class="ltx_tr">
<th id="S4.T2.1.1.3.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t" style="padding-top:2.5pt;padding-bottom:2.5pt;">Zero-shot</th>
<td id="S4.T2.1.1.3.1.2" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:2.5pt;padding-bottom:2.5pt;"><span id="S4.T2.1.1.3.1.2.1" class="ltx_text ltx_font_bold">27.65</span></td>
<td id="S4.T2.1.1.3.1.3" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:2.5pt;padding-bottom:2.5pt;">23.79</td>
<td id="S4.T2.1.1.3.1.4" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:2.5pt;padding-bottom:2.5pt;"><span id="S4.T2.1.1.3.1.4.1" class="ltx_text ltx_font_bold">3.35</span></td>
<td id="S4.T2.1.1.3.1.5" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:2.5pt;padding-bottom:2.5pt;">0.14</td>
<td id="S4.T2.1.1.3.1.6" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:2.5pt;padding-bottom:2.5pt;"><span id="S4.T2.1.1.3.1.6.1" class="ltx_text ltx_font_bold">11.04</span></td>
<td id="S4.T2.1.1.3.1.7" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:2.5pt;padding-bottom:2.5pt;">9.15</td>
<td id="S4.T2.1.1.3.1.8" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:2.5pt;padding-bottom:2.5pt;"><span id="S4.T2.1.1.3.1.8.1" class="ltx_text ltx_font_bold">20.83</span></td>
<td id="S4.T2.1.1.3.1.9" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:2.5pt;padding-bottom:2.5pt;">17.00</td>
</tr>
<tr id="S4.T2.1.1.4.2" class="ltx_tr">
<th id="S4.T2.1.1.4.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row" style="padding-top:2.5pt;padding-bottom:2.5pt;">P-tuning</th>
<td id="S4.T2.1.1.4.2.2" class="ltx_td ltx_align_center" style="padding-top:2.5pt;padding-bottom:2.5pt;">23.25</td>
<td id="S4.T2.1.1.4.2.3" class="ltx_td ltx_align_center" style="padding-top:2.5pt;padding-bottom:2.5pt;"><span id="S4.T2.1.1.4.2.3.1" class="ltx_text ltx_font_bold">47.18</span></td>
<td id="S4.T2.1.1.4.2.4" class="ltx_td ltx_align_center" style="padding-top:2.5pt;padding-bottom:2.5pt;">16.68</td>
<td id="S4.T2.1.1.4.2.5" class="ltx_td ltx_align_center" style="padding-top:2.5pt;padding-bottom:2.5pt;"><span id="S4.T2.1.1.4.2.5.1" class="ltx_text ltx_font_bold">31.93</span></td>
<td id="S4.T2.1.1.4.2.6" class="ltx_td ltx_align_center" style="padding-top:2.5pt;padding-bottom:2.5pt;"><span id="S4.T2.1.1.4.2.6.1" class="ltx_text ltx_font_bold">15.88</span></td>
<td id="S4.T2.1.1.4.2.7" class="ltx_td ltx_align_center" style="padding-top:2.5pt;padding-bottom:2.5pt;">15.40</td>
<td id="S4.T2.1.1.4.2.8" class="ltx_td ltx_align_center" style="padding-top:2.5pt;padding-bottom:2.5pt;">20.75</td>
<td id="S4.T2.1.1.4.2.9" class="ltx_td ltx_align_center" style="padding-top:2.5pt;padding-bottom:2.5pt;"><span id="S4.T2.1.1.4.2.9.1" class="ltx_text ltx_font_bold">30.44</span></td>
</tr>
<tr id="S4.T2.1.1.5.3" class="ltx_tr">
<th id="S4.T2.1.1.5.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row" style="padding-top:2.5pt;padding-bottom:2.5pt;">Adapter</th>
<td id="S4.T2.1.1.5.3.2" class="ltx_td ltx_align_center" style="padding-top:2.5pt;padding-bottom:2.5pt;">22.64</td>
<td id="S4.T2.1.1.5.3.3" class="ltx_td ltx_align_center" style="padding-top:2.5pt;padding-bottom:2.5pt;"><span id="S4.T2.1.1.5.3.3.1" class="ltx_text ltx_font_bold">52.75</span></td>
<td id="S4.T2.1.1.5.3.4" class="ltx_td ltx_align_center" style="padding-top:2.5pt;padding-bottom:2.5pt;">29.87</td>
<td id="S4.T2.1.1.5.3.5" class="ltx_td ltx_align_center" style="padding-top:2.5pt;padding-bottom:2.5pt;"><span id="S4.T2.1.1.5.3.5.1" class="ltx_text ltx_font_bold">31.98</span></td>
<td id="S4.T2.1.1.5.3.6" class="ltx_td ltx_align_center" style="padding-top:2.5pt;padding-bottom:2.5pt;">15.06</td>
<td id="S4.T2.1.1.5.3.7" class="ltx_td ltx_align_center" style="padding-top:2.5pt;padding-bottom:2.5pt;"><span id="S4.T2.1.1.5.3.7.1" class="ltx_text ltx_font_bold">15.16</span></td>
<td id="S4.T2.1.1.5.3.8" class="ltx_td ltx_align_center" style="padding-top:2.5pt;padding-bottom:2.5pt;">22.58</td>
<td id="S4.T2.1.1.5.3.9" class="ltx_td ltx_align_center" style="padding-top:2.5pt;padding-bottom:2.5pt;"><span id="S4.T2.1.1.5.3.9.1" class="ltx_text ltx_font_bold">34.65</span></td>
</tr>
<tr id="S4.T2.1.1.6.4" class="ltx_tr">
<th id="S4.T2.1.1.6.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb" style="padding-top:2.5pt;padding-bottom:2.5pt;">LoRA</th>
<td id="S4.T2.1.1.6.4.2" class="ltx_td ltx_align_center ltx_border_bb" style="padding-top:2.5pt;padding-bottom:2.5pt;">26.53</td>
<td id="S4.T2.1.1.6.4.3" class="ltx_td ltx_align_center ltx_border_bb" style="padding-top:2.5pt;padding-bottom:2.5pt;"><span id="S4.T2.1.1.6.4.3.1" class="ltx_text ltx_font_bold">52.80</span></td>
<td id="S4.T2.1.1.6.4.4" class="ltx_td ltx_align_center ltx_border_bb" style="padding-top:2.5pt;padding-bottom:2.5pt;">24.21</td>
<td id="S4.T2.1.1.6.4.5" class="ltx_td ltx_align_center ltx_border_bb" style="padding-top:2.5pt;padding-bottom:2.5pt;"><span id="S4.T2.1.1.6.4.5.1" class="ltx_text ltx_font_bold">32.41</span></td>
<td id="S4.T2.1.1.6.4.6" class="ltx_td ltx_align_center ltx_border_bb" style="padding-top:2.5pt;padding-bottom:2.5pt;"><span id="S4.T2.1.1.6.4.6.1" class="ltx_text ltx_font_bold">15.40</span></td>
<td id="S4.T2.1.1.6.4.7" class="ltx_td ltx_align_center ltx_border_bb" style="padding-top:2.5pt;padding-bottom:2.5pt;">15.30</td>
<td id="S4.T2.1.1.6.4.8" class="ltx_td ltx_align_center ltx_border_bb" style="padding-top:2.5pt;padding-bottom:2.5pt;">24.29</td>
<td id="S4.T2.1.1.6.4.9" class="ltx_td ltx_align_center ltx_border_bb" style="padding-top:2.5pt;padding-bottom:2.5pt;"><span id="S4.T2.1.1.6.4.9.1" class="ltx_text ltx_font_bold">34.24</span></td>
</tr>
</tbody>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 2: </span>Instruction-tuned RETRO evaluation results.</figcaption>
</figure>
</section>
</section>
<section id="S5" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">5 </span>Conclusion</h2>

<div id="S5.p1" class="ltx_para">
<p id="S5.p1.1" class="ltx_p">This study explores Parameter-Efficient Fine-Tuning (PEFT) methods applied to Retrieval-Augmented Generation (RAG) models, comparing GPT and RETRO architectures. RETRO generally outperforms GPT in zero-shot settings due to their pre-training process that integrates external retrieval, enhancing contextual understanding. However, GPT models show a higher performance potential with PEFT, indicating more room for improvement during fine-tuning. Both RETRO and GPT models perform optimally around the 8B parameter mark, balancing cost and performance. While P-tuning is effective in larger models, it lags behind other methods in smaller models, particularly for RETRO. Applying PEFT to Instruction-tuned RETRO yields limited improvement compared to base RETRO, suggesting a saturation point in leveraging pre-training and fine-tuning benefits. Our comprehensive analysis offers valuable insights for optimizing large language models with PEFT and RAG to the community.</p>
</div>
</section>
<section id="Sx1" class="ltx_section">
<h2 class="ltx_title ltx_title_section">Limitations</h2>

<div id="Sx1.p1" class="ltx_para">
<p id="Sx1.p1.1" class="ltx_p">Due to the breadth of experiments covered in this work we had to prioritze certain experiments over others. This resulted in us using only the medium sized GPT and RETRO models for additional finetuning and Instruction tuning experiments. We believe these results generalize to the other model sizes but leave that to be validated in future work.</p>
</div>
</section>
<section id="Sx2" class="ltx_section">
<h2 class="ltx_title ltx_title_section">Potential Risks</h2>

<div id="Sx2.p1" class="ltx_para">
<p id="Sx2.p1.1" class="ltx_p">The environmental impact associated with training and fine-tuning large models is not negligable as it involves substantial computational resources and energy consumption. While PEFT aims to alleviate this by reducing the number of tunable parameters, works like ours still require significant compute to distinguish which methods are more promising.</p>
</div>
</section>
<section id="bib" class="ltx_bibliography">
<h2 class="ltx_title ltx_title_bibliography">References</h2>

<ul class="ltx_biblist">
<li id="bib.bib1" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Borgeaud et al. (2022)</span>
<span class="ltx_bibblock">
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. 2022.

</span>
<span class="ltx_bibblock">Improving language models by retrieving from trillions of tokens.

</span>
<span class="ltx_bibblock">In <em id="bib.bib1.1.1" class="ltx_emph ltx_font_italic">International conference on machine learning</em>, pages 2206–2240. PMLR.

</span>
</li>
<li id="bib.bib2" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Dasigi et al. (2021)</span>
<span class="ltx_bibblock">
Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A Smith, and Matt Gardner. 2021.

</span>
<span class="ltx_bibblock">A dataset of information-seeking questions and answers anchored in research papers.

</span>
<span class="ltx_bibblock"><em id="bib.bib2.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2105.03011</em>.

</span>
</li>
<li id="bib.bib3" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Dettmers et al. (2024)</span>
<span class="ltx_bibblock">
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2024.

</span>
<span class="ltx_bibblock">Qlora: Efficient finetuning of quantized llms.

</span>
<span class="ltx_bibblock"><em id="bib.bib3.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>, 36.

</span>
</li>
<li id="bib.bib4" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Gao et al. (2023)</span>
<span class="ltx_bibblock">
Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang. 2023.

</span>
<span class="ltx_bibblock">Retrieval-augmented generation for large language models: A survey.

</span>
<span class="ltx_bibblock"><em id="bib.bib4.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2312.10997</em>.

</span>
</li>
<li id="bib.bib5" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Gupta et al. (2024)</span>
<span class="ltx_bibblock">
Aman Gupta, Anup Shirgaonkar, Angels de Luis Balaguer, Bruno Silva, Daniel Holstein, Dawei Li, Jennifer Marsman, Leonardo O Nunes, Mahsa Rouzbahman, Morris Sharp, et al. 2024.

</span>
<span class="ltx_bibblock">Rag vs fine-tuning: Pipelines, tradeoffs, and a case study on agriculture.

</span>
<span class="ltx_bibblock"><em id="bib.bib5.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2401.08406</em>.

</span>
</li>
<li id="bib.bib6" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Guu et al. (2020)</span>
<span class="ltx_bibblock">
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. 2020.

</span>
<span class="ltx_bibblock">Retrieval augmented language model pre-training.

</span>
<span class="ltx_bibblock">In <em id="bib.bib6.1.1" class="ltx_emph ltx_font_italic">International conference on machine learning</em>, pages 3929–3938. PMLR.

</span>
</li>
<li id="bib.bib7" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Han et al. (2024)</span>
<span class="ltx_bibblock">
Zeyu Han, Chao Gao, Jinyang Liu, Sai Qian Zhang, et al. 2024.

</span>
<span class="ltx_bibblock">Parameter-efficient fine-tuning for large models: A comprehensive survey.

</span>
<span class="ltx_bibblock"><em id="bib.bib7.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2403.14608</em>.

</span>
</li>
<li id="bib.bib8" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Houlsby et al. (2019)</span>
<span class="ltx_bibblock">
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019.

</span>
<span class="ltx_bibblock">Parameter-efficient transfer learning for nlp.

</span>
<span class="ltx_bibblock">In <em id="bib.bib8.1.1" class="ltx_emph ltx_font_italic">International conference on machine learning</em>, pages 2790–2799. PMLR.

</span>
</li>
<li id="bib.bib9" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hu et al. (2021)</span>
<span class="ltx_bibblock">
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021.

</span>
<span class="ltx_bibblock">Lora: Low-rank adaptation of large language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib9.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2106.09685</em>.

</span>
</li>
<li id="bib.bib10" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Izacard et al. (2023)</span>
<span class="ltx_bibblock">
Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. 2023.

</span>
<span class="ltx_bibblock">Atlas: Few-shot learning with retrieval augmented language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib10.1.1" class="ltx_emph ltx_font_italic">Journal of Machine Learning Research</em>, 24(251):1–43.

</span>
</li>
<li id="bib.bib11" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Joshi et al. (2017)</span>
<span class="ltx_bibblock">
Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. 2017.

</span>
<span class="ltx_bibblock">Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension.

</span>
<span class="ltx_bibblock"><em id="bib.bib11.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:1705.03551</em>.

</span>
</li>
<li id="bib.bib12" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Karpukhin et al. (2020)</span>
<span class="ltx_bibblock">
Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020.

</span>
<span class="ltx_bibblock">Dense passage retrieval for open-domain question answering.

</span>
<span class="ltx_bibblock"><em id="bib.bib12.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2004.04906</em>.

</span>
</li>
<li id="bib.bib13" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Kočiskỳ et al. (2018)</span>
<span class="ltx_bibblock">
Tomáš Kočiskỳ, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. 2018.

</span>
<span class="ltx_bibblock">The narrativeqa reading comprehension challenge.

</span>
<span class="ltx_bibblock"><em id="bib.bib13.1.1" class="ltx_emph ltx_font_italic">Transactions of the Association for Computational Linguistics</em>, 6:317–328.

</span>
</li>
<li id="bib.bib14" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Kwiatkowski et al. (2019)</span>
<span class="ltx_bibblock">
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019.

</span>
<span class="ltx_bibblock">Natural questions: a benchmark for question answering research.

</span>
<span class="ltx_bibblock"><em id="bib.bib14.1.1" class="ltx_emph ltx_font_italic">Transactions of the Association for Computational Linguistics</em>, 7:453–466.

</span>
</li>
<li id="bib.bib15" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lakatos et al. (2024)</span>
<span class="ltx_bibblock">
Robert Lakatos, Peter Pollner, Andras Hajdu, and Tamas Joo. 2024.

</span>
<span class="ltx_bibblock">Investigating the performance of retrieval-augmented generation and fine-tuning for the development of ai-driven knowledge-based systems.

</span>
<span class="ltx_bibblock"><em id="bib.bib15.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2403.09727</em>.

</span>
</li>
<li id="bib.bib16" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lin (2004)</span>
<span class="ltx_bibblock">
Chin-Yew Lin. 2004.

</span>
<span class="ltx_bibblock">Rouge: A package for automatic evaluation of summaries.

</span>
<span class="ltx_bibblock">In <em id="bib.bib16.1.1" class="ltx_emph ltx_font_italic">Text summarization branches out</em>, pages 74–81.

</span>
</li>
<li id="bib.bib17" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lin et al. (2023)</span>
<span class="ltx_bibblock">
Sheng-Chieh Lin, Akari Asai, Minghan Li, Barlas Oguz, Jimmy Lin, Yashar Mehdad, Wen-tau Yih, and Xilun Chen. 2023.

</span>
<span class="ltx_bibblock">How to train your dragon: Diverse augmentation towards generalizable dense retrieval.

</span>
<span class="ltx_bibblock"><em id="bib.bib17.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2302.07452</em>.

</span>
</li>
<li id="bib.bib18" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Liu et al. (2024)</span>
<span class="ltx_bibblock">
Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024.

</span>
<span class="ltx_bibblock">Lost in the middle: How language models use long contexts.

</span>
<span class="ltx_bibblock"><em id="bib.bib18.1.1" class="ltx_emph ltx_font_italic">Transactions of the Association for Computational Linguistics</em>, 12:157–173.

</span>
</li>
<li id="bib.bib19" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Liu et al. (2023)</span>
<span class="ltx_bibblock">
Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. 2023.

</span>
<span class="ltx_bibblock">Gpt understands, too.

</span>
<span class="ltx_bibblock"><em id="bib.bib19.1.1" class="ltx_emph ltx_font_italic">AI Open</em>.

</span>
</li>
<li id="bib.bib20" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Nakano et al. (2021)</span>
<span class="ltx_bibblock">
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. 2021.

</span>
<span class="ltx_bibblock">Webgpt: Browser-assisted question-answering with human feedback, 2021.

</span>
<span class="ltx_bibblock"><em id="bib.bib20.1.1" class="ltx_emph ltx_font_italic">URL https://arxiv. org/abs/2112.09332</em>.

</span>
</li>
<li id="bib.bib21" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ovadia et al. (2023)</span>
<span class="ltx_bibblock">
Oded Ovadia, Menachem Brief, Moshik Mishaeli, and Oren Elisha. 2023.

</span>
<span class="ltx_bibblock">Fine-tuning or retrieval? comparing knowledge injection in llms.

</span>
<span class="ltx_bibblock"><em id="bib.bib21.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2312.05934</em>.

</span>
</li>
<li id="bib.bib22" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Page et al. (1999)</span>
<span class="ltx_bibblock">
Lawrence Page, Sergey Brin, Rajeev Motwani, Terry Winograd, et al. 1999.

</span>
<span class="ltx_bibblock">The pagerank citation ranking: Bringing order to the web.

</span>
</li>
<li id="bib.bib23" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Pang et al. (2021)</span>
<span class="ltx_bibblock">
Richard Yuanzhe Pang, Alicia Parrish, Nitish Joshi, Nikita Nangia, Jason Phang, Angelica Chen, Vishakh Padmakumar, Johnny Ma, Jana Thompson, He He, et al. 2021.

</span>
<span class="ltx_bibblock">Qual-ity: Question answering with long input texts, yes.

</span>
<span class="ltx_bibblock"><em id="bib.bib23.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2112.08608</em>.

</span>
</li>
<li id="bib.bib24" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Rangan and Yin (2024)</span>
<span class="ltx_bibblock">
Keshav Rangan and Yiqiao Yin. 2024.

</span>
<span class="ltx_bibblock">A fine-tuning enhanced rag system with quantized influence measure as ai judge.

</span>
<span class="ltx_bibblock"><em id="bib.bib24.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2402.17081</em>.

</span>
</li>
<li id="bib.bib25" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Robertson et al. (2009)</span>
<span class="ltx_bibblock">
Stephen Robertson, Hugo Zaragoza, et al. 2009.

</span>
<span class="ltx_bibblock">The probabilistic relevance framework: Bm25 and beyond.

</span>
<span class="ltx_bibblock"><em id="bib.bib25.1.1" class="ltx_emph ltx_font_italic">Foundations and Trends® in Information Retrieval</em>, 3(4):333–389.

</span>
</li>
<li id="bib.bib26" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Soudani et al. (2024)</span>
<span class="ltx_bibblock">
Heydar Soudani, Evangelos Kanoulas, and Faegheh Hasibi. 2024.

</span>
<span class="ltx_bibblock">Fine tuning vs. retrieval augmented generation for less popular knowledge.

</span>
<span class="ltx_bibblock"><em id="bib.bib26.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2403.01432</em>.

</span>
</li>
<li id="bib.bib27" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wang et al. (2023a)</span>
<span class="ltx_bibblock">
Boxin Wang, Wei Ping, Lawrence McAfee, Peng Xu, Bo Li, Mohammad Shoeybi, and Bryan Catanzaro. 2023a.

</span>
<span class="ltx_bibblock">Instructretro: Instruction tuning post retrieval-augmented pretraining.

</span>
<span class="ltx_bibblock"><em id="bib.bib27.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2310.07713</em>.

</span>
</li>
<li id="bib.bib28" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wang et al. (2023b)</span>
<span class="ltx_bibblock">
Boxin Wang, Wei Ping, Peng Xu, Lawrence McAfee, Zihan Liu, Mohammad Shoeybi, Yi Dong, Oleksii Kuchaiev, Bo Li, Chaowei Xiao, et al. 2023b.

</span>
<span class="ltx_bibblock">Shall we pretrain autoregressive language models with retrieval? a comprehensive study.

</span>
<span class="ltx_bibblock"><em id="bib.bib28.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2304.06762</em>.

</span>
</li>
<li id="bib.bib29" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Xu et al. (2023)</span>
<span class="ltx_bibblock">
Peng Xu, Wei Ping, Xianchao Wu, Lawrence McAfee, Chen Zhu, Zihan Liu, Sandeep Subramanian, Evelina Bakhturina, Mohammad Shoeybi, and Bryan Catanzaro. 2023.

</span>
<span class="ltx_bibblock">Retrieval meets long context large language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib29.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2310.03025</em>.

</span>
</li>
<li id="bib.bib30" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhang et al. (2024a)</span>
<span class="ltx_bibblock">
Liang Zhang, Katherine Jijo, Spurthi Setty, Eden Chung, Fatima Javid, Natan Vidra, and Tommy Clifford. 2024a.

</span>
<span class="ltx_bibblock">Enhancing large language model performance to answer questions and extract information more accurately.

</span>
<span class="ltx_bibblock"><em id="bib.bib30.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2402.01722</em>.

</span>
</li>
<li id="bib.bib31" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhang et al. (2024b)</span>
<span class="ltx_bibblock">
Tianjun Zhang, Shishir G Patil, Naman Jain, Sheng Shen, Matei Zaharia, Ion Stoica, and Joseph E Gonzalez. 2024b.

</span>
<span class="ltx_bibblock">Raft: Adapting language model to domain specific rag.

</span>
<span class="ltx_bibblock"><em id="bib.bib31.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2403.10131</em>.

</span>
</li>
<li id="bib.bib32" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhong et al. (2021)</span>
<span class="ltx_bibblock">
Ming Zhong, Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Hassan Awadallah, Asli Celikyilmaz, Yang Liu, Xipeng Qiu, et al. 2021.

</span>
<span class="ltx_bibblock">Qmsum: A new benchmark for query-based multi-domain meeting summarization.

</span>
<span class="ltx_bibblock"><em id="bib.bib32.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2104.05938</em>.

</span>
</li>
</ul>
</section>
<section id="A1" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix A </span>Details on Experimental Setup</h2>

<section id="A1.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">A.1 </span>Hyperparameter Tuning</h3>

<div id="A1.SS1.p1" class="ltx_para">
<p id="A1.SS1.p1.1" class="ltx_p">We conducted hyperparemter search by selectively modifying number of virtual tokens among the values of 40, 50, 90 and 100, Adapter/LoRA dimensions 16, 32 or 64 and learning rates 1e-4 to 1e-6.</p>
</div>
</section>
<section id="A1.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">A.2 </span>Resource Utilization</h3>

<div id="A1.SS2.p1" class="ltx_para">
<p id="A1.SS2.p1.1" class="ltx_p">In our experiments, we used up to 16 compute nodes, each with 8 A100-80GB SXM GPUs. When model is smaller, we increased the data parallelism size, using tools in <a target="_blank" href="https://github.com/NVIDIA/NeMo" title="" class="ltx_ref ltx_href">NeMo framework</a>.</p>
</div>
</section>
<section id="A1.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">A.3 </span>Prompt Template</h3>

<div id="A1.SS3.p1" class="ltx_para">
<p id="A1.SS3.p1.1" class="ltx_p">The template we used to present context to GPT models is as follows.</p>
</div>
<div id="A1.SS3.p2" class="ltx_para">
<pre id="A1.SS3.p2.1" class="ltx_verbatim ltx_font_typewriter">
title: {title}
source: {source}
title: {title}
source: {source}
title: {title}
source: {source}
title: {title}
source: {source}
title: {title}
source: {source}

Question: {question} Answer: The answer is
</pre>
</div>
</section>
</section>
<section id="A2" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix B </span>Dataset Statistics</h2>

<figure id="A2.T3" class="ltx_table">
<div id="A2.T3.1" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:433.6pt;height:92.1pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(47.3pt,-10.0pt) scale(1.27911266929516,1.27911266929516) ;">
<table id="A2.T3.1.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A2.T3.1.1.1.1" class="ltx_tr">
<th id="A2.T3.1.1.1.1.1" class="ltx_td ltx_th ltx_th_row ltx_border_tt"></th>
<th id="A2.T3.1.1.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">NQ</th>
<th id="A2.T3.1.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">TQA</th>
<th id="A2.T3.1.1.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">NQA</th>
<th id="A2.T3.1.1.1.1.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">QASPER</th>
<th id="A2.T3.1.1.1.1.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">QUALITY</th>
<th id="A2.T3.1.1.1.1.7" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">QMSUM</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A2.T3.1.1.2.1" class="ltx_tr">
<th id="A2.T3.1.1.2.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">Train</th>
<td id="A2.T3.1.1.2.1.2" class="ltx_td ltx_align_center ltx_border_t">79168</td>
<td id="A2.T3.1.1.2.1.3" class="ltx_td ltx_align_center ltx_border_t">78785</td>
<td id="A2.T3.1.1.2.1.4" class="ltx_td ltx_align_center ltx_border_t">44002</td>
<td id="A2.T3.1.1.2.1.5" class="ltx_td ltx_align_center ltx_border_t">2053</td>
<td id="A2.T3.1.1.2.1.6" class="ltx_td ltx_align_center ltx_border_t">2018</td>
<td id="A2.T3.1.1.2.1.7" class="ltx_td ltx_align_center ltx_border_t">1005</td>
</tr>
<tr id="A2.T3.1.1.3.2" class="ltx_tr">
<th id="A2.T3.1.1.3.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">Validation</th>
<td id="A2.T3.1.1.3.2.2" class="ltx_td ltx_align_center">8757</td>
<td id="A2.T3.1.1.3.2.3" class="ltx_td ltx_align_center">8837</td>
<td id="A2.T3.1.1.3.2.4" class="ltx_td ltx_align_center">11001</td>
<td id="A2.T3.1.1.3.2.5" class="ltx_td ltx_align_center">514</td>
<td id="A2.T3.1.1.3.2.6" class="ltx_td ltx_align_center">505</td>
<td id="A2.T3.1.1.3.2.7" class="ltx_td ltx_align_center">252</td>
</tr>
<tr id="A2.T3.1.1.4.3" class="ltx_tr">
<th id="A2.T3.1.1.4.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb">Test</th>
<td id="A2.T3.1.1.4.3.2" class="ltx_td ltx_align_center ltx_border_bb">3610</td>
<td id="A2.T3.1.1.4.3.3" class="ltx_td ltx_align_center ltx_border_bb">11313</td>
<td id="A2.T3.1.1.4.3.4" class="ltx_td ltx_align_center ltx_border_bb">5859</td>
<td id="A2.T3.1.1.4.3.5" class="ltx_td ltx_align_center ltx_border_bb">1726</td>
<td id="A2.T3.1.1.4.3.6" class="ltx_td ltx_align_center ltx_border_bb">2086</td>
<td id="A2.T3.1.1.4.3.7" class="ltx_td ltx_align_center ltx_border_bb">272</td>
</tr>
</tbody>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 3: </span>Number of samples in train/validation/test split for each dataset.</figcaption>
</figure>
</section>
<section id="A3" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix C </span>Supplementary Figures and Tables</h2>

<figure id="A3.F3" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="A3.F3.sf1" class="ltx_figure ltx_figure_panel"><img src="/html/2407.04528/assets/x4.png" id="A3.F3.sf1.g1" class="ltx_graphics ltx_img_square" width="230" height="200" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(a) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="A3.F3.sf2" class="ltx_figure ltx_figure_panel"><img src="/html/2407.04528/assets/x5.png" id="A3.F3.sf2.g1" class="ltx_graphics ltx_img_square" width="230" height="200" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(b) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
</div>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">Figure 3: </span>GPT vs RETRO comparisons on Extra Small and Medium sized models.</figcaption>
</figure>
<figure id="A3.F4" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="A3.F4.sf1" class="ltx_figure ltx_figure_panel"><img src="/html/2407.04528/assets/x6.png" id="A3.F4.sf1.g1" class="ltx_graphics ltx_img_landscape" width="230" height="111" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(a) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="A3.F4.sf2" class="ltx_figure ltx_figure_panel"><img src="/html/2407.04528/assets/x7.png" id="A3.F4.sf2.g1" class="ltx_graphics ltx_img_landscape" width="230" height="111" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(b) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="A3.F4.sf3" class="ltx_figure ltx_figure_panel"><img src="/html/2407.04528/assets/x8.png" id="A3.F4.sf3.g1" class="ltx_graphics ltx_img_landscape" width="230" height="111" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(c) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="A3.F4.sf4" class="ltx_figure ltx_figure_panel"><img src="/html/2407.04528/assets/x9.png" id="A3.F4.sf4.g1" class="ltx_graphics ltx_img_landscape" width="230" height="111" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(d) </span></figcaption>
</figure>
</div>
</div>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">Figure 4: </span>GPT vs RETRO seperate method comparisons.</figcaption>
</figure>
<figure id="A3.T4" class="ltx_table">
<div id="A3.T4.1" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:433.6pt;height:60.6pt;vertical-align:-0.6pt;"><span class="ltx_transformed_inner" style="transform:translate(-173.5pt,24.0pt) scale(0.555467607062901,0.555467607062901) ;">
<table id="A3.T4.1.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A3.T4.1.1.1.1" class="ltx_tr">
<th id="A3.T4.1.1.1.1.1" class="ltx_td ltx_th ltx_th_row ltx_border_tt" style="padding-top:2.5pt;padding-bottom:2.5pt;"></th>
<th id="A3.T4.1.1.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding-top:2.5pt;padding-bottom:2.5pt;" colspan="2"><span id="A3.T4.1.1.1.1.2.1" class="ltx_text ltx_font_bold">NQ</span></th>
<th id="A3.T4.1.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding-top:2.5pt;padding-bottom:2.5pt;" colspan="2"><span id="A3.T4.1.1.1.1.3.1" class="ltx_text ltx_font_bold">TQA</span></th>
<th id="A3.T4.1.1.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding-top:2.5pt;padding-bottom:2.5pt;" colspan="2"><span id="A3.T4.1.1.1.1.4.1" class="ltx_text ltx_font_bold">NQA</span></th>
<th id="A3.T4.1.1.1.1.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding-top:2.5pt;padding-bottom:2.5pt;" colspan="2"><span id="A3.T4.1.1.1.1.5.1" class="ltx_text ltx_font_bold">QASPER</span></th>
<th id="A3.T4.1.1.1.1.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding-top:2.5pt;padding-bottom:2.5pt;" colspan="2"><span id="A3.T4.1.1.1.1.6.1" class="ltx_text ltx_font_bold">QUALITY</span></th>
<th id="A3.T4.1.1.1.1.7" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding-top:2.5pt;padding-bottom:2.5pt;" colspan="2"><span id="A3.T4.1.1.1.1.7.1" class="ltx_text ltx_font_bold">QMSUM</span></th>
<th id="A3.T4.1.1.1.1.8" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding-top:2.5pt;padding-bottom:2.5pt;" colspan="2"><span id="A3.T4.1.1.1.1.8.1" class="ltx_text ltx_font_bold">Average</span></th>
</tr>
<tr id="A3.T4.1.1.2.2" class="ltx_tr">
<th id="A3.T4.1.1.2.2.1" class="ltx_td ltx_th ltx_th_row" style="padding-top:2.5pt;padding-bottom:2.5pt;"></th>
<th id="A3.T4.1.1.2.2.2" class="ltx_td ltx_align_center ltx_th ltx_th_column" style="padding-top:2.5pt;padding-bottom:2.5pt;">I-RETRO</th>
<th id="A3.T4.1.1.2.2.3" class="ltx_td ltx_align_center ltx_th ltx_th_column" style="padding-top:2.5pt;padding-bottom:2.5pt;">RETRO</th>
<th id="A3.T4.1.1.2.2.4" class="ltx_td ltx_align_center ltx_th ltx_th_column" style="padding-top:2.5pt;padding-bottom:2.5pt;">I-RETRO</th>
<th id="A3.T4.1.1.2.2.5" class="ltx_td ltx_align_center ltx_th ltx_th_column" style="padding-top:2.5pt;padding-bottom:2.5pt;">RETRO</th>
<th id="A3.T4.1.1.2.2.6" class="ltx_td ltx_align_center ltx_th ltx_th_column" style="padding-top:2.5pt;padding-bottom:2.5pt;">I-RETRO</th>
<th id="A3.T4.1.1.2.2.7" class="ltx_td ltx_align_center ltx_th ltx_th_column" style="padding-top:2.5pt;padding-bottom:2.5pt;">RETRO</th>
<th id="A3.T4.1.1.2.2.8" class="ltx_td ltx_align_center ltx_th ltx_th_column" style="padding-top:2.5pt;padding-bottom:2.5pt;">I-RETRO</th>
<th id="A3.T4.1.1.2.2.9" class="ltx_td ltx_align_center ltx_th ltx_th_column" style="padding-top:2.5pt;padding-bottom:2.5pt;">RETRO</th>
<th id="A3.T4.1.1.2.2.10" class="ltx_td ltx_align_center ltx_th ltx_th_column" style="padding-top:2.5pt;padding-bottom:2.5pt;">I-RETRO</th>
<th id="A3.T4.1.1.2.2.11" class="ltx_td ltx_align_center ltx_th ltx_th_column" style="padding-top:2.5pt;padding-bottom:2.5pt;">RETRO</th>
<th id="A3.T4.1.1.2.2.12" class="ltx_td ltx_align_center ltx_th ltx_th_column" style="padding-top:2.5pt;padding-bottom:2.5pt;">I-RETRO</th>
<th id="A3.T4.1.1.2.2.13" class="ltx_td ltx_align_center ltx_th ltx_th_column" style="padding-top:2.5pt;padding-bottom:2.5pt;">RETRO</th>
<th id="A3.T4.1.1.2.2.14" class="ltx_td ltx_align_center ltx_th ltx_th_column" style="padding-top:2.5pt;padding-bottom:2.5pt;">I-RETRO</th>
<th id="A3.T4.1.1.2.2.15" class="ltx_td ltx_align_center ltx_th ltx_th_column" style="padding-top:2.5pt;padding-bottom:2.5pt;">RETRO</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A3.T4.1.1.3.1" class="ltx_tr">
<th id="A3.T4.1.1.3.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t" style="padding-top:2.5pt;padding-bottom:2.5pt;">Zero-shot</th>
<td id="A3.T4.1.1.3.1.2" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:2.5pt;padding-bottom:2.5pt;"><span id="A3.T4.1.1.3.1.2.1" class="ltx_text ltx_font_bold">30.39</span></td>
<td id="A3.T4.1.1.3.1.3" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:2.5pt;padding-bottom:2.5pt;">24.11</td>
<td id="A3.T4.1.1.3.1.4" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:2.5pt;padding-bottom:2.5pt;"><span id="A3.T4.1.1.3.1.4.1" class="ltx_text ltx_font_bold">53.25</span></td>
<td id="A3.T4.1.1.3.1.5" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:2.5pt;padding-bottom:2.5pt;">52.17</td>
<td id="A3.T4.1.1.3.1.6" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:2.5pt;padding-bottom:2.5pt;"><span id="A3.T4.1.1.3.1.6.1" class="ltx_text ltx_font_bold">12.23</span></td>
<td id="A3.T4.1.1.3.1.7" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:2.5pt;padding-bottom:2.5pt;">6.39</td>
<td id="A3.T4.1.1.3.1.8" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:2.5pt;padding-bottom:2.5pt;"><span id="A3.T4.1.1.3.1.8.1" class="ltx_text ltx_font_bold">14.72</span></td>
<td id="A3.T4.1.1.3.1.9" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:2.5pt;padding-bottom:2.5pt;">10.04</td>
<td id="A3.T4.1.1.3.1.10" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:2.5pt;padding-bottom:2.5pt;"><span id="A3.T4.1.1.3.1.10.1" class="ltx_text ltx_font_bold">3.35</span></td>
<td id="A3.T4.1.1.3.1.11" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:2.5pt;padding-bottom:2.5pt;">0.14</td>
<td id="A3.T4.1.1.3.1.12" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:2.5pt;padding-bottom:2.5pt;"><span id="A3.T4.1.1.3.1.12.1" class="ltx_text ltx_font_bold">11.04</span></td>
<td id="A3.T4.1.1.3.1.13" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:2.5pt;padding-bottom:2.5pt;">9.15</td>
<td id="A3.T4.1.1.3.1.14" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:2.5pt;padding-bottom:2.5pt;"><span id="A3.T4.1.1.3.1.14.1" class="ltx_text ltx_font_bold">20.83</span></td>
<td id="A3.T4.1.1.3.1.15" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:2.5pt;padding-bottom:2.5pt;">17.00</td>
</tr>
<tr id="A3.T4.1.1.4.2" class="ltx_tr">
<th id="A3.T4.1.1.4.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row" style="padding-top:2.5pt;padding-bottom:2.5pt;">P-tuning</th>
<td id="A3.T4.1.1.4.2.2" class="ltx_td ltx_align_center" style="padding-top:2.5pt;padding-bottom:2.5pt;">19.55</td>
<td id="A3.T4.1.1.4.2.3" class="ltx_td ltx_align_center" style="padding-top:2.5pt;padding-bottom:2.5pt;"><span id="A3.T4.1.1.4.2.3.1" class="ltx_text ltx_font_bold">24.18</span></td>
<td id="A3.T4.1.1.4.2.4" class="ltx_td ltx_align_center" style="padding-top:2.5pt;padding-bottom:2.5pt;">41.95</td>
<td id="A3.T4.1.1.4.2.5" class="ltx_td ltx_align_center" style="padding-top:2.5pt;padding-bottom:2.5pt;"><span id="A3.T4.1.1.4.2.5.1" class="ltx_text ltx_font_bold">67.94</span></td>
<td id="A3.T4.1.1.4.2.6" class="ltx_td ltx_align_center" style="padding-top:2.5pt;padding-bottom:2.5pt;"><span id="A3.T4.1.1.4.2.6.1" class="ltx_text ltx_font_bold">20.17</span></td>
<td id="A3.T4.1.1.4.2.7" class="ltx_td ltx_align_center" style="padding-top:2.5pt;padding-bottom:2.5pt;">19.02</td>
<td id="A3.T4.1.1.4.2.8" class="ltx_td ltx_align_center" style="padding-top:2.5pt;padding-bottom:2.5pt;">11.34</td>
<td id="A3.T4.1.1.4.2.9" class="ltx_td ltx_align_center" style="padding-top:2.5pt;padding-bottom:2.5pt;"><span id="A3.T4.1.1.4.2.9.1" class="ltx_text ltx_font_bold">24.20</span></td>
<td id="A3.T4.1.1.4.2.10" class="ltx_td ltx_align_center" style="padding-top:2.5pt;padding-bottom:2.5pt;">16.68</td>
<td id="A3.T4.1.1.4.2.11" class="ltx_td ltx_align_center" style="padding-top:2.5pt;padding-bottom:2.5pt;"><span id="A3.T4.1.1.4.2.11.1" class="ltx_text ltx_font_bold">31.93</span></td>
<td id="A3.T4.1.1.4.2.12" class="ltx_td ltx_align_center" style="padding-top:2.5pt;padding-bottom:2.5pt;"><span id="A3.T4.1.1.4.2.12.1" class="ltx_text ltx_font_bold">15.88</span></td>
<td id="A3.T4.1.1.4.2.13" class="ltx_td ltx_align_center" style="padding-top:2.5pt;padding-bottom:2.5pt;">15.40</td>
<td id="A3.T4.1.1.4.2.14" class="ltx_td ltx_align_center" style="padding-top:2.5pt;padding-bottom:2.5pt;">20.75</td>
<td id="A3.T4.1.1.4.2.15" class="ltx_td ltx_align_center" style="padding-top:2.5pt;padding-bottom:2.5pt;"><span id="A3.T4.1.1.4.2.15.1" class="ltx_text ltx_font_bold">30.44</span></td>
</tr>
<tr id="A3.T4.1.1.5.3" class="ltx_tr">
<th id="A3.T4.1.1.5.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row" style="padding-top:2.5pt;padding-bottom:2.5pt;">Adapter</th>
<td id="A3.T4.1.1.5.3.2" class="ltx_td ltx_align_center" style="padding-top:2.5pt;padding-bottom:2.5pt;">18.81</td>
<td id="A3.T4.1.1.5.3.3" class="ltx_td ltx_align_center" style="padding-top:2.5pt;padding-bottom:2.5pt;"><span id="A3.T4.1.1.5.3.3.1" class="ltx_text ltx_font_bold">43.01</span></td>
<td id="A3.T4.1.1.5.3.4" class="ltx_td ltx_align_center" style="padding-top:2.5pt;padding-bottom:2.5pt;">38.83</td>
<td id="A3.T4.1.1.5.3.5" class="ltx_td ltx_align_center" style="padding-top:2.5pt;padding-bottom:2.5pt;"><span id="A3.T4.1.1.5.3.5.1" class="ltx_text ltx_font_bold">71.35</span></td>
<td id="A3.T4.1.1.5.3.6" class="ltx_td ltx_align_center" style="padding-top:2.5pt;padding-bottom:2.5pt;">20.30</td>
<td id="A3.T4.1.1.5.3.7" class="ltx_td ltx_align_center" style="padding-top:2.5pt;padding-bottom:2.5pt;"><span id="A3.T4.1.1.5.3.7.1" class="ltx_text ltx_font_bold">20.51</span></td>
<td id="A3.T4.1.1.5.3.8" class="ltx_td ltx_align_center" style="padding-top:2.5pt;padding-bottom:2.5pt;">12.64</td>
<td id="A3.T4.1.1.5.3.9" class="ltx_td ltx_align_center" style="padding-top:2.5pt;padding-bottom:2.5pt;"><span id="A3.T4.1.1.5.3.9.1" class="ltx_text ltx_font_bold">25.90</span></td>
<td id="A3.T4.1.1.5.3.10" class="ltx_td ltx_align_center" style="padding-top:2.5pt;padding-bottom:2.5pt;">29.87</td>
<td id="A3.T4.1.1.5.3.11" class="ltx_td ltx_align_center" style="padding-top:2.5pt;padding-bottom:2.5pt;"><span id="A3.T4.1.1.5.3.11.1" class="ltx_text ltx_font_bold">31.98</span></td>
<td id="A3.T4.1.1.5.3.12" class="ltx_td ltx_align_center" style="padding-top:2.5pt;padding-bottom:2.5pt;">15.06</td>
<td id="A3.T4.1.1.5.3.13" class="ltx_td ltx_align_center" style="padding-top:2.5pt;padding-bottom:2.5pt;"><span id="A3.T4.1.1.5.3.13.1" class="ltx_text ltx_font_bold">15.16</span></td>
<td id="A3.T4.1.1.5.3.14" class="ltx_td ltx_align_center" style="padding-top:2.5pt;padding-bottom:2.5pt;">22.58</td>
<td id="A3.T4.1.1.5.3.15" class="ltx_td ltx_align_center" style="padding-top:2.5pt;padding-bottom:2.5pt;"><span id="A3.T4.1.1.5.3.15.1" class="ltx_text ltx_font_bold">34.65</span></td>
</tr>
<tr id="A3.T4.1.1.6.4" class="ltx_tr">
<th id="A3.T4.1.1.6.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb" style="padding-top:2.5pt;padding-bottom:2.5pt;">LoRA</th>
<td id="A3.T4.1.1.6.4.2" class="ltx_td ltx_align_center ltx_border_bb" style="padding-top:2.5pt;padding-bottom:2.5pt;">21.56</td>
<td id="A3.T4.1.1.6.4.3" class="ltx_td ltx_align_center ltx_border_bb" style="padding-top:2.5pt;padding-bottom:2.5pt;"><span id="A3.T4.1.1.6.4.3.1" class="ltx_text ltx_font_bold">42.11</span></td>
<td id="A3.T4.1.1.6.4.4" class="ltx_td ltx_align_center ltx_border_bb" style="padding-top:2.5pt;padding-bottom:2.5pt;">47.89</td>
<td id="A3.T4.1.1.6.4.5" class="ltx_td ltx_align_center ltx_border_bb" style="padding-top:2.5pt;padding-bottom:2.5pt;"><span id="A3.T4.1.1.6.4.5.1" class="ltx_text ltx_font_bold">70.75</span></td>
<td id="A3.T4.1.1.6.4.6" class="ltx_td ltx_align_center ltx_border_bb" style="padding-top:2.5pt;padding-bottom:2.5pt;">19.23</td>
<td id="A3.T4.1.1.6.4.7" class="ltx_td ltx_align_center ltx_border_bb" style="padding-top:2.5pt;padding-bottom:2.5pt;"><span id="A3.T4.1.1.6.4.7.1" class="ltx_text ltx_font_bold">20.42</span></td>
<td id="A3.T4.1.1.6.4.8" class="ltx_td ltx_align_center ltx_border_bb" style="padding-top:2.5pt;padding-bottom:2.5pt;">17.45</td>
<td id="A3.T4.1.1.6.4.9" class="ltx_td ltx_align_center ltx_border_bb" style="padding-top:2.5pt;padding-bottom:2.5pt;"><span id="A3.T4.1.1.6.4.9.1" class="ltx_text ltx_font_bold">24.48</span></td>
<td id="A3.T4.1.1.6.4.10" class="ltx_td ltx_align_center ltx_border_bb" style="padding-top:2.5pt;padding-bottom:2.5pt;">24.21</td>
<td id="A3.T4.1.1.6.4.11" class="ltx_td ltx_align_center ltx_border_bb" style="padding-top:2.5pt;padding-bottom:2.5pt;"><span id="A3.T4.1.1.6.4.11.1" class="ltx_text ltx_font_bold">32.41</span></td>
<td id="A3.T4.1.1.6.4.12" class="ltx_td ltx_align_center ltx_border_bb" style="padding-top:2.5pt;padding-bottom:2.5pt;"><span id="A3.T4.1.1.6.4.12.1" class="ltx_text ltx_font_bold">15.40</span></td>
<td id="A3.T4.1.1.6.4.13" class="ltx_td ltx_align_center ltx_border_bb" style="padding-top:2.5pt;padding-bottom:2.5pt;">15.30</td>
<td id="A3.T4.1.1.6.4.14" class="ltx_td ltx_align_center ltx_border_bb" style="padding-top:2.5pt;padding-bottom:2.5pt;">24.29</td>
<td id="A3.T4.1.1.6.4.15" class="ltx_td ltx_align_center ltx_border_bb" style="padding-top:2.5pt;padding-bottom:2.5pt;"><span id="A3.T4.1.1.6.4.15.1" class="ltx_text ltx_font_bold">34.24</span></td>
</tr>
</tbody>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 4: </span>Full results with Instruction-tuned RETRO. <span id="A3.T4.3.1" class="ltx_text ltx_font_bold">Bold</span> indicates the better result in each head-to-head comparison.</figcaption>
</figure>
</section>
</article>
</div>
<div class="ar5iv-footer"><a href="/html/2407.04527" class="ar5iv-nav-button ar5iv-nav-button-prev">◄</a>
    <a class="ar5iv-home-button" href="/"><img height="40" alt="ar5iv homepage" src="/assets/ar5iv.png"></a>
    <a href="/feeling_lucky" class="ar5iv-text-button">Feeling<br>lucky?</a>
    <a href="/log/2407.04528" class="ar5iv-text-button ar5iv-severity-warning">Conversion<br>report</a>
    <a class="ar5iv-text-button" target="_blank" href="https://github.com/dginev/ar5iv/issues/new?template=improve-article--arxiv-id-.md&title=Improve+article+2407.04528">Report<br>an issue</a>
    <a href="https://arxiv.org/abs/2407.04528" class="ar5iv-text-button arxiv-ui-theme">View&nbsp;original<br>on&nbsp;arXiv</a><a href="/html/2407.04529" class="ar5iv-nav-button ar5iv-nav-button-next">►</a>
</div><footer class="ltx_page_footer">
<a class="ar5iv-toggle-color-scheme" href="javascript:toggleColorScheme()" title="Toggle ar5iv color scheme"><span class="color-scheme-icon"></span></a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/license" target="_blank">Copyright</a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/policies/privacy_policy" target="_blank">Privacy Policy</a>

<div class="ltx_page_logo">Generated  on Mon Aug  5 16:30:32 2024 by <a target="_blank" href="http://dlmf.nist.gov/LaTeXML/" class="ltx_LaTeXML_logo"><span style="letter-spacing:-0.2em; margin-right:0.1em;">L<span class="ltx_font_smallcaps" style="position:relative; bottom:2.2pt;">a</span>T<span class="ltx_font_smallcaps" style="font-size:120%;position:relative; bottom:-0.2ex;">e</span></span><span style="font-size:90%; position:relative; bottom:-0.2ex;">XML</span><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAsAAAAOCAYAAAD5YeaVAAAAAXNSR0IArs4c6QAAAAZiS0dEAP8A/wD/oL2nkwAAAAlwSFlzAAALEwAACxMBAJqcGAAAAAd0SU1FB9wKExQZLWTEaOUAAAAddEVYdENvbW1lbnQAQ3JlYXRlZCB3aXRoIFRoZSBHSU1Q72QlbgAAAdpJREFUKM9tkL+L2nAARz9fPZNCKFapUn8kyI0e4iRHSR1Kb8ng0lJw6FYHFwv2LwhOpcWxTjeUunYqOmqd6hEoRDhtDWdA8ApRYsSUCDHNt5ul13vz4w0vWCgUnnEc975arX6ORqN3VqtVZbfbTQC4uEHANM3jSqXymFI6yWazP2KxWAXAL9zCUa1Wy2tXVxheKA9YNoR8Pt+aTqe4FVVVvz05O6MBhqUIBGk8Hn8HAOVy+T+XLJfLS4ZhTiRJgqIoVBRFIoric47jPnmeB1mW/9rr9ZpSSn3Lsmir1fJZlqWlUonKsvwWwD8ymc/nXwVBeLjf7xEKhdBut9Hr9WgmkyGEkJwsy5eHG5vN5g0AKIoCAEgkEkin0wQAfN9/cXPdheu6P33fBwB4ngcAcByHJpPJl+fn54mD3Gg0NrquXxeLRQAAwzAYj8cwTZPwPH9/sVg8PXweDAauqqr2cDjEer1GJBLBZDJBs9mE4zjwfZ85lAGg2+06hmGgXq+j3+/DsixYlgVN03a9Xu8jgCNCyIegIAgx13Vfd7vdu+FweG8YRkjXdWy329+dTgeSJD3ieZ7RNO0VAXAPwDEAO5VKndi2fWrb9jWl9Esul6PZbDY9Go1OZ7PZ9z/lyuD3OozU2wAAAABJRU5ErkJggg==" alt="Mascot Sammy"></a>
</div></footer>
</div>

    <script>
      var canMathML = typeof(MathMLElement) == "function";
      if (!canMathML) {
        var body = document.querySelector("body");
        body.firstElementChild.setAttribute('style', 'opacity: 0;');
        var loading = document.createElement("div");
        loading.setAttribute("id", "mathjax-loading-spinner");
        var message = document.createElement("div");
        message.setAttribute("id", "mathjax-loading-message");
        message.innerText = "Typesetting Equations...";
        body.prepend(loading);
        body.prepend(message);

        var el = document.createElement("script");
        el.src = "https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js";
        document.querySelector("head").appendChild(el);

        window.MathJax = {
          startup: {
            pageReady: () => {
              return MathJax.startup.defaultPageReady().then(() => {
                body.removeChild(loading);
                body.removeChild(message);
                body.firstElementChild.removeAttribute('style');
              }); } } };
      }
    </script>
    <script>
    // Auxiliary function, building the preview feature when
    // an inline citation is clicked
    function clicked_cite(e) {
      e.preventDefault();
      let cite = this.closest('.ltx_cite');
      let next = cite.nextSibling;
      if (next && next.nodeType == Node.ELEMENT_NODE && next.getAttribute('class') == "ar5iv-bibitem-preview") {
        next.remove();
        return; }
      // Before adding a preview modal,
      // cleanup older previews, in case they're still open
      document.querySelectorAll('span.ar5iv-bibitem-preview').forEach(function(node) {
        node.remove();
      })

      // Create the preview
      preview = document.createElement('span');
      preview.setAttribute('class','ar5iv-bibitem-preview');
      let target = document.getElementById(this.getAttribute('href').slice(1));
      target.childNodes.forEach(function (child) {
        preview.append(child.cloneNode(true));
      });
      let close_x = document.createElement('button');
      close_x.setAttribute("aria-label","Close modal for bibliography item preview");
      close_x.textContent = "×";
      close_x.setAttribute('class', 'ar5iv-button-close-preview');
      close_x.setAttribute('onclick','this.parentNode.remove()');
      preview.append(close_x);
      preview.querySelectorAll('.ltx_tag_bibitem').forEach(function(node) {
        node.remove();
      });
      cite.parentNode.insertBefore(preview, cite.nextSibling);
      return;
    }
    // Global Document initialization:
    // - assign the preview feature to all inline citation links
    document.querySelectorAll(".ltx_cite .ltx_ref").forEach(function (link) {
      link.addEventListener("click", clicked_cite);
    });
    </script>
    </body>
</html>

2407.04528
Parameter-Efficient Fine-Tuning (PEFT)  and Retrieval-Augmented Generation (RAG)  have become popular methods for adapting large language models while minimizing compute requirements. In this paper, we apply PEFT methods (P-tuning, Adapters, and LoRA)  to a modified Retrieval-Enhanced Transformer (RETRO)  and a baseline GPT model across several sizes, ranging from 823 million to 48 billion parameters. We show that RETRO models outperform GPT models in zero-shot settings due to their unique pre-training process but GPT models have higher performance potential with PEFT. Additionally, our study indicates that 8B parameter models strike an optimal balance between cost and performance and P-tuning lags behind other PEFT techniques. We further provide a comparative analysis of between applying PEFT to an Instruction-tuned RETRO model and base RETRO model. This work presents the first comprehensive comparison of various PEFT methods integrated with RAG, applied to both GPT and RETRO models, highlighting their relative performance.
abstract
GPT vs RETRO: Exploring the Intersection of Retrieval and Parameter-Efficient Fine-Tuning
Abstract
2407.04528
1IntroductionPre-trained large language models have made a demonstrable impact across applications in academia and industry. Many use cases, however, require LLMs adapted to specific tasks and unique information but lack the resources for extensive retraining. To address this, Parameter-Efficient Fine-Tuning (PEFT) and Retrieval-Augmented Generation (RAG) have become popular methods due to their effectiveness and efficiency, inspiring new lines of research.Figure 1:Average GPT vs RETRO scores of six datasets across model sizes of 823M to 48B parameters.PEFT has been proven to be a comparable substitute to Supervised Fine-Tuning (SFT)  by achieving competitive performance at a fraction of the number of updated parameters. In this paper we select P-tuning, Adapter modulesand Low-Rank Adaptation (LoRA) as representative PEFT methods. P-tuning involves training continuous prompt embeddings to guide output for specific tasks without modifying base model parameters. Adapters operate by training fully connected layers inserted throughout the base model while keeping the remaining parameters frozen. LoRA employs a similar strategy but further decomposes the inserted layers into low-rank matrices, enhancing efficiency.Retrieval-augmented generation (RAG)  improves model quality by incorporating external knowledge through mechanisms like BM-25 or TF-IDF, online web search, or trained dense retriever models. Any LLM can be transformed into a retrieval-augmented model by concatenating retrieved sources with the input query, provided it fits within the model’s context window.found that retrieval significantly improves GPT model quality on long context tasks, reducing the "lost in the middle" effectand offering inherent efficiency benefits.Alternatively, there exist multiple worksthat have integrated retrieval as part of model pretraining or fine-tuning to notable success when compared to typical GPT models despite being a much lesser explored domain. RETROis of particular interest due to its unique approach of incorporating a retrieval module directly into the transformer architecture via a chunked-cross attention mechanism and ability to scale to trillions of tokens resulting in reduced perplexity. Subsequently,showed that RETRO at sizes up to 9.5 billion parameters largely outperforms GPT on specific knowledge-intensive tasks. Furthermore,illustrated that when scaled up to 48 billion parameters and instruction-tuned, RETRO performed better than equivalent GPT models on several question answering, reading comprehension and summarization tasks.In this paper we continue the exploration of RETRO versus GPT through the lens of parameter efficient finetuning. We apply P-tuning, Adapter modules and LoRA to multiple tasks with retrieval for both RETRO and GPT models. To our knowledge, this paper provides the first in-depth comparison of various Parameter Efficient Fine-Tuning integrated with Retrieval-Augmented Generation, uniquely applied to both GPT and RETRO models.
introduction
GPT vs RETRO: Exploring the Intersection of Retrieval and Parameter-Efficient Fine-Tuning
1Introduction
2407.04528
2Related WorkIn this section we focus on recent work that combine finetuning with retrieval. A comprehensive surveysynthetized multiple comparative studies on PEFT and RAG, underscoring the potential benefits of combining these approaches as a promising direction for future investigation. There are multiple works that provide methods to combine RAG with fine-tuning to improve accuracy. Multiple studies have explored the comparison between fine-tuning and retrieval.andreported improved accuracy using RAG over fine-tuning GPT models, while also noting suboptimal results when combining the two methods.demonstrated improved outcomes by integrating both approaches for specific agriculture and geography tasks. Additionally,compared the efficacy of these methods, including full and QLoRA fine-tuning, in low-frequency entity question-answering tasks.
These studies collectively suggest the need for comprehensive investigation into multiple PEFT techniques combined with RAG and maintain retrieval pretrained LLMs with PEFT to be unexplored, thereby motivating our research.NQTQANQAQASPERQUALITYQMSUMAVERAGEGPTRETROGPTRETROGPTRETROGPTRETROGPTRETROGPTRETROGPTRETROExtra SmallZero-shot2.958.289.9919.267.074.879.0010.790.380.489.837.786.548.58P-tuning24.747.6063.6324.6116.746.6924.5411.9424.5917.4518.2513.6328.7513.65Adapter38.4823.6967.9959.6017.7615.4223.5220.9624.2625.9319.7414.4231.9626.67LoRA37.0922.1367.3159.0218.0815.8123.5419.8524.9325.6519.2713.7931.7026.04SmallZero-shot11.6518.7729.8838.427.077.1212.3112.420.001.0112.359.2512.2114.50P-tuning39.2718.5870.3161.1319.9815.1324.7520.3422.7724.1118.7614.6132.6425.65Adapter42.2923.6873.2164.9121.4018.1027.2920.5524.9325.0720.1715.0334.8827.89LoRA39.2728.0672.3464.5920.9817.9024.8321.2825.7924.6920.3114.4633.9228.50MediumZero-shot23.6724.1151.0052.178.906.399.0110.041.440.1411.289.1517.5517.00P-tuning45.5224.1877.0067.9424.5019.0233.3124.2032.7431.9320.3715.4038.9130.44Adapter46.7143.0178.0571.3524.3020.5132.5325.9040.8431.9820.0315.6140.4134.65LoRA46.8142.1178.2670.7525.1720.4231.8424.4841.5632.4121.4715.3040.8534.24Fine-tuning41.3429.7979.8268.8422.3319.3749.6723.5337.0133.5621.9515.2942.0231.73LargeZero-shot25.3731.4348.6860.3013.927.988.7310.522.971.876.309.3317.6620.24P-tuning45.2015.7878.3373.2225.2121.5834.2424.5047.6539.9320.0715.0041.7831.67Adapter47.4844.4379.6873.5726.3722.0332.1226.0946.7438.0620.8115.2242.2036.57LoRA47.3344.4879.7973.6325.8521.4932.2525.2142.6239.3121.6715.0241.5836.53Extra LargeZero-shot26.9733.4944.7162.8711.8910.0711.5813.383.070.967.659.9917.6521.79P-tuning47.2724.5380.2774.3827.0922.4834.0824.9357.1938.0621.1715.5344.5133.32Adapter49.6846.4181.6475.1026.9422.2433.9426.3854.6542.6221.1915.7146.8238.08LoRA49.2144.5381.8774.9227.3122.1631.9827.4949.1939.6522.7715.7343.7237.41Table 1:A comprehensive comparison between GPT vs RETRO on six datasets.Boldindicates the better result in each head-to-head comparison.
body
GPT vs RETRO: Exploring the Intersection of Retrieval and Parameter-Efficient Fine-Tuning
2Related Work
2407.04528
3Experimental Setup3.1DatasetsTo cover several task categories, we use six datasets suited to benefit from retrieval and finetuning. We selectNatural Questions(NQ) ,TriviaQA(TQA) ,NarrativeQA(NQA) andQasperfor document question answering,QuALITYfor multiple-choice question answering, andQMSumfor query-based summarization (see detail statistics in AppendixB) . Each of these datasets contain necessary external knowledge that must be filtered via retrieval and response behaviour that encourages finetuning. Following the official metrics, we use F1 score for evaluating document QA, exact match for mutliple-choice QA and the geometric mean of ROUGE-1/2/Lfor summarization.3.2ModelsIn order to undertand the effect of model scales, we use base GPT models of sizes 823M (Extra Small) , 2.25B (Small) , 8.5B (Medium) , 22B (Large) , and 43B (Extra Large) , as introduced in, which were pretrained on a massive dataset of 1.2 trillion tokens. We employ the corresponding RETRO models from the same work as the foundation for our retrieval pretrained LLM experiments. Notably, the RETRO architecture features an encoder that extracts neighbors from an external database, which increases the total model size to 877M, 2.47B, 9.5B, 24B, and 48B, respectively.found ablating the encoder after pretraining led to comparable results. In our paper we include it so that adapter modules and LoRA
layers are added throughout decoder and encoder components. For more on the base models, we refer readers to the original work.3.3RetrievalWe followto use Dragon+as a retriever. Dragon+ is a dual encoder model that consists of a query encoder and a context encoder.
We first chunk each context document with 100 words, and then encode
both the questions and all chunks independently with corresponding encoders. The most relevant
5 chunks, ranked by the dot product of the question embedding and chunk embedding, are retrieved as neighbors.
For GPT models, they are concatenated together (following the left to right order from the most relevant to least relevant)  as the
context of the prompt for generation. For RETRO models, they interact with the question during generation through chunked cross-attention.3.4Parameter Efficient Fine-TuningWe implement P-tuning in RETRO akin to GPT. Virtual tokens are added to the beginning of the decoder. Based on the design of chunked cross-attention, left padding is added to ensure the length of input (virtual tokens + context + question)  is a multiple of chunk size. Adapter and LoRA layers are inserted within the transformer blocks in both RETRO and GPT at the feedforward and attention layers. In RETRO they are also inserted in the transformer encoder ingesting the retrieved neighbors. We provide additional hyperparameter tuning, resource utilization and prompt template details in AppendixA.
body
GPT vs RETRO: Exploring the Intersection of Retrieval and Parameter-Efficient Fine-Tuning
3Experimental Setup
2407.04528
4Results4.1Main ResultsTable1shows the comprehensive comparison between GPT and RETRO models across five sizes and six datasets. From this table we observe:(1)  RETRO is generally better than GPT at zero-shot settings. This superiority stems from its unique pre-training approach. By learning to extract salient information from retrieved text and integrate it into its generation process, RETRO develops the capability to harness relevant contextual knowledge, ultimately leading to its good zero-shot performance. In contrast, GPT relies on an auto-regressive loss during pre-training, focusing on accurately predicting next tokens without the benefit of external retrievals. As a result, GPT’s ability to learn context-aware question-answering is limited to the presence of relevant data within the pre-training corpus, resulting in significantly less targeted training compared to RETRO.(2)  Both RETRO and GPT models exhibit saturation points on these datasets around the 8B mark, with a similar pattern emerging between the two models, albeit with RETRO performing less well. This can be seen in Figure1and suggests that, for a specific task, a medium-sized PEFT model strikes the optimal balance between cost and performance, making it a sweet spot for many applications.(3)  For GPT models, P-tuning performs worse at smaller models (<=8B) , but it performs as good or better than LoRA and Adapter at the large model sizes. This difference is visualized in Figure2and Figure3(AppendixC) . However, for RETRO models, P-tuning generally under performs the other PEFT methods across all model sizes.(4)  The performance ceiling for PEFT-tuned models is significantly higher for GPT compared to RETRO as seen in Figure4(AppendixC) . For example, using medium-sized models, the average score of LoRA with GPT is 40.85, while with RETRO it is 34.24. This disparity suggests that GPT has more room for improvement with PEFT tuning. This phenomenon can also be possibly explained by the two different pre-training strategies. Since GPT has been less targeted on retrieval-augmented generation during pre-training, it opens a larger room for improvement during fine-tuning.(5)  We also conduct full fine-tuning on medium-sized GPT and RETRO models. For GPT model, we find that full fine-tuning achieves slightly better performance than PEFT on average, with a notable exception on the QASPER dataset, where the difference is more pronounced and full fine-tuning yields significantly better results.
In contrast, for RETRO model, full fine-tuning generally performs worse than PEFT, demonstrating the effectiveness of PEFT. These findings are consistent with the observations of.Figure 2:Comparison of Extra Large GPT and RETRO results averaged across 6 datasets.4.2Comparing to Instruction-tuned RETROInstruction tuning post retrieval-augmented pre-traininghas been demonstrated to improve zero-shot performance on RETRO models. A natural thought is that whether Instruction-tuned RETRO (I-RETRO)  serve as a better foundation for applying PEFT compared to the base RETRO. To investigate this, we additionally apply PEFT to a medium-sized I-RETRO model and show overall results in Table2and more granular results in Table4(AppendixC) . Our findings reveal that while I-RETRO exhibits improved performance in the zero-shot setting, it has limited scope for further improvement using PEFT. Even with substantial hyperparameter tuning, the average scores across six datasets, using each of the three PEFT methods, demonstrate an approximately 10% gap between I-RETRO and base RETRO. We hypothesize that conceptually both models should be tunable to similar performance but will leave that exploration to future work.Average QAQUALITYQMSUMAverageI-RETRORETROI-RETRORETROI-RETRORETROI-RETRORETROZero-shot27.6523.793.350.1411.049.1520.8317.00P-tuning23.2547.1816.6831.9315.8815.4020.7530.44Adapter22.6452.7529.8731.9815.0615.1622.5834.65LoRA26.5352.8024.2132.4115.4015.3024.2934.24Table 2:Instruction-tuned RETRO evaluation results.
body
GPT vs RETRO: Exploring the Intersection of Retrieval and Parameter-Efficient Fine-Tuning
4Results
2407.04528
5ConclusionThis study explores Parameter-Efficient Fine-Tuning (PEFT)  methods applied to Retrieval-Augmented Generation (RAG)  models, comparing GPT and RETRO architectures. RETRO generally outperforms GPT in zero-shot settings due to their pre-training process that integrates external retrieval, enhancing contextual understanding. However, GPT models show a higher performance potential with PEFT, indicating more room for improvement during fine-tuning. Both RETRO and GPT models perform optimally around the 8B parameter mark, balancing cost and performance. While P-tuning is effective in larger models, it lags behind other methods in smaller models, particularly for RETRO. Applying PEFT to Instruction-tuned RETRO yields limited improvement compared to base RETRO, suggesting a saturation point in leveraging pre-training and fine-tuning benefits. Our comprehensive analysis offers valuable insights for optimizing large language models with PEFT and RAG to the community.
body
GPT vs RETRO: Exploring the Intersection of Retrieval and Parameter-Efficient Fine-Tuning
5Conclusion
2407.04528
LimitationsDue to the breadth of experiments covered in this work we had to prioritze certain experiments over others. This resulted in us using only the medium sized GPT and RETRO models for additional finetuning and Instruction tuning experiments. We believe these results generalize to the other model sizes but leave that to be validated in future work.
body
GPT vs RETRO: Exploring the Intersection of Retrieval and Parameter-Efficient Fine-Tuning
Limitations
2407.04528
Potential RisksThe environmental impact associated with training and fine-tuning large models is not negligable as it involves substantial computational resources and energy consumption. While PEFT aims to alleviate this by reducing the number of tunable parameters, works like ours still require significant compute to distinguish which methods are more promising.
conclusion
GPT vs RETRO: Exploring the Intersection of Retrieval and Parameter-Efficient Fine-Tuning
Potential Risks