<!DOCTYPE html><html lang="en">
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<title>[2404.19553] Extending Llama-3’s Context Ten-Fold Overnight</title><meta property="og:description" content="We extend the context length of Llama-3-8B-Instruct from 8K to 80K via QLoRA fine-tuning111The model is noted as Llama-3-8B-Instruct-80K-QLoRA given its max context length during fine-tuning. However, users could apply…">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="twitter:card" content="summary">
<meta name="twitter:title" content="Extending Llama-3’s Context Ten-Fold Overnight">
<meta name="twitter:image:src" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta name="twitter:image:alt" content="ar5iv logo">
<meta property="og:title" content="Extending Llama-3’s Context Ten-Fold Overnight">
<meta property="og:site_name" content="ar5iv">
<meta property="og:image" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta property="og:type" content="article">
<meta property="og:url" content="https://ar5iv.labs.arxiv.org/html/2404.19553">

<!--Generated on Sun May  5 17:55:32 2024 by LaTeXML (version 0.8.8) http://dlmf.nist.gov/LaTeXML/.-->
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">

<script>
  function detectColorScheme(){
    var theme="light";
    var current_theme = localStorage.getItem("ar5iv_theme");
    if(current_theme){
      if(current_theme == "dark"){
        theme = "dark";
      } }
    else if(!window.matchMedia) { return false; }
    else if(window.matchMedia("(prefers-color-scheme: dark)").matches) {
      theme = "dark"; }
    if (theme=="dark") {
      document.documentElement.setAttribute("data-theme", "dark");
    } else {
      document.documentElement.setAttribute("data-theme", "light"); } }

  detectColorScheme();

  function toggleColorScheme(){
    var current_theme = localStorage.getItem("ar5iv_theme");
    if (current_theme) {
      if (current_theme == "light") {
        localStorage.setItem("ar5iv_theme", "dark"); }
      else {
        localStorage.setItem("ar5iv_theme", "light"); } }
    else {
        localStorage.setItem("ar5iv_theme", "dark"); }
    detectColorScheme(); }
</script>
<link media="all" rel="stylesheet" href="/assets/ar5iv-fonts.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv-site.0.2.2.css">
</head>
<body>
<div class="ltx_page_main">
<div class="ltx_page_content">
<article class="ltx_document ltx_authors_1line">
<h1 class="ltx_title ltx_title_document">Extending Llama-3’s Context Ten-Fold Overnight</h1>
<div class="ltx_authors">
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Peitian Zhang<sup id="id10.10.id1" class="ltx_sup"><span id="id10.10.id1.1" class="ltx_text ltx_font_italic">1,2</span></sup>,  Ninglu Shao<sup id="id11.11.id2" class="ltx_sup"><span id="id11.11.id2.1" class="ltx_text ltx_font_italic">1,2</span></sup>,  Zheng Liu<sup id="id12.12.id3" class="ltx_sup">1</sup>,  Shitao Xiao<sup id="id13.13.id4" class="ltx_sup">1</sup>,  <span id="id5.5.1" class="ltx_text ltx_font_bold">Hongjin Qian<sup id="id5.5.1.1" class="ltx_sup"><span id="id5.5.1.1.1" class="ltx_text ltx_font_medium ltx_font_italic">1,2</span></sup></span>,
<br class="ltx_break"><span id="id6.6.2" class="ltx_text ltx_font_bold">Qiwei Ye<sup id="id6.6.2.1" class="ltx_sup"><span id="id6.6.2.1.1" class="ltx_text ltx_font_medium">1</span></sup>,</span>   <span id="id8.8.4" class="ltx_text ltx_font_bold">Zhicheng Dou<sup id="id8.8.4.1" class="ltx_sup"><span id="id8.8.4.1.1" class="ltx_text ltx_font_medium">2</span></sup>
<br class="ltx_break"><sup id="id8.8.4.2" class="ltx_sup"><span id="id8.8.4.2.1" class="ltx_text ltx_font_medium">1</span></sup></span> Beijing Academy of Artificial Intelligence 
<br class="ltx_break"><sup id="id14.14.id5" class="ltx_sup">2</sup> Gaoling School of Artificial Intelligence, Renmin University of China 
<br class="ltx_break"><span id="id15.15.id6" class="ltx_text ltx_font_typewriter">namespace.pt@gmail.com</span>  <span id="id16.16.id7" class="ltx_text ltx_font_typewriter">zhengliu1026@gmail.com</span> 
<br class="ltx_break">
</span><span class="ltx_author_notes">Corresponding author.</span></span>
</div>

<div class="ltx_abstract">
<h6 class="ltx_title ltx_title_abstract">Abstract</h6>
<p id="id17.id1" class="ltx_p">We extend the context length of Llama-3-8B-Instruct from 8K to 80K via QLoRA fine-tuning<span id="footnote1" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">1</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">1</sup><span class="ltx_tag ltx_tag_note">1</span>The model is noted as Llama-3-8B-Instruct-80K-QLoRA given its max context length during fine-tuning. However, users could apply the model for even longer contexts via extrapolation.</span></span></span>. The entire training cycle is super efficient, which takes 8 hours on one 8xA800 (80G) GPU machine. The resulted model exhibits superior performances across a broad range of evaluation tasks, such as NIHS, topic retrieval, and long-context language understanding; meanwhile, it also well preserves the original capability over short contexts. The dramatic context extension is mainly attributed to merely 3.5K synthetic training samples generated by GPT-4 , which indicates the LLMs’ inherent (yet largely underestimated) potential to extend its original context length. In fact, the context length could be extended far beyond 80K with more computation resources. Therefore, the team will publicly release the entire resources (including data, model, data generation pipeline, training code) so as to facilitate the future research from the community: <a target="_blank" href="https://github.com/FlagOpen/FlagEmbedding" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/FlagOpen/FlagEmbedding</a>.</p>
</div>
<section id="S1" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">1 </span>Introduction</h2>

<div id="S1.p1" class="ltx_para">
<p id="S1.p1.1" class="ltx_p">Recently, considerable attention has been directed towards long-context large language models, where different approaches are adopted to establish long-context capabilities for large language models <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib4" title="" class="ltx_ref">4</a>, <a href="#bib.bib14" title="" class="ltx_ref">14</a>, <a href="#bib.bib5" title="" class="ltx_ref">5</a>, <a href="#bib.bib8" title="" class="ltx_ref">8</a>, <a href="#bib.bib9" title="" class="ltx_ref">9</a>, <a href="#bib.bib16" title="" class="ltx_ref">16</a>, <a href="#bib.bib2" title="" class="ltx_ref">2</a>]</cite>. However, most of them require significant compute and resources to accomplish.</p>
</div>
<div id="S1.p2" class="ltx_para">
<p id="S1.p2.1" class="ltx_p">In this technical report, we propose an efficient solution for entitling the long-context capabilities for LLMs, with which we extend the context length of Llama-3-8B-Instruct<span id="footnote2" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">2</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">2</sup><span class="ltx_tag ltx_tag_note">2</span><a target="_blank" href="https://llama.meta.com/llama3/" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://llama.meta.com/llama3/</a></span></span></span> from 8K to 80K. Specifically, we use GPT-4 <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib13" title="" class="ltx_ref">13</a>]</cite> to synthesize 3.5K long-context training data, covering three long-context tasks:</p>
<ol id="S1.I1" class="ltx_enumerate">
<li id="S1.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">1.</span> 
<div id="S1.I1.i1.p1" class="ltx_para">
<p id="S1.I1.i1.p1.1" class="ltx_p"><span id="S1.I1.i1.p1.1.1" class="ltx_text ltx_font_bold">Single-Detail QA</span>: the inquiry targets on one specific detail in a long context. To construct data for this task, we slice out a short segment (e.g., a chunk with less than 4096 tokens) from a long context (e.g., a book or a long paper) and prompt GPT-4 to generate multiple question-answer pairs based on this segment.</p>
</div>
</li>
<li id="S1.I1.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">2.</span> 
<div id="S1.I1.i2.p1" class="ltx_para">
<p id="S1.I1.i2.p1.1" class="ltx_p"><span id="S1.I1.i2.p1.1.1" class="ltx_text ltx_font_bold">Multi-Detail QA</span>: the inquiry requires information aggregation and reasoning over multiple details in a long context. We define two types of long context. The <span id="S1.I1.i2.p1.1.2" class="ltx_text ltx_font_bold">homogeneous</span> context contains a coherent text, such as a book or a long paper. We prompt GPT-4 to generate multiple question-answer pairs that require aggregating and analyzing information from different locations in the context. The <span id="S1.I1.i2.p1.1.3" class="ltx_text ltx_font_bold">heterogeneous</span> context consists of multiple independent texts. Notably, we perform clustering over a large corpus then extract texts from the same cluster to form each heterogeneous context. Therefore, the grouped texts share some semantic similarity. We then prompt GPT-4 to ask about the similarities/dissimilarities across these texts.</p>
</div>
</li>
<li id="S1.I1.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">3.</span> 
<div id="S1.I1.i3.p1" class="ltx_para">
<p id="S1.I1.i3.p1.1" class="ltx_p"><span id="S1.I1.i3.p1.1.1" class="ltx_text ltx_font_bold">Biography Summarization</span>: we prompt GPT-4 to write a biography for each main character in a given book.</p>
</div>
</li>
</ol>
</div>
<div id="S1.p3" class="ltx_para">
<p id="S1.p3.1" class="ltx_p">For all three tasks, the length of context is between 64K to 80K. Note that longer data can also be synthesized following the same methodology. When training, we organize the question-answer pairs for the same context in one multi-turn conversation then fine-tune the LLM to correctly answer the questions given the entire long context as input.
Following previous work<span id="footnote3" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">3</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">3</sup><span class="ltx_tag ltx_tag_note">3</span>https://www.together.ai/blog/llama-2-7b-32k</span></span></span>, we mix 5K instances randomly chosen from RedPajama <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib6" title="" class="ltx_ref">6</a>]</cite> to mitigate forgetting. We also mix LongAlpaca <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib5" title="" class="ltx_ref">5</a>]</cite> in the training set, which contains 12K instruction tuning instances with 16K length at maximum. Therefore, the entire training dataset contains 20K instances.</p>
</div>
<div id="S1.p4" class="ltx_para">
<p id="S1.p4.1" class="ltx_p">We use QLoRA <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib7" title="" class="ltx_ref">7</a>]</cite> to efficiently fine-tune the model. We apply LoRA on all Q,K,V,O projections and additionally train the embedding layer. We set LoRA rank to 32 and alpha to 16. The learning rate is 5e-5 with linear decay and no warmups. The batch size is 8. Gradient checkpointing is enabled. No parallel strategy is required thanks to the efficient implementation from Unsloth <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib1" title="" class="ltx_ref">1</a>]</cite>. We train the model for 1 epoch, which takes 8 hours to complete on a 8xA800 (80G) machine. Importantly, we expand the RoPE base from 500K to 200M in training.</p>
</div>
<div id="S1.p5" class="ltx_para">
<p id="S1.p5.1" class="ltx_p">Our contributions are highlighted as follows:</p>
<ul id="S1.I2" class="ltx_itemize">
<li id="S1.I2.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S1.I2.i1.p1" class="ltx_para">
<p id="S1.I2.i1.p1.1" class="ltx_p">We release Llama-3-8B-Instruct-80K-QLoRA, which extends the context length of Llama-3-8B-Instruct from 8K to 80K. The entire resources including the model, training data, and code are all publicly available, which may advance the field of training long-context LLMs.</p>
</div>
</li>
<li id="S1.I2.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S1.I2.i2.p1" class="ltx_para">
<p id="S1.I2.i2.p1.1" class="ltx_p">Our training recipe is simple and efficient, while the resulted model demonstrates remarkable performance on downstream long-context tasks. Further research can be made to improve our approach.</p>
</div>
</li>
</ul>
</div>
</section>
<section id="S2" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">2 </span>Experiments</h2>

<figure id="S2.F1" class="ltx_figure"><img src="/html/2404.19553/assets/x1.png" id="S2.F1.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="231" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 1: </span>The accuracy score of Llama-3-8B-Instruct-80K-QLoRA on Needle-In-A-HayStack task. The blue vertical line indicates the training length, i.e. 80K.</figcaption>
</figure>
<figure id="S2.F2" class="ltx_figure"><img src="/html/2404.19553/assets/x2.png" id="S2.F2.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="415" height="187" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 2: </span>The accuracy of Topic Retrieval task.</figcaption>
</figure>
<figure id="S2.T1" class="ltx_table">
<table id="S2.T1.1" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S2.T1.1.1.1" class="ltx_tr">
<th id="S2.T1.1.1.1.1" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S2.T1.1.1.1.1.1" class="ltx_text" style="font-size:70%;">Model</span></th>
<th id="S2.T1.1.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S2.T1.1.1.1.2.1" class="ltx_text" style="font-size:70%;">Single-Doc</span></th>
<th id="S2.T1.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S2.T1.1.1.1.3.1" class="ltx_text" style="font-size:70%;">Multi-Doc</span></th>
<th id="S2.T1.1.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S2.T1.1.1.1.4.1" class="ltx_text" style="font-size:70%;">Summ.</span></th>
<th id="S2.T1.1.1.1.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S2.T1.1.1.1.5.1" class="ltx_text" style="font-size:70%;">Few-Shot</span></th>
<th id="S2.T1.1.1.1.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S2.T1.1.1.1.6.1" class="ltx_text" style="font-size:70%;">Synthetic</span></th>
<th id="S2.T1.1.1.1.7" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S2.T1.1.1.1.7.1" class="ltx_text" style="font-size:70%;">Code</span></th>
<th id="S2.T1.1.1.1.8" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S2.T1.1.1.1.8.1" class="ltx_text" style="font-size:70%;">Avg</span></th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S2.T1.1.2.1" class="ltx_tr">
<td id="S2.T1.1.2.1.1" class="ltx_td ltx_align_center ltx_border_t"><span id="S2.T1.1.2.1.1.1" class="ltx_text" style="font-size:70%;">Llama-3-8B-Instruct</span></td>
<td id="S2.T1.1.2.1.2" class="ltx_td ltx_align_center ltx_border_t"><span id="S2.T1.1.2.1.2.1" class="ltx_text" style="font-size:70%;">37.33</span></td>
<td id="S2.T1.1.2.1.3" class="ltx_td ltx_align_center ltx_border_t"><span id="S2.T1.1.2.1.3.1" class="ltx_text" style="font-size:70%;">36.04</span></td>
<td id="S2.T1.1.2.1.4" class="ltx_td ltx_align_center ltx_border_t"><span id="S2.T1.1.2.1.4.1" class="ltx_text" style="font-size:70%;">26.83</span></td>
<td id="S2.T1.1.2.1.5" class="ltx_td ltx_align_center ltx_border_t"><span id="S2.T1.1.2.1.5.1" class="ltx_text ltx_font_bold" style="font-size:70%;">69.56</span></td>
<td id="S2.T1.1.2.1.6" class="ltx_td ltx_align_center ltx_border_t"><span id="S2.T1.1.2.1.6.1" class="ltx_text" style="font-size:70%;">37.75</span></td>
<td id="S2.T1.1.2.1.7" class="ltx_td ltx_align_center ltx_border_t"><span id="S2.T1.1.2.1.7.1" class="ltx_text" style="font-size:70%;">53.24</span></td>
<td id="S2.T1.1.2.1.8" class="ltx_td ltx_align_center ltx_border_t"><span id="S2.T1.1.2.1.8.1" class="ltx_text" style="font-size:70%;">43.20</span></td>
</tr>
<tr id="S2.T1.1.3.2" class="ltx_tr">
<td id="S2.T1.1.3.2.1" class="ltx_td ltx_align_center"><span id="S2.T1.1.3.2.1.1" class="ltx_text" style="font-size:70%;">Llama-3-8B-Instruct-262K</span></td>
<td id="S2.T1.1.3.2.2" class="ltx_td ltx_align_center"><span id="S2.T1.1.3.2.2.1" class="ltx_text" style="font-size:70%;">37.29</span></td>
<td id="S2.T1.1.3.2.3" class="ltx_td ltx_align_center"><span id="S2.T1.1.3.2.3.1" class="ltx_text" style="font-size:70%;">31.20</span></td>
<td id="S2.T1.1.3.2.4" class="ltx_td ltx_align_center"><span id="S2.T1.1.3.2.4.1" class="ltx_text" style="font-size:70%;">26.18</span></td>
<td id="S2.T1.1.3.2.5" class="ltx_td ltx_align_center"><span id="S2.T1.1.3.2.5.1" class="ltx_text" style="font-size:70%;">67.25</span></td>
<td id="S2.T1.1.3.2.6" class="ltx_td ltx_align_center"><span id="S2.T1.1.3.2.6.1" class="ltx_text" style="font-size:70%;">44.25</span></td>
<td id="S2.T1.1.3.2.7" class="ltx_td ltx_align_center"><span id="S2.T1.1.3.2.7.1" class="ltx_text ltx_font_bold" style="font-size:70%;">62.71</span></td>
<td id="S2.T1.1.3.2.8" class="ltx_td ltx_align_center"><span id="S2.T1.1.3.2.8.1" class="ltx_text" style="font-size:70%;">43.73</span></td>
</tr>
<tr id="S2.T1.1.4.3" class="ltx_tr">
<td id="S2.T1.1.4.3.1" class="ltx_td ltx_align_center ltx_border_bb"><span id="S2.T1.1.4.3.1.1" class="ltx_text" style="font-size:70%;">Llama-3-8B-Instruct-80K-QLoRA</span></td>
<td id="S2.T1.1.4.3.2" class="ltx_td ltx_align_center ltx_border_bb"><span id="S2.T1.1.4.3.2.1" class="ltx_text ltx_font_bold" style="font-size:70%;">43.57</span></td>
<td id="S2.T1.1.4.3.3" class="ltx_td ltx_align_center ltx_border_bb"><span id="S2.T1.1.4.3.3.1" class="ltx_text ltx_font_bold" style="font-size:70%;">43.07</span></td>
<td id="S2.T1.1.4.3.4" class="ltx_td ltx_align_center ltx_border_bb"><span id="S2.T1.1.4.3.4.1" class="ltx_text ltx_font_bold" style="font-size:70%;">28.93</span></td>
<td id="S2.T1.1.4.3.5" class="ltx_td ltx_align_center ltx_border_bb"><span id="S2.T1.1.4.3.5.1" class="ltx_text" style="font-size:70%;">69.15</span></td>
<td id="S2.T1.1.4.3.6" class="ltx_td ltx_align_center ltx_border_bb"><span id="S2.T1.1.4.3.6.1" class="ltx_text ltx_font_bold" style="font-size:70%;">48.50</span></td>
<td id="S2.T1.1.4.3.7" class="ltx_td ltx_align_center ltx_border_bb"><span id="S2.T1.1.4.3.7.1" class="ltx_text" style="font-size:70%;">51.95</span></td>
<td id="S2.T1.1.4.3.8" class="ltx_td ltx_align_center ltx_border_bb"><span id="S2.T1.1.4.3.8.1" class="ltx_text ltx_font_bold" style="font-size:70%;">47.19</span></td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering" style="font-size:70%;"><span class="ltx_tag ltx_tag_table">Table 1: </span>Evaluation results on LongBench. For Llama-3-8B-Instruct, we use 8K context length.</figcaption>
</figure>
<figure id="S2.T2" class="ltx_table">
<table id="S2.T2.1" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S2.T2.1.1.1" class="ltx_tr">
<th id="S2.T2.1.1.1.1" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">Model</th>
<th id="S2.T2.1.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">LongBookQA Eng</th>
<th id="S2.T2.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">LongBookSum Eng</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S2.T2.1.2.1" class="ltx_tr">
<td id="S2.T2.1.2.1.1" class="ltx_td ltx_align_center ltx_border_t">GPT-4</td>
<td id="S2.T2.1.2.1.2" class="ltx_td ltx_align_center ltx_border_t">22.22</td>
<td id="S2.T2.1.2.1.3" class="ltx_td ltx_align_center ltx_border_t">14.73</td>
</tr>
<tr id="S2.T2.1.3.2" class="ltx_tr">
<td id="S2.T2.1.3.2.1" class="ltx_td ltx_align_center">Llama-3-8B-Instruct</td>
<td id="S2.T2.1.3.2.2" class="ltx_td ltx_align_center">7.00</td>
<td id="S2.T2.1.3.2.3" class="ltx_td ltx_align_center"><span id="S2.T2.1.3.2.3.1" class="ltx_text ltx_font_bold">16.40</span></td>
</tr>
<tr id="S2.T2.1.4.3" class="ltx_tr">
<td id="S2.T2.1.4.3.1" class="ltx_td ltx_align_center">Llama-3-8B-Instruct-262K</td>
<td id="S2.T2.1.4.3.2" class="ltx_td ltx_align_center">20.30</td>
<td id="S2.T2.1.4.3.3" class="ltx_td ltx_align_center">10.34</td>
</tr>
<tr id="S2.T2.1.5.4" class="ltx_tr">
<td id="S2.T2.1.5.4.1" class="ltx_td ltx_align_center ltx_border_bb">Llama-3-8B-Instruct-80K-QLoRA</td>
<td id="S2.T2.1.5.4.2" class="ltx_td ltx_align_center ltx_border_bb"><span id="S2.T2.1.5.4.2.1" class="ltx_text ltx_font_bold">30.92</span></td>
<td id="S2.T2.1.5.4.3" class="ltx_td ltx_align_center ltx_border_bb">14.73</td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 2: </span>Evaluation results on InfBench. For Llama-3-8B-Instruct, we use 8K context length. The results of GPT-4 is copied from the paper <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib17" title="" class="ltx_ref">17</a>]</cite>.</figcaption>
</figure>
<figure id="S2.T3" class="ltx_table">
<table id="S2.T3.1" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S2.T3.1.1.1" class="ltx_tr">
<th id="S2.T3.1.1.1.1" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">Model</th>
<th id="S2.T3.1.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">STEM</th>
<th id="S2.T3.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">Social</th>
<th id="S2.T3.1.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">Humanities</th>
<th id="S2.T3.1.1.1.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">Others</th>
<th id="S2.T3.1.1.1.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">Avg</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S2.T3.1.2.1" class="ltx_tr">
<td id="S2.T3.1.2.1.1" class="ltx_td ltx_align_center ltx_border_t">Llama-2-7B-Chat</td>
<td id="S2.T3.1.2.1.2" class="ltx_td ltx_align_center ltx_border_t">35.92</td>
<td id="S2.T3.1.2.1.3" class="ltx_td ltx_align_center ltx_border_t">54.37</td>
<td id="S2.T3.1.2.1.4" class="ltx_td ltx_align_center ltx_border_t">51.74</td>
<td id="S2.T3.1.2.1.5" class="ltx_td ltx_align_center ltx_border_t">51.42</td>
<td id="S2.T3.1.2.1.6" class="ltx_td ltx_align_center ltx_border_t">47.22</td>
</tr>
<tr id="S2.T3.1.3.2" class="ltx_tr">
<td id="S2.T3.1.3.2.1" class="ltx_td ltx_align_center">Mistral-7B-v0.2-Instruct</td>
<td id="S2.T3.1.3.2.2" class="ltx_td ltx_align_center">48.79</td>
<td id="S2.T3.1.3.2.3" class="ltx_td ltx_align_center">69.95</td>
<td id="S2.T3.1.3.2.4" class="ltx_td ltx_align_center">64.99</td>
<td id="S2.T3.1.3.2.5" class="ltx_td ltx_align_center">61.64</td>
<td id="S2.T3.1.3.2.6" class="ltx_td ltx_align_center">60.10</td>
</tr>
<tr id="S2.T3.1.4.3" class="ltx_tr">
<td id="S2.T3.1.4.3.1" class="ltx_td ltx_align_center">Llama-3-8B-Instruct</td>
<td id="S2.T3.1.4.3.2" class="ltx_td ltx_align_center"><span id="S2.T3.1.4.3.2.1" class="ltx_text ltx_font_bold">53.87</span></td>
<td id="S2.T3.1.4.3.3" class="ltx_td ltx_align_center"><span id="S2.T3.1.4.3.3.1" class="ltx_text ltx_font_bold">75.66</span></td>
<td id="S2.T3.1.4.3.4" class="ltx_td ltx_align_center"><span id="S2.T3.1.4.3.4.1" class="ltx_text ltx_font_bold">69.44</span></td>
<td id="S2.T3.1.4.3.5" class="ltx_td ltx_align_center">69.75</td>
<td id="S2.T3.1.4.3.6" class="ltx_td ltx_align_center"><span id="S2.T3.1.4.3.6.1" class="ltx_text ltx_font_bold">65.91</span></td>
</tr>
<tr id="S2.T3.1.5.4" class="ltx_tr">
<td id="S2.T3.1.5.4.1" class="ltx_td ltx_align_center">Llama-3-8B-Instruct-262K</td>
<td id="S2.T3.1.5.4.2" class="ltx_td ltx_align_center">52.10</td>
<td id="S2.T3.1.5.4.3" class="ltx_td ltx_align_center">73.26</td>
<td id="S2.T3.1.5.4.4" class="ltx_td ltx_align_center">67.15</td>
<td id="S2.T3.1.5.4.5" class="ltx_td ltx_align_center"><span id="S2.T3.1.5.4.5.1" class="ltx_text ltx_font_bold">69.80</span></td>
<td id="S2.T3.1.5.4.6" class="ltx_td ltx_align_center">64.34</td>
</tr>
<tr id="S2.T3.1.6.5" class="ltx_tr">
<td id="S2.T3.1.6.5.1" class="ltx_td ltx_align_center ltx_border_bb">Llama-3-8B-Instruct-80K-QLoRA</td>
<td id="S2.T3.1.6.5.2" class="ltx_td ltx_align_center ltx_border_bb">53.10</td>
<td id="S2.T3.1.6.5.3" class="ltx_td ltx_align_center ltx_border_bb">73.24</td>
<td id="S2.T3.1.6.5.4" class="ltx_td ltx_align_center ltx_border_bb">67.32</td>
<td id="S2.T3.1.6.5.5" class="ltx_td ltx_align_center ltx_border_bb">68.79</td>
<td id="S2.T3.1.6.5.6" class="ltx_td ltx_align_center ltx_border_bb">64.44</td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 3: </span>Zero-shot performance on MMLU.</figcaption>
</figure>
<div id="S2.p1" class="ltx_para">
<p id="S2.p1.1" class="ltx_p">We evaluate our model on popular long-context benchmarks, then compare it with the original Llama-3-8B-Instruct model and the long-context Llama-3-8B-Instruct-262K from the community<span id="footnote4" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">4</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">4</sup><span class="ltx_tag ltx_tag_note">4</span><a target="_blank" href="https://huggingface.co/gradientai/Llama-3-8B-Instruct-262k" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://huggingface.co/gradientai/Llama-3-8B-Instruct-262k</a></span></span></span>.</p>
</div>
<div id="S2.p2" class="ltx_para">
<p id="S2.p2.1" class="ltx_p">Firstly, we leverage the Needle-In-A-Haystack task, which aims to recall an irrelevant piece of information (a.k.a. needle) inserted into a lengthy context (a.k.a. haystack). The accuracy is evaluated with GPT3.5. We use the same needle and haystack as in the official repository<span id="footnote5" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">5</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">5</sup><span class="ltx_tag ltx_tag_note">5</span><a target="_blank" href="https://github.com/gkamradt/LLMTest_NeedleInAHaystack" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/gkamradt/LLMTest_NeedleInAHaystack</a></span></span></span>. Our model achieves 100% accuracy over all its training context length. Besides, the model generalizes well to the unseen positions (80K<math id="S2.p2.1.m1.1" class="ltx_Math" alttext="\sim" display="inline"><semantics id="S2.p2.1.m1.1a"><mo id="S2.p2.1.m1.1.1" xref="S2.p2.1.m1.1.1.cmml">∼</mo><annotation-xml encoding="MathML-Content" id="S2.p2.1.m1.1b"><csymbol cd="latexml" id="S2.p2.1.m1.1.1.cmml" xref="S2.p2.1.m1.1.1">similar-to</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S2.p2.1.m1.1c">\sim</annotation></semantics></math>128K).</p>
</div>
<div id="S2.p3" class="ltx_para">
<p id="S2.p3.1" class="ltx_p">Secondly, we report the Topic Retrieval <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib12" title="" class="ltx_ref">12</a>]</cite> accuracy in Figure <a href="#S2.F2" title="Figure 2 ‣ 2 Experiments ‣ Extending Llama-3’s Context Ten-Fold Overnight" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>. This task synthesizes a long conversation with multiple independent discussions of a certain topic between the user and the assistant. Then the LLM is required to repeat the first topic as is in the conversation. We use the conversations made up of [5,10,15,20,25,30,40,50,60,70] topics for evaluation. It can be observed that Llama-3-8B-Instruct fails to remember the topic when the context is longer than 9K. However, the accuracy of our model remains 100% throughout all context lengths.</p>
</div>
<div id="S2.p4" class="ltx_para">
<p id="S2.p4.1" class="ltx_p">Thirdly, we evaluate our model on LongBench <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib3" title="" class="ltx_ref">3</a>]</cite>, which contains a variety of real-world long-context tasks. Most context on this benchmark is shorter than 32K. Thus, we use 32K context length by default and 8K for Llama-3-8B-Instruct. The results are shown in Table <a href="#S2.T1" title="Table 1 ‣ 2 Experiments ‣ Extending Llama-3’s Context Ten-Fold Overnight" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>. Our model significantly and consistently outperforms all baselines except on the code completion task. Mixing more code data in training may mitigate this problem.</p>
</div>
<div id="S2.p5" class="ltx_para">
<p id="S2.p5.1" class="ltx_p">Forthly, we employ the English Long-Book QA and the Long-Book Summarization task from InfiniteBench <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib17" title="" class="ltx_ref">17</a>]</cite> to assess the model’s performance on really long context. The testing instances are usually longer than 100K. We truncate them to 80K. According to Table <a href="#S2.T2" title="Table 2 ‣ 2 Experiments ‣ Extending Llama-3’s Context Ten-Fold Overnight" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>, Llama-3-8B-Instruct-80K-QLoRA excels on answering the questions based on the long context. It also achieves competitive performance against GPT-4 in terms of summarization. Interestingly, Llama-3-8B-Instruct with 8K context outperforms GPT-4 with 128K context on summarization. This is likely to be a metric-oriented issue (currently rouge-f1 is used) since the summary may have different paraphrases, which may not necessarily overlap with the ground truth.</p>
</div>
<div id="S2.p6" class="ltx_para">
<p id="S2.p6.1" class="ltx_p">Lastly, in Table <a href="#S2.T3" title="Table 3 ‣ 2 Experiments ‣ Extending Llama-3’s Context Ten-Fold Overnight" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>, we compare the zero-shot performance of our model and the baselines on MMLU <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib10" title="" class="ltx_ref">10</a>]</cite> benchmark. We also include Llama-2-7B-Chat <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib15" title="" class="ltx_ref">15</a>]</cite> and Mistral-7B-Instruct-v0.2 <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib11" title="" class="ltx_ref">11</a>]</cite> for comparison. It can be observed that both long-context models underperform the original Llama-3-8B-Instruct, indicating that context extension may compromise the model’s short-context capability. This observation is in line with previous research <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib14" title="" class="ltx_ref">14</a>]</cite>. However, our model’s performance is still superior to other open-source models at the same scale.</p>
</div>
</section>
<section id="bib" class="ltx_bibliography">
<h2 class="ltx_title ltx_title_bibliography">References</h2>

<ul class="ltx_biblist">
<li id="bib.bib1" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[1]</span>
<span class="ltx_bibblock">
Unsloth.ai.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://github.com/unslothai/unsloth" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/unslothai/unsloth</a>, 2023.

</span>
</li>
<li id="bib.bib2" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[2]</span>
<span class="ltx_bibblock">
S. An, Z. Ma, Z. Lin, N. Zheng, and J.-G. Lou.

</span>
<span class="ltx_bibblock">Make your llm fully utilize the context, 2024.

</span>
</li>
<li id="bib.bib3" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[3]</span>
<span class="ltx_bibblock">
Y. Bai, X. Lv, J. Zhang, H. Lyu, J. Tang, Z. Huang, Z. Du, X. Liu, A. Zeng, L. Hou, Y. Dong, J. Tang, and J. Li.

</span>
<span class="ltx_bibblock">Longbench: A bilingual, multitask benchmark for long context understanding, 2023.

</span>
</li>
<li id="bib.bib4" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[4]</span>
<span class="ltx_bibblock">
S. Chen, S. Wong, L. Chen, and Y. Tian.

</span>
<span class="ltx_bibblock">Extending context window of large language models via positional interpolation, 2023.

</span>
</li>
<li id="bib.bib5" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[5]</span>
<span class="ltx_bibblock">
Y. Chen, S. Qian, H. Tang, X. Lai, Z. Liu, S. Han, and J. Jia.

</span>
<span class="ltx_bibblock">Longlora: Efficient fine-tuning of long-context large language models, 2024.

</span>
</li>
<li id="bib.bib6" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[6]</span>
<span class="ltx_bibblock">
T. Computer.

</span>
<span class="ltx_bibblock">Redpajama: An open source recipe to reproduce llama training dataset, 2023.

</span>
</li>
<li id="bib.bib7" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[7]</span>
<span class="ltx_bibblock">
T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer.

</span>
<span class="ltx_bibblock">Qlora: Efficient finetuning of quantized llms, 2023.

</span>
</li>
<li id="bib.bib8" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[8]</span>
<span class="ltx_bibblock">
Y. Ding, L. L. Zhang, C. Zhang, Y. Xu, N. Shang, J. Xu, F. Yang, and M. Yang.

</span>
<span class="ltx_bibblock">Longrope: Extending llm context window beyond 2 million tokens, 2024.

</span>
</li>
<li id="bib.bib9" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[9]</span>
<span class="ltx_bibblock">
Y. Fu, R. Panda, X. Niu, X. Yue, H. Hajishirzi, Y. Kim, and H. Peng.

</span>
<span class="ltx_bibblock">Data engineering for scaling language models to 128k context, 2024.

</span>
</li>
<li id="bib.bib10" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[10]</span>
<span class="ltx_bibblock">
D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt.

</span>
<span class="ltx_bibblock">Measuring massive multitask language understanding, 2021.

</span>
</li>
<li id="bib.bib11" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[11]</span>
<span class="ltx_bibblock">
A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. de las Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, L. R. Lavaud, M.-A. Lachaux, P. Stock, T. L. Scao, T. Lavril, T. Wang, T. Lacroix, and W. E. Sayed.

</span>
<span class="ltx_bibblock">Mistral 7b, 2023.

</span>
</li>
<li id="bib.bib12" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[12]</span>
<span class="ltx_bibblock">
D. Li*, R. Shao*, A. Xie, Y. Sheng, L. Zheng, J. E. Gonzalez, I. Stoica, X. Ma, , and H. Zhang.

</span>
<span class="ltx_bibblock">How long can open-source llms truly promise on context length?, June 2023.

</span>
</li>
<li id="bib.bib13" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[13]</span>
<span class="ltx_bibblock">
OpenAI.

</span>
<span class="ltx_bibblock">Gpt-4 technical report, 2024.

</span>
</li>
<li id="bib.bib14" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[14]</span>
<span class="ltx_bibblock">
B. Peng, J. Quesnelle, H. Fan, and E. Shippole.

</span>
<span class="ltx_bibblock">Yarn: Efficient context window extension of large language models, 2023.

</span>
</li>
<li id="bib.bib15" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[15]</span>
<span class="ltx_bibblock">
H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. C. Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan, M. Kardas, V. Kerkez, M. Khabsa, I. Kloumann, A. Korenev, P. S. Koura, M.-A. Lachaux, T. Lavril, J. Lee, D. Liskovich, Y. Lu, Y. Mao, X. Martinet, T. Mihaylov, P. Mishra, I. Molybog, Y. Nie, A. Poulton, J. Reizenstein, R. Rungta, K. Saladi, A. Schelten, R. Silva, E. M. Smith, R. Subramanian, X. E. Tan, B. Tang, R. Taylor, A. Williams, J. X. Kuan, P. Xu, Z. Yan, I. Zarov, Y. Zhang, A. Fan, M. Kambadur, S. Narang, A. Rodriguez, R. Stojnic, S. Edunov, and T. Scialom.

</span>
<span class="ltx_bibblock">Llama 2: Open foundation and fine-tuned chat models, 2023.

</span>
</li>
<li id="bib.bib16" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[16]</span>
<span class="ltx_bibblock">
P. Zhang, Z. Liu, S. Xiao, N. Shao, Q. Ye, and Z. Dou.

</span>
<span class="ltx_bibblock">Soaring from 4k to 400k: Extending llm’s context with activation beacon, 2024.

</span>
</li>
<li id="bib.bib17" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[17]</span>
<span class="ltx_bibblock">
X. Zhang, Y. Chen, S. Hu, Z. Xu, J. Chen, M. K. Hao, X. Han, Z. L. Thai, S. Wang, Z. Liu, and M. Sun.

</span>
<span class="ltx_bibblock"><math id="bib.bib17.1.m1.1" class="ltx_Math" alttext="\infty" display="inline"><semantics id="bib.bib17.1.m1.1a"><mi mathvariant="normal" id="bib.bib17.1.m1.1.1" xref="bib.bib17.1.m1.1.1.cmml">∞</mi><annotation-xml encoding="MathML-Content" id="bib.bib17.1.m1.1b"><infinity id="bib.bib17.1.m1.1.1.cmml" xref="bib.bib17.1.m1.1.1"></infinity></annotation-xml><annotation encoding="application/x-tex" id="bib.bib17.1.m1.1c">\infty</annotation></semantics></math>bench: Extending long context evaluation beyond 100k tokens, 2024.

</span>
</li>
</ul>
</section>
</article>
</div>
<div class="ar5iv-footer"><a href="/html/2404.19552" class="ar5iv-nav-button ar5iv-nav-button-prev">◄</a>
    <a class="ar5iv-home-button" href="/"><img height="40" alt="ar5iv homepage" src="/assets/ar5iv.png"></a>
    <a href="/feeling_lucky" class="ar5iv-text-button">Feeling<br>lucky?</a>
    <a href="/log/2404.19553" class="ar5iv-text-button ar5iv-severity-ok">Conversion<br>report</a>
    <a class="ar5iv-text-button" target="_blank" href="https://github.com/dginev/ar5iv/issues/new?template=improve-article--arxiv-id-.md&title=Improve+article+2404.19553">Report<br>an issue</a>
    <a href="https://arxiv.org/abs/2404.19553" class="ar5iv-text-button arxiv-ui-theme">View&nbsp;original<br>on&nbsp;arXiv</a><a href="/html/2404.19554" class="ar5iv-nav-button ar5iv-nav-button-next">►</a>
</div><footer class="ltx_page_footer">
<a class="ar5iv-toggle-color-scheme" href="javascript:toggleColorScheme()" title="Toggle ar5iv color scheme"><span class="color-scheme-icon"></span></a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/license" target="_blank">Copyright</a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/policies/privacy_policy" target="_blank">Privacy Policy</a>

<div class="ltx_page_logo">Generated  on Sun May  5 17:55:32 2024 by <a target="_blank" href="http://dlmf.nist.gov/LaTeXML/" class="ltx_LaTeXML_logo"><span style="letter-spacing:-0.2em; margin-right:0.1em;">L<span class="ltx_font_smallcaps" style="position:relative; bottom:2.2pt;">a</span>T<span class="ltx_font_smallcaps" style="font-size:120%;position:relative; bottom:-0.2ex;">e</span></span><span style="font-size:90%; position:relative; bottom:-0.2ex;">XML</span><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAsAAAAOCAYAAAD5YeaVAAAAAXNSR0IArs4c6QAAAAZiS0dEAP8A/wD/oL2nkwAAAAlwSFlzAAALEwAACxMBAJqcGAAAAAd0SU1FB9wKExQZLWTEaOUAAAAddEVYdENvbW1lbnQAQ3JlYXRlZCB3aXRoIFRoZSBHSU1Q72QlbgAAAdpJREFUKM9tkL+L2nAARz9fPZNCKFapUn8kyI0e4iRHSR1Kb8ng0lJw6FYHFwv2LwhOpcWxTjeUunYqOmqd6hEoRDhtDWdA8ApRYsSUCDHNt5ul13vz4w0vWCgUnnEc975arX6ORqN3VqtVZbfbTQC4uEHANM3jSqXymFI6yWazP2KxWAXAL9zCUa1Wy2tXVxheKA9YNoR8Pt+aTqe4FVVVvz05O6MBhqUIBGk8Hn8HAOVy+T+XLJfLS4ZhTiRJgqIoVBRFIoric47jPnmeB1mW/9rr9ZpSSn3Lsmir1fJZlqWlUonKsvwWwD8ymc/nXwVBeLjf7xEKhdBut9Hr9WgmkyGEkJwsy5eHG5vN5g0AKIoCAEgkEkin0wQAfN9/cXPdheu6P33fBwB4ngcAcByHJpPJl+fn54mD3Gg0NrquXxeLRQAAwzAYj8cwTZPwPH9/sVg8PXweDAauqqr2cDjEer1GJBLBZDJBs9mE4zjwfZ85lAGg2+06hmGgXq+j3+/DsixYlgVN03a9Xu8jgCNCyIegIAgx13Vfd7vdu+FweG8YRkjXdWy329+dTgeSJD3ieZ7RNO0VAXAPwDEAO5VKndi2fWrb9jWl9Esul6PZbDY9Go1OZ7PZ9z/lyuD3OozU2wAAAABJRU5ErkJggg==" alt="Mascot Sammy"></a>
</div></footer>
</div>

    <script>
      var canMathML = typeof(MathMLElement) == "function";
      if (!canMathML) {
        var body = document.querySelector("body");
        body.firstElementChild.setAttribute('style', 'opacity: 0;');
        var loading = document.createElement("div");
        loading.setAttribute("id", "mathjax-loading-spinner");
        var message = document.createElement("div");
        message.setAttribute("id", "mathjax-loading-message");
        message.innerText = "Typesetting Equations...";
        body.prepend(loading);
        body.prepend(message);

        var el = document.createElement("script");
        el.src = "https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js";
        document.querySelector("head").appendChild(el);

        window.MathJax = {
          startup: {
            pageReady: () => {
              return MathJax.startup.defaultPageReady().then(() => {
                body.removeChild(loading);
                body.removeChild(message);
                body.firstElementChild.removeAttribute('style');
              }); } } };
      }
    </script>
    <script>
    // Auxiliary function, building the preview feature when
    // an inline citation is clicked
    function clicked_cite(e) {
      e.preventDefault();
      let cite = this.closest('.ltx_cite');
      let next = cite.nextSibling;
      if (next && next.nodeType == Node.ELEMENT_NODE && next.getAttribute('class') == "ar5iv-bibitem-preview") {
        next.remove();
        return; }
      // Before adding a preview modal,
      // cleanup older previews, in case they're still open
      document.querySelectorAll('span.ar5iv-bibitem-preview').forEach(function(node) {
        node.remove();
      })

      // Create the preview
      preview = document.createElement('span');
      preview.setAttribute('class','ar5iv-bibitem-preview');
      let target = document.getElementById(this.getAttribute('href').slice(1));
      target.childNodes.forEach(function (child) {
        preview.append(child.cloneNode(true));
      });
      let close_x = document.createElement('button');
      close_x.setAttribute("aria-label","Close modal for bibliography item preview");
      close_x.textContent = "×";
      close_x.setAttribute('class', 'ar5iv-button-close-preview');
      close_x.setAttribute('onclick','this.parentNode.remove()');
      preview.append(close_x);
      preview.querySelectorAll('.ltx_tag_bibitem').forEach(function(node) {
        node.remove();
      });
      cite.parentNode.insertBefore(preview, cite.nextSibling);
      return;
    }
    // Global Document initialization:
    // - assign the preview feature to all inline citation links
    document.querySelectorAll(".ltx_cite .ltx_ref").forEach(function (link) {
      link.addEventListener("click", clicked_cite);
    });
    </script>
    </body>
</html>

2404.19553
We extend the context length of Llama-3-8B-Instruct from 8K to 80K via QLoRA fine-tuning111The model is noted as Llama-3-8B-Instruct-80K-QLoRA given its max context length during fine-tuning. However, users could apply the model for even longer contexts via extrapolation.. The entire training cycle is super efficient, which takes 8 hours on one 8xA800 (80G)  GPU machine. The resulted model exhibits superior performances across a broad range of evaluation tasks, such as NIHS, topic retrieval, and long-context language understanding; meanwhile, it also well preserves the original capability over short contexts. The dramatic context extension is mainly attributed to merely 3.5K synthetic training samples generated by GPT-4 , which indicates the LLMs’ inherent (yet largely underestimated)  potential to extend its original context length. In fact, the context length could be extended far beyond 80K with more computation resources. Therefore, the team will publicly release the entire resources (including data, model, data generation pipeline, training code)  so as to facilitate the future research from the community:https://github.com/FlagOpen/FlagEmbedding.
abstract
Extending Llama-3’s Context Ten-Fold Overnight
Abstract
2404.19553
1IntroductionRecently, considerable attention has been directed towards long-context large language models, where different approaches are adopted to establish long-context capabilities for large language models. However, most of them require significant compute and resources to accomplish.In this technical report, we propose an efficient solution for entitling the long-context capabilities for LLMs, with which we extend the context length of Llama-3-8B-Instruct222https://llama.meta.com/llama3/from 8K to 80K. Specifically, we use GPT-4to synthesize 3.5K long-context training data, covering three long-context tasks:1.Single-Detail QA: the inquiry targets on one specific detail in a long context. To construct data for this task, we slice out a short segment (e.g., a chunk with less than 4096 tokens)  from a long context (e.g., a book or a long paper)  and prompt GPT-4 to generate multiple question-answer pairs based on this segment.2.Multi-Detail QA: the inquiry requires information aggregation and reasoning over multiple details in a long context. We define two types of long context. Thehomogeneouscontext contains a coherent text, such as a book or a long paper. We prompt GPT-4 to generate multiple question-answer pairs that require aggregating and analyzing information from different locations in the context. Theheterogeneouscontext consists of multiple independent texts. Notably, we perform clustering over a large corpus then extract texts from the same cluster to form each heterogeneous context. Therefore, the grouped texts share some semantic similarity. We then prompt GPT-4 to ask about the similarities/dissimilarities across these texts.3.Biography Summarization: we prompt GPT-4 to write a biography for each main character in a given book.For all three tasks, the length of context is between 64K to 80K. Note that longer data can also be synthesized following the same methodology. When training, we organize the question-answer pairs for the same context in one multi-turn conversation then fine-tune the LLM to correctly answer the questions given the entire long context as input.
Following previous work333https://www.together.ai/blog/llama-2-7b-32k, we mix 5K instances randomly chosen from RedPajamato mitigate forgetting. We also mix LongAlpacain the training set, which contains 12K instruction tuning instances with 16K length at maximum. Therefore, the entire training dataset contains 20K instances.We use QLoRAto efficiently fine-tune the model. We apply LoRA on all Q,K,V,O projections and additionally train the embedding layer. We set LoRA rank to 32 and alpha to 16. The learning rate is 5e-5 with linear decay and no warmups. The batch size is 8. Gradient checkpointing is enabled. No parallel strategy is required thanks to the efficient implementation from Unsloth. We train the model for 1 epoch, which takes 8 hours to complete on a 8xA800 (80G)  machine. Importantly, we expand the RoPE base from 500K to 200M in training.Our contributions are highlighted as follows:•We release Llama-3-8B-Instruct-80K-QLoRA, which extends the context length of Llama-3-8B-Instruct from 8K to 80K. The entire resources including the model, training data, and code are all publicly available, which may advance the field of training long-context LLMs.•Our training recipe is simple and efficient, while the resulted model demonstrates remarkable performance on downstream long-context tasks. Further research can be made to improve our approach.
introduction
Extending Llama-3’s Context Ten-Fold Overnight
1Introduction
2404.19553
2ExperimentsFigure 1:The accuracy score of Llama-3-8B-Instruct-80K-QLoRA on Needle-In-A-HayStack task. The blue vertical line indicates the training length, i.e. 80K.Figure 2:The accuracy of Topic Retrieval task.ModelSingle-DocMulti-DocSumm.Few-ShotSyntheticCodeAvgLlama-3-8B-Instruct37.3336.0426.8369.5637.7553.2443.20Llama-3-8B-Instruct-262K37.2931.2026.1867.2544.2562.7143.73Llama-3-8B-Instruct-80K-QLoRA43.5743.0728.9369.1548.5051.9547.19Table 1:Evaluation results on LongBench. For Llama-3-8B-Instruct, we use 8K context length.ModelLongBookQA EngLongBookSum EngGPT-422.2214.73Llama-3-8B-Instruct7.0016.40Llama-3-8B-Instruct-262K20.3010.34Llama-3-8B-Instruct-80K-QLoRA30.9214.73Table 2:Evaluation results on InfBench. For Llama-3-8B-Instruct, we use 8K context length. The results of GPT-4 is copied from the paper.ModelSTEMSocialHumanitiesOthersAvgLlama-2-7B-Chat35.9254.3751.7451.4247.22Mistral-7B-v0.2-Instruct48.7969.9564.9961.6460.10Llama-3-8B-Instruct53.8775.6669.4469.7565.91Llama-3-8B-Instruct-262K52.1073.2667.1569.8064.34Llama-3-8B-Instruct-80K-QLoRA53.1073.2467.3268.7964.44Table 3:Zero-shot performance on MMLU.We evaluate our model on popular long-context benchmarks, then compare it with the original Llama-3-8B-Instruct model and the long-context Llama-3-8B-Instruct-262K from the community444https://huggingface.co/gradientai/Llama-3-8B-Instruct-262k.Firstly, we leverage the Needle-In-A-Haystack task, which aims to recall an irrelevant piece of information (a.k.a. needle)  inserted into a lengthy context (a.k.a. haystack) . The accuracy is evaluated with GPT3.5. We use the same needle and haystack as in the official repository555https://github.com/gkamradt/LLMTest_NeedleInAHaystack. Our model achieves 100% accuracy over all its training context length. Besides, the model generalizes well to the unseen positions (80K128K) .Secondly, we report the Topic Retrievalaccuracy in Figure2. This task synthesizes a long conversation with multiple independent discussions of a certain topic between the user and the assistant. Then the LLM is required to repeat the first topic as is in the conversation. We use the conversations made up of [5,10,15,20,25,30,40,50,60,70] topics for evaluation. It can be observed that Llama-3-8B-Instruct fails to remember the topic when the context is longer than 9K. However, the accuracy of our model remains 100% throughout all context lengths.Thirdly, we evaluate our model on LongBench, which contains a variety of real-world long-context tasks. Most context on this benchmark is shorter than 32K. Thus, we use 32K context length by default and 8K for Llama-3-8B-Instruct. The results are shown in Table1. Our model significantly and consistently outperforms all baselines except on the code completion task. Mixing more code data in training may mitigate this problem.Forthly, we employ the English Long-Book QA and the Long-Book Summarization task from InfiniteBenchto assess the model’s performance on really long context. The testing instances are usually longer than 100K. We truncate them to 80K. According to Table2, Llama-3-8B-Instruct-80K-QLoRA excels on answering the questions based on the long context. It also achieves competitive performance against GPT-4 in terms of summarization. Interestingly, Llama-3-8B-Instruct with 8K context outperforms GPT-4 with 128K context on summarization. This is likely to be a metric-oriented issue (currently rouge-f1 is used)  since the summary may have different paraphrases, which may not necessarily overlap with the ground truth.Lastly, in Table3, we compare the zero-shot performance of our model and the baselines on MMLUbenchmark. We also include Llama-2-7B-Chatand Mistral-7B-Instruct-v0.2for comparison. It can be observed that both long-context models underperform the original Llama-3-8B-Instruct, indicating that context extension may compromise the model’s short-context capability. This observation is in line with previous research. However, our model’s performance is still superior to other open-source models at the same scale.
conclusion
Extending Llama-3’s Context Ten-Fold Overnight
2Experiments