<!DOCTYPE html><html lang="en">
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<title>[2402.17834] Stable LM 2 1.6B Technical Report</title><meta property="og:description" content="We introduce StableLM 2 1.6B, the first in a new generation of our language model series.
In this technical report, we present in detail the data and training procedure leading to the base and instruction-tuned version…">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="twitter:card" content="summary">
<meta name="twitter:title" content="Stable LM 2 1.6B Technical Report">
<meta name="twitter:image:src" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta name="twitter:image:alt" content="ar5iv logo">
<meta property="og:title" content="Stable LM 2 1.6B Technical Report">
<meta property="og:site_name" content="ar5iv">
<meta property="og:image" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta property="og:type" content="article">
<meta property="og:url" content="https://ar5iv.labs.arxiv.org/html/2402.17834">

<!--Generated on Tue Mar  5 15:56:42 2024 by LaTeXML (version 0.8.8) http://dlmf.nist.gov/LaTeXML/.-->
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">

<script>
  function detectColorScheme(){
    var theme="light";
    var current_theme = localStorage.getItem("ar5iv_theme");
    if(current_theme){
      if(current_theme == "dark"){
        theme = "dark";
      } }
    else if(!window.matchMedia) { return false; }
    else if(window.matchMedia("(prefers-color-scheme: dark)").matches) {
      theme = "dark"; }
    if (theme=="dark") {
      document.documentElement.setAttribute("data-theme", "dark");
    } else {
      document.documentElement.setAttribute("data-theme", "light"); } }

  detectColorScheme();

  function toggleColorScheme(){
    var current_theme = localStorage.getItem("ar5iv_theme");
    if (current_theme) {
      if (current_theme == "light") {
        localStorage.setItem("ar5iv_theme", "dark"); }
      else {
        localStorage.setItem("ar5iv_theme", "light"); } }
    else {
        localStorage.setItem("ar5iv_theme", "dark"); }
    detectColorScheme(); }
</script>
<link media="all" rel="stylesheet" href="/assets/ar5iv-fonts.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv-site.0.2.2.css">
</head>
<body>
<div class="ltx_page_main">
<div class="ltx_page_content">
<article class="ltx_document ltx_authors_1line" lang="en">
<h1 class="ltx_title ltx_title_document">Stable LM 2 1.6B Technical Report</h1>
<div class="ltx_authors">
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">
Marco Bellagente<sup id="id1.1.id1" class="ltx_sup">*</sup>  Jonathan Tow<sup id="id2.2.id2" class="ltx_sup">*</sup>  Dakota Mahan  Duy Phung  Maksym Zhuravinskyi 
<br class="ltx_break">Reshinth Adithyan  James Baicoianu  Ben Brooks  Nathan Cooper  Ashish Datta 
<br class="ltx_break">Meng Lee  Emad Mostaque  Michael Pieler  Nikhil Pinnaparju  Paulo Rocha 
<br class="ltx_break">Harry Saini  Hannah Teufel  Niccolo Zanichelli  Carlos Riquelme

<br class="ltx_break">
<br class="ltx_break"><span id="id3.3.id3" class="ltx_text ltx_font_bold">Stability AI Language Team</span>
</span><span class="ltx_author_notes">Equal contribution. Correspondance to: {marco.bellagente, jonathantow, carlos.riquelme}@stability.ai</span></span>
</div>

<div class="ltx_abstract">
<h6 class="ltx_title ltx_title_abstract">Abstract</h6>
<p id="id4.id1" class="ltx_p"><span id="id4.id1.1" class="ltx_text">We introduce StableLM 2 1.6B, the first in a new generation of our language model series.
In this technical report, we present in detail the data and training procedure leading to the base and instruction-tuned versions of StableLM 2 1.6B.
The weights for both models are available via Hugging Face for anyone to download and use <span id="footnote1" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">1</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">1</sup><span class="ltx_tag ltx_tag_note">1</span><a target="_blank" href="https://huggingface.co/stabilityai/stablelm-2-1_6b" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://huggingface.co/stabilityai/stablelm-2-1_6b</a></span></span></span><span id="footnote2" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">2</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">2</sup><span class="ltx_tag ltx_tag_note">2</span><a target="_blank" href="https://huggingface.co/stabilityai/stablelm-2-zephyr-1_6b" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://huggingface.co/stabilityai/stablelm-2-zephyr-1_6b</a></span></span></span>.
The report contains thorough evaluations of these models, including zero- and few-shot benchmarks, multilingual benchmarks, and the MT benchmark focusing on multi-turn dialogues.
At the time of publishing this report, StableLM 2 1.6B was the state-of-the-art open model under 2B parameters by a significant margin.
Given its appealing small size, we also provide throughput measurements on a number of edge devices.
In addition, we open source several quantized checkpoints and provide their performance metrics compared to the original model.
</span></p>
</div>
<section id="S1" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">1 </span>Introduction</h2>

<div id="S1.p1" class="ltx_para">
<p id="S1.p1.1" class="ltx_p">Following the development of the Transformer architecture <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib71" title="" class="ltx_ref">71</a>]</cite>, a remarkable number of proprietary and open-source large language models have been trained and deployed.
While countless new ideas and artifacts are announced weekly or daily, some key aspects remain opaque – especially around the most powerful models.
Often, the training data is not disclosed.
This poses a fundamental challenge in times where society demands transparency as it faces a brand-new disruptive technology that is hard to inspect and audit.
In this report, we explain in a reproducible manner how to train a modest-size but state-of-the-art language model.
All the data we used is public (see Table <a href="#S2.T1" title="Table 1 ‣ 2 Model Pre-Training ‣ Stable LM 2 1.6B Technical Report" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>) and its training required around 92k GPU hours of training – worth around $322k on popular cloud providers (assuming $3.5 per GPU hour).
We hope this work contributes to the open AI community and helps set the standard for upcoming transparent models.</p>
</div>
<div id="S1.p2" class="ltx_para">
<p id="S1.p2.1" class="ltx_p">This report is organized as follows: Section <a href="#S2" title="2 Model Pre-Training ‣ Stable LM 2 1.6B Technical Report" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a> details the process of pre-training Stable LM 2 1.6B. We devote Section <a href="#S3" title="3 Fine-tuning and Alignment ‣ Stable LM 2 1.6B Technical Report" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a> to fine-tuning and human preference alignment.
Section <a href="#S4" title="4 Experimental Results and Benchmarks ‣ Stable LM 2 1.6B Technical Report" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a> presents model evaluations on standard downstream benchmarks.
Compiling and running inference Stable LM 2 on several edge devices is outlined in Section <a href="#S5" title="5 Inference and Quantization ‣ Stable LM 2 1.6B Technical Report" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a>.
We consider several follow-up directions in Section <a href="#S6" title="6 Future Work ‣ Stable LM 2 1.6B Technical Report" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a>. Carbon emissions and societal impacts related to the training and release of Stable LM 2 are covered in Section <a href="#S7" title="7 Environmental and Societal Impact ‣ Stable LM 2 1.6B Technical Report" class="ltx_ref"><span class="ltx_text ltx_ref_tag">7</span></a>.
Finally, Section <a href="#S8" title="8 Conclusion ‣ Stable LM 2 1.6B Technical Report" class="ltx_ref"><span class="ltx_text ltx_ref_tag">8</span></a> concludes and summarizes this work.</p>
</div>
</section>
<section id="S2" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">2 </span>Model Pre-Training</h2>

<div id="S2.p1" class="ltx_para">
<p id="S2.p1.1" class="ltx_p">The first stage in training large language models (LLMs) focuses on learning to predict the next token in a sequence using a vast and diverse array of data sources. We refer to this stage as <em id="S2.p1.1.1" class="ltx_emph ltx_font_italic">pre-training</em>.
It enables models to build general-purpose internal representations suitable for basic language capabilities and even more advanced generation and comprehension tasks. In fact, it has been hypothesized that the majority of model knowledge and capabilities are learned during pre-training <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib88" title="" class="ltx_ref">88</a>]</cite>. In this section, we introduce the design principles and ablations that influenced the creation of our training set, as well as details about the model architecture and training procedure. While many similar reports exist for other cutting-edge models, they often omit critical details, such as the particular data sources, sampling weights, or the complete set of ablations they performed.
As a result, the open-source community cannot accurately reproduce these models.
On the other hand, we present a fully transparent log of our model training details. We are confident that researchers and practitioners will find valuable insights in this comprehensive account.</p>
</div>
<figure id="S2.T1" class="ltx_table">
<table id="S2.T1.2" class="ltx_tabular ltx_align_middle">
<tr id="S2.T1.2.1" class="ltx_tr">
<td id="S2.T1.2.1.1" class="ltx_td ltx_align_center ltx_align_middle ltx_border_tt" style="padding-bottom:4.30554pt;"><span id="S2.T1.2.1.1.1" class="ltx_text ltx_font_bold">Dataset</span></td>
<td id="S2.T1.2.1.2" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_tt" style="width:36.1pt;padding-bottom:4.30554pt;">
<span id="S2.T1.2.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="S2.T1.2.1.2.1.1" class="ltx_p"><span id="S2.T1.2.1.2.1.1.1" class="ltx_text ltx_font_bold">Sampling Weight</span></span>
</span>
</td>
<td id="S2.T1.2.1.3" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_tt" style="width:72.3pt;padding-bottom:4.30554pt;">
<span id="S2.T1.2.1.3.1" class="ltx_inline-block ltx_align_top">
<span id="S2.T1.2.1.3.1.1" class="ltx_p"><span id="S2.T1.2.1.3.1.1.1" class="ltx_text ltx_font_bold">Num Tokens</span></span>
</span>
</td>
<td id="S2.T1.2.1.4" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_tt" style="width:36.1pt;padding-bottom:4.30554pt;">
<span id="S2.T1.2.1.4.1" class="ltx_inline-block ltx_align_top">
<span id="S2.T1.2.1.4.1.1" class="ltx_p"><span id="S2.T1.2.1.4.1.1.1" class="ltx_text ltx_font_bold">Epochs</span></span>
</span>
</td>
<td id="S2.T1.2.1.5" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_tt" style="width:36.1pt;padding-bottom:4.30554pt;">
<span id="S2.T1.2.1.5.1" class="ltx_inline-block ltx_align_top">
<span id="S2.T1.2.1.5.1.1" class="ltx_p"><span id="S2.T1.2.1.5.1.1.1" class="ltx_text ltx_font_bold">Category</span></span>
</span>
</td>
</tr>
<tr id="S2.T1.2.2" class="ltx_tr" style="background-color:#CCE6FF;">
<td id="S2.T1.2.2.1" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:72.3pt;padding-bottom:0.86108pt;"><span id="S2.T1.2.2.1.1" class="ltx_text ltx_font_bold" style="background-color:#CCE6FF;">Arxiv</span></td>
<td id="S2.T1.2.2.2" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:36.1pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.2.2.1" class="ltx_inline-block ltx_align_top" style="background-color:#CCE6FF;">
<span id="S2.T1.2.2.2.1.1" class="ltx_p">0.0079</span>
</span>
</td>
<td id="S2.T1.2.2.3" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:72.3pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.2.3.1" class="ltx_inline-block ltx_align_top" style="background-color:#CCE6FF;">
<span id="S2.T1.2.2.3.1.1" class="ltx_p">15,852,446,656</span>
</span>
</td>
<td id="S2.T1.2.2.4" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:36.1pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.2.4.1" class="ltx_inline-block ltx_align_top" style="background-color:#CCE6FF;">
<span id="S2.T1.2.2.4.1.1" class="ltx_p">0.75</span>
</span>
</td>
<td id="S2.T1.2.2.5" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:36.1pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.2.5.1" class="ltx_inline-block ltx_align_top" style="background-color:#CCE6FF;">
<span id="S2.T1.2.2.5.1.1" class="ltx_p">Academic</span>
</span>
</td>
</tr>
<tr id="S2.T1.2.3" class="ltx_tr" style="background-color:#CCE6FF;">
<td id="S2.T1.2.3.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:72.3pt;padding-bottom:0.86108pt;"><span id="S2.T1.2.3.1.1" class="ltx_text ltx_font_bold" style="background-color:#CCE6FF;">PubMed</span></td>
<td id="S2.T1.2.3.2" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.3.2.1" class="ltx_inline-block ltx_align_top" style="background-color:#CCE6FF;">
<span id="S2.T1.2.3.2.1.1" class="ltx_p">0.012</span>
</span>
</td>
<td id="S2.T1.2.3.3" class="ltx_td ltx_align_justify ltx_align_middle" style="width:72.3pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.3.3.1" class="ltx_inline-block ltx_align_top" style="background-color:#CCE6FF;">
<span id="S2.T1.2.3.3.1.1" class="ltx_p">24,126,600,626</span>
</span>
</td>
<td id="S2.T1.2.3.4" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.3.4.1" class="ltx_inline-block ltx_align_top" style="background-color:#CCE6FF;">
<span id="S2.T1.2.3.4.1.1" class="ltx_p">1.0</span>
</span>
</td>
<td id="S2.T1.2.3.5" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.3.5.1" class="ltx_inline-block ltx_align_top" style="background-color:#CCE6FF;">
<span id="S2.T1.2.3.5.1.1" class="ltx_p">Academic</span>
</span>
</td>
</tr>
<tr id="S2.T1.2.4" class="ltx_tr" style="background-color:#CCE6FF;">
<td id="S2.T1.2.4.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:72.3pt;padding-bottom:0.86108pt;"><span id="S2.T1.2.4.1.1" class="ltx_text ltx_font_bold" style="background-color:#CCE6FF;">S2ORC</span></td>
<td id="S2.T1.2.4.2" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.4.2.1" class="ltx_inline-block ltx_align_top" style="background-color:#CCE6FF;">
<span id="S2.T1.2.4.2.1.1" class="ltx_p">0.0318</span>
</span>
</td>
<td id="S2.T1.2.4.3" class="ltx_td ltx_align_justify ltx_align_middle" style="width:72.3pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.4.3.1" class="ltx_inline-block ltx_align_top" style="background-color:#CCE6FF;">
<span id="S2.T1.2.4.3.1.1" class="ltx_p">63,910,567,858</span>
</span>
</td>
<td id="S2.T1.2.4.4" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.4.4.1" class="ltx_inline-block ltx_align_top" style="background-color:#CCE6FF;">
<span id="S2.T1.2.4.4.1.1" class="ltx_p">1.0</span>
</span>
</td>
<td id="S2.T1.2.4.5" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.4.5.1" class="ltx_inline-block ltx_align_top" style="background-color:#CCE6FF;">
<span id="S2.T1.2.4.5.1.1" class="ltx_p">Academic</span>
</span>
</td>
</tr>
<tr id="S2.T1.2.5" class="ltx_tr" style="background-color:#CCE6FF;">
<td id="S2.T1.2.5.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:72.3pt;padding-bottom:0.86108pt;"><span id="S2.T1.2.5.1.1" class="ltx_text ltx_font_bold" style="background-color:#CCE6FF;">PhilPapers</span></td>
<td id="S2.T1.2.5.2" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.5.2.1" class="ltx_inline-block ltx_align_top" style="background-color:#CCE6FF;">
<span id="S2.T1.2.5.2.1.1" class="ltx_p">0.0013</span>
</span>
</td>
<td id="S2.T1.2.5.3" class="ltx_td ltx_align_justify ltx_align_middle" style="width:72.3pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.5.3.1" class="ltx_inline-block ltx_align_top" style="background-color:#CCE6FF;">
<span id="S2.T1.2.5.3.1.1" class="ltx_p">2,591,115,796</span>
</span>
</td>
<td id="S2.T1.2.5.4" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.5.4.1" class="ltx_inline-block ltx_align_top" style="background-color:#CCE6FF;">
<span id="S2.T1.2.5.4.1.1" class="ltx_p">4.0</span>
</span>
</td>
<td id="S2.T1.2.5.5" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.5.5.1" class="ltx_inline-block ltx_align_top" style="background-color:#CCE6FF;">
<span id="S2.T1.2.5.5.1.1" class="ltx_p">Academic</span>
</span>
</td>
</tr>
<tr id="S2.T1.2.6" class="ltx_tr" style="background-color:#FFB482;">
<td id="S2.T1.2.6.1" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:72.3pt;padding-bottom:0.86108pt;"><span id="S2.T1.2.6.1.1" class="ltx_text ltx_font_bold" style="background-color:#FFB482;">BookCorpusOpen</span></td>
<td id="S2.T1.2.6.2" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:36.1pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.6.2.1" class="ltx_inline-block ltx_align_top" style="background-color:#FFB482;">
<span id="S2.T1.2.6.2.1.1" class="ltx_p">0.0045</span>
</span>
</td>
<td id="S2.T1.2.6.3" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:72.3pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.6.3.1" class="ltx_inline-block ltx_align_top" style="background-color:#FFB482;">
<span id="S2.T1.2.6.3.1.1" class="ltx_p">9,135,954,810</span>
</span>
</td>
<td id="S2.T1.2.6.4" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:36.1pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.6.4.1" class="ltx_inline-block ltx_align_top" style="background-color:#FFB482;">
<span id="S2.T1.2.6.4.1.1" class="ltx_p">6.0</span>
</span>
</td>
<td id="S2.T1.2.6.5" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:36.1pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.6.5.1" class="ltx_inline-block ltx_align_top" style="background-color:#FFB482;">
<span id="S2.T1.2.6.5.1.1" class="ltx_p">Books</span>
</span>
</td>
</tr>
<tr id="S2.T1.2.7" class="ltx_tr" style="background-color:#FFB482;">
<td id="S2.T1.2.7.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:72.3pt;padding-bottom:0.86108pt;"><span id="S2.T1.2.7.1.1" class="ltx_text ltx_font_bold" style="background-color:#FFB482;">PG-19</span></td>
<td id="S2.T1.2.7.2" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.7.2.1" class="ltx_inline-block ltx_align_top" style="background-color:#FFB482;">
<span id="S2.T1.2.7.2.1.1" class="ltx_p">0.0091</span>
</span>
</td>
<td id="S2.T1.2.7.3" class="ltx_td ltx_align_justify ltx_align_middle" style="width:72.3pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.7.3.1" class="ltx_inline-block ltx_align_top" style="background-color:#FFB482;">
<span id="S2.T1.2.7.3.1.1" class="ltx_p">18,293,226,468</span>
</span>
</td>
<td id="S2.T1.2.7.4" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.7.4.1" class="ltx_inline-block ltx_align_top" style="background-color:#FFB482;">
<span id="S2.T1.2.7.4.1.1" class="ltx_p">4.0</span>
</span>
</td>
<td id="S2.T1.2.7.5" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.7.5.1" class="ltx_inline-block ltx_align_top" style="background-color:#FFB482;">
<span id="S2.T1.2.7.5.1.1" class="ltx_p">Books</span>
</span>
</td>
</tr>
<tr id="S2.T1.2.8" class="ltx_tr" style="background-color:#FFB482;">
<td id="S2.T1.2.8.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:72.3pt;padding-bottom:0.86108pt;"><span id="S2.T1.2.8.1.1" class="ltx_text ltx_font_bold" style="background-color:#FFB482;">FanFics</span></td>
<td id="S2.T1.2.8.2" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.8.2.1" class="ltx_inline-block ltx_align_top" style="background-color:#FFB482;">
<span id="S2.T1.2.8.2.1.1" class="ltx_p">0.0018</span>
</span>
</td>
<td id="S2.T1.2.8.3" class="ltx_td ltx_align_justify ltx_align_middle" style="width:72.3pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.8.3.1" class="ltx_inline-block ltx_align_top" style="background-color:#FFB482;">
<span id="S2.T1.2.8.3.1.1" class="ltx_p">3,644,583,700</span>
</span>
</td>
<td id="S2.T1.2.8.4" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.8.4.1" class="ltx_inline-block ltx_align_top" style="background-color:#FFB482;">
<span id="S2.T1.2.8.4.1.1" class="ltx_p">4.0</span>
</span>
</td>
<td id="S2.T1.2.8.5" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.8.5.1" class="ltx_inline-block ltx_align_top" style="background-color:#FFB482;">
<span id="S2.T1.2.8.5.1.1" class="ltx_p">Books</span>
</span>
</td>
</tr>
<tr id="S2.T1.2.9" class="ltx_tr" style="background-color:#CCFFCC;">
<td id="S2.T1.2.9.1" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:72.3pt;padding-bottom:0.86108pt;"><span id="S2.T1.2.9.1.1" class="ltx_text ltx_font_bold" style="background-color:#CCFFCC;">Cultura-X (EN)</span></td>
<td id="S2.T1.2.9.2" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:36.1pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.9.2.1" class="ltx_inline-block ltx_align_top" style="background-color:#CCFFCC;">
<span id="S2.T1.2.9.2.1.1" class="ltx_p">0.2521</span>
</span>
</td>
<td id="S2.T1.2.9.3" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:72.3pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.9.3.1" class="ltx_inline-block ltx_align_top" style="background-color:#CCFFCC;">
<span id="S2.T1.2.9.3.1.1" class="ltx_p">506,625,211,049</span>
</span>
</td>
<td id="S2.T1.2.9.4" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:36.1pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.9.4.1" class="ltx_inline-block ltx_align_top" style="background-color:#CCFFCC;">
<span id="S2.T1.2.9.4.1.1" class="ltx_p">0.72</span>
</span>
</td>
<td id="S2.T1.2.9.5" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:36.1pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.9.5.1" class="ltx_inline-block ltx_align_top" style="background-color:#CCFFCC;">
<span id="S2.T1.2.9.5.1.1" class="ltx_p">Web</span>
</span>
</td>
</tr>
<tr id="S2.T1.2.10" class="ltx_tr" style="background-color:#CCFFCC;">
<td id="S2.T1.2.10.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:72.3pt;padding-bottom:0.86108pt;"><span id="S2.T1.2.10.1.1" class="ltx_text ltx_font_bold" style="background-color:#CCFFCC;">Cultura-X (ES)</span></td>
<td id="S2.T1.2.10.2" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.10.2.1" class="ltx_inline-block ltx_align_top" style="background-color:#CCFFCC;">
<span id="S2.T1.2.10.2.1.1" class="ltx_p">0.0155</span>
</span>
</td>
<td id="S2.T1.2.10.3" class="ltx_td ltx_align_justify ltx_align_middle" style="width:72.3pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.10.3.1" class="ltx_inline-block ltx_align_top" style="background-color:#CCFFCC;">
<span id="S2.T1.2.10.3.1.1" class="ltx_p">31,241,701,156</span>
</span>
</td>
<td id="S2.T1.2.10.4" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.10.4.1" class="ltx_inline-block ltx_align_top" style="background-color:#CCFFCC;">
<span id="S2.T1.2.10.4.1.1" class="ltx_p">0.4</span>
</span>
</td>
<td id="S2.T1.2.10.5" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.10.5.1" class="ltx_inline-block ltx_align_top" style="background-color:#CCFFCC;">
<span id="S2.T1.2.10.5.1.1" class="ltx_p">Web</span>
</span>
</td>
</tr>
<tr id="S2.T1.2.11" class="ltx_tr" style="background-color:#CCFFCC;">
<td id="S2.T1.2.11.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:72.3pt;padding-bottom:0.86108pt;"><span id="S2.T1.2.11.1.1" class="ltx_text ltx_font_bold" style="background-color:#CCFFCC;">Cultura-X (DE)</span></td>
<td id="S2.T1.2.11.2" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.11.2.1" class="ltx_inline-block ltx_align_top" style="background-color:#CCFFCC;">
<span id="S2.T1.2.11.2.1.1" class="ltx_p">0.0152</span>
</span>
</td>
<td id="S2.T1.2.11.3" class="ltx_td ltx_align_justify ltx_align_middle" style="width:72.3pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.11.3.1" class="ltx_inline-block ltx_align_top" style="background-color:#CCFFCC;">
<span id="S2.T1.2.11.3.1.1" class="ltx_p">30,628,813,934</span>
</span>
</td>
<td id="S2.T1.2.11.4" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.11.4.1" class="ltx_inline-block ltx_align_top" style="background-color:#CCFFCC;">
<span id="S2.T1.2.11.4.1.1" class="ltx_p">0.32</span>
</span>
</td>
<td id="S2.T1.2.11.5" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.11.5.1" class="ltx_inline-block ltx_align_top" style="background-color:#CCFFCC;">
<span id="S2.T1.2.11.5.1.1" class="ltx_p">Web</span>
</span>
</td>
</tr>
<tr id="S2.T1.2.12" class="ltx_tr" style="background-color:#CCFFCC;">
<td id="S2.T1.2.12.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:72.3pt;padding-bottom:0.86108pt;"><span id="S2.T1.2.12.1.1" class="ltx_text ltx_font_bold" style="background-color:#CCFFCC;">Cultura-X (FR)</span></td>
<td id="S2.T1.2.12.2" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.12.2.1" class="ltx_inline-block ltx_align_top" style="background-color:#CCFFCC;">
<span id="S2.T1.2.12.2.1.1" class="ltx_p">0.0097</span>
</span>
</td>
<td id="S2.T1.2.12.3" class="ltx_td ltx_align_justify ltx_align_middle" style="width:72.3pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.12.3.1" class="ltx_inline-block ltx_align_top" style="background-color:#CCFFCC;">
<span id="S2.T1.2.12.3.1.1" class="ltx_p">19,466,611,808</span>
</span>
</td>
<td id="S2.T1.2.12.4" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.12.4.1" class="ltx_inline-block ltx_align_top" style="background-color:#CCFFCC;">
<span id="S2.T1.2.12.4.1.1" class="ltx_p">0.26</span>
</span>
</td>
<td id="S2.T1.2.12.5" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.12.5.1" class="ltx_inline-block ltx_align_top" style="background-color:#CCFFCC;">
<span id="S2.T1.2.12.5.1.1" class="ltx_p">Web</span>
</span>
</td>
</tr>
<tr id="S2.T1.2.13" class="ltx_tr" style="background-color:#CCFFCC;">
<td id="S2.T1.2.13.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:72.3pt;padding-bottom:0.86108pt;"><span id="S2.T1.2.13.1.1" class="ltx_text ltx_font_bold" style="background-color:#CCFFCC;">Cultura-X (IT)</span></td>
<td id="S2.T1.2.13.2" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.13.2.1" class="ltx_inline-block ltx_align_top" style="background-color:#CCFFCC;">
<span id="S2.T1.2.13.2.1.1" class="ltx_p">0.0096</span>
</span>
</td>
<td id="S2.T1.2.13.3" class="ltx_td ltx_align_justify ltx_align_middle" style="width:72.3pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.13.3.1" class="ltx_inline-block ltx_align_top" style="background-color:#CCFFCC;">
<span id="S2.T1.2.13.3.1.1" class="ltx_p">19,202,903,491</span>
</span>
</td>
<td id="S2.T1.2.13.4" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.13.4.1" class="ltx_inline-block ltx_align_top" style="background-color:#CCFFCC;">
<span id="S2.T1.2.13.4.1.1" class="ltx_p">0.4</span>
</span>
</td>
<td id="S2.T1.2.13.5" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.13.5.1" class="ltx_inline-block ltx_align_top" style="background-color:#CCFFCC;">
<span id="S2.T1.2.13.5.1.1" class="ltx_p">Web</span>
</span>
</td>
</tr>
<tr id="S2.T1.2.14" class="ltx_tr" style="background-color:#CCFFCC;">
<td id="S2.T1.2.14.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:72.3pt;padding-bottom:0.86108pt;"><span id="S2.T1.2.14.1.1" class="ltx_text ltx_font_bold" style="background-color:#CCFFCC;">Cultura-X (NL)</span></td>
<td id="S2.T1.2.14.2" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.14.2.1" class="ltx_inline-block ltx_align_top" style="background-color:#CCFFCC;">
<span id="S2.T1.2.14.2.1.1" class="ltx_p">0.0097</span>
</span>
</td>
<td id="S2.T1.2.14.3" class="ltx_td ltx_align_justify ltx_align_middle" style="width:72.3pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.14.3.1" class="ltx_inline-block ltx_align_top" style="background-color:#CCFFCC;">
<span id="S2.T1.2.14.3.1.1" class="ltx_p">19,511,322,386</span>
</span>
</td>
<td id="S2.T1.2.14.4" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.14.4.1" class="ltx_inline-block ltx_align_top" style="background-color:#CCFFCC;">
<span id="S2.T1.2.14.4.1.1" class="ltx_p">0.62</span>
</span>
</td>
<td id="S2.T1.2.14.5" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.14.5.1" class="ltx_inline-block ltx_align_top" style="background-color:#CCFFCC;">
<span id="S2.T1.2.14.5.1.1" class="ltx_p">Web</span>
</span>
</td>
</tr>
<tr id="S2.T1.2.15" class="ltx_tr" style="background-color:#CCFFCC;">
<td id="S2.T1.2.15.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:72.3pt;padding-bottom:0.86108pt;"><span id="S2.T1.2.15.1.1" class="ltx_text ltx_font_bold" style="background-color:#CCFFCC;">Cultura-X (PT)</span></td>
<td id="S2.T1.2.15.2" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.15.2.1" class="ltx_inline-block ltx_align_top" style="background-color:#CCFFCC;">
<span id="S2.T1.2.15.2.1.1" class="ltx_p">0.01</span>
</span>
</td>
<td id="S2.T1.2.15.3" class="ltx_td ltx_align_justify ltx_align_middle" style="width:72.3pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.15.3.1" class="ltx_inline-block ltx_align_top" style="background-color:#CCFFCC;">
<span id="S2.T1.2.15.3.1.1" class="ltx_p">20,063,968,694</span>
</span>
</td>
<td id="S2.T1.2.15.4" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.15.4.1" class="ltx_inline-block ltx_align_top" style="background-color:#CCFFCC;">
<span id="S2.T1.2.15.4.1.1" class="ltx_p">0.78</span>
</span>
</td>
<td id="S2.T1.2.15.5" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.15.5.1" class="ltx_inline-block ltx_align_top" style="background-color:#CCFFCC;">
<span id="S2.T1.2.15.5.1.1" class="ltx_p">Web</span>
</span>
</td>
</tr>
<tr id="S2.T1.2.16" class="ltx_tr" style="background-color:#CCFFCC;">
<td id="S2.T1.2.16.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:72.3pt;padding-bottom:0.86108pt;"><span id="S2.T1.2.16.1.1" class="ltx_text ltx_font_bold" style="background-color:#CCFFCC;">C4</span></td>
<td id="S2.T1.2.16.2" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.16.2.1" class="ltx_inline-block ltx_align_top" style="background-color:#CCFFCC;">
<span id="S2.T1.2.16.2.1.1" class="ltx_p">0.0855</span>
</span>
</td>
<td id="S2.T1.2.16.3" class="ltx_td ltx_align_justify ltx_align_middle" style="width:72.3pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.16.3.1" class="ltx_inline-block ltx_align_top" style="background-color:#CCFFCC;">
<span id="S2.T1.2.16.3.1.1" class="ltx_p">171,782,178,108</span>
</span>
</td>
<td id="S2.T1.2.16.4" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.16.4.1" class="ltx_inline-block ltx_align_top" style="background-color:#CCFFCC;">
<span id="S2.T1.2.16.4.1.1" class="ltx_p">1.0</span>
</span>
</td>
<td id="S2.T1.2.16.5" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.16.5.1" class="ltx_inline-block ltx_align_top" style="background-color:#CCFFCC;">
<span id="S2.T1.2.16.5.1.1" class="ltx_p">Web</span>
</span>
</td>
</tr>
<tr id="S2.T1.2.17" class="ltx_tr" style="background-color:#CCFFCC;">
<td id="S2.T1.2.17.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:72.3pt;padding-bottom:0.86108pt;"><span id="S2.T1.2.17.1.1" class="ltx_text ltx_font_bold" style="background-color:#CCFFCC;">OpenWebText2</span></td>
<td id="S2.T1.2.17.2" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.17.2.1" class="ltx_inline-block ltx_align_top" style="background-color:#CCFFCC;">
<span id="S2.T1.2.17.2.1.1" class="ltx_p">0.0130</span>
</span>
</td>
<td id="S2.T1.2.17.3" class="ltx_td ltx_align_justify ltx_align_middle" style="width:72.3pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.17.3.1" class="ltx_inline-block ltx_align_top" style="background-color:#CCFFCC;">
<span id="S2.T1.2.17.3.1.1" class="ltx_p">26,161,864,434</span>
</span>
</td>
<td id="S2.T1.2.17.4" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.17.4.1" class="ltx_inline-block ltx_align_top" style="background-color:#CCFFCC;">
<span id="S2.T1.2.17.4.1.1" class="ltx_p">3.0</span>
</span>
</td>
<td id="S2.T1.2.17.5" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.17.5.1" class="ltx_inline-block ltx_align_top" style="background-color:#CCFFCC;">
<span id="S2.T1.2.17.5.1.1" class="ltx_p">Web</span>
</span>
</td>
</tr>
<tr id="S2.T1.2.18" class="ltx_tr" style="background-color:#CCFFCC;">
<td id="S2.T1.2.18.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:72.3pt;padding-bottom:0.86108pt;"><span id="S2.T1.2.18.1.1" class="ltx_text ltx_font_bold" style="background-color:#CCFFCC;">RefinedWeb</span></td>
<td id="S2.T1.2.18.2" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.18.2.1" class="ltx_inline-block ltx_align_top" style="background-color:#CCFFCC;">
<span id="S2.T1.2.18.2.1.1" class="ltx_p">0.3292</span>
</span>
</td>
<td id="S2.T1.2.18.3" class="ltx_td ltx_align_justify ltx_align_middle" style="width:72.3pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.18.3.1" class="ltx_inline-block ltx_align_top" style="background-color:#CCFFCC;">
<span id="S2.T1.2.18.3.1.1" class="ltx_p">661,591,178,339</span>
</span>
</td>
<td id="S2.T1.2.18.4" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.18.4.1" class="ltx_inline-block ltx_align_top" style="background-color:#CCFFCC;">
<span id="S2.T1.2.18.4.1.1" class="ltx_p">1.15</span>
</span>
</td>
<td id="S2.T1.2.18.5" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.18.5.1" class="ltx_inline-block ltx_align_top" style="background-color:#CCFFCC;">
<span id="S2.T1.2.18.5.1.1" class="ltx_p">Web</span>
</span>
</td>
</tr>
<tr id="S2.T1.2.19" class="ltx_tr" style="background-color:#D0BBFF;">
<td id="S2.T1.2.19.1" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:72.3pt;padding-bottom:0.86108pt;"><span id="S2.T1.2.19.1.1" class="ltx_text ltx_font_bold" style="background-color:#D0BBFF;">StackExchange</span></td>
<td id="S2.T1.2.19.2" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:36.1pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.19.2.1" class="ltx_inline-block ltx_align_top" style="background-color:#D0BBFF;">
<span id="S2.T1.2.19.2.1.1" class="ltx_p">0.0231</span>
</span>
</td>
<td id="S2.T1.2.19.3" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:72.3pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.19.3.1" class="ltx_inline-block ltx_align_top" style="background-color:#D0BBFF;">
<span id="S2.T1.2.19.3.1.1" class="ltx_p">46,302,993,820</span>
</span>
</td>
<td id="S2.T1.2.19.4" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:36.1pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.19.4.1" class="ltx_inline-block ltx_align_top" style="background-color:#D0BBFF;">
<span id="S2.T1.2.19.4.1.1" class="ltx_p">2.5</span>
</span>
</td>
<td id="S2.T1.2.19.5" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:36.1pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.19.5.1" class="ltx_inline-block ltx_align_top" style="background-color:#D0BBFF;">
<span id="S2.T1.2.19.5.1.1" class="ltx_p">Social</span>
</span>
</td>
</tr>
<tr id="S2.T1.2.20" class="ltx_tr" style="background-color:#D0BBFF;">
<td id="S2.T1.2.20.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:72.3pt;padding-bottom:0.86108pt;"><span id="S2.T1.2.20.1.1" class="ltx_text ltx_font_bold" style="background-color:#D0BBFF;">HackerNews</span></td>
<td id="S2.T1.2.20.2" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.20.2.1" class="ltx_inline-block ltx_align_top" style="background-color:#D0BBFF;">
<span id="S2.T1.2.20.2.1.1" class="ltx_p">0.0019</span>
</span>
</td>
<td id="S2.T1.2.20.3" class="ltx_td ltx_align_justify ltx_align_middle" style="width:72.3pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.20.3.1" class="ltx_inline-block ltx_align_top" style="background-color:#D0BBFF;">
<span id="S2.T1.2.20.3.1.1" class="ltx_p">3,817,060,582</span>
</span>
</td>
<td id="S2.T1.2.20.4" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.20.4.1" class="ltx_inline-block ltx_align_top" style="background-color:#D0BBFF;">
<span id="S2.T1.2.20.4.1.1" class="ltx_p">2.0</span>
</span>
</td>
<td id="S2.T1.2.20.5" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.20.5.1" class="ltx_inline-block ltx_align_top" style="background-color:#D0BBFF;">
<span id="S2.T1.2.20.5.1.1" class="ltx_p">Social</span>
</span>
</td>
</tr>
<tr id="S2.T1.2.21" class="ltx_tr" style="background-color:#FFB5C1;">
<td id="S2.T1.2.21.1" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:72.3pt;padding-bottom:0.86108pt;"><span id="S2.T1.2.21.1.1" class="ltx_text ltx_font_bold" style="background-color:#FFB5C1;">EuroParl</span></td>
<td id="S2.T1.2.21.2" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:36.1pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.21.2.1" class="ltx_inline-block ltx_align_top" style="background-color:#FFB5C1;">
<span id="S2.T1.2.21.2.1.1" class="ltx_p">0.0023</span>
</span>
</td>
<td id="S2.T1.2.21.3" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:72.3pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.21.3.1" class="ltx_inline-block ltx_align_top" style="background-color:#FFB5C1;">
<span id="S2.T1.2.21.3.1.1" class="ltx_p">4,678,506,882</span>
</span>
</td>
<td id="S2.T1.2.21.4" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:36.1pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.21.4.1" class="ltx_inline-block ltx_align_top" style="background-color:#FFB5C1;">
<span id="S2.T1.2.21.4.1.1" class="ltx_p">3.0</span>
</span>
</td>
<td id="S2.T1.2.21.5" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:36.1pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.21.5.1" class="ltx_inline-block ltx_align_top" style="background-color:#FFB5C1;">
<span id="S2.T1.2.21.5.1.1" class="ltx_p">Law</span>
</span>
</td>
</tr>
<tr id="S2.T1.2.22" class="ltx_tr" style="background-color:#FFB5C1;">
<td id="S2.T1.2.22.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:72.3pt;padding-bottom:0.86108pt;"><span id="S2.T1.2.22.1.1" class="ltx_text ltx_font_bold" style="background-color:#FFB5C1;">FreeLaw</span></td>
<td id="S2.T1.2.22.2" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.22.2.1" class="ltx_inline-block ltx_align_top" style="background-color:#FFB5C1;">
<span id="S2.T1.2.22.2.1.1" class="ltx_p">0.0088</span>
</span>
</td>
<td id="S2.T1.2.22.3" class="ltx_td ltx_align_justify ltx_align_middle" style="width:72.3pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.22.3.1" class="ltx_inline-block ltx_align_top" style="background-color:#FFB5C1;">
<span id="S2.T1.2.22.3.1.1" class="ltx_p">17,694,697,577</span>
</span>
</td>
<td id="S2.T1.2.22.4" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.22.4.1" class="ltx_inline-block ltx_align_top" style="background-color:#FFB5C1;">
<span id="S2.T1.2.22.4.1.1" class="ltx_p">1.2</span>
</span>
</td>
<td id="S2.T1.2.22.5" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.22.5.1" class="ltx_inline-block ltx_align_top" style="background-color:#FFB5C1;">
<span id="S2.T1.2.22.5.1.1" class="ltx_p">Law</span>
</span>
</td>
</tr>
<tr id="S2.T1.2.23" class="ltx_tr" style="background-color:#FFB5C1;">
<td id="S2.T1.2.23.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:72.3pt;padding-bottom:0.86108pt;"><span id="S2.T1.2.23.1.1" class="ltx_text ltx_font_bold" style="background-color:#FFB5C1;">PileOfLaw</span></td>
<td id="S2.T1.2.23.2" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.23.2.1" class="ltx_inline-block ltx_align_top" style="background-color:#FFB5C1;">
<span id="S2.T1.2.23.2.1.1" class="ltx_p">0.0063</span>
</span>
</td>
<td id="S2.T1.2.23.3" class="ltx_td ltx_align_justify ltx_align_middle" style="width:72.3pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.23.3.1" class="ltx_inline-block ltx_align_top" style="background-color:#FFB5C1;">
<span id="S2.T1.2.23.3.1.1" class="ltx_p">12,663,061,642</span>
</span>
</td>
<td id="S2.T1.2.23.4" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.23.4.1" class="ltx_inline-block ltx_align_top" style="background-color:#FFB5C1;">
<span id="S2.T1.2.23.4.1.1" class="ltx_p">0.75</span>
</span>
</td>
<td id="S2.T1.2.23.5" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.23.5.1" class="ltx_inline-block ltx_align_top" style="background-color:#FFB5C1;">
<span id="S2.T1.2.23.5.1.1" class="ltx_p">Law</span>
</span>
</td>
</tr>
<tr id="S2.T1.2.24" class="ltx_tr" style="background-color:#E0FFFF;">
<td id="S2.T1.2.24.1" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:72.3pt;padding-bottom:0.86108pt;"><span id="S2.T1.2.24.1.1" class="ltx_text ltx_font_bold" style="background-color:#E0FFFF;">DM Math</span></td>
<td id="S2.T1.2.24.2" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:36.1pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.24.2.1" class="ltx_inline-block ltx_align_top" style="background-color:#E0FFFF;">
<span id="S2.T1.2.24.2.1.1" class="ltx_p">0.0066</span>
</span>
</td>
<td id="S2.T1.2.24.3" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:72.3pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.24.3.1" class="ltx_inline-block ltx_align_top" style="background-color:#E0FFFF;">
<span id="S2.T1.2.24.3.1.1" class="ltx_p">13,321,872,138</span>
</span>
</td>
<td id="S2.T1.2.24.4" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:36.1pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.24.4.1" class="ltx_inline-block ltx_align_top" style="background-color:#E0FFFF;">
<span id="S2.T1.2.24.4.1.1" class="ltx_p">3.0</span>
</span>
</td>
<td id="S2.T1.2.24.5" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:36.1pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.24.5.1" class="ltx_inline-block ltx_align_top" style="background-color:#E0FFFF;">
<span id="S2.T1.2.24.5.1.1" class="ltx_p">Math</span>
</span>
</td>
</tr>
<tr id="S2.T1.2.25" class="ltx_tr" style="background-color:#E0FFFF;">
<td id="S2.T1.2.25.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:72.3pt;padding-bottom:0.86108pt;"><span id="S2.T1.2.25.1.1" class="ltx_text ltx_font_bold" style="background-color:#E0FFFF;">AMPS</span></td>
<td id="S2.T1.2.25.2" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.25.2.1" class="ltx_inline-block ltx_align_top" style="background-color:#E0FFFF;">
<span id="S2.T1.2.25.2.1.1" class="ltx_p">0.0011</span>
</span>
</td>
<td id="S2.T1.2.25.3" class="ltx_td ltx_align_justify ltx_align_middle" style="width:72.3pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.25.3.1" class="ltx_inline-block ltx_align_top" style="background-color:#E0FFFF;">
<span id="S2.T1.2.25.3.1.1" class="ltx_p">2,126,719,278</span>
</span>
</td>
<td id="S2.T1.2.25.4" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.25.4.1" class="ltx_inline-block ltx_align_top" style="background-color:#E0FFFF;">
<span id="S2.T1.2.25.4.1.1" class="ltx_p">6.0</span>
</span>
</td>
<td id="S2.T1.2.25.5" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.25.5.1" class="ltx_inline-block ltx_align_top" style="background-color:#E0FFFF;">
<span id="S2.T1.2.25.5.1.1" class="ltx_p">Math</span>
</span>
</td>
</tr>
<tr id="S2.T1.2.26" class="ltx_tr" style="background-color:#E0FFFF;">
<td id="S2.T1.2.26.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:72.3pt;padding-bottom:0.86108pt;"><span id="S2.T1.2.26.1.1" class="ltx_text ltx_font_bold" style="background-color:#E0FFFF;">OpenWebMath</span></td>
<td id="S2.T1.2.26.2" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.26.2.1" class="ltx_inline-block ltx_align_top" style="background-color:#E0FFFF;">
<span id="S2.T1.2.26.2.1.1" class="ltx_p">0.0132</span>
</span>
</td>
<td id="S2.T1.2.26.3" class="ltx_td ltx_align_justify ltx_align_middle" style="width:72.3pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.26.3.1" class="ltx_inline-block ltx_align_top" style="background-color:#E0FFFF;">
<span id="S2.T1.2.26.3.1.1" class="ltx_p">26,530,341,292</span>
</span>
</td>
<td id="S2.T1.2.26.4" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.26.4.1" class="ltx_inline-block ltx_align_top" style="background-color:#E0FFFF;">
<span id="S2.T1.2.26.4.1.1" class="ltx_p">2.0</span>
</span>
</td>
<td id="S2.T1.2.26.5" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.26.5.1" class="ltx_inline-block ltx_align_top" style="background-color:#E0FFFF;">
<span id="S2.T1.2.26.5.1.1" class="ltx_p">Math</span>
</span>
</td>
</tr>
<tr id="S2.T1.2.27" class="ltx_tr" style="background-color:#FAB0E4;">
<td id="S2.T1.2.27.1" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:72.3pt;padding-bottom:0.86108pt;"><span id="S2.T1.2.27.1.1" class="ltx_text ltx_font_bold" style="background-color:#FAB0E4;">RedPajama Wiki</span></td>
<td id="S2.T1.2.27.2" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:36.1pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.27.2.1" class="ltx_inline-block ltx_align_top" style="background-color:#FAB0E4;">
<span id="S2.T1.2.27.2.1.1" class="ltx_p">0.0363</span>
</span>
</td>
<td id="S2.T1.2.27.3" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:72.3pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.27.3.1" class="ltx_inline-block ltx_align_top" style="background-color:#FAB0E4;">
<span id="S2.T1.2.27.3.1.1" class="ltx_p">72,866,870,472</span>
</span>
</td>
<td id="S2.T1.2.27.4" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:36.1pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.27.4.1" class="ltx_inline-block ltx_align_top" style="background-color:#FAB0E4;">
<span id="S2.T1.2.27.4.1.1" class="ltx_p">3.0</span>
</span>
</td>
<td id="S2.T1.2.27.5" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:36.1pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.27.5.1" class="ltx_inline-block ltx_align_top" style="background-color:#FAB0E4;">
<span id="S2.T1.2.27.5.1.1" class="ltx_p">Wiki</span>
</span>
</td>
</tr>
<tr id="S2.T1.2.28" class="ltx_tr" style="background-color:#FFFEA3;">
<td id="S2.T1.2.28.1" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:72.3pt;padding-bottom:0.86108pt;"><span id="S2.T1.2.28.1.1" class="ltx_text ltx_font_bold" style="background-color:#FFFEA3;">Starcoder</span></td>
<td id="S2.T1.2.28.2" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:36.1pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.28.2.1" class="ltx_inline-block ltx_align_top" style="background-color:#FFFEA3;">
<span id="S2.T1.2.28.2.1.1" class="ltx_p">0.0724</span>
</span>
</td>
<td id="S2.T1.2.28.3" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:72.3pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.28.3.1" class="ltx_inline-block ltx_align_top" style="background-color:#FFFEA3;">
<span id="S2.T1.2.28.3.1.1" class="ltx_p">145,586,775,301</span>
</span>
</td>
<td id="S2.T1.2.28.4" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:36.1pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.28.4.1" class="ltx_inline-block ltx_align_top" style="background-color:#FFFEA3;">
<span id="S2.T1.2.28.4.1.1" class="ltx_p">0.74</span>
</span>
</td>
<td id="S2.T1.2.28.5" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:36.1pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.28.5.1" class="ltx_inline-block ltx_align_top" style="background-color:#FFFEA3;">
<span id="S2.T1.2.28.5.1.1" class="ltx_p">Code</span>
</span>
</td>
</tr>
<tr id="S2.T1.2.29" class="ltx_tr" style="background-color:#DEBB9B;">
<td id="S2.T1.2.29.1" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:72.3pt;padding-bottom:0.86108pt;"><span id="S2.T1.2.29.1.1" class="ltx_text ltx_font_bold" style="background-color:#DEBB9B;">Restruct-v1</span></td>
<td id="S2.T1.2.29.2" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:36.1pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.29.2.1" class="ltx_inline-block ltx_align_top" style="background-color:#DEBB9B;">
<span id="S2.T1.2.29.2.1.1" class="ltx_p">0.0102</span>
</span>
</td>
<td id="S2.T1.2.29.3" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:72.3pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.29.3.1" class="ltx_inline-block ltx_align_top" style="background-color:#DEBB9B;">
<span id="S2.T1.2.29.3.1.1" class="ltx_p">20,412,655,632</span>
</span>
</td>
<td id="S2.T1.2.29.4" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:36.1pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.29.4.1" class="ltx_inline-block ltx_align_top" style="background-color:#DEBB9B;">
<span id="S2.T1.2.29.4.1.1" class="ltx_p">3.0</span>
</span>
</td>
<td id="S2.T1.2.29.5" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:36.1pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.29.5.1" class="ltx_inline-block ltx_align_top" style="background-color:#DEBB9B;">
<span id="S2.T1.2.29.5.1.1" class="ltx_p">Instruction</span>
</span>
</td>
</tr>
<tr id="S2.T1.2.30" class="ltx_tr">
<td id="S2.T1.2.30.1" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:72.3pt;padding-bottom:0.86108pt;"><span id="S2.T1.2.30.1.1" class="ltx_text ltx_font_bold">Total</span></td>
<td id="S2.T1.2.30.2" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:36.1pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.30.2.1" class="ltx_inline-block ltx_align_top">
<span id="S2.T1.2.30.2.1.1" class="ltx_p">1</span>
</span>
</td>
<td id="S2.T1.2.30.3" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:72.3pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.30.3.1" class="ltx_inline-block ltx_align_top">
<span id="S2.T1.2.30.3.1.1" class="ltx_p">2,009,831,803,933</span>
</span>
</td>
<td id="S2.T1.2.30.4" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:36.1pt;padding-bottom:0.86108pt;"></td>
<td id="S2.T1.2.30.5" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:36.1pt;padding-bottom:0.86108pt;">
<span id="S2.T1.2.30.5.1" class="ltx_inline-block ltx_align_top">
<span id="S2.T1.2.30.5.1.1" class="ltx_p">-</span>
</span>
</td>
</tr>
</table>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table"><span id="S2.T1.4.1.1" class="ltx_text" style="font-size:90%;">Table 1</span>: </span><span id="S2.T1.5.2" class="ltx_text" style="font-size:90%;">The complete Stable LM 2 training set with sampling weights. The tokens count refers to the <span id="S2.T1.5.2.1" class="ltx_text ltx_font_bold">Arcade100k</span> tokenizer introduced in Sec. <a href="#S2.SS3" title="2.3 Tokenizer ‣ 2 Model Pre-Training ‣ Stable LM 2 1.6B Technical Report" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2.3</span></a>.
The number of tokens in the table already includes the number of epochs shown next to it.
For instance, in the case of BookCorpusOpen, we use around 9B tokens, corresponding to 6 epochs of the original dataset (that is, each epoch is around 1.5B tokens).
Similarly, if epochs are below one, the number of tokens shown is a subset of the total dataset.
</span></figcaption>
</figure>
<figure id="S2.F1" class="ltx_figure"><img src="/html/2402.17834/assets/x1.png" id="S2.F1.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="438" height="292" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S2.F1.2.1.1" class="ltx_text" style="font-size:90%;">Figure 1</span>: </span><span id="S2.F1.3.2" class="ltx_text" style="font-size:90%;">Percentage of effective training tokens by domain in the Stable LM 2 pre-training dataset.</span></figcaption>
</figure>
<section id="S2.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.1 </span>Training</h3>

<div id="S2.SS1.p1" class="ltx_para">
<p id="S2.SS1.p1.2" class="ltx_p">We train Stable LM 2 to predict the next token following standard autoregressive sequence modeling <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib58" title="" class="ltx_ref">58</a>]</cite>. We
train our model from scratch with a context length of 4096 and benefit from the efficient sequence-wise parallelism optimizations of FlashAttention-2 <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib17" title="" class="ltx_ref">17</a>, <a href="#bib.bib16" title="" class="ltx_ref">16</a>]</cite>.
Training is performed in BFloat16 mixed precision while keeping all-reduce operations in FP32. <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib10" title="" class="ltx_ref">10</a>, <a href="#bib.bib74" title="" class="ltx_ref">74</a>]</cite> found it beneficial to add a z-loss regularization term on the softmax normalizer, <math id="S2.SS1.p1.1.m1.1" class="ltx_Math" alttext="z_{\mathrm{loss}}\propto\log^{2}Z" display="inline"><semantics id="S2.SS1.p1.1.m1.1a"><mrow id="S2.SS1.p1.1.m1.1.1" xref="S2.SS1.p1.1.m1.1.1.cmml"><msub id="S2.SS1.p1.1.m1.1.1.2" xref="S2.SS1.p1.1.m1.1.1.2.cmml"><mi id="S2.SS1.p1.1.m1.1.1.2.2" xref="S2.SS1.p1.1.m1.1.1.2.2.cmml">z</mi><mi id="S2.SS1.p1.1.m1.1.1.2.3" xref="S2.SS1.p1.1.m1.1.1.2.3.cmml">loss</mi></msub><mo id="S2.SS1.p1.1.m1.1.1.1" xref="S2.SS1.p1.1.m1.1.1.1.cmml">∝</mo><mrow id="S2.SS1.p1.1.m1.1.1.3" xref="S2.SS1.p1.1.m1.1.1.3.cmml"><msup id="S2.SS1.p1.1.m1.1.1.3.1" xref="S2.SS1.p1.1.m1.1.1.3.1.cmml"><mi id="S2.SS1.p1.1.m1.1.1.3.1.2" xref="S2.SS1.p1.1.m1.1.1.3.1.2.cmml">log</mi><mn id="S2.SS1.p1.1.m1.1.1.3.1.3" xref="S2.SS1.p1.1.m1.1.1.3.1.3.cmml">2</mn></msup><mo lspace="0.167em" id="S2.SS1.p1.1.m1.1.1.3a" xref="S2.SS1.p1.1.m1.1.1.3.cmml">⁡</mo><mi id="S2.SS1.p1.1.m1.1.1.3.2" xref="S2.SS1.p1.1.m1.1.1.3.2.cmml">Z</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS1.p1.1.m1.1b"><apply id="S2.SS1.p1.1.m1.1.1.cmml" xref="S2.SS1.p1.1.m1.1.1"><csymbol cd="latexml" id="S2.SS1.p1.1.m1.1.1.1.cmml" xref="S2.SS1.p1.1.m1.1.1.1">proportional-to</csymbol><apply id="S2.SS1.p1.1.m1.1.1.2.cmml" xref="S2.SS1.p1.1.m1.1.1.2"><csymbol cd="ambiguous" id="S2.SS1.p1.1.m1.1.1.2.1.cmml" xref="S2.SS1.p1.1.m1.1.1.2">subscript</csymbol><ci id="S2.SS1.p1.1.m1.1.1.2.2.cmml" xref="S2.SS1.p1.1.m1.1.1.2.2">𝑧</ci><ci id="S2.SS1.p1.1.m1.1.1.2.3.cmml" xref="S2.SS1.p1.1.m1.1.1.2.3">loss</ci></apply><apply id="S2.SS1.p1.1.m1.1.1.3.cmml" xref="S2.SS1.p1.1.m1.1.1.3"><apply id="S2.SS1.p1.1.m1.1.1.3.1.cmml" xref="S2.SS1.p1.1.m1.1.1.3.1"><csymbol cd="ambiguous" id="S2.SS1.p1.1.m1.1.1.3.1.1.cmml" xref="S2.SS1.p1.1.m1.1.1.3.1">superscript</csymbol><log id="S2.SS1.p1.1.m1.1.1.3.1.2.cmml" xref="S2.SS1.p1.1.m1.1.1.3.1.2"></log><cn type="integer" id="S2.SS1.p1.1.m1.1.1.3.1.3.cmml" xref="S2.SS1.p1.1.m1.1.1.3.1.3">2</cn></apply><ci id="S2.SS1.p1.1.m1.1.1.3.2.cmml" xref="S2.SS1.p1.1.m1.1.1.3.2">𝑍</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p1.1.m1.1c">z_{\mathrm{loss}}\propto\log^{2}Z</annotation></semantics></math>, to mitigate instability stemming from output logits divergence.
While it did not hurt performance in our ablations, the improvements to stability were minimal.
Accordingly, it was not applied in the final run.
We employ a standard AdamW optimizer with the following hyperparameters: <math id="S2.SS1.p1.2.m2.3" class="ltx_Math" alttext="\beta_{1}=0.9,\beta_{2}=0.95,\epsilon=1e-8,\lambda(\text{weight decay})=0.1" display="inline"><semantics id="S2.SS1.p1.2.m2.3a"><mrow id="S2.SS1.p1.2.m2.3.3.2" xref="S2.SS1.p1.2.m2.3.3.3.cmml"><mrow id="S2.SS1.p1.2.m2.2.2.1.1" xref="S2.SS1.p1.2.m2.2.2.1.1.cmml"><msub id="S2.SS1.p1.2.m2.2.2.1.1.2" xref="S2.SS1.p1.2.m2.2.2.1.1.2.cmml"><mi id="S2.SS1.p1.2.m2.2.2.1.1.2.2" xref="S2.SS1.p1.2.m2.2.2.1.1.2.2.cmml">β</mi><mn id="S2.SS1.p1.2.m2.2.2.1.1.2.3" xref="S2.SS1.p1.2.m2.2.2.1.1.2.3.cmml">1</mn></msub><mo id="S2.SS1.p1.2.m2.2.2.1.1.1" xref="S2.SS1.p1.2.m2.2.2.1.1.1.cmml">=</mo><mn id="S2.SS1.p1.2.m2.2.2.1.1.3" xref="S2.SS1.p1.2.m2.2.2.1.1.3.cmml">0.9</mn></mrow><mo id="S2.SS1.p1.2.m2.3.3.2.3" xref="S2.SS1.p1.2.m2.3.3.3a.cmml">,</mo><mrow id="S2.SS1.p1.2.m2.3.3.2.2.2" xref="S2.SS1.p1.2.m2.3.3.2.2.3.cmml"><mrow id="S2.SS1.p1.2.m2.3.3.2.2.1.1" xref="S2.SS1.p1.2.m2.3.3.2.2.1.1.cmml"><msub id="S2.SS1.p1.2.m2.3.3.2.2.1.1.2" xref="S2.SS1.p1.2.m2.3.3.2.2.1.1.2.cmml"><mi id="S2.SS1.p1.2.m2.3.3.2.2.1.1.2.2" xref="S2.SS1.p1.2.m2.3.3.2.2.1.1.2.2.cmml">β</mi><mn id="S2.SS1.p1.2.m2.3.3.2.2.1.1.2.3" xref="S2.SS1.p1.2.m2.3.3.2.2.1.1.2.3.cmml">2</mn></msub><mo id="S2.SS1.p1.2.m2.3.3.2.2.1.1.1" xref="S2.SS1.p1.2.m2.3.3.2.2.1.1.1.cmml">=</mo><mn id="S2.SS1.p1.2.m2.3.3.2.2.1.1.3" xref="S2.SS1.p1.2.m2.3.3.2.2.1.1.3.cmml">0.95</mn></mrow><mo id="S2.SS1.p1.2.m2.3.3.2.2.2.3" xref="S2.SS1.p1.2.m2.3.3.2.2.3a.cmml">,</mo><mrow id="S2.SS1.p1.2.m2.3.3.2.2.2.2.2" xref="S2.SS1.p1.2.m2.3.3.2.2.2.2.3.cmml"><mrow id="S2.SS1.p1.2.m2.3.3.2.2.2.2.1.1" xref="S2.SS1.p1.2.m2.3.3.2.2.2.2.1.1.cmml"><mi id="S2.SS1.p1.2.m2.3.3.2.2.2.2.1.1.2" xref="S2.SS1.p1.2.m2.3.3.2.2.2.2.1.1.2.cmml">ϵ</mi><mo id="S2.SS1.p1.2.m2.3.3.2.2.2.2.1.1.1" xref="S2.SS1.p1.2.m2.3.3.2.2.2.2.1.1.1.cmml">=</mo><mrow id="S2.SS1.p1.2.m2.3.3.2.2.2.2.1.1.3" xref="S2.SS1.p1.2.m2.3.3.2.2.2.2.1.1.3.cmml"><mrow id="S2.SS1.p1.2.m2.3.3.2.2.2.2.1.1.3.2" xref="S2.SS1.p1.2.m2.3.3.2.2.2.2.1.1.3.2.cmml"><mn id="S2.SS1.p1.2.m2.3.3.2.2.2.2.1.1.3.2.2" xref="S2.SS1.p1.2.m2.3.3.2.2.2.2.1.1.3.2.2.cmml">1</mn><mo lspace="0em" rspace="0em" id="S2.SS1.p1.2.m2.3.3.2.2.2.2.1.1.3.2.1" xref="S2.SS1.p1.2.m2.3.3.2.2.2.2.1.1.3.2.1.cmml">​</mo><mi id="S2.SS1.p1.2.m2.3.3.2.2.2.2.1.1.3.2.3" xref="S2.SS1.p1.2.m2.3.3.2.2.2.2.1.1.3.2.3.cmml">e</mi></mrow><mo id="S2.SS1.p1.2.m2.3.3.2.2.2.2.1.1.3.1" xref="S2.SS1.p1.2.m2.3.3.2.2.2.2.1.1.3.1.cmml">−</mo><mn id="S2.SS1.p1.2.m2.3.3.2.2.2.2.1.1.3.3" xref="S2.SS1.p1.2.m2.3.3.2.2.2.2.1.1.3.3.cmml">8</mn></mrow></mrow><mo id="S2.SS1.p1.2.m2.3.3.2.2.2.2.2.3" xref="S2.SS1.p1.2.m2.3.3.2.2.2.2.3a.cmml">,</mo><mrow id="S2.SS1.p1.2.m2.3.3.2.2.2.2.2.2" xref="S2.SS1.p1.2.m2.3.3.2.2.2.2.2.2.cmml"><mrow id="S2.SS1.p1.2.m2.3.3.2.2.2.2.2.2.2" xref="S2.SS1.p1.2.m2.3.3.2.2.2.2.2.2.2.cmml"><mi id="S2.SS1.p1.2.m2.3.3.2.2.2.2.2.2.2.2" xref="S2.SS1.p1.2.m2.3.3.2.2.2.2.2.2.2.2.cmml">λ</mi><mo lspace="0em" rspace="0em" id="S2.SS1.p1.2.m2.3.3.2.2.2.2.2.2.2.1" xref="S2.SS1.p1.2.m2.3.3.2.2.2.2.2.2.2.1.cmml">​</mo><mrow id="S2.SS1.p1.2.m2.3.3.2.2.2.2.2.2.2.3.2" xref="S2.SS1.p1.2.m2.1.1a.cmml"><mo stretchy="false" id="S2.SS1.p1.2.m2.3.3.2.2.2.2.2.2.2.3.2.1" xref="S2.SS1.p1.2.m2.1.1a.cmml">(</mo><mtext id="S2.SS1.p1.2.m2.1.1" xref="S2.SS1.p1.2.m2.1.1.cmml">weight decay</mtext><mo stretchy="false" id="S2.SS1.p1.2.m2.3.3.2.2.2.2.2.2.2.3.2.2" xref="S2.SS1.p1.2.m2.1.1a.cmml">)</mo></mrow></mrow><mo id="S2.SS1.p1.2.m2.3.3.2.2.2.2.2.2.1" xref="S2.SS1.p1.2.m2.3.3.2.2.2.2.2.2.1.cmml">=</mo><mn id="S2.SS1.p1.2.m2.3.3.2.2.2.2.2.2.3" xref="S2.SS1.p1.2.m2.3.3.2.2.2.2.2.2.3.cmml">0.1</mn></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS1.p1.2.m2.3b"><apply id="S2.SS1.p1.2.m2.3.3.3.cmml" xref="S2.SS1.p1.2.m2.3.3.2"><csymbol cd="ambiguous" id="S2.SS1.p1.2.m2.3.3.3a.cmml" xref="S2.SS1.p1.2.m2.3.3.2.3">formulae-sequence</csymbol><apply id="S2.SS1.p1.2.m2.2.2.1.1.cmml" xref="S2.SS1.p1.2.m2.2.2.1.1"><eq id="S2.SS1.p1.2.m2.2.2.1.1.1.cmml" xref="S2.SS1.p1.2.m2.2.2.1.1.1"></eq><apply id="S2.SS1.p1.2.m2.2.2.1.1.2.cmml" xref="S2.SS1.p1.2.m2.2.2.1.1.2"><csymbol cd="ambiguous" id="S2.SS1.p1.2.m2.2.2.1.1.2.1.cmml" xref="S2.SS1.p1.2.m2.2.2.1.1.2">subscript</csymbol><ci id="S2.SS1.p1.2.m2.2.2.1.1.2.2.cmml" xref="S2.SS1.p1.2.m2.2.2.1.1.2.2">𝛽</ci><cn type="integer" id="S2.SS1.p1.2.m2.2.2.1.1.2.3.cmml" xref="S2.SS1.p1.2.m2.2.2.1.1.2.3">1</cn></apply><cn type="float" id="S2.SS1.p1.2.m2.2.2.1.1.3.cmml" xref="S2.SS1.p1.2.m2.2.2.1.1.3">0.9</cn></apply><apply id="S2.SS1.p1.2.m2.3.3.2.2.3.cmml" xref="S2.SS1.p1.2.m2.3.3.2.2.2"><csymbol cd="ambiguous" id="S2.SS1.p1.2.m2.3.3.2.2.3a.cmml" xref="S2.SS1.p1.2.m2.3.3.2.2.2.3">formulae-sequence</csymbol><apply id="S2.SS1.p1.2.m2.3.3.2.2.1.1.cmml" xref="S2.SS1.p1.2.m2.3.3.2.2.1.1"><eq id="S2.SS1.p1.2.m2.3.3.2.2.1.1.1.cmml" xref="S2.SS1.p1.2.m2.3.3.2.2.1.1.1"></eq><apply id="S2.SS1.p1.2.m2.3.3.2.2.1.1.2.cmml" xref="S2.SS1.p1.2.m2.3.3.2.2.1.1.2"><csymbol cd="ambiguous" id="S2.SS1.p1.2.m2.3.3.2.2.1.1.2.1.cmml" xref="S2.SS1.p1.2.m2.3.3.2.2.1.1.2">subscript</csymbol><ci id="S2.SS1.p1.2.m2.3.3.2.2.1.1.2.2.cmml" xref="S2.SS1.p1.2.m2.3.3.2.2.1.1.2.2">𝛽</ci><cn type="integer" id="S2.SS1.p1.2.m2.3.3.2.2.1.1.2.3.cmml" xref="S2.SS1.p1.2.m2.3.3.2.2.1.1.2.3">2</cn></apply><cn type="float" id="S2.SS1.p1.2.m2.3.3.2.2.1.1.3.cmml" xref="S2.SS1.p1.2.m2.3.3.2.2.1.1.3">0.95</cn></apply><apply id="S2.SS1.p1.2.m2.3.3.2.2.2.2.3.cmml" xref="S2.SS1.p1.2.m2.3.3.2.2.2.2.2"><csymbol cd="ambiguous" id="S2.SS1.p1.2.m2.3.3.2.2.2.2.3a.cmml" xref="S2.SS1.p1.2.m2.3.3.2.2.2.2.2.3">formulae-sequence</csymbol><apply id="S2.SS1.p1.2.m2.3.3.2.2.2.2.1.1.cmml" xref="S2.SS1.p1.2.m2.3.3.2.2.2.2.1.1"><eq id="S2.SS1.p1.2.m2.3.3.2.2.2.2.1.1.1.cmml" xref="S2.SS1.p1.2.m2.3.3.2.2.2.2.1.1.1"></eq><ci id="S2.SS1.p1.2.m2.3.3.2.2.2.2.1.1.2.cmml" xref="S2.SS1.p1.2.m2.3.3.2.2.2.2.1.1.2">italic-ϵ</ci><apply id="S2.SS1.p1.2.m2.3.3.2.2.2.2.1.1.3.cmml" xref="S2.SS1.p1.2.m2.3.3.2.2.2.2.1.1.3"><minus id="S2.SS1.p1.2.m2.3.3.2.2.2.2.1.1.3.1.cmml" xref="S2.SS1.p1.2.m2.3.3.2.2.2.2.1.1.3.1"></minus><apply id="S2.SS1.p1.2.m2.3.3.2.2.2.2.1.1.3.2.cmml" xref="S2.SS1.p1.2.m2.3.3.2.2.2.2.1.1.3.2"><times id="S2.SS1.p1.2.m2.3.3.2.2.2.2.1.1.3.2.1.cmml" xref="S2.SS1.p1.2.m2.3.3.2.2.2.2.1.1.3.2.1"></times><cn type="integer" id="S2.SS1.p1.2.m2.3.3.2.2.2.2.1.1.3.2.2.cmml" xref="S2.SS1.p1.2.m2.3.3.2.2.2.2.1.1.3.2.2">1</cn><ci id="S2.SS1.p1.2.m2.3.3.2.2.2.2.1.1.3.2.3.cmml" xref="S2.SS1.p1.2.m2.3.3.2.2.2.2.1.1.3.2.3">𝑒</ci></apply><cn type="integer" id="S2.SS1.p1.2.m2.3.3.2.2.2.2.1.1.3.3.cmml" xref="S2.SS1.p1.2.m2.3.3.2.2.2.2.1.1.3.3">8</cn></apply></apply><apply id="S2.SS1.p1.2.m2.3.3.2.2.2.2.2.2.cmml" xref="S2.SS1.p1.2.m2.3.3.2.2.2.2.2.2"><eq id="S2.SS1.p1.2.m2.3.3.2.2.2.2.2.2.1.cmml" xref="S2.SS1.p1.2.m2.3.3.2.2.2.2.2.2.1"></eq><apply id="S2.SS1.p1.2.m2.3.3.2.2.2.2.2.2.2.cmml" xref="S2.SS1.p1.2.m2.3.3.2.2.2.2.2.2.2"><times id="S2.SS1.p1.2.m2.3.3.2.2.2.2.2.2.2.1.cmml" xref="S2.SS1.p1.2.m2.3.3.2.2.2.2.2.2.2.1"></times><ci id="S2.SS1.p1.2.m2.3.3.2.2.2.2.2.2.2.2.cmml" xref="S2.SS1.p1.2.m2.3.3.2.2.2.2.2.2.2.2">𝜆</ci><ci id="S2.SS1.p1.2.m2.1.1a.cmml" xref="S2.SS1.p1.2.m2.3.3.2.2.2.2.2.2.2.3.2"><mtext id="S2.SS1.p1.2.m2.1.1.cmml" xref="S2.SS1.p1.2.m2.1.1">weight decay</mtext></ci></apply><cn type="float" id="S2.SS1.p1.2.m2.3.3.2.2.2.2.2.2.3.cmml" xref="S2.SS1.p1.2.m2.3.3.2.2.2.2.2.2.3">0.1</cn></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p1.2.m2.3c">\beta_{1}=0.9,\beta_{2}=0.95,\epsilon=1e-8,\lambda(\text{weight decay})=0.1</annotation></semantics></math>.
Sec. <a href="#S2.SS5" title="2.5 Learning Rate Scheduler ‣ 2 Model Pre-Training ‣ Stable LM 2 1.6B Technical Report" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2.5</span></a> offers details regarding the custom learning rate scheduler that we applied.</p>
</div>
</section>
<section id="S2.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.2 </span>Data</h3>

<div id="S2.SS2.p1" class="ltx_para">
<p id="S2.SS2.p1.1" class="ltx_p">Model performance is affected by pre-training data design decisions, including both the source selection and the sampling weights <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib48" title="" class="ltx_ref">48</a>]</cite>. Our approach is close to that of <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib67" title="" class="ltx_ref">67</a>]</cite>: the majority of our training data consists of sources utilized in the training of other LLMs, such as RefinedWeb <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib57" title="" class="ltx_ref">57</a>]</cite>, subsets of the Pile <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib22" title="" class="ltx_ref">22</a>]</cite>, RedPajama <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib13" title="" class="ltx_ref">13</a>]</cite> and the Stack <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib39" title="" class="ltx_ref">39</a>]</cite>. We supplement these with OpenWebText <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib24" title="" class="ltx_ref">24</a>]</cite>, OpenWebMath <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib56" title="" class="ltx_ref">56</a>]</cite>, and parts of CulturaX <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib54" title="" class="ltx_ref">54</a>]</cite>. While inspecting randomly sampled documents from the mC4 subset of CulturaX, we encountered frequent HTML boilerplate and decided to remove this portion entirely, finally keeping only the OSCAR subset.
Additionally, we incorporate FanFics<span id="footnote3" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">3</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">3</sup><span class="ltx_tag ltx_tag_note">3</span><a target="_blank" href="https://huggingface.co/datasets/atom-in-the-universe/fanfics-10k-50k" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://huggingface.co/datasets/atom-in-the-universe/fanfics-10k-50k</a></span></span></span>, a subset of 50k documents from <em id="S2.SS2.p1.1.1" class="ltx_emph ltx_font_italic">fanfiction.net</em> selected by lowest perplexity scores according to a KenLM<span id="footnote4" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">4</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">4</sup><span class="ltx_tag ltx_tag_note">4</span><a target="_blank" href="https://huggingface.co/edugp/kenlm" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://huggingface.co/edugp/kenlm</a></span></span></span>. Finally, following <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib81" title="" class="ltx_ref">81</a>]</cite>, we restructure several raw datasets into rich fixed forms for downstream tasks such as summarization, question-answering, sentiment analysis, etc., and add instruction data from <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib47" title="" class="ltx_ref">47</a>]</cite>, the aggregate of which we collectively call Restruct-v1. The list of sources used in Restruct-v1 is made available in Tab. <a href="#A5.T11" title="Table 11 ‣ Appendix E Restructed Pre-training Sources ‣ Stable LM 2 1.6B Technical Report" class="ltx_ref"><span class="ltx_text ltx_ref_tag">11</span></a>.
Stable LM’s training set is comprised entirely of open-source datasets compatible with commercial usage, most of which are hosted on the Hugging Face Hub.
The only exception to the latter aspect (HFH), Restruct-v1, can easily be reproduced by following the approaches and prompt templates provided by <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib81" title="" class="ltx_ref">81</a>]</cite>.</p>
</div>
<div id="S2.SS2.p2" class="ltx_para">
<p id="S2.SS2.p2.1" class="ltx_p">Carefully selecting the mixture proportions of the various data domains is critical, particularly with respect to the amount of non-English and code data.
We trained several models on different mixes and evaluated them on downstream benchmarks to pick our final dataset.
The full set of ablations is available in Appendix <a href="#A1" title="Appendix A Data Ablations ‣ Stable LM 2 1.6B Technical Report" class="ltx_ref"><span class="ltx_text ltx_ref_tag">A</span></a>, together with rationales for selecting these particular mixes.
Based on the results of the ablations, we trained our model with the mix shown in Table <a href="#S2.T1" title="Table 1 ‣ 2 Model Pre-Training ‣ Stable LM 2 1.6B Technical Report" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>, which accounts for around 2 trillion tokens.
Note that it includes multilingual data in German (DE), Spanish (ES), French (FR), Italian (IT), Dutch (NL), and Portuguese (PT). The split of our dataset across different domains is visualized in Fig. <a href="#S2.F1" title="Figure 1 ‣ 2 Model Pre-Training ‣ Stable LM 2 1.6B Technical Report" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>.</p>
</div>
<figure id="S2.T3" class="ltx_table">
<div class="ltx_flex_figure ltx_flex_table">
<div class="ltx_flex_cell ltx_flex_size_1">
<table id="S2.T3.2" class="ltx_tabular ltx_figure_panel ltx_align_middle">
<tr id="S2.T3.2.1" class="ltx_tr">
<td id="S2.T3.2.1.1" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_tt" style="width:57.8pt;padding-bottom:0.43057pt;">
<span id="S2.T3.2.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="S2.T3.2.1.1.1.1" class="ltx_p"><span id="S2.T3.2.1.1.1.1.1" class="ltx_text ltx_font_bold">Parameters</span></span>
</span>
</td>
<td id="S2.T3.2.1.2" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_tt" style="width:57.8pt;padding-bottom:0.43057pt;">
<span id="S2.T3.2.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="S2.T3.2.1.2.1.1" class="ltx_p"><span id="S2.T3.2.1.2.1.1.1" class="ltx_text ltx_font_bold">Hidden Size</span></span>
</span>
</td>
<td id="S2.T3.2.1.3" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_tt" style="width:57.8pt;padding-bottom:0.43057pt;">
<span id="S2.T3.2.1.3.1" class="ltx_inline-block ltx_align_top">
<span id="S2.T3.2.1.3.1.1" class="ltx_p"><span id="S2.T3.2.1.3.1.1.1" class="ltx_text ltx_font_bold">Layers</span></span>
</span>
</td>
<td id="S2.T3.2.1.4" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_tt" style="width:57.8pt;padding-bottom:0.43057pt;">
<span id="S2.T3.2.1.4.1" class="ltx_inline-block ltx_align_top">
<span id="S2.T3.2.1.4.1.1" class="ltx_p"><span id="S2.T3.2.1.4.1.1.1" class="ltx_text ltx_font_bold">Heads</span></span>
</span>
</td>
<td id="S2.T3.2.1.5" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_tt" style="width:72.3pt;padding-bottom:0.43057pt;">
<span id="S2.T3.2.1.5.1" class="ltx_inline-block ltx_align_top">
<span id="S2.T3.2.1.5.1.1" class="ltx_p"><span id="S2.T3.2.1.5.1.1.1" class="ltx_text ltx_font_bold">Sequence Length</span></span>
</span>
</td>
</tr>
<tr id="S2.T3.2.2" class="ltx_tr">
<td id="S2.T3.2.2.1" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_b ltx_border_t" style="width:57.8pt;padding-bottom:4.30554pt;">
<span id="S2.T3.2.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="S2.T3.2.2.1.1.1" class="ltx_p">1,644,417,024</span>
</span>
</td>
<td id="S2.T3.2.2.2" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_b ltx_border_t" style="width:57.8pt;padding-bottom:4.30554pt;">
<span id="S2.T3.2.2.2.1" class="ltx_inline-block ltx_align_top">
<span id="S2.T3.2.2.2.1.1" class="ltx_p">2048</span>
</span>
</td>
<td id="S2.T3.2.2.3" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_b ltx_border_t" style="width:57.8pt;padding-bottom:4.30554pt;">
<span id="S2.T3.2.2.3.1" class="ltx_inline-block ltx_align_top">
<span id="S2.T3.2.2.3.1.1" class="ltx_p">24</span>
</span>
</td>
<td id="S2.T3.2.2.4" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_b ltx_border_t" style="width:57.8pt;padding-bottom:4.30554pt;">
<span id="S2.T3.2.2.4.1" class="ltx_inline-block ltx_align_top">
<span id="S2.T3.2.2.4.1.1" class="ltx_p">32</span>
</span>
</td>
<td id="S2.T3.2.2.5" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_b ltx_border_t" style="width:72.3pt;padding-bottom:4.30554pt;">
<span id="S2.T3.2.2.5.1" class="ltx_inline-block ltx_align_top">
<span id="S2.T3.2.2.5.1.1" class="ltx_p">4096</span>
</span>
</td>
</tr>
</table>
</div>
</div>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table"><span id="S2.T3.3.1.1" class="ltx_text" style="font-size:90%;">Table 2</span>: </span><span id="S2.T3.4.2" class="ltx_text" style="font-size:90%;">Stable LM 2 model architecture.</span></figcaption><div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_1">
<table id="S2.T3.5" class="ltx_tabular ltx_figure_panel ltx_align_middle">
<tr id="S2.T3.5.1" class="ltx_tr">
<td id="S2.T3.5.1.1" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_tt" style="width:79.5pt;padding-bottom:0.43057pt;">
<span id="S2.T3.5.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="S2.T3.5.1.1.1.1" class="ltx_p"><span id="S2.T3.5.1.1.1.1.1" class="ltx_text ltx_font_bold">Data Parallel Degree</span></span>
</span>
</td>
<td id="S2.T3.5.1.2" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_tt" style="width:79.5pt;padding-bottom:0.43057pt;">
<span id="S2.T3.5.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="S2.T3.5.1.2.1.1" class="ltx_p"><span id="S2.T3.5.1.2.1.1.1" class="ltx_text ltx_font_bold">Micro Batch Size</span></span>
</span>
</td>
<td id="S2.T3.5.1.3" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_tt" style="width:79.5pt;padding-bottom:0.43057pt;">
<span id="S2.T3.5.1.3.1" class="ltx_inline-block ltx_align_top">
<span id="S2.T3.5.1.3.1.1" class="ltx_p"><span id="S2.T3.5.1.3.1.1.1" class="ltx_text ltx_font_bold">Gradient Accumulation Steps</span></span>
</span>
</td>
<td id="S2.T3.5.1.4" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_tt" style="width:79.5pt;padding-bottom:0.43057pt;">
<span id="S2.T3.5.1.4.1" class="ltx_inline-block ltx_align_top">
<span id="S2.T3.5.1.4.1.1" class="ltx_p"><span id="S2.T3.5.1.4.1.1.1" class="ltx_text ltx_font_bold">Activation Checkpointing</span></span>
</span>
</td>
</tr>
<tr id="S2.T3.5.2" class="ltx_tr">
<td id="S2.T3.5.2.1" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_b ltx_border_t" style="width:79.5pt;padding-bottom:0.43057pt;">
<span id="S2.T3.5.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="S2.T3.5.2.1.1.1" class="ltx_p">512</span>
</span>
</td>
<td id="S2.T3.5.2.2" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_b ltx_border_t" style="width:79.5pt;padding-bottom:0.43057pt;">
<span id="S2.T3.5.2.2.1" class="ltx_inline-block ltx_align_top">
<span id="S2.T3.5.2.2.1.1" class="ltx_p">2</span>
</span>
</td>
<td id="S2.T3.5.2.3" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_b ltx_border_t" style="width:79.5pt;padding-bottom:0.43057pt;">
<span id="S2.T3.5.2.3.1" class="ltx_inline-block ltx_align_top">
<span id="S2.T3.5.2.3.1.1" class="ltx_p">2</span>
</span>
</td>
<td id="S2.T3.5.2.4" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_b ltx_border_t" style="width:79.5pt;padding-bottom:0.43057pt;">
<span id="S2.T3.5.2.4.1" class="ltx_inline-block ltx_align_top">
<span id="S2.T3.5.2.4.1.1" class="ltx_p">disabled</span>
</span>
</td>
</tr>
</table>
</div>
</div>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table"><span id="S2.T3.6.1.1" class="ltx_text" style="font-size:90%;">Table 3</span>: </span><span id="S2.T3.7.2" class="ltx_text" style="font-size:90%;">Stable LM 2 training configuration.</span></figcaption>
</figure>
</section>
<section id="S2.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.3 </span>Tokenizer</h3>

<div id="S2.SS3.p1" class="ltx_para">
<p id="S2.SS3.p1.1" class="ltx_p">We use <span id="S2.SS3.p1.1.1" class="ltx_text ltx_font_bold">Arcade100k</span>, a BPE tokenizer extended from OpenAI’s <code id="S2.SS3.p1.1.2" class="ltx_verbatim ltx_font_typewriter">tiktoken.cl100k_base</code> to include special tokens for code <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib40" title="" class="ltx_ref">40</a>]</cite> and digit-split handling <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib45" title="" class="ltx_ref">45</a>, <a href="#bib.bib4" title="" class="ltx_ref">4</a>]</cite>. The vocabulary consists of 100,289 tokens and is padded to the nearest multiple of 64 (100,352) during training to meet the recommended Tensor Cores alignment on NVIDIA A100 devices. In preliminary experiments, we did not observe statistically significant deviations in downstream natural language performance tasks when compared against a hyperparameter matching model trained with the smaller GPT-NeoX tokenizer <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib6" title="" class="ltx_ref">6</a>]</cite>. Increased compression rates for code and non-English languages influenced our decision to use <span id="S2.SS3.p1.1.3" class="ltx_text ltx_font_bold">Arcade100k</span> over existing tokenizers.</p>
</div>
</section>
<section id="S2.SS4" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.4 </span>Architecture and Training Layout</h3>

<div id="S2.SS4.p1" class="ltx_para">
<p id="S2.SS4.p1.1" class="ltx_p">The model is a causal decoder-only transformer similar in design to the LLaMA architecture <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib67" title="" class="ltx_ref">67</a>]</cite>. Table <a href="#S2.T3" title="Table 3 ‣ 2.2 Data ‣ 2 Model Pre-Training ‣ Stable LM 2 1.6B Technical Report" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a> shows some of the key architectural details.
In particular, the main differences with respect to LLaMA are the following:</p>
<ul id="S2.I1" class="ltx_itemize">
<li id="S2.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S2.I1.i1.p1" class="ltx_para">
<p id="S2.I1.i1.p1.1" class="ltx_p"><span id="S2.I1.i1.p1.1.1" class="ltx_text ltx_font_bold">Position Embeddings</span>. Rotary Position Embeddings <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib66" title="" class="ltx_ref">66</a>]</cite> applied to the first <math id="S2.I1.i1.p1.1.m1.1" class="ltx_Math" alttext="25\%" display="inline"><semantics id="S2.I1.i1.p1.1.m1.1a"><mrow id="S2.I1.i1.p1.1.m1.1.1" xref="S2.I1.i1.p1.1.m1.1.1.cmml"><mn id="S2.I1.i1.p1.1.m1.1.1.2" xref="S2.I1.i1.p1.1.m1.1.1.2.cmml">25</mn><mo id="S2.I1.i1.p1.1.m1.1.1.1" xref="S2.I1.i1.p1.1.m1.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S2.I1.i1.p1.1.m1.1b"><apply id="S2.I1.i1.p1.1.m1.1.1.cmml" xref="S2.I1.i1.p1.1.m1.1.1"><csymbol cd="latexml" id="S2.I1.i1.p1.1.m1.1.1.1.cmml" xref="S2.I1.i1.p1.1.m1.1.1.1">percent</csymbol><cn type="integer" id="S2.I1.i1.p1.1.m1.1.1.2.cmml" xref="S2.I1.i1.p1.1.m1.1.1.2">25</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.I1.i1.p1.1.m1.1c">25\%</annotation></semantics></math> of head embedding dimensions for improved throughput following <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib6" title="" class="ltx_ref">6</a>]</cite>.</p>
</div>
</li>
<li id="S2.I1.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S2.I1.i2.p1" class="ltx_para">
<p id="S2.I1.i2.p1.1" class="ltx_p"><span id="S2.I1.i2.p1.1.1" class="ltx_text ltx_font_bold">Normalization</span>. LayerNorm <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib3" title="" class="ltx_ref">3</a>]</cite> with learned bias terms as opposed to RMSNorm <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib84" title="" class="ltx_ref">84</a>]</cite>.</p>
</div>
</li>
<li id="S2.I1.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S2.I1.i3.p1" class="ltx_para">
<p id="S2.I1.i3.p1.1" class="ltx_p"><span id="S2.I1.i3.p1.1.1" class="ltx_text ltx_font_bold">Biases</span>. We remove all bias terms from the feed-forward networks and multi-head self-attention layers, except for the biases of the key, query, and value projections <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib4" title="" class="ltx_ref">4</a>]</cite>.</p>
</div>
</li>
</ul>
</div>
<div id="S2.SS4.p2" class="ltx_para">
<p id="S2.SS4.p2.5" class="ltx_p">Stable LM 2 1.6B is trained on 64 Amazon P4d instances comprising 512 NVIDIA A100 (40GB HBM2) GPUs. The size of our model, together with ZeRO stage 1 distributed optimization <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib61" title="" class="ltx_ref">61</a>]</cite>, eliminates the need for model sharding. Still, different triplets of <span id="S2.SS4.p2.5.1" class="ltx_text ltx_font_bold">micro batch size</span>, <span id="S2.SS4.p2.5.2" class="ltx_text ltx_font_bold">gradient accumulation steps</span>, and <span id="S2.SS4.p2.5.3" class="ltx_text ltx_font_bold">activation checkpointing</span> granularity lead to different speed metrics.
Following the recommendations in <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib28" title="" class="ltx_ref">28</a>]</cite>, we obtain our final configuration by finding the micro-batch size that allows us to completely remove activation checkpointing. We then determine the gradient accumulation steps based on our target batch size and <span id="S2.SS4.p2.5.4" class="ltx_text ltx_font_bold">data parallel</span> degree. We employ a batch size of <math id="S2.SS4.p2.1.m1.3" class="ltx_Math" alttext="8,388,608" display="inline"><semantics id="S2.SS4.p2.1.m1.3a"><mrow id="S2.SS4.p2.1.m1.3.4.2" xref="S2.SS4.p2.1.m1.3.4.1.cmml"><mn id="S2.SS4.p2.1.m1.1.1" xref="S2.SS4.p2.1.m1.1.1.cmml">8</mn><mo id="S2.SS4.p2.1.m1.3.4.2.1" xref="S2.SS4.p2.1.m1.3.4.1.cmml">,</mo><mn id="S2.SS4.p2.1.m1.2.2" xref="S2.SS4.p2.1.m1.2.2.cmml">388</mn><mo id="S2.SS4.p2.1.m1.3.4.2.2" xref="S2.SS4.p2.1.m1.3.4.1.cmml">,</mo><mn id="S2.SS4.p2.1.m1.3.3" xref="S2.SS4.p2.1.m1.3.3.cmml">608</mn></mrow><annotation-xml encoding="MathML-Content" id="S2.SS4.p2.1.m1.3b"><list id="S2.SS4.p2.1.m1.3.4.1.cmml" xref="S2.SS4.p2.1.m1.3.4.2"><cn type="integer" id="S2.SS4.p2.1.m1.1.1.cmml" xref="S2.SS4.p2.1.m1.1.1">8</cn><cn type="integer" id="S2.SS4.p2.1.m1.2.2.cmml" xref="S2.SS4.p2.1.m1.2.2">388</cn><cn type="integer" id="S2.SS4.p2.1.m1.3.3.cmml" xref="S2.SS4.p2.1.m1.3.3">608</cn></list></annotation-xml><annotation encoding="application/x-tex" id="S2.SS4.p2.1.m1.3c">8,388,608</annotation></semantics></math> tokens, based on the observations in Appendix <a href="#A4" title="Appendix D Global Batch Size ‣ Stable LM 2 1.6B Technical Report" class="ltx_ref"><span class="ltx_text ltx_ref_tag">D</span></a>. With the setup in Table <a href="#S2.T3" title="Table 3 ‣ 2.2 Data ‣ 2 Model Pre-Training ‣ Stable LM 2 1.6B Technical Report" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>, we achieve <math id="S2.SS4.p2.2.m2.1" class="ltx_Math" alttext="\approx" display="inline"><semantics id="S2.SS4.p2.2.m2.1a"><mo id="S2.SS4.p2.2.m2.1.1" xref="S2.SS4.p2.2.m2.1.1.cmml">≈</mo><annotation-xml encoding="MathML-Content" id="S2.SS4.p2.2.m2.1b"><approx id="S2.SS4.p2.2.m2.1.1.cmml" xref="S2.SS4.p2.2.m2.1.1"></approx></annotation-xml><annotation encoding="application/x-tex" id="S2.SS4.p2.2.m2.1c">\approx</annotation></semantics></math>170 TFLOPs/s per device, or <math id="S2.SS4.p2.3.m3.1" class="ltx_Math" alttext="54.5\%" display="inline"><semantics id="S2.SS4.p2.3.m3.1a"><mrow id="S2.SS4.p2.3.m3.1.1" xref="S2.SS4.p2.3.m3.1.1.cmml"><mn id="S2.SS4.p2.3.m3.1.1.2" xref="S2.SS4.p2.3.m3.1.1.2.cmml">54.5</mn><mo id="S2.SS4.p2.3.m3.1.1.1" xref="S2.SS4.p2.3.m3.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S2.SS4.p2.3.m3.1b"><apply id="S2.SS4.p2.3.m3.1.1.cmml" xref="S2.SS4.p2.3.m3.1.1"><csymbol cd="latexml" id="S2.SS4.p2.3.m3.1.1.1.cmml" xref="S2.SS4.p2.3.m3.1.1.1">percent</csymbol><cn type="float" id="S2.SS4.p2.3.m3.1.1.2.cmml" xref="S2.SS4.p2.3.m3.1.1.2">54.5</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS4.p2.3.m3.1c">54.5\%</annotation></semantics></math> model flops utilization (MFU). A higher hardware utilization of <math id="S2.SS4.p2.4.m4.1" class="ltx_Math" alttext="\approx" display="inline"><semantics id="S2.SS4.p2.4.m4.1a"><mo id="S2.SS4.p2.4.m4.1.1" xref="S2.SS4.p2.4.m4.1.1.cmml">≈</mo><annotation-xml encoding="MathML-Content" id="S2.SS4.p2.4.m4.1b"><approx id="S2.SS4.p2.4.m4.1.1.cmml" xref="S2.SS4.p2.4.m4.1.1"></approx></annotation-xml><annotation encoding="application/x-tex" id="S2.SS4.p2.4.m4.1c">\approx</annotation></semantics></math>200 TFLOPs/s (<math id="S2.SS4.p2.5.m5.1" class="ltx_Math" alttext="64\%" display="inline"><semantics id="S2.SS4.p2.5.m5.1a"><mrow id="S2.SS4.p2.5.m5.1.1" xref="S2.SS4.p2.5.m5.1.1.cmml"><mn id="S2.SS4.p2.5.m5.1.1.2" xref="S2.SS4.p2.5.m5.1.1.2.cmml">64</mn><mo id="S2.SS4.p2.5.m5.1.1.1" xref="S2.SS4.p2.5.m5.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S2.SS4.p2.5.m5.1b"><apply id="S2.SS4.p2.5.m5.1.1.cmml" xref="S2.SS4.p2.5.m5.1.1"><csymbol cd="latexml" id="S2.SS4.p2.5.m5.1.1.1.cmml" xref="S2.SS4.p2.5.m5.1.1.1">percent</csymbol><cn type="integer" id="S2.SS4.p2.5.m5.1.1.2.cmml" xref="S2.SS4.p2.5.m5.1.1.2">64</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS4.p2.5.m5.1c">64\%</annotation></semantics></math> MFU) can be trivially achieved by decreasing the degree of data parallelism and correspondingly increasing the number of gradient accumulation steps, at the cost of an increased iteration time.</p>
</div>
</section>
<section id="S2.SS5" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.5 </span>Learning Rate Scheduler</h3>

<div id="S2.SS5.p1" class="ltx_para">
<p id="S2.SS5.p1.1" class="ltx_p">We propose a new learning rate scheduler.
It consists of multiple stages and is designed to favor flexibility in terms of continued pre-training.
We begin by linearly increasing the learning rate to its max value of <math id="S2.SS5.p1.1.m1.1" class="ltx_Math" alttext="1e-3" display="inline"><semantics id="S2.SS5.p1.1.m1.1a"><mrow id="S2.SS5.p1.1.m1.1.1" xref="S2.SS5.p1.1.m1.1.1.cmml"><mrow id="S2.SS5.p1.1.m1.1.1.2" xref="S2.SS5.p1.1.m1.1.1.2.cmml"><mn id="S2.SS5.p1.1.m1.1.1.2.2" xref="S2.SS5.p1.1.m1.1.1.2.2.cmml">1</mn><mo lspace="0em" rspace="0em" id="S2.SS5.p1.1.m1.1.1.2.1" xref="S2.SS5.p1.1.m1.1.1.2.1.cmml">​</mo><mi id="S2.SS5.p1.1.m1.1.1.2.3" xref="S2.SS5.p1.1.m1.1.1.2.3.cmml">e</mi></mrow><mo id="S2.SS5.p1.1.m1.1.1.1" xref="S2.SS5.p1.1.m1.1.1.1.cmml">−</mo><mn id="S2.SS5.p1.1.m1.1.1.3" xref="S2.SS5.p1.1.m1.1.1.3.cmml">3</mn></mrow><annotation-xml encoding="MathML-Content" id="S2.SS5.p1.1.m1.1b"><apply id="S2.SS5.p1.1.m1.1.1.cmml" xref="S2.SS5.p1.1.m1.1.1"><minus id="S2.SS5.p1.1.m1.1.1.1.cmml" xref="S2.SS5.p1.1.m1.1.1.1"></minus><apply id="S2.SS5.p1.1.m1.1.1.2.cmml" xref="S2.SS5.p1.1.m1.1.1.2"><times id="S2.SS5.p1.1.m1.1.1.2.1.cmml" xref="S2.SS5.p1.1.m1.1.1.2.1"></times><cn type="integer" id="S2.SS5.p1.1.m1.1.1.2.2.cmml" xref="S2.SS5.p1.1.m1.1.1.2.2">1</cn><ci id="S2.SS5.p1.1.m1.1.1.2.3.cmml" xref="S2.SS5.p1.1.m1.1.1.2.3">𝑒</ci></apply><cn type="integer" id="S2.SS5.p1.1.m1.1.1.3.cmml" xref="S2.SS5.p1.1.m1.1.1.3">3</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS5.p1.1.m1.1c">1e-3</annotation></semantics></math> over 9720 steps. This <em id="S2.SS5.p1.1.1" class="ltx_emph ltx_font_italic">warm up</em> stage is followed by the main training phase in which the learning rate is decreased according to Eq. <a href="#S2.E1" title="In 2.5 Learning Rate Scheduler ‣ 2 Model Pre-Training ‣ Stable LM 2 1.6B Technical Report" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>:</p>
<table id="S2.E1" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S2.E1.m1.8" class="ltx_math_unparsed" alttext="\left\{\begin{aligned} &amp;m+\frac{(M-m)}{2}*\left[\cos\left(2\pi*\frac{i}{N}\right)+1\right]\quad&amp;\text{if}\quad i\leq N/4&amp;\qquad\text{\emph{cosine} decay}\\
&amp;\frac{\alpha}{\sqrt{i+\beta}}\quad&amp;\text{if}\quad i&gt;N/4&amp;\qquad\text{\emph{rsqrt} decay}\end{aligned}\right." display="block"><semantics id="S2.E1.m1.8a"><mrow id="S2.E1.m1.8b"><mo id="S2.E1.m1.8.9">{</mo><mtable columnspacing="0pt" displaystyle="true" rowspacing="0pt" id="S2.E1.m1.8.8"><mtr id="S2.E1.m1.8.8a"><mtd id="S2.E1.m1.8.8b"></mtd><mtd class="ltx_align_left" columnalign="left" id="S2.E1.m1.8.8c"><mrow id="S2.E1.m1.3.3.3.3.3.3"><mrow id="S2.E1.m1.3.3.3.3.3.3.1"><mi id="S2.E1.m1.3.3.3.3.3.3.1.3">m</mi><mo id="S2.E1.m1.3.3.3.3.3.3.1.2">+</mo><mrow id="S2.E1.m1.3.3.3.3.3.3.1.1"><mfrac id="S2.E1.m1.1.1.1.1.1.1"><mrow id="S2.E1.m1.1.1.1.1.1.1.1.1"><mo stretchy="false" id="S2.E1.m1.1.1.1.1.1.1.1.1.2">(</mo><mrow id="S2.E1.m1.1.1.1.1.1.1.1.1.1"><mi id="S2.E1.m1.1.1.1.1.1.1.1.1.1.2">M</mi><mo id="S2.E1.m1.1.1.1.1.1.1.1.1.1.1">−</mo><mi id="S2.E1.m1.1.1.1.1.1.1.1.1.1.3">m</mi></mrow><mo stretchy="false" id="S2.E1.m1.1.1.1.1.1.1.1.1.3">)</mo></mrow><mn id="S2.E1.m1.1.1.1.1.1.1.3">2</mn></mfrac><mo lspace="0.222em" rspace="0.222em" id="S2.E1.m1.3.3.3.3.3.3.1.1.2">∗</mo><mrow id="S2.E1.m1.3.3.3.3.3.3.1.1.1.1"><mo id="S2.E1.m1.3.3.3.3.3.3.1.1.1.1.2">[</mo><mrow id="S2.E1.m1.3.3.3.3.3.3.1.1.1.1.1"><mrow id="S2.E1.m1.3.3.3.3.3.3.1.1.1.1.1.1.1"><mi id="S2.E1.m1.2.2.2.2.2.2">cos</mi><mo id="S2.E1.m1.3.3.3.3.3.3.1.1.1.1.1.1.1a">⁡</mo><mrow id="S2.E1.m1.3.3.3.3.3.3.1.1.1.1.1.1.1.1"><mo id="S2.E1.m1.3.3.3.3.3.3.1.1.1.1.1.1.1.1.2">(</mo><mrow id="S2.E1.m1.3.3.3.3.3.3.1.1.1.1.1.1.1.1.1"><mrow id="S2.E1.m1.3.3.3.3.3.3.1.1.1.1.1.1.1.1.1.2"><mn id="S2.E1.m1.3.3.3.3.3.3.1.1.1.1.1.1.1.1.1.2.2">2</mn><mo lspace="0em" rspace="0em" id="S2.E1.m1.3.3.3.3.3.3.1.1.1.1.1.1.1.1.1.2.1">​</mo><mi id="S2.E1.m1.3.3.3.3.3.3.1.1.1.1.1.1.1.1.1.2.3">π</mi></mrow><mo lspace="0.222em" rspace="0.222em" id="S2.E1.m1.3.3.3.3.3.3.1.1.1.1.1.1.1.1.1.1">∗</mo><mfrac id="S2.E1.m1.3.3.3.3.3.3.1.1.1.1.1.1.1.1.1.3"><mi id="S2.E1.m1.3.3.3.3.3.3.1.1.1.1.1.1.1.1.1.3.2">i</mi><mi id="S2.E1.m1.3.3.3.3.3.3.1.1.1.1.1.1.1.1.1.3.3">N</mi></mfrac></mrow><mo id="S2.E1.m1.3.3.3.3.3.3.1.1.1.1.1.1.1.1.3">)</mo></mrow></mrow><mo id="S2.E1.m1.3.3.3.3.3.3.1.1.1.1.1.2">+</mo><mn id="S2.E1.m1.3.3.3.3.3.3.1.1.1.1.1.3">1</mn></mrow><mo id="S2.E1.m1.3.3.3.3.3.3.1.1.1.1.3">]</mo></mrow></mrow></mrow><mspace width="1em" id="S2.E1.m1.3.3.3.3.3.3.2"></mspace></mrow></mtd><mtd class="ltx_align_right" columnalign="right" id="S2.E1.m1.8.8d"><mrow id="S2.E1.m1.5.5.5.5.2"><mrow id="S2.E1.m1.5.5.5.5.2.4.2"><mtext id="S2.E1.m1.4.4.4.4.1.1">if</mtext><mspace width="1em" id="S2.E1.m1.5.5.5.5.2.4.2.1"></mspace><mi id="S2.E1.m1.5.5.5.5.2.2">i</mi></mrow><mo id="S2.E1.m1.5.5.5.5.2.3">≤</mo><mrow id="S2.E1.m1.5.5.5.5.2.5"><mi id="S2.E1.m1.5.5.5.5.2.5.2">N</mi><mo id="S2.E1.m1.5.5.5.5.2.5.1">/</mo><mn id="S2.E1.m1.5.5.5.5.2.5.3">4</mn></mrow></mrow></mtd><mtd class="ltx_align_left" columnalign="left" id="S2.E1.m1.8.8e"><mrow id="S2.E1.m1.5.5.5.7.1"><mtext class="ltx_mathvariant_italic" id="S2.E1.m1.5.5.5.7.1a">cosine</mtext><mtext id="S2.E1.m1.5.5.5.7.1b"> decay</mtext></mrow></mtd></mtr><mtr id="S2.E1.m1.8.8f"><mtd id="S2.E1.m1.8.8g"></mtd><mtd class="ltx_align_left" columnalign="left" id="S2.E1.m1.8.8h"><mrow id="S2.E1.m1.6.6.6.1.1.3"><mfrac id="S2.E1.m1.6.6.6.1.1.1"><mi id="S2.E1.m1.6.6.6.1.1.1.2">α</mi><msqrt id="S2.E1.m1.6.6.6.1.1.1.3"><mrow id="S2.E1.m1.6.6.6.1.1.1.3.2"><mi id="S2.E1.m1.6.6.6.1.1.1.3.2.2">i</mi><mo id="S2.E1.m1.6.6.6.1.1.1.3.2.1">+</mo><mi id="S2.E1.m1.6.6.6.1.1.1.3.2.3">β</mi></mrow></msqrt></mfrac><mspace width="1em" id="S2.E1.m1.6.6.6.1.1.3.1"></mspace></mrow></mtd><mtd class="ltx_align_right" columnalign="right" id="S2.E1.m1.8.8i"><mrow id="S2.E1.m1.8.8.8.3.2"><mrow id="S2.E1.m1.8.8.8.3.2.4.2"><mtext id="S2.E1.m1.7.7.7.2.1.1">if</mtext><mspace width="1em" id="S2.E1.m1.8.8.8.3.2.4.2.1"></mspace><mi id="S2.E1.m1.8.8.8.3.2.2">i</mi></mrow><mo id="S2.E1.m1.8.8.8.3.2.3">&gt;</mo><mrow id="S2.E1.m1.8.8.8.3.2.5"><mi id="S2.E1.m1.8.8.8.3.2.5.2">N</mi><mo id="S2.E1.m1.8.8.8.3.2.5.1">/</mo><mn id="S2.E1.m1.8.8.8.3.2.5.3">4</mn></mrow></mrow></mtd><mtd class="ltx_align_left" columnalign="left" id="S2.E1.m1.8.8j"><mrow id="S2.E1.m1.8.8.8.5.1"><mtext class="ltx_mathvariant_italic" id="S2.E1.m1.8.8.8.5.1a">rsqrt</mtext><mtext id="S2.E1.m1.8.8.8.5.1b"> decay</mtext></mrow></mtd></mtr></mtable></mrow><annotation encoding="application/x-tex" id="S2.E1.m1.8c">\left\{\begin{aligned} &amp;m+\frac{(M-m)}{2}*\left[\cos\left(2\pi*\frac{i}{N}\right)+1\right]\quad&amp;\text{if}\quad i\leq N/4&amp;\qquad\text{\emph{cosine} decay}\\
&amp;\frac{\alpha}{\sqrt{i+\beta}}\quad&amp;\text{if}\quad i&gt;N/4&amp;\qquad\text{\emph{rsqrt} decay}\end{aligned}\right.</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(1)</span></td>
</tr></tbody>
</table>
<p id="S2.SS5.p1.8" class="ltx_p">where <math id="S2.SS5.p1.2.m1.1" class="ltx_Math" alttext="m" display="inline"><semantics id="S2.SS5.p1.2.m1.1a"><mi id="S2.SS5.p1.2.m1.1.1" xref="S2.SS5.p1.2.m1.1.1.cmml">m</mi><annotation-xml encoding="MathML-Content" id="S2.SS5.p1.2.m1.1b"><ci id="S2.SS5.p1.2.m1.1.1.cmml" xref="S2.SS5.p1.2.m1.1.1">𝑚</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS5.p1.2.m1.1c">m</annotation></semantics></math> and <math id="S2.SS5.p1.3.m2.1" class="ltx_Math" alttext="M" display="inline"><semantics id="S2.SS5.p1.3.m2.1a"><mi id="S2.SS5.p1.3.m2.1.1" xref="S2.SS5.p1.3.m2.1.1.cmml">M</mi><annotation-xml encoding="MathML-Content" id="S2.SS5.p1.3.m2.1b"><ci id="S2.SS5.p1.3.m2.1.1.cmml" xref="S2.SS5.p1.3.m2.1.1">𝑀</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS5.p1.3.m2.1c">M</annotation></semantics></math> are respectively the minimum and maximum learning rates, <math id="S2.SS5.p1.4.m3.1" class="ltx_Math" alttext="i" display="inline"><semantics id="S2.SS5.p1.4.m3.1a"><mi id="S2.SS5.p1.4.m3.1.1" xref="S2.SS5.p1.4.m3.1.1.cmml">i</mi><annotation-xml encoding="MathML-Content" id="S2.SS5.p1.4.m3.1b"><ci id="S2.SS5.p1.4.m3.1.1.cmml" xref="S2.SS5.p1.4.m3.1.1">𝑖</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS5.p1.4.m3.1c">i</annotation></semantics></math> is the current step, and <math id="S2.SS5.p1.5.m4.1" class="ltx_Math" alttext="N" display="inline"><semantics id="S2.SS5.p1.5.m4.1a"><mi id="S2.SS5.p1.5.m4.1.1" xref="S2.SS5.p1.5.m4.1.1.cmml">N</mi><annotation-xml encoding="MathML-Content" id="S2.SS5.p1.5.m4.1b"><ci id="S2.SS5.p1.5.m4.1.1.cmml" xref="S2.SS5.p1.5.m4.1.1">𝑁</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS5.p1.5.m4.1c">N</annotation></semantics></math> is the total number of steps. The free parameters <math id="S2.SS5.p1.6.m5.1" class="ltx_Math" alttext="\alpha" display="inline"><semantics id="S2.SS5.p1.6.m5.1a"><mi id="S2.SS5.p1.6.m5.1.1" xref="S2.SS5.p1.6.m5.1.1.cmml">α</mi><annotation-xml encoding="MathML-Content" id="S2.SS5.p1.6.m5.1b"><ci id="S2.SS5.p1.6.m5.1.1.cmml" xref="S2.SS5.p1.6.m5.1.1">𝛼</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS5.p1.6.m5.1c">\alpha</annotation></semantics></math> and <math id="S2.SS5.p1.7.m6.1" class="ltx_Math" alttext="\beta" display="inline"><semantics id="S2.SS5.p1.7.m6.1a"><mi id="S2.SS5.p1.7.m6.1.1" xref="S2.SS5.p1.7.m6.1.1.cmml">β</mi><annotation-xml encoding="MathML-Content" id="S2.SS5.p1.7.m6.1b"><ci id="S2.SS5.p1.7.m6.1.1.cmml" xref="S2.SS5.p1.7.m6.1.1">𝛽</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS5.p1.7.m6.1c">\beta</annotation></semantics></math> have been arbitrarily chosen to enforce the continuity of the scheduler and its derivative at <math id="S2.SS5.p1.8.m7.1" class="ltx_Math" alttext="i=N/4" display="inline"><semantics id="S2.SS5.p1.8.m7.1a"><mrow id="S2.SS5.p1.8.m7.1.1" xref="S2.SS5.p1.8.m7.1.1.cmml"><mi id="S2.SS5.p1.8.m7.1.1.2" xref="S2.SS5.p1.8.m7.1.1.2.cmml">i</mi><mo id="S2.SS5.p1.8.m7.1.1.1" xref="S2.SS5.p1.8.m7.1.1.1.cmml">=</mo><mrow id="S2.SS5.p1.8.m7.1.1.3" xref="S2.SS5.p1.8.m7.1.1.3.cmml"><mi id="S2.SS5.p1.8.m7.1.1.3.2" xref="S2.SS5.p1.8.m7.1.1.3.2.cmml">N</mi><mo id="S2.SS5.p1.8.m7.1.1.3.1" xref="S2.SS5.p1.8.m7.1.1.3.1.cmml">/</mo><mn id="S2.SS5.p1.8.m7.1.1.3.3" xref="S2.SS5.p1.8.m7.1.1.3.3.cmml">4</mn></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS5.p1.8.m7.1b"><apply id="S2.SS5.p1.8.m7.1.1.cmml" xref="S2.SS5.p1.8.m7.1.1"><eq id="S2.SS5.p1.8.m7.1.1.1.cmml" xref="S2.SS5.p1.8.m7.1.1.1"></eq><ci id="S2.SS5.p1.8.m7.1.1.2.cmml" xref="S2.SS5.p1.8.m7.1.1.2">𝑖</ci><apply id="S2.SS5.p1.8.m7.1.1.3.cmml" xref="S2.SS5.p1.8.m7.1.1.3"><divide id="S2.SS5.p1.8.m7.1.1.3.1.cmml" xref="S2.SS5.p1.8.m7.1.1.3.1"></divide><ci id="S2.SS5.p1.8.m7.1.1.3.2.cmml" xref="S2.SS5.p1.8.m7.1.1.3.2">𝑁</ci><cn type="integer" id="S2.SS5.p1.8.m7.1.1.3.3.cmml" xref="S2.SS5.p1.8.m7.1.1.3.3">4</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS5.p1.8.m7.1c">i=N/4</annotation></semantics></math>.
We finalize training by linearly cooling the learning rate down to zero over 80k steps, which corresponds to around 670B tokens.
The full scheduler is illustrated in Fig. <a href="#S2.F2" title="Figure 2 ‣ 2.5 Learning Rate Scheduler ‣ 2 Model Pre-Training ‣ Stable LM 2 1.6B Technical Report" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>; further details and ablations can be found in Appendix <a href="#A2" title="Appendix B Scheduler Ablations ‣ Stable LM 2 1.6B Technical Report" class="ltx_ref"><span class="ltx_text ltx_ref_tag">B</span></a>.</p>
</div>
<figure id="S2.F2" class="ltx_figure"><img src="/html/2402.17834/assets/x2.png" id="S2.F2.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="242" height="181" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S2.F2.2.1.1" class="ltx_text" style="font-size:90%;">Figure 2</span>: </span><span id="S2.F2.3.2" class="ltx_text" style="font-size:90%;">Multi-stage infinite scheduler proposed and applied in this work.</span></figcaption>
</figure>
</section>
</section>
<section id="S3" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">3 </span>Fine-tuning and Alignment</h2>

<div id="S3.p1" class="ltx_para">
<p id="S3.p1.1" class="ltx_p">Following pre-training, we further develop the conversational skills of our model via a fine-tuning stage that consists of three main steps: supervised fine-tuning (SFT), direct preference optimization (DPO), and self-knowledge learning.
Importantly, we do not use multilingual data at this stage.
We now describe each of them in detail, and in Section <a href="#S4" title="4 Experimental Results and Benchmarks ‣ Stable LM 2 1.6B Technical Report" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a> we report the results after all three steps.</p>
</div>
<figure id="S3.T4" class="ltx_table">
<table id="S3.T4.2" class="ltx_tabular ltx_align_middle">
<tr id="S3.T4.2.1" class="ltx_tr">
<td id="S3.T4.2.1.1" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_tt" style="width:21.7pt;padding-bottom:0.43057pt;"><span id="S3.T4.2.1.1.1" class="ltx_text ltx_font_bold">Model</span></td>
<td id="S3.T4.2.1.2" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_tt" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T4.2.1.2.1.1" class="ltx_p"><span id="S3.T4.2.1.2.1.1.1" class="ltx_text ltx_font_bold">Size</span></span>
</span>
</td>
<td id="S3.T4.2.1.3" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_tt" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.1.3.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T4.2.1.3.1.1" class="ltx_p"><span id="S3.T4.2.1.3.1.1.1" class="ltx_text ltx_font_bold">Avg</span></span>
</span>
</td>
<td id="S3.T4.2.1.4" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_tt" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.1.4.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T4.2.1.4.1.1" class="ltx_p"><span id="S3.T4.2.1.4.1.1.1" class="ltx_text ltx_font_bold">ARC</span></span>
</span>
</td>
<td id="S3.T4.2.1.5" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_tt" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.1.5.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T4.2.1.5.1.1" class="ltx_p"><span id="S3.T4.2.1.5.1.1.1" class="ltx_text ltx_font_bold">HS</span></span>
</span>
</td>
<td id="S3.T4.2.1.6" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_tt" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.1.6.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T4.2.1.6.1.1" class="ltx_p"><span id="S3.T4.2.1.6.1.1.1" class="ltx_text ltx_font_bold">MMLU</span></span>
</span>
</td>
<td id="S3.T4.2.1.7" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_tt" style="width:36.1pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.1.7.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T4.2.1.7.1.1" class="ltx_p"><span id="S3.T4.2.1.7.1.1.1" class="ltx_text ltx_font_bold">TQA</span></span>
</span>
</td>
<td id="S3.T4.2.1.8" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_tt" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.1.8.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T4.2.1.8.1.1" class="ltx_p"><span id="S3.T4.2.1.8.1.1.1" class="ltx_text ltx_font_bold">Wino</span></span>
</span>
</td>
<td id="S3.T4.2.1.9" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_tt" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.1.9.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T4.2.1.9.1.1" class="ltx_p"><span id="S3.T4.2.1.9.1.1.1" class="ltx_text ltx_font_bold">GSM</span></span>
</span>
</td>
</tr>
<tr id="S3.T4.2.2" class="ltx_tr">
<td id="S3.T4.2.2.1" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.2.1.1" class="ltx_text ltx_font_bold">phi-2</span>†</td>
<td id="S3.T4.2.2.2" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.2.2.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T4.2.2.2.1.1" class="ltx_p">2.8B</span>
</span>
</td>
<td id="S3.T4.2.2.3" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.2.3.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T4.2.2.3.1.1" class="ltx_p">61.3</span>
</span>
</td>
<td id="S3.T4.2.2.4" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.2.4.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T4.2.2.4.1.1" class="ltx_p">61.1</span>
</span>
</td>
<td id="S3.T4.2.2.5" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.2.5.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T4.2.2.5.1.1" class="ltx_p">75.1</span>
</span>
</td>
<td id="S3.T4.2.2.6" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.2.6.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T4.2.2.6.1.1" class="ltx_p">58.1</span>
</span>
</td>
<td id="S3.T4.2.2.7" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:36.1pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.2.7.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T4.2.2.7.1.1" class="ltx_p">44.5</span>
</span>
</td>
<td id="S3.T4.2.2.8" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.2.8.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T4.2.2.8.1.1" class="ltx_p">74.4</span>
</span>
</td>
<td id="S3.T4.2.2.9" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.2.9.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T4.2.2.9.1.1" class="ltx_p">54.8</span>
</span>
</td>
</tr>
<tr id="S3.T4.2.3" class="ltx_tr" style="background-color:#CCE6FF;">
<td id="S3.T4.2.3.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;"><span id="S3.T4.2.3.1.1" class="ltx_text ltx_font_bold" style="background-color:#CCE6FF;">stablelm-2-zephyr-1_6b<span id="S3.T4.2.3.1.1.1" class="ltx_text ltx_font_medium">†</span></span></td>
<td id="S3.T4.2.3.2" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.3.2.1" class="ltx_inline-block ltx_align_top" style="background-color:#CCE6FF;">
<span id="S3.T4.2.3.2.1.1" class="ltx_p">1.6B</span>
</span>
</td>
<td id="S3.T4.2.3.3" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.3.3.1" class="ltx_inline-block ltx_align_top" style="background-color:#CCE6FF;">
<span id="S3.T4.2.3.3.1.1" class="ltx_p">49.7</span>
</span>
</td>
<td id="S3.T4.2.3.4" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.3.4.1" class="ltx_inline-block ltx_align_top" style="background-color:#CCE6FF;">
<span id="S3.T4.2.3.4.1.1" class="ltx_p">43.3</span>
</span>
</td>
<td id="S3.T4.2.3.5" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.3.5.1" class="ltx_inline-block ltx_align_top" style="background-color:#CCE6FF;">
<span id="S3.T4.2.3.5.1.1" class="ltx_p">69.3</span>
</span>
</td>
<td id="S3.T4.2.3.6" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.3.6.1" class="ltx_inline-block ltx_align_top" style="background-color:#CCE6FF;">
<span id="S3.T4.2.3.6.1.1" class="ltx_p">41.8</span>
</span>
</td>
<td id="S3.T4.2.3.7" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.3.7.1" class="ltx_inline-block ltx_align_top" style="background-color:#CCE6FF;">
<span id="S3.T4.2.3.7.1.1" class="ltx_p">45.6</span>
</span>
</td>
<td id="S3.T4.2.3.8" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.3.8.1" class="ltx_inline-block ltx_align_top" style="background-color:#CCE6FF;">
<span id="S3.T4.2.3.8.1.1" class="ltx_p">63.6</span>
</span>
</td>
<td id="S3.T4.2.3.9" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.3.9.1" class="ltx_inline-block ltx_align_top" style="background-color:#CCE6FF;">
<span id="S3.T4.2.3.9.1.1" class="ltx_p">34.8</span>
</span>
</td>
</tr>
<tr id="S3.T4.2.4" class="ltx_tr">
<td id="S3.T4.2.4.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.4.1.1" class="ltx_text ltx_font_bold">phi-1_5</span>†</td>
<td id="S3.T4.2.4.2" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.4.2.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T4.2.4.2.1.1" class="ltx_p">1.3B</span>
</span>
</td>
<td id="S3.T4.2.4.3" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.4.3.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T4.2.4.3.1.1" class="ltx_p">47.7</span>
</span>
</td>
<td id="S3.T4.2.4.4" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.4.4.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T4.2.4.4.1.1" class="ltx_p">52.9</span>
</span>
</td>
<td id="S3.T4.2.4.5" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.4.5.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T4.2.4.5.1.1" class="ltx_p">63.8</span>
</span>
</td>
<td id="S3.T4.2.4.6" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.4.6.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T4.2.4.6.1.1" class="ltx_p">43.9</span>
</span>
</td>
<td id="S3.T4.2.4.7" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.4.7.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T4.2.4.7.1.1" class="ltx_p">40.9</span>
</span>
</td>
<td id="S3.T4.2.4.8" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.4.8.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T4.2.4.8.1.1" class="ltx_p">72.2</span>
</span>
</td>
<td id="S3.T4.2.4.9" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.4.9.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T4.2.4.9.1.1" class="ltx_p">12.4</span>
</span>
</td>
</tr>
<tr id="S3.T4.2.5" class="ltx_tr">
<td id="S3.T4.2.5.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;"><span id="S3.T4.2.5.1.1" class="ltx_text ltx_font_bold">stablelm-3b-4e1t</span></td>
<td id="S3.T4.2.5.2" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.5.2.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T4.2.5.2.1.1" class="ltx_p">2.7B</span>
</span>
</td>
<td id="S3.T4.2.5.3" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.5.3.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T4.2.5.3.1.1" class="ltx_p">46.6</span>
</span>
</td>
<td id="S3.T4.2.5.4" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.5.4.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T4.2.5.4.1.1" class="ltx_p">46.6</span>
</span>
</td>
<td id="S3.T4.2.5.5" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.5.5.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T4.2.5.5.1.1" class="ltx_p">75.9</span>
</span>
</td>
<td id="S3.T4.2.5.6" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.5.6.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T4.2.5.6.1.1" class="ltx_p">45.2</span>
</span>
</td>
<td id="S3.T4.2.5.7" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.5.7.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T4.2.5.7.1.1" class="ltx_p">37.2</span>
</span>
</td>
<td id="S3.T4.2.5.8" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.5.8.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T4.2.5.8.1.1" class="ltx_p">71.2</span>
</span>
</td>
<td id="S3.T4.2.5.9" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.5.9.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T4.2.5.9.1.1" class="ltx_p">3.3</span>
</span>
</td>
</tr>
<tr id="S3.T4.2.6" class="ltx_tr">
<td id="S3.T4.2.6.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;"><span id="S3.T4.2.6.1.1" class="ltx_text ltx_font_bold">Qwen-1.5-1.8B</span></td>
<td id="S3.T4.2.6.2" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.6.2.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T4.2.6.2.1.1" class="ltx_p">1.8B</span>
</span>
</td>
<td id="S3.T4.2.6.3" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.6.3.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T4.2.6.3.1.1" class="ltx_p">46.6</span>
</span>
</td>
<td id="S3.T4.2.6.4" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.6.4.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T4.2.6.4.1.1" class="ltx_p">37.9</span>
</span>
</td>
<td id="S3.T4.2.6.5" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.6.5.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T4.2.6.5.1.1" class="ltx_p">61.4</span>
</span>
</td>
<td id="S3.T4.2.6.6" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.6.6.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T4.2.6.6.1.1" class="ltx_p">46.7</span>
</span>
</td>
<td id="S3.T4.2.6.7" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.6.7.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T4.2.6.7.1.1" class="ltx_p">39.4</span>
</span>
</td>
<td id="S3.T4.2.6.8" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.6.8.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T4.2.6.8.1.1" class="ltx_p">60.3</span>
</span>
</td>
<td id="S3.T4.2.6.9" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.6.9.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T4.2.6.9.1.1" class="ltx_p">33.6</span>
</span>
</td>
</tr>
<tr id="S3.T4.2.7" class="ltx_tr">
<td id="S3.T4.2.7.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;"><span id="S3.T4.2.7.1.1" class="ltx_text ltx_font_bold">gemma-2b</span></td>
<td id="S3.T4.2.7.2" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.7.2.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T4.2.7.2.1.1" class="ltx_p">2.5B</span>
</span>
</td>
<td id="S3.T4.2.7.3" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.7.3.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T4.2.7.3.1.1" class="ltx_p">46.5</span>
</span>
</td>
<td id="S3.T4.2.7.4" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.7.4.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T4.2.7.4.1.1" class="ltx_p">48.5</span>
</span>
</td>
<td id="S3.T4.2.7.5" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.7.5.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T4.2.7.5.1.1" class="ltx_p">71.7</span>
</span>
</td>
<td id="S3.T4.2.7.6" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.7.6.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T4.2.7.6.1.1" class="ltx_p">41.7</span>
</span>
</td>
<td id="S3.T4.2.7.7" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.7.7.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T4.2.7.7.1.1" class="ltx_p">33.1</span>
</span>
</td>
<td id="S3.T4.2.7.8" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.7.8.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T4.2.7.8.1.1" class="ltx_p">66.8</span>
</span>
</td>
<td id="S3.T4.2.7.9" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.7.9.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T4.2.7.9.1.1" class="ltx_p">17.4</span>
</span>
</td>
</tr>
<tr id="S3.T4.2.8" class="ltx_tr" style="background-color:#CCE6FF;">
<td id="S3.T4.2.8.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;"><span id="S3.T4.2.8.1.1" class="ltx_text ltx_font_bold" style="background-color:#CCE6FF;">stablelm-2-1_6b</span></td>
<td id="S3.T4.2.8.2" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.8.2.1" class="ltx_inline-block ltx_align_top" style="background-color:#CCE6FF;">
<span id="S3.T4.2.8.2.1.1" class="ltx_p">1.6B</span>
</span>
</td>
<td id="S3.T4.2.8.3" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.8.3.1" class="ltx_inline-block ltx_align_top" style="background-color:#CCE6FF;">
<span id="S3.T4.2.8.3.1.1" class="ltx_p">45.3</span>
</span>
</td>
<td id="S3.T4.2.8.4" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.8.4.1" class="ltx_inline-block ltx_align_top" style="background-color:#CCE6FF;">
<span id="S3.T4.2.8.4.1.1" class="ltx_p">43.3</span>
</span>
</td>
<td id="S3.T4.2.8.5" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.8.5.1" class="ltx_inline-block ltx_align_top" style="background-color:#CCE6FF;">
<span id="S3.T4.2.8.5.1.1" class="ltx_p">70.5</span>
</span>
</td>
<td id="S3.T4.2.8.6" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.8.6.1" class="ltx_inline-block ltx_align_top" style="background-color:#CCE6FF;">
<span id="S3.T4.2.8.6.1.1" class="ltx_p">38.9</span>
</span>
</td>
<td id="S3.T4.2.8.7" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.8.7.1" class="ltx_inline-block ltx_align_top" style="background-color:#CCE6FF;">
<span id="S3.T4.2.8.7.1.1" class="ltx_p">36.8</span>
</span>
</td>
<td id="S3.T4.2.8.8" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.8.8.1" class="ltx_inline-block ltx_align_top" style="background-color:#CCE6FF;">
<span id="S3.T4.2.8.8.1.1" class="ltx_p">64.6</span>
</span>
</td>
<td id="S3.T4.2.8.9" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.8.9.1" class="ltx_inline-block ltx_align_top" style="background-color:#CCE6FF;">
<span id="S3.T4.2.8.9.1.1" class="ltx_p">17.4</span>
</span>
</td>
</tr>
<tr id="S3.T4.2.9" class="ltx_tr">
<td id="S3.T4.2.9.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.9.1.1" class="ltx_text ltx_font_bold">gemma-2b-it</span>†</td>
<td id="S3.T4.2.9.2" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.9.2.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T4.2.9.2.1.1" class="ltx_p">2.5B</span>
</span>
</td>
<td id="S3.T4.2.9.3" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.9.3.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T4.2.9.3.1.1" class="ltx_p">42.7</span>
</span>
</td>
<td id="S3.T4.2.9.4" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.9.4.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T4.2.9.4.1.1" class="ltx_p">43.9</span>
</span>
</td>
<td id="S3.T4.2.9.5" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.9.5.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T4.2.9.5.1.1" class="ltx_p">62.7</span>
</span>
</td>
<td id="S3.T4.2.9.6" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.9.6.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T4.2.9.6.1.1" class="ltx_p">37.6</span>
</span>
</td>
<td id="S3.T4.2.9.7" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.9.7.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T4.2.9.7.1.1" class="ltx_p">45.8</span>
</span>
</td>
<td id="S3.T4.2.9.8" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.9.8.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T4.2.9.8.1.1" class="ltx_p">60.9</span>
</span>
</td>
<td id="S3.T4.2.9.9" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.9.9.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T4.2.9.9.1.1" class="ltx_p">5.5</span>
</span>
</td>
</tr>
<tr id="S3.T4.2.10" class="ltx_tr">
<td id="S3.T4.2.10.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;"><span id="S3.T4.2.10.1.1" class="ltx_text ltx_font_bold">open_llama_3b</span></td>
<td id="S3.T4.2.10.2" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.10.2.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T4.2.10.2.1.1" class="ltx_p">3B</span>
</span>
</td>
<td id="S3.T4.2.10.3" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.10.3.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T4.2.10.3.1.1" class="ltx_p">40.3</span>
</span>
</td>
<td id="S3.T4.2.10.4" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.10.4.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T4.2.10.4.1.1" class="ltx_p">39.9</span>
</span>
</td>
<td id="S3.T4.2.10.5" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.10.5.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T4.2.10.5.1.1" class="ltx_p">71.6</span>
</span>
</td>
<td id="S3.T4.2.10.6" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.10.6.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T4.2.10.6.1.1" class="ltx_p">27.1</span>
</span>
</td>
<td id="S3.T4.2.10.7" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.10.7.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T4.2.10.7.1.1" class="ltx_p">34.8</span>
</span>
</td>
<td id="S3.T4.2.10.8" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.10.8.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T4.2.10.8.1.1" class="ltx_p">67.0</span>
</span>
</td>
<td id="S3.T4.2.10.9" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.10.9.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T4.2.10.9.1.1" class="ltx_p">0.9</span>
</span>
</td>
</tr>
<tr id="S3.T4.2.11" class="ltx_tr">
<td id="S3.T4.2.11.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;"><span id="S3.T4.2.11.1.1" class="ltx_text ltx_font_bold">falcon-rw-1b</span></td>
<td id="S3.T4.2.11.2" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.11.2.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T4.2.11.2.1.1" class="ltx_p">1.3B</span>
</span>
</td>
<td id="S3.T4.2.11.3" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.11.3.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T4.2.11.3.1.1" class="ltx_p">37.1</span>
</span>
</td>
<td id="S3.T4.2.11.4" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.11.4.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T4.2.11.4.1.1" class="ltx_p">35.1</span>
</span>
</td>
<td id="S3.T4.2.11.5" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.11.5.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T4.2.11.5.1.1" class="ltx_p">63.6</span>
</span>
</td>
<td id="S3.T4.2.11.6" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.11.6.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T4.2.11.6.1.1" class="ltx_p">25.3</span>
</span>
</td>
<td id="S3.T4.2.11.7" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.11.7.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T4.2.11.7.1.1" class="ltx_p">35.9</span>
</span>
</td>
<td id="S3.T4.2.11.8" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.11.8.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T4.2.11.8.1.1" class="ltx_p">62.0</span>
</span>
</td>
<td id="S3.T4.2.11.9" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.11.9.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T4.2.11.9.1.1" class="ltx_p">0.5</span>
</span>
</td>
</tr>
<tr id="S3.T4.2.12" class="ltx_tr">
<td id="S3.T4.2.12.1" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_b" style="width:21.7pt;padding-bottom:0.43057pt;"><span id="S3.T4.2.12.1.1" class="ltx_text ltx_font_bold">TinyLLama-1.1B</span></td>
<td id="S3.T4.2.12.2" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_b" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.12.2.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T4.2.12.2.1.1" class="ltx_p">1.1B</span>
</span>
</td>
<td id="S3.T4.2.12.3" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_b" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.12.3.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T4.2.12.3.1.1" class="ltx_p">36.4</span>
</span>
</td>
<td id="S3.T4.2.12.4" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_b" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.12.4.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T4.2.12.4.1.1" class="ltx_p">33.9</span>
</span>
</td>
<td id="S3.T4.2.12.5" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_b" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.12.5.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T4.2.12.5.1.1" class="ltx_p">60.3</span>
</span>
</td>
<td id="S3.T4.2.12.6" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_b" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.12.6.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T4.2.12.6.1.1" class="ltx_p">26.0</span>
</span>
</td>
<td id="S3.T4.2.12.7" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_b" style="width:36.1pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.12.7.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T4.2.12.7.1.1" class="ltx_p">37.3</span>
</span>
</td>
<td id="S3.T4.2.12.8" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_b" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.12.8.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T4.2.12.8.1.1" class="ltx_p">59.5</span>
</span>
</td>
<td id="S3.T4.2.12.9" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_b" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T4.2.12.9.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T4.2.12.9.1.1" class="ltx_p">1.4</span>
</span>
</td>
</tr>
</table>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table"><span id="S3.T4.4.1.1" class="ltx_text" style="font-size:90%;">Table 4</span>: </span><span id="S3.T4.5.2" class="ltx_text" style="font-size:90%;">Comparison of Open LLM leaderboard evals. † denotes aligned models. Note that in this table, as well as in Tab. <a href="#S3.T5" title="Table 5 ‣ 3 Fine-tuning and Alignment ‣ Stable LM 2 1.6B Technical Report" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a>, we marked the Phi series of models <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib26" title="" class="ltx_ref">26</a>]</cite> as <em id="S3.T4.5.2.1" class="ltx_emph ltx_font_italic">aligned</em>. While we acknowledge that they may have only been pre-trained, by the nature of the training data used, and the self-disclaimed intended use for QA and chat formats, we believe this to be a fair statement.</span></figcaption>
</figure>
<figure id="S3.T5" class="ltx_table">
<table id="S3.T5.2" class="ltx_tabular ltx_align_middle">
<tr id="S3.T5.2.1" class="ltx_tr">
<td id="S3.T5.2.1.1" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_tt" style="width:28.9pt;padding-bottom:0.43057pt;"><span id="S3.T5.2.1.1.1" class="ltx_text ltx_font_bold">Model</span></td>
<td id="S3.T5.2.1.2" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_tt" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T5.2.1.2.1.1" class="ltx_p"><span id="S3.T5.2.1.2.1.1.1" class="ltx_text ltx_font_bold">All</span></span>
</span>
</td>
<td id="S3.T5.2.1.3" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_tt" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.1.3.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T5.2.1.3.1.1" class="ltx_p"><span id="S3.T5.2.1.3.1.1.1" class="ltx_text ltx_font_bold">EN</span></span>
</span>
</td>
<td id="S3.T5.2.1.4" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_tt" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.1.4.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T5.2.1.4.1.1" class="ltx_p"><span id="S3.T5.2.1.4.1.1.1" class="ltx_text ltx_font_bold">DE</span></span>
</span>
</td>
<td id="S3.T5.2.1.5" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_tt" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.1.5.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T5.2.1.5.1.1" class="ltx_p"><span id="S3.T5.2.1.5.1.1.1" class="ltx_text ltx_font_bold">ES</span></span>
</span>
</td>
<td id="S3.T5.2.1.6" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_tt" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.1.6.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T5.2.1.6.1.1" class="ltx_p"><span id="S3.T5.2.1.6.1.1.1" class="ltx_text ltx_font_bold">FR</span></span>
</span>
</td>
<td id="S3.T5.2.1.7" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_tt" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.1.7.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T5.2.1.7.1.1" class="ltx_p"><span id="S3.T5.2.1.7.1.1.1" class="ltx_text ltx_font_bold">IT</span></span>
</span>
</td>
<td id="S3.T5.2.1.8" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_tt" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.1.8.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T5.2.1.8.1.1" class="ltx_p"><span id="S3.T5.2.1.8.1.1.1" class="ltx_text ltx_font_bold">NL</span></span>
</span>
</td>
<td id="S3.T5.2.1.9" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_tt" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.1.9.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T5.2.1.9.1.1" class="ltx_p"><span id="S3.T5.2.1.9.1.1.1" class="ltx_text ltx_font_bold">PT</span></span>
</span>
</td>
</tr>
<tr id="S3.T5.2.2" class="ltx_tr">
<td id="S3.T5.2.2.1" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:28.9pt;padding-bottom:0.43057pt;"><span id="S3.T5.2.2.1.1" class="ltx_text ltx_font_bold">stablelm-3b-4e1t</span></td>
<td id="S3.T5.2.2.2" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.2.2.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T5.2.2.2.1.1" class="ltx_p">41.7</span>
</span>
</td>
<td id="S3.T5.2.2.3" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.2.3.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T5.2.2.3.1.1" class="ltx_p">50.9</span>
</span>
</td>
<td id="S3.T5.2.2.4" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.2.4.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T5.2.2.4.1.1" class="ltx_p">39.7</span>
</span>
</td>
<td id="S3.T5.2.2.5" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.2.5.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T5.2.2.5.1.1" class="ltx_p">40.2</span>
</span>
</td>
<td id="S3.T5.2.2.6" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.2.6.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T5.2.2.6.1.1" class="ltx_p">41.2</span>
</span>
</td>
<td id="S3.T5.2.2.7" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.2.7.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T5.2.2.7.1.1" class="ltx_p">41.1</span>
</span>
</td>
<td id="S3.T5.2.2.8" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.2.8.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T5.2.2.8.1.1" class="ltx_p">36.7</span>
</span>
</td>
<td id="S3.T5.2.2.9" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.2.9.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T5.2.2.9.1.1" class="ltx_p">41.9</span>
</span>
</td>
</tr>
<tr id="S3.T5.2.3" class="ltx_tr" style="background-color:#CCE6FF;">
<td id="S3.T5.2.3.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:28.9pt;padding-bottom:0.43057pt;"><span id="S3.T5.2.3.1.1" class="ltx_text ltx_font_bold" style="background-color:#CCE6FF;">stablelm-2-zephyr-1_6b<span id="S3.T5.2.3.1.1.1" class="ltx_text ltx_font_medium">†</span></span></td>
<td id="S3.T5.2.3.2" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.3.2.1" class="ltx_inline-block ltx_align_top" style="background-color:#CCE6FF;">
<span id="S3.T5.2.3.2.1.1" class="ltx_p">41.5</span>
</span>
</td>
<td id="S3.T5.2.3.3" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.3.3.1" class="ltx_inline-block ltx_align_top" style="background-color:#CCE6FF;">
<span id="S3.T5.2.3.3.1.1" class="ltx_p">49.5</span>
</span>
</td>
<td id="S3.T5.2.3.4" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.3.4.1" class="ltx_inline-block ltx_align_top" style="background-color:#CCE6FF;">
<span id="S3.T5.2.3.4.1.1" class="ltx_p">40.2</span>
</span>
</td>
<td id="S3.T5.2.3.5" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.3.5.1" class="ltx_inline-block ltx_align_top" style="background-color:#CCE6FF;">
<span id="S3.T5.2.3.5.1.1" class="ltx_p">40.0</span>
</span>
</td>
<td id="S3.T5.2.3.6" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.3.6.1" class="ltx_inline-block ltx_align_top" style="background-color:#CCE6FF;">
<span id="S3.T5.2.3.6.1.1" class="ltx_p">39.8</span>
</span>
</td>
<td id="S3.T5.2.3.7" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.3.7.1" class="ltx_inline-block ltx_align_top" style="background-color:#CCE6FF;">
<span id="S3.T5.2.3.7.1.1" class="ltx_p">39.9</span>
</span>
</td>
<td id="S3.T5.2.3.8" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.3.8.1" class="ltx_inline-block ltx_align_top" style="background-color:#CCE6FF;">
<span id="S3.T5.2.3.8.1.1" class="ltx_p">38.8</span>
</span>
</td>
<td id="S3.T5.2.3.9" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.3.9.1" class="ltx_inline-block ltx_align_top" style="background-color:#CCE6FF;">
<span id="S3.T5.2.3.9.1.1" class="ltx_p">42.0</span>
</span>
</td>
</tr>
<tr id="S3.T5.2.4" class="ltx_tr" style="background-color:#CCE6FF;">
<td id="S3.T5.2.4.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:28.9pt;padding-bottom:0.43057pt;"><span id="S3.T5.2.4.1.1" class="ltx_text ltx_font_bold" style="background-color:#CCE6FF;">stablelm-2-1_6b</span></td>
<td id="S3.T5.2.4.2" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.4.2.1" class="ltx_inline-block ltx_align_top" style="background-color:#CCE6FF;">
<span id="S3.T5.2.4.2.1.1" class="ltx_p">40.5</span>
</span>
</td>
<td id="S3.T5.2.4.3" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.4.3.1" class="ltx_inline-block ltx_align_top" style="background-color:#CCE6FF;">
<span id="S3.T5.2.4.3.1.1" class="ltx_p">48.7</span>
</span>
</td>
<td id="S3.T5.2.4.4" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.4.4.1" class="ltx_inline-block ltx_align_top" style="background-color:#CCE6FF;">
<span id="S3.T5.2.4.4.1.1" class="ltx_p">39.1</span>
</span>
</td>
<td id="S3.T5.2.4.5" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.4.5.1" class="ltx_inline-block ltx_align_top" style="background-color:#CCE6FF;">
<span id="S3.T5.2.4.5.1.1" class="ltx_p">39.0</span>
</span>
</td>
<td id="S3.T5.2.4.6" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.4.6.1" class="ltx_inline-block ltx_align_top" style="background-color:#CCE6FF;">
<span id="S3.T5.2.4.6.1.1" class="ltx_p">39.3</span>
</span>
</td>
<td id="S3.T5.2.4.7" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.4.7.1" class="ltx_inline-block ltx_align_top" style="background-color:#CCE6FF;">
<span id="S3.T5.2.4.7.1.1" class="ltx_p">38.8</span>
</span>
</td>
<td id="S3.T5.2.4.8" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.4.8.1" class="ltx_inline-block ltx_align_top" style="background-color:#CCE6FF;">
<span id="S3.T5.2.4.8.1.1" class="ltx_p">37.8</span>
</span>
</td>
<td id="S3.T5.2.4.9" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.4.9.1" class="ltx_inline-block ltx_align_top" style="background-color:#CCE6FF;">
<span id="S3.T5.2.4.9.1.1" class="ltx_p">41.2</span>
</span>
</td>
</tr>
<tr id="S3.T5.2.5" class="ltx_tr">
<td id="S3.T5.2.5.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:28.9pt;padding-bottom:0.43057pt;"><span id="S3.T5.2.5.1.1" class="ltx_text ltx_font_bold">gemma-2b</span></td>
<td id="S3.T5.2.5.2" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.5.2.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T5.2.5.2.1.1" class="ltx_p">39.8</span>
</span>
</td>
<td id="S3.T5.2.5.3" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.5.3.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T5.2.5.3.1.1" class="ltx_p">48.6</span>
</span>
</td>
<td id="S3.T5.2.5.4" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.5.4.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T5.2.5.4.1.1" class="ltx_p">38.3</span>
</span>
</td>
<td id="S3.T5.2.5.5" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.5.5.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T5.2.5.5.1.1" class="ltx_p">38.7</span>
</span>
</td>
<td id="S3.T5.2.5.6" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.5.6.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T5.2.5.6.1.1" class="ltx_p">38.7</span>
</span>
</td>
<td id="S3.T5.2.5.7" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.5.7.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T5.2.5.7.1.1" class="ltx_p">38.4</span>
</span>
</td>
<td id="S3.T5.2.5.8" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.5.8.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T5.2.5.8.1.1" class="ltx_p">35.1</span>
</span>
</td>
<td id="S3.T5.2.5.9" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.5.9.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T5.2.5.9.1.1" class="ltx_p">40.5</span>
</span>
</td>
</tr>
<tr id="S3.T5.2.6" class="ltx_tr">
<td id="S3.T5.2.6.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:28.9pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.6.1.1" class="ltx_text ltx_font_bold">gemma-2b-it</span>†</td>
<td id="S3.T5.2.6.2" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.6.2.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T5.2.6.2.1.1" class="ltx_p">38.2</span>
</span>
</td>
<td id="S3.T5.2.6.3" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.6.3.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T5.2.6.3.1.1" class="ltx_p">49.4</span>
</span>
</td>
<td id="S3.T5.2.6.4" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.6.4.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T5.2.6.4.1.1" class="ltx_p">36.8</span>
</span>
</td>
<td id="S3.T5.2.6.5" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.6.5.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T5.2.6.5.1.1" class="ltx_p">38.0</span>
</span>
</td>
<td id="S3.T5.2.6.6" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.6.6.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T5.2.6.6.1.1" class="ltx_p">37.5</span>
</span>
</td>
<td id="S3.T5.2.6.7" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.6.7.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T5.2.6.7.1.1" class="ltx_p">35.5</span>
</span>
</td>
<td id="S3.T5.2.6.8" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.6.8.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T5.2.6.8.1.1" class="ltx_p">32.1</span>
</span>
</td>
<td id="S3.T5.2.6.9" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.6.9.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T5.2.6.9.1.1" class="ltx_p">38.1</span>
</span>
</td>
</tr>
<tr id="S3.T5.2.7" class="ltx_tr">
<td id="S3.T5.2.7.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:28.9pt;padding-bottom:0.43057pt;"><span id="S3.T5.2.7.1.1" class="ltx_text ltx_font_bold">open_llama_3b</span></td>
<td id="S3.T5.2.7.2" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.7.2.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T5.2.7.2.1.1" class="ltx_p">37.5</span>
</span>
</td>
<td id="S3.T5.2.7.3" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.7.3.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T5.2.7.3.1.1" class="ltx_p">47.3</span>
</span>
</td>
<td id="S3.T5.2.7.4" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.7.4.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T5.2.7.4.1.1" class="ltx_p">35.2</span>
</span>
</td>
<td id="S3.T5.2.7.5" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.7.5.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T5.2.7.5.1.1" class="ltx_p">36.4</span>
</span>
</td>
<td id="S3.T5.2.7.6" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.7.6.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T5.2.7.6.1.1" class="ltx_p">37.6</span>
</span>
</td>
<td id="S3.T5.2.7.7" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.7.7.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T5.2.7.7.1.1" class="ltx_p">37.1</span>
</span>
</td>
<td id="S3.T5.2.7.8" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.7.8.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T5.2.7.8.1.1" class="ltx_p">32.2</span>
</span>
</td>
<td id="S3.T5.2.7.9" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.7.9.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T5.2.7.9.1.1" class="ltx_p">36.8</span>
</span>
</td>
</tr>
<tr id="S3.T5.2.8" class="ltx_tr">
<td id="S3.T5.2.8.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:28.9pt;padding-bottom:0.43057pt;"><span id="S3.T5.2.8.1.1" class="ltx_text ltx_font_bold">Qwen-1.5-1.8B-Chat</span></td>
<td id="S3.T5.2.8.2" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.8.2.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T5.2.8.2.1.1" class="ltx_p">35.5</span>
</span>
</td>
<td id="S3.T5.2.8.3" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.8.3.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T5.2.8.3.1.1" class="ltx_p">46.2</span>
</span>
</td>
<td id="S3.T5.2.8.4" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.8.4.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T5.2.8.4.1.1" class="ltx_p">33.2</span>
</span>
</td>
<td id="S3.T5.2.8.5" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.8.5.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T5.2.8.5.1.1" class="ltx_p">35.1</span>
</span>
</td>
<td id="S3.T5.2.8.6" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.8.6.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T5.2.8.6.1.1" class="ltx_p">34.3</span>
</span>
</td>
<td id="S3.T5.2.8.7" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.8.7.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T5.2.8.7.1.1" class="ltx_p">33.2</span>
</span>
</td>
<td id="S3.T5.2.8.8" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.8.8.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T5.2.8.8.1.1" class="ltx_p">30.9</span>
</span>
</td>
<td id="S3.T5.2.8.9" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.8.9.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T5.2.8.9.1.1" class="ltx_p">35.7</span>
</span>
</td>
</tr>
<tr id="S3.T5.2.9" class="ltx_tr">
<td id="S3.T5.2.9.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:28.9pt;padding-bottom:0.43057pt;"><span id="S3.T5.2.9.1.1" class="ltx_text ltx_font_bold">Qwen-1.5-1.8B</span></td>
<td id="S3.T5.2.9.2" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.9.2.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T5.2.9.2.1.1" class="ltx_p">34.8</span>
</span>
</td>
<td id="S3.T5.2.9.3" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.9.3.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T5.2.9.3.1.1" class="ltx_p">46.3</span>
</span>
</td>
<td id="S3.T5.2.9.4" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.9.4.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T5.2.9.4.1.1" class="ltx_p">31.8</span>
</span>
</td>
<td id="S3.T5.2.9.5" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.9.5.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T5.2.9.5.1.1" class="ltx_p">34.0</span>
</span>
</td>
<td id="S3.T5.2.9.6" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.9.6.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T5.2.9.6.1.1" class="ltx_p">34.2</span>
</span>
</td>
<td id="S3.T5.2.9.7" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.9.7.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T5.2.9.7.1.1" class="ltx_p">32.8</span>
</span>
</td>
<td id="S3.T5.2.9.8" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.9.8.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T5.2.9.8.1.1" class="ltx_p">29.7</span>
</span>
</td>
<td id="S3.T5.2.9.9" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.9.9.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T5.2.9.9.1.1" class="ltx_p">35.0</span>
</span>
</td>
</tr>
<tr id="S3.T5.2.10" class="ltx_tr">
<td id="S3.T5.2.10.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:28.9pt;padding-bottom:0.43057pt;"><span id="S3.T5.2.10.1.1" class="ltx_text ltx_font_bold">TinyLLama-1.1B</span></td>
<td id="S3.T5.2.10.2" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.10.2.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T5.2.10.2.1.1" class="ltx_p">34.8</span>
</span>
</td>
<td id="S3.T5.2.10.3" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.10.3.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T5.2.10.3.1.1" class="ltx_p">42.4</span>
</span>
</td>
<td id="S3.T5.2.10.4" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.10.4.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T5.2.10.4.1.1" class="ltx_p">33.0</span>
</span>
</td>
<td id="S3.T5.2.10.5" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.10.5.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T5.2.10.5.1.1" class="ltx_p">33.8</span>
</span>
</td>
<td id="S3.T5.2.10.6" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.10.6.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T5.2.10.6.1.1" class="ltx_p">34.7</span>
</span>
</td>
<td id="S3.T5.2.10.7" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.10.7.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T5.2.10.7.1.1" class="ltx_p">33.5</span>
</span>
</td>
<td id="S3.T5.2.10.8" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.10.8.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T5.2.10.8.1.1" class="ltx_p">31.0</span>
</span>
</td>
<td id="S3.T5.2.10.9" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.10.9.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T5.2.10.9.1.1" class="ltx_p">35.0</span>
</span>
</td>
</tr>
<tr id="S3.T5.2.11" class="ltx_tr">
<td id="S3.T5.2.11.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:28.9pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.11.1.1" class="ltx_text ltx_font_bold">phi-2</span>†</td>
<td id="S3.T5.2.11.2" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.11.2.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T5.2.11.2.1.1" class="ltx_p">34.6</span>
</span>
</td>
<td id="S3.T5.2.11.3" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.11.3.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T5.2.11.3.1.1" class="ltx_p">55.8</span>
</span>
</td>
<td id="S3.T5.2.11.4" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.11.4.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T5.2.11.4.1.1" class="ltx_p">29.0</span>
</span>
</td>
<td id="S3.T5.2.11.5" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.11.5.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T5.2.11.5.1.1" class="ltx_p">34.3</span>
</span>
</td>
<td id="S3.T5.2.11.6" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.11.6.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T5.2.11.6.1.1" class="ltx_p">32.9</span>
</span>
</td>
<td id="S3.T5.2.11.7" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.11.7.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T5.2.11.7.1.1" class="ltx_p">29.9</span>
</span>
</td>
<td id="S3.T5.2.11.8" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.11.8.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T5.2.11.8.1.1" class="ltx_p">27.1</span>
</span>
</td>
<td id="S3.T5.2.11.9" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.11.9.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T5.2.11.9.1.1" class="ltx_p">33.4</span>
</span>
</td>
</tr>
<tr id="S3.T5.2.12" class="ltx_tr">
<td id="S3.T5.2.12.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:28.9pt;padding-bottom:0.43057pt;"><span id="S3.T5.2.12.1.1" class="ltx_text ltx_font_bold">falcon-rw-1b</span></td>
<td id="S3.T5.2.12.2" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.12.2.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T5.2.12.2.1.1" class="ltx_p">29.9</span>
</span>
</td>
<td id="S3.T5.2.12.3" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.12.3.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T5.2.12.3.1.1" class="ltx_p">42.2</span>
</span>
</td>
<td id="S3.T5.2.12.4" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.12.4.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T5.2.12.4.1.1" class="ltx_p">27.4</span>
</span>
</td>
<td id="S3.T5.2.12.5" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.12.5.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T5.2.12.5.1.1" class="ltx_p">28.6</span>
</span>
</td>
<td id="S3.T5.2.12.6" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.12.6.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T5.2.12.6.1.1" class="ltx_p">28.3</span>
</span>
</td>
<td id="S3.T5.2.12.7" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.12.7.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T5.2.12.7.1.1" class="ltx_p">28.0</span>
</span>
</td>
<td id="S3.T5.2.12.8" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.12.8.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T5.2.12.8.1.1" class="ltx_p">25.9</span>
</span>
</td>
<td id="S3.T5.2.12.9" class="ltx_td ltx_align_justify ltx_align_middle" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.12.9.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T5.2.12.9.1.1" class="ltx_p">29.1</span>
</span>
</td>
</tr>
<tr id="S3.T5.2.13" class="ltx_tr">
<td id="S3.T5.2.13.1" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_b" style="width:28.9pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.13.1.1" class="ltx_text ltx_font_bold">phi-1_5</span>†</td>
<td id="S3.T5.2.13.2" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_b" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.13.2.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T5.2.13.2.1.1" class="ltx_p">29.7</span>
</span>
</td>
<td id="S3.T5.2.13.3" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_b" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.13.3.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T5.2.13.3.1.1" class="ltx_p">47.1</span>
</span>
</td>
<td id="S3.T5.2.13.4" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_b" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.13.4.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T5.2.13.4.1.1" class="ltx_p">25.3</span>
</span>
</td>
<td id="S3.T5.2.13.5" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_b" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.13.5.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T5.2.13.5.1.1" class="ltx_p">28.7</span>
</span>
</td>
<td id="S3.T5.2.13.6" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_b" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.13.6.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T5.2.13.6.1.1" class="ltx_p">26.8</span>
</span>
</td>
<td id="S3.T5.2.13.7" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_b" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.13.7.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T5.2.13.7.1.1" class="ltx_p">27.2</span>
</span>
</td>
<td id="S3.T5.2.13.8" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_b" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.13.8.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T5.2.13.8.1.1" class="ltx_p">24.1</span>
</span>
</td>
<td id="S3.T5.2.13.9" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_b" style="width:21.7pt;padding-bottom:0.43057pt;">
<span id="S3.T5.2.13.9.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T5.2.13.9.1.1" class="ltx_p">29.0</span>
</span>
</td>
</tr>
</table>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table"><span id="S3.T5.3.1.1" class="ltx_text" style="font-size:90%;">Table 5</span>: </span><span id="S3.T5.4.2" class="ltx_text" style="font-size:90%;">Multilingual evaluations. As in the previous table, † denotes aligned models.
</span></figcaption>
</figure>
<section id="S3.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.1 </span>SFT</h3>

<div id="S3.SS1.p1" class="ltx_para">
<p id="S3.SS1.p1.1" class="ltx_p">The first step is supervised fine-tuning.
We fine-tune the pre-trained model on a number of instruction datasets publicly available on the Hugging Face Hub.
In particular, we use the following <em id="S3.SS1.p1.1.1" class="ltx_emph ltx_font_italic">conversational</em> datasets: UltraChat <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib18" title="" class="ltx_ref">18</a>]</cite>, WizardLM <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib77" title="" class="ltx_ref">77</a>]</cite>, SlimOrca <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib41" title="" class="ltx_ref">41</a>]</cite>, ShareGPT <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib72" title="" class="ltx_ref">72</a>]</cite>, Capybara <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib15" title="" class="ltx_ref">15</a>]</cite>, Deita <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib46" title="" class="ltx_ref">46</a>]</cite>, and MetaMathQA <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib80" title="" class="ltx_ref">80</a>]</cite>.
We removed any samples that exceeded eight turns, leading to a total of 826,938 samples.</p>
</div>
<figure id="S3.T6" class="ltx_table">
<table id="S3.T6.2" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="S3.T6.2.1" class="ltx_tr">
<td id="S3.T6.2.1.1" class="ltx_td ltx_align_left ltx_border_tt" style="padding-bottom:4.30554pt;"><span id="S3.T6.2.1.1.1" class="ltx_text ltx_font_bold">Dataset</span></td>
<td id="S3.T6.2.1.2" class="ltx_td ltx_align_left ltx_border_tt" style="padding-bottom:4.30554pt;"><span id="S3.T6.2.1.2.1" class="ltx_text ltx_font_bold">Type</span></td>
<td id="S3.T6.2.1.3" class="ltx_td ltx_align_left ltx_border_tt" style="padding-bottom:4.30554pt;"><span id="S3.T6.2.1.3.1" class="ltx_text ltx_font_bold">Source</span></td>
<td id="S3.T6.2.1.4" class="ltx_td ltx_align_right ltx_border_tt" style="padding-bottom:4.30554pt;"><span id="S3.T6.2.1.4.1" class="ltx_text ltx_font_bold">Number of Samples</span></td>
</tr>
<tr id="S3.T6.2.2" class="ltx_tr">
<td id="S3.T6.2.2.1" class="ltx_td ltx_align_left ltx_border_t" style="padding-bottom:0.86108pt;">UltraChat</td>
<td id="S3.T6.2.2.2" class="ltx_td ltx_align_left ltx_border_t" style="padding-bottom:0.86108pt;">SFT</td>
<td id="S3.T6.2.2.3" class="ltx_td ltx_align_left ltx_border_t" style="padding-bottom:0.86108pt;">HuggingFaceH4/ultrachat_200k</td>
<td id="S3.T6.2.2.4" class="ltx_td ltx_align_right ltx_border_t" style="padding-bottom:0.86108pt;">194,409</td>
</tr>
<tr id="S3.T6.2.3" class="ltx_tr">
<td id="S3.T6.2.3.1" class="ltx_td ltx_align_left" style="padding-bottom:0.86108pt;">WizardLM</td>
<td id="S3.T6.2.3.2" class="ltx_td ltx_align_left" style="padding-bottom:0.86108pt;">SFT</td>
<td id="S3.T6.2.3.3" class="ltx_td ltx_align_left" style="padding-bottom:0.86108pt;">WizardLM/WizardLM_evol_instruct_V2_196k</td>
<td id="S3.T6.2.3.4" class="ltx_td ltx_align_right" style="padding-bottom:0.86108pt;">80,662</td>
</tr>
<tr id="S3.T6.2.4" class="ltx_tr">
<td id="S3.T6.2.4.1" class="ltx_td ltx_align_left" style="padding-bottom:0.86108pt;">SlimOrca</td>
<td id="S3.T6.2.4.2" class="ltx_td ltx_align_left" style="padding-bottom:0.86108pt;">SFT</td>
<td id="S3.T6.2.4.3" class="ltx_td ltx_align_left" style="padding-bottom:0.86108pt;">Open-Orca/SlimOrca-Dedup</td>
<td id="S3.T6.2.4.4" class="ltx_td ltx_align_right" style="padding-bottom:0.86108pt;">143,789</td>
</tr>
<tr id="S3.T6.2.5" class="ltx_tr">
<td id="S3.T6.2.5.1" class="ltx_td ltx_align_left" style="padding-bottom:0.86108pt;">ShareGPT</td>
<td id="S3.T6.2.5.2" class="ltx_td ltx_align_left" style="padding-bottom:0.86108pt;">SFT</td>
<td id="S3.T6.2.5.3" class="ltx_td ltx_align_left" style="padding-bottom:0.86108pt;">openchat/openchat_sharegpt4_dataset</td>
<td id="S3.T6.2.5.4" class="ltx_td ltx_align_right" style="padding-bottom:0.86108pt;">3,509</td>
</tr>
<tr id="S3.T6.2.6" class="ltx_tr">
<td id="S3.T6.2.6.1" class="ltx_td ltx_align_left" style="padding-bottom:0.86108pt;">Capybara</td>
<td id="S3.T6.2.6.2" class="ltx_td ltx_align_left" style="padding-bottom:0.86108pt;">SFT</td>
<td id="S3.T6.2.6.3" class="ltx_td ltx_align_left" style="padding-bottom:0.86108pt;">LDJnr/Capybara</td>
<td id="S3.T6.2.6.4" class="ltx_td ltx_align_right" style="padding-bottom:0.86108pt;">7,291</td>
</tr>
<tr id="S3.T6.2.7" class="ltx_tr">
<td id="S3.T6.2.7.1" class="ltx_td ltx_align_left" style="padding-bottom:0.86108pt;">Deita</td>
<td id="S3.T6.2.7.2" class="ltx_td ltx_align_left" style="padding-bottom:0.86108pt;">SFT</td>
<td id="S3.T6.2.7.3" class="ltx_td ltx_align_left" style="padding-bottom:0.86108pt;">hkust-nlp/deita-10k-v0</td>
<td id="S3.T6.2.7.4" class="ltx_td ltx_align_right" style="padding-bottom:0.86108pt;">2,860</td>
</tr>
<tr id="S3.T6.2.8" class="ltx_tr">
<td id="S3.T6.2.8.1" class="ltx_td ltx_align_left" style="padding-bottom:0.86108pt;">MetaMathQA</td>
<td id="S3.T6.2.8.2" class="ltx_td ltx_align_left" style="padding-bottom:0.86108pt;">SFT</td>
<td id="S3.T6.2.8.3" class="ltx_td ltx_align_left" style="padding-bottom:0.86108pt;">meta-math/MetaMathQA</td>
<td id="S3.T6.2.8.4" class="ltx_td ltx_align_right" style="padding-bottom:0.86108pt;">394,418</td>
</tr>
<tr id="S3.T6.2.9" class="ltx_tr">
<td id="S3.T6.2.9.1" class="ltx_td ltx_align_left ltx_border_t" style="padding-bottom:0.86108pt;">Ultrafeedback</td>
<td id="S3.T6.2.9.2" class="ltx_td ltx_align_left ltx_border_t" style="padding-bottom:0.86108pt;">DPO</td>
<td id="S3.T6.2.9.3" class="ltx_td ltx_align_left ltx_border_t" style="padding-bottom:0.86108pt;">argilla/ultrafeedback-binarized-preferences</td>
<td id="S3.T6.2.9.4" class="ltx_td ltx_align_right ltx_border_t" style="padding-bottom:0.86108pt;">63,169</td>
</tr>
<tr id="S3.T6.2.10" class="ltx_tr">
<td id="S3.T6.2.10.1" class="ltx_td ltx_align_left ltx_border_b" style="padding-bottom:0.86108pt;">Orca Pairs</td>
<td id="S3.T6.2.10.2" class="ltx_td ltx_align_left ltx_border_b" style="padding-bottom:0.86108pt;">DPO</td>
<td id="S3.T6.2.10.3" class="ltx_td ltx_align_left ltx_border_b" style="padding-bottom:0.86108pt;">Intel/orca_dpo_pairs</td>
<td id="S3.T6.2.10.4" class="ltx_td ltx_align_right ltx_border_b" style="padding-bottom:0.86108pt;">12,859</td>
</tr>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table"><span id="S3.T6.3.1.1" class="ltx_text" style="font-size:90%;">Table 6</span>: </span><span id="S3.T6.4.2" class="ltx_text" style="font-size:90%;">Fine-tuning datasets</span></figcaption>
</figure>
<div id="S3.SS1.p2" class="ltx_para">
<p id="S3.SS1.p2.2" class="ltx_p">We train our SFT models for three epochs using a cosine learning rate scheduler. A warm-up phase of <math id="S3.SS1.p2.1.m1.1" class="ltx_Math" alttext="10\%" display="inline"><semantics id="S3.SS1.p2.1.m1.1a"><mrow id="S3.SS1.p2.1.m1.1.1" xref="S3.SS1.p2.1.m1.1.1.cmml"><mn id="S3.SS1.p2.1.m1.1.1.2" xref="S3.SS1.p2.1.m1.1.1.2.cmml">10</mn><mo id="S3.SS1.p2.1.m1.1.1.1" xref="S3.SS1.p2.1.m1.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p2.1.m1.1b"><apply id="S3.SS1.p2.1.m1.1.1.cmml" xref="S3.SS1.p2.1.m1.1.1"><csymbol cd="latexml" id="S3.SS1.p2.1.m1.1.1.1.cmml" xref="S3.SS1.p2.1.m1.1.1.1">percent</csymbol><cn type="integer" id="S3.SS1.p2.1.m1.1.1.2.cmml" xref="S3.SS1.p2.1.m1.1.1.2">10</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p2.1.m1.1c">10\%</annotation></semantics></math> of the training duration is employed before reaching the peak learning rate of <math id="S3.SS1.p2.2.m2.1" class="ltx_Math" alttext="5e-5" display="inline"><semantics id="S3.SS1.p2.2.m2.1a"><mrow id="S3.SS1.p2.2.m2.1.1" xref="S3.SS1.p2.2.m2.1.1.cmml"><mrow id="S3.SS1.p2.2.m2.1.1.2" xref="S3.SS1.p2.2.m2.1.1.2.cmml"><mn id="S3.SS1.p2.2.m2.1.1.2.2" xref="S3.SS1.p2.2.m2.1.1.2.2.cmml">5</mn><mo lspace="0em" rspace="0em" id="S3.SS1.p2.2.m2.1.1.2.1" xref="S3.SS1.p2.2.m2.1.1.2.1.cmml">​</mo><mi id="S3.SS1.p2.2.m2.1.1.2.3" xref="S3.SS1.p2.2.m2.1.1.2.3.cmml">e</mi></mrow><mo id="S3.SS1.p2.2.m2.1.1.1" xref="S3.SS1.p2.2.m2.1.1.1.cmml">−</mo><mn id="S3.SS1.p2.2.m2.1.1.3" xref="S3.SS1.p2.2.m2.1.1.3.cmml">5</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p2.2.m2.1b"><apply id="S3.SS1.p2.2.m2.1.1.cmml" xref="S3.SS1.p2.2.m2.1.1"><minus id="S3.SS1.p2.2.m2.1.1.1.cmml" xref="S3.SS1.p2.2.m2.1.1.1"></minus><apply id="S3.SS1.p2.2.m2.1.1.2.cmml" xref="S3.SS1.p2.2.m2.1.1.2"><times id="S3.SS1.p2.2.m2.1.1.2.1.cmml" xref="S3.SS1.p2.2.m2.1.1.2.1"></times><cn type="integer" id="S3.SS1.p2.2.m2.1.1.2.2.cmml" xref="S3.SS1.p2.2.m2.1.1.2.2">5</cn><ci id="S3.SS1.p2.2.m2.1.1.2.3.cmml" xref="S3.SS1.p2.2.m2.1.1.2.3">𝑒</ci></apply><cn type="integer" id="S3.SS1.p2.2.m2.1.1.3.cmml" xref="S3.SS1.p2.2.m2.1.1.3">5</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p2.2.m2.1c">5e-5</annotation></semantics></math>. We set the global batch size to 512 sequences and pack inputs into sequences of up to 4096 tokens in length.</p>
</div>
</section>
<section id="S3.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.2 </span>DPO</h3>

<div id="S3.SS2.p1" class="ltx_para">
<p id="S3.SS2.p1.3" class="ltx_p">Direct Preference Optimization (DPO) <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib58" title="" class="ltx_ref">58</a>]</cite> has been a fundamental tool in recent strong models such as Zephyr-7B <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib70" title="" class="ltx_ref">70</a>]</cite>, Neural-Chat-7B, and Tulu-2-DPO-70B <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib32" title="" class="ltx_ref">32</a>]</cite>.
Accordingly, after applying SFT, we aligned the resulting model via DPO. We use two datasets at this stage: UltraFeedback  <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib14" title="" class="ltx_ref">14</a>]</cite> and Intel Orca Pairs. We filter the datasets by removing pairs with tied ranking, pairs with duplicated content, and pairs in which the score for the chosen response is less than eight out of ten. We train the model with DPO following the Zephyr recipe <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib70" title="" class="ltx_ref">70</a>]</cite> and borrowing most of its hyperparameters, except for <math id="S3.SS2.p1.1.m1.1" class="ltx_Math" alttext="\beta" display="inline"><semantics id="S3.SS2.p1.1.m1.1a"><mi id="S3.SS2.p1.1.m1.1.1" xref="S3.SS2.p1.1.m1.1.1.cmml">β</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.1.m1.1b"><ci id="S3.SS2.p1.1.m1.1.1.cmml" xref="S3.SS2.p1.1.m1.1.1">𝛽</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.1.m1.1c">\beta</annotation></semantics></math>, which we lower to <math id="S3.SS2.p1.2.m2.1" class="ltx_Math" alttext="0.01" display="inline"><semantics id="S3.SS2.p1.2.m2.1a"><mn id="S3.SS2.p1.2.m2.1.1" xref="S3.SS2.p1.2.m2.1.1.cmml">0.01</mn><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.2.m2.1b"><cn type="float" id="S3.SS2.p1.2.m2.1.1.cmml" xref="S3.SS2.p1.2.m2.1.1">0.01</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.2.m2.1c">0.01</annotation></semantics></math>, and the learning rate, which we lower to <math id="S3.SS2.p1.3.m3.1" class="ltx_Math" alttext="4e-6" display="inline"><semantics id="S3.SS2.p1.3.m3.1a"><mrow id="S3.SS2.p1.3.m3.1.1" xref="S3.SS2.p1.3.m3.1.1.cmml"><mrow id="S3.SS2.p1.3.m3.1.1.2" xref="S3.SS2.p1.3.m3.1.1.2.cmml"><mn id="S3.SS2.p1.3.m3.1.1.2.2" xref="S3.SS2.p1.3.m3.1.1.2.2.cmml">4</mn><mo lspace="0em" rspace="0em" id="S3.SS2.p1.3.m3.1.1.2.1" xref="S3.SS2.p1.3.m3.1.1.2.1.cmml">​</mo><mi id="S3.SS2.p1.3.m3.1.1.2.3" xref="S3.SS2.p1.3.m3.1.1.2.3.cmml">e</mi></mrow><mo id="S3.SS2.p1.3.m3.1.1.1" xref="S3.SS2.p1.3.m3.1.1.1.cmml">−</mo><mn id="S3.SS2.p1.3.m3.1.1.3" xref="S3.SS2.p1.3.m3.1.1.3.cmml">6</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.3.m3.1b"><apply id="S3.SS2.p1.3.m3.1.1.cmml" xref="S3.SS2.p1.3.m3.1.1"><minus id="S3.SS2.p1.3.m3.1.1.1.cmml" xref="S3.SS2.p1.3.m3.1.1.1"></minus><apply id="S3.SS2.p1.3.m3.1.1.2.cmml" xref="S3.SS2.p1.3.m3.1.1.2"><times id="S3.SS2.p1.3.m3.1.1.2.1.cmml" xref="S3.SS2.p1.3.m3.1.1.2.1"></times><cn type="integer" id="S3.SS2.p1.3.m3.1.1.2.2.cmml" xref="S3.SS2.p1.3.m3.1.1.2.2">4</cn><ci id="S3.SS2.p1.3.m3.1.1.2.3.cmml" xref="S3.SS2.p1.3.m3.1.1.2.3">𝑒</ci></apply><cn type="integer" id="S3.SS2.p1.3.m3.1.1.3.cmml" xref="S3.SS2.p1.3.m3.1.1.3">6</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.3.m3.1c">4e-6</annotation></semantics></math>, both of which helped improve the stability of training and final performance. This stage of training is performed using the Alignment Handbook <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib69" title="" class="ltx_ref">69</a>]</cite>.</p>
</div>
<figure id="S3.F3" class="ltx_figure"><img src="/html/2402.17834/assets/figs/mt_plot_paper.png" id="S3.F3.g1" class="ltx_graphics ltx_centering ltx_img_square" width="471" height="466" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S3.F3.2.1.1" class="ltx_text" style="font-size:90%;">Figure 3</span>: </span><span id="S3.F3.3.2" class="ltx_text" style="font-size:90%;">Stable LM 2 1.6B shows competitive performance, matching or even surpassing significantly larger models on MT-Bench.
</span></figcaption>
</figure>
</section>
<section id="S3.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.3 </span>Self-Knowledge</h3>

<div id="S3.SS3.p1" class="ltx_para">
<p id="S3.SS3.p1.1" class="ltx_p">The model after the output of the DPO <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib58" title="" class="ltx_ref">58</a>]</cite> stage does not have knowledge about who created it, or even what limitations a language model has. To remedy this, we were inspired by the data generation method of Reinforcement Learning from Contrast Distillation (RLCD) <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib78" title="" class="ltx_ref">78</a>]</cite> and the training method of Conditioned Reinforcement Learning Fine-Tuning (C-RLFT) <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib72" title="" class="ltx_ref">72</a>]</cite>, which we apply to self-knowledge training.</p>
</div>
<div id="S3.SS3.p2" class="ltx_para">
<p id="S3.SS3.p2.1" class="ltx_p">To generate the initial prompts, we use the base model to generate 10k random first messages to a language model with no duplicates. To generate continuations, we use a few-shot prompt with self-knowledge in the previous chat turns as a positive completion. For the negative prompt, we simply sample from the prompt with no additional prompting or few-shot turns.</p>
</div>
<div id="S3.SS3.p3" class="ltx_para">
<p id="S3.SS3.p3.1" class="ltx_p">We train with unpacked examples for 6 epochs using a batch size of 256, a warmup stage for 100 steps to a maximum LR of 3e-6 followed by a cosine decay to zero. The positive prompts are trained in the same way as the SFT step, while the negative prompts are trained with a negative token instead of the Assistant token.</p>
</div>
</section>
</section>
<section id="S4" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">4 </span>Experimental Results and Benchmarks</h2>

<div id="S4.p1" class="ltx_para">
<p id="S4.p1.1" class="ltx_p">This section presents the main experimental results for Stable LM 2 1.6B.
We compare with similarly-sized open-source models showing strong improvements on various tasks, including multilingual capabilities in Spanish, German, French, Italian, Portuguese, and Dutch.
For context, we also provide comparisons with much larger models.
We split our experiments into three sets: few- and zero-shot evaluations in English (as commonly done in the Hugging Face Open LLM leaderboard), multilingual evaluations, and conversational evaluations.</p>
</div>
<section id="S4.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.1 </span>Few-shot and Zero-shot Evaluations</h3>

<div id="S4.SS1.p1" class="ltx_para">
<p id="S4.SS1.p1.1" class="ltx_p">First, we assess the 0-shot and few-shot capabilities of Stable LM 2 by evaluating our model over popular benchmarks and comparing results against similarly sized open-source pre-trained models.
Table <a href="#S3.T4" title="Table 4 ‣ 3 Fine-tuning and Alignment ‣ Stable LM 2 1.6B Technical Report" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a> presents model evaluations in English.
Our results cover the 6 benchmarks from the Open LLM Leaderboard (<cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib5" title="" class="ltx_ref">5</a>]</cite>): ARC-Challenge 25-shot <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib11" title="" class="ltx_ref">11</a>]</cite> (<span id="S4.SS1.p1.1.1" class="ltx_text ltx_font_bold">ARC</span>), HellaSwag 10-shot <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib83" title="" class="ltx_ref">83</a>]</cite> (<span id="S4.SS1.p1.1.2" class="ltx_text ltx_font_bold">HS</span>), MMLU 5-shot <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib30" title="" class="ltx_ref">30</a>]</cite> (<span id="S4.SS1.p1.1.3" class="ltx_text ltx_font_bold">MMLU</span>), TruthfulQA 0-shot <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib43" title="" class="ltx_ref">43</a>]</cite> (<span id="S4.SS1.p1.1.4" class="ltx_text ltx_font_bold">TQA</span>), WinoGrande 5-shot <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib62" title="" class="ltx_ref">62</a>]</cite> (<span id="S4.SS1.p1.1.5" class="ltx_text ltx_font_bold">Wino</span>) and GSM8K 5-shot <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib12" title="" class="ltx_ref">12</a>]</cite> (<span id="S4.SS1.p1.1.6" class="ltx_text ltx_font_bold">GSM</span>).
Also, as Stable LM 2 is a general-purpose foundational model, we further assess natural language understanding capabilities by evaluating English and machine-translated versions of LAMBADA.
All evaluations are performed with the Language Model Evaluation Harness framework <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib23" title="" class="ltx_ref">23</a>]</cite><span id="footnote5" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">5</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">5</sup><span class="ltx_tag ltx_tag_note">5</span><a target="_blank" href="https://github.com/Stability-AI/lm-evaluation-harness/tree/stablelm-2/multilingual-bench" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/Stability-AI/lm-evaluation-harness/tree/stablelm-2/multilingual-bench</a></span></span></span>.
As shown in Table <a href="#S3.T4" title="Table 4 ‣ 3 Fine-tuning and Alignment ‣ Stable LM 2 1.6B Technical Report" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>, Stable LM 2 1.6B (<span id="S4.SS1.p1.1.7" class="ltx_text ltx_font_italic">stablelm-2-1-6b</span>) outperforms other base models by a significant margin.
Similarly, the instruction-tuned version (<span id="S4.SS1.p1.1.8" class="ltx_text ltx_font_italic">stablelm-2-1-6b-dpo</span>) improves on Microsoft’s Phi-1.5 by two average points while lagging behind the larger Phi-2.0 on few-shot accuracy.
Performance versus Google’s Gemma 2B (2.5B parameters) is also remarkable.</p>
</div>
<figure id="S4.T7" class="ltx_table">
<table id="S4.T7.2" class="ltx_tabular ltx_align_middle">
<tr id="S4.T7.2.1" class="ltx_tr">
<td id="S4.T7.2.1.1" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_tt" style="width:72.3pt;"><span id="S4.T7.2.1.1.1" class="ltx_text ltx_font_bold">Model</span></td>
<td id="S4.T7.2.1.2" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_tt" style="width:72.3pt;">
<span id="S4.T7.2.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T7.2.1.2.1.1" class="ltx_p"><span id="S4.T7.2.1.2.1.1.1" class="ltx_text ltx_font_bold">Size</span></span>
</span>
</td>
<td id="S4.T7.2.1.3" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_tt" style="width:72.3pt;">
<span id="S4.T7.2.1.3.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T7.2.1.3.1.1" class="ltx_p"><span id="S4.T7.2.1.3.1.1.1" class="ltx_text ltx_font_bold">MT-Bench</span></span>
</span>
</td>
</tr>
<tr id="S4.T7.2.2" class="ltx_tr">
<td id="S4.T7.2.2.1" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:72.3pt;"><span id="S4.T7.2.2.1.1" class="ltx_text ltx_font_bold">Mistral-7B-Instruct-v0.2</span></td>
<td id="S4.T7.2.2.2" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:72.3pt;">
<span id="S4.T7.2.2.2.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T7.2.2.2.1.1" class="ltx_p">7B</span>
</span>
</td>
<td id="S4.T7.2.2.3" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:72.3pt;">
<span id="S4.T7.2.2.3.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T7.2.2.3.1.1" class="ltx_p">7.61</span>
</span>
</td>
</tr>
<tr id="S4.T7.2.3" class="ltx_tr">
<td id="S4.T7.2.3.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:72.3pt;"><span id="S4.T7.2.3.1.1" class="ltx_text ltx_font_bold">Llama2-Chat</span></td>
<td id="S4.T7.2.3.2" class="ltx_td ltx_align_justify ltx_align_middle" style="width:72.3pt;">
<span id="S4.T7.2.3.2.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T7.2.3.2.1.1" class="ltx_p">70B</span>
</span>
</td>
<td id="S4.T7.2.3.3" class="ltx_td ltx_align_justify ltx_align_middle" style="width:72.3pt;">
<span id="S4.T7.2.3.3.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T7.2.3.3.1.1" class="ltx_p">6.86</span>
</span>
</td>
</tr>
<tr id="S4.T7.2.4" class="ltx_tr">
<td id="S4.T7.2.4.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:72.3pt;"><span id="S4.T7.2.4.1.1" class="ltx_text ltx_font_bold">stablelm-zephyr-3b</span></td>
<td id="S4.T7.2.4.2" class="ltx_td ltx_align_justify ltx_align_middle" style="width:72.3pt;">
<span id="S4.T7.2.4.2.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T7.2.4.2.1.1" class="ltx_p">3B</span>
</span>
</td>
<td id="S4.T7.2.4.3" class="ltx_td ltx_align_justify ltx_align_middle" style="width:72.3pt;">
<span id="S4.T7.2.4.3.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T7.2.4.3.1.1" class="ltx_p">6.64</span>
</span>
</td>
</tr>
<tr id="S4.T7.2.5" class="ltx_tr">
<td id="S4.T7.2.5.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:72.3pt;"><span id="S4.T7.2.5.1.1" class="ltx_text ltx_font_bold">MPT-30B-Chat</span></td>
<td id="S4.T7.2.5.2" class="ltx_td ltx_align_justify ltx_align_middle" style="width:72.3pt;">
<span id="S4.T7.2.5.2.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T7.2.5.2.1.1" class="ltx_p">30B</span>
</span>
</td>
<td id="S4.T7.2.5.3" class="ltx_td ltx_align_justify ltx_align_middle" style="width:72.3pt;">
<span id="S4.T7.2.5.3.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T7.2.5.3.1.1" class="ltx_p">6.39</span>
</span>
</td>
</tr>
<tr id="S4.T7.2.6" class="ltx_tr" style="background-color:#CCE6FF;">
<td id="S4.T7.2.6.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:72.3pt;"><span id="S4.T7.2.6.1.1" class="ltx_text ltx_font_bold" style="background-color:#CCE6FF;">stablelm-2-zephyr-1.6b</span></td>
<td id="S4.T7.2.6.2" class="ltx_td ltx_align_justify ltx_align_middle" style="width:72.3pt;">
<span id="S4.T7.2.6.2.1" class="ltx_inline-block ltx_align_top" style="background-color:#CCE6FF;">
<span id="S4.T7.2.6.2.1.1" class="ltx_p">1.6B</span>
</span>
</td>
<td id="S4.T7.2.6.3" class="ltx_td ltx_align_justify ltx_align_middle" style="width:72.3pt;">
<span id="S4.T7.2.6.3.1" class="ltx_inline-block ltx_align_top" style="background-color:#CCE6FF;">
<span id="S4.T7.2.6.3.1.1" class="ltx_p">5.42</span>
</span>
</td>
</tr>
<tr id="S4.T7.2.7" class="ltx_tr">
<td id="S4.T7.2.7.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:72.3pt;"><span id="S4.T7.2.7.1.1" class="ltx_text ltx_font_bold">Qwen-1.5-1.8B-Chat</span></td>
<td id="S4.T7.2.7.2" class="ltx_td ltx_align_justify ltx_align_middle" style="width:72.3pt;">
<span id="S4.T7.2.7.2.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T7.2.7.2.1.1" class="ltx_p">1.8B</span>
</span>
</td>
<td id="S4.T7.2.7.3" class="ltx_td ltx_align_justify ltx_align_middle" style="width:72.3pt;">
<span id="S4.T7.2.7.3.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T7.2.7.3.1.1" class="ltx_p">5.29</span>
</span>
</td>
</tr>
<tr id="S4.T7.2.8" class="ltx_tr">
<td id="S4.T7.2.8.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:72.3pt;"><span id="S4.T7.2.8.1.1" class="ltx_text ltx_font_bold">gemma-2b-it</span></td>
<td id="S4.T7.2.8.2" class="ltx_td ltx_align_justify ltx_align_middle" style="width:72.3pt;">
<span id="S4.T7.2.8.2.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T7.2.8.2.1.1" class="ltx_p">2.5B</span>
</span>
</td>
<td id="S4.T7.2.8.3" class="ltx_td ltx_align_justify ltx_align_middle" style="width:72.3pt;">
<span id="S4.T7.2.8.3.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T7.2.8.3.1.1" class="ltx_p">5.19</span>
</span>
</td>
</tr>
<tr id="S4.T7.2.9" class="ltx_tr">
<td id="S4.T7.2.9.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:72.3pt;"><span id="S4.T7.2.9.1.1" class="ltx_text ltx_font_bold">Falcon-40B-Instruct</span></td>
<td id="S4.T7.2.9.2" class="ltx_td ltx_align_justify ltx_align_middle" style="width:72.3pt;">
<span id="S4.T7.2.9.2.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T7.2.9.2.1.1" class="ltx_p">40B</span>
</span>
</td>
<td id="S4.T7.2.9.3" class="ltx_td ltx_align_justify ltx_align_middle" style="width:72.3pt;">
<span id="S4.T7.2.9.3.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T7.2.9.3.1.1" class="ltx_p">5.17</span>
</span>
</td>
</tr>
<tr id="S4.T7.2.10" class="ltx_tr">
<td id="S4.T7.2.10.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:72.3pt;"><span id="S4.T7.2.10.1.1" class="ltx_text ltx_font_bold">dolphin-2.6-phi-2</span></td>
<td id="S4.T7.2.10.2" class="ltx_td ltx_align_justify ltx_align_middle" style="width:72.3pt;">
<span id="S4.T7.2.10.2.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T7.2.10.2.1.1" class="ltx_p">2.7B</span>
</span>
</td>
<td id="S4.T7.2.10.3" class="ltx_td ltx_align_justify ltx_align_middle" style="width:72.3pt;">
<span id="S4.T7.2.10.3.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T7.2.10.3.1.1" class="ltx_p">4.93</span>
</span>
</td>
</tr>
<tr id="S4.T7.2.11" class="ltx_tr">
<td id="S4.T7.2.11.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:72.3pt;"><span id="S4.T7.2.11.1.1" class="ltx_text ltx_font_bold">phi-2</span></td>
<td id="S4.T7.2.11.2" class="ltx_td ltx_align_justify ltx_align_middle" style="width:72.3pt;">
<span id="S4.T7.2.11.2.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T7.2.11.2.1.1" class="ltx_p">2.7B</span>
</span>
</td>
<td id="S4.T7.2.11.3" class="ltx_td ltx_align_justify ltx_align_middle" style="width:72.3pt;">
<span id="S4.T7.2.11.3.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T7.2.11.3.1.1" class="ltx_p">4.29</span>
</span>
</td>
</tr>
<tr id="S4.T7.2.12" class="ltx_tr">
<td id="S4.T7.2.12.1" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_b" style="width:72.3pt;"><span id="S4.T7.2.12.1.1" class="ltx_text ltx_font_bold">TinyLlama-1.1B-Chat-v1.0</span></td>
<td id="S4.T7.2.12.2" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_b" style="width:72.3pt;">
<span id="S4.T7.2.12.2.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T7.2.12.2.1.1" class="ltx_p">1.1B</span>
</span>
</td>
<td id="S4.T7.2.12.3" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_b" style="width:72.3pt;">
<span id="S4.T7.2.12.3.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T7.2.12.3.1.1" class="ltx_p">3.46</span>
</span>
</td>
</tr>
</table>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table"><span id="S4.T7.3.1.1" class="ltx_text" style="font-size:90%;">Table 7</span>: </span><span id="S4.T7.4.2" class="ltx_text" style="font-size:90%;">MT-Bench results</span></figcaption>
</figure>
</section>
<section id="S4.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.2 </span>Multilingual Evaluations</h3>

<div id="S4.SS2.p1" class="ltx_para">
<p id="S4.SS2.p1.1" class="ltx_p">We assess knowledge and reasoning in the multilingual setting for non-English languages seen during pre-training by evaluating on ChatGPT-translated versions of ARC, HS, TQA, and MMLU (<cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib38" title="" class="ltx_ref">38</a>]</cite>). In addition, we test next-word prediction capabilities using the machine-translated LAMBADA datasets from <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib23" title="" class="ltx_ref">23</a>]</cite>.
After manual inspection by native speakers, we have deemed existing machine translations <span id="footnote6" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">6</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">6</sup><span class="ltx_tag ltx_tag_note">6</span><a target="_blank" href="https://huggingface.co/datasets/EleutherAI/lambada_openai" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://huggingface.co/datasets/EleutherAI/lambada_openai</a></span></span></span> too noisy to draw accurate performance signals from. We instead evaluate multilingual next-word prediction with new translations which are made available for researchers <span id="footnote7" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">7</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">7</sup><span class="ltx_tag ltx_tag_note">7</span><a target="_blank" href="https://huggingface.co/datasets/marcob/lambada_multilingual" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://huggingface.co/datasets/marcob/lambada_multilingual</a></span></span></span>.</p>
</div>
<div id="S4.SS2.p2" class="ltx_para">
<p id="S4.SS2.p2.1" class="ltx_p">The zero-shot results are presented in Tab. <a href="#S3.T5" title="Table 5 ‣ 3 Fine-tuning and Alignment ‣ Stable LM 2 1.6B Technical Report" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a> and highlight Stable LM 2’s superior performance compared to models even twice its size.</p>
</div>
</section>
<section id="S4.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.3 </span>MT Benchmark Evaluations</h3>

<div id="S4.SS3.p1" class="ltx_para">
<p id="S4.SS3.p1.1" class="ltx_p">Finally, we also test the conversational skills of our model on the popular multi-turn benchmark <span id="S4.SS3.p1.1.1" class="ltx_text ltx_font_bold">MT-Bench</span> <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib86" title="" class="ltx_ref">86</a>]</cite>. The results are provided in Fig. <a href="#S3.F3" title="Figure 3 ‣ 3.2 DPO ‣ 3 Fine-tuning and Alignment ‣ Stable LM 2 1.6B Technical Report" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a> and Tab. <a href="#S4.T7" title="Table 7 ‣ 4.1 Few-shot and Zero-shot Evaluations ‣ 4 Experimental Results and Benchmarks ‣ Stable LM 2 1.6B Technical Report" class="ltx_ref"><span class="ltx_text ltx_ref_tag">7</span></a>. While lagging behind much more powerful models such as Mistral 7B Instruct v0.2 (more than 4x the size of Stable LM 2), our model delivers better chat performance and beats both Phi-2, Gemma 2B and TinyLLaMA 1.1B by a wide margin despite the larger size of the former.</p>
</div>
</section>
</section>
<section id="S5" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">5 </span>Inference and Quantization</h2>

<div id="S5.p1" class="ltx_para">
<p id="S5.p1.1" class="ltx_p">This model represents a substantial leap towards making advanced generation capabilities available directly on-device without the computational overhead of larger models. We believe this model strikes a great balance between remarkable efficiency and effectiveness in inference tasks when paired with inference frameworks and quantization methods. As part of our release, we provide quantized weights of <span id="S5.p1.1.1" class="ltx_text ltx_font_italic">stablelm-2-1-6b</span> supported on popular inference libraries such as llama.cpp <span id="footnote8" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">8</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">8</sup><span class="ltx_tag ltx_tag_note">8</span><a target="_blank" href="https://github.com/ggerganov/llama.cpp" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/ggerganov/llama.cpp</a></span></span></span>, Apple MLX <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib29" title="" class="ltx_ref">29</a>]</cite> and Intel OpenVINO <span id="footnote9" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">9</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">9</sup><span class="ltx_tag ltx_tag_note">9</span><a target="_blank" href="https://github.com/openvinotoolkit/openvino" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/openvinotoolkit/openvino</a></span></span></span>.</p>
</div>
<section id="S5.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">5.1 </span>Quantization</h3>

<div id="S5.SS1.p1" class="ltx_para">
<p id="S5.SS1.p1.1" class="ltx_p">We make available quantization files for various models and formats to support easier
integration with different inference frameworks, including:</p>
<ul id="S5.I1" class="ltx_itemize">
<li id="S5.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S5.I1.i1.p1" class="ltx_para">
<p id="S5.I1.i1.p1.1" class="ltx_p">Two 4-bit quantized models: Q4_0, Q4_1 and a 5-bit quantized model: Q5_K_M GGUF</p>
</div>
</li>
<li id="S5.I1.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S5.I1.i2.p1" class="ltx_para">
<p id="S5.I1.i2.p1.1" class="ltx_p">INT4 for OpenVINO quantized with Intel’s Neural Network Compression Framework (NNCF)</p>
</div>
</li>
<li id="S5.I1.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S5.I1.i3.p1" class="ltx_para">
<p id="S5.I1.i3.p1.1" class="ltx_p">INT4 for MLX quantized with MLX</p>
</div>
</li>
</ul>
</div>
<div id="S5.SS1.p2" class="ltx_para">
<p id="S5.SS1.p2.1" class="ltx_p">These quantization files can be found in the model’s
<a target="_blank" href="https://huggingface.co/stabilityai/stablelm-2-zephyr-1_6b/tree/main" title="" class="ltx_ref ltx_href">Hugging Face repository</a> for the
convenience of developers and researchers working with our models. We aim to facilitate smoother deployment experiences across various deep-learning framework
ecosystems by offering a range of quantization formats.</p>
</div>
</section>
<section id="S5.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">5.2 </span>Throughput</h3>

<div id="S5.SS2.p1" class="ltx_para">
<p id="S5.SS2.p1.1" class="ltx_p">In Tab. <a href="#S5.T8" title="Table 8 ‣ 5.2 Throughput ‣ 5 Inference and Quantization ‣ Stable LM 2 1.6B Technical Report" class="ltx_ref"><span class="ltx_text ltx_ref_tag">8</span></a> we provide throughput numbers obtained from our model running on consumer-grade devices and the system environments utilized.
Our initial runs showcase that when using a lower precision, we are able to achieve almost 2x performance in throughput.
Note that these figures are provided as a reference, and they are not the result of rigorous benchmarking but are rather intended to give users a practical insight into what they can expect in terms of performance on commonly used devices. Likewise, as lower precision quantization is expected to reduce the model’s performance, we encourage researchers and developers to assess the potential degradation in real-world scenarios.</p>
</div>
<figure id="S5.T8" class="ltx_table">
<table id="S5.T8.2" class="ltx_tabular ltx_align_middle">
<tr id="S5.T8.2.1" class="ltx_tr">
<td id="S5.T8.2.1.1" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_tt" style="width:72.3pt;padding-bottom:0.43057pt;"><span id="S5.T8.2.1.1.1" class="ltx_text ltx_font_bold">Framework</span></td>
<td id="S5.T8.2.1.2" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_tt" style="width:72.3pt;padding-bottom:0.43057pt;">
<span id="S5.T8.2.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="S5.T8.2.1.2.1.1" class="ltx_p"><span id="S5.T8.2.1.2.1.1.1" class="ltx_text ltx_font_bold">CPU</span></span>
</span>
</td>
<td id="S5.T8.2.1.3" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_tt" style="width:36.1pt;padding-bottom:0.43057pt;">
<span id="S5.T8.2.1.3.1" class="ltx_inline-block ltx_align_top">
<span id="S5.T8.2.1.3.1.1" class="ltx_p"><span id="S5.T8.2.1.3.1.1.1" class="ltx_text ltx_font_bold">Precision</span></span>
</span>
</td>
<td id="S5.T8.2.1.4" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_tt" style="width:72.3pt;padding-bottom:0.43057pt;">
<span id="S5.T8.2.1.4.1" class="ltx_inline-block ltx_align_top">
<span id="S5.T8.2.1.4.1.1" class="ltx_p"><span id="S5.T8.2.1.4.1.1.1" class="ltx_text ltx_font_bold">Throughput (Tok/s)</span></span>
</span>
</td>
<td id="S5.T8.2.1.5" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_tt" style="width:79.5pt;padding-bottom:0.43057pt;">
<span id="S5.T8.2.1.5.1" class="ltx_inline-block ltx_align_top">
<span id="S5.T8.2.1.5.1.1" class="ltx_p"><span id="S5.T8.2.1.5.1.1.1" class="ltx_text ltx_font_bold">Power consumption (W)</span></span>
</span>
</td>
</tr>
<tr id="S5.T8.2.2" class="ltx_tr">
<td id="S5.T8.2.2.1" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:72.3pt;padding-bottom:0.43057pt;">MLX</td>
<td id="S5.T8.2.2.2" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:72.3pt;padding-bottom:0.43057pt;">
<span id="S5.T8.2.2.2.1" class="ltx_inline-block ltx_align_top">
<span id="S5.T8.2.2.2.1.1" class="ltx_p">M2</span>
</span>
</td>
<td id="S5.T8.2.2.3" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:36.1pt;padding-bottom:0.43057pt;">
<span id="S5.T8.2.2.3.1" class="ltx_inline-block ltx_align_top">
<span id="S5.T8.2.2.3.1.1" class="ltx_p">FP16</span>
</span>
</td>
<td id="S5.T8.2.2.4" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:72.3pt;padding-bottom:0.43057pt;">
<span id="S5.T8.2.2.4.1" class="ltx_inline-block ltx_align_top">
<span id="S5.T8.2.2.4.1.1" class="ltx_p">71</span>
</span>
</td>
<td id="S5.T8.2.2.5" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:79.5pt;padding-bottom:0.43057pt;">
<span id="S5.T8.2.2.5.1" class="ltx_inline-block ltx_align_top">
<span id="S5.T8.2.2.5.1.1" class="ltx_p">6</span>
</span>
</td>
</tr>
<tr id="S5.T8.2.3" class="ltx_tr">
<td id="S5.T8.2.3.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:72.3pt;padding-bottom:0.43057pt;">Mac Mini</td>
<td id="S5.T8.2.3.2" class="ltx_td ltx_align_justify ltx_align_middle" style="width:72.3pt;padding-bottom:0.43057pt;">
<span id="S5.T8.2.3.2.1" class="ltx_inline-block ltx_align_top">
<span id="S5.T8.2.3.2.1.1" class="ltx_p">(8GB)</span>
</span>
</td>
<td id="S5.T8.2.3.3" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;padding-bottom:0.43057pt;">
<span id="S5.T8.2.3.3.1" class="ltx_inline-block ltx_align_top">
<span id="S5.T8.2.3.3.1.1" class="ltx_p">INT4</span>
</span>
</td>
<td id="S5.T8.2.3.4" class="ltx_td ltx_align_justify ltx_align_middle" style="width:72.3pt;padding-bottom:0.43057pt;">
<span id="S5.T8.2.3.4.1" class="ltx_inline-block ltx_align_top">
<span id="S5.T8.2.3.4.1.1" class="ltx_p">127</span>
</span>
</td>
<td id="S5.T8.2.3.5" class="ltx_td ltx_align_justify ltx_align_middle" style="width:79.5pt;padding-bottom:0.43057pt;">
<span id="S5.T8.2.3.5.1" class="ltx_inline-block ltx_align_top">
<span id="S5.T8.2.3.5.1.1" class="ltx_p">11</span>
</span>
</td>
</tr>
<tr id="S5.T8.2.4" class="ltx_tr">
<td id="S5.T8.2.4.1" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:72.3pt;padding-bottom:0.43057pt;">GGUF</td>
<td id="S5.T8.2.4.2" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:72.3pt;padding-bottom:0.43057pt;">
<span id="S5.T8.2.4.2.1" class="ltx_inline-block ltx_align_top">
<span id="S5.T8.2.4.2.1.1" class="ltx_p ltx_align_center">M2 Pro Max</span>
</span>
</td>
<td id="S5.T8.2.4.3" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:36.1pt;padding-bottom:0.43057pt;">
<span id="S5.T8.2.4.3.1" class="ltx_inline-block ltx_align_top">
<span id="S5.T8.2.4.3.1.1" class="ltx_p">FP16</span>
</span>
</td>
<td id="S5.T8.2.4.4" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:72.3pt;padding-bottom:0.43057pt;">
<span id="S5.T8.2.4.4.1" class="ltx_inline-block ltx_align_top">
<span id="S5.T8.2.4.4.1.1" class="ltx_p">46</span>
</span>
</td>
<td id="S5.T8.2.4.5" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:79.5pt;padding-bottom:0.43057pt;">
<span id="S5.T8.2.4.5.1" class="ltx_inline-block ltx_align_top">
<span id="S5.T8.2.4.5.1.1" class="ltx_p">14</span>
</span>
</td>
</tr>
<tr id="S5.T8.2.5" class="ltx_tr">
<td id="S5.T8.2.5.1" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_b" style="width:72.3pt;padding-bottom:0.43057pt;">2023 MacBook Pro</td>
<td id="S5.T8.2.5.2" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_b" style="width:72.3pt;padding-bottom:0.43057pt;">
<span id="S5.T8.2.5.2.1" class="ltx_inline-block ltx_align_top">
<span id="S5.T8.2.5.2.1.1" class="ltx_p">(16GB)</span>
</span>
</td>
<td id="S5.T8.2.5.3" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_b" style="width:36.1pt;padding-bottom:0.43057pt;">
<span id="S5.T8.2.5.3.1" class="ltx_inline-block ltx_align_top">
<span id="S5.T8.2.5.3.1.1" class="ltx_p">INT4</span>
</span>
</td>
<td id="S5.T8.2.5.4" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_b" style="width:72.3pt;padding-bottom:0.43057pt;">
<span id="S5.T8.2.5.4.1" class="ltx_inline-block ltx_align_top">
<span id="S5.T8.2.5.4.1.1" class="ltx_p">99</span>
</span>
</td>
<td id="S5.T8.2.5.5" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_b" style="width:79.5pt;padding-bottom:0.43057pt;">
<span id="S5.T8.2.5.5.1" class="ltx_inline-block ltx_align_top">
<span id="S5.T8.2.5.5.1.1" class="ltx_p">14</span>
</span>
</td>
</tr>
</table>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table"><span id="S5.T8.3.1.1" class="ltx_text" style="font-size:90%;">Table 8</span>: </span><span id="S5.T8.4.2" class="ltx_text" style="font-size:90%;">Throughput and power usage on various devices using different quantization frameworks. We employ a batch size of 1 for all benchmarks. INT4 numbers for GGUF were collected using Q4_0 quantization. </span></figcaption>
</figure>
</section>
</section>
<section id="S6" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">6 </span>Future Work</h2>

<div id="S6.p1" class="ltx_para">
<p id="S6.p1.1" class="ltx_p">There is a number of research avenues we would like to explore to further improve the model:</p>
<ol id="S6.I1" class="ltx_enumerate">
<li id="S6.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">1.</span> 
<div id="S6.I1.i1.p1" class="ltx_para">
<p id="S6.I1.i1.p1.1" class="ltx_p"><span id="S6.I1.i1.p1.1.1" class="ltx_text ltx_font_bold">Data</span>. In this work, we focused on publicly available data. In particular, most of the data comes from web-crawled content, as is common for most models. This data is known to contain many low-quality documents <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib20" title="" class="ltx_ref">20</a>]</cite> that can potentially harm training.
We believe there is significant potential in smart filtering, re-writing, and synthetic data generation with strong models.</p>
</div>
</li>
<li id="S6.I1.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">2.</span> 
<div id="S6.I1.i2.p1" class="ltx_para">
<p id="S6.I1.i2.p1.1" class="ltx_p"><span id="S6.I1.i2.p1.1.1" class="ltx_text ltx_font_bold">Hallucination Mitigation</span>.
Language models are prone to generating incorrect or misleading information, and small language models are even more prone to doing so. Finding reliable ways to detect hallucinations in these models will unlock new use cases in areas that are sensitive to hallucinations.</p>
</div>
</li>
<li id="S6.I1.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">3.</span> 
<div id="S6.I1.i3.p1" class="ltx_para">
<p id="S6.I1.i3.p1.1" class="ltx_p"><span id="S6.I1.i3.p1.1.1" class="ltx_text ltx_font_bold">Long Contexts and Retrieval</span>.
The ability to retrieve information across long context windows is essential for applications such as chat models or dataset integration. Accordingly, in Appendix <a href="#A3" title="Appendix C Evaluation of Performance Across Different Sized Context Windows ‣ Stable LM 2 1.6B Technical Report" class="ltx_ref"><span class="ltx_text ltx_ref_tag">C</span></a>, we explore the current capabilities and limitations of StableLM2 1.6B on the Needle-in-the-Haystack task. Going forward we plan to further build upon this work as well as to extend our models to context lengths beyond 4k.</p>
</div>
</li>
<li id="S6.I1.i4" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">4.</span> 
<div id="S6.I1.i4.p1" class="ltx_para">
<p id="S6.I1.i4.p1.1" class="ltx_p"><span id="S6.I1.i4.p1.1.1" class="ltx_text ltx_font_bold">Conditional Computation</span>. Small models are often capacity-constrained – that is, with the current training approaches, they lack the capacity to process and exploit all of the training data. Recently, ideas such as Mixture of Experts have been successfully applied to take a dense model and extend it to contain more parameters that are selectively applied to certain inputs (for instance, via sparse upcycling <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib37" title="" class="ltx_ref">37</a>]</cite>).
Importantly, if each token selects only one expert, the overall inference FLOPs do not significantly change.
Applying this to the Stable LM 2 1.6B model is a natural extension we will investigate.</p>
</div>
</li>
</ol>
</div>
</section>
<section id="S7" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">7 </span>Environmental and Societal Impact</h2>

<section id="S7.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">7.1 </span>Carbon Footprint</h3>

<div id="S7.SS1.p1" class="ltx_para">
<p id="S7.SS1.p1.5" class="ltx_p">The training of Stable LM 2 has consumed energy with associated carbon dioxide emissions. In line with <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib67" title="" class="ltx_ref">67</a>]</cite> we report our carbon footprint based on the formula</p>
<table id="A5.EGx1" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="S7.Ex1"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="S7.Ex1.m1.1" class="ltx_Math" alttext="\displaystyle\text{Total Wh}=\text{GPU-h}\times\text{power consumption}\times\text{PUE}" display="inline"><semantics id="S7.Ex1.m1.1a"><mrow id="S7.Ex1.m1.1.1" xref="S7.Ex1.m1.1.1.cmml"><mtext id="S7.Ex1.m1.1.1.2" xref="S7.Ex1.m1.1.1.2a.cmml">Total Wh</mtext><mo id="S7.Ex1.m1.1.1.1" xref="S7.Ex1.m1.1.1.1.cmml">=</mo><mrow id="S7.Ex1.m1.1.1.3" xref="S7.Ex1.m1.1.1.3.cmml"><mtext id="S7.Ex1.m1.1.1.3.2" xref="S7.Ex1.m1.1.1.3.2a.cmml">GPU-h</mtext><mo lspace="0.222em" rspace="0.222em" id="S7.Ex1.m1.1.1.3.1" xref="S7.Ex1.m1.1.1.3.1.cmml">×</mo><mtext id="S7.Ex1.m1.1.1.3.3" xref="S7.Ex1.m1.1.1.3.3a.cmml">power consumption</mtext><mo lspace="0.222em" rspace="0.222em" id="S7.Ex1.m1.1.1.3.1a" xref="S7.Ex1.m1.1.1.3.1.cmml">×</mo><mtext id="S7.Ex1.m1.1.1.3.4" xref="S7.Ex1.m1.1.1.3.4a.cmml">PUE</mtext></mrow></mrow><annotation-xml encoding="MathML-Content" id="S7.Ex1.m1.1b"><apply id="S7.Ex1.m1.1.1.cmml" xref="S7.Ex1.m1.1.1"><eq id="S7.Ex1.m1.1.1.1.cmml" xref="S7.Ex1.m1.1.1.1"></eq><ci id="S7.Ex1.m1.1.1.2a.cmml" xref="S7.Ex1.m1.1.1.2"><mtext id="S7.Ex1.m1.1.1.2.cmml" xref="S7.Ex1.m1.1.1.2">Total Wh</mtext></ci><apply id="S7.Ex1.m1.1.1.3.cmml" xref="S7.Ex1.m1.1.1.3"><times id="S7.Ex1.m1.1.1.3.1.cmml" xref="S7.Ex1.m1.1.1.3.1"></times><ci id="S7.Ex1.m1.1.1.3.2a.cmml" xref="S7.Ex1.m1.1.1.3.2"><mtext id="S7.Ex1.m1.1.1.3.2.cmml" xref="S7.Ex1.m1.1.1.3.2">GPU-h</mtext></ci><ci id="S7.Ex1.m1.1.1.3.3a.cmml" xref="S7.Ex1.m1.1.1.3.3"><mtext id="S7.Ex1.m1.1.1.3.3.cmml" xref="S7.Ex1.m1.1.1.3.3">power consumption</mtext></ci><ci id="S7.Ex1.m1.1.1.3.4a.cmml" xref="S7.Ex1.m1.1.1.3.4"><mtext id="S7.Ex1.m1.1.1.3.4.cmml" xref="S7.Ex1.m1.1.1.3.4">PUE</mtext></ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S7.Ex1.m1.1c">\displaystyle\text{Total Wh}=\text{GPU-h}\times\text{power consumption}\times\text{PUE}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
<p id="S7.SS1.p1.4" class="ltx_p">where the Power Usage Effectiveness is set to 1.1. We trained Stable LM 2 for <math id="S7.SS1.p1.1.m1.2" class="ltx_Math" alttext="\approx 92,000" display="inline"><semantics id="S7.SS1.p1.1.m1.2a"><mrow id="S7.SS1.p1.1.m1.2.3" xref="S7.SS1.p1.1.m1.2.3.cmml"><mi id="S7.SS1.p1.1.m1.2.3.2" xref="S7.SS1.p1.1.m1.2.3.2.cmml"></mi><mo id="S7.SS1.p1.1.m1.2.3.1" xref="S7.SS1.p1.1.m1.2.3.1.cmml">≈</mo><mrow id="S7.SS1.p1.1.m1.2.3.3.2" xref="S7.SS1.p1.1.m1.2.3.3.1.cmml"><mn id="S7.SS1.p1.1.m1.1.1" xref="S7.SS1.p1.1.m1.1.1.cmml">92</mn><mo id="S7.SS1.p1.1.m1.2.3.3.2.1" xref="S7.SS1.p1.1.m1.2.3.3.1.cmml">,</mo><mn id="S7.SS1.p1.1.m1.2.2" xref="S7.SS1.p1.1.m1.2.2.cmml">000</mn></mrow></mrow><annotation-xml encoding="MathML-Content" id="S7.SS1.p1.1.m1.2b"><apply id="S7.SS1.p1.1.m1.2.3.cmml" xref="S7.SS1.p1.1.m1.2.3"><approx id="S7.SS1.p1.1.m1.2.3.1.cmml" xref="S7.SS1.p1.1.m1.2.3.1"></approx><csymbol cd="latexml" id="S7.SS1.p1.1.m1.2.3.2.cmml" xref="S7.SS1.p1.1.m1.2.3.2">absent</csymbol><list id="S7.SS1.p1.1.m1.2.3.3.1.cmml" xref="S7.SS1.p1.1.m1.2.3.3.2"><cn type="integer" id="S7.SS1.p1.1.m1.1.1.cmml" xref="S7.SS1.p1.1.m1.1.1">92</cn><cn type="integer" id="S7.SS1.p1.1.m1.2.2.cmml" xref="S7.SS1.p1.1.m1.2.2">000</cn></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S7.SS1.p1.1.m1.2c">\approx 92,000</annotation></semantics></math> GPU-hours, giving a total power consumption of 30MWh considering our average power usage. The tons of emitted carbon tCO<sub id="S7.SS1.p1.4.1" class="ltx_sub">2</sub>eq can be estimated using the US national average carbon intensity factor of 0.385 kg CO<sub id="S7.SS1.p1.4.2" class="ltx_sub">2</sub>eq/KWh, leading to a final figure of 11 tCO<sub id="S7.SS1.p1.4.3" class="ltx_sub">2</sub>eq.</p>
</div>
</section>
<section id="S7.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">7.2 </span>Societal impact</h3>

<div id="S7.SS2.p1" class="ltx_para">
<p id="S7.SS2.p1.1" class="ltx_p">Stability AI is committed to releasing open models to help improve access to foundational AI technology. Open access to model weights enables researchers to inspect models for suitability and vulnerabilities, test the effectiveness of different optimization strategies, and correct for biases observed in the model. To that end, this model is released under an open non-commercial license. However, open release can introduce challenges in assessing the societal impact of a model. For example, Stability AI does not have direct visibility into downstream applications of Stable LM 2 1.6B, the distribution of applications by sector, or the distribution of model usage by geography. Since the model is released with a noncommercial license, we expect a limited number of applications outside of fine-tuning or evaluation interfaces and a limited number of third parties affected by the model.</p>
</div>
<div id="S7.SS2.p2" class="ltx_para">
<p id="S7.SS2.p2.1" class="ltx_p">We will continue to monitor openly released fine-tuned models to understand the extent of fine-tuning research or development activity that uses Stable LM 2 1.6B as a base model, including the evaluation results of these derivative models.
</p>
</div>
</section>
</section>
<section id="S8" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">8 </span>Conclusion</h2>

<div id="S8.p1" class="ltx_para">
<p id="S8.p1.1" class="ltx_p">In this report, we introduced Stable LM 2 1.6B, a compact decoder-only language model trained on multilingual datasets. It fluidly handles up to seven languages: English, Spanish, German, Italian, French, Portuguese, and Dutch.
To ensure that the community can reproduce our run, we detail all datasets used during training –with the <em id="S8.p1.1.1" class="ltx_emph ltx_font_italic">exact</em> data mix– and our newly designed learning rate schedule.
We also conduct extensive model evaluations and comparisons with other similarly-sized models, demonstrating Stable LM 2 1.6B’s exceptional performance.
Finally, we profile the model on common edge computing architectures.
We hope the current report contributes to the improvement and further research on small language models.</p>
</div>
</section>
<section id="Sx1" class="ltx_section">
<h2 class="ltx_title ltx_title_section">Acknowledgments</h2>

<div id="Sx1.p1" class="ltx_para">
<p id="Sx1.p1.1" class="ltx_p">We thank our awesome MLOps team members, particularly Richard Vencu, for the support provided.
We also thank Christian Laforte, Cedric Wagrez, and Jerry Chi for their feedback, useful ideas, and comments.</p>
</div>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
<section id="bib" class="ltx_bibliography">
<h2 class="ltx_title ltx_title_bibliography" style="font-size:90%;">References</h2>

<ul class="ltx_biblist">
<li id="bib.bib1" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[1]</span>
<span class="ltx_bibblock"><span id="bib.bib1.1.1" class="ltx_text" style="font-size:90%;">
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib1.2.1" class="ltx_text" style="font-size:90%;">Program synthesis with large language models.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib1.3.1" class="ltx_text ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2108.07732</span><span id="bib.bib1.4.2" class="ltx_text" style="font-size:90%;">, 2021.
</span>
</span>
</li>
<li id="bib.bib2" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[2]</span>
<span class="ltx_bibblock"><span id="bib.bib2.1.1" class="ltx_text" style="font-size:90%;">
b-mc2.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib2.2.1" class="ltx_text" style="font-size:90%;">sql-create-context dataset, 2023.
</span>
</span>
</li>
<li id="bib.bib3" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[3]</span>
<span class="ltx_bibblock"><span id="bib.bib3.1.1" class="ltx_text" style="font-size:90%;">
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib3.2.1" class="ltx_text" style="font-size:90%;">Layer normalization, 2016.
</span>
</span>
</li>
<li id="bib.bib4" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[4]</span>
<span class="ltx_bibblock"><span id="bib.bib4.1.1" class="ltx_text" style="font-size:90%;">
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib4.2.1" class="ltx_text" style="font-size:90%;">Qwen technical report, 2023.
</span>
</span>
</li>
<li id="bib.bib5" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[5]</span>
<span class="ltx_bibblock"><span id="bib.bib5.1.1" class="ltx_text" style="font-size:90%;">
Edward Beeching, Clémentine Fourrier, Nathan Habib, Sheon Han, Nathan Lambert, Nazneen Rajani, Omar Sanseviero, Lewis Tunstall, and Thomas Wolf.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib5.2.1" class="ltx_text" style="font-size:90%;">Open llm leaderboard.
</span>
</span>
<span class="ltx_bibblock"><a target="_blank" href="https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard" title="" class="ltx_ref ltx_url ltx_font_typewriter" style="font-size:90%;">https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard</a><span id="bib.bib5.3.1" class="ltx_text" style="font-size:90%;">, 2023.
</span>
</span>
</li>
<li id="bib.bib6" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[6]</span>
<span class="ltx_bibblock"><span id="bib.bib6.1.1" class="ltx_text" style="font-size:90%;">
Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, USVSN Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib6.2.1" class="ltx_text" style="font-size:90%;">Gpt-neox-20b: An open-source autoregressive language model, 2022.
</span>
</span>
</li>
<li id="bib.bib7" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[7]</span>
<span class="ltx_bibblock"><span id="bib.bib7.1.1" class="ltx_text" style="font-size:90%;">
British Library, Victoria Morris, Daniel van Strien, Giorgia Tolfo, Lora Afric, Stewart Robertson, Patricia Tiney, Annelies Dogterom, and Ildi Wollner.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib7.2.1" class="ltx_text" style="font-size:90%;">19th century books - metadata with additional crowdsourced annotations, 2021.
</span>
</span>
</li>
<li id="bib.bib8" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[8]</span>
<span class="ltx_bibblock"><span id="bib.bib8.1.1" class="ltx_text" style="font-size:90%;">
Isabel Cachola, Kyle Lo, Arman Cohan, and Daniel S. Weld.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib8.2.1" class="ltx_text" style="font-size:90%;">TLDR: Extreme summarization of scientific documents.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib8.3.1" class="ltx_text ltx_font_italic" style="font-size:90%;">arXiv:2004.15011</span><span id="bib.bib8.4.2" class="ltx_text" style="font-size:90%;">, 2020.
</span>
</span>
</li>
<li id="bib.bib9" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[9]</span>
<span class="ltx_bibblock"><span id="bib.bib9.1.1" class="ltx_text" style="font-size:90%;">
Iñigo Casanueva, Tadas Temcinas, Daniela Gerz, Matthew Henderson, and Ivan Vulic.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib9.2.1" class="ltx_text" style="font-size:90%;">Efficient intent detection with dual sentence encoders.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib9.3.1" class="ltx_text" style="font-size:90%;">In </span><span id="bib.bib9.4.2" class="ltx_text ltx_font_italic" style="font-size:90%;">Proceedings of the 2nd Workshop on NLP for ConvAI - ACL 2020</span><span id="bib.bib9.5.3" class="ltx_text" style="font-size:90%;">, mar 2020.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib9.6.1" class="ltx_text" style="font-size:90%;">Data available at https://github.com/PolyAI-LDN/task-specific-datasets.
</span>
</span>
</li>
<li id="bib.bib10" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[10]</span>
<span class="ltx_bibblock"><span id="bib.bib10.1.1" class="ltx_text" style="font-size:90%;">
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib10.2.1" class="ltx_text" style="font-size:90%;">Palm: Scaling language modeling with pathways, 2022.
</span>
</span>
</li>
<li id="bib.bib11" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[11]</span>
<span class="ltx_bibblock"><span id="bib.bib11.1.1" class="ltx_text" style="font-size:90%;">
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib11.2.1" class="ltx_text" style="font-size:90%;">Think you have solved question answering? try arc, the ai2 reasoning challenge, 2018.
</span>
</span>
</li>
<li id="bib.bib12" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[12]</span>
<span class="ltx_bibblock"><span id="bib.bib12.1.1" class="ltx_text" style="font-size:90%;">
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib12.2.1" class="ltx_text" style="font-size:90%;">Training verifiers to solve math word problems, 2021.
</span>
</span>
</li>
<li id="bib.bib13" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[13]</span>
<span class="ltx_bibblock"><span id="bib.bib13.1.1" class="ltx_text" style="font-size:90%;">
Together Computer.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib13.2.1" class="ltx_text" style="font-size:90%;">Redpajama: An open source recipe to reproduce llama training dataset, 2023.
</span>
</span>
</li>
<li id="bib.bib14" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[14]</span>
<span class="ltx_bibblock"><span id="bib.bib14.1.1" class="ltx_text" style="font-size:90%;">
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib14.2.1" class="ltx_text" style="font-size:90%;">Ultrafeedback: Boosting language models with high-quality feedback, 2023.
</span>
</span>
</li>
<li id="bib.bib15" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[15]</span>
<span class="ltx_bibblock"><span id="bib.bib15.1.1" class="ltx_text" style="font-size:90%;">
Luigi Daniele and Suphavadeeprasit.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib15.2.1" class="ltx_text" style="font-size:90%;">Amplify-instruct: Synthetically generated diverse multi-turn conversations for effecient llm training.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib15.3.1" class="ltx_text ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:(coming soon)</span><span id="bib.bib15.4.2" class="ltx_text" style="font-size:90%;">, 2023.
</span>
</span>
</li>
<li id="bib.bib16" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[16]</span>
<span class="ltx_bibblock"><span id="bib.bib16.1.1" class="ltx_text" style="font-size:90%;">
Tri Dao.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib16.2.1" class="ltx_text" style="font-size:90%;">FlashAttention-2: Faster attention with better parallelism and work partitioning, 2023.
</span>
</span>
</li>
<li id="bib.bib17" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[17]</span>
<span class="ltx_bibblock"><span id="bib.bib17.1.1" class="ltx_text" style="font-size:90%;">
Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib17.2.1" class="ltx_text" style="font-size:90%;">FlashAttention: Fast and memory-efficient exact attention with IO-awareness.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib17.3.1" class="ltx_text" style="font-size:90%;">In </span><span id="bib.bib17.4.2" class="ltx_text ltx_font_italic" style="font-size:90%;">Advances in Neural Information Processing Systems</span><span id="bib.bib17.5.3" class="ltx_text" style="font-size:90%;">, 2022.
</span>
</span>
</li>
<li id="bib.bib18" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[18]</span>
<span class="ltx_bibblock"><span id="bib.bib18.1.1" class="ltx_text" style="font-size:90%;">
Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib18.2.1" class="ltx_text" style="font-size:90%;">Enhancing chat language models by scaling high-quality instructional conversations, 2023.
</span>
</span>
</li>
<li id="bib.bib19" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[19]</span>
<span class="ltx_bibblock"><span id="bib.bib19.1.1" class="ltx_text" style="font-size:90%;">
Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten Bosma, Zongwei Zhou, Tao Wang, Yu Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, Kathleen Meier-Hellstern, Toju Duke, Lucas Dixon, Kun Zhang, Quoc V Le, Yonghui Wu, Zhifeng Chen, and Claire Cui.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib19.2.1" class="ltx_text" style="font-size:90%;">Glam: Efficient scaling of language models with mixture-of-experts, 2022.
</span>
</span>
</li>
<li id="bib.bib20" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[20]</span>
<span class="ltx_bibblock"><span id="bib.bib20.1.1" class="ltx_text" style="font-size:90%;">
Yanai Elazar, Akshita Bhagia, Ian Magnusson, Abhilasha Ravichander, Dustin Schwenk, Alane Suhr, Pete Walsh, Dirk Groeneveld, Luca Soldaini, Sameer Singh, Hanna Hajishirzi, Noah A. Smith, and Jesse Dodge.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib20.2.1" class="ltx_text" style="font-size:90%;">What’s in my big data?, 2023.
</span>
</span>
</li>
<li id="bib.bib21" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[21]</span>
<span class="ltx_bibblock"><span id="bib.bib21.1.1" class="ltx_text" style="font-size:90%;">
Yao Fu, Rameswar Panda, Xinyao Niu, Xiang Yue, Hannaneh Hajishirzi, Yoon Kim, and Hao Peng.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib21.2.1" class="ltx_text" style="font-size:90%;">Data engineering for scaling language models to 128k context, 2024.
</span>
</span>
</li>
<li id="bib.bib22" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[22]</span>
<span class="ltx_bibblock"><span id="bib.bib22.1.1" class="ltx_text" style="font-size:90%;">
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib22.2.1" class="ltx_text" style="font-size:90%;">The pile: An 800gb dataset of diverse text for language modeling, 2020.
</span>
</span>
</li>
<li id="bib.bib23" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[23]</span>
<span class="ltx_bibblock"><span id="bib.bib23.1.1" class="ltx_text" style="font-size:90%;">
Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib23.2.1" class="ltx_text" style="font-size:90%;">A framework for few-shot language model evaluation, September 2021.
</span>
</span>
</li>
<li id="bib.bib24" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[24]</span>
<span class="ltx_bibblock"><span id="bib.bib24.1.1" class="ltx_text" style="font-size:90%;">
Aaron Gokaslan, Vanya Cohen, Ellie Pavlick, and Stefanie Tellex.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib24.2.1" class="ltx_text" style="font-size:90%;">Openwebtext corpus, 2019.
</span>
</span>
</li>
<li id="bib.bib25" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[25]</span>
<span class="ltx_bibblock"><span id="bib.bib25.1.1" class="ltx_text" style="font-size:90%;">
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib25.2.1" class="ltx_text" style="font-size:90%;">Accurate, large minibatch sgd: Training imagenet in 1 hour, 2018.
</span>
</span>
</li>
<li id="bib.bib26" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[26]</span>
<span class="ltx_bibblock"><span id="bib.bib26.1.1" class="ltx_text" style="font-size:90%;">
Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, et al.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib26.2.1" class="ltx_text" style="font-size:90%;">Textbooks are all you need.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib26.3.1" class="ltx_text ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2306.11644</span><span id="bib.bib26.4.2" class="ltx_text" style="font-size:90%;">, 2023.
</span>
</span>
</li>
<li id="bib.bib27" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[27]</span>
<span class="ltx_bibblock"><span id="bib.bib27.1.1" class="ltx_text" style="font-size:90%;">
Himanshu Gupta, Neeraj Varshney, Swaroop Mishra, Kuntal Kumar Pal, Saurabh Arjun Sawant, Kevin Scaria, Siddharth Goyal, and Chitta Baral.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib27.2.1" class="ltx_text" style="font-size:90%;">“john is 50 years old, can his son be 65?” evaluating NLP models’ understanding of feasibility.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib27.3.1" class="ltx_text" style="font-size:90%;">In Andreas Vlachos and Isabelle Augenstein, editors, </span><span id="bib.bib27.4.2" class="ltx_text ltx_font_italic" style="font-size:90%;">Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics</span><span id="bib.bib27.5.3" class="ltx_text" style="font-size:90%;">, pages 407–417, Dubrovnik, Croatia, May 2023. Association for Computational Linguistics.
</span>
</span>
</li>
<li id="bib.bib28" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[28]</span>
<span class="ltx_bibblock"><span id="bib.bib28.1.1" class="ltx_text" style="font-size:90%;">
Johannes Hagemann, Samuel Weinbach, Konstantin Dobler, Maximilian Schall, and Gerard de Melo.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib28.2.1" class="ltx_text" style="font-size:90%;">Efficient parallelization layouts for large-scale distributed model training, 2023.
</span>
</span>
</li>
<li id="bib.bib29" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[29]</span>
<span class="ltx_bibblock"><span id="bib.bib29.1.1" class="ltx_text" style="font-size:90%;">
Awni Hannun, Jagrit Digani, Angelos Katharopoulos, and Ronan Collobert.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib29.2.1" class="ltx_text" style="font-size:90%;">MLX: Efficient and flexible machine learning on apple silicon, 2023.
</span>
</span>
</li>
<li id="bib.bib30" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[30]</span>
<span class="ltx_bibblock"><span id="bib.bib30.1.1" class="ltx_text" style="font-size:90%;">
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib30.2.1" class="ltx_text" style="font-size:90%;">Measuring massive multitask language understanding, 2021.
</span>
</span>
</li>
<li id="bib.bib31" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[31]</span>
<span class="ltx_bibblock"><span id="bib.bib31.1.1" class="ltx_text" style="font-size:90%;">
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib31.2.1" class="ltx_text" style="font-size:90%;">Training compute-optimal large language models, 2022.
</span>
</span>
</li>
<li id="bib.bib32" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[32]</span>
<span class="ltx_bibblock"><span id="bib.bib32.1.1" class="ltx_text" style="font-size:90%;">
Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A. Smith, Iz Beltagy, and Hannaneh Hajishirzi.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib32.2.1" class="ltx_text" style="font-size:90%;">Camels in a changing climate: Enhancing lm adaptation with tulu 2, 2023.
</span>
</span>
</li>
<li id="bib.bib33" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[33]</span>
<span class="ltx_bibblock"><span id="bib.bib33.1.1" class="ltx_text" style="font-size:90%;">
Mingi Jeon, Seung-Yeop Baik, Joonghyuk Hahn, Yo-Sub Han, and Sang-Ki Ko.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib33.2.1" class="ltx_text" style="font-size:90%;">Deep Learning-based Code Complexity Prediction, 2022.
</span>
</span>
</li>
<li id="bib.bib34" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[34]</span>
<span class="ltx_bibblock"><span id="bib.bib34.1.1" class="ltx_text" style="font-size:90%;">
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib34.2.1" class="ltx_text" style="font-size:90%;">triviaqa: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib34.3.1" class="ltx_text ltx_font_italic" style="font-size:90%;">arXiv e-prints</span><span id="bib.bib34.4.2" class="ltx_text" style="font-size:90%;">, page arXiv:1705.03551, 2017.
</span>
</span>
</li>
<li id="bib.bib35" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[35]</span>
<span class="ltx_bibblock"><span id="bib.bib35.1.1" class="ltx_text" style="font-size:90%;">
Greg Kamradt and Ikko Ashimine.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib35.2.1" class="ltx_text" style="font-size:90%;">Needle in a haystack - pressure testing llms, 2023.
</span>
</span>
</li>
<li id="bib.bib36" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[36]</span>
<span class="ltx_bibblock"><span id="bib.bib36.1.1" class="ltx_text" style="font-size:90%;">
Tomáš Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib36.2.1" class="ltx_text" style="font-size:90%;">The NarrativeQA reading comprehension challenge.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib36.3.1" class="ltx_text ltx_font_italic" style="font-size:90%;">Transactions of the Association for Computational Linguistics</span><span id="bib.bib36.4.2" class="ltx_text" style="font-size:90%;">, 6:317–328, 2018.
</span>
</span>
</li>
<li id="bib.bib37" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[37]</span>
<span class="ltx_bibblock"><span id="bib.bib37.1.1" class="ltx_text" style="font-size:90%;">
Aran Komatsuzaki, Joan Puigcerver, James Lee-Thorp, Carlos Riquelme Ruiz, Basil Mustafa, Joshua Ainslie, Yi Tay, Mostafa Dehghani, and Neil Houlsby.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib37.2.1" class="ltx_text" style="font-size:90%;">Sparse upcycling: Training mixture-of-experts from dense checkpoints.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib37.3.1" class="ltx_text ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2212.05055</span><span id="bib.bib37.4.2" class="ltx_text" style="font-size:90%;">, 2022.
</span>
</span>
</li>
<li id="bib.bib38" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[38]</span>
<span class="ltx_bibblock"><span id="bib.bib38.1.1" class="ltx_text" style="font-size:90%;">
Viet Dac Lai, Chien Van Nguyen, Nghia Trung Ngo, Thuat Nguyen, Franck Dernoncourt, Ryan A Rossi, and Thien Huu Nguyen.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib38.2.1" class="ltx_text" style="font-size:90%;">Okapi: Instruction-tuned large language models in multiple languages with reinforcement learning from human feedback.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib38.3.1" class="ltx_text ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2307.16039</span><span id="bib.bib38.4.2" class="ltx_text" style="font-size:90%;">, 2023.
</span>
</span>
</li>
<li id="bib.bib39" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[39]</span>
<span class="ltx_bibblock"><span id="bib.bib39.1.1" class="ltx_text" style="font-size:90%;">
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj, Joel Lamy-Poirier, João Monteiro, Oleh Shliazhko, Nicolas Gontier, Nicholas Meade, Armel Zebaze, Ming-Ho Yee, Logesh Kumar Umapathi, Jian Zhu, Benjamin Lipkin, Muhtasham Oblokulov, Zhiruo Wang, Rudra Murthy, Jason Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang, Nour Fahmy, Urvashi Bhattacharyya, Wenhao Yu, Swayam Singh, Sasha Luccioni, Paulo Villegas, Maxim Kunakov, Fedor Zhdanov, Manuel Romero, Tony Lee, Nadav Timor, Jennifer Ding, Claire Schlesinger, Hailey Schoelkopf, Jan Ebert, Tri Dao, Mayank Mishra, Alex Gu, Jennifer Robinson, Carolyn Jane Anderson, Brendan Dolan-Gavitt, Danish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von
Werra, and Harm de Vries.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib39.2.1" class="ltx_text" style="font-size:90%;">Starcoder: may the source be with you!, 2023.
</span>
</span>
</li>
<li id="bib.bib40" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[40]</span>
<span class="ltx_bibblock"><span id="bib.bib40.1.1" class="ltx_text" style="font-size:90%;">
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj, Joel Lamy-Poirier, João Monteiro, Oleh Shliazhko, Nicolas Gontier, Nicholas Meade, Armel Zebaze, Ming-Ho Yee, Logesh Kumar Umapathi, Jian Zhu, Benjamin Lipkin, Muhtasham Oblokulov, Zhiruo Wang, Rudra Murthy, Jason Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang, Nour Fahmy, Urvashi Bhattacharyya, Wenhao Yu, Swayam Singh, Sasha Luccioni, Paulo Villegas, Maxim Kunakov, Fedor Zhdanov, Manuel Romero, Tony Lee, Nadav Timor, Jennifer Ding, Claire Schlesinger, Hailey Schoelkopf, Jan Ebert, Tri Dao, Mayank Mishra, Alex Gu, Jennifer Robinson, Carolyn Jane Anderson, Brendan Dolan-Gavitt, Danish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von
Werra, and Harm de Vries.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib40.2.1" class="ltx_text" style="font-size:90%;">StarCoder: may the source be with you!, 2023.
</span>
</span>
</li>
<li id="bib.bib41" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[41]</span>
<span class="ltx_bibblock"><span id="bib.bib41.1.1" class="ltx_text" style="font-size:90%;">
Wing Lian, Guan Wang, Bleys Goodson, Eugene Pentland, Austin Cook, Chanvichet Vong, and "Teknium".
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib41.2.1" class="ltx_text" style="font-size:90%;">Slimorca: An open dataset of gpt-4 augmented flan reasoning traces, with verification, 2023.
</span>
</span>
</li>
<li id="bib.bib42" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[42]</span>
<span class="ltx_bibblock"><span id="bib.bib42.1.1" class="ltx_text" style="font-size:90%;">
Bill Yuchen Lin, Wangchunshu Zhou, Ming Shen, Pei Zhou, Chandra Bhagavatula, Yejin Choi, and Xiang Ren.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib42.2.1" class="ltx_text" style="font-size:90%;">CommonGen: A constrained text generation challenge for generative commonsense reasoning.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib42.3.1" class="ltx_text" style="font-size:90%;">In </span><span id="bib.bib42.4.2" class="ltx_text ltx_font_italic" style="font-size:90%;">Findings of the Association for Computational Linguistics: EMNLP 2020</span><span id="bib.bib42.5.3" class="ltx_text" style="font-size:90%;">, pages 1823–1840, Online, November 2020. Association for Computational Linguistics.
</span>
</span>
</li>
<li id="bib.bib43" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[43]</span>
<span class="ltx_bibblock"><span id="bib.bib43.1.1" class="ltx_text" style="font-size:90%;">
Stephanie Lin, Jacob Hilton, and Owain Evans.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib43.2.1" class="ltx_text" style="font-size:90%;">Truthfulqa: Measuring how models mimic human falsehoods, 2022.
</span>
</span>
</li>
<li id="bib.bib44" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[44]</span>
<span class="ltx_bibblock"><span id="bib.bib44.1.1" class="ltx_text" style="font-size:90%;">
Emmy Liu, Chen Cui, Kenneth Zheng, and Graham Neubig.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib44.2.1" class="ltx_text" style="font-size:90%;">Testing the ability of language models to interpret figurative language, 2022.
</span>
</span>
</li>
<li id="bib.bib45" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[45]</span>
<span class="ltx_bibblock"><span id="bib.bib45.1.1" class="ltx_text" style="font-size:90%;">
Tiedong Liu and Bryan Kian Hsiang Low.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib45.2.1" class="ltx_text" style="font-size:90%;">Goat: Fine-tuned llama outperforms gpt-4 on arithmetic tasks, 2023.
</span>
</span>
</li>
<li id="bib.bib46" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[46]</span>
<span class="ltx_bibblock"><span id="bib.bib46.1.1" class="ltx_text" style="font-size:90%;">
Wei Liu, Weihao Zeng, Keqing He, Yong Jiang, and Junxian He.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib46.2.1" class="ltx_text" style="font-size:90%;">What makes good data for alignment? a comprehensive study of automatic data selection in instruction tuning, 2023.
</span>
</span>
</li>
<li id="bib.bib47" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[47]</span>
<span class="ltx_bibblock"><span id="bib.bib47.1.1" class="ltx_text" style="font-size:90%;">
Shayne Longpre, Robert Mahari, Anthony Chen, Naana Obeng-Marnu, Damien Sileo, William Brannon, Niklas Muennighoff, Nathan Khazam, Jad Kabbara, Kartik Perisetla, et al.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib47.2.1" class="ltx_text" style="font-size:90%;">The data provenance initiative: A large scale audit of dataset licensing &amp; attribution in ai.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib47.3.1" class="ltx_text ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2310.16787</span><span id="bib.bib47.4.2" class="ltx_text" style="font-size:90%;">, 2023.
</span>
</span>
</li>
<li id="bib.bib48" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[48]</span>
<span class="ltx_bibblock"><span id="bib.bib48.1.1" class="ltx_text" style="font-size:90%;">
Shayne Longpre, Gregory Yauney, Emily Reif, Katherine Lee, Adam Roberts, Barret Zoph, Denny Zhou, Jason Wei, Kevin Robinson, David Mimno, and Daphne Ippolito.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib48.2.1" class="ltx_text" style="font-size:90%;">A pretrainer’s guide to training data: Measuring the effects of data age, domain coverage, quality, &amp; toxicity, 2023.
</span>
</span>
</li>
<li id="bib.bib49" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[49]</span>
<span class="ltx_bibblock"><span id="bib.bib49.1.1" class="ltx_text" style="font-size:90%;">
Risto Luukkonen, Ville Komulainen, Jouni Luoma, Anni Eskelinen, Jenna Kanerva, Hanna-Mari Kupari, Filip Ginter, Veronika Laippala, Niklas Muennighoff, Aleksandra Piktus, Thomas Wang, Nouamane Tazi, Teven Le Scao, Thomas Wolf, Osma Suominen, Samuli Sairanen, Mikko Merioksa, Jyrki Heinonen, Aija Vahtola, Samuel Antao, and Sampo Pyysalo.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib49.2.1" class="ltx_text" style="font-size:90%;">Fingpt: Large generative models for a small language, 2023.
</span>
</span>
</li>
<li id="bib.bib50" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[50]</span>
<span class="ltx_bibblock"><span id="bib.bib50.1.1" class="ltx_text" style="font-size:90%;">
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib50.2.1" class="ltx_text" style="font-size:90%;">Learning word vectors for sentiment analysis.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib50.3.1" class="ltx_text" style="font-size:90%;">In </span><span id="bib.bib50.4.2" class="ltx_text ltx_font_italic" style="font-size:90%;">Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies</span><span id="bib.bib50.5.3" class="ltx_text" style="font-size:90%;">, pages 142–150, Portland, Oregon, USA, June 2011. Association for Computational Linguistics.
</span>
</span>
</li>
<li id="bib.bib51" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[51]</span>
<span class="ltx_bibblock"><span id="bib.bib51.1.1" class="ltx_text" style="font-size:90%;">
Sam McCandlish, Jared Kaplan, Dario Amodei, and OpenAI Dota Team.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib51.2.1" class="ltx_text" style="font-size:90%;">An empirical model of large-batch training, 2018.
</span>
</span>
</li>
<li id="bib.bib52" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[52]</span>
<span class="ltx_bibblock"><span id="bib.bib52.1.1" class="ltx_text" style="font-size:90%;">
John P. McCrae, Alexandre Rademaker, Francis Bond, Ewa Rudnicka, and Christiane Fellbaum.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib52.2.1" class="ltx_text" style="font-size:90%;">English WordNet 2019 – an open-source WordNet for English.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib52.3.1" class="ltx_text" style="font-size:90%;">In Piek Vossen and Christiane Fellbaum, editors, </span><span id="bib.bib52.4.2" class="ltx_text ltx_font_italic" style="font-size:90%;">Proceedings of the 10th Global Wordnet Conference</span><span id="bib.bib52.5.3" class="ltx_text" style="font-size:90%;">, pages 245–252, Wroclaw, Poland, July 2019. Global Wordnet Association.
</span>
</span>
</li>
<li id="bib.bib53" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[53]</span>
<span class="ltx_bibblock"><span id="bib.bib53.1.1" class="ltx_text" style="font-size:90%;">
Niklas Muennighoff, Alexander M. Rush, Boaz Barak, Teven Le Scao, Aleksandra Piktus, Nouamane Tazi, Sampo Pyysalo, Thomas Wolf, and Colin Raffel.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib53.2.1" class="ltx_text" style="font-size:90%;">Scaling data-constrained language models, 2023.
</span>
</span>
</li>
<li id="bib.bib54" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[54]</span>
<span class="ltx_bibblock"><span id="bib.bib54.1.1" class="ltx_text" style="font-size:90%;">
Thuat Nguyen, Chien Van Nguyen, Viet Dac Lai, Hieu Man, Nghia Trung Ngo, Franck Dernoncourt, Ryan A. Rossi, and Thien Huu Nguyen.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib54.2.1" class="ltx_text" style="font-size:90%;">Culturax: A cleaned, enormous, and multilingual dataset for large language models in 167 languages, 2023.
</span>
</span>
</li>
<li id="bib.bib55" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[55]</span>
<span class="ltx_bibblock"><span id="bib.bib55.1.1" class="ltx_text" style="font-size:90%;">
OpenAI, :, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mo Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao,
Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Łukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Łukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe,
Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O’Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder,
Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao
Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib55.2.1" class="ltx_text" style="font-size:90%;">Gpt-4 technical report, 2023.
</span>
</span>
</li>
<li id="bib.bib56" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[56]</span>
<span class="ltx_bibblock"><span id="bib.bib56.1.1" class="ltx_text" style="font-size:90%;">
Keiran Paster, Marco Dos Santos, Zhangir Azerbayev, and Jimmy Ba.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib56.2.1" class="ltx_text" style="font-size:90%;">Openwebmath: An open dataset of high-quality mathematical web text, 2023.
</span>
</span>
</li>
<li id="bib.bib57" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[57]</span>
<span class="ltx_bibblock"><span id="bib.bib57.1.1" class="ltx_text" style="font-size:90%;">
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib57.2.1" class="ltx_text" style="font-size:90%;">The refinedweb dataset for falcon llm: Outperforming curated corpora with web data, and web data only, 2023.
</span>
</span>
</li>
<li id="bib.bib58" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[58]</span>
<span class="ltx_bibblock"><span id="bib.bib58.1.1" class="ltx_text" style="font-size:90%;">
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib58.2.1" class="ltx_text" style="font-size:90%;">Improving language understanding by generative pre-training, 2018.
</span>
</span>
</li>
<li id="bib.bib59" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[59]</span>
<span class="ltx_bibblock"><span id="bib.bib59.1.1" class="ltx_text" style="font-size:90%;">
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib59.2.1" class="ltx_text" style="font-size:90%;">Exploring the limits of transfer learning with a unified text-to-text transformer, 2023.
</span>
</span>
</li>
<li id="bib.bib60" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[60]</span>
<span class="ltx_bibblock"><span id="bib.bib60.1.1" class="ltx_text" style="font-size:90%;">
Vipul Raheja, Dhruv Kumar, Ryan Koo, and Dongyeop Kang.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib60.2.1" class="ltx_text" style="font-size:90%;">Coedit: Text editing by task-specific instruction tuning, 2023.
</span>
</span>
</li>
<li id="bib.bib61" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[61]</span>
<span class="ltx_bibblock"><span id="bib.bib61.1.1" class="ltx_text" style="font-size:90%;">
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib61.2.1" class="ltx_text" style="font-size:90%;">Zero: Memory optimizations toward training trillion parameter models, 2020.
</span>
</span>
</li>
<li id="bib.bib62" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[62]</span>
<span class="ltx_bibblock"><span id="bib.bib62.1.1" class="ltx_text" style="font-size:90%;">
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib62.2.1" class="ltx_text" style="font-size:90%;">Winogrande: An adversarial winograd schema challenge at scale, 2019.
</span>
</span>
</li>
<li id="bib.bib63" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[63]</span>
<span class="ltx_bibblock"><span id="bib.bib63.1.1" class="ltx_text" style="font-size:90%;">
Eva Sharma, Chen Li, and Lu Wang.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib63.2.1" class="ltx_text" style="font-size:90%;">BIGPATENT: A large-scale dataset for abstractive and coherent summarization.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib63.3.1" class="ltx_text ltx_font_italic" style="font-size:90%;">CoRR</span><span id="bib.bib63.4.2" class="ltx_text" style="font-size:90%;">, abs/1906.03741, 2019.
</span>
</span>
</li>
<li id="bib.bib64" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[64]</span>
<span class="ltx_bibblock"><span id="bib.bib64.1.1" class="ltx_text" style="font-size:90%;">
Zhengxiang Shi, Qiang Zhang, and Aldo Lipani.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib64.2.1" class="ltx_text" style="font-size:90%;">Stepgame: A new benchmark for robust multi-hop spatial reasoning in texts.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib64.3.1" class="ltx_text" style="font-size:90%;">In </span><span id="bib.bib64.4.2" class="ltx_text ltx_font_italic" style="font-size:90%;">Proceedings of the AAAI Conference on Artificial Intelligence</span><span id="bib.bib64.5.3" class="ltx_text" style="font-size:90%;">, volume 36, pages 11321–11329, Jun. 2022.
</span>
</span>
</li>
<li id="bib.bib65" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[65]</span>
<span class="ltx_bibblock"><span id="bib.bib65.1.1" class="ltx_text" style="font-size:90%;">
Gizem Soğancıoğlu, Hakime "Ozt"urk, and Arzucan "Ozg"ur.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib65.2.1" class="ltx_text" style="font-size:90%;">Biosses: a semantic sentence similarity estimation system for the biomedical domain.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib65.3.1" class="ltx_text ltx_font_italic" style="font-size:90%;">Bioinformatics</span><span id="bib.bib65.4.2" class="ltx_text" style="font-size:90%;">, 33(14):i49–i58, 2017.
</span>
</span>
</li>
<li id="bib.bib66" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[66]</span>
<span class="ltx_bibblock"><span id="bib.bib66.1.1" class="ltx_text" style="font-size:90%;">
Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib66.2.1" class="ltx_text" style="font-size:90%;">Roformer: Enhanced transformer with rotary position embedding, 2023.
</span>
</span>
</li>
<li id="bib.bib67" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[67]</span>
<span class="ltx_bibblock"><span id="bib.bib67.1.1" class="ltx_text" style="font-size:90%;">
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib67.2.1" class="ltx_text" style="font-size:90%;">Llama: Open and efficient foundation language models, 2023.
</span>
</span>
</li>
<li id="bib.bib68" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[68]</span>
<span class="ltx_bibblock"><span id="bib.bib68.1.1" class="ltx_text" style="font-size:90%;">
Jonathan Tow, Marco Bellagente, Dakota Mahan, and Carlos Riquelme.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib68.2.1" class="ltx_text" style="font-size:90%;">Stablelm 3b 4e1t, 2023.
</span>
</span>
</li>
<li id="bib.bib69" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[69]</span>
<span class="ltx_bibblock"><span id="bib.bib69.1.1" class="ltx_text" style="font-size:90%;">
Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Shengyi Huang, Kashif Rasul, Alexander M. Rush, and Thomas Wolf.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib69.2.1" class="ltx_text" style="font-size:90%;">The alignment handbook.
</span>
</span>
<span class="ltx_bibblock"><a target="_blank" href="https://github.com/huggingface/alignment-handbook" title="" class="ltx_ref ltx_url ltx_font_typewriter" style="font-size:90%;">https://github.com/huggingface/alignment-handbook</a><span id="bib.bib69.3.1" class="ltx_text" style="font-size:90%;">, 2023.
</span>
</span>
</li>
<li id="bib.bib70" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[70]</span>
<span class="ltx_bibblock"><span id="bib.bib70.1.1" class="ltx_text" style="font-size:90%;">
Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, et al.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib70.2.1" class="ltx_text" style="font-size:90%;">Zephyr: Direct distillation of lm alignment.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib70.3.1" class="ltx_text ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2310.16944</span><span id="bib.bib70.4.2" class="ltx_text" style="font-size:90%;">, 2023.
</span>
</span>
</li>
<li id="bib.bib71" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[71]</span>
<span class="ltx_bibblock"><span id="bib.bib71.1.1" class="ltx_text" style="font-size:90%;">
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib71.2.1" class="ltx_text" style="font-size:90%;">Attention is all you need.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib71.3.1" class="ltx_text ltx_font_italic" style="font-size:90%;">Advances in neural information processing systems</span><span id="bib.bib71.4.2" class="ltx_text" style="font-size:90%;">, 30, 2017.
</span>
</span>
</li>
<li id="bib.bib72" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[72]</span>
<span class="ltx_bibblock"><span id="bib.bib72.1.1" class="ltx_text" style="font-size:90%;">
Guan Wang, Sijie Cheng, Xianyuan Zhan, Xiangang Li, Sen Song, and Yang Liu.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib72.2.1" class="ltx_text" style="font-size:90%;">Openchat: Advancing open-source language models with mixed-quality data, 2023.
</span>
</span>
</li>
<li id="bib.bib73" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[73]</span>
<span class="ltx_bibblock"><span id="bib.bib73.1.1" class="ltx_text" style="font-size:90%;">
Zhilin Wang, Yi Dong, Jiaqi Zeng, Virginia Adams, Makesh Narsimhan Sreedhar, Daniel Egert, Olivier Delalleau, Jane Polak Scowcroft, Neel Kant, Aidan Swope, and Oleksii Kuchaiev.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib73.2.1" class="ltx_text" style="font-size:90%;">Helpsteer: Multi-attribute helpfulness dataset for steerlm, 2023.
</span>
</span>
</li>
<li id="bib.bib74" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[74]</span>
<span class="ltx_bibblock"><span id="bib.bib74.1.1" class="ltx_text" style="font-size:90%;">
Mitchell Wortsman, Peter J. Liu, Lechao Xiao, Katie Everett, Alex Alemi, Ben Adlam, John D. Co-Reyes, Izzeddin Gur, Abhishek Kumar, Roman Novak, Jeffrey Pennington, Jascha Sohl-dickstein, Kelvin Xu, Jaehoon Lee, Justin Gilmer, and Simon Kornblith.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib74.2.1" class="ltx_text" style="font-size:90%;">Small-scale proxies for large-scale transformer training instabilities, 2023.
</span>
</span>
</li>
<li id="bib.bib75" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[75]</span>
<span class="ltx_bibblock"><span id="bib.bib75.1.1" class="ltx_text" style="font-size:90%;">
Qizhe Xie, Guokun Lai, Zihang Dai, and Eduard H. Hovy.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib75.2.1" class="ltx_text" style="font-size:90%;">Large-scale cloze test dataset designed by teachers.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib75.3.1" class="ltx_text ltx_font_italic" style="font-size:90%;">CoRR</span><span id="bib.bib75.4.2" class="ltx_text" style="font-size:90%;">, abs/1711.03225, 2017.
</span>
</span>
</li>
<li id="bib.bib76" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[76]</span>
<span class="ltx_bibblock"><span id="bib.bib76.1.1" class="ltx_text" style="font-size:90%;">
Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy Liang, Quoc V. Le, Tengyu Ma, and Adams Wei Yu.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib76.2.1" class="ltx_text" style="font-size:90%;">Doremi: Optimizing data mixtures speeds up language model pretraining, 2023.
</span>
</span>
</li>
<li id="bib.bib77" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[77]</span>
<span class="ltx_bibblock"><span id="bib.bib77.1.1" class="ltx_text" style="font-size:90%;">
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib77.2.1" class="ltx_text" style="font-size:90%;">Wizardlm: Empowering large language models to follow complex instructions.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib77.3.1" class="ltx_text ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2304.12244</span><span id="bib.bib77.4.2" class="ltx_text" style="font-size:90%;">, 2023.
</span>
</span>
</li>
<li id="bib.bib78" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[78]</span>
<span class="ltx_bibblock"><span id="bib.bib78.1.1" class="ltx_text" style="font-size:90%;">
Kevin Yang, Dan Klein, Asli Celikyilmaz, Nanyun Peng, and Yuandong Tian.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib78.2.1" class="ltx_text" style="font-size:90%;">Rlcd: Reinforcement learning from contrast distillation for language model alignment, 2023.
</span>
</span>
</li>
<li id="bib.bib79" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[79]</span>
<span class="ltx_bibblock"><span id="bib.bib79.1.1" class="ltx_text" style="font-size:90%;">
Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib79.2.1" class="ltx_text" style="font-size:90%;">Large batch optimization for deep learning: Training bert in 76 minutes, 2020.
</span>
</span>
</li>
<li id="bib.bib80" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[80]</span>
<span class="ltx_bibblock"><span id="bib.bib80.1.1" class="ltx_text" style="font-size:90%;">
Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib80.2.1" class="ltx_text" style="font-size:90%;">Metamath: Bootstrap your own mathematical questions for large language models.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib80.3.1" class="ltx_text ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2309.12284</span><span id="bib.bib80.4.2" class="ltx_text" style="font-size:90%;">, 2023.
</span>
</span>
</li>
<li id="bib.bib81" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[81]</span>
<span class="ltx_bibblock"><span id="bib.bib81.1.1" class="ltx_text" style="font-size:90%;">
Weizhe Yuan and Pengfei Liu.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib81.2.1" class="ltx_text" style="font-size:90%;">restructured pre-training, 2022.
</span>
</span>
</li>
<li id="bib.bib82" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[82]</span>
<span class="ltx_bibblock"><span id="bib.bib82.1.1" class="ltx_text" style="font-size:90%;">
Armel Randy Zebaze.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib82.2.1" class="ltx_text" style="font-size:90%;">Self-instruct-starcoder.
</span>
</span>
</li>
<li id="bib.bib83" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[83]</span>
<span class="ltx_bibblock"><span id="bib.bib83.1.1" class="ltx_text" style="font-size:90%;">
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib83.2.1" class="ltx_text" style="font-size:90%;">HellaSwag: Can a machine really finish your sentence?
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib83.3.1" class="ltx_text" style="font-size:90%;">In Anna Korhonen, David Traum, and Lluís Màrquez, editors, </span><span id="bib.bib83.4.2" class="ltx_text ltx_font_italic" style="font-size:90%;">Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics</span><span id="bib.bib83.5.3" class="ltx_text" style="font-size:90%;">, pages 4791–4800, Florence, Italy, July 2019. Association for Computational Linguistics.
</span>
</span>
</li>
<li id="bib.bib84" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[84]</span>
<span class="ltx_bibblock"><span id="bib.bib84.1.1" class="ltx_text" style="font-size:90%;">
Biao Zhang and Rico Sennrich.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib84.2.1" class="ltx_text" style="font-size:90%;">Root mean square layer normalization, 2019.
</span>
</span>
</li>
<li id="bib.bib85" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[85]</span>
<span class="ltx_bibblock"><span id="bib.bib85.1.1" class="ltx_text" style="font-size:90%;">
Jingmiao Zhao and Carolyn Jane Anderson.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib85.2.1" class="ltx_text" style="font-size:90%;">Solving and generating npr sunday puzzles with large language models, 2023.
</span>
</span>
</li>
<li id="bib.bib86" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[86]</span>
<span class="ltx_bibblock"><span id="bib.bib86.1.1" class="ltx_text" style="font-size:90%;">
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib86.2.1" class="ltx_text" style="font-size:90%;">Judging llm-as-a-judge with mt-bench and chatbot arena, 2023.
</span>
</span>
</li>
<li id="bib.bib87" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[87]</span>
<span class="ltx_bibblock"><span id="bib.bib87.1.1" class="ltx_text" style="font-size:90%;">
Ben Zhou, Kyle Richardson, Qiang Ning, Tushar Khot, Ashish Sabharwal, and Dan Roth.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib87.2.1" class="ltx_text" style="font-size:90%;">Temporal reasoning on implicit events from distant supervision.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib87.3.1" class="ltx_text" style="font-size:90%;">In </span><span id="bib.bib87.4.2" class="ltx_text ltx_font_italic" style="font-size:90%;">NAACL</span><span id="bib.bib87.5.3" class="ltx_text" style="font-size:90%;">, 2021.
</span>
</span>
</li>
<li id="bib.bib88" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[88]</span>
<span class="ltx_bibblock"><span id="bib.bib88.1.1" class="ltx_text" style="font-size:90%;">
Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib88.2.1" class="ltx_text" style="font-size:90%;">Lima: Less is more for alignment, 2023.
</span>
</span>
</li>
</ul>
</section>
<div class="ltx_pagination ltx_role_newpage"></div>
<section id="A1" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix A </span>Data Ablations</h2>

<div id="A1.p1" class="ltx_para">
<p id="A1.p1.1" class="ltx_p">How to select the optimal training mix for pre-training from a set of sources is an open problem. Tuning weights based on downstream tasks <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib10" title="" class="ltx_ref">10</a>, <a href="#bib.bib19" title="" class="ltx_ref">19</a>]</cite> can be extremely costly and bears the risk of overfitting on particular tasks as well as exploiting data leakage. While the computationally cheap, principled approach introduced in <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib76" title="" class="ltx_ref">76</a>]</cite> is promising, we found it delivers sub-optimal weights when the data sources are highly imbalanced and have different information content (e.g., large web sources vs curated datasets).
Furthermore, multilingual evaluations introduce a more explicit dependence on the tokenizer, with increased noise due to the lack of high-quality, non-machine-translated benchmarks.
We, therefore, aim to find general guiding principles that are expected to hold against changes in the tokenizer or in the absence of high-quality benchmarks for each data category while keeping the cost of these ablations low.</p>
</div>
<div id="A1.p2" class="ltx_para">
<p id="A1.p2.1" class="ltx_p">We trained a set of 1B models on a total of 100B tokens sampled according to Tab. <a href="#A1.T9" title="Table 9 ‣ Appendix A Data Ablations ‣ Stable LM 2 1.6B Technical Report" class="ltx_ref"><span class="ltx_text ltx_ref_tag">9</span></a>.</p>
</div>
<figure id="A1.T9" class="ltx_table">
<table id="A1.T9.2" class="ltx_tabular ltx_align_middle">
<tr id="A1.T9.2.1" class="ltx_tr">
<td id="A1.T9.2.1.1" class="ltx_td ltx_align_center ltx_align_middle ltx_border_tt" style="padding-bottom:4.30554pt;" colspan="7"><span id="A1.T9.2.1.1.1" class="ltx_text ltx_font_bold">Sampling weights</span></td>
</tr>
<tr id="A1.T9.2.2" class="ltx_tr">
<td id="A1.T9.2.2.1" class="ltx_td ltx_align_center ltx_align_middle" style="padding-bottom:4.30554pt;"><span id="A1.T9.2.2.1.1" class="ltx_text ltx_font_bold">Source</span></td>
<td id="A1.T9.2.2.2" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;background-color:#CCE6FF;padding-bottom:4.30554pt;">
<span id="A1.T9.2.2.2.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T9.2.2.2.1.1" class="ltx_p"><span id="A1.T9.2.2.2.1.1.1" class="ltx_text ltx_font_bold" style="background-color:#CCE6FF;">Control</span></span>
</span>
</td>
<td id="A1.T9.2.2.3" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;background-color:#CCFFCC;padding-bottom:4.30554pt;">
<span id="A1.T9.2.2.3.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T9.2.2.3.1.1" class="ltx_p"><span id="A1.T9.2.2.3.1.1.1" class="ltx_text ltx_font_bold" style="background-color:#CCFFCC;">Mix 1</span></span>
</span>
</td>
<td id="A1.T9.2.2.4" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;background-color:#CCFFCC;padding-bottom:4.30554pt;">
<span id="A1.T9.2.2.4.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T9.2.2.4.1.1" class="ltx_p"><span id="A1.T9.2.2.4.1.1.1" class="ltx_text ltx_font_bold" style="background-color:#CCFFCC;">Mix 2</span></span>
</span>
</td>
<td id="A1.T9.2.2.5" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;background-color:#CCFFCC;padding-bottom:4.30554pt;">
<span id="A1.T9.2.2.5.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T9.2.2.5.1.1" class="ltx_p"><span id="A1.T9.2.2.5.1.1.1" class="ltx_text ltx_font_bold" style="background-color:#CCFFCC;">Mix 3</span></span>
</span>
</td>
<td id="A1.T9.2.2.6" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;background-color:#FFB5C1;padding-bottom:4.30554pt;">
<span id="A1.T9.2.2.6.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T9.2.2.6.1.1" class="ltx_p"><span id="A1.T9.2.2.6.1.1.1" class="ltx_text ltx_font_bold" style="background-color:#FFB5C1;">Mix 4</span></span>
</span>
</td>
<td id="A1.T9.2.2.7" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;background-color:#CCCCCC;padding-bottom:4.30554pt;">
<span id="A1.T9.2.2.7.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T9.2.2.7.1.1" class="ltx_p"><span id="A1.T9.2.2.7.1.1.1" class="ltx_text ltx_font_bold" style="background-color:#CCCCCC;">Mix 5</span></span>
</span>
</td>
</tr>
<tr id="A1.T9.2.3" class="ltx_tr">
<td id="A1.T9.2.3.1" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:36.1pt;padding-bottom:4.30554pt;">Cultura-x En</td>
<td id="A1.T9.2.3.2" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:36.1pt;background-color:#CCE6FF;padding-bottom:4.30554pt;">
<span id="A1.T9.2.3.2.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T9.2.3.2.1.1" class="ltx_p ltx_align_center"><span id="A1.T9.2.3.2.1.1.1" class="ltx_text" style="background-color:#CCE6FF;">0.6894</span></span>
</span>
</td>
<td id="A1.T9.2.3.3" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:36.1pt;background-color:#CCFFCC;padding-bottom:4.30554pt;">
<span id="A1.T9.2.3.3.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T9.2.3.3.1.1" class="ltx_p"><span id="A1.T9.2.3.3.1.1.1" class="ltx_text" style="background-color:#CCFFCC;">0.5694</span></span>
</span>
</td>
<td id="A1.T9.2.3.4" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:36.1pt;background-color:#CCFFCC;padding-bottom:4.30554pt;">
<span id="A1.T9.2.3.4.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T9.2.3.4.1.1" class="ltx_p"><span id="A1.T9.2.3.4.1.1.1" class="ltx_text" style="background-color:#CCFFCC;">0.3294</span></span>
</span>
</td>
<td id="A1.T9.2.3.5" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:36.1pt;background-color:#CCFFCC;padding-bottom:4.30554pt;">
<span id="A1.T9.2.3.5.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T9.2.3.5.1.1" class="ltx_p"><span id="A1.T9.2.3.5.1.1.1" class="ltx_text" style="background-color:#CCFFCC;">0.1494</span></span>
</span>
</td>
<td id="A1.T9.2.3.6" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:36.1pt;background-color:#FFB5C1;padding-bottom:4.30554pt;">
<span id="A1.T9.2.3.6.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T9.2.3.6.1.1" class="ltx_p"><span id="A1.T9.2.3.6.1.1.1" class="ltx_text" style="background-color:#FFB5C1;">0.49</span></span>
</span>
</td>
<td id="A1.T9.2.3.7" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:36.1pt;background-color:#CCCCCC;padding-bottom:4.30554pt;">
<span id="A1.T9.2.3.7.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T9.2.3.7.1.1" class="ltx_p"><span id="A1.T9.2.3.7.1.1.1" class="ltx_text" style="background-color:#CCCCCC;">0.49</span></span>
</span>
</td>
</tr>
<tr id="A1.T9.2.4" class="ltx_tr">
<td id="A1.T9.2.4.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;padding-bottom:4.30554pt;">Cultura-x De</td>
<td id="A1.T9.2.4.2" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;background-color:#CCE6FF;padding-bottom:4.30554pt;">
<span id="A1.T9.2.4.2.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T9.2.4.2.1.1" class="ltx_p"><span id="A1.T9.2.4.2.1.1.1" class="ltx_text" style="background-color:#CCE6FF;">0.</span></span>
</span>
</td>
<td id="A1.T9.2.4.3" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;background-color:#CCFFCC;padding-bottom:4.30554pt;">
<span id="A1.T9.2.4.3.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T9.2.4.3.1.1" class="ltx_p"><span id="A1.T9.2.4.3.1.1.1" class="ltx_text" style="background-color:#CCFFCC;">0.03</span></span>
</span>
</td>
<td id="A1.T9.2.4.4" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;background-color:#CCFFCC;padding-bottom:4.30554pt;">
<span id="A1.T9.2.4.4.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T9.2.4.4.1.1" class="ltx_p"><span id="A1.T9.2.4.4.1.1.1" class="ltx_text" style="background-color:#CCFFCC;">0.06</span></span>
</span>
</td>
<td id="A1.T9.2.4.5" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;background-color:#CCFFCC;padding-bottom:4.30554pt;">
<span id="A1.T9.2.4.5.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T9.2.4.5.1.1" class="ltx_p"><span id="A1.T9.2.4.5.1.1.1" class="ltx_text" style="background-color:#CCFFCC;">0.09</span></span>
</span>
</td>
<td id="A1.T9.2.4.6" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;background-color:#FFB5C1;padding-bottom:4.30554pt;">
<span id="A1.T9.2.4.6.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T9.2.4.6.1.1" class="ltx_p"><span id="A1.T9.2.4.6.1.1.1" class="ltx_text" style="background-color:#FFB5C1;">0.</span></span>
</span>
</td>
<td id="A1.T9.2.4.7" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;background-color:#CCCCCC;padding-bottom:4.30554pt;">
<span id="A1.T9.2.4.7.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T9.2.4.7.1.1" class="ltx_p"><span id="A1.T9.2.4.7.1.1.1" class="ltx_text" style="background-color:#CCCCCC;">0.</span></span>
</span>
</td>
</tr>
<tr id="A1.T9.2.5" class="ltx_tr">
<td id="A1.T9.2.5.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;padding-bottom:4.30554pt;">Cultura-x Es</td>
<td id="A1.T9.2.5.2" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;background-color:#CCE6FF;padding-bottom:4.30554pt;">
<span id="A1.T9.2.5.2.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T9.2.5.2.1.1" class="ltx_p"><span id="A1.T9.2.5.2.1.1.1" class="ltx_text" style="background-color:#CCE6FF;">0.</span></span>
</span>
</td>
<td id="A1.T9.2.5.3" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;background-color:#CCFFCC;padding-bottom:4.30554pt;">
<span id="A1.T9.2.5.3.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T9.2.5.3.1.1" class="ltx_p"><span id="A1.T9.2.5.3.1.1.1" class="ltx_text" style="background-color:#CCFFCC;">0.03</span></span>
</span>
</td>
<td id="A1.T9.2.5.4" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;background-color:#CCFFCC;padding-bottom:4.30554pt;">
<span id="A1.T9.2.5.4.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T9.2.5.4.1.1" class="ltx_p"><span id="A1.T9.2.5.4.1.1.1" class="ltx_text" style="background-color:#CCFFCC;">0.06</span></span>
</span>
</td>
<td id="A1.T9.2.5.5" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;background-color:#CCFFCC;padding-bottom:4.30554pt;">
<span id="A1.T9.2.5.5.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T9.2.5.5.1.1" class="ltx_p"><span id="A1.T9.2.5.5.1.1.1" class="ltx_text" style="background-color:#CCFFCC;">0.09</span></span>
</span>
</td>
<td id="A1.T9.2.5.6" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;background-color:#FFB5C1;padding-bottom:4.30554pt;">
<span id="A1.T9.2.5.6.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T9.2.5.6.1.1" class="ltx_p"><span id="A1.T9.2.5.6.1.1.1" class="ltx_text" style="background-color:#FFB5C1;">0.</span></span>
</span>
</td>
<td id="A1.T9.2.5.7" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;background-color:#CCCCCC;padding-bottom:4.30554pt;">
<span id="A1.T9.2.5.7.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T9.2.5.7.1.1" class="ltx_p"><span id="A1.T9.2.5.7.1.1.1" class="ltx_text" style="background-color:#CCCCCC;">0.</span></span>
</span>
</td>
</tr>
<tr id="A1.T9.2.6" class="ltx_tr">
<td id="A1.T9.2.6.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;padding-bottom:4.30554pt;">Cultura-x Fr</td>
<td id="A1.T9.2.6.2" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;background-color:#CCE6FF;padding-bottom:4.30554pt;">
<span id="A1.T9.2.6.2.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T9.2.6.2.1.1" class="ltx_p"><span id="A1.T9.2.6.2.1.1.1" class="ltx_text" style="background-color:#CCE6FF;">0.</span></span>
</span>
</td>
<td id="A1.T9.2.6.3" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;background-color:#CCFFCC;padding-bottom:4.30554pt;">
<span id="A1.T9.2.6.3.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T9.2.6.3.1.1" class="ltx_p"><span id="A1.T9.2.6.3.1.1.1" class="ltx_text" style="background-color:#CCFFCC;">0.03</span></span>
</span>
</td>
<td id="A1.T9.2.6.4" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;background-color:#CCFFCC;padding-bottom:4.30554pt;">
<span id="A1.T9.2.6.4.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T9.2.6.4.1.1" class="ltx_p"><span id="A1.T9.2.6.4.1.1.1" class="ltx_text" style="background-color:#CCFFCC;">0.06</span></span>
</span>
</td>
<td id="A1.T9.2.6.5" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;background-color:#CCFFCC;padding-bottom:4.30554pt;">
<span id="A1.T9.2.6.5.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T9.2.6.5.1.1" class="ltx_p"><span id="A1.T9.2.6.5.1.1.1" class="ltx_text" style="background-color:#CCFFCC;">0.09</span></span>
</span>
</td>
<td id="A1.T9.2.6.6" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;background-color:#FFB5C1;padding-bottom:4.30554pt;">
<span id="A1.T9.2.6.6.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T9.2.6.6.1.1" class="ltx_p"><span id="A1.T9.2.6.6.1.1.1" class="ltx_text" style="background-color:#FFB5C1;">0.</span></span>
</span>
</td>
<td id="A1.T9.2.6.7" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;background-color:#CCCCCC;padding-bottom:4.30554pt;">
<span id="A1.T9.2.6.7.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T9.2.6.7.1.1" class="ltx_p"><span id="A1.T9.2.6.7.1.1.1" class="ltx_text" style="background-color:#CCCCCC;">0.</span></span>
</span>
</td>
</tr>
<tr id="A1.T9.2.7" class="ltx_tr">
<td id="A1.T9.2.7.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;padding-bottom:4.30554pt;">Cultura-x It</td>
<td id="A1.T9.2.7.2" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;background-color:#CCE6FF;padding-bottom:4.30554pt;">
<span id="A1.T9.2.7.2.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T9.2.7.2.1.1" class="ltx_p"><span id="A1.T9.2.7.2.1.1.1" class="ltx_text" style="background-color:#CCE6FF;">0.</span></span>
</span>
</td>
<td id="A1.T9.2.7.3" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;background-color:#CCFFCC;padding-bottom:4.30554pt;">
<span id="A1.T9.2.7.3.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T9.2.7.3.1.1" class="ltx_p"><span id="A1.T9.2.7.3.1.1.1" class="ltx_text" style="background-color:#CCFFCC;">0.03</span></span>
</span>
</td>
<td id="A1.T9.2.7.4" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;background-color:#CCFFCC;padding-bottom:4.30554pt;">
<span id="A1.T9.2.7.4.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T9.2.7.4.1.1" class="ltx_p"><span id="A1.T9.2.7.4.1.1.1" class="ltx_text" style="background-color:#CCFFCC;">0.06</span></span>
</span>
</td>
<td id="A1.T9.2.7.5" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;background-color:#CCFFCC;padding-bottom:4.30554pt;">
<span id="A1.T9.2.7.5.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T9.2.7.5.1.1" class="ltx_p"><span id="A1.T9.2.7.5.1.1.1" class="ltx_text" style="background-color:#CCFFCC;">0.09</span></span>
</span>
</td>
<td id="A1.T9.2.7.6" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;background-color:#FFB5C1;padding-bottom:4.30554pt;">
<span id="A1.T9.2.7.6.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T9.2.7.6.1.1" class="ltx_p"><span id="A1.T9.2.7.6.1.1.1" class="ltx_text" style="background-color:#FFB5C1;">0.</span></span>
</span>
</td>
<td id="A1.T9.2.7.7" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;background-color:#CCCCCC;padding-bottom:4.30554pt;">
<span id="A1.T9.2.7.7.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T9.2.7.7.1.1" class="ltx_p"><span id="A1.T9.2.7.7.1.1.1" class="ltx_text" style="background-color:#CCCCCC;">0.</span></span>
</span>
</td>
</tr>
<tr id="A1.T9.2.8" class="ltx_tr">
<td id="A1.T9.2.8.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;padding-bottom:4.30554pt;">Cultura-x Pt</td>
<td id="A1.T9.2.8.2" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;background-color:#CCE6FF;padding-bottom:4.30554pt;">
<span id="A1.T9.2.8.2.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T9.2.8.2.1.1" class="ltx_p"><span id="A1.T9.2.8.2.1.1.1" class="ltx_text" style="background-color:#CCE6FF;">0.</span></span>
</span>
</td>
<td id="A1.T9.2.8.3" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;background-color:#CCFFCC;padding-bottom:4.30554pt;">
<span id="A1.T9.2.8.3.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T9.2.8.3.1.1" class="ltx_p"><span id="A1.T9.2.8.3.1.1.1" class="ltx_text" style="background-color:#CCFFCC;">0.03</span></span>
</span>
</td>
<td id="A1.T9.2.8.4" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;background-color:#CCFFCC;padding-bottom:4.30554pt;">
<span id="A1.T9.2.8.4.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T9.2.8.4.1.1" class="ltx_p"><span id="A1.T9.2.8.4.1.1.1" class="ltx_text" style="background-color:#CCFFCC;">0.06</span></span>
</span>
</td>
<td id="A1.T9.2.8.5" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;background-color:#CCFFCC;padding-bottom:4.30554pt;">
<span id="A1.T9.2.8.5.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T9.2.8.5.1.1" class="ltx_p"><span id="A1.T9.2.8.5.1.1.1" class="ltx_text" style="background-color:#CCFFCC;">0.09</span></span>
</span>
</td>
<td id="A1.T9.2.8.6" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;background-color:#FFB5C1;padding-bottom:4.30554pt;">
<span id="A1.T9.2.8.6.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T9.2.8.6.1.1" class="ltx_p"><span id="A1.T9.2.8.6.1.1.1" class="ltx_text" style="background-color:#FFB5C1;">0.</span></span>
</span>
</td>
<td id="A1.T9.2.8.7" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;background-color:#CCCCCC;padding-bottom:4.30554pt;">
<span id="A1.T9.2.8.7.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T9.2.8.7.1.1" class="ltx_p"><span id="A1.T9.2.8.7.1.1.1" class="ltx_text" style="background-color:#CCCCCC;">0.</span></span>
</span>
</td>
</tr>
<tr id="A1.T9.2.9" class="ltx_tr">
<td id="A1.T9.2.9.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;padding-bottom:4.30554pt;">Cultura-x Nl</td>
<td id="A1.T9.2.9.2" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;background-color:#CCE6FF;padding-bottom:4.30554pt;">
<span id="A1.T9.2.9.2.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T9.2.9.2.1.1" class="ltx_p"><span id="A1.T9.2.9.2.1.1.1" class="ltx_text" style="background-color:#CCE6FF;">0.</span></span>
</span>
</td>
<td id="A1.T9.2.9.3" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;background-color:#CCFFCC;padding-bottom:4.30554pt;">
<span id="A1.T9.2.9.3.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T9.2.9.3.1.1" class="ltx_p"><span id="A1.T9.2.9.3.1.1.1" class="ltx_text" style="background-color:#CCFFCC;">0.03</span></span>
</span>
</td>
<td id="A1.T9.2.9.4" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;background-color:#CCFFCC;padding-bottom:4.30554pt;">
<span id="A1.T9.2.9.4.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T9.2.9.4.1.1" class="ltx_p"><span id="A1.T9.2.9.4.1.1.1" class="ltx_text" style="background-color:#CCFFCC;">0.06</span></span>
</span>
</td>
<td id="A1.T9.2.9.5" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;background-color:#CCFFCC;padding-bottom:4.30554pt;">
<span id="A1.T9.2.9.5.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T9.2.9.5.1.1" class="ltx_p"><span id="A1.T9.2.9.5.1.1.1" class="ltx_text" style="background-color:#CCFFCC;">0.09</span></span>
</span>
</td>
<td id="A1.T9.2.9.6" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;background-color:#FFB5C1;padding-bottom:4.30554pt;">
<span id="A1.T9.2.9.6.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T9.2.9.6.1.1" class="ltx_p"><span id="A1.T9.2.9.6.1.1.1" class="ltx_text" style="background-color:#FFB5C1;">0.</span></span>
</span>
</td>
<td id="A1.T9.2.9.7" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;background-color:#CCCCCC;padding-bottom:4.30554pt;">
<span id="A1.T9.2.9.7.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T9.2.9.7.1.1" class="ltx_p"><span id="A1.T9.2.9.7.1.1.1" class="ltx_text" style="background-color:#CCCCCC;">0.</span></span>
</span>
</td>
</tr>
<tr id="A1.T9.2.10" class="ltx_tr">
<td id="A1.T9.2.10.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;padding-bottom:4.30554pt;">Starcoder</td>
<td id="A1.T9.2.10.2" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;background-color:#CCE6FF;padding-bottom:4.30554pt;">
<span id="A1.T9.2.10.2.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T9.2.10.2.1.1" class="ltx_p"><span id="A1.T9.2.10.2.1.1.1" class="ltx_text" style="background-color:#CCE6FF;">0.0702</span></span>
</span>
</td>
<td id="A1.T9.2.10.3" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;background-color:#CCFFCC;padding-bottom:4.30554pt;">
<span id="A1.T9.2.10.3.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T9.2.10.3.1.1" class="ltx_p"><span id="A1.T9.2.10.3.1.1.1" class="ltx_text" style="background-color:#CCFFCC;">0.0702</span></span>
</span>
</td>
<td id="A1.T9.2.10.4" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;background-color:#CCFFCC;padding-bottom:4.30554pt;">
<span id="A1.T9.2.10.4.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T9.2.10.4.1.1" class="ltx_p"><span id="A1.T9.2.10.4.1.1.1" class="ltx_text" style="background-color:#CCFFCC;">0.0702</span></span>
</span>
</td>
<td id="A1.T9.2.10.5" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;background-color:#CCFFCC;padding-bottom:4.30554pt;">
<span id="A1.T9.2.10.5.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T9.2.10.5.1.1" class="ltx_p"><span id="A1.T9.2.10.5.1.1.1" class="ltx_text" style="background-color:#CCFFCC;">0.0702</span></span>
</span>
</td>
<td id="A1.T9.2.10.6" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;background-color:#FFB5C1;padding-bottom:4.30554pt;">
<span id="A1.T9.2.10.6.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T9.2.10.6.1.1" class="ltx_p"><span id="A1.T9.2.10.6.1.1.1" class="ltx_text" style="background-color:#FFB5C1;">0.0497</span></span>
</span>
</td>
<td id="A1.T9.2.10.7" class="ltx_td ltx_align_justify ltx_align_middle" style="width:36.1pt;background-color:#CCCCCC;padding-bottom:4.30554pt;">
<span id="A1.T9.2.10.7.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T9.2.10.7.1.1" class="ltx_p"><span id="A1.T9.2.10.7.1.1.1" class="ltx_text" style="background-color:#CCCCCC;">0.28</span></span>
</span>
</td>
</tr>
<tr id="A1.T9.2.11" class="ltx_tr">
<td id="A1.T9.2.11.1" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_b" style="width:36.1pt;padding-bottom:4.30554pt;">Others</td>
<td id="A1.T9.2.11.2" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_b" style="width:36.1pt;background-color:#CCE6FF;padding-bottom:4.30554pt;">
<span id="A1.T9.2.11.2.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T9.2.11.2.1.1" class="ltx_p"><span id="A1.T9.2.11.2.1.1.1" class="ltx_text" style="background-color:#CCE6FF;">0.2292</span></span>
</span>
</td>
<td id="A1.T9.2.11.3" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_b" style="width:36.1pt;background-color:#CCFFCC;padding-bottom:4.30554pt;">
<span id="A1.T9.2.11.3.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T9.2.11.3.1.1" class="ltx_p"><span id="A1.T9.2.11.3.1.1.1" class="ltx_text" style="background-color:#CCFFCC;">0.2292</span></span>
</span>
</td>
<td id="A1.T9.2.11.4" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_b" style="width:36.1pt;background-color:#CCFFCC;padding-bottom:4.30554pt;">
<span id="A1.T9.2.11.4.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T9.2.11.4.1.1" class="ltx_p"><span id="A1.T9.2.11.4.1.1.1" class="ltx_text" style="background-color:#CCFFCC;">0.2292</span></span>
</span>
</td>
<td id="A1.T9.2.11.5" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_b" style="width:36.1pt;background-color:#CCFFCC;padding-bottom:4.30554pt;">
<span id="A1.T9.2.11.5.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T9.2.11.5.1.1" class="ltx_p"><span id="A1.T9.2.11.5.1.1.1" class="ltx_text" style="background-color:#CCFFCC;">0.2292</span></span>
</span>
</td>
<td id="A1.T9.2.11.6" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_b" style="width:36.1pt;background-color:#FFB5C1;padding-bottom:4.30554pt;">
<span id="A1.T9.2.11.6.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T9.2.11.6.1.1" class="ltx_p"><span id="A1.T9.2.11.6.1.1.1" class="ltx_text" style="background-color:#FFB5C1;">0.4648</span></span>
</span>
</td>
<td id="A1.T9.2.11.7" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_b" style="width:36.1pt;background-color:#CCCCCC;padding-bottom:4.30554pt;">
<span id="A1.T9.2.11.7.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T9.2.11.7.1.1" class="ltx_p"><span id="A1.T9.2.11.7.1.1.1" class="ltx_text" style="background-color:#CCCCCC;">0.2292</span></span>
</span>
</td>
</tr>
</table>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table"><span id="A1.T9.8.1.1" class="ltx_text" style="font-size:90%;">Table 9</span>: </span><span id="A1.T9.9.2" class="ltx_text" style="font-size:90%;">
Data ablations with corresponding sampling weights. Column 2 (<span id="A1.T9.9.2.1" class="ltx_text ltx_font_bold">Control</span>) is our reference, containing only English and code data, with a standard, significant amount of web data. <span id="A1.T9.9.2.2" class="ltx_text ltx_font_bold">Mix 1-3</span> test our strategy for adding multilingual data, capping non-English sources to a fixed percentage. In <span id="A1.T9.9.2.3" class="ltx_text ltx_font_bold">Mix 4-5</span> we reduce the amount of web data, increasing respectively books and academic sources and code.
Source <span id="A1.T9.9.2.4" class="ltx_text ltx_font_bold">Others</span> contains the same data as in Tab. <a href="#S2.T1" title="Table 1 ‣ 2 Model Pre-Training ‣ Stable LM 2 1.6B Technical Report" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a> from the categories: Academic, Books, Social, Law, Math, and Wiki. In <span id="A1.T9.9.2.5" class="ltx_text ltx_font_bold">Mix 4</span>, we uniformly upsample Academic and Books sources.
</span></figcaption>
</figure>
<div id="A1.p3" class="ltx_para">
<p id="A1.p3.1" class="ltx_p">Evaluations of each model on English and non-English benchmarks are shown in Tab. <a href="#A1.T10" title="Table 10 ‣ Appendix A Data Ablations ‣ Stable LM 2 1.6B Technical Report" class="ltx_ref"><span class="ltx_text ltx_ref_tag">10</span></a>.
We observe the following trends</p>
<ul id="A1.I1" class="ltx_itemize">
<li id="A1.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A1.I1.i1.p1" class="ltx_para">
<p id="A1.I1.i1.p1.1" class="ltx_p">Contrary to <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib53" title="" class="ltx_ref">53</a>]</cite>, we find less conclusive evidence that code can be used as a neutral filler for training data as increasing the amount of code leads to a degradation in language model performances. We leave a more thorough exploration of this to future work, including math and reasoning tasks that may benefit from a higher proportion of code data.</p>
</div>
</li>
<li id="A1.I1.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A1.I1.i2.p1" class="ltx_para">
<p id="A1.I1.i2.p1.1" class="ltx_p">Performance on non-English benchmarks increases for each language by adding any amount of data in that same language. However, this increase saturates very fast and we observe only modest gains beyond <math id="A1.I1.i2.p1.1.m1.1" class="ltx_Math" alttext="6\%" display="inline"><semantics id="A1.I1.i2.p1.1.m1.1a"><mrow id="A1.I1.i2.p1.1.m1.1.1" xref="A1.I1.i2.p1.1.m1.1.1.cmml"><mn id="A1.I1.i2.p1.1.m1.1.1.2" xref="A1.I1.i2.p1.1.m1.1.1.2.cmml">6</mn><mo id="A1.I1.i2.p1.1.m1.1.1.1" xref="A1.I1.i2.p1.1.m1.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="A1.I1.i2.p1.1.m1.1b"><apply id="A1.I1.i2.p1.1.m1.1.1.cmml" xref="A1.I1.i2.p1.1.m1.1.1"><csymbol cd="latexml" id="A1.I1.i2.p1.1.m1.1.1.1.cmml" xref="A1.I1.i2.p1.1.m1.1.1.1">percent</csymbol><cn type="integer" id="A1.I1.i2.p1.1.m1.1.1.2.cmml" xref="A1.I1.i2.p1.1.m1.1.1.2">6</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.I1.i2.p1.1.m1.1c">6\%</annotation></semantics></math>.
We hypothesise that this might be due to the lack of high-quality, structured data sources in non-English languages, which we only sample from the web.</p>
</div>
</li>
<li id="A1.I1.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A1.I1.i3.p1" class="ltx_para">
<p id="A1.I1.i3.p1.1" class="ltx_p">Upsampling Academic and Books sources improves downstream performance over the control run, particularly in natural language understanding.</p>
</div>
</li>
</ul>
</div>
<figure id="A1.T10" class="ltx_table">
<table id="A1.T10.2" class="ltx_tabular ltx_align_middle">
<tr id="A1.T10.2.1" class="ltx_tr">
<td id="A1.T10.2.1.1" class="ltx_td ltx_align_center ltx_align_middle ltx_border_tt" style="padding-bottom:4.30554pt;"><span id="A1.T10.2.1.1.1" class="ltx_text ltx_font_bold">Data</span></td>
<td id="A1.T10.2.1.2" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_tt" style="width:57.8pt;padding-bottom:4.30554pt;">
<span id="A1.T10.2.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T10.2.1.2.1.1" class="ltx_p"><span id="A1.T10.2.1.2.1.1.1" class="ltx_text ltx_font_bold">Avg</span></span>
</span>
</td>
<td id="A1.T10.2.1.3" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_tt" style="width:72.3pt;padding-bottom:4.30554pt;">
<span id="A1.T10.2.1.3.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T10.2.1.3.1.1" class="ltx_p"><span id="A1.T10.2.1.3.1.1.1" class="ltx_text ltx_font_bold">English Commonsense</span></span>
</span>
</td>
<td id="A1.T10.2.1.4" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_tt" style="width:57.8pt;padding-bottom:4.30554pt;">
<span id="A1.T10.2.1.4.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T10.2.1.4.1.1" class="ltx_p"><span id="A1.T10.2.1.4.1.1.1" class="ltx_text ltx_font_bold">LAMBADA</span></span>
</span>
</td>
<td id="A1.T10.2.1.5" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_tt" style="width:57.8pt;padding-bottom:4.30554pt;">
<span id="A1.T10.2.1.5.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T10.2.1.5.1.1" class="ltx_p"><span id="A1.T10.2.1.5.1.1.1" class="ltx_text ltx_font_bold">Okapi</span></span>
</span>
</td>
</tr>
<tr id="A1.T10.2.2" class="ltx_tr">
<td id="A1.T10.2.2.1" class="ltx_td ltx_align_center ltx_align_middle ltx_border_t" style="padding-bottom:4.30554pt;"><span id="A1.T10.2.2.1.1" class="ltx_text ltx_font_bold">Control</span></td>
<td id="A1.T10.2.2.2" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:57.8pt;padding-bottom:4.30554pt;">
<span id="A1.T10.2.2.2.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T10.2.2.2.1.1" class="ltx_p">38.76</span>
</span>
</td>
<td id="A1.T10.2.2.3" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:72.3pt;padding-bottom:4.30554pt;">
<span id="A1.T10.2.2.3.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T10.2.2.3.1.1" class="ltx_p"><span id="A1.T10.2.2.3.1.1.1" class="ltx_text ltx_font_bold">66.51</span></span>
</span>
</td>
<td id="A1.T10.2.2.4" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:57.8pt;padding-bottom:4.30554pt;">
<span id="A1.T10.2.2.4.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T10.2.2.4.1.1" class="ltx_p">30.93</span>
</span>
</td>
<td id="A1.T10.2.2.5" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:57.8pt;padding-bottom:4.30554pt;">
<span id="A1.T10.2.2.5.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T10.2.2.5.1.1" class="ltx_p">33.98</span>
</span>
</td>
</tr>
<tr id="A1.T10.2.3" class="ltx_tr">
<td id="A1.T10.2.3.1" class="ltx_td ltx_align_center ltx_align_middle" style="padding-bottom:4.30554pt;"><span id="A1.T10.2.3.1.1" class="ltx_text ltx_font_bold">Mix 1</span></td>
<td id="A1.T10.2.3.2" class="ltx_td ltx_align_justify ltx_align_middle" style="width:57.8pt;padding-bottom:4.30554pt;">
<span id="A1.T10.2.3.2.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T10.2.3.2.1.1" class="ltx_p">39.91</span>
</span>
</td>
<td id="A1.T10.2.3.3" class="ltx_td ltx_align_justify ltx_align_middle" style="width:72.3pt;padding-bottom:4.30554pt;">
<span id="A1.T10.2.3.3.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T10.2.3.3.1.1" class="ltx_p">63.92</span>
</span>
</td>
<td id="A1.T10.2.3.4" class="ltx_td ltx_align_justify ltx_align_middle" style="width:57.8pt;padding-bottom:4.30554pt;">
<span id="A1.T10.2.3.4.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T10.2.3.4.1.1" class="ltx_p">34.25</span>
</span>
</td>
<td id="A1.T10.2.3.5" class="ltx_td ltx_align_justify ltx_align_middle" style="width:57.8pt;padding-bottom:4.30554pt;">
<span id="A1.T10.2.3.5.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T10.2.3.5.1.1" class="ltx_p">35.39</span>
</span>
</td>
</tr>
<tr id="A1.T10.2.4" class="ltx_tr">
<td id="A1.T10.2.4.1" class="ltx_td ltx_align_center ltx_align_middle" style="padding-bottom:4.30554pt;"><span id="A1.T10.2.4.1.1" class="ltx_text ltx_font_bold">Mix 2</span></td>
<td id="A1.T10.2.4.2" class="ltx_td ltx_align_justify ltx_align_middle" style="width:57.8pt;padding-bottom:4.30554pt;">
<span id="A1.T10.2.4.2.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T10.2.4.2.1.1" class="ltx_p"><span id="A1.T10.2.4.2.1.1.1" class="ltx_text ltx_font_bold">40.69</span></span>
</span>
</td>
<td id="A1.T10.2.4.3" class="ltx_td ltx_align_justify ltx_align_middle" style="width:72.3pt;padding-bottom:4.30554pt;">
<span id="A1.T10.2.4.3.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T10.2.4.3.1.1" class="ltx_p">64.71</span>
</span>
</td>
<td id="A1.T10.2.4.4" class="ltx_td ltx_align_justify ltx_align_middle" style="width:57.8pt;padding-bottom:4.30554pt;">
<span id="A1.T10.2.4.4.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T10.2.4.4.1.1" class="ltx_p"><span id="A1.T10.2.4.4.1.1.1" class="ltx_text ltx_font_bold">35.74</span></span>
</span>
</td>
<td id="A1.T10.2.4.5" class="ltx_td ltx_align_justify ltx_align_middle" style="width:57.8pt;padding-bottom:4.30554pt;">
<span id="A1.T10.2.4.5.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T10.2.4.5.1.1" class="ltx_p"><span id="A1.T10.2.4.5.1.1.1" class="ltx_text ltx_font_bold">35.93</span></span>
</span>
</td>
</tr>
<tr id="A1.T10.2.5" class="ltx_tr">
<td id="A1.T10.2.5.1" class="ltx_td ltx_align_center ltx_align_middle" style="padding-bottom:4.30554pt;"><span id="A1.T10.2.5.1.1" class="ltx_text ltx_font_bold">Mix 3</span></td>
<td id="A1.T10.2.5.2" class="ltx_td ltx_align_justify ltx_align_middle" style="width:57.8pt;padding-bottom:4.30554pt;">
<span id="A1.T10.2.5.2.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T10.2.5.2.1.1" class="ltx_p">39.87</span>
</span>
</td>
<td id="A1.T10.2.5.3" class="ltx_td ltx_align_justify ltx_align_middle" style="width:72.3pt;padding-bottom:4.30554pt;">
<span id="A1.T10.2.5.3.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T10.2.5.3.1.1" class="ltx_p">63.02</span>
</span>
</td>
<td id="A1.T10.2.5.4" class="ltx_td ltx_align_justify ltx_align_middle" style="width:57.8pt;padding-bottom:4.30554pt;">
<span id="A1.T10.2.5.4.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T10.2.5.4.1.1" class="ltx_p">35.22</span>
</span>
</td>
<td id="A1.T10.2.5.5" class="ltx_td ltx_align_justify ltx_align_middle" style="width:57.8pt;padding-bottom:4.30554pt;">
<span id="A1.T10.2.5.5.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T10.2.5.5.1.1" class="ltx_p">35.25</span>
</span>
</td>
</tr>
<tr id="A1.T10.2.6" class="ltx_tr">
<td id="A1.T10.2.6.1" class="ltx_td ltx_align_center ltx_align_middle" style="padding-bottom:4.30554pt;"><span id="A1.T10.2.6.1.1" class="ltx_text ltx_font_bold">Mix 4</span></td>
<td id="A1.T10.2.6.2" class="ltx_td ltx_align_justify ltx_align_middle" style="width:57.8pt;padding-bottom:4.30554pt;">
<span id="A1.T10.2.6.2.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T10.2.6.2.1.1" class="ltx_p">39.41</span>
</span>
</td>
<td id="A1.T10.2.6.3" class="ltx_td ltx_align_justify ltx_align_middle" style="width:72.3pt;padding-bottom:4.30554pt;">
<span id="A1.T10.2.6.3.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T10.2.6.3.1.1" class="ltx_p">65.87</span>
</span>
</td>
<td id="A1.T10.2.6.4" class="ltx_td ltx_align_justify ltx_align_middle" style="width:57.8pt;padding-bottom:4.30554pt;">
<span id="A1.T10.2.6.4.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T10.2.6.4.1.1" class="ltx_p">32.73</span>
</span>
</td>
<td id="A1.T10.2.6.5" class="ltx_td ltx_align_justify ltx_align_middle" style="width:57.8pt;padding-bottom:4.30554pt;">
<span id="A1.T10.2.6.5.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T10.2.6.5.1.1" class="ltx_p">34.58</span>
</span>
</td>
</tr>
<tr id="A1.T10.2.7" class="ltx_tr">
<td id="A1.T10.2.7.1" class="ltx_td ltx_align_center ltx_align_middle ltx_border_b" style="padding-bottom:4.30554pt;"><span id="A1.T10.2.7.1.1" class="ltx_text ltx_font_bold">Mix 5</span></td>
<td id="A1.T10.2.7.2" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_b" style="width:57.8pt;padding-bottom:4.30554pt;">
<span id="A1.T10.2.7.2.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T10.2.7.2.1.1" class="ltx_p">38.66</span>
</span>
</td>
<td id="A1.T10.2.7.3" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_b" style="width:72.3pt;padding-bottom:4.30554pt;">
<span id="A1.T10.2.7.3.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T10.2.7.3.1.1" class="ltx_p">64.95</span>
</span>
</td>
<td id="A1.T10.2.7.4" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_b" style="width:57.8pt;padding-bottom:4.30554pt;">
<span id="A1.T10.2.7.4.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T10.2.7.4.1.1" class="ltx_p">31.4</span>
</span>
</td>
<td id="A1.T10.2.7.5" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_b" style="width:57.8pt;padding-bottom:4.30554pt;">
<span id="A1.T10.2.7.5.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T10.2.7.5.1.1" class="ltx_p">34.07</span>
</span>
</td>
</tr>
</table>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table"><span id="A1.T10.7.1.1" class="ltx_text" style="font-size:90%;">Table 10</span>: </span><span id="A1.T10.8.2" class="ltx_text" style="font-size:90%;">
Downstream evaluations of the models considered in our data ablations. English commonsense includes: <span id="A1.T10.8.2.1" class="ltx_text ltx_font_bold">ARC-Easy</span>, <span id="A1.T10.8.2.2" class="ltx_text ltx_font_bold">PIQA</span>, <span id="A1.T10.8.2.3" class="ltx_text ltx_font_bold">SciQ</span>, <span id="A1.T10.8.2.4" class="ltx_text ltx_font_bold">WinoGrande</span>. Each score is averaged over EN, DE, ES, FR, and IT. </span></figcaption>
</figure>
</section>
<section id="A2" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix B </span>Scheduler Ablations</h2>

<div id="A2.p1" class="ltx_para">
<p id="A2.p1.3" class="ltx_p"><cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib31" title="" class="ltx_ref">31</a>]</cite> shows that the widely adopted cosine learning rate decay achieves optimal performance only when performing a full cosine period, forcing practitioners to fix the number of steps beforehand. As multi-epoch training performs well for LLMs <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib53" title="" class="ltx_ref">53</a>, <a href="#bib.bib49" title="" class="ltx_ref">49</a>, <a href="#bib.bib68" title="" class="ltx_ref">68</a>]</cite>, and larger and cleaner data sources are made accessible by the OS community, it becomes more and more important to alleviate this limitation.
To this end, we experimented with the "inverse square root" (rsqrt) learning rate scheduler <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib59" title="" class="ltx_ref">59</a>]</cite> Eq. <a href="#A2.E2" title="In Appendix B Scheduler Ablations ‣ Stable LM 2 1.6B Technical Report" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a></p>
<table id="A5.EGx2" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="A2.E2"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="A2.E2.m1.2" class="ltx_Math" alttext="\displaystyle\frac{1}{\sqrt{\text{max}\left(i,k\right)}}" display="inline"><semantics id="A2.E2.m1.2a"><mstyle displaystyle="true" id="A2.E2.m1.2.2" xref="A2.E2.m1.2.2.cmml"><mfrac id="A2.E2.m1.2.2a" xref="A2.E2.m1.2.2.cmml"><mn id="A2.E2.m1.2.2.4" xref="A2.E2.m1.2.2.4.cmml">1</mn><msqrt id="A2.E2.m1.2.2.2" xref="A2.E2.m1.2.2.2.cmml"><mrow id="A2.E2.m1.2.2.2.2.2" xref="A2.E2.m1.2.2.2.2.2.cmml"><mtext id="A2.E2.m1.2.2.2.2.2.4" xref="A2.E2.m1.2.2.2.2.2.4a.cmml">max</mtext><mo lspace="0em" rspace="0em" id="A2.E2.m1.2.2.2.2.2.3" xref="A2.E2.m1.2.2.2.2.2.3.cmml">​</mo><mrow id="A2.E2.m1.2.2.2.2.2.5.2" xref="A2.E2.m1.2.2.2.2.2.5.1.cmml"><mo id="A2.E2.m1.2.2.2.2.2.5.2.1" xref="A2.E2.m1.2.2.2.2.2.5.1.cmml">(</mo><mi id="A2.E2.m1.1.1.1.1.1.1" xref="A2.E2.m1.1.1.1.1.1.1.cmml">i</mi><mo id="A2.E2.m1.2.2.2.2.2.5.2.2" xref="A2.E2.m1.2.2.2.2.2.5.1.cmml">,</mo><mi id="A2.E2.m1.2.2.2.2.2.2" xref="A2.E2.m1.2.2.2.2.2.2.cmml">k</mi><mo id="A2.E2.m1.2.2.2.2.2.5.2.3" xref="A2.E2.m1.2.2.2.2.2.5.1.cmml">)</mo></mrow></mrow></msqrt></mfrac></mstyle><annotation-xml encoding="MathML-Content" id="A2.E2.m1.2b"><apply id="A2.E2.m1.2.2.cmml" xref="A2.E2.m1.2.2"><divide id="A2.E2.m1.2.2.3.cmml" xref="A2.E2.m1.2.2"></divide><cn type="integer" id="A2.E2.m1.2.2.4.cmml" xref="A2.E2.m1.2.2.4">1</cn><apply id="A2.E2.m1.2.2.2.cmml" xref="A2.E2.m1.2.2.2"><root id="A2.E2.m1.2.2.2a.cmml" xref="A2.E2.m1.2.2.2"></root><apply id="A2.E2.m1.2.2.2.2.2.cmml" xref="A2.E2.m1.2.2.2.2.2"><times id="A2.E2.m1.2.2.2.2.2.3.cmml" xref="A2.E2.m1.2.2.2.2.2.3"></times><ci id="A2.E2.m1.2.2.2.2.2.4a.cmml" xref="A2.E2.m1.2.2.2.2.2.4"><mtext id="A2.E2.m1.2.2.2.2.2.4.cmml" xref="A2.E2.m1.2.2.2.2.2.4">max</mtext></ci><interval closure="open" id="A2.E2.m1.2.2.2.2.2.5.1.cmml" xref="A2.E2.m1.2.2.2.2.2.5.2"><ci id="A2.E2.m1.1.1.1.1.1.1.cmml" xref="A2.E2.m1.1.1.1.1.1.1">𝑖</ci><ci id="A2.E2.m1.2.2.2.2.2.2.cmml" xref="A2.E2.m1.2.2.2.2.2.2">𝑘</ci></interval></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.E2.m1.2c">\displaystyle\frac{1}{\sqrt{\text{max}\left(i,k\right)}}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(2)</span></td>
</tr></tbody>
</table>
<p id="A2.p1.2" class="ltx_p">where <math id="A2.p1.1.m1.1" class="ltx_Math" alttext="i" display="inline"><semantics id="A2.p1.1.m1.1a"><mi id="A2.p1.1.m1.1.1" xref="A2.p1.1.m1.1.1.cmml">i</mi><annotation-xml encoding="MathML-Content" id="A2.p1.1.m1.1b"><ci id="A2.p1.1.m1.1.1.cmml" xref="A2.p1.1.m1.1.1">𝑖</ci></annotation-xml><annotation encoding="application/x-tex" id="A2.p1.1.m1.1c">i</annotation></semantics></math> is the current iteration and <math id="A2.p1.2.m2.1" class="ltx_Math" alttext="k" display="inline"><semantics id="A2.p1.2.m2.1a"><mi id="A2.p1.2.m2.1.1" xref="A2.p1.2.m2.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="A2.p1.2.m2.1b"><ci id="A2.p1.2.m2.1.1.cmml" xref="A2.p1.2.m2.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="A2.p1.2.m2.1c">k</annotation></semantics></math> the number of warmup steps. As the scheduler is strictly convex and reaches zero asymptotically, it can be used to train for infinite iterations.</p>
</div>
<div id="A2.p2" class="ltx_para">
<p id="A2.p2.1" class="ltx_p">However, in standard scenarios, of which we show an example in Fig. <a href="#A3.F4" title="Figure 4 ‣ Appendix C Evaluation of Performance Across Different Sized Context Windows ‣ Stable LM 2 1.6B Technical Report" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>, rsqrt consistently underperforms cosine. We make the comparison by decaying the learning rate to 0 in both cases, with a linear cool down for the last <math id="A2.p2.1.m1.1" class="ltx_Math" alttext="10\%" display="inline"><semantics id="A2.p2.1.m1.1a"><mrow id="A2.p2.1.m1.1.1" xref="A2.p2.1.m1.1.1.cmml"><mn id="A2.p2.1.m1.1.1.2" xref="A2.p2.1.m1.1.1.2.cmml">10</mn><mo id="A2.p2.1.m1.1.1.1" xref="A2.p2.1.m1.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="A2.p2.1.m1.1b"><apply id="A2.p2.1.m1.1.1.cmml" xref="A2.p2.1.m1.1.1"><csymbol cd="latexml" id="A2.p2.1.m1.1.1.1.cmml" xref="A2.p2.1.m1.1.1.1">percent</csymbol><cn type="integer" id="A2.p2.1.m1.1.1.2.cmml" xref="A2.p2.1.m1.1.1.2">10</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.p2.1.m1.1c">10\%</annotation></semantics></math> of the steps for the rsqrt scheduler.</p>
</div>
<div id="A2.p3" class="ltx_para">
<p id="A2.p3.1" class="ltx_p">A sharp difference between the two schedulers is how they start from the peak learning rate: with a flat slope, negative second derivative for cosine and a large slope, positive second derivative for rsqrt. This allows rsqrt to escape the high learning rate region quickly, which we identified as the main cause of the performance gap.
We then get the best of both worlds by combining both schedulers into a single function that is again the cosine for the first section, and smoothly switches to rsqrt, as described in Eq. <a href="#S2.E1" title="In 2.5 Learning Rate Scheduler ‣ 2 Model Pre-Training ‣ Stable LM 2 1.6B Technical Report" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>.</p>
</div>
<div id="A2.p4" class="ltx_para">
<p id="A2.p4.6" class="ltx_p">Finally, we experimented with two different versions of our scheduler, whose performance we show in the right panels of Fig. <a href="#A3.F4" title="Figure 4 ‣ Appendix C Evaluation of Performance Across Different Sized Context Windows ‣ Stable LM 2 1.6B Technical Report" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>: hybrid (<span id="A2.p4.6.2" class="ltx_text ltx_font_bold">hyb</span>) corresponds to our final scheduler defined in Eq. <a href="#S2.E1" title="In 2.5 Learning Rate Scheduler ‣ 2 Model Pre-Training ‣ Stable LM 2 1.6B Technical Report" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>, while for hybrid<sub id="A2.p4.6.3" class="ltx_sub">2</sub> (<span id="A2.p4.2.1" class="ltx_text ltx_font_bold">hyb<sub id="A2.p4.2.1.1" class="ltx_sub"><span id="A2.p4.2.1.1.1" class="ltx_text ltx_font_medium">2</span></sub></span>) we double the cosine period and move the turning point at half the training steps instead of at a fourth. We attribute the difference in performance between the two versions to the different average learning rates. For the set of experiments in Fig. <a href="#A3.F4" title="Figure 4 ‣ Appendix C Evaluation of Performance Across Different Sized Context Windows ‣ Stable LM 2 1.6B Technical Report" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>, integrating the scheduler over the training volume, we obtain an average learning rate of <math id="A2.p4.3.m2.1" class="ltx_Math" alttext="1.5e-4" display="inline"><semantics id="A2.p4.3.m2.1a"><mrow id="A2.p4.3.m2.1.1" xref="A2.p4.3.m2.1.1.cmml"><mrow id="A2.p4.3.m2.1.1.2" xref="A2.p4.3.m2.1.1.2.cmml"><mn id="A2.p4.3.m2.1.1.2.2" xref="A2.p4.3.m2.1.1.2.2.cmml">1.5</mn><mo lspace="0em" rspace="0em" id="A2.p4.3.m2.1.1.2.1" xref="A2.p4.3.m2.1.1.2.1.cmml">​</mo><mi id="A2.p4.3.m2.1.1.2.3" xref="A2.p4.3.m2.1.1.2.3.cmml">e</mi></mrow><mo id="A2.p4.3.m2.1.1.1" xref="A2.p4.3.m2.1.1.1.cmml">−</mo><mn id="A2.p4.3.m2.1.1.3" xref="A2.p4.3.m2.1.1.3.cmml">4</mn></mrow><annotation-xml encoding="MathML-Content" id="A2.p4.3.m2.1b"><apply id="A2.p4.3.m2.1.1.cmml" xref="A2.p4.3.m2.1.1"><minus id="A2.p4.3.m2.1.1.1.cmml" xref="A2.p4.3.m2.1.1.1"></minus><apply id="A2.p4.3.m2.1.1.2.cmml" xref="A2.p4.3.m2.1.1.2"><times id="A2.p4.3.m2.1.1.2.1.cmml" xref="A2.p4.3.m2.1.1.2.1"></times><cn type="float" id="A2.p4.3.m2.1.1.2.2.cmml" xref="A2.p4.3.m2.1.1.2.2">1.5</cn><ci id="A2.p4.3.m2.1.1.2.3.cmml" xref="A2.p4.3.m2.1.1.2.3">𝑒</ci></apply><cn type="integer" id="A2.p4.3.m2.1.1.3.cmml" xref="A2.p4.3.m2.1.1.3">4</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.p4.3.m2.1c">1.5e-4</annotation></semantics></math>, <math id="A2.p4.4.m3.1" class="ltx_Math" alttext="1.13e-4" display="inline"><semantics id="A2.p4.4.m3.1a"><mrow id="A2.p4.4.m3.1.1" xref="A2.p4.4.m3.1.1.cmml"><mrow id="A2.p4.4.m3.1.1.2" xref="A2.p4.4.m3.1.1.2.cmml"><mn id="A2.p4.4.m3.1.1.2.2" xref="A2.p4.4.m3.1.1.2.2.cmml">1.13</mn><mo lspace="0em" rspace="0em" id="A2.p4.4.m3.1.1.2.1" xref="A2.p4.4.m3.1.1.2.1.cmml">​</mo><mi id="A2.p4.4.m3.1.1.2.3" xref="A2.p4.4.m3.1.1.2.3.cmml">e</mi></mrow><mo id="A2.p4.4.m3.1.1.1" xref="A2.p4.4.m3.1.1.1.cmml">−</mo><mn id="A2.p4.4.m3.1.1.3" xref="A2.p4.4.m3.1.1.3.cmml">4</mn></mrow><annotation-xml encoding="MathML-Content" id="A2.p4.4.m3.1b"><apply id="A2.p4.4.m3.1.1.cmml" xref="A2.p4.4.m3.1.1"><minus id="A2.p4.4.m3.1.1.1.cmml" xref="A2.p4.4.m3.1.1.1"></minus><apply id="A2.p4.4.m3.1.1.2.cmml" xref="A2.p4.4.m3.1.1.2"><times id="A2.p4.4.m3.1.1.2.1.cmml" xref="A2.p4.4.m3.1.1.2.1"></times><cn type="float" id="A2.p4.4.m3.1.1.2.2.cmml" xref="A2.p4.4.m3.1.1.2.2">1.13</cn><ci id="A2.p4.4.m3.1.1.2.3.cmml" xref="A2.p4.4.m3.1.1.2.3">𝑒</ci></apply><cn type="integer" id="A2.p4.4.m3.1.1.3.cmml" xref="A2.p4.4.m3.1.1.3">4</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.p4.4.m3.1c">1.13e-4</annotation></semantics></math> and <math id="A2.p4.5.m4.1" class="ltx_Math" alttext="1.67e-4" display="inline"><semantics id="A2.p4.5.m4.1a"><mrow id="A2.p4.5.m4.1.1" xref="A2.p4.5.m4.1.1.cmml"><mrow id="A2.p4.5.m4.1.1.2" xref="A2.p4.5.m4.1.1.2.cmml"><mn id="A2.p4.5.m4.1.1.2.2" xref="A2.p4.5.m4.1.1.2.2.cmml">1.67</mn><mo lspace="0em" rspace="0em" id="A2.p4.5.m4.1.1.2.1" xref="A2.p4.5.m4.1.1.2.1.cmml">​</mo><mi id="A2.p4.5.m4.1.1.2.3" xref="A2.p4.5.m4.1.1.2.3.cmml">e</mi></mrow><mo id="A2.p4.5.m4.1.1.1" xref="A2.p4.5.m4.1.1.1.cmml">−</mo><mn id="A2.p4.5.m4.1.1.3" xref="A2.p4.5.m4.1.1.3.cmml">4</mn></mrow><annotation-xml encoding="MathML-Content" id="A2.p4.5.m4.1b"><apply id="A2.p4.5.m4.1.1.cmml" xref="A2.p4.5.m4.1.1"><minus id="A2.p4.5.m4.1.1.1.cmml" xref="A2.p4.5.m4.1.1.1"></minus><apply id="A2.p4.5.m4.1.1.2.cmml" xref="A2.p4.5.m4.1.1.2"><times id="A2.p4.5.m4.1.1.2.1.cmml" xref="A2.p4.5.m4.1.1.2.1"></times><cn type="float" id="A2.p4.5.m4.1.1.2.2.cmml" xref="A2.p4.5.m4.1.1.2.2">1.67</cn><ci id="A2.p4.5.m4.1.1.2.3.cmml" xref="A2.p4.5.m4.1.1.2.3">𝑒</ci></apply><cn type="integer" id="A2.p4.5.m4.1.1.3.cmml" xref="A2.p4.5.m4.1.1.3">4</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.p4.5.m4.1c">1.67e-4</annotation></semantics></math> for cosine, hyb and hyb<sub id="A2.p4.6.4" class="ltx_sub">2</sub> respectively. We leave to future work a proof that schedulers with the same average value lead to statistically equivalent models under mild conditions, such as monotonicity and fixed endpoints.</p>
</div>
</section>
<section id="A3" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix C </span>Evaluation of Performance Across Different Sized Context Windows</h2>

<figure id="A3.F4" class="ltx_figure"><img src="/html/2402.17834/assets/x3.png" id="A3.F4.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="346" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="A3.F4.4.2.1" class="ltx_text" style="font-size:90%;">Figure 4</span>: </span><span id="A3.F4.2.1" class="ltx_text" style="font-size:90%;">Learning rate scheduler ablations. Left figures: comparison between rsqrt and cosine decay scheduler. Right figures: final loss of models trained with the two variants of the hybrid scheduler and cosine for different number of training tokens (top). The difference in final loss between the 3 schedulers is <math id="A3.F4.2.1.m1.1" class="ltx_Math" alttext="&lt;1\%" display="inline"><semantics id="A3.F4.2.1.m1.1b"><mrow id="A3.F4.2.1.m1.1.1" xref="A3.F4.2.1.m1.1.1.cmml"><mi id="A3.F4.2.1.m1.1.1.2" xref="A3.F4.2.1.m1.1.1.2.cmml"></mi><mo id="A3.F4.2.1.m1.1.1.1" xref="A3.F4.2.1.m1.1.1.1.cmml">&lt;</mo><mrow id="A3.F4.2.1.m1.1.1.3" xref="A3.F4.2.1.m1.1.1.3.cmml"><mn id="A3.F4.2.1.m1.1.1.3.2" xref="A3.F4.2.1.m1.1.1.3.2.cmml">1</mn><mo id="A3.F4.2.1.m1.1.1.3.1" xref="A3.F4.2.1.m1.1.1.3.1.cmml">%</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A3.F4.2.1.m1.1c"><apply id="A3.F4.2.1.m1.1.1.cmml" xref="A3.F4.2.1.m1.1.1"><lt id="A3.F4.2.1.m1.1.1.1.cmml" xref="A3.F4.2.1.m1.1.1.1"></lt><csymbol cd="latexml" id="A3.F4.2.1.m1.1.1.2.cmml" xref="A3.F4.2.1.m1.1.1.2">absent</csymbol><apply id="A3.F4.2.1.m1.1.1.3.cmml" xref="A3.F4.2.1.m1.1.1.3"><csymbol cd="latexml" id="A3.F4.2.1.m1.1.1.3.1.cmml" xref="A3.F4.2.1.m1.1.1.3.1">percent</csymbol><cn type="integer" id="A3.F4.2.1.m1.1.1.3.2.cmml" xref="A3.F4.2.1.m1.1.1.3.2">1</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.F4.2.1.m1.1d">&lt;1\%</annotation></semantics></math> (bottom).</span></figcaption>
</figure>
<div id="A3.p1" class="ltx_para">
<p id="A3.p1.1" class="ltx_p">The Needle-in-a-haystack test, as introduced in <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib35" title="" class="ltx_ref">35</a>]</cite>, is commonly used to assess the retrieval capabilities of LLMs across different context window sizes. Following the methodology of <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib35" title="" class="ltx_ref">35</a>]</cite>, a fact ("the needle") is embedded within a context of unrelated essays ("the haystack").
The model under evaluation is tasked to answer a question requiring the retrieval of the needle from the context.
The evaluation is carried out systematically by placing the needle at 35 different depths within the context and comparing the results across context window sizes from 500 to 4000 tokens. Notably, these window sizes are specifically chosen to assess the performance within our trained context size of 4096 tokens. An AI judge, typically GPT-4 <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib55" title="" class="ltx_ref">55</a>]</cite>, scores the answers from 1 to 10 based on wether or not the fact was correctly retrieved.</p>
</div>
<div id="A3.p2" class="ltx_para">
<p id="A3.p2.2" class="ltx_p">While performing the evaluation, we noticed that the order of the context was not fixed or controlled by a seed, leading in some cases to significantly different scores. For this reason, we show in Fig. <a href="#A3.F5" title="Figure 5 ‣ Appendix C Evaluation of Performance Across Different Sized Context Windows ‣ Stable LM 2 1.6B Technical Report" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a> the average results over 10 different runs, together with the corresponding standard deviation. For Stable LM 2, we employ the prompt of <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib21" title="" class="ltx_ref">21</a>]</cite>, whereas for our fine-tuned version we use the official repository as is.
Averaging the scores over the evaluated grid, we observe a slight degradation from <math id="A3.p2.1.m1.1" class="ltx_Math" alttext="\approx 7.7" display="inline"><semantics id="A3.p2.1.m1.1a"><mrow id="A3.p2.1.m1.1.1" xref="A3.p2.1.m1.1.1.cmml"><mi id="A3.p2.1.m1.1.1.2" xref="A3.p2.1.m1.1.1.2.cmml"></mi><mo id="A3.p2.1.m1.1.1.1" xref="A3.p2.1.m1.1.1.1.cmml">≈</mo><mn id="A3.p2.1.m1.1.1.3" xref="A3.p2.1.m1.1.1.3.cmml">7.7</mn></mrow><annotation-xml encoding="MathML-Content" id="A3.p2.1.m1.1b"><apply id="A3.p2.1.m1.1.1.cmml" xref="A3.p2.1.m1.1.1"><approx id="A3.p2.1.m1.1.1.1.cmml" xref="A3.p2.1.m1.1.1.1"></approx><csymbol cd="latexml" id="A3.p2.1.m1.1.1.2.cmml" xref="A3.p2.1.m1.1.1.2">absent</csymbol><cn type="float" id="A3.p2.1.m1.1.1.3.cmml" xref="A3.p2.1.m1.1.1.3">7.7</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.p2.1.m1.1c">\approx 7.7</annotation></semantics></math> for our base model to <math id="A3.p2.2.m2.1" class="ltx_Math" alttext="\approx 6.8" display="inline"><semantics id="A3.p2.2.m2.1a"><mrow id="A3.p2.2.m2.1.1" xref="A3.p2.2.m2.1.1.cmml"><mi id="A3.p2.2.m2.1.1.2" xref="A3.p2.2.m2.1.1.2.cmml"></mi><mo id="A3.p2.2.m2.1.1.1" xref="A3.p2.2.m2.1.1.1.cmml">≈</mo><mn id="A3.p2.2.m2.1.1.3" xref="A3.p2.2.m2.1.1.3.cmml">6.8</mn></mrow><annotation-xml encoding="MathML-Content" id="A3.p2.2.m2.1b"><apply id="A3.p2.2.m2.1.1.cmml" xref="A3.p2.2.m2.1.1"><approx id="A3.p2.2.m2.1.1.1.cmml" xref="A3.p2.2.m2.1.1.1"></approx><csymbol cd="latexml" id="A3.p2.2.m2.1.1.2.cmml" xref="A3.p2.2.m2.1.1.2">absent</csymbol><cn type="float" id="A3.p2.2.m2.1.1.3.cmml" xref="A3.p2.2.m2.1.1.3">6.8</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.p2.2.m2.1c">\approx 6.8</annotation></semantics></math> for the fine-tuned version. The different prompt structures make it hard to directly compare the results, however, we will investigate in future work how different elements such as the distribution of the document lengths and the attention mask correlate with this behavior.</p>
</div>
<figure id="A3.F5" class="ltx_figure"><img src="/html/2402.17834/assets/x4.png" id="A3.F5.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="346" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="A3.F5.2.1.1" class="ltx_text" style="font-size:90%;">Figure 5</span>: </span><span id="A3.F5.3.2" class="ltx_text" style="font-size:90%;">Needle-in-a-haystack evaluation of Stable LM 2 on context window sizes from 500 to 4000 tokens.</span></figcaption>
</figure>
<figure id="A3.F6" class="ltx_figure"><img src="/html/2402.17834/assets/x5.png" id="A3.F6.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="272" height="204" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="A3.F6.2.1.1" class="ltx_text" style="font-size:90%;">Figure 6</span>: </span><span id="A3.F6.3.2" class="ltx_text" style="font-size:90%;">Number of iterations performed at various batch sizes to match the same final loss.</span></figcaption>
</figure>
</section>
<section id="A4" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix D </span>Global Batch Size</h2>

<div id="A4.p1" class="ltx_para">
<p id="A4.p1.5" class="ltx_p">The role of the batch size in stochastic optimization convergence is illustrated in <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib51" title="" class="ltx_ref">51</a>]</cite> as follows.
The loss <math id="A4.p1.1.m1.1" class="ltx_Math" alttext="L({\theta})" display="inline"><semantics id="A4.p1.1.m1.1a"><mrow id="A4.p1.1.m1.1.2" xref="A4.p1.1.m1.1.2.cmml"><mi id="A4.p1.1.m1.1.2.2" xref="A4.p1.1.m1.1.2.2.cmml">L</mi><mo lspace="0em" rspace="0em" id="A4.p1.1.m1.1.2.1" xref="A4.p1.1.m1.1.2.1.cmml">​</mo><mrow id="A4.p1.1.m1.1.2.3.2" xref="A4.p1.1.m1.1.2.cmml"><mo stretchy="false" id="A4.p1.1.m1.1.2.3.2.1" xref="A4.p1.1.m1.1.2.cmml">(</mo><mi id="A4.p1.1.m1.1.1" xref="A4.p1.1.m1.1.1.cmml">θ</mi><mo stretchy="false" id="A4.p1.1.m1.1.2.3.2.2" xref="A4.p1.1.m1.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A4.p1.1.m1.1b"><apply id="A4.p1.1.m1.1.2.cmml" xref="A4.p1.1.m1.1.2"><times id="A4.p1.1.m1.1.2.1.cmml" xref="A4.p1.1.m1.1.2.1"></times><ci id="A4.p1.1.m1.1.2.2.cmml" xref="A4.p1.1.m1.1.2.2">𝐿</ci><ci id="A4.p1.1.m1.1.1.cmml" xref="A4.p1.1.m1.1.1">𝜃</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.p1.1.m1.1c">L({\theta})</annotation></semantics></math> of a model parameterized by <math id="A4.p1.2.m2.1" class="ltx_Math" alttext="\theta" display="inline"><semantics id="A4.p1.2.m2.1a"><mi id="A4.p1.2.m2.1.1" xref="A4.p1.2.m2.1.1.cmml">θ</mi><annotation-xml encoding="MathML-Content" id="A4.p1.2.m2.1b"><ci id="A4.p1.2.m2.1.1.cmml" xref="A4.p1.2.m2.1.1">𝜃</ci></annotation-xml><annotation encoding="application/x-tex" id="A4.p1.2.m2.1c">\theta</annotation></semantics></math> over a training set <math id="A4.p1.3.m3.1" class="ltx_Math" alttext="D" display="inline"><semantics id="A4.p1.3.m3.1a"><mi id="A4.p1.3.m3.1.1" xref="A4.p1.3.m3.1.1.cmml">D</mi><annotation-xml encoding="MathML-Content" id="A4.p1.3.m3.1b"><ci id="A4.p1.3.m3.1.1.cmml" xref="A4.p1.3.m3.1.1">𝐷</ci></annotation-xml><annotation encoding="application/x-tex" id="A4.p1.3.m3.1c">D</annotation></semantics></math> is estimated by independently drawing random samples from <math id="A4.p1.4.m4.1" class="ltx_Math" alttext="D" display="inline"><semantics id="A4.p1.4.m4.1a"><mi id="A4.p1.4.m4.1.1" xref="A4.p1.4.m4.1.1.cmml">D</mi><annotation-xml encoding="MathML-Content" id="A4.p1.4.m4.1b"><ci id="A4.p1.4.m4.1.1.cmml" xref="A4.p1.4.m4.1.1">𝐷</ci></annotation-xml><annotation encoding="application/x-tex" id="A4.p1.4.m4.1c">D</annotation></semantics></math> to form a batch <math id="A4.p1.5.m5.1" class="ltx_Math" alttext="B" display="inline"><semantics id="A4.p1.5.m5.1a"><mi id="A4.p1.5.m5.1.1" xref="A4.p1.5.m5.1.1.cmml">B</mi><annotation-xml encoding="MathML-Content" id="A4.p1.5.m5.1b"><ci id="A4.p1.5.m5.1.1.cmml" xref="A4.p1.5.m5.1.1">𝐵</ci></annotation-xml><annotation encoding="application/x-tex" id="A4.p1.5.m5.1c">B</annotation></semantics></math>.
Thus, the loss gradient used to update the model’s parameters is given by</p>
<table id="A5.EGx3" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="A4.E3"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="A4.E3.m1.4" class="ltx_Math" alttext="\displaystyle\nabla L\left({\theta}\right)=\frac{1}{|B|}\sum_{i=1}^{|B|}\nabla_{\theta}L_{x_{i}}\left(\theta\right)" display="inline"><semantics id="A4.E3.m1.4a"><mrow id="A4.E3.m1.4.5" xref="A4.E3.m1.4.5.cmml"><mrow id="A4.E3.m1.4.5.2" xref="A4.E3.m1.4.5.2.cmml"><mrow id="A4.E3.m1.4.5.2.2" xref="A4.E3.m1.4.5.2.2.cmml"><mo rspace="0.167em" id="A4.E3.m1.4.5.2.2.1" xref="A4.E3.m1.4.5.2.2.1.cmml">∇</mo><mi id="A4.E3.m1.4.5.2.2.2" xref="A4.E3.m1.4.5.2.2.2.cmml">L</mi></mrow><mo lspace="0em" rspace="0em" id="A4.E3.m1.4.5.2.1" xref="A4.E3.m1.4.5.2.1.cmml">​</mo><mrow id="A4.E3.m1.4.5.2.3.2" xref="A4.E3.m1.4.5.2.cmml"><mo id="A4.E3.m1.4.5.2.3.2.1" xref="A4.E3.m1.4.5.2.cmml">(</mo><mi id="A4.E3.m1.3.3" xref="A4.E3.m1.3.3.cmml">θ</mi><mo id="A4.E3.m1.4.5.2.3.2.2" xref="A4.E3.m1.4.5.2.cmml">)</mo></mrow></mrow><mo id="A4.E3.m1.4.5.1" xref="A4.E3.m1.4.5.1.cmml">=</mo><mrow id="A4.E3.m1.4.5.3" xref="A4.E3.m1.4.5.3.cmml"><mstyle displaystyle="true" id="A4.E3.m1.1.1" xref="A4.E3.m1.1.1.cmml"><mfrac id="A4.E3.m1.1.1a" xref="A4.E3.m1.1.1.cmml"><mn id="A4.E3.m1.1.1.3" xref="A4.E3.m1.1.1.3.cmml">1</mn><mrow id="A4.E3.m1.1.1.1.3" xref="A4.E3.m1.1.1.1.2.cmml"><mo stretchy="false" id="A4.E3.m1.1.1.1.3.1" xref="A4.E3.m1.1.1.1.2.1.cmml">|</mo><mi id="A4.E3.m1.1.1.1.1" xref="A4.E3.m1.1.1.1.1.cmml">B</mi><mo stretchy="false" id="A4.E3.m1.1.1.1.3.2" xref="A4.E3.m1.1.1.1.2.1.cmml">|</mo></mrow></mfrac></mstyle><mo lspace="0em" rspace="0em" id="A4.E3.m1.4.5.3.1" xref="A4.E3.m1.4.5.3.1.cmml">​</mo><mrow id="A4.E3.m1.4.5.3.2" xref="A4.E3.m1.4.5.3.2.cmml"><mstyle displaystyle="true" id="A4.E3.m1.4.5.3.2.1" xref="A4.E3.m1.4.5.3.2.1.cmml"><munderover id="A4.E3.m1.4.5.3.2.1a" xref="A4.E3.m1.4.5.3.2.1.cmml"><mo movablelimits="false" id="A4.E3.m1.4.5.3.2.1.2.2" xref="A4.E3.m1.4.5.3.2.1.2.2.cmml">∑</mo><mrow id="A4.E3.m1.4.5.3.2.1.2.3" xref="A4.E3.m1.4.5.3.2.1.2.3.cmml"><mi id="A4.E3.m1.4.5.3.2.1.2.3.2" xref="A4.E3.m1.4.5.3.2.1.2.3.2.cmml">i</mi><mo id="A4.E3.m1.4.5.3.2.1.2.3.1" xref="A4.E3.m1.4.5.3.2.1.2.3.1.cmml">=</mo><mn id="A4.E3.m1.4.5.3.2.1.2.3.3" xref="A4.E3.m1.4.5.3.2.1.2.3.3.cmml">1</mn></mrow><mrow id="A4.E3.m1.2.2.1.3" xref="A4.E3.m1.2.2.1.2.cmml"><mo stretchy="false" id="A4.E3.m1.2.2.1.3.1" xref="A4.E3.m1.2.2.1.2.1.cmml">|</mo><mi id="A4.E3.m1.2.2.1.1" xref="A4.E3.m1.2.2.1.1.cmml">B</mi><mo stretchy="false" id="A4.E3.m1.2.2.1.3.2" xref="A4.E3.m1.2.2.1.2.1.cmml">|</mo></mrow></munderover></mstyle><mrow id="A4.E3.m1.4.5.3.2.2" xref="A4.E3.m1.4.5.3.2.2.cmml"><mrow id="A4.E3.m1.4.5.3.2.2.2" xref="A4.E3.m1.4.5.3.2.2.2.cmml"><msub id="A4.E3.m1.4.5.3.2.2.2.1" xref="A4.E3.m1.4.5.3.2.2.2.1.cmml"><mo lspace="0.167em" rspace="0.167em" id="A4.E3.m1.4.5.3.2.2.2.1.2" xref="A4.E3.m1.4.5.3.2.2.2.1.2.cmml">∇</mo><mi id="A4.E3.m1.4.5.3.2.2.2.1.3" xref="A4.E3.m1.4.5.3.2.2.2.1.3.cmml">θ</mi></msub><msub id="A4.E3.m1.4.5.3.2.2.2.2" xref="A4.E3.m1.4.5.3.2.2.2.2.cmml"><mi id="A4.E3.m1.4.5.3.2.2.2.2.2" xref="A4.E3.m1.4.5.3.2.2.2.2.2.cmml">L</mi><msub id="A4.E3.m1.4.5.3.2.2.2.2.3" xref="A4.E3.m1.4.5.3.2.2.2.2.3.cmml"><mi id="A4.E3.m1.4.5.3.2.2.2.2.3.2" xref="A4.E3.m1.4.5.3.2.2.2.2.3.2.cmml">x</mi><mi id="A4.E3.m1.4.5.3.2.2.2.2.3.3" xref="A4.E3.m1.4.5.3.2.2.2.2.3.3.cmml">i</mi></msub></msub></mrow><mo lspace="0em" rspace="0em" id="A4.E3.m1.4.5.3.2.2.1" xref="A4.E3.m1.4.5.3.2.2.1.cmml">​</mo><mrow id="A4.E3.m1.4.5.3.2.2.3.2" xref="A4.E3.m1.4.5.3.2.2.cmml"><mo id="A4.E3.m1.4.5.3.2.2.3.2.1" xref="A4.E3.m1.4.5.3.2.2.cmml">(</mo><mi id="A4.E3.m1.4.4" xref="A4.E3.m1.4.4.cmml">θ</mi><mo id="A4.E3.m1.4.5.3.2.2.3.2.2" xref="A4.E3.m1.4.5.3.2.2.cmml">)</mo></mrow></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="A4.E3.m1.4b"><apply id="A4.E3.m1.4.5.cmml" xref="A4.E3.m1.4.5"><eq id="A4.E3.m1.4.5.1.cmml" xref="A4.E3.m1.4.5.1"></eq><apply id="A4.E3.m1.4.5.2.cmml" xref="A4.E3.m1.4.5.2"><times id="A4.E3.m1.4.5.2.1.cmml" xref="A4.E3.m1.4.5.2.1"></times><apply id="A4.E3.m1.4.5.2.2.cmml" xref="A4.E3.m1.4.5.2.2"><ci id="A4.E3.m1.4.5.2.2.1.cmml" xref="A4.E3.m1.4.5.2.2.1">∇</ci><ci id="A4.E3.m1.4.5.2.2.2.cmml" xref="A4.E3.m1.4.5.2.2.2">𝐿</ci></apply><ci id="A4.E3.m1.3.3.cmml" xref="A4.E3.m1.3.3">𝜃</ci></apply><apply id="A4.E3.m1.4.5.3.cmml" xref="A4.E3.m1.4.5.3"><times id="A4.E3.m1.4.5.3.1.cmml" xref="A4.E3.m1.4.5.3.1"></times><apply id="A4.E3.m1.1.1.cmml" xref="A4.E3.m1.1.1"><divide id="A4.E3.m1.1.1.2.cmml" xref="A4.E3.m1.1.1"></divide><cn type="integer" id="A4.E3.m1.1.1.3.cmml" xref="A4.E3.m1.1.1.3">1</cn><apply id="A4.E3.m1.1.1.1.2.cmml" xref="A4.E3.m1.1.1.1.3"><abs id="A4.E3.m1.1.1.1.2.1.cmml" xref="A4.E3.m1.1.1.1.3.1"></abs><ci id="A4.E3.m1.1.1.1.1.cmml" xref="A4.E3.m1.1.1.1.1">𝐵</ci></apply></apply><apply id="A4.E3.m1.4.5.3.2.cmml" xref="A4.E3.m1.4.5.3.2"><apply id="A4.E3.m1.4.5.3.2.1.cmml" xref="A4.E3.m1.4.5.3.2.1"><csymbol cd="ambiguous" id="A4.E3.m1.4.5.3.2.1.1.cmml" xref="A4.E3.m1.4.5.3.2.1">superscript</csymbol><apply id="A4.E3.m1.4.5.3.2.1.2.cmml" xref="A4.E3.m1.4.5.3.2.1"><csymbol cd="ambiguous" id="A4.E3.m1.4.5.3.2.1.2.1.cmml" xref="A4.E3.m1.4.5.3.2.1">subscript</csymbol><sum id="A4.E3.m1.4.5.3.2.1.2.2.cmml" xref="A4.E3.m1.4.5.3.2.1.2.2"></sum><apply id="A4.E3.m1.4.5.3.2.1.2.3.cmml" xref="A4.E3.m1.4.5.3.2.1.2.3"><eq id="A4.E3.m1.4.5.3.2.1.2.3.1.cmml" xref="A4.E3.m1.4.5.3.2.1.2.3.1"></eq><ci id="A4.E3.m1.4.5.3.2.1.2.3.2.cmml" xref="A4.E3.m1.4.5.3.2.1.2.3.2">𝑖</ci><cn type="integer" id="A4.E3.m1.4.5.3.2.1.2.3.3.cmml" xref="A4.E3.m1.4.5.3.2.1.2.3.3">1</cn></apply></apply><apply id="A4.E3.m1.2.2.1.2.cmml" xref="A4.E3.m1.2.2.1.3"><abs id="A4.E3.m1.2.2.1.2.1.cmml" xref="A4.E3.m1.2.2.1.3.1"></abs><ci id="A4.E3.m1.2.2.1.1.cmml" xref="A4.E3.m1.2.2.1.1">𝐵</ci></apply></apply><apply id="A4.E3.m1.4.5.3.2.2.cmml" xref="A4.E3.m1.4.5.3.2.2"><times id="A4.E3.m1.4.5.3.2.2.1.cmml" xref="A4.E3.m1.4.5.3.2.2.1"></times><apply id="A4.E3.m1.4.5.3.2.2.2.cmml" xref="A4.E3.m1.4.5.3.2.2.2"><apply id="A4.E3.m1.4.5.3.2.2.2.1.cmml" xref="A4.E3.m1.4.5.3.2.2.2.1"><csymbol cd="ambiguous" id="A4.E3.m1.4.5.3.2.2.2.1.1.cmml" xref="A4.E3.m1.4.5.3.2.2.2.1">subscript</csymbol><ci id="A4.E3.m1.4.5.3.2.2.2.1.2.cmml" xref="A4.E3.m1.4.5.3.2.2.2.1.2">∇</ci><ci id="A4.E3.m1.4.5.3.2.2.2.1.3.cmml" xref="A4.E3.m1.4.5.3.2.2.2.1.3">𝜃</ci></apply><apply id="A4.E3.m1.4.5.3.2.2.2.2.cmml" xref="A4.E3.m1.4.5.3.2.2.2.2"><csymbol cd="ambiguous" id="A4.E3.m1.4.5.3.2.2.2.2.1.cmml" xref="A4.E3.m1.4.5.3.2.2.2.2">subscript</csymbol><ci id="A4.E3.m1.4.5.3.2.2.2.2.2.cmml" xref="A4.E3.m1.4.5.3.2.2.2.2.2">𝐿</ci><apply id="A4.E3.m1.4.5.3.2.2.2.2.3.cmml" xref="A4.E3.m1.4.5.3.2.2.2.2.3"><csymbol cd="ambiguous" id="A4.E3.m1.4.5.3.2.2.2.2.3.1.cmml" xref="A4.E3.m1.4.5.3.2.2.2.2.3">subscript</csymbol><ci id="A4.E3.m1.4.5.3.2.2.2.2.3.2.cmml" xref="A4.E3.m1.4.5.3.2.2.2.2.3.2">𝑥</ci><ci id="A4.E3.m1.4.5.3.2.2.2.2.3.3.cmml" xref="A4.E3.m1.4.5.3.2.2.2.2.3.3">𝑖</ci></apply></apply></apply><ci id="A4.E3.m1.4.4.cmml" xref="A4.E3.m1.4.4">𝜃</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.E3.m1.4c">\displaystyle\nabla L\left({\theta}\right)=\frac{1}{|B|}\sum_{i=1}^{|B|}\nabla_{\theta}L_{x_{i}}\left(\theta\right)</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(3)</span></td>
</tr></tbody>
</table>
<p id="A4.p1.6" class="ltx_p">and its variance scales with <math id="A4.p1.6.m1.1" class="ltx_Math" alttext="1/|B|" display="inline"><semantics id="A4.p1.6.m1.1a"><mrow id="A4.p1.6.m1.1.2" xref="A4.p1.6.m1.1.2.cmml"><mn id="A4.p1.6.m1.1.2.2" xref="A4.p1.6.m1.1.2.2.cmml">1</mn><mo id="A4.p1.6.m1.1.2.1" xref="A4.p1.6.m1.1.2.1.cmml">/</mo><mrow id="A4.p1.6.m1.1.2.3.2" xref="A4.p1.6.m1.1.2.3.1.cmml"><mo stretchy="false" id="A4.p1.6.m1.1.2.3.2.1" xref="A4.p1.6.m1.1.2.3.1.1.cmml">|</mo><mi id="A4.p1.6.m1.1.1" xref="A4.p1.6.m1.1.1.cmml">B</mi><mo stretchy="false" id="A4.p1.6.m1.1.2.3.2.2" xref="A4.p1.6.m1.1.2.3.1.1.cmml">|</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A4.p1.6.m1.1b"><apply id="A4.p1.6.m1.1.2.cmml" xref="A4.p1.6.m1.1.2"><divide id="A4.p1.6.m1.1.2.1.cmml" xref="A4.p1.6.m1.1.2.1"></divide><cn type="integer" id="A4.p1.6.m1.1.2.2.cmml" xref="A4.p1.6.m1.1.2.2">1</cn><apply id="A4.p1.6.m1.1.2.3.1.cmml" xref="A4.p1.6.m1.1.2.3.2"><abs id="A4.p1.6.m1.1.2.3.1.1.cmml" xref="A4.p1.6.m1.1.2.3.2.1"></abs><ci id="A4.p1.6.m1.1.1.cmml" xref="A4.p1.6.m1.1.1">𝐵</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.p1.6.m1.1c">1/|B|</annotation></semantics></math>. In other words, a bigger batch has a lower variance and thus provides a more accurate estimate of the gradient.
A more accurate gradient suggests that we should increase the learning rate accordingly to converge faster; however, in practice, this is far from trivial and requires special handling such as fine-tuning the learning rate scheduler or performing per-layer updates <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib25" title="" class="ltx_ref">25</a>, <a href="#bib.bib79" title="" class="ltx_ref">79</a>]</cite>.</p>
</div>
<div id="A4.p2" class="ltx_para">
<p id="A4.p2.3" class="ltx_p">To choose the global batch size for Stable LM 2, we make the following assumptions:</p>
<ol id="A4.I1" class="ltx_enumerate">
<li id="A4.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">1.</span> 
<div id="A4.I1.i1.p1" class="ltx_para">
<p id="A4.I1.i1.p1.1" class="ltx_p">We do not change the learning rate based on the batch size</p>
</div>
</li>
<li id="A4.I1.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">2.</span> 
<div id="A4.I1.i2.p1" class="ltx_para">
<p id="A4.I1.i2.p1.1" class="ltx_p">The batch size can be approximately scaled at no cost.</p>
</div>
</li>
</ol>
<p id="A4.p2.2" class="ltx_p">With <math id="A4.p2.1.m1.1" class="ltx_Math" alttext="1." display="inline"><semantics id="A4.p2.1.m1.1a"><mrow id="A4.p2.1.m1.1.2.2"><mn id="A4.p2.1.m1.1.1" xref="A4.p2.1.m1.1.1.cmml">1</mn><mo lspace="0em" id="A4.p2.1.m1.1.2.2.1">.</mo></mrow><annotation-xml encoding="MathML-Content" id="A4.p2.1.m1.1b"><cn type="integer" id="A4.p2.1.m1.1.1.cmml" xref="A4.p2.1.m1.1.1">1</cn></annotation-xml><annotation encoding="application/x-tex" id="A4.p2.1.m1.1c">1.</annotation></semantics></math>, we are, in principle, giving up on potential further gains by tuning the step size to the gradient noise. In practice, we notice that using a large learning rate at the boundary between convergence and divergence of our ablations is more than enough to compensate for this.
<math id="A4.p2.2.m2.1" class="ltx_Math" alttext="2." display="inline"><semantics id="A4.p2.2.m2.1a"><mrow id="A4.p2.2.m2.1.2.2"><mn id="A4.p2.2.m2.1.1" xref="A4.p2.2.m2.1.1.cmml">2</mn><mo lspace="0em" id="A4.p2.2.m2.1.2.2.1">.</mo></mrow><annotation-xml encoding="MathML-Content" id="A4.p2.2.m2.1b"><cn type="integer" id="A4.p2.2.m2.1.1.cmml" xref="A4.p2.2.m2.1.1">2</cn></annotation-xml><annotation encoding="application/x-tex" id="A4.p2.2.m2.1c">2.</annotation></semantics></math> follows from a combination of theoretical results, hardware optimizations, and increased training data availability. <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib53" title="" class="ltx_ref">53</a>]</cite> empirically demonstrated how multiple epochs of repeated data are as good as fresh data in LLMs pre-training, while the computational overhead of increasing data parallel workers is minimal.</p>
</div>
<div id="A4.p3" class="ltx_para">
<p id="A4.p3.4" class="ltx_p">The data volume available for pre-training is consistently increasing through larger datasets, and there are promising results of multi-epoch training. Hence,
we explore training with larger batch sizes, which requires more overall training tokens to reach the final loss but significantly decreases training time.
To determine our final batch size, we start by training a baseline with a batch <math id="A4.p3.1.m1.1" class="ltx_Math" alttext="B_{0}" display="inline"><semantics id="A4.p3.1.m1.1a"><msub id="A4.p3.1.m1.1.1" xref="A4.p3.1.m1.1.1.cmml"><mi id="A4.p3.1.m1.1.1.2" xref="A4.p3.1.m1.1.1.2.cmml">B</mi><mn id="A4.p3.1.m1.1.1.3" xref="A4.p3.1.m1.1.1.3.cmml">0</mn></msub><annotation-xml encoding="MathML-Content" id="A4.p3.1.m1.1b"><apply id="A4.p3.1.m1.1.1.cmml" xref="A4.p3.1.m1.1.1"><csymbol cd="ambiguous" id="A4.p3.1.m1.1.1.1.cmml" xref="A4.p3.1.m1.1.1">subscript</csymbol><ci id="A4.p3.1.m1.1.1.2.cmml" xref="A4.p3.1.m1.1.1.2">𝐵</ci><cn type="integer" id="A4.p3.1.m1.1.1.3.cmml" xref="A4.p3.1.m1.1.1.3">0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.p3.1.m1.1c">B_{0}</annotation></semantics></math> of 4M tokens on <math id="A4.p3.2.m2.1" class="ltx_Math" alttext="T_{0}=50B" display="inline"><semantics id="A4.p3.2.m2.1a"><mrow id="A4.p3.2.m2.1.1" xref="A4.p3.2.m2.1.1.cmml"><msub id="A4.p3.2.m2.1.1.2" xref="A4.p3.2.m2.1.1.2.cmml"><mi id="A4.p3.2.m2.1.1.2.2" xref="A4.p3.2.m2.1.1.2.2.cmml">T</mi><mn id="A4.p3.2.m2.1.1.2.3" xref="A4.p3.2.m2.1.1.2.3.cmml">0</mn></msub><mo id="A4.p3.2.m2.1.1.1" xref="A4.p3.2.m2.1.1.1.cmml">=</mo><mrow id="A4.p3.2.m2.1.1.3" xref="A4.p3.2.m2.1.1.3.cmml"><mn id="A4.p3.2.m2.1.1.3.2" xref="A4.p3.2.m2.1.1.3.2.cmml">50</mn><mo lspace="0em" rspace="0em" id="A4.p3.2.m2.1.1.3.1" xref="A4.p3.2.m2.1.1.3.1.cmml">​</mo><mi id="A4.p3.2.m2.1.1.3.3" xref="A4.p3.2.m2.1.1.3.3.cmml">B</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="A4.p3.2.m2.1b"><apply id="A4.p3.2.m2.1.1.cmml" xref="A4.p3.2.m2.1.1"><eq id="A4.p3.2.m2.1.1.1.cmml" xref="A4.p3.2.m2.1.1.1"></eq><apply id="A4.p3.2.m2.1.1.2.cmml" xref="A4.p3.2.m2.1.1.2"><csymbol cd="ambiguous" id="A4.p3.2.m2.1.1.2.1.cmml" xref="A4.p3.2.m2.1.1.2">subscript</csymbol><ci id="A4.p3.2.m2.1.1.2.2.cmml" xref="A4.p3.2.m2.1.1.2.2">𝑇</ci><cn type="integer" id="A4.p3.2.m2.1.1.2.3.cmml" xref="A4.p3.2.m2.1.1.2.3">0</cn></apply><apply id="A4.p3.2.m2.1.1.3.cmml" xref="A4.p3.2.m2.1.1.3"><times id="A4.p3.2.m2.1.1.3.1.cmml" xref="A4.p3.2.m2.1.1.3.1"></times><cn type="integer" id="A4.p3.2.m2.1.1.3.2.cmml" xref="A4.p3.2.m2.1.1.3.2">50</cn><ci id="A4.p3.2.m2.1.1.3.3.cmml" xref="A4.p3.2.m2.1.1.3.3">𝐵</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.p3.2.m2.1c">T_{0}=50B</annotation></semantics></math> total training tokens. Subsequently, we train new models from scratch with batches <math id="A4.p3.3.m3.1" class="ltx_Math" alttext="B_{i}=" display="inline"><semantics id="A4.p3.3.m3.1a"><mrow id="A4.p3.3.m3.1.1" xref="A4.p3.3.m3.1.1.cmml"><msub id="A4.p3.3.m3.1.1.2" xref="A4.p3.3.m3.1.1.2.cmml"><mi id="A4.p3.3.m3.1.1.2.2" xref="A4.p3.3.m3.1.1.2.2.cmml">B</mi><mi id="A4.p3.3.m3.1.1.2.3" xref="A4.p3.3.m3.1.1.2.3.cmml">i</mi></msub><mo id="A4.p3.3.m3.1.1.1" xref="A4.p3.3.m3.1.1.1.cmml">=</mo><mi id="A4.p3.3.m3.1.1.3" xref="A4.p3.3.m3.1.1.3.cmml"></mi></mrow><annotation-xml encoding="MathML-Content" id="A4.p3.3.m3.1b"><apply id="A4.p3.3.m3.1.1.cmml" xref="A4.p3.3.m3.1.1"><eq id="A4.p3.3.m3.1.1.1.cmml" xref="A4.p3.3.m3.1.1.1"></eq><apply id="A4.p3.3.m3.1.1.2.cmml" xref="A4.p3.3.m3.1.1.2"><csymbol cd="ambiguous" id="A4.p3.3.m3.1.1.2.1.cmml" xref="A4.p3.3.m3.1.1.2">subscript</csymbol><ci id="A4.p3.3.m3.1.1.2.2.cmml" xref="A4.p3.3.m3.1.1.2.2">𝐵</ci><ci id="A4.p3.3.m3.1.1.2.3.cmml" xref="A4.p3.3.m3.1.1.2.3">𝑖</ci></apply><csymbol cd="latexml" id="A4.p3.3.m3.1.1.3.cmml" xref="A4.p3.3.m3.1.1.3">absent</csymbol></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.p3.3.m3.1c">B_{i}=</annotation></semantics></math> 8M, 12M, and 16M, increasing <math id="A4.p3.4.m4.1" class="ltx_Math" alttext="T_{i}" display="inline"><semantics id="A4.p3.4.m4.1a"><msub id="A4.p3.4.m4.1.1" xref="A4.p3.4.m4.1.1.cmml"><mi id="A4.p3.4.m4.1.1.2" xref="A4.p3.4.m4.1.1.2.cmml">T</mi><mi id="A4.p3.4.m4.1.1.3" xref="A4.p3.4.m4.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="A4.p3.4.m4.1b"><apply id="A4.p3.4.m4.1.1.cmml" xref="A4.p3.4.m4.1.1"><csymbol cd="ambiguous" id="A4.p3.4.m4.1.1.1.cmml" xref="A4.p3.4.m4.1.1">subscript</csymbol><ci id="A4.p3.4.m4.1.1.2.cmml" xref="A4.p3.4.m4.1.1.2">𝑇</ci><ci id="A4.p3.4.m4.1.1.3.cmml" xref="A4.p3.4.m4.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.p3.4.m4.1c">T_{i}</annotation></semantics></math> tokens until the same final loss is reached.
We do this with an rsqrt scheduler as the number of required training steps is unknown beforehand.</p>
</div>
<div id="A4.p4" class="ltx_para">
<p id="A4.p4.1" class="ltx_p">In Fig. <a href="#A3.F6" title="Figure 6 ‣ Appendix C Evaluation of Performance Across Different Sized Context Windows ‣ Stable LM 2 1.6B Technical Report" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a>, we show the number of iterations <math id="A4.p4.1.m1.1" class="ltx_Math" alttext="B_{i}/T_{i}" display="inline"><semantics id="A4.p4.1.m1.1a"><mrow id="A4.p4.1.m1.1.1" xref="A4.p4.1.m1.1.1.cmml"><msub id="A4.p4.1.m1.1.1.2" xref="A4.p4.1.m1.1.1.2.cmml"><mi id="A4.p4.1.m1.1.1.2.2" xref="A4.p4.1.m1.1.1.2.2.cmml">B</mi><mi id="A4.p4.1.m1.1.1.2.3" xref="A4.p4.1.m1.1.1.2.3.cmml">i</mi></msub><mo id="A4.p4.1.m1.1.1.1" xref="A4.p4.1.m1.1.1.1.cmml">/</mo><msub id="A4.p4.1.m1.1.1.3" xref="A4.p4.1.m1.1.1.3.cmml"><mi id="A4.p4.1.m1.1.1.3.2" xref="A4.p4.1.m1.1.1.3.2.cmml">T</mi><mi id="A4.p4.1.m1.1.1.3.3" xref="A4.p4.1.m1.1.1.3.3.cmml">i</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="A4.p4.1.m1.1b"><apply id="A4.p4.1.m1.1.1.cmml" xref="A4.p4.1.m1.1.1"><divide id="A4.p4.1.m1.1.1.1.cmml" xref="A4.p4.1.m1.1.1.1"></divide><apply id="A4.p4.1.m1.1.1.2.cmml" xref="A4.p4.1.m1.1.1.2"><csymbol cd="ambiguous" id="A4.p4.1.m1.1.1.2.1.cmml" xref="A4.p4.1.m1.1.1.2">subscript</csymbol><ci id="A4.p4.1.m1.1.1.2.2.cmml" xref="A4.p4.1.m1.1.1.2.2">𝐵</ci><ci id="A4.p4.1.m1.1.1.2.3.cmml" xref="A4.p4.1.m1.1.1.2.3">𝑖</ci></apply><apply id="A4.p4.1.m1.1.1.3.cmml" xref="A4.p4.1.m1.1.1.3"><csymbol cd="ambiguous" id="A4.p4.1.m1.1.1.3.1.cmml" xref="A4.p4.1.m1.1.1.3">subscript</csymbol><ci id="A4.p4.1.m1.1.1.3.2.cmml" xref="A4.p4.1.m1.1.1.3.2">𝑇</ci><ci id="A4.p4.1.m1.1.1.3.3.cmml" xref="A4.p4.1.m1.1.1.3.3">𝑖</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.p4.1.m1.1c">B_{i}/T_{i}</annotation></semantics></math> which are required to match the baseline loss. Assuming for simplicity that the iteration time is independent of the batch size, this gives an upper bound on the speed up we can achieve by increasing the batch.</p>
</div>
<div id="A4.p5" class="ltx_para">
<p id="A4.p5.3" class="ltx_p">In the regime considered, we observe that an increase in batch size leads to a decrease in the number of iterations required to match the baseline loss all the way to the biggest batch considered of <math id="A4.p5.1.m1.1" class="ltx_Math" alttext="16.7" display="inline"><semantics id="A4.p5.1.m1.1a"><mn id="A4.p5.1.m1.1.1" xref="A4.p5.1.m1.1.1.cmml">16.7</mn><annotation-xml encoding="MathML-Content" id="A4.p5.1.m1.1b"><cn type="float" id="A4.p5.1.m1.1.1.cmml" xref="A4.p5.1.m1.1.1">16.7</cn></annotation-xml><annotation encoding="application/x-tex" id="A4.p5.1.m1.1c">16.7</annotation></semantics></math>M tokens, which speeds up training by a factor 2x. However, to achieve an equal loss, we require <math id="A4.p5.2.m2.1" class="ltx_Math" alttext="T_{i}=96B" display="inline"><semantics id="A4.p5.2.m2.1a"><mrow id="A4.p5.2.m2.1.1" xref="A4.p5.2.m2.1.1.cmml"><msub id="A4.p5.2.m2.1.1.2" xref="A4.p5.2.m2.1.1.2.cmml"><mi id="A4.p5.2.m2.1.1.2.2" xref="A4.p5.2.m2.1.1.2.2.cmml">T</mi><mi id="A4.p5.2.m2.1.1.2.3" xref="A4.p5.2.m2.1.1.2.3.cmml">i</mi></msub><mo id="A4.p5.2.m2.1.1.1" xref="A4.p5.2.m2.1.1.1.cmml">=</mo><mrow id="A4.p5.2.m2.1.1.3" xref="A4.p5.2.m2.1.1.3.cmml"><mn id="A4.p5.2.m2.1.1.3.2" xref="A4.p5.2.m2.1.1.3.2.cmml">96</mn><mo lspace="0em" rspace="0em" id="A4.p5.2.m2.1.1.3.1" xref="A4.p5.2.m2.1.1.3.1.cmml">​</mo><mi id="A4.p5.2.m2.1.1.3.3" xref="A4.p5.2.m2.1.1.3.3.cmml">B</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="A4.p5.2.m2.1b"><apply id="A4.p5.2.m2.1.1.cmml" xref="A4.p5.2.m2.1.1"><eq id="A4.p5.2.m2.1.1.1.cmml" xref="A4.p5.2.m2.1.1.1"></eq><apply id="A4.p5.2.m2.1.1.2.cmml" xref="A4.p5.2.m2.1.1.2"><csymbol cd="ambiguous" id="A4.p5.2.m2.1.1.2.1.cmml" xref="A4.p5.2.m2.1.1.2">subscript</csymbol><ci id="A4.p5.2.m2.1.1.2.2.cmml" xref="A4.p5.2.m2.1.1.2.2">𝑇</ci><ci id="A4.p5.2.m2.1.1.2.3.cmml" xref="A4.p5.2.m2.1.1.2.3">𝑖</ci></apply><apply id="A4.p5.2.m2.1.1.3.cmml" xref="A4.p5.2.m2.1.1.3"><times id="A4.p5.2.m2.1.1.3.1.cmml" xref="A4.p5.2.m2.1.1.3.1"></times><cn type="integer" id="A4.p5.2.m2.1.1.3.2.cmml" xref="A4.p5.2.m2.1.1.3.2">96</cn><ci id="A4.p5.2.m2.1.1.3.3.cmml" xref="A4.p5.2.m2.1.1.3.3">𝐵</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.p5.2.m2.1c">T_{i}=96B</annotation></semantics></math> training tokens, which is a factor 1.96 increase compared to the baseline.
Therefore to train Stable LM 2, we opt for a global batch of <math id="A4.p5.3.m3.3" class="ltx_Math" alttext="8,388,608" display="inline"><semantics id="A4.p5.3.m3.3a"><mrow id="A4.p5.3.m3.3.4.2" xref="A4.p5.3.m3.3.4.1.cmml"><mn id="A4.p5.3.m3.1.1" xref="A4.p5.3.m3.1.1.cmml">8</mn><mo id="A4.p5.3.m3.3.4.2.1" xref="A4.p5.3.m3.3.4.1.cmml">,</mo><mn id="A4.p5.3.m3.2.2" xref="A4.p5.3.m3.2.2.cmml">388</mn><mo id="A4.p5.3.m3.3.4.2.2" xref="A4.p5.3.m3.3.4.1.cmml">,</mo><mn id="A4.p5.3.m3.3.3" xref="A4.p5.3.m3.3.3.cmml">608</mn></mrow><annotation-xml encoding="MathML-Content" id="A4.p5.3.m3.3b"><list id="A4.p5.3.m3.3.4.1.cmml" xref="A4.p5.3.m3.3.4.2"><cn type="integer" id="A4.p5.3.m3.1.1.cmml" xref="A4.p5.3.m3.1.1">8</cn><cn type="integer" id="A4.p5.3.m3.2.2.cmml" xref="A4.p5.3.m3.2.2">388</cn><cn type="integer" id="A4.p5.3.m3.3.3.cmml" xref="A4.p5.3.m3.3.3">608</cn></list></annotation-xml><annotation encoding="application/x-tex" id="A4.p5.3.m3.3c">8,388,608</annotation></semantics></math>, which with the layout of Tab. <a href="#S2.T3" title="Table 3 ‣ 2.2 Data ‣ 2 Model Pre-Training ‣ Stable LM 2 1.6B Technical Report" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a> offers the best compromise between decrease in training time and additional required training tokens.</p>
</div>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
<section id="A5" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix E </span>Restructed Pre-training Sources</h2>

<div id="A5.p1" class="ltx_para">
<p id="A5.p1.1" class="ltx_p">While many sources are restructured in <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib81" title="" class="ltx_ref">81</a>]</cite>, in this work, we have only considered those listed in Tab.<a href="#A5.T11" title="Table 11 ‣ Appendix E Restructed Pre-training Sources ‣ Stable LM 2 1.6B Technical Report" class="ltx_ref"><span class="ltx_text ltx_ref_tag">11</span></a> with commercially viable licenses such as BIGPATENT, CLOTH, SciTLDR, TriviaQA, WordNet, WikiHow, etc. We also restructure additional sources with similar licensing compatibility and list them below.</p>
</div>
<figure id="A5.T11" class="ltx_table">
<table id="A5.T11.2" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="A5.T11.2.1" class="ltx_tr">
<td id="A5.T11.2.1.1" class="ltx_td ltx_align_left ltx_border_tt"><span id="A5.T11.2.1.1.1" class="ltx_text ltx_font_bold">Dataset</span></td>
<td id="A5.T11.2.1.2" class="ltx_td ltx_align_left ltx_border_tt"><span id="A5.T11.2.1.2.1" class="ltx_text ltx_font_bold">Prefix</span></td>
</tr>
<tr id="A5.T11.2.2" class="ltx_tr">
<td id="A5.T11.2.2.1" class="ltx_td ltx_align_left ltx_border_t">Banking77 <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib9" title="" class="ltx_ref">9</a>]</cite>
</td>
<td id="A5.T11.2.2.2" class="ltx_td ltx_align_left ltx_border_t">banking77</td>
</tr>
<tr id="A5.T11.2.3" class="ltx_tr">
<td id="A5.T11.2.3.1" class="ltx_td ltx_align_left">BigPatent <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib63" title="" class="ltx_ref">63</a>]</cite>
</td>
<td id="A5.T11.2.3.2" class="ltx_td ltx_align_left">big_patent</td>
</tr>
<tr id="A5.T11.2.4" class="ltx_tr">
<td id="A5.T11.2.4.1" class="ltx_td ltx_align_left">BIOSSES <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib65" title="" class="ltx_ref">65</a>]</cite>
</td>
<td id="A5.T11.2.4.2" class="ltx_td ltx_align_left">biosses</td>
</tr>
<tr id="A5.T11.2.5" class="ltx_tr">
<td id="A5.T11.2.5.1" class="ltx_td ltx_align_left">BLBooksGenre <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib7" title="" class="ltx_ref">7</a>]</cite>
</td>
<td id="A5.T11.2.5.2" class="ltx_td ltx_align_left">TheBritishLibrary/blbooksgenre</td>
</tr>
<tr id="A5.T11.2.6" class="ltx_tr">
<td id="A5.T11.2.6.1" class="ltx_td ltx_align_left">CodeComplex <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib33" title="" class="ltx_ref">33</a>]</cite>
</td>
<td id="A5.T11.2.6.2" class="ltx_td ltx_align_left">codeparrot/codecomplex</td>
</tr>
<tr id="A5.T11.2.7" class="ltx_tr">
<td id="A5.T11.2.7.1" class="ltx_td ltx_align_left">CoEdIT <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib60" title="" class="ltx_ref">60</a>]</cite>
</td>
<td id="A5.T11.2.7.2" class="ltx_td ltx_align_left">grammarly/coedit</td>
</tr>
<tr id="A5.T11.2.8" class="ltx_tr">
<td id="A5.T11.2.8.1" class="ltx_td ltx_align_left">CLOTH <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib75" title="" class="ltx_ref">75</a>]</cite>
</td>
<td id="A5.T11.2.8.2" class="ltx_td ltx_align_left">AndyChiang/cloth</td>
</tr>
<tr id="A5.T11.2.9" class="ltx_tr">
<td id="A5.T11.2.9.1" class="ltx_td ltx_align_left">CommonGen <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib42" title="" class="ltx_ref">42</a>]</cite>
</td>
<td id="A5.T11.2.9.2" class="ltx_td ltx_align_left">common_gen</td>
</tr>
<tr id="A5.T11.2.10" class="ltx_tr">
<td id="A5.T11.2.10.1" class="ltx_td ltx_align_left">FigQA <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib44" title="" class="ltx_ref">44</a>]</cite>
</td>
<td id="A5.T11.2.10.2" class="ltx_td ltx_align_left">nightingal3/fig-qa</td>
</tr>
<tr id="A5.T11.2.11" class="ltx_tr">
<td id="A5.T11.2.11.1" class="ltx_td ltx_align_left">FeasibilityQA <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib27" title="" class="ltx_ref">27</a>]</cite>
</td>
<td id="A5.T11.2.11.2" class="ltx_td ltx_align_left">jon-tow/feasibility_qa</td>
</tr>
<tr id="A5.T11.2.12" class="ltx_tr">
<td id="A5.T11.2.12.1" class="ltx_td ltx_align_left">Flan 2021 <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib47" title="" class="ltx_ref">47</a>]</cite>
</td>
<td id="A5.T11.2.12.2" class="ltx_td ltx_align_left">DataProvenanceInitiative/flan2021_submix_original</td>
</tr>
<tr id="A5.T11.2.13" class="ltx_tr">
<td id="A5.T11.2.13.1" class="ltx_td ltx_align_left">Flan Chain of Thought <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib47" title="" class="ltx_ref">47</a>]</cite>
</td>
<td id="A5.T11.2.13.2" class="ltx_td ltx_align_left">DataProvenanceInitiative/cot_submix_original</td>
</tr>
<tr id="A5.T11.2.14" class="ltx_tr">
<td id="A5.T11.2.14.1" class="ltx_td ltx_align_left">Flan NIv2 <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib47" title="" class="ltx_ref">47</a>]</cite>
</td>
<td id="A5.T11.2.14.2" class="ltx_td ltx_align_left">DataProvenanceInitiative/niv2_submix_original</td>
</tr>
<tr id="A5.T11.2.15" class="ltx_tr">
<td id="A5.T11.2.15.1" class="ltx_td ltx_align_left">Flan T0 <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib47" title="" class="ltx_ref">47</a>]</cite>
</td>
<td id="A5.T11.2.15.2" class="ltx_td ltx_align_left">DataProvenanceInitiative/t0_submix_original</td>
</tr>
<tr id="A5.T11.2.16" class="ltx_tr">
<td id="A5.T11.2.16.1" class="ltx_td ltx_align_left">HelpSteer <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib73" title="" class="ltx_ref">73</a>]</cite>
</td>
<td id="A5.T11.2.16.2" class="ltx_td ltx_align_left">nvidia/HelpSteer</td>
</tr>
<tr id="A5.T11.2.17" class="ltx_tr">
<td id="A5.T11.2.17.1" class="ltx_td ltx_align_left">IMDB Review <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib50" title="" class="ltx_ref">50</a>]</cite>
</td>
<td id="A5.T11.2.17.2" class="ltx_td ltx_align_left">ajaykarthick/imdb-movie-reviews</td>
</tr>
<tr id="A5.T11.2.18" class="ltx_tr">
<td id="A5.T11.2.18.1" class="ltx_td ltx_align_left">Joke Explanation <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib47" title="" class="ltx_ref">47</a>]</cite>
</td>
<td id="A5.T11.2.18.2" class="ltx_td ltx_align_left">dim/joke_explaination</td>
</tr>
<tr id="A5.T11.2.19" class="ltx_tr">
<td id="A5.T11.2.19.1" class="ltx_td ltx_align_left">MBPP <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib1" title="" class="ltx_ref">1</a>]</cite>
</td>
<td id="A5.T11.2.19.2" class="ltx_td ltx_align_left">mbpp</td>
</tr>
<tr id="A5.T11.2.20" class="ltx_tr">
<td id="A5.T11.2.20.1" class="ltx_td ltx_align_left">NarrativeQA <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib36" title="" class="ltx_ref">36</a>]</cite>
</td>
<td id="A5.T11.2.20.2" class="ltx_td ltx_align_left">narrativeqa</td>
</tr>
<tr id="A5.T11.2.21" class="ltx_tr">
<td id="A5.T11.2.21.1" class="ltx_td ltx_align_left">PuzzleQA <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib85" title="" class="ltx_ref">85</a>]</cite>
</td>
<td id="A5.T11.2.21.2" class="ltx_td ltx_align_left">Jingmiao/PUZZLEQA</td>
</tr>
<tr id="A5.T11.2.22" class="ltx_tr">
<td id="A5.T11.2.22.1" class="ltx_td ltx_align_left">SciTLDR <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib8" title="" class="ltx_ref">8</a>]</cite>
</td>
<td id="A5.T11.2.22.2" class="ltx_td ltx_align_left">allenai/scitldr</td>
</tr>
<tr id="A5.T11.2.23" class="ltx_tr">
<td id="A5.T11.2.23.1" class="ltx_td ltx_align_left">Self-Instruct Starcoder <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib82" title="" class="ltx_ref">82</a>]</cite>
</td>
<td id="A5.T11.2.23.2" class="ltx_td ltx_align_left">codeparrot/self-instruct-starcoder</td>
</tr>
<tr id="A5.T11.2.24" class="ltx_tr">
<td id="A5.T11.2.24.1" class="ltx_td ltx_align_left">SQL Create Context <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib2" title="" class="ltx_ref">2</a>]</cite>
</td>
<td id="A5.T11.2.24.2" class="ltx_td ltx_align_left">b-mc2/sql-create-context</td>
</tr>
<tr id="A5.T11.2.25" class="ltx_tr">
<td id="A5.T11.2.25.1" class="ltx_td ltx_align_left">StepGame <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib64" title="" class="ltx_ref">64</a>]</cite>
</td>
<td id="A5.T11.2.25.2" class="ltx_td ltx_align_left">tasksource/stepgame</td>
</tr>
<tr id="A5.T11.2.26" class="ltx_tr">
<td id="A5.T11.2.26.1" class="ltx_td ltx_align_left">TRACIE <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib87" title="" class="ltx_ref">87</a>]</cite>
</td>
<td id="A5.T11.2.26.2" class="ltx_td ltx_align_left">tasksource/tracie</td>
</tr>
<tr id="A5.T11.2.27" class="ltx_tr">
<td id="A5.T11.2.27.1" class="ltx_td ltx_align_left">TriviaQA <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib34" title="" class="ltx_ref">34</a>]</cite>
</td>
<td id="A5.T11.2.27.2" class="ltx_td ltx_align_left">trivia_qa</td>
</tr>
<tr id="A5.T11.2.28" class="ltx_tr">
<td id="A5.T11.2.28.1" class="ltx_td ltx_align_left">WikiHow</td>
<td id="A5.T11.2.28.2" class="ltx_td ltx_align_left">wikihow</td>
</tr>
<tr id="A5.T11.2.29" class="ltx_tr">
<td id="A5.T11.2.29.1" class="ltx_td ltx_align_left">WordNet <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib52" title="" class="ltx_ref">52</a>]</cite>
</td>
<td id="A5.T11.2.29.2" class="ltx_td ltx_align_left">jon-tow/open-english-wordnet-synset-2023</td>
</tr>
<tr id="A5.T11.2.30" class="ltx_tr">
<td id="A5.T11.2.30.1" class="ltx_td ltx_align_left ltx_border_b">Yahoo Answers Topics</td>
<td id="A5.T11.2.30.2" class="ltx_td ltx_align_left ltx_border_b">yahoo_answers_topics</td>
</tr>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table"><span id="A5.T11.3.1.1" class="ltx_text" style="font-size:90%;">Table 11</span>: </span><span id="A5.T11.4.2" class="ltx_text" style="font-size:90%;">The original sources for restructured and instruction pre-training datasets can be found at <a target="_blank" href="https://huggingface.co/datasets/" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://huggingface.co/datasets/</a> followed by the provided prefix.</span></figcaption>
</figure>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
</article>
</div>
<div class="ar5iv-footer"><a href="/html/2402.17833" class="ar5iv-nav-button ar5iv-nav-button-prev">◄</a>
    <a class="ar5iv-home-button" href="/"><img height="40" alt="ar5iv homepage" src="/assets/ar5iv.png"></a>
    <a href="/feeling_lucky" class="ar5iv-text-button">Feeling<br>lucky?</a>
    <a href="/log/2402.17834" class="ar5iv-text-button ar5iv-severity-error">Conversion<br>report</a>
    <a class="ar5iv-text-button" target="_blank" href="https://github.com/dginev/ar5iv/issues/new?template=improve-article--arxiv-id-.md&title=Improve+article+2402.17834">Report<br>an issue</a>
    <a href="https://arxiv.org/abs/2402.17834" class="ar5iv-text-button arxiv-ui-theme">View&nbsp;original<br>on&nbsp;arXiv</a><a href="/html/2402.17835" class="ar5iv-nav-button ar5iv-nav-button-next">►</a>
</div><footer class="ltx_page_footer">
<a class="ar5iv-toggle-color-scheme" href="javascript:toggleColorScheme()" title="Toggle ar5iv color scheme"><span class="color-scheme-icon"></span></a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/license" target="_blank">Copyright</a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/policies/privacy_policy" target="_blank">Privacy Policy</a>

<div class="ltx_page_logo">Generated  on Tue Mar  5 15:56:42 2024 by <a target="_blank" href="http://dlmf.nist.gov/LaTeXML/" class="ltx_LaTeXML_logo"><span style="letter-spacing:-0.2em; margin-right:0.1em;">L<span class="ltx_font_smallcaps" style="position:relative; bottom:2.2pt;">a</span>T<span class="ltx_font_smallcaps" style="font-size:120%;position:relative; bottom:-0.2ex;">e</span></span><span style="font-size:90%; position:relative; bottom:-0.2ex;">XML</span><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAsAAAAOCAYAAAD5YeaVAAAAAXNSR0IArs4c6QAAAAZiS0dEAP8A/wD/oL2nkwAAAAlwSFlzAAALEwAACxMBAJqcGAAAAAd0SU1FB9wKExQZLWTEaOUAAAAddEVYdENvbW1lbnQAQ3JlYXRlZCB3aXRoIFRoZSBHSU1Q72QlbgAAAdpJREFUKM9tkL+L2nAARz9fPZNCKFapUn8kyI0e4iRHSR1Kb8ng0lJw6FYHFwv2LwhOpcWxTjeUunYqOmqd6hEoRDhtDWdA8ApRYsSUCDHNt5ul13vz4w0vWCgUnnEc975arX6ORqN3VqtVZbfbTQC4uEHANM3jSqXymFI6yWazP2KxWAXAL9zCUa1Wy2tXVxheKA9YNoR8Pt+aTqe4FVVVvz05O6MBhqUIBGk8Hn8HAOVy+T+XLJfLS4ZhTiRJgqIoVBRFIoric47jPnmeB1mW/9rr9ZpSSn3Lsmir1fJZlqWlUonKsvwWwD8ymc/nXwVBeLjf7xEKhdBut9Hr9WgmkyGEkJwsy5eHG5vN5g0AKIoCAEgkEkin0wQAfN9/cXPdheu6P33fBwB4ngcAcByHJpPJl+fn54mD3Gg0NrquXxeLRQAAwzAYj8cwTZPwPH9/sVg8PXweDAauqqr2cDjEer1GJBLBZDJBs9mE4zjwfZ85lAGg2+06hmGgXq+j3+/DsixYlgVN03a9Xu8jgCNCyIegIAgx13Vfd7vdu+FweG8YRkjXdWy329+dTgeSJD3ieZ7RNO0VAXAPwDEAO5VKndi2fWrb9jWl9Esul6PZbDY9Go1OZ7PZ9z/lyuD3OozU2wAAAABJRU5ErkJggg==" alt="Mascot Sammy"></a>
</div></footer>
</div>

    <script>
      var canMathML = typeof(MathMLElement) == "function";
      if (!canMathML) {
        var body = document.querySelector("body");
        body.firstElementChild.setAttribute('style', 'opacity: 0;');
        var loading = document.createElement("div");
        loading.setAttribute("id", "mathjax-loading-spinner");
        var message = document.createElement("div");
        message.setAttribute("id", "mathjax-loading-message");
        message.innerText = "Typesetting Equations...";
        body.prepend(loading);
        body.prepend(message);

        var el = document.createElement("script");
        el.src = "https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js";
        document.querySelector("head").appendChild(el);

        window.MathJax = {
          startup: {
            pageReady: () => {
              return MathJax.startup.defaultPageReady().then(() => {
                body.removeChild(loading);
                body.removeChild(message);
                body.firstElementChild.removeAttribute('style');
              }); } } };
      }
    </script>
    <script>
    // Auxiliary function, building the preview feature when
    // an inline citation is clicked
    function clicked_cite(e) {
      e.preventDefault();
      let cite = this.closest('.ltx_cite');
      let next = cite.nextSibling;
      if (next && next.nodeType == Node.ELEMENT_NODE && next.getAttribute('class') == "ar5iv-bibitem-preview") {
        next.remove();
        return; }
      // Before adding a preview modal,
      // cleanup older previews, in case they're still open
      document.querySelectorAll('span.ar5iv-bibitem-preview').forEach(function(node) {
        node.remove();
      })

      // Create the preview
      preview = document.createElement('span');
      preview.setAttribute('class','ar5iv-bibitem-preview');
      let target = document.getElementById(this.getAttribute('href').slice(1));
      target.childNodes.forEach(function (child) {
        preview.append(child.cloneNode(true));
      });
      let close_x = document.createElement('button');
      close_x.setAttribute("aria-label","Close modal for bibliography item preview");
      close_x.textContent = "×";
      close_x.setAttribute('class', 'ar5iv-button-close-preview');
      close_x.setAttribute('onclick','this.parentNode.remove()');
      preview.append(close_x);
      preview.querySelectorAll('.ltx_tag_bibitem').forEach(function(node) {
        node.remove();
      });
      cite.parentNode.insertBefore(preview, cite.nextSibling);
      return;
    }
    // Global Document initialization:
    // - assign the preview feature to all inline citation links
    document.querySelectorAll(".ltx_cite .ltx_ref").forEach(function (link) {
      link.addEventListener("click", clicked_cite);
    });
    </script>
    </body>
</html>

2402.17834
We introduce StableLM 2 1.6B, the first in a new generation of our language model series.
In this technical report, we present in detail the data and training procedure leading to the base and instruction-tuned versions of StableLM 2 1.6B.
The weights for both models are available via Hugging Face for anyone to download and use111https://huggingface.co/stabilityai/stablelm-2-1_6b222https://huggingface.co/stabilityai/stablelm-2-zephyr-1_6b.
The report contains thorough evaluations of these models, including zero- and few-shot benchmarks, multilingual benchmarks, and the MT benchmark focusing on multi-turn dialogues.
At the time of publishing this report, StableLM 2 1.6B was the state-of-the-art open model under 2B parameters by a significant margin.
Given its appealing small size, we also provide throughput measurements on a number of edge devices.
In addition, we open source several quantized checkpoints and provide their performance metrics compared to the original model.
abstract
Stable LM 2 1.6B Technical Report
Abstract
2402.17834
1IntroductionFollowing the development of the Transformer architecture, a remarkable number of proprietary and open-source large language models have been trained and deployed.
While countless new ideas and artifacts are announced weekly or daily, some key aspects remain opaque – especially around the most powerful models.
Often, the training data is not disclosed.
This poses a fundamental challenge in times where society demands transparency as it faces a brand-new disruptive technology that is hard to inspect and audit.
In this report, we explain in a reproducible manner how to train a modest-size but state-of-the-art language model.
All the data we used is public (see Table1)  and its training required around 92k GPU hours of training – worth around $322k on popular cloud providers (assuming $3.5 per GPU hour) .
We hope this work contributes to the open AI community and helps set the standard for upcoming transparent models.This report is organized as follows: Section2details the process of pre-training Stable LM 2 1.6B. We devote Section3to fine-tuning and human preference alignment.
Section4presents model evaluations on standard downstream benchmarks.
Compiling and running inference Stable LM 2 on several edge devices is outlined in Section5.
We consider several follow-up directions in Section6. Carbon emissions and societal impacts related to the training and release of Stable LM 2 are covered in Section7.
Finally, Section8concludes and summarizes this work.
introduction
Stable LM 2 1.6B Technical Report
1Introduction
2402.17834
2Model Pre-TrainingThe first stage in training large language models (LLMs)  focuses on learning to predict the next token in a sequence using a vast and diverse array of data sources. We refer to this stage aspre-training.
It enables models to build general-purpose internal representations suitable for basic language capabilities and even more advanced generation and comprehension tasks. In fact, it has been hypothesized that the majority of model knowledge and capabilities are learned during pre-training. In this section, we introduce the design principles and ablations that influenced the creation of our training set, as well as details about the model architecture and training procedure. While many similar reports exist for other cutting-edge models, they often omit critical details, such as the particular data sources, sampling weights, or the complete set of ablations they performed.
As a result, the open-source community cannot accurately reproduce these models.
On the other hand, we present a fully transparent log of our model training details. We are confident that researchers and practitioners will find valuable insights in this comprehensive account.DatasetSampling WeightNum TokensEpochsCategoryArxiv0.007915,852,446,6560.75AcademicPubMed0.01224,126,600,6261.0AcademicS2ORC0.031863,910,567,8581.0AcademicPhilPapers0.00132,591,115,7964.0AcademicBookCorpusOpen0.00459,135,954,8106.0BooksPG-190.009118,293,226,4684.0BooksFanFics0.00183,644,583,7004.0BooksCultura-X (EN) 0.2521506,625,211,0490.72WebCultura-X (ES) 0.015531,241,701,1560.4WebCultura-X (DE) 0.015230,628,813,9340.32WebCultura-X (FR) 0.009719,466,611,8080.26WebCultura-X (IT) 0.009619,202,903,4910.4WebCultura-X (NL) 0.009719,511,322,3860.62WebCultura-X (PT) 0.0120,063,968,6940.78WebC40.0855171,782,178,1081.0WebOpenWebText20.013026,161,864,4343.0WebRefinedWeb0.3292661,591,178,3391.15WebStackExchange0.023146,302,993,8202.5SocialHackerNews0.00193,817,060,5822.0SocialEuroParl0.00234,678,506,8823.0LawFreeLaw0.008817,694,697,5771.2LawPileOfLaw0.006312,663,061,6420.75LawDM Math0.006613,321,872,1383.0MathAMPS0.00112,126,719,2786.0MathOpenWebMath0.013226,530,341,2922.0MathRedPajama Wiki0.036372,866,870,4723.0WikiStarcoder0.0724145,586,775,3010.74CodeRestruct-v10.010220,412,655,6323.0InstructionTotal12,009,831,803,933-Table 1:The complete Stable LM 2 training set with sampling weights. The tokens count refers to theArcade100ktokenizer introduced in Sec.2.3.
The number of tokens in the table already includes the number of epochs shown next to it.
For instance, in the case of BookCorpusOpen, we use around 9B tokens, corresponding to 6 epochs of the original dataset (that is, each epoch is around 1.5B tokens) .
Similarly, if epochs are below one, the number of tokens shown is a subset of the total dataset.Figure 1:Percentage of effective training tokens by domain in the Stable LM 2 pre-training dataset.2.1TrainingWe train Stable LM 2 to predict the next token following standard autoregressive sequence modeling. We
train our model from scratch with a context length of 4096 and benefit from the efficient sequence-wise parallelism optimizations of FlashAttention-2.
Training is performed in BFloat16 mixed precision while keeping all-reduce operations in FP32.found it beneficial to add a z-loss regularization term on the softmax normalizer,, to mitigate instability stemming from output logits divergence.
While it did not hurt performance in our ablations, the improvements to stability were minimal.
Accordingly, it was not applied in the final run.
We employ a standard AdamW optimizer with the following hyperparameters:.
Sec.2.5offers details regarding the custom learning rate scheduler that we applied.2.2DataModel performance is affected by pre-training data design decisions, including both the source selection and the sampling weights. Our approach is close to that of: the majority of our training data consists of sources utilized in the training of other LLMs, such as RefinedWeb, subsets of the Pile, RedPajamaand the Stack. We supplement these with OpenWebText, OpenWebMath, and parts of CulturaX. While inspecting randomly sampled documents from the mC4 subset of CulturaX, we encountered frequent HTML boilerplate and decided to remove this portion entirely, finally keeping only the OSCAR subset.
Additionally, we incorporate FanFics333https://huggingface.co/datasets/atom-in-the-universe/fanfics-10k-50k, a subset of 50k documents fromfanfiction.netselected by lowest perplexity scores according to a KenLM444https://huggingface.co/edugp/kenlm. Finally, following, we restructure several raw datasets into rich fixed forms for downstream tasks such as summarization, question-answering, sentiment analysis, etc., and add instruction data from, the aggregate of which we collectively call Restruct-v1. The list of sources used in Restruct-v1 is made available in Tab.11.
Stable LM’s training set is comprised entirely of open-source datasets compatible with commercial usage, most of which are hosted on the Hugging Face Hub.
The only exception to the latter aspect (HFH) , Restruct-v1, can easily be reproduced by following the approaches and prompt templates provided by.Carefully selecting the mixture proportions of the various data domains is critical, particularly with respect to the amount of non-English and code data.
We trained several models on different mixes and evaluated them on downstream benchmarks to pick our final dataset.
The full set of ablations is available in AppendixA, together with rationales for selecting these particular mixes.
Based on the results of the ablations, we trained our model with the mix shown in Table1, which accounts for around 2 trillion tokens.
Note that it includes multilingual data in German (DE) , Spanish (ES) , French (FR) , Italian (IT) , Dutch (NL) , and Portuguese (PT) . The split of our dataset across different domains is visualized in Fig.1.ParametersHidden SizeLayersHeadsSequence Length1,644,417,024204824324096Table 2:Stable LM 2 model architecture.Data Parallel DegreeMicro Batch SizeGradient Accumulation StepsActivation Checkpointing51222disabledTable 3:Stable LM 2 training configuration.2.3TokenizerWe useArcade100k, a BPE tokenizer extended from OpenAI’stiktoken.cl100k_baseto include special tokens for codeand digit-split handling. The vocabulary consists of 100,289 tokens and is padded to the nearest multiple of 64 (100,352)  during training to meet the recommended Tensor Cores alignment on NVIDIA A100 devices. In preliminary experiments, we did not observe statistically significant deviations in downstream natural language performance tasks when compared against a hyperparameter matching model trained with the smaller GPT-NeoX tokenizer. Increased compression rates for code and non-English languages influenced our decision to useArcade100kover existing tokenizers.2.4Architecture and Training LayoutThe model is a causal decoder-only transformer similar in design to the LLaMA architecture. Table3shows some of the key architectural details.
In particular, the main differences with respect to LLaMA are the following:•Position Embeddings. Rotary Position Embeddingsapplied to the firstof head embedding dimensions for improved throughput following.•Normalization. LayerNormwith learned bias terms as opposed to RMSNorm.•Biases. We remove all bias terms from the feed-forward networks and multi-head self-attention layers, except for the biases of the key, query, and value projections.Stable LM 2 1.6B is trained on 64 Amazon P4d instances comprising 512 NVIDIA A100 (40GB HBM2)  GPUs. The size of our model, together with ZeRO stage 1 distributed optimization, eliminates the need for model sharding. Still, different triplets ofmicro batch size,gradient accumulation steps, andactivation checkpointinggranularity lead to different speed metrics.
Following the recommendations in, we obtain our final configuration by finding the micro-batch size that allows us to completely remove activation checkpointing. We then determine the gradient accumulation steps based on our target batch size anddata paralleldegree. We employ a batch size oftokens, based on the observations in AppendixD. With the setup in Table3, we achieve170 TFLOPs/s per device, ormodel flops utilization (MFU) . A higher hardware utilization of200 TFLOPs/s (MFU)  can be trivially achieved by decreasing the degree of data parallelism and correspondingly increasing the number of gradient accumulation steps, at the cost of an increased iteration time.2.5Learning Rate SchedulerWe propose a new learning rate scheduler.
It consists of multiple stages and is designed to favor flexibility in terms of continued pre-training.
We begin by linearly increasing the learning rate to its max value ofover 9720 steps. Thiswarm upstage is followed by the main training phase in which the learning rate is decreased according to Eq.1:(1) whereandare respectively the minimum and maximum learning rates,is the current step, andis the total number of steps. The free parametersandhave been arbitrarily chosen to enforce the continuity of the scheduler and its derivative at.
We finalize training by linearly cooling the learning rate down to zero over 80k steps, which corresponds to around 670B tokens.
The full scheduler is illustrated in Fig.2; further details and ablations can be found in AppendixB.Figure 2:Multi-stage infinite scheduler proposed and applied in this work.
body
Stable LM 2 1.6B Technical Report
2Model Pre-Training
2402.17834
3Fine-tuning and AlignmentFollowing pre-training, we further develop the conversational skills of our model via a fine-tuning stage that consists of three main steps: supervised fine-tuning (SFT) , direct preference optimization (DPO) , and self-knowledge learning.
Importantly, we do not use multilingual data at this stage.
We now describe each of them in detail, and in Section4we report the results after all three steps.ModelSizeAvgARCHSMMLUTQAWinoGSMphi-2†2.8B61.361.175.158.144.574.454.8stablelm-2-zephyr-1_6b†1.6B49.743.369.341.845.663.634.8phi-1_5†1.3B47.752.963.843.940.972.212.4stablelm-3b-4e1t2.7B46.646.675.945.237.271.23.3Qwen-1.5-1.8B1.8B46.637.961.446.739.460.333.6gemma-2b2.5B46.548.571.741.733.166.817.4stablelm-2-1_6b1.6B45.343.370.538.936.864.617.4gemma-2b-it†2.5B42.743.962.737.645.860.95.5open_llama_3b3B40.339.971.627.134.867.00.9falcon-rw-1b1.3B37.135.163.625.335.962.00.5TinyLLama-1.1B1.1B36.433.960.326.037.359.51.4Table 4:Comparison of Open LLM leaderboard evals. † denotes aligned models. Note that in this table, as well as in Tab.5, we marked the Phi series of modelsasaligned. While we acknowledge that they may have only been pre-trained, by the nature of the training data used, and the self-disclaimed intended use for QA and chat formats, we believe this to be a fair statement.ModelAllENDEESFRITNLPTstablelm-3b-4e1t41.750.939.740.241.241.136.741.9stablelm-2-zephyr-1_6b†41.549.540.240.039.839.938.842.0stablelm-2-1_6b40.548.739.139.039.338.837.841.2gemma-2b39.848.638.338.738.738.435.140.5gemma-2b-it†38.249.436.838.037.535.532.138.1open_llama_3b37.547.335.236.437.637.132.236.8Qwen-1.5-1.8B-Chat35.546.233.235.134.333.230.935.7Qwen-1.5-1.8B34.846.331.834.034.232.829.735.0TinyLLama-1.1B34.842.433.033.834.733.531.035.0phi-2†34.655.829.034.332.929.927.133.4falcon-rw-1b29.942.227.428.628.328.025.929.1phi-1_5†29.747.125.328.726.827.224.129.0Table 5:Multilingual evaluations. As in the previous table, † denotes aligned models.3.1SFTThe first step is supervised fine-tuning.
We fine-tune the pre-trained model on a number of instruction datasets publicly available on the Hugging Face Hub.
In particular, we use the followingconversationaldatasets: UltraChat, WizardLM, SlimOrca, ShareGPT, Capybara, Deita, and MetaMathQA.
We removed any samples that exceeded eight turns, leading to a total of 826,938 samples.DatasetTypeSourceNumber of SamplesUltraChatSFTHuggingFaceH4/ultrachat_200k194,409WizardLMSFTWizardLM/WizardLM_evol_instruct_V2_196k80,662SlimOrcaSFTOpen-Orca/SlimOrca-Dedup143,789ShareGPTSFTopenchat/openchat_sharegpt4_dataset3,509CapybaraSFTLDJnr/Capybara7,291DeitaSFThkust-nlp/deita-10k-v02,860MetaMathQASFTmeta-math/MetaMathQA394,418UltrafeedbackDPOargilla/ultrafeedback-binarized-preferences63,169Orca PairsDPOIntel/orca_dpo_pairs12,859Table 6:Fine-tuning datasetsWe train our SFT models for three epochs using a cosine learning rate scheduler. A warm-up phase ofof the training duration is employed before reaching the peak learning rate of. We set the global batch size to 512 sequences and pack inputs into sequences of up to 4096 tokens in length.3.2DPODirect Preference Optimization (DPO) has been a fundamental tool in recent strong models such as Zephyr-7B, Neural-Chat-7B, and Tulu-2-DPO-70B.
Accordingly, after applying SFT, we aligned the resulting model via DPO. We use two datasets at this stage: UltraFeedbackand Intel Orca Pairs. We filter the datasets by removing pairs with tied ranking, pairs with duplicated content, and pairs in which the score for the chosen response is less than eight out of ten. We train the model with DPO following the Zephyr recipeand borrowing most of its hyperparameters, except for, which we lower to, and the learning rate, which we lower to, both of which helped improve the stability of training and final performance. This stage of training is performed using the Alignment Handbook.Figure 3:Stable LM 2 1.6B shows competitive performance, matching or even surpassing significantly larger models on MT-Bench.3.3Self-KnowledgeThe model after the output of the DPOstage does not have knowledge about who created it, or even what limitations a language model has. To remedy this, we were inspired by the data generation method of Reinforcement Learning from Contrast Distillation (RLCD) and the training method of Conditioned Reinforcement Learning Fine-Tuning (C-RLFT) , which we apply to self-knowledge training.To generate the initial prompts, we use the base model to generate 10k random first messages to a language model with no duplicates. To generate continuations, we use a few-shot prompt with self-knowledge in the previous chat turns as a positive completion. For the negative prompt, we simply sample from the prompt with no additional prompting or few-shot turns.We train with unpacked examples for 6 epochs using a batch size of 256, a warmup stage for 100 steps to a maximum LR of 3e-6 followed by a cosine decay to zero. The positive prompts are trained in the same way as the SFT step, while the negative prompts are trained with a negative token instead of the Assistant token.
body
Stable LM 2 1.6B Technical Report
3Fine-tuning and Alignment
2402.17834
4Experimental Results and BenchmarksThis section presents the main experimental results for Stable LM 2 1.6B.
We compare with similarly-sized open-source models showing strong improvements on various tasks, including multilingual capabilities in Spanish, German, French, Italian, Portuguese, and Dutch.
For context, we also provide comparisons with much larger models.
We split our experiments into three sets: few- and zero-shot evaluations in English (as commonly done in the Hugging Face Open LLM leaderboard) , multilingual evaluations, and conversational evaluations.4.1Few-shot and Zero-shot EvaluationsFirst, we assess the 0-shot and few-shot capabilities of Stable LM 2 by evaluating our model over popular benchmarks and comparing results against similarly sized open-source pre-trained models.
Table4presents model evaluations in English.
Our results cover the 6 benchmarks from the Open LLM Leaderboard () : ARC-Challenge 25-shot(ARC) , HellaSwag 10-shot(HS) , MMLU 5-shot(MMLU) , TruthfulQA 0-shot(TQA) , WinoGrande 5-shot(Wino)  and GSM8K 5-shot(GSM) .
Also, as Stable LM 2 is a general-purpose foundational model, we further assess natural language understanding capabilities by evaluating English and machine-translated versions of LAMBADA.
All evaluations are performed with the Language Model Evaluation Harness framework555https://github.com/Stability-AI/lm-evaluation-harness/tree/stablelm-2/multilingual-bench.
As shown in Table4, Stable LM 2 1.6B (stablelm-2-1-6b)  outperforms other base models by a significant margin.
Similarly, the instruction-tuned version (stablelm-2-1-6b-dpo)  improves on Microsoft’s Phi-1.5 by two average points while lagging behind the larger Phi-2.0 on few-shot accuracy.
Performance versus Google’s Gemma 2B (2.5B parameters)  is also remarkable.ModelSizeMT-BenchMistral-7B-Instruct-v0.27B7.61Llama2-Chat70B6.86stablelm-zephyr-3b3B6.64MPT-30B-Chat30B6.39stablelm-2-zephyr-1.6b1.6B5.42Qwen-1.5-1.8B-Chat1.8B5.29gemma-2b-it2.5B5.19Falcon-40B-Instruct40B5.17dolphin-2.6-phi-22.7B4.93phi-22.7B4.29TinyLlama-1.1B-Chat-v1.01.1B3.46Table 7:MT-Bench results4.2Multilingual EvaluationsWe assess knowledge and reasoning in the multilingual setting for non-English languages seen during pre-training by evaluating on ChatGPT-translated versions of ARC, HS, TQA, and MMLU () . In addition, we test next-word prediction capabilities using the machine-translated LAMBADA datasets from.
After manual inspection by native speakers, we have deemed existing machine translations666https://huggingface.co/datasets/EleutherAI/lambada_openaitoo noisy to draw accurate performance signals from. We instead evaluate multilingual next-word prediction with new translations which are made available for researchers777https://huggingface.co/datasets/marcob/lambada_multilingual.The zero-shot results are presented in Tab.5and highlight Stable LM 2’s superior performance compared to models even twice its size.4.3MT Benchmark EvaluationsFinally, we also test the conversational skills of our model on the popular multi-turn benchmarkMT-Bench. The results are provided in Fig.3and Tab.7. While lagging behind much more powerful models such as Mistral 7B Instruct v0.2 (more than 4x the size of Stable LM 2) , our model delivers better chat performance and beats both Phi-2, Gemma 2B and TinyLLaMA 1.1B by a wide margin despite the larger size of the former.
body
Stable LM 2 1.6B Technical Report
4Experimental Results and Benchmarks
2402.17834
5Inference and QuantizationThis model represents a substantial leap towards making advanced generation capabilities available directly on-device without the computational overhead of larger models. We believe this model strikes a great balance between remarkable efficiency and effectiveness in inference tasks when paired with inference frameworks and quantization methods. As part of our release, we provide quantized weights ofstablelm-2-1-6bsupported on popular inference libraries such as llama.cpp888https://github.com/ggerganov/llama.cpp, Apple MLXand Intel OpenVINO999https://github.com/openvinotoolkit/openvino.5.1QuantizationWe make available quantization files for various models and formats to support easier
integration with different inference frameworks, including:•Two 4-bit quantized models: Q4_0, Q4_1 and a 5-bit quantized model: Q5_K_M GGUF•INT4 for OpenVINO quantized with Intel’s Neural Network Compression Framework (NNCF) •INT4 for MLX quantized with MLXThese quantization files can be found in the model’sHugging Face repositoryfor the
convenience of developers and researchers working with our models. We aim to facilitate smoother deployment experiences across various deep-learning framework
ecosystems by offering a range of quantization formats.5.2ThroughputIn Tab.8we provide throughput numbers obtained from our model running on consumer-grade devices and the system environments utilized.
Our initial runs showcase that when using a lower precision, we are able to achieve almost 2x performance in throughput.
Note that these figures are provided as a reference, and they are not the result of rigorous benchmarking but are rather intended to give users a practical insight into what they can expect in terms of performance on commonly used devices. Likewise, as lower precision quantization is expected to reduce the model’s performance, we encourage researchers and developers to assess the potential degradation in real-world scenarios.FrameworkCPUPrecisionThroughput (Tok/s) Power consumption (W) MLXM2FP16716Mac Mini(8GB) INT412711GGUFM2 Pro MaxFP1646142023 MacBook Pro(16GB) INT49914Table 8:Throughput and power usage on various devices using different quantization frameworks. We employ a batch size of 1 for all benchmarks. INT4 numbers for GGUF were collected using Q4_0 quantization.
body
Stable LM 2 1.6B Technical Report
5Inference and Quantization
2402.17834
6Future WorkThere is a number of research avenues we would like to explore to further improve the model:1.Data. In this work, we focused on publicly available data. In particular, most of the data comes from web-crawled content, as is common for most models. This data is known to contain many low-quality documentsthat can potentially harm training.
We believe there is significant potential in smart filtering, re-writing, and synthetic data generation with strong models.2.Hallucination Mitigation.
Language models are prone to generating incorrect or misleading information, and small language models are even more prone to doing so. Finding reliable ways to detect hallucinations in these models will unlock new use cases in areas that are sensitive to hallucinations.3.Long Contexts and Retrieval.
The ability to retrieve information across long context windows is essential for applications such as chat models or dataset integration. Accordingly, in AppendixC, we explore the current capabilities and limitations of StableLM2 1.6B on the Needle-in-the-Haystack task. Going forward we plan to further build upon this work as well as to extend our models to context lengths beyond 4k.4.Conditional Computation. Small models are often capacity-constrained – that is, with the current training approaches, they lack the capacity to process and exploit all of the training data. Recently, ideas such as Mixture of Experts have been successfully applied to take a dense model and extend it to contain more parameters that are selectively applied to certain inputs (for instance, via sparse upcycling) .
Importantly, if each token selects only one expert, the overall inference FLOPs do not significantly change.
Applying this to the Stable LM 2 1.6B model is a natural extension we will investigate.
body
Stable LM 2 1.6B Technical Report
6Future Work
2402.17834
7Environmental and Societal Impact7.1Carbon FootprintThe training of Stable LM 2 has consumed energy with associated carbon dioxide emissions. In line withwe report our carbon footprint based on the formulawhere the Power Usage Effectiveness is set to 1.1. We trained Stable LM 2 forGPU-hours, giving a total power consumption of 30MWh considering our average power usage. The tons of emitted carbon tCO2eq can be estimated using the US national average carbon intensity factor of 0.385 kg CO2eq/KWh, leading to a final figure of 11 tCO2eq.7.2Societal impactStability AI is committed to releasing open models to help improve access to foundational AI technology. Open access to model weights enables researchers to inspect models for suitability and vulnerabilities, test the effectiveness of different optimization strategies, and correct for biases observed in the model. To that end, this model is released under an open non-commercial license. However, open release can introduce challenges in assessing the societal impact of a model. For example, Stability AI does not have direct visibility into downstream applications of Stable LM 2 1.6B, the distribution of applications by sector, or the distribution of model usage by geography. Since the model is released with a noncommercial license, we expect a limited number of applications outside of fine-tuning or evaluation interfaces and a limited number of third parties affected by the model.We will continue to monitor openly released fine-tuned models to understand the extent of fine-tuning research or development activity that uses Stable LM 2 1.6B as a base model, including the evaluation results of these derivative models.
body
Stable LM 2 1.6B Technical Report
7Environmental and Societal Impact
2402.17834
8ConclusionIn this report, we introduced Stable LM 2 1.6B, a compact decoder-only language model trained on multilingual datasets. It fluidly handles up to seven languages: English, Spanish, German, Italian, French, Portuguese, and Dutch.
To ensure that the community can reproduce our run, we detail all datasets used during training –with theexactdata mix– and our newly designed learning rate schedule.
We also conduct extensive model evaluations and comparisons with other similarly-sized models, demonstrating Stable LM 2 1.6B’s exceptional performance.
Finally, we profile the model on common edge computing architectures.
We hope the current report contributes to the improvement and further research on small language models.
body
Stable LM 2 1.6B Technical Report
8Conclusion
2402.17834
AcknowledgmentsWe thank our awesome MLOps team members, particularly Richard Vencu, for the support provided.
We also thank Christian Laforte, Cedric Wagrez, and Jerry Chi for their feedback, useful ideas, and comments.
conclusion
Stable LM 2 1.6B Technical Report
Acknowledgments