<!DOCTYPE html><html lang="en">
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<title>[2402.14905] MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases</title><meta property="og:description" content="This paper addresses the growing need for efficient large language models (LLMs) on mobile devices, driven by increasing cloud costs and latency concerns. We focus on designing top-quality LLMs with fewer than a billio…">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="twitter:card" content="summary">
<meta name="twitter:title" content="MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases">
<meta name="twitter:image:src" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta name="twitter:image:alt" content="ar5iv logo">
<meta property="og:title" content="MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases">
<meta property="og:site_name" content="ar5iv">
<meta property="og:image" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta property="og:type" content="article">
<meta property="og:url" content="https://ar5iv.labs.arxiv.org/html/2402.14905">

<!--Generated on Tue Mar  5 18:33:22 2024 by LaTeXML (version 0.8.8) http://dlmf.nist.gov/LaTeXML/.-->
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
<meta name="keywords" lang="en" content="Machine Learning,  ICML">

<script>
  function detectColorScheme(){
    var theme="light";
    var current_theme = localStorage.getItem("ar5iv_theme");
    if(current_theme){
      if(current_theme == "dark"){
        theme = "dark";
      } }
    else if(!window.matchMedia) { return false; }
    else if(window.matchMedia("(prefers-color-scheme: dark)").matches) {
      theme = "dark"; }
    if (theme=="dark") {
      document.documentElement.setAttribute("data-theme", "dark");
    } else {
      document.documentElement.setAttribute("data-theme", "light"); } }

  detectColorScheme();

  function toggleColorScheme(){
    var current_theme = localStorage.getItem("ar5iv_theme");
    if (current_theme) {
      if (current_theme == "light") {
        localStorage.setItem("ar5iv_theme", "dark"); }
      else {
        localStorage.setItem("ar5iv_theme", "light"); } }
    else {
        localStorage.setItem("ar5iv_theme", "dark"); }
    detectColorScheme(); }
</script>
<link media="all" rel="stylesheet" href="/assets/ar5iv-fonts.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv-site.0.2.2.css">
</head>
<body>
<div class="ltx_page_main">
<div class="ltx_page_content">
<article class="ltx_document ltx_pruned_first">
<h1 class="ltx_title ltx_title_document">MobileLLM: Optimizing Sub-billion Parameter Language Models 
<br class="ltx_break">for On-Device Use Cases</h1>
<div class="ltx_authors">
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Zechun Liu
</span></span>
<span class="ltx_author_before">  </span><span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Changsheng Zhao
</span></span>
<span class="ltx_author_before">  </span><span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Forrest Iandola
</span></span>
<span class="ltx_author_before">  </span><span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Chen Lai
</span></span>
<span class="ltx_author_before">  </span><span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Yuandong Tian
</span></span>
<span class="ltx_author_before">  </span><span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Igor Fedorov
</span></span>
<span class="ltx_author_before">  </span><span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Yunyang Xiong
</span></span>
<span class="ltx_author_before">  </span><span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Ernie Chang
</span></span>
<span class="ltx_author_before">  </span><span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Yangyang Shi
</span></span>
<span class="ltx_author_before">  </span><span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Raghuraman Krishnamoorthi
</span></span>
<span class="ltx_author_before">  </span><span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Liangzhen Lai
</span></span>
<span class="ltx_author_before">  </span><span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Vikas Chandra
</span></span>
</div>

<div class="ltx_abstract">
<h6 class="ltx_title ltx_title_abstract">Abstract</h6>
<p id="id4.4" class="ltx_p">This paper addresses the growing need for efficient large language models (LLMs) on mobile devices, driven by increasing cloud costs and latency concerns. We focus on designing top-quality LLMs with fewer than a billion parameters, a practical choice for mobile deployment.
Contrary to prevailing belief emphasizing the pivotal role of data and parameter quantity in determining model quality, our investigation underscores the significance of model architecture for sub-billion scale LLMs. Leveraging deep and thin architectures, coupled with embedding sharing and grouped-query attention mechanisms, we establish a strong baseline network denoted as <span id="id4.4.1" class="ltx_text ltx_markedasmath ltx_font_typewriter">MobileLLM</span>, which attains a remarkable 2.7%/4.3% accuracy boost over preceding 125M/350M state-of-the-art models. Additionally, we propose an immediate block-wise weight sharing approach with no increase in model size and only marginal latency overhead. The resultant models, denoted as <span id="id4.4.2" class="ltx_text ltx_markedasmath ltx_font_typewriter">MobileLLM-LS</span>, demonstrate a further accuracy enhancement of 0.7%/0.8% than <span id="id4.4.3" class="ltx_text ltx_markedasmath ltx_font_typewriter">MobileLLM</span> 125M/350M.
Moreover, <span id="id4.4.4" class="ltx_text ltx_markedasmath ltx_font_typewriter">MobileLLM</span> model family shows significant improvements compared to previous sub-billion models on chat benchmarks, and demonstrates close correctness to LLaMA-v2 7B in API calling tasks, highlighting the capability of small models for common on-device use cases.</p>
</div>
<div class="ltx_keywords">Machine Learning, ICML
</div>
<div id="p2" class="ltx_para">
<br class="ltx_break">
</div>
<section id="S1" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">1 </span>Introduction</h2>

<figure id="S1.F1" class="ltx_figure"><img src="/html/2402.14905/assets/x1.png" id="S1.F1.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="348" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S1.F1.2.1.1" class="ltx_text" style="font-size:90%;">Figure 1</span>: </span><span id="S1.F1.3.2" class="ltx_text" style="font-size:90%;">Average score on zero-shot common sense tasks for LLMs smaller than 1B parameters. Each bubble’s area is proportional to the model size of a variant in a model family. The results of previous methods were evaluated using open-source Hugging Face models to ensure consistent evaluation procedures. The full list of tasks is in Table <a href="#S3.SS2" title="3.2 Main Results ‣ 3 Experiments ‣ 2.3 Layer Sharing ‣ 2 Improving Sub-billion Scale LLM Design ‣ MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.2</span></a>.</span></figcaption>
</figure>
<div id="S1.p1" class="ltx_para">
<p id="S1.p1.1" class="ltx_p">Large language models (LLMs) are permeating various facets of human life, influencing not only the way people communicate and work but also shaping everyday entertainment experiences. Prominent examples of contemporary LLM products, such as ChatGPT and Perplexity AI, primarily operate in cloud environments. Leading models such as ChatGPT4 exceed 1 trillion parameters.<span id="footnote1" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">1</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">1</sup><span class="ltx_tag ltx_tag_note">1</span>https://the-decoder.com/gpt-4-has-a-trillion-parameters</span></span></span> However, envisioning a future scenario characterized by widespread human reliance of LLMs in both front-end conversational interfaces and back-end operations like recommendation system, equating to <math id="S1.p1.1.m1.1" class="ltx_Math" alttext="\sim" display="inline"><semantics id="S1.p1.1.m1.1a"><mo id="S1.p1.1.m1.1.1" xref="S1.p1.1.m1.1.1.cmml">∼</mo><annotation-xml encoding="MathML-Content" id="S1.p1.1.m1.1b"><csymbol cd="latexml" id="S1.p1.1.m1.1.1.cmml" xref="S1.p1.1.m1.1.1">similar-to</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S1.p1.1.m1.1c">\sim</annotation></semantics></math>5% of individuals’ daily time. In this hypothetical scenario, employing GPT-4 at processing rate of 50 tokens/s entails the deployment of around one hundred million H100 GPUs<span id="footnote2" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">2</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">2</sup><span class="ltx_tag ltx_tag_note">2</span>Detailed calculation can be found in appendix.</span></span></span>, each capable of 60 TFLOPs/s<span id="footnote3" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">3</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">3</sup><span class="ltx_tag ltx_tag_note">3</span>https://www.nvidia.com/en-us/data-center/h100/</span></span></span>. This computation scale, excluding communication and data transmission overhead, is on par with 160 Meta-scale companies<span id="footnote4" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">4</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">4</sup><span class="ltx_tag ltx_tag_note">4</span>https://twitter.com/soumithchintala/status/1748074223187173724</span></span></span>. The ensuing energy consumption and carbon dioxide emissions would present staggering environmental challenges. Consequently, it is imperative that we downsize LLMs.</p>
</div>
<figure id="S1.F2" class="ltx_figure"><img src="/html/2402.14905/assets/x2.png" id="S1.F2.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="206" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S1.F2.2.1.1" class="ltx_text" style="font-size:90%;">Figure 2</span>: </span><span id="S1.F2.3.2" class="ltx_text" style="font-size:90%;">Memory hierarchy in prevalent mobile devices. Despite adequate Flash storage, the operational memory for executing high-speed applications predominantly resides in DRAM, typically constrained to 6-12 GB.</span></figcaption>
</figure>
<div id="S1.p2" class="ltx_para">
<p id="S1.p2.1" class="ltx_p">Furthermore, considerations of <span id="S1.p2.1.1" class="ltx_text ltx_font_italic">portability</span> and <span id="S1.p2.1.2" class="ltx_text ltx_font_italic">computational cost</span> propel the necessity to deploy LLMs on smartphones and mobile devices. In the current landscape of mobile technology, integrating a LLM like the LLaMA-v2 7B <cite class="ltx_cite ltx_citemacro_citep">(Touvron et al., <a href="#bib.bib47" title="" class="ltx_ref">2023b</a>)</cite> with 8-bit weights proves prohibitively expensive due to limitations in main-memory (DRAM) capacity source. A prevalent memory hierarchy in mobile devices is depicted in Figure <a href="#S1.F2" title="Figure 2 ‣ 1 Introduction ‣ MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>.
With DRAM capacities ranging from 6 GB for the iPhone 15 and 12 GB for the Google Pixel 8 Pro <cite class="ltx_cite ltx_citemacro_citep">(Hristov, <a href="#bib.bib19" title="" class="ltx_ref">2022</a>; Google, <a href="#bib.bib15" title="" class="ltx_ref">2023</a>)</cite>, a mobile app should not exceed 10% of the DRAM, since DRAM is shared with the operating system and other applications <cite class="ltx_cite ltx_citemacro_citep">(Malladi et al., <a href="#bib.bib32" title="" class="ltx_ref">2012</a>)</cite>. This motivates deploying sub-billion parameter LLMs. Additionally, factoring in LLM energy consumption (0.1 J/token per billion in model parameters <cite class="ltx_cite ltx_citemacro_citep">(Han et al., <a href="#bib.bib17" title="" class="ltx_ref">2016</a>; Malladi et al., <a href="#bib.bib32" title="" class="ltx_ref">2012</a>)</cite>), a 7B-parameter LLM consumes 0.7 J/token. A fully charged iPhone, with approximately 50kJ of energy, can sustain this model in conversation for less than 2 hours at a rate of 10 tokens/s, with every 64 tokens draining 0.2% of the battery.</p>
</div>
<div id="S1.p3" class="ltx_para">
<p id="S1.p3.1" class="ltx_p">These demands converge on a singular imperative: the adoption of compact models for on-device execution.
By utilizing a sub-billion model, such as a 350M 8-bit model consuming only 0.035 J/token, an iPhone can support conversational use an entire day.
Moreover, the decoding speed can be significantly enhanced, as exemplified by the benchmark results of the 125M model, capable of operating at of 50 tokens/s, comparing to the state-of-the-art iPhone App MLC Chat utilizing the LLaMA 7B model at 3<math id="S1.p3.1.m1.1" class="ltx_Math" alttext="\sim" display="inline"><semantics id="S1.p3.1.m1.1a"><mo id="S1.p3.1.m1.1.1" xref="S1.p3.1.m1.1.1.cmml">∼</mo><annotation-xml encoding="MathML-Content" id="S1.p3.1.m1.1b"><csymbol cd="latexml" id="S1.p3.1.m1.1.1.cmml" xref="S1.p3.1.m1.1.1">similar-to</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S1.p3.1.m1.1c">\sim</annotation></semantics></math>6 tokens/second<span id="footnote5" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">5</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">5</sup><span class="ltx_tag ltx_tag_note">5</span>https://llm.mlc.ai</span></span></span>. In light of these considerations, our paper is motivated by the design and implementation of LLMs with parameters less than 1 billion.</p>
</div>
<div id="S1.p4" class="ltx_para">
<p id="S1.p4.1" class="ltx_p">We make following contributions to build the most accurate LLMs to date under 1 billion parameters.</p>
<ul id="S1.I1" class="ltx_itemize">
<li id="S1.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S1.I1.i1.p1" class="ltx_para">
<p id="S1.I1.i1.p1.1" class="ltx_p">Contradictory to the scaling law <cite class="ltx_cite ltx_citemacro_citep">(Kaplan et al., <a href="#bib.bib22" title="" class="ltx_ref">2020</a>)</cite>, we demonstrate that depth is more important than width for small LLMs. A deep-and-thin model structure excels in capturing abstract concepts, resulting in superior final performance.</p>
</div>
</li>
<li id="S1.I1.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S1.I1.i2.p1" class="ltx_para">
<p id="S1.I1.i2.p1.1" class="ltx_p">We revisit embedding sharing methods <cite class="ltx_cite ltx_citemacro_citep">(Zhang et al., <a href="#bib.bib57" title="" class="ltx_ref">2022</a>)</cite> and implement grouped query attention <cite class="ltx_cite ltx_citemacro_citep">(Ainslie et al., <a href="#bib.bib1" title="" class="ltx_ref">2023</a>)</cite> in small LLMs to maximize weight utilization.</p>
</div>
</li>
<li id="S1.I1.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S1.I1.i3.p1" class="ltx_para">
<p id="S1.I1.i3.p1.1" class="ltx_p">We propose immediate block-wise weight sharing. In scenarios where memory movement is the latency bottleneck, weight sharing between two adjacent blocks avoids weight movement, requiring only computing the block twice and incurring minimal latency overhead.</p>
</div>
</li>
<li id="S1.I1.i4" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S1.I1.i4.p1" class="ltx_para">
<p id="S1.I1.i4.p1.2" class="ltx_p">We propose a new family of models, <span id="S1.I1.i4.p1.2.1" class="ltx_text ltx_markedasmath ltx_font_typewriter">MobileLLM</span>, showcasing SOTA performance. In a suite of zero-shot tasks, <span id="S1.I1.i4.p1.2.2" class="ltx_text ltx_markedasmath ltx_font_typewriter">MobileLLM</span> outperforms the previous SOTA 125M/350M models by 2.7%/4.3%.</p>
</div>
</li>
<li id="S1.I1.i5" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S1.I1.i5.p1" class="ltx_para">
<p id="S1.I1.i5.p1.2" class="ltx_p">In downstream tasks, such as Chat and API calling, <span id="S1.I1.i5.p1.2.1" class="ltx_text ltx_markedasmath ltx_font_typewriter">MobileLLM</span> model family significantly outperforms equivalently-sized models. In an API calling task, <span id="S1.I1.i5.p1.2.2" class="ltx_text ltx_markedasmath ltx_font_typewriter">MobileLLM</span> even achieves comparable exact-match score as considerably larger LLaMA-v2 7B model.</p>
</div>
</li>
</ul>
</div>
</section>
<section id="S2" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">2 </span>Improving Sub-billion Scale LLM Design</h2>

<div id="S2.p1" class="ltx_para">
<p id="S2.p1.2" class="ltx_p">In this section, we present the evolutionary path from a baseline sub-billion parameter model to the new state-of-the-art models (Figure <a href="#S2.F3" title="Figure 3 ‣ 2.1 Training Setup ‣ 2 Improving Sub-billion Scale LLM Design ‣ MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>). We explore both 125M and 350M models and demonstrate consistent improvements in both cases. For on-device use cases where model size is a major constraint, how to allocate the limited weight parameters effectively becomes more critical than ever. We first propose a strong baseline model named <span id="S2.p1.2.1" class="ltx_text ltx_markedasmath ltx_font_typewriter">MobileLLM</span> by testing out four model design techniques that are beneficial for sub-billion scale LLMs, including (1) adopting SwiGLU FFN  <cite class="ltx_cite ltx_citemacro_citep">(Dauphin et al., <a href="#bib.bib9" title="" class="ltx_ref">2017</a>)</cite>; (2) forcing <span id="S2.p1.2.2" class="ltx_text ltx_font_italic">lanky</span> (deep and thin) architectures (3) revisiting embedding sharing method <cite class="ltx_cite ltx_citemacro_citep">(Zhang et al., <a href="#bib.bib57" title="" class="ltx_ref">2022</a>)</cite> (4) utilizing grouped query attention <cite class="ltx_cite ltx_citemacro_citep">(Chowdhery et al., <a href="#bib.bib5" title="" class="ltx_ref">2023</a>)</cite>. Then we develop an immediate block-wise layer sharing method that further boosts accuracy without incurring any additional memory overhead and with only slight latency overhead in the memory-bounded LM decoding process. We denote our model with layer sharing as <span id="S2.p1.2.3" class="ltx_text ltx_markedasmath ltx_font_typewriter">MobileLLM-LS</span>.</p>
</div>
<section id="S2.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.1 </span>Training Setup</h3>

<div id="S2.SS1.p1" class="ltx_para">
<p id="S2.SS1.p1.1" class="ltx_p">Our experiments are conducted on 32 A100 GPUs, with each GPU having a batch size of 32. We performed exploratory experiments with 120k iterations on 0.25T tokens. Subsequently, the top models reported in Table <a href="#S3.SS2" title="3.2 Main Results ‣ 3 Experiments ‣ 2.3 Layer Sharing ‣ 2 Improving Sub-billion Scale LLM Design ‣ MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.2</span></a> and Table <a href="#S3.SS2" title="3.2 Main Results ‣ 3 Experiments ‣ 2.3 Layer Sharing ‣ 2 Improving Sub-billion Scale LLM Design ‣ MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.2</span></a>, are trained with 480k iterations on 1T tokens.</p>
</div>
<div id="S2.SS1.p2" class="ltx_para">
<p id="S2.SS1.p2.1" class="ltx_p">We evaluate the pre-trained model on zero-shot common sense reasoning tasks, including ARC-easy, ARC-challenge <cite class="ltx_cite ltx_citemacro_citep">(Clark et al., <a href="#bib.bib7" title="" class="ltx_ref">2018</a>)</cite>, BoolQ <cite class="ltx_cite ltx_citemacro_citep">(Clark et al., <a href="#bib.bib6" title="" class="ltx_ref">2019</a>)</cite>, PIQA <cite class="ltx_cite ltx_citemacro_citep">(Bisk et al., <a href="#bib.bib3" title="" class="ltx_ref">2020</a>)</cite>, SIQA <cite class="ltx_cite ltx_citemacro_citep">(Sap et al., <a href="#bib.bib38" title="" class="ltx_ref">2019</a>)</cite>, HellaSwag <cite class="ltx_cite ltx_citemacro_citep">(Zellers et al., <a href="#bib.bib56" title="" class="ltx_ref">2019</a>)</cite>, OBQA <cite class="ltx_cite ltx_citemacro_citep">(Mihaylov et al., <a href="#bib.bib33" title="" class="ltx_ref">2018</a>)</cite>, WinoGrande <cite class="ltx_cite ltx_citemacro_citep">(Sakaguchi et al., <a href="#bib.bib37" title="" class="ltx_ref">2021</a>)</cite>, as well as question answering and reading comprehension tasks using TQA <cite class="ltx_cite ltx_citemacro_citep">(Joshi et al., <a href="#bib.bib21" title="" class="ltx_ref">2017</a>)</cite> and RACE dataset <cite class="ltx_cite ltx_citemacro_citep">(Lai et al., <a href="#bib.bib27" title="" class="ltx_ref">2017</a>)</cite>.</p>
</div>
<figure id="S2.F3" class="ltx_figure"><img src="/html/2402.14905/assets/x3.png" id="S2.F3.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="371" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S2.F3.2.1.1" class="ltx_text" style="font-size:90%;">Figure 3</span>: </span><span id="S2.F3.3.2" class="ltx_text" style="font-size:90%;">Design roadmap of sub-billion sized transformer models. The foreground and background bars represents the averaged accuracy on zero-shot common sense reasoning tasks for 125M and 350M models, respectively. The 125M model, initially a 12-layer 768-dimension structure, is enhanced through improving feed-forward network design, network depth adjustments, and weight sharing strategies. The detailed accuracy on each modification can be found in the appendix.</span></figcaption>
</figure>
</section>
<section id="S2.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.2 </span>Building a Strong Baseline</h3>

<section id="S2.SS2.SSS1" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">2.2.1 </span>Feed-forward Network Choice</h4>

<div id="S2.SS2.SSS1.p1" class="ltx_para">
<p id="S2.SS2.SSS1.p1.1" class="ltx_p">We first investigate activation functions commonly used in feed-forward networks (FFNs) and find that the state-of-the-art SwiGLU <cite class="ltx_cite ltx_citemacro_citep">(Dauphin et al., <a href="#bib.bib9" title="" class="ltx_ref">2017</a>)</cite> is also beneficial for small models. By changing the vanilla FFN (<math id="S2.SS2.SSS1.p1.1.m1.1" class="ltx_Math" alttext="FC\rightarrow ReLU\rightarrow FC" display="inline"><semantics id="S2.SS2.SSS1.p1.1.m1.1a"><mrow id="S2.SS2.SSS1.p1.1.m1.1b"><mi id="S2.SS2.SSS1.p1.1.m1.1.1" xref="S2.SS2.SSS1.p1.1.m1.1.1.cmml">F</mi><mi id="S2.SS2.SSS1.p1.1.m1.1.2" xref="S2.SS2.SSS1.p1.1.m1.1.2.cmml">C</mi><mo stretchy="false" id="S2.SS2.SSS1.p1.1.m1.1.3" xref="S2.SS2.SSS1.p1.1.m1.1.3.cmml">→</mo><mi id="S2.SS2.SSS1.p1.1.m1.1.4" xref="S2.SS2.SSS1.p1.1.m1.1.4.cmml">R</mi><mi id="S2.SS2.SSS1.p1.1.m1.1.5" xref="S2.SS2.SSS1.p1.1.m1.1.5.cmml">e</mi><mi id="S2.SS2.SSS1.p1.1.m1.1.6" xref="S2.SS2.SSS1.p1.1.m1.1.6.cmml">L</mi><mi id="S2.SS2.SSS1.p1.1.m1.1.7" xref="S2.SS2.SSS1.p1.1.m1.1.7.cmml">U</mi><mo stretchy="false" id="S2.SS2.SSS1.p1.1.m1.1.8" xref="S2.SS2.SSS1.p1.1.m1.1.8.cmml">→</mo><mi id="S2.SS2.SSS1.p1.1.m1.1.9" xref="S2.SS2.SSS1.p1.1.m1.1.9.cmml">F</mi><mi id="S2.SS2.SSS1.p1.1.m1.1.10" xref="S2.SS2.SSS1.p1.1.m1.1.10.cmml">C</mi></mrow><annotation-xml encoding="MathML-Content" id="S2.SS2.SSS1.p1.1.m1.1c"><cerror id="S2.SS2.SSS1.p1.1.m1.1d"><csymbol cd="ambiguous" id="S2.SS2.SSS1.p1.1.m1.1e">fragments</csymbol><csymbol cd="unknown" id="S2.SS2.SSS1.p1.1.m1.1.1.cmml" xref="S2.SS2.SSS1.p1.1.m1.1.1">F</csymbol><csymbol cd="unknown" id="S2.SS2.SSS1.p1.1.m1.1.2.cmml" xref="S2.SS2.SSS1.p1.1.m1.1.2">C</csymbol><ci id="S2.SS2.SSS1.p1.1.m1.1.3.cmml" xref="S2.SS2.SSS1.p1.1.m1.1.3">→</ci><csymbol cd="unknown" id="S2.SS2.SSS1.p1.1.m1.1.4.cmml" xref="S2.SS2.SSS1.p1.1.m1.1.4">R</csymbol><csymbol cd="unknown" id="S2.SS2.SSS1.p1.1.m1.1.5.cmml" xref="S2.SS2.SSS1.p1.1.m1.1.5">e</csymbol><csymbol cd="unknown" id="S2.SS2.SSS1.p1.1.m1.1.6.cmml" xref="S2.SS2.SSS1.p1.1.m1.1.6">L</csymbol><csymbol cd="unknown" id="S2.SS2.SSS1.p1.1.m1.1.7.cmml" xref="S2.SS2.SSS1.p1.1.m1.1.7">U</csymbol><ci id="S2.SS2.SSS1.p1.1.m1.1.8.cmml" xref="S2.SS2.SSS1.p1.1.m1.1.8">→</ci><csymbol cd="unknown" id="S2.SS2.SSS1.p1.1.m1.1.9.cmml" xref="S2.SS2.SSS1.p1.1.m1.1.9">F</csymbol><csymbol cd="unknown" id="S2.SS2.SSS1.p1.1.m1.1.10.cmml" xref="S2.SS2.SSS1.p1.1.m1.1.10">C</csymbol></cerror></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.SSS1.p1.1.m1.1f">FC\rightarrow ReLU\rightarrow FC</annotation></semantics></math>) to SwiGLU, The average performance on zero-shot reasoning tasks is boost from 42.6 to 43.9 for the 125M model. Therefore, we use SwiGLU in FFN for the experiments afterwards.</p>
</div>
</section>
<section id="S2.SS2.SSS2" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">2.2.2 </span>Architecture Depth vs Width</h4>

<figure id="S2.F4" class="ltx_figure"><img src="/html/2402.14905/assets/x4.png" id="S2.F4.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="156" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S2.F4.2.1.1" class="ltx_text" style="font-size:90%;">Figure 4</span>: </span><span id="S2.F4.3.2" class="ltx_text" style="font-size:90%;">Under comparable model sizes, deeper and thinner models generally outperform their wider and shallower counterparts across various tasks such as zero-shot common sense reasoning, question answering, and reading comprehension.</span></figcaption>
</figure>
<div id="S2.SS2.SSS2.p1" class="ltx_para">
<p id="S2.SS2.SSS2.p1.1" class="ltx_p">A prevalent belief <cite class="ltx_cite ltx_citemacro_citep">(Kaplan et al., <a href="#bib.bib22" title="" class="ltx_ref">2020</a>)</cite> in the field suggests that the performance of transformer models is primarily determined by number of parameters, the size of the training dataset, and the number of training iterations. This belief posits that architectural designs have negligible impact on the transformer model’s performance. However, our findings indicate that this may not hold true for smaller models.</p>
</div>
<div id="S2.SS2.SSS2.p2" class="ltx_para">
<p id="S2.SS2.SSS2.p2.2" class="ltx_p">Our experimental results, specifically for small models with limited model capacity, reveals that going deeper is more crucial than going wider for performance improvement.
We conducted an extensive study involving the training of 19 models, including 9 models with <math id="S2.SS2.SSS2.p2.1.m1.1" class="ltx_Math" alttext="\sim" display="inline"><semantics id="S2.SS2.SSS2.p2.1.m1.1a"><mo id="S2.SS2.SSS2.p2.1.m1.1.1" xref="S2.SS2.SSS2.p2.1.m1.1.1.cmml">∼</mo><annotation-xml encoding="MathML-Content" id="S2.SS2.SSS2.p2.1.m1.1b"><csymbol cd="latexml" id="S2.SS2.SSS2.p2.1.m1.1.1.cmml" xref="S2.SS2.SSS2.p2.1.m1.1.1">similar-to</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.SSS2.p2.1.m1.1c">\sim</annotation></semantics></math>125M parameters and 10 models with <math id="S2.SS2.SSS2.p2.2.m2.1" class="ltx_Math" alttext="\sim" display="inline"><semantics id="S2.SS2.SSS2.p2.2.m2.1a"><mo id="S2.SS2.SSS2.p2.2.m2.1.1" xref="S2.SS2.SSS2.p2.2.m2.1.1.cmml">∼</mo><annotation-xml encoding="MathML-Content" id="S2.SS2.SSS2.p2.2.m2.1b"><csymbol cd="latexml" id="S2.SS2.SSS2.p2.2.m2.1.1.cmml" xref="S2.SS2.SSS2.p2.2.m2.1.1">similar-to</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.SSS2.p2.2.m2.1c">\sim</annotation></semantics></math>350M parameters. Each model is designed with similar size but varied in terms of depth and width. We experiment on eight zero-shot common sense reasoning tasks, as well as question answering and reading comprehension benchmarks. Our findings consistently demonstrate that deeper and thinner models outperform their shallower and wider counterparts. Figure <a href="#S2.F4" title="Figure 4 ‣ 2.2.2 Architecture Depth vs Width ‣ 2.2 Building a Strong Baseline ‣ 2 Improving Sub-billion Scale LLM Design ‣ MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a> (a) and (b) illustrate the superior performance of deeper networks across most zero-shot reasoning tasks, including ARC-easy, ARC-challenge, PIQA, HellaSwag, OBQA, WinoGrande. Particularly, this trend is even more pronounced on the TQA and RACE datasets, as shown in Figure <a href="#S2.F4" title="Figure 4 ‣ 2.2.2 Architecture Depth vs Width ‣ 2.2 Building a Strong Baseline ‣ 2 Improving Sub-billion Scale LLM Design ‣ MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a> (c)-(f). Detailed model configurations and results can be seen in appendix. Our findings suggest that models with 30 or even 42 layers perform significantly better than those with 12 layers for transformer models sized around 125M. This finding is surprising considering the number of layers in most previous 125M models <cite class="ltx_cite ltx_citemacro_citep">(Zhang et al., <a href="#bib.bib57" title="" class="ltx_ref">2022</a>; Black et al., <a href="#bib.bib4" title="" class="ltx_ref">2022</a>)</cite> is limited to 12.</p>
</div>
</section>
<section id="S2.SS2.SSS3" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">2.2.3 </span>Embedding Sharing</h4>

<figure id="S2.T1" class="ltx_table">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table"><span id="S2.T1.5.2.1" class="ltx_text" style="font-size:90%;">Table 1</span>: </span><span id="S2.T1.2.1" class="ltx_text" style="font-size:90%;">Ablation study on input-output embedding sharing with a 30-layer model with 512 embedding dimension, on zero-shot common-sense reasoning tasks. Increased depth (<math id="S2.T1.2.1.m1.1" class="ltx_Math" alttext="\uparrow" display="inline"><semantics id="S2.T1.2.1.m1.1b"><mo stretchy="false" id="S2.T1.2.1.m1.1.1" xref="S2.T1.2.1.m1.1.1.cmml">↑</mo><annotation-xml encoding="MathML-Content" id="S2.T1.2.1.m1.1c"><ci id="S2.T1.2.1.m1.1.1.cmml" xref="S2.T1.2.1.m1.1.1">↑</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.2.1.m1.1d">\uparrow</annotation></semantics></math> depth) model has 32 layers.</span></figcaption>
<div id="S2.T1.3" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:368.6pt;height:55.4pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(-55.3pt,8.3pt) scale(0.76904597614895,0.76904597614895) ;">
<table id="S2.T1.3.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S2.T1.3.1.2.1" class="ltx_tr">
<th id="S2.T1.3.1.2.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_tt" style="padding:-2pt 3.4pt;"><span id="S2.T1.3.1.2.1.1.1" class="ltx_text ltx_font_bold">Model</span></th>
<th id="S2.T1.3.1.2.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 3.4pt;"><span id="S2.T1.3.1.2.1.2.1" class="ltx_text ltx_font_bold"># Params</span></th>
<th id="S2.T1.3.1.2.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 3.4pt;"><span id="S2.T1.3.1.2.1.3.1" class="ltx_text ltx_font_bold">ARC-e</span></th>
<th id="S2.T1.3.1.2.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 3.4pt;"><span id="S2.T1.3.1.2.1.4.1" class="ltx_text ltx_font_bold">ARC-c</span></th>
<th id="S2.T1.3.1.2.1.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 3.4pt;"><span id="S2.T1.3.1.2.1.5.1" class="ltx_text ltx_font_bold">BoolQ</span></th>
<th id="S2.T1.3.1.2.1.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 3.4pt;"><span id="S2.T1.3.1.2.1.6.1" class="ltx_text ltx_font_bold">PIQA</span></th>
<th id="S2.T1.3.1.2.1.7" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 3.4pt;"><span id="S2.T1.3.1.2.1.7.1" class="ltx_text ltx_font_bold">SIQA</span></th>
<th id="S2.T1.3.1.2.1.8" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 3.4pt;"><span id="S2.T1.3.1.2.1.8.1" class="ltx_text ltx_font_bold">HS</span></th>
<th id="S2.T1.3.1.2.1.9" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 3.4pt;"><span id="S2.T1.3.1.2.1.9.1" class="ltx_text ltx_font_bold">OBQA</span></th>
<th id="S2.T1.3.1.2.1.10" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 3.4pt;"><span id="S2.T1.3.1.2.1.10.1" class="ltx_text ltx_font_bold">WinoGrande</span></th>
<th id="S2.T1.3.1.2.1.11" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 3.4pt;"><span id="S2.T1.3.1.2.1.11.1" class="ltx_text ltx_font_bold">Avg.</span></th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S2.T1.3.1.3.1" class="ltx_tr">
<th id="S2.T1.3.1.3.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t" style="padding:-2pt 3.4pt;">Without emb-share</th>
<td id="S2.T1.3.1.3.1.2" class="ltx_td ltx_align_center ltx_border_t" style="padding:-2pt 3.4pt;">135M</td>
<td id="S2.T1.3.1.3.1.3" class="ltx_td ltx_align_center ltx_border_t" style="padding:-2pt 3.4pt;">43.6</td>
<td id="S2.T1.3.1.3.1.4" class="ltx_td ltx_align_center ltx_border_t" style="padding:-2pt 3.4pt;">26.1</td>
<td id="S2.T1.3.1.3.1.5" class="ltx_td ltx_align_center ltx_border_t" style="padding:-2pt 3.4pt;">58.0</td>
<td id="S2.T1.3.1.3.1.6" class="ltx_td ltx_align_center ltx_border_t" style="padding:-2pt 3.4pt;">62.5</td>
<td id="S2.T1.3.1.3.1.7" class="ltx_td ltx_align_center ltx_border_t" style="padding:-2pt 3.4pt;">42.6</td>
<td id="S2.T1.3.1.3.1.8" class="ltx_td ltx_align_center ltx_border_t" style="padding:-2pt 3.4pt;">36.5</td>
<td id="S2.T1.3.1.3.1.9" class="ltx_td ltx_align_center ltx_border_t" style="padding:-2pt 3.4pt;">37.5</td>
<td id="S2.T1.3.1.3.1.10" class="ltx_td ltx_align_center ltx_border_t" style="padding:-2pt 3.4pt;">51.5</td>
<td id="S2.T1.3.1.3.1.11" class="ltx_td ltx_align_center ltx_border_t" style="padding:-2pt 3.4pt;">44.8</td>
</tr>
<tr id="S2.T1.3.1.4.2" class="ltx_tr">
<th id="S2.T1.3.1.4.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row" style="padding:-2pt 3.4pt;">+ emb-share</th>
<td id="S2.T1.3.1.4.2.2" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">119M</td>
<td id="S2.T1.3.1.4.2.3" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">44.4</td>
<td id="S2.T1.3.1.4.2.4" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">26.0</td>
<td id="S2.T1.3.1.4.2.5" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">56.2</td>
<td id="S2.T1.3.1.4.2.6" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">62.8</td>
<td id="S2.T1.3.1.4.2.7" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">43.1</td>
<td id="S2.T1.3.1.4.2.8" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">35.9</td>
<td id="S2.T1.3.1.4.2.9" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">36.0</td>
<td id="S2.T1.3.1.4.2.10" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">52.6</td>
<td id="S2.T1.3.1.4.2.11" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">44.6</td>
</tr>
<tr id="S2.T1.3.1.1" class="ltx_tr">
<th id="S2.T1.3.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb" style="padding:-2pt 3.4pt;">+ emb-share, <math id="S2.T1.3.1.1.1.m1.1" class="ltx_Math" alttext="\uparrow" display="inline"><semantics id="S2.T1.3.1.1.1.m1.1a"><mo stretchy="false" id="S2.T1.3.1.1.1.m1.1.1" xref="S2.T1.3.1.1.1.m1.1.1.cmml">↑</mo><annotation-xml encoding="MathML-Content" id="S2.T1.3.1.1.1.m1.1b"><ci id="S2.T1.3.1.1.1.m1.1.1.cmml" xref="S2.T1.3.1.1.1.m1.1.1">↑</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.3.1.1.1.m1.1c">\uparrow</annotation></semantics></math> depth</th>
<td id="S2.T1.3.1.1.2" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 3.4pt;">125M</td>
<td id="S2.T1.3.1.1.3" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 3.4pt;">43.3</td>
<td id="S2.T1.3.1.1.4" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 3.4pt;">26.4</td>
<td id="S2.T1.3.1.1.5" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 3.4pt;">54.4</td>
<td id="S2.T1.3.1.1.6" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 3.4pt;">64.7</td>
<td id="S2.T1.3.1.1.7" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 3.4pt;">43.5</td>
<td id="S2.T1.3.1.1.8" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 3.4pt;">36.9</td>
<td id="S2.T1.3.1.1.9" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 3.4pt;">38.5</td>
<td id="S2.T1.3.1.1.10" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 3.4pt;">52.6</td>
<td id="S2.T1.3.1.1.11" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 3.4pt;">45.0</td>
</tr>
</tbody>
</table>
</span></div>
</figure>
<div id="S2.SS2.SSS3.p1" class="ltx_para">
<p id="S2.SS2.SSS3.p1.1" class="ltx_p">In sub-billion scale language models, the embedding layers constitute a significant portion of the parameter count. For instance, with an embedding dimension of 512 and a vocabulary size of 32k, the input and output embedding layers each comprise 16 million parameters. Together, these embedding layers account for more than 20% the total parameters of a 125M-parameter model.
Contrastingly, this proportion is considerably lower in larger language models. For example, the input and output embeddings only account for 3.7% of the total number of parameters in the LLaMA-7B model <cite class="ltx_cite ltx_citemacro_citep">(Touvron et al., <a href="#bib.bib46" title="" class="ltx_ref">2023a</a>)</cite> and a mere 0.7% in the LLaMA-70B model. This disparity might elucidate why embedding sharing was initially proposed and implemented in OPT models <cite class="ltx_cite ltx_citemacro_citep">(Zhang et al., <a href="#bib.bib57" title="" class="ltx_ref">2022</a>)</cite> but was subsequently disregarded in recent designs of LLMs.</p>
</div>
<div id="S2.SS2.SSS3.p2" class="ltx_para">
<p id="S2.SS2.SSS3.p2.2" class="ltx_p">In the development of sub-billion scale language models, we revisit the concept of input-output embedding sharing. The input embedding in LLM models maps the token id in the vocabulary to the corresponding token embedding and has a dimension of <math id="S2.SS2.SSS3.p2.1.m1.1" class="ltx_Math" alttext="(vocab\_size,embedding\_dim)" display="inline"><semantics id="S2.SS2.SSS3.p2.1.m1.1a"><mrow id="S2.SS2.SSS3.p2.1.m1.1b"><mo stretchy="false" id="S2.SS2.SSS3.p2.1.m1.1.1" xref="S2.SS2.SSS3.p2.1.m1.1.1.cmml">(</mo><mi id="S2.SS2.SSS3.p2.1.m1.1.2" xref="S2.SS2.SSS3.p2.1.m1.1.2.cmml">v</mi><mi id="S2.SS2.SSS3.p2.1.m1.1.3" xref="S2.SS2.SSS3.p2.1.m1.1.3.cmml">o</mi><mi id="S2.SS2.SSS3.p2.1.m1.1.4" xref="S2.SS2.SSS3.p2.1.m1.1.4.cmml">c</mi><mi id="S2.SS2.SSS3.p2.1.m1.1.5" xref="S2.SS2.SSS3.p2.1.m1.1.5.cmml">a</mi><mi id="S2.SS2.SSS3.p2.1.m1.1.6" xref="S2.SS2.SSS3.p2.1.m1.1.6.cmml">b</mi><mi mathvariant="normal" id="S2.SS2.SSS3.p2.1.m1.1.7" xref="S2.SS2.SSS3.p2.1.m1.1.7.cmml">_</mi><mi id="S2.SS2.SSS3.p2.1.m1.1.8" xref="S2.SS2.SSS3.p2.1.m1.1.8.cmml">s</mi><mi id="S2.SS2.SSS3.p2.1.m1.1.9" xref="S2.SS2.SSS3.p2.1.m1.1.9.cmml">i</mi><mi id="S2.SS2.SSS3.p2.1.m1.1.10" xref="S2.SS2.SSS3.p2.1.m1.1.10.cmml">z</mi><mi id="S2.SS2.SSS3.p2.1.m1.1.11" xref="S2.SS2.SSS3.p2.1.m1.1.11.cmml">e</mi><mo id="S2.SS2.SSS3.p2.1.m1.1.12" xref="S2.SS2.SSS3.p2.1.m1.1.12.cmml">,</mo><mi id="S2.SS2.SSS3.p2.1.m1.1.13" xref="S2.SS2.SSS3.p2.1.m1.1.13.cmml">e</mi><mi id="S2.SS2.SSS3.p2.1.m1.1.14" xref="S2.SS2.SSS3.p2.1.m1.1.14.cmml">m</mi><mi id="S2.SS2.SSS3.p2.1.m1.1.15" xref="S2.SS2.SSS3.p2.1.m1.1.15.cmml">b</mi><mi id="S2.SS2.SSS3.p2.1.m1.1.16" xref="S2.SS2.SSS3.p2.1.m1.1.16.cmml">e</mi><mi id="S2.SS2.SSS3.p2.1.m1.1.17" xref="S2.SS2.SSS3.p2.1.m1.1.17.cmml">d</mi><mi id="S2.SS2.SSS3.p2.1.m1.1.18" xref="S2.SS2.SSS3.p2.1.m1.1.18.cmml">d</mi><mi id="S2.SS2.SSS3.p2.1.m1.1.19" xref="S2.SS2.SSS3.p2.1.m1.1.19.cmml">i</mi><mi id="S2.SS2.SSS3.p2.1.m1.1.20" xref="S2.SS2.SSS3.p2.1.m1.1.20.cmml">n</mi><mi id="S2.SS2.SSS3.p2.1.m1.1.21" xref="S2.SS2.SSS3.p2.1.m1.1.21.cmml">g</mi><mi mathvariant="normal" id="S2.SS2.SSS3.p2.1.m1.1.22" xref="S2.SS2.SSS3.p2.1.m1.1.22.cmml">_</mi><mi id="S2.SS2.SSS3.p2.1.m1.1.23" xref="S2.SS2.SSS3.p2.1.m1.1.23.cmml">d</mi><mi id="S2.SS2.SSS3.p2.1.m1.1.24" xref="S2.SS2.SSS3.p2.1.m1.1.24.cmml">i</mi><mi id="S2.SS2.SSS3.p2.1.m1.1.25" xref="S2.SS2.SSS3.p2.1.m1.1.25.cmml">m</mi><mo stretchy="false" id="S2.SS2.SSS3.p2.1.m1.1.26" xref="S2.SS2.SSS3.p2.1.m1.1.26.cmml">)</mo></mrow><annotation-xml encoding="MathML-Content" id="S2.SS2.SSS3.p2.1.m1.1c"><cerror id="S2.SS2.SSS3.p2.1.m1.1d"><csymbol cd="ambiguous" id="S2.SS2.SSS3.p2.1.m1.1e">fragments</csymbol><ci id="S2.SS2.SSS3.p2.1.m1.1.1.cmml" xref="S2.SS2.SSS3.p2.1.m1.1.1">(</ci><csymbol cd="unknown" id="S2.SS2.SSS3.p2.1.m1.1.2.cmml" xref="S2.SS2.SSS3.p2.1.m1.1.2">v</csymbol><csymbol cd="unknown" id="S2.SS2.SSS3.p2.1.m1.1.3.cmml" xref="S2.SS2.SSS3.p2.1.m1.1.3">o</csymbol><csymbol cd="unknown" id="S2.SS2.SSS3.p2.1.m1.1.4.cmml" xref="S2.SS2.SSS3.p2.1.m1.1.4">c</csymbol><csymbol cd="unknown" id="S2.SS2.SSS3.p2.1.m1.1.5.cmml" xref="S2.SS2.SSS3.p2.1.m1.1.5">a</csymbol><csymbol cd="unknown" id="S2.SS2.SSS3.p2.1.m1.1.6.cmml" xref="S2.SS2.SSS3.p2.1.m1.1.6">b</csymbol><csymbol cd="unknown" id="S2.SS2.SSS3.p2.1.m1.1.7.cmml" xref="S2.SS2.SSS3.p2.1.m1.1.7">_</csymbol><csymbol cd="unknown" id="S2.SS2.SSS3.p2.1.m1.1.8.cmml" xref="S2.SS2.SSS3.p2.1.m1.1.8">s</csymbol><csymbol cd="unknown" id="S2.SS2.SSS3.p2.1.m1.1.9.cmml" xref="S2.SS2.SSS3.p2.1.m1.1.9">i</csymbol><csymbol cd="unknown" id="S2.SS2.SSS3.p2.1.m1.1.10.cmml" xref="S2.SS2.SSS3.p2.1.m1.1.10">z</csymbol><csymbol cd="unknown" id="S2.SS2.SSS3.p2.1.m1.1.11.cmml" xref="S2.SS2.SSS3.p2.1.m1.1.11">e</csymbol><ci id="S2.SS2.SSS3.p2.1.m1.1.12.cmml" xref="S2.SS2.SSS3.p2.1.m1.1.12">,</ci><csymbol cd="unknown" id="S2.SS2.SSS3.p2.1.m1.1.13.cmml" xref="S2.SS2.SSS3.p2.1.m1.1.13">e</csymbol><csymbol cd="unknown" id="S2.SS2.SSS3.p2.1.m1.1.14.cmml" xref="S2.SS2.SSS3.p2.1.m1.1.14">m</csymbol><csymbol cd="unknown" id="S2.SS2.SSS3.p2.1.m1.1.15.cmml" xref="S2.SS2.SSS3.p2.1.m1.1.15">b</csymbol><csymbol cd="unknown" id="S2.SS2.SSS3.p2.1.m1.1.16.cmml" xref="S2.SS2.SSS3.p2.1.m1.1.16">e</csymbol><csymbol cd="unknown" id="S2.SS2.SSS3.p2.1.m1.1.17.cmml" xref="S2.SS2.SSS3.p2.1.m1.1.17">d</csymbol><csymbol cd="unknown" id="S2.SS2.SSS3.p2.1.m1.1.18.cmml" xref="S2.SS2.SSS3.p2.1.m1.1.18">d</csymbol><csymbol cd="unknown" id="S2.SS2.SSS3.p2.1.m1.1.19.cmml" xref="S2.SS2.SSS3.p2.1.m1.1.19">i</csymbol><csymbol cd="unknown" id="S2.SS2.SSS3.p2.1.m1.1.20.cmml" xref="S2.SS2.SSS3.p2.1.m1.1.20">n</csymbol><csymbol cd="unknown" id="S2.SS2.SSS3.p2.1.m1.1.21.cmml" xref="S2.SS2.SSS3.p2.1.m1.1.21">g</csymbol><csymbol cd="unknown" id="S2.SS2.SSS3.p2.1.m1.1.22.cmml" xref="S2.SS2.SSS3.p2.1.m1.1.22">_</csymbol><csymbol cd="unknown" id="S2.SS2.SSS3.p2.1.m1.1.23.cmml" xref="S2.SS2.SSS3.p2.1.m1.1.23">d</csymbol><csymbol cd="unknown" id="S2.SS2.SSS3.p2.1.m1.1.24.cmml" xref="S2.SS2.SSS3.p2.1.m1.1.24">i</csymbol><csymbol cd="unknown" id="S2.SS2.SSS3.p2.1.m1.1.25.cmml" xref="S2.SS2.SSS3.p2.1.m1.1.25">m</csymbol><ci id="S2.SS2.SSS3.p2.1.m1.1.26.cmml" xref="S2.SS2.SSS3.p2.1.m1.1.26">)</ci></cerror></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.SSS3.p2.1.m1.1f">(vocab\_size,embedding\_dim)</annotation></semantics></math>. Conversely, the output fully-connected layer maps the embedding dimension back to the logits prediction across the vocabulary, with weight size of <math id="S2.SS2.SSS3.p2.2.m2.1" class="ltx_Math" alttext="(vocab\_size,embedding\_dim)" display="inline"><semantics id="S2.SS2.SSS3.p2.2.m2.1a"><mrow id="S2.SS2.SSS3.p2.2.m2.1b"><mo stretchy="false" id="S2.SS2.SSS3.p2.2.m2.1.1" xref="S2.SS2.SSS3.p2.2.m2.1.1.cmml">(</mo><mi id="S2.SS2.SSS3.p2.2.m2.1.2" xref="S2.SS2.SSS3.p2.2.m2.1.2.cmml">v</mi><mi id="S2.SS2.SSS3.p2.2.m2.1.3" xref="S2.SS2.SSS3.p2.2.m2.1.3.cmml">o</mi><mi id="S2.SS2.SSS3.p2.2.m2.1.4" xref="S2.SS2.SSS3.p2.2.m2.1.4.cmml">c</mi><mi id="S2.SS2.SSS3.p2.2.m2.1.5" xref="S2.SS2.SSS3.p2.2.m2.1.5.cmml">a</mi><mi id="S2.SS2.SSS3.p2.2.m2.1.6" xref="S2.SS2.SSS3.p2.2.m2.1.6.cmml">b</mi><mi mathvariant="normal" id="S2.SS2.SSS3.p2.2.m2.1.7" xref="S2.SS2.SSS3.p2.2.m2.1.7.cmml">_</mi><mi id="S2.SS2.SSS3.p2.2.m2.1.8" xref="S2.SS2.SSS3.p2.2.m2.1.8.cmml">s</mi><mi id="S2.SS2.SSS3.p2.2.m2.1.9" xref="S2.SS2.SSS3.p2.2.m2.1.9.cmml">i</mi><mi id="S2.SS2.SSS3.p2.2.m2.1.10" xref="S2.SS2.SSS3.p2.2.m2.1.10.cmml">z</mi><mi id="S2.SS2.SSS3.p2.2.m2.1.11" xref="S2.SS2.SSS3.p2.2.m2.1.11.cmml">e</mi><mo id="S2.SS2.SSS3.p2.2.m2.1.12" xref="S2.SS2.SSS3.p2.2.m2.1.12.cmml">,</mo><mi id="S2.SS2.SSS3.p2.2.m2.1.13" xref="S2.SS2.SSS3.p2.2.m2.1.13.cmml">e</mi><mi id="S2.SS2.SSS3.p2.2.m2.1.14" xref="S2.SS2.SSS3.p2.2.m2.1.14.cmml">m</mi><mi id="S2.SS2.SSS3.p2.2.m2.1.15" xref="S2.SS2.SSS3.p2.2.m2.1.15.cmml">b</mi><mi id="S2.SS2.SSS3.p2.2.m2.1.16" xref="S2.SS2.SSS3.p2.2.m2.1.16.cmml">e</mi><mi id="S2.SS2.SSS3.p2.2.m2.1.17" xref="S2.SS2.SSS3.p2.2.m2.1.17.cmml">d</mi><mi id="S2.SS2.SSS3.p2.2.m2.1.18" xref="S2.SS2.SSS3.p2.2.m2.1.18.cmml">d</mi><mi id="S2.SS2.SSS3.p2.2.m2.1.19" xref="S2.SS2.SSS3.p2.2.m2.1.19.cmml">i</mi><mi id="S2.SS2.SSS3.p2.2.m2.1.20" xref="S2.SS2.SSS3.p2.2.m2.1.20.cmml">n</mi><mi id="S2.SS2.SSS3.p2.2.m2.1.21" xref="S2.SS2.SSS3.p2.2.m2.1.21.cmml">g</mi><mi mathvariant="normal" id="S2.SS2.SSS3.p2.2.m2.1.22" xref="S2.SS2.SSS3.p2.2.m2.1.22.cmml">_</mi><mi id="S2.SS2.SSS3.p2.2.m2.1.23" xref="S2.SS2.SSS3.p2.2.m2.1.23.cmml">d</mi><mi id="S2.SS2.SSS3.p2.2.m2.1.24" xref="S2.SS2.SSS3.p2.2.m2.1.24.cmml">i</mi><mi id="S2.SS2.SSS3.p2.2.m2.1.25" xref="S2.SS2.SSS3.p2.2.m2.1.25.cmml">m</mi><mo stretchy="false" id="S2.SS2.SSS3.p2.2.m2.1.26" xref="S2.SS2.SSS3.p2.2.m2.1.26.cmml">)</mo></mrow><annotation-xml encoding="MathML-Content" id="S2.SS2.SSS3.p2.2.m2.1c"><cerror id="S2.SS2.SSS3.p2.2.m2.1d"><csymbol cd="ambiguous" id="S2.SS2.SSS3.p2.2.m2.1e">fragments</csymbol><ci id="S2.SS2.SSS3.p2.2.m2.1.1.cmml" xref="S2.SS2.SSS3.p2.2.m2.1.1">(</ci><csymbol cd="unknown" id="S2.SS2.SSS3.p2.2.m2.1.2.cmml" xref="S2.SS2.SSS3.p2.2.m2.1.2">v</csymbol><csymbol cd="unknown" id="S2.SS2.SSS3.p2.2.m2.1.3.cmml" xref="S2.SS2.SSS3.p2.2.m2.1.3">o</csymbol><csymbol cd="unknown" id="S2.SS2.SSS3.p2.2.m2.1.4.cmml" xref="S2.SS2.SSS3.p2.2.m2.1.4">c</csymbol><csymbol cd="unknown" id="S2.SS2.SSS3.p2.2.m2.1.5.cmml" xref="S2.SS2.SSS3.p2.2.m2.1.5">a</csymbol><csymbol cd="unknown" id="S2.SS2.SSS3.p2.2.m2.1.6.cmml" xref="S2.SS2.SSS3.p2.2.m2.1.6">b</csymbol><csymbol cd="unknown" id="S2.SS2.SSS3.p2.2.m2.1.7.cmml" xref="S2.SS2.SSS3.p2.2.m2.1.7">_</csymbol><csymbol cd="unknown" id="S2.SS2.SSS3.p2.2.m2.1.8.cmml" xref="S2.SS2.SSS3.p2.2.m2.1.8">s</csymbol><csymbol cd="unknown" id="S2.SS2.SSS3.p2.2.m2.1.9.cmml" xref="S2.SS2.SSS3.p2.2.m2.1.9">i</csymbol><csymbol cd="unknown" id="S2.SS2.SSS3.p2.2.m2.1.10.cmml" xref="S2.SS2.SSS3.p2.2.m2.1.10">z</csymbol><csymbol cd="unknown" id="S2.SS2.SSS3.p2.2.m2.1.11.cmml" xref="S2.SS2.SSS3.p2.2.m2.1.11">e</csymbol><ci id="S2.SS2.SSS3.p2.2.m2.1.12.cmml" xref="S2.SS2.SSS3.p2.2.m2.1.12">,</ci><csymbol cd="unknown" id="S2.SS2.SSS3.p2.2.m2.1.13.cmml" xref="S2.SS2.SSS3.p2.2.m2.1.13">e</csymbol><csymbol cd="unknown" id="S2.SS2.SSS3.p2.2.m2.1.14.cmml" xref="S2.SS2.SSS3.p2.2.m2.1.14">m</csymbol><csymbol cd="unknown" id="S2.SS2.SSS3.p2.2.m2.1.15.cmml" xref="S2.SS2.SSS3.p2.2.m2.1.15">b</csymbol><csymbol cd="unknown" id="S2.SS2.SSS3.p2.2.m2.1.16.cmml" xref="S2.SS2.SSS3.p2.2.m2.1.16">e</csymbol><csymbol cd="unknown" id="S2.SS2.SSS3.p2.2.m2.1.17.cmml" xref="S2.SS2.SSS3.p2.2.m2.1.17">d</csymbol><csymbol cd="unknown" id="S2.SS2.SSS3.p2.2.m2.1.18.cmml" xref="S2.SS2.SSS3.p2.2.m2.1.18">d</csymbol><csymbol cd="unknown" id="S2.SS2.SSS3.p2.2.m2.1.19.cmml" xref="S2.SS2.SSS3.p2.2.m2.1.19">i</csymbol><csymbol cd="unknown" id="S2.SS2.SSS3.p2.2.m2.1.20.cmml" xref="S2.SS2.SSS3.p2.2.m2.1.20">n</csymbol><csymbol cd="unknown" id="S2.SS2.SSS3.p2.2.m2.1.21.cmml" xref="S2.SS2.SSS3.p2.2.m2.1.21">g</csymbol><csymbol cd="unknown" id="S2.SS2.SSS3.p2.2.m2.1.22.cmml" xref="S2.SS2.SSS3.p2.2.m2.1.22">_</csymbol><csymbol cd="unknown" id="S2.SS2.SSS3.p2.2.m2.1.23.cmml" xref="S2.SS2.SSS3.p2.2.m2.1.23">d</csymbol><csymbol cd="unknown" id="S2.SS2.SSS3.p2.2.m2.1.24.cmml" xref="S2.SS2.SSS3.p2.2.m2.1.24">i</csymbol><csymbol cd="unknown" id="S2.SS2.SSS3.p2.2.m2.1.25.cmml" xref="S2.SS2.SSS3.p2.2.m2.1.25">m</csymbol><ci id="S2.SS2.SSS3.p2.2.m2.1.26.cmml" xref="S2.SS2.SSS3.p2.2.m2.1.26">)</ci></cerror></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.SSS3.p2.2.m2.1f">(vocab\_size,embedding\_dim)</annotation></semantics></math>. By sharing the embedding, we reuse the input embedding weights as the output fully connected layer weights, resulting in a more efficient and compact model architecture.</p>
</div>
<div id="S2.SS2.SSS3.p3" class="ltx_para">
<p id="S2.SS2.SSS3.p3.1" class="ltx_p">We experiment on a 30-layer 125M model. In Table <a href="#S2.T1" title="Table 1 ‣ 2.2.3 Embedding Sharing ‣ 2.2 Building a Strong Baseline ‣ 2 Improving Sub-billion Scale LLM Design ‣ MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>, we demonstrate that sharing the input and output embeddings reduces the number of parameters by 16M, approximately 11.8% of total parameters with a 0.2 points drop in average accuracy. The marginal accuracy drop can be readily restored by reallocating the saved parameters to add more layers. Increasing the depth to 32 layers produces a 0.4 points accuracy gain while still maintaining 10M fewer parameters compared to the original 135M model. Similar results are also observed in 350M models.
These findings further suggest that embedding sharing is a valuable technique for maximizing weight utilization and optimizing model performance given limited model storage budget.</p>
</div>
</section>
<section id="S2.SS2.SSS4" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">2.2.4 </span>Number of Heads and KV Heads</h4>

<figure id="S2.F5" class="ltx_figure"><img src="/html/2402.14905/assets/x5.png" id="S2.F5.g1" class="ltx_graphics ltx_img_landscape" width="484" height="183" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure"><span id="S2.F5.2.1.1" class="ltx_text" style="font-size:90%;">Figure 5</span>: </span><span id="S2.F5.3.2" class="ltx_text" style="font-size:90%;">Ablation study on number of heads and kv-heads. Here, ratio denotes the number of heads divided by number of kv-heads. Averaged accuracy on zero-shot reasoning tasks is reported. </span></figcaption>
</figure>
<div id="S2.SS2.SSS4.p1" class="ltx_para">
<p id="S2.SS2.SSS4.p1.3" class="ltx_p">We now investigate the optimal head size for small transformer models. The trade-off between more semantics per head dimension and more non-linear combinations of multiple heads is a key consideration in choosing the head size. In addition, most previous studies have typically used an identical number of key-value heads to query heads in sub-billion parameter language models. Instead, we found that grouped query attention, which is initially designed for reducing key-value cache size in LLMs <cite class="ltx_cite ltx_citemacro_citep">(Chowdhery et al., <a href="#bib.bib5" title="" class="ltx_ref">2023</a>; Ainslie et al., <a href="#bib.bib1" title="" class="ltx_ref">2023</a>)</cite>, that can also effectively reduce redundancy in key-value heads in small LMs. Grouped query attention can be viewed as another form of weight-sharing for weight re-utilization, where the number of key-value heads is <math id="S2.SS2.SSS4.p1.1.m1.1" class="ltx_Math" alttext="1/n" display="inline"><semantics id="S2.SS2.SSS4.p1.1.m1.1a"><mrow id="S2.SS2.SSS4.p1.1.m1.1b"><mn id="S2.SS2.SSS4.p1.1.m1.1.1" xref="S2.SS2.SSS4.p1.1.m1.1.1.cmml">1</mn><mo id="S2.SS2.SSS4.p1.1.m1.1.2" xref="S2.SS2.SSS4.p1.1.m1.1.2.cmml">/</mo><mi id="S2.SS2.SSS4.p1.1.m1.1.3" xref="S2.SS2.SSS4.p1.1.m1.1.3.cmml">n</mi></mrow><annotation-xml encoding="MathML-Content" id="S2.SS2.SSS4.p1.1.m1.1c"><cerror id="S2.SS2.SSS4.p1.1.m1.1d"><csymbol cd="ambiguous" id="S2.SS2.SSS4.p1.1.m1.1e">fragments</csymbol><cn type="integer" id="S2.SS2.SSS4.p1.1.m1.1.1.cmml" xref="S2.SS2.SSS4.p1.1.m1.1.1">1</cn><divide id="S2.SS2.SSS4.p1.1.m1.1.2.cmml" xref="S2.SS2.SSS4.p1.1.m1.1.2"></divide><csymbol cd="unknown" id="S2.SS2.SSS4.p1.1.m1.1.3.cmml" xref="S2.SS2.SSS4.p1.1.m1.1.3">n</csymbol></cerror></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.SSS4.p1.1.m1.1f">1/n</annotation></semantics></math> that of query heads, and the kv head is repeated <math id="S2.SS2.SSS4.p1.2.m2.1" class="ltx_Math" alttext="n" display="inline"><semantics id="S2.SS2.SSS4.p1.2.m2.1a"><mi id="S2.SS2.SSS4.p1.2.m2.1.1" xref="S2.SS2.SSS4.p1.2.m2.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.SSS4.p1.2.m2.1b"><ci id="S2.SS2.SSS4.p1.2.m2.1.1.cmml" xref="S2.SS2.SSS4.p1.2.m2.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.SSS4.p1.2.m2.1c">n</annotation></semantics></math> times in computing attention scores and output together with the query. Here, <math id="S2.SS2.SSS4.p1.3.m3.1" class="ltx_Math" alttext="n\in\mathbb{Z}^{+}" display="inline"><semantics id="S2.SS2.SSS4.p1.3.m3.1a"><mrow id="S2.SS2.SSS4.p1.3.m3.1b"><mi id="S2.SS2.SSS4.p1.3.m3.1.1" xref="S2.SS2.SSS4.p1.3.m3.1.1.cmml">n</mi><mo id="S2.SS2.SSS4.p1.3.m3.1.2" xref="S2.SS2.SSS4.p1.3.m3.1.2.cmml">∈</mo><mi id="S2.SS2.SSS4.p1.3.m3.1.3" xref="S2.SS2.SSS4.p1.3.m3.1.3.cmml">ℤ</mi><msup id="S2.SS2.SSS4.p1.3.m3.1.4" xref="S2.SS2.SSS4.p1.3.m3.1.4.cmml"><mi id="S2.SS2.SSS4.p1.3.m3.1.4a" xref="S2.SS2.SSS4.p1.3.m3.1.4.cmml"></mi><mo id="S2.SS2.SSS4.p1.3.m3.1.4.1.1" xref="S2.SS2.SSS4.p1.3.m3.1.4.1.1.cmml">+</mo></msup></mrow><annotation-xml encoding="MathML-Content" id="S2.SS2.SSS4.p1.3.m3.1c"><cerror id="S2.SS2.SSS4.p1.3.m3.1d"><csymbol cd="ambiguous" id="S2.SS2.SSS4.p1.3.m3.1e">fragments</csymbol><csymbol cd="unknown" id="S2.SS2.SSS4.p1.3.m3.1.1.cmml" xref="S2.SS2.SSS4.p1.3.m3.1.1">n</csymbol><in id="S2.SS2.SSS4.p1.3.m3.1.2.cmml" xref="S2.SS2.SSS4.p1.3.m3.1.2"></in><csymbol cd="unknown" id="S2.SS2.SSS4.p1.3.m3.1.3.cmml" xref="S2.SS2.SSS4.p1.3.m3.1.3">Z</csymbol><apply id="S2.SS2.SSS4.p1.3.m3.1.4.cmml" xref="S2.SS2.SSS4.p1.3.m3.1.4"><plus id="S2.SS2.SSS4.p1.3.m3.1.4.1.1.cmml" xref="S2.SS2.SSS4.p1.3.m3.1.4.1.1"></plus></apply></cerror></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.SSS4.p1.3.m3.1f">n\in\mathbb{Z}^{+}</annotation></semantics></math> denotes a positive integer that the number of query heads are divisible by.</p>
</div>
<div id="S2.SS2.SSS4.p2" class="ltx_para">
<p id="S2.SS2.SSS4.p2.1" class="ltx_p">To establish a solid foundation for a state-of-the-art small transformer model, we conducted experiments to determine the desirable head size on 125M and 350M models. Results in Figure <a href="#S2.F5" title="Figure 5 ‣ 2.2.4 Number of Heads and KV Heads ‣ 2.2 Building a Strong Baseline ‣ 2 Improving Sub-billion Scale LLM Design ‣ MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a> show that using 16 query heads produces the best results. Additionally, reducing the number of kv-heads from 16 to 4 resulted in comparable accuracy for the 125M model and only 0.2 points accuracy drop in the 350M model with almost 10% model size reduction. These results serve as a guideline in our model architecture design. By adopting the grouped query attention (GQA) and meanwhile increasing the embedding dimension to maintain the model size, the accuracy of 125M further increases 0.4 points, indicating GQA as a favorable method to further squeeze out small model’s potential.</p>
</div>
<div id="S2.SS2.SSS4.p3" class="ltx_para">
<p id="S2.SS2.SSS4.p3.1" class="ltx_p">In summary, we tested out four state-of-the-art techniques beneficial to small model designs, including FFN with SwiGLU, deep and thin architecture, embedding sharing, and grouped query attention. Combining these techniques, we build a strong baseline small LLM and we name it <span id="S2.SS2.SSS4.p3.1.1" class="ltx_text ltx_markedasmath ltx_font_typewriter">MobileLLM</span>.</p>
</div>
</section>
</section>
<section id="S2.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.3 </span>Layer Sharing</h3>

<figure id="S2.SS3.1" class="ltx_table">

<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table"><span id="S2.SS3.1.2.1.1" class="ltx_text" style="font-size:90%;">Table 2</span>: </span><span id="S2.SS3.1.3.2" class="ltx_text" style="font-size:90%;">Ablation study of layer-sharing strategy on zero-shot common sense reasoning tasks.</span></figcaption><div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_1">
<div id="S2.SS3.1.4" class="ltx_inline-block ltx_figure_panel ltx_transformed_outer" style="width:411.9pt;height:114pt;vertical-align:-0.7pt;"><span class="ltx_transformed_inner" style="transform:translate(-88.5pt,24.3pt) scale(0.699394329249242,0.699394329249242) ;">
<table id="S2.SS3.1.4.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S2.SS3.1.4.1.1.1" class="ltx_tr">
<th id="S2.SS3.1.4.1.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_tt" style="padding:-2pt 4.3pt;"><span id="S2.SS3.1.4.1.1.1.1.1" class="ltx_text ltx_font_bold">Model</span></th>
<th id="S2.SS3.1.4.1.1.1.2" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_tt" style="padding:-2pt 4.3pt;"><span id="S2.SS3.1.4.1.1.1.2.1" class="ltx_text ltx_font_bold">Sharing method</span></th>
<th id="S2.SS3.1.4.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 4.3pt;"><span id="S2.SS3.1.4.1.1.1.3.1" class="ltx_text ltx_font_bold">ARC-e</span></th>
<th id="S2.SS3.1.4.1.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 4.3pt;"><span id="S2.SS3.1.4.1.1.1.4.1" class="ltx_text ltx_font_bold">ARC-c</span></th>
<th id="S2.SS3.1.4.1.1.1.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 4.3pt;"><span id="S2.SS3.1.4.1.1.1.5.1" class="ltx_text ltx_font_bold">BoolQ</span></th>
<th id="S2.SS3.1.4.1.1.1.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 4.3pt;"><span id="S2.SS3.1.4.1.1.1.6.1" class="ltx_text ltx_font_bold">PIQA</span></th>
<th id="S2.SS3.1.4.1.1.1.7" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 4.3pt;"><span id="S2.SS3.1.4.1.1.1.7.1" class="ltx_text ltx_font_bold">SIQA</span></th>
<th id="S2.SS3.1.4.1.1.1.8" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 4.3pt;"><span id="S2.SS3.1.4.1.1.1.8.1" class="ltx_text ltx_font_bold">HellaSwag</span></th>
<th id="S2.SS3.1.4.1.1.1.9" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 4.3pt;"><span id="S2.SS3.1.4.1.1.1.9.1" class="ltx_text ltx_font_bold">OBQA</span></th>
<th id="S2.SS3.1.4.1.1.1.10" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 4.3pt;"><span id="S2.SS3.1.4.1.1.1.10.1" class="ltx_text ltx_font_bold">WinoGrande</span></th>
<th id="S2.SS3.1.4.1.1.1.11" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 4.3pt;"><span id="S2.SS3.1.4.1.1.1.11.1" class="ltx_text ltx_font_bold">Avg.</span></th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S2.SS3.1.4.1.2.1" class="ltx_tr">
<th id="S2.SS3.1.4.1.2.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t" style="padding:-2pt 4.3pt;" rowspan="4"><span id="S2.SS3.1.4.1.2.1.1.1" class="ltx_text">125M</span></th>
<th id="S2.SS3.1.4.1.2.1.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t" style="padding:-2pt 4.3pt;">baseline</th>
<td id="S2.SS3.1.4.1.2.1.3" class="ltx_td ltx_align_center ltx_border_t" style="padding:-2pt 4.3pt;">41.6</td>
<td id="S2.SS3.1.4.1.2.1.4" class="ltx_td ltx_align_center ltx_border_t" style="padding:-2pt 4.3pt;">25.7</td>
<td id="S2.SS3.1.4.1.2.1.5" class="ltx_td ltx_align_center ltx_border_t" style="padding:-2pt 4.3pt;">61.1</td>
<td id="S2.SS3.1.4.1.2.1.6" class="ltx_td ltx_align_center ltx_border_t" style="padding:-2pt 4.3pt;">62.4</td>
<td id="S2.SS3.1.4.1.2.1.7" class="ltx_td ltx_align_center ltx_border_t" style="padding:-2pt 4.3pt;">43.1</td>
<td id="S2.SS3.1.4.1.2.1.8" class="ltx_td ltx_align_center ltx_border_t" style="padding:-2pt 4.3pt;">34.4</td>
<td id="S2.SS3.1.4.1.2.1.9" class="ltx_td ltx_align_center ltx_border_t" style="padding:-2pt 4.3pt;">36.9</td>
<td id="S2.SS3.1.4.1.2.1.10" class="ltx_td ltx_align_center ltx_border_t" style="padding:-2pt 4.3pt;">51.6</td>
<td id="S2.SS3.1.4.1.2.1.11" class="ltx_td ltx_align_center ltx_border_t" style="padding:-2pt 4.3pt;">44.6</td>
</tr>
<tr id="S2.SS3.1.4.1.3.2" class="ltx_tr">
<th id="S2.SS3.1.4.1.3.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row" style="padding:-2pt 4.3pt;">Immediate block-wise share</th>
<td id="S2.SS3.1.4.1.3.2.2" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">43.9</td>
<td id="S2.SS3.1.4.1.3.2.3" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">27.9</td>
<td id="S2.SS3.1.4.1.3.2.4" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">61.5</td>
<td id="S2.SS3.1.4.1.3.2.5" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">64.3</td>
<td id="S2.SS3.1.4.1.3.2.6" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">41.5</td>
<td id="S2.SS3.1.4.1.3.2.7" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">35.5</td>
<td id="S2.SS3.1.4.1.3.2.8" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">35.1</td>
<td id="S2.SS3.1.4.1.3.2.9" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">50.2</td>
<td id="S2.SS3.1.4.1.3.2.10" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">45.0</td>
</tr>
<tr id="S2.SS3.1.4.1.4.3" class="ltx_tr">
<th id="S2.SS3.1.4.1.4.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row" style="padding:-2pt 4.3pt;">Repeat-all-over share</th>
<td id="S2.SS3.1.4.1.4.3.2" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">43.6</td>
<td id="S2.SS3.1.4.1.4.3.3" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">27.1</td>
<td id="S2.SS3.1.4.1.4.3.4" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">60.7</td>
<td id="S2.SS3.1.4.1.4.3.5" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">63.4</td>
<td id="S2.SS3.1.4.1.4.3.6" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">42.6</td>
<td id="S2.SS3.1.4.1.4.3.7" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">35.5</td>
<td id="S2.SS3.1.4.1.4.3.8" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">36.9</td>
<td id="S2.SS3.1.4.1.4.3.9" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">51.7</td>
<td id="S2.SS3.1.4.1.4.3.10" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;"><span id="S2.SS3.1.4.1.4.3.10.1" class="ltx_text ltx_font_bold">45.2</span></td>
</tr>
<tr id="S2.SS3.1.4.1.5.4" class="ltx_tr">
<th id="S2.SS3.1.4.1.5.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row" style="padding:-2pt 4.3pt;">Reverse share</th>
<td id="S2.SS3.1.4.1.5.4.2" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">43.8</td>
<td id="S2.SS3.1.4.1.5.4.3" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">26.0</td>
<td id="S2.SS3.1.4.1.5.4.4" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">58.9</td>
<td id="S2.SS3.1.4.1.5.4.5" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">62.9</td>
<td id="S2.SS3.1.4.1.5.4.6" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">42.2</td>
<td id="S2.SS3.1.4.1.5.4.7" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">35.2</td>
<td id="S2.SS3.1.4.1.5.4.8" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">36.8</td>
<td id="S2.SS3.1.4.1.5.4.9" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">52.2</td>
<td id="S2.SS3.1.4.1.5.4.10" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">44.8</td>
</tr>
<tr id="S2.SS3.1.4.1.6.5" class="ltx_tr">
<th id="S2.SS3.1.4.1.6.5.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb" style="padding:-2pt 4.3pt;" rowspan="4">
<span id="S2.SS3.1.4.1.6.5.1.1" class="ltx_ERROR undefined">\hdashline</span>[0.8pt/1pt]
<span id="S2.SS3.1.4.1.6.5.1.2" class="ltx_text">350M</span>
</th>
<th id="S2.SS3.1.4.1.6.5.2" class="ltx_td ltx_align_left ltx_th ltx_th_row" style="padding:-2pt 4.3pt;">baseline</th>
<td id="S2.SS3.1.4.1.6.5.3" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">50.8</td>
<td id="S2.SS3.1.4.1.6.5.4" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">30.6</td>
<td id="S2.SS3.1.4.1.6.5.5" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">62.3</td>
<td id="S2.SS3.1.4.1.6.5.6" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">68.6</td>
<td id="S2.SS3.1.4.1.6.5.7" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">43.5</td>
<td id="S2.SS3.1.4.1.6.5.8" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">45.1</td>
<td id="S2.SS3.1.4.1.6.5.9" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">43.8</td>
<td id="S2.SS3.1.4.1.6.5.10" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">52.4</td>
<td id="S2.SS3.1.4.1.6.5.11" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">49.6</td>
</tr>
<tr id="S2.SS3.1.4.1.7.6" class="ltx_tr">
<th id="S2.SS3.1.4.1.7.6.1" class="ltx_td ltx_align_left ltx_th ltx_th_row" style="padding:-2pt 4.3pt;">Immediate block-wise share</th>
<td id="S2.SS3.1.4.1.7.6.2" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">51.5</td>
<td id="S2.SS3.1.4.1.7.6.3" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">30.8</td>
<td id="S2.SS3.1.4.1.7.6.4" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">59.6</td>
<td id="S2.SS3.1.4.1.7.6.5" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">68.2</td>
<td id="S2.SS3.1.4.1.7.6.6" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">43.9</td>
<td id="S2.SS3.1.4.1.7.6.7" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">47.7</td>
<td id="S2.SS3.1.4.1.7.6.8" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">44.7</td>
<td id="S2.SS3.1.4.1.7.6.9" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">55.0</td>
<td id="S2.SS3.1.4.1.7.6.10" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">50.2</td>
</tr>
<tr id="S2.SS3.1.4.1.8.7" class="ltx_tr">
<th id="S2.SS3.1.4.1.8.7.1" class="ltx_td ltx_align_left ltx_th ltx_th_row" style="padding:-2pt 4.3pt;">Repeat-all-over share</th>
<td id="S2.SS3.1.4.1.8.7.2" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">53.5</td>
<td id="S2.SS3.1.4.1.8.7.3" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">33.0</td>
<td id="S2.SS3.1.4.1.8.7.4" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">61.2</td>
<td id="S2.SS3.1.4.1.8.7.5" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">69.4</td>
<td id="S2.SS3.1.4.1.8.7.6" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">43.2</td>
<td id="S2.SS3.1.4.1.8.7.7" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">48.3</td>
<td id="S2.SS3.1.4.1.8.7.8" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">42.2</td>
<td id="S2.SS3.1.4.1.8.7.9" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">54.6</td>
<td id="S2.SS3.1.4.1.8.7.10" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;"><span id="S2.SS3.1.4.1.8.7.10.1" class="ltx_text ltx_font_bold">50.7</span></td>
</tr>
<tr id="S2.SS3.1.4.1.9.8" class="ltx_tr">
<th id="S2.SS3.1.4.1.9.8.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb" style="padding:-2pt 4.3pt;">Reverse share</th>
<td id="S2.SS3.1.4.1.9.8.2" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 4.3pt;">50.7</td>
<td id="S2.SS3.1.4.1.9.8.3" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 4.3pt;">32.2</td>
<td id="S2.SS3.1.4.1.9.8.4" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 4.3pt;">61.0</td>
<td id="S2.SS3.1.4.1.9.8.5" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 4.3pt;">68.8</td>
<td id="S2.SS3.1.4.1.9.8.6" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 4.3pt;">43.8</td>
<td id="S2.SS3.1.4.1.9.8.7" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 4.3pt;">47.4</td>
<td id="S2.SS3.1.4.1.9.8.8" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 4.3pt;">43.1</td>
<td id="S2.SS3.1.4.1.9.8.9" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 4.3pt;">53.8</td>
<td id="S2.SS3.1.4.1.9.8.10" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 4.3pt;">50.1</td>
</tr>
</tbody>
</table>
</span></div>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<p id="S2.SS3.1.5" class="ltx_p ltx_figure_panel">The findings in Section <a href="#S2.SS2.SSS2" title="2.2.2 Architecture Depth vs Width ‣ 2.2 Building a Strong Baseline ‣ 2 Improving Sub-billion Scale LLM Design ‣ MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2.2.2</span></a> on the impact of layer depth versus width suggest deeper layers are favorable for small transformer models. This motivates us to investigate layer sharing as a strategy to increase the number of hidden layers without additional model storage cost. This approach is particularly helpful in on-device scenarios where model size is a major constraint.</p>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<p id="S2.SS3.1.1" class="ltx_p ltx_figure_panel">Surprisingly, the experimental findings show that accuracy enhancement can be achieved by simply replicating transformer blocks, without necessitating architectural modifications or an enlargement of the model size.
We further examined three different weight sharing strategies, illustrated in Figure <a href="#S2.F6" title="Figure 6 ‣ 2.3 Layer Sharing ‣ 2 Improving Sub-billion Scale LLM Design ‣ MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a>. Results in Table <a href="#S2.SS3" title="2.3 Layer Sharing ‣ 2 Improving Sub-billion Scale LLM Design ‣ MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2.3</span></a> indicate that the repeat-over layer-sharing strategy produces the best performance among immediate block-wise repeat, repeat all-over, and reverse sharing strategies. However, considering the hardware memory hierarchy (Figure <a href="#S1.F2" title="Figure 2 ‣ 1 Introduction ‣ MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>), the SRAM for computing is typically limited to around 20MB. This capacity is usually only sufficient to hold a single transformer block. Therefore, placing shared weights in the cache and computing twice immediately can avoid the need to transfer weights between the SRAM and DRAM, resulting in improved overall execution speed for auto-regressive inference. Consequently, we have opted for the immediate block-wise sharing strategy in our model design. We denote the proposed model with layer sharing as <span id="S2.SS3.1.1.1" class="ltx_text ltx_markedasmath ltx_font_typewriter">MobileLLM-LS</span>.</p>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="S2.F6" class="ltx_figure ltx_figure_panel"><img src="/html/2402.14905/assets/x6.png" id="S2.F6.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="368" height="288" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S2.F6.2.1.1" class="ltx_text" style="font-size:90%;">Figure 6</span>: </span><span id="S2.F6.3.2" class="ltx_text" style="font-size:90%;">(a) Baseline model without layer sharing; (b) Immediate block-wise sharing; (c) Repeat-all-over sharing; (d) Reverse sharing. A transformer block contains the multi-head self-attention (MHSA) and the feed-forward network (FFN). While repeat-all-over sharing has slightly higher performance, immediate block-wise sharing best utilize the cache because the shared weights can stay in the cache and be immediately computed twice.</span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<section id="S3" class="ltx_section ltx_figure_panel">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">3 </span>Experiments</h2>

<section id="S3.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.1 </span>Experimental Settings</h3>

<div id="S3.SS1.p1" class="ltx_para">
<p id="S3.SS1.p1.1" class="ltx_p">We train <span id="S3.SS1.p1.1.1" class="ltx_text ltx_markedasmath ltx_font_typewriter">MobileLLM</span> from scratch using Adam optimizer <cite class="ltx_cite ltx_citemacro_citep">(Kingma &amp; Ba, <a href="#bib.bib25" title="" class="ltx_ref">2014</a>)</cite> with weight decay of 0.1. The experiments are conducted using 32 A100 GPUs, with a batch size of 32 on each GPU. The initial learning rate is set to 2e-3 and follows a cosine learning-rate decay strategy. We perform quick exploration experiments with 120k iterations on 0.25T tokens and train the best models reported in Tables <a href="#S3.SS2" title="3.2 Main Results ‣ 3 Experiments ‣ 2.3 Layer Sharing ‣ 2 Improving Sub-billion Scale LLM Design ‣ MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.2</span></a> and <a href="#S3.SS2" title="3.2 Main Results ‣ 3 Experiments ‣ 2.3 Layer Sharing ‣ 2 Improving Sub-billion Scale LLM Design ‣ MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.2</span></a> with 480k iterations on 1T tokens.</p>
</div>
</section>
<section id="S3.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.2 </span>Main Results</h3>

<figure id="S3.SS2.21" class="ltx_table">

<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table"><span id="S3.SS2.21.24.3.1" class="ltx_text" style="font-size:90%;">Table 3</span>: </span><span id="S3.SS2.4.4.2" class="ltx_text" style="font-size:90%;">Zero-shot performance on Common Sense Reasoning tasks. <span id="S3.SS2.4.4.2.1" class="ltx_text ltx_markedasmath ltx_font_typewriter">MobileLLM</span> denotes the proposed baseline model and <span id="S3.SS2.4.4.2.2" class="ltx_text ltx_markedasmath ltx_font_typewriter">MobileLLM-LS</span> is integrated with layer sharing with the #layer counting layers with distinct weights.</span></figcaption><div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_1">
<div id="S3.SS2.8.8" class="ltx_inline-block ltx_figure_panel ltx_transformed_outer" style="width:433.6pt;height:226.8pt;vertical-align:-0.7pt;"><span class="ltx_transformed_inner" style="transform:translate(-93.9pt,49.0pt) scale(0.697714644076707,0.697714644076707) ;">
<table id="S3.SS2.8.8.4" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S3.SS2.8.8.4.5.1" class="ltx_tr">
<th id="S3.SS2.8.8.4.5.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_tt" style="padding:-2pt 4.3pt;"><span id="S3.SS2.8.8.4.5.1.1.1" class="ltx_text ltx_font_bold">Model</span></th>
<th id="S3.SS2.8.8.4.5.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_th_row ltx_border_tt" style="padding:-2pt 4.3pt;"><span id="S3.SS2.8.8.4.5.1.2.1" class="ltx_text ltx_font_bold">#Layers</span></th>
<th id="S3.SS2.8.8.4.5.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 4.3pt;"><span id="S3.SS2.8.8.4.5.1.3.1" class="ltx_text ltx_font_bold">#Params</span></th>
<th id="S3.SS2.8.8.4.5.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 4.3pt;"><span id="S3.SS2.8.8.4.5.1.4.1" class="ltx_text ltx_font_bold">ARC-e</span></th>
<th id="S3.SS2.8.8.4.5.1.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 4.3pt;"><span id="S3.SS2.8.8.4.5.1.5.1" class="ltx_text ltx_font_bold">ARC-c</span></th>
<th id="S3.SS2.8.8.4.5.1.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 4.3pt;"><span id="S3.SS2.8.8.4.5.1.6.1" class="ltx_text ltx_font_bold">BoolQ</span></th>
<th id="S3.SS2.8.8.4.5.1.7" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 4.3pt;"><span id="S3.SS2.8.8.4.5.1.7.1" class="ltx_text ltx_font_bold">PIQA</span></th>
<th id="S3.SS2.8.8.4.5.1.8" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 4.3pt;"><span id="S3.SS2.8.8.4.5.1.8.1" class="ltx_text ltx_font_bold">SIQA</span></th>
<th id="S3.SS2.8.8.4.5.1.9" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 4.3pt;"><span id="S3.SS2.8.8.4.5.1.9.1" class="ltx_text ltx_font_bold">HellaSwag</span></th>
<th id="S3.SS2.8.8.4.5.1.10" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 4.3pt;"><span id="S3.SS2.8.8.4.5.1.10.1" class="ltx_text ltx_font_bold">OBQA</span></th>
<th id="S3.SS2.8.8.4.5.1.11" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 4.3pt;"><span id="S3.SS2.8.8.4.5.1.11.1" class="ltx_text ltx_font_bold">WinoGrande</span></th>
<th id="S3.SS2.8.8.4.5.1.12" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 4.3pt;"><span id="S3.SS2.8.8.4.5.1.12.1" class="ltx_text ltx_font_bold">Avg.</span></th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S3.SS2.8.8.4.6.1" class="ltx_tr">
<th id="S3.SS2.8.8.4.6.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t" style="padding:-2pt 4.3pt;">Cerebras-GPT-111M</th>
<th id="S3.SS2.8.8.4.6.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_t" style="padding:-2pt 4.3pt;">10</th>
<td id="S3.SS2.8.8.4.6.1.3" class="ltx_td ltx_align_center ltx_border_t" style="padding:-2pt 4.3pt;">111M</td>
<td id="S3.SS2.8.8.4.6.1.4" class="ltx_td ltx_align_center ltx_border_t" style="padding:-2pt 4.3pt;">35.8</td>
<td id="S3.SS2.8.8.4.6.1.5" class="ltx_td ltx_align_center ltx_border_t" style="padding:-2pt 4.3pt;">20.2</td>
<td id="S3.SS2.8.8.4.6.1.6" class="ltx_td ltx_align_center ltx_border_t" style="padding:-2pt 4.3pt;"><span id="S3.SS2.8.8.4.6.1.6.1" class="ltx_text ltx_font_bold">62.0</span></td>
<td id="S3.SS2.8.8.4.6.1.7" class="ltx_td ltx_align_center ltx_border_t" style="padding:-2pt 4.3pt;">58.0</td>
<td id="S3.SS2.8.8.4.6.1.8" class="ltx_td ltx_align_center ltx_border_t" style="padding:-2pt 4.3pt;">39.8</td>
<td id="S3.SS2.8.8.4.6.1.9" class="ltx_td ltx_align_center ltx_border_t" style="padding:-2pt 4.3pt;">26.7</td>
<td id="S3.SS2.8.8.4.6.1.10" class="ltx_td ltx_align_center ltx_border_t" style="padding:-2pt 4.3pt;">29.0</td>
<td id="S3.SS2.8.8.4.6.1.11" class="ltx_td ltx_align_center ltx_border_t" style="padding:-2pt 4.3pt;">48.8</td>
<td id="S3.SS2.8.8.4.6.1.12" class="ltx_td ltx_align_center ltx_border_t" style="padding:-2pt 4.3pt;">40.0</td>
</tr>
<tr id="S3.SS2.8.8.4.7.2" class="ltx_tr">
<th id="S3.SS2.8.8.4.7.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row" style="padding:-2pt 4.3pt;">LaMini-GPT-124M</th>
<th id="S3.SS2.8.8.4.7.2.2" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 4.3pt;">12</th>
<td id="S3.SS2.8.8.4.7.2.3" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">124M</td>
<td id="S3.SS2.8.8.4.7.2.4" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">43.6</td>
<td id="S3.SS2.8.8.4.7.2.5" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">26.0</td>
<td id="S3.SS2.8.8.4.7.2.6" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">51.8</td>
<td id="S3.SS2.8.8.4.7.2.7" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">62.7</td>
<td id="S3.SS2.8.8.4.7.2.8" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">42.1</td>
<td id="S3.SS2.8.8.4.7.2.9" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">30.2</td>
<td id="S3.SS2.8.8.4.7.2.10" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">29.6</td>
<td id="S3.SS2.8.8.4.7.2.11" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">49.2</td>
<td id="S3.SS2.8.8.4.7.2.12" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">41.9</td>
</tr>
<tr id="S3.SS2.8.8.4.8.3" class="ltx_tr">
<th id="S3.SS2.8.8.4.8.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row" style="padding:-2pt 4.3pt;">Galactica-125M</th>
<th id="S3.SS2.8.8.4.8.3.2" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 4.3pt;">12</th>
<td id="S3.SS2.8.8.4.8.3.3" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">125M</td>
<td id="S3.SS2.8.8.4.8.3.4" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;"><span id="S3.SS2.8.8.4.8.3.4.1" class="ltx_text ltx_font_bold">44.0</span></td>
<td id="S3.SS2.8.8.4.8.3.5" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">26.2</td>
<td id="S3.SS2.8.8.4.8.3.6" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">54.9</td>
<td id="S3.SS2.8.8.4.8.3.7" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">55.4</td>
<td id="S3.SS2.8.8.4.8.3.8" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">38.9</td>
<td id="S3.SS2.8.8.4.8.3.9" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">29.6</td>
<td id="S3.SS2.8.8.4.8.3.10" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">28.2</td>
<td id="S3.SS2.8.8.4.8.3.11" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">49.6</td>
<td id="S3.SS2.8.8.4.8.3.12" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">40.9</td>
</tr>
<tr id="S3.SS2.8.8.4.9.4" class="ltx_tr">
<th id="S3.SS2.8.8.4.9.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row" style="padding:-2pt 4.3pt;">OPT-125M</th>
<th id="S3.SS2.8.8.4.9.4.2" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 4.3pt;">12</th>
<td id="S3.SS2.8.8.4.9.4.3" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">125M</td>
<td id="S3.SS2.8.8.4.9.4.4" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">41.3</td>
<td id="S3.SS2.8.8.4.9.4.5" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">25.2</td>
<td id="S3.SS2.8.8.4.9.4.6" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">57.5</td>
<td id="S3.SS2.8.8.4.9.4.7" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">62.0</td>
<td id="S3.SS2.8.8.4.9.4.8" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">41.9</td>
<td id="S3.SS2.8.8.4.9.4.9" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">31.1</td>
<td id="S3.SS2.8.8.4.9.4.10" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">31.2</td>
<td id="S3.SS2.8.8.4.9.4.11" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">50.8</td>
<td id="S3.SS2.8.8.4.9.4.12" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">42.6</td>
</tr>
<tr id="S3.SS2.8.8.4.10.5" class="ltx_tr">
<th id="S3.SS2.8.8.4.10.5.1" class="ltx_td ltx_align_left ltx_th ltx_th_row" style="padding:-2pt 4.3pt;">GPT-neo-125M</th>
<th id="S3.SS2.8.8.4.10.5.2" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 4.3pt;">12</th>
<td id="S3.SS2.8.8.4.10.5.3" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">125M</td>
<td id="S3.SS2.8.8.4.10.5.4" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">40.7</td>
<td id="S3.SS2.8.8.4.10.5.5" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">24.8</td>
<td id="S3.SS2.8.8.4.10.5.6" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;"><span id="S3.SS2.8.8.4.10.5.6.1" class="ltx_text ltx_font_bold">61.3</span></td>
<td id="S3.SS2.8.8.4.10.5.7" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">62.5</td>
<td id="S3.SS2.8.8.4.10.5.8" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">41.9</td>
<td id="S3.SS2.8.8.4.10.5.9" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">29.7</td>
<td id="S3.SS2.8.8.4.10.5.10" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">31.6</td>
<td id="S3.SS2.8.8.4.10.5.11" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">50.7</td>
<td id="S3.SS2.8.8.4.10.5.12" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">42.9</td>
</tr>
<tr id="S3.SS2.8.8.4.11.6" class="ltx_tr">
<th id="S3.SS2.8.8.4.11.6.1" class="ltx_td ltx_align_left ltx_th ltx_th_row" style="padding:-2pt 4.3pt;">Pythia-160M</th>
<th id="S3.SS2.8.8.4.11.6.2" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 4.3pt;">12</th>
<td id="S3.SS2.8.8.4.11.6.3" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">162M</td>
<td id="S3.SS2.8.8.4.11.6.4" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">40.0</td>
<td id="S3.SS2.8.8.4.11.6.5" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">25.3</td>
<td id="S3.SS2.8.8.4.11.6.6" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">59.5</td>
<td id="S3.SS2.8.8.4.11.6.7" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">62.0</td>
<td id="S3.SS2.8.8.4.11.6.8" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">41.5</td>
<td id="S3.SS2.8.8.4.11.6.9" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">29.9</td>
<td id="S3.SS2.8.8.4.11.6.10" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">31.2</td>
<td id="S3.SS2.8.8.4.11.6.11" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">50.9</td>
<td id="S3.SS2.8.8.4.11.6.12" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">42.5</td>
</tr>
<tr id="S3.SS2.8.8.4.12.7" class="ltx_tr">
<th id="S3.SS2.8.8.4.12.7.1" class="ltx_td ltx_align_left ltx_th ltx_th_row" style="padding:-2pt 4.3pt;">RWKV-169M</th>
<th id="S3.SS2.8.8.4.12.7.2" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 4.3pt;">12</th>
<td id="S3.SS2.8.8.4.12.7.3" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">169M</td>
<td id="S3.SS2.8.8.4.12.7.4" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">42.5</td>
<td id="S3.SS2.8.8.4.12.7.5" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">25.3</td>
<td id="S3.SS2.8.8.4.12.7.6" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">59.1</td>
<td id="S3.SS2.8.8.4.12.7.7" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">63.9</td>
<td id="S3.SS2.8.8.4.12.7.8" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">40.7</td>
<td id="S3.SS2.8.8.4.12.7.9" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">31.9</td>
<td id="S3.SS2.8.8.4.12.7.10" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">33.8</td>
<td id="S3.SS2.8.8.4.12.7.11" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">51.5</td>
<td id="S3.SS2.8.8.4.12.7.12" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">43.6</td>
</tr>
<tr id="S3.SS2.5.5.1.1" class="ltx_tr">
<th id="S3.SS2.5.5.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row" style="padding:-2pt 4.3pt;">
<span id="S3.SS2.5.5.1.1.1.1" class="ltx_text ltx_markedasmath ltx_font_typewriter">MobileLLM</span>-125M</th>
<th id="S3.SS2.5.5.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 4.3pt;">30</th>
<td id="S3.SS2.5.5.1.1.3" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">125M</td>
<td id="S3.SS2.5.5.1.1.4" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">43.9</td>
<td id="S3.SS2.5.5.1.1.5" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;"><span id="S3.SS2.5.5.1.1.5.1" class="ltx_text ltx_font_bold">27.1</span></td>
<td id="S3.SS2.5.5.1.1.6" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">60.2</td>
<td id="S3.SS2.5.5.1.1.7" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;"><span id="S3.SS2.5.5.1.1.7.1" class="ltx_text ltx_font_bold">65.3</span></td>
<td id="S3.SS2.5.5.1.1.8" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;"><span id="S3.SS2.5.5.1.1.8.1" class="ltx_text ltx_font_bold">42.4</span></td>
<td id="S3.SS2.5.5.1.1.9" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;"><span id="S3.SS2.5.5.1.1.9.1" class="ltx_text ltx_font_bold">38.9</span></td>
<td id="S3.SS2.5.5.1.1.10" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;"><span id="S3.SS2.5.5.1.1.10.1" class="ltx_text ltx_font_bold">39.5</span></td>
<td id="S3.SS2.5.5.1.1.11" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;"><span id="S3.SS2.5.5.1.1.11.1" class="ltx_text ltx_font_bold">53.1</span></td>
<td id="S3.SS2.5.5.1.1.12" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;"><span id="S3.SS2.5.5.1.1.12.1" class="ltx_text ltx_font_bold">46.3</span></td>
</tr>
<tr id="S3.SS2.6.6.2.2" class="ltx_tr">
<th id="S3.SS2.6.6.2.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row" style="padding:-2pt 4.3pt;">
<span id="S3.SS2.6.6.2.2.1.1" class="ltx_text ltx_markedasmath ltx_font_typewriter">MobileLLM-LS</span>-125M</th>
<th id="S3.SS2.6.6.2.2.2" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 4.3pt;">30</th>
<td id="S3.SS2.6.6.2.2.3" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">125M</td>
<td id="S3.SS2.6.6.2.2.4" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;"><span id="S3.SS2.6.6.2.2.4.1" class="ltx_text ltx_font_bold">45.8</span></td>
<td id="S3.SS2.6.6.2.2.5" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;"><span id="S3.SS2.6.6.2.2.5.1" class="ltx_text ltx_font_bold">28.7</span></td>
<td id="S3.SS2.6.6.2.2.6" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">60.4</td>
<td id="S3.SS2.6.6.2.2.7" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;"><span id="S3.SS2.6.6.2.2.7.1" class="ltx_text ltx_font_bold">65.7</span></td>
<td id="S3.SS2.6.6.2.2.8" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;"><span id="S3.SS2.6.6.2.2.8.1" class="ltx_text ltx_font_bold">42.9</span></td>
<td id="S3.SS2.6.6.2.2.9" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;"><span id="S3.SS2.6.6.2.2.9.1" class="ltx_text ltx_font_bold">39.5</span></td>
<td id="S3.SS2.6.6.2.2.10" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;"><span id="S3.SS2.6.6.2.2.10.1" class="ltx_text ltx_font_bold">41.1</span></td>
<td id="S3.SS2.6.6.2.2.11" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;"><span id="S3.SS2.6.6.2.2.11.1" class="ltx_text ltx_font_bold">52.1</span></td>
<td id="S3.SS2.6.6.2.2.12" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;"><span id="S3.SS2.6.6.2.2.12.1" class="ltx_text ltx_font_bold">47.0</span></td>
</tr>
<tr id="S3.SS2.8.8.4.13.8" class="ltx_tr">
<th id="S3.SS2.8.8.4.13.8.1" class="ltx_td ltx_align_left ltx_th ltx_th_row" style="padding:-2pt 4.3pt;">
<span id="S3.SS2.8.8.4.13.8.1.1" class="ltx_ERROR undefined">\hdashline</span>[0.8pt/1pt]
Cerebras-GPT-256M</th>
<th id="S3.SS2.8.8.4.13.8.2" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 4.3pt;">14</th>
<td id="S3.SS2.8.8.4.13.8.3" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">256M</td>
<td id="S3.SS2.8.8.4.13.8.4" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">37.9</td>
<td id="S3.SS2.8.8.4.13.8.5" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">23.2</td>
<td id="S3.SS2.8.8.4.13.8.6" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">60.3</td>
<td id="S3.SS2.8.8.4.13.8.7" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">61.4</td>
<td id="S3.SS2.8.8.4.13.8.8" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">40.6</td>
<td id="S3.SS2.8.8.4.13.8.9" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">28.3</td>
<td id="S3.SS2.8.8.4.13.8.10" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">31.8</td>
<td id="S3.SS2.8.8.4.13.8.11" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">50.5</td>
<td id="S3.SS2.8.8.4.13.8.12" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">41.8</td>
</tr>
<tr id="S3.SS2.8.8.4.14.9" class="ltx_tr">
<th id="S3.SS2.8.8.4.14.9.1" class="ltx_td ltx_align_left ltx_th ltx_th_row" style="padding:-2pt 4.3pt;">OPT-350M</th>
<th id="S3.SS2.8.8.4.14.9.2" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 4.3pt;">24</th>
<td id="S3.SS2.8.8.4.14.9.3" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">331M</td>
<td id="S3.SS2.8.8.4.14.9.4" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">41.9</td>
<td id="S3.SS2.8.8.4.14.9.5" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">25.7</td>
<td id="S3.SS2.8.8.4.14.9.6" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">54.0</td>
<td id="S3.SS2.8.8.4.14.9.7" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">64.8</td>
<td id="S3.SS2.8.8.4.14.9.8" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">42.6</td>
<td id="S3.SS2.8.8.4.14.9.9" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">36.2</td>
<td id="S3.SS2.8.8.4.14.9.10" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">33.3</td>
<td id="S3.SS2.8.8.4.14.9.11" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">52.4</td>
<td id="S3.SS2.8.8.4.14.9.12" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">43.9</td>
</tr>
<tr id="S3.SS2.8.8.4.15.10" class="ltx_tr">
<th id="S3.SS2.8.8.4.15.10.1" class="ltx_td ltx_align_left ltx_th ltx_th_row" style="padding:-2pt 4.3pt;">RWKV-430M</th>
<th id="S3.SS2.8.8.4.15.10.2" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 4.3pt;">24</th>
<td id="S3.SS2.8.8.4.15.10.3" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">430M</td>
<td id="S3.SS2.8.8.4.15.10.4" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">48.9</td>
<td id="S3.SS2.8.8.4.15.10.5" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">32.0</td>
<td id="S3.SS2.8.8.4.15.10.6" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">53.4</td>
<td id="S3.SS2.8.8.4.15.10.7" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">68.1</td>
<td id="S3.SS2.8.8.4.15.10.8" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">43.6</td>
<td id="S3.SS2.8.8.4.15.10.9" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">40.6</td>
<td id="S3.SS2.8.8.4.15.10.10" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">37.8</td>
<td id="S3.SS2.8.8.4.15.10.11" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">51.6</td>
<td id="S3.SS2.8.8.4.15.10.12" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">47.0</td>
</tr>
<tr id="S3.SS2.8.8.4.16.11" class="ltx_tr">
<th id="S3.SS2.8.8.4.16.11.1" class="ltx_td ltx_align_left ltx_th ltx_th_row" style="padding:-2pt 4.3pt;">Pythia-410M</th>
<th id="S3.SS2.8.8.4.16.11.2" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 4.3pt;">24</th>
<td id="S3.SS2.8.8.4.16.11.3" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">405M</td>
<td id="S3.SS2.8.8.4.16.11.4" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">47.1</td>
<td id="S3.SS2.8.8.4.16.11.5" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">30.3</td>
<td id="S3.SS2.8.8.4.16.11.6" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">55.3</td>
<td id="S3.SS2.8.8.4.16.11.7" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">67.2</td>
<td id="S3.SS2.8.8.4.16.11.8" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">43.1</td>
<td id="S3.SS2.8.8.4.16.11.9" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">40.1</td>
<td id="S3.SS2.8.8.4.16.11.10" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">36.2</td>
<td id="S3.SS2.8.8.4.16.11.11" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">53.4</td>
<td id="S3.SS2.8.8.4.16.11.12" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">46.6</td>
</tr>
<tr id="S3.SS2.8.8.4.17.12" class="ltx_tr">
<th id="S3.SS2.8.8.4.17.12.1" class="ltx_td ltx_align_left ltx_th ltx_th_row" style="padding:-2pt 4.3pt;">BLOOM-560M</th>
<th id="S3.SS2.8.8.4.17.12.2" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 4.3pt;">24</th>
<td id="S3.SS2.8.8.4.17.12.3" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">559M</td>
<td id="S3.SS2.8.8.4.17.12.4" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">43.7</td>
<td id="S3.SS2.8.8.4.17.12.5" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">27.5</td>
<td id="S3.SS2.8.8.4.17.12.6" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">53.7</td>
<td id="S3.SS2.8.8.4.17.12.7" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">65.1</td>
<td id="S3.SS2.8.8.4.17.12.8" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">42.5</td>
<td id="S3.SS2.8.8.4.17.12.9" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">36.5</td>
<td id="S3.SS2.8.8.4.17.12.10" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">32.6</td>
<td id="S3.SS2.8.8.4.17.12.11" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">52.2</td>
<td id="S3.SS2.8.8.4.17.12.12" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">44.2</td>
</tr>
<tr id="S3.SS2.8.8.4.18.13" class="ltx_tr">
<th id="S3.SS2.8.8.4.18.13.1" class="ltx_td ltx_align_left ltx_th ltx_th_row" style="padding:-2pt 4.3pt;">Cerebras-GPT-590M</th>
<th id="S3.SS2.8.8.4.18.13.2" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 4.3pt;">18</th>
<td id="S3.SS2.8.8.4.18.13.3" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">590M</td>
<td id="S3.SS2.8.8.4.18.13.4" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">42.6</td>
<td id="S3.SS2.8.8.4.18.13.5" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">24.9</td>
<td id="S3.SS2.8.8.4.18.13.6" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">57.7</td>
<td id="S3.SS2.8.8.4.18.13.7" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">62.8</td>
<td id="S3.SS2.8.8.4.18.13.8" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">40.9</td>
<td id="S3.SS2.8.8.4.18.13.9" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">32.0</td>
<td id="S3.SS2.8.8.4.18.13.10" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">33.2</td>
<td id="S3.SS2.8.8.4.18.13.11" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">49.7</td>
<td id="S3.SS2.8.8.4.18.13.12" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">43.0</td>
</tr>
<tr id="S3.SS2.7.7.3.3" class="ltx_tr">
<th id="S3.SS2.7.7.3.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row" style="padding:-2pt 4.3pt;">
<span id="S3.SS2.7.7.3.3.1.1" class="ltx_text ltx_markedasmath ltx_font_typewriter">MobileLLM</span>-350M</th>
<th id="S3.SS2.7.7.3.3.2" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 4.3pt;">32</th>
<td id="S3.SS2.7.7.3.3.3" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">345M</td>
<td id="S3.SS2.7.7.3.3.4" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;"><span id="S3.SS2.7.7.3.3.4.1" class="ltx_text ltx_font_bold">53.8</span></td>
<td id="S3.SS2.7.7.3.3.5" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;"><span id="S3.SS2.7.7.3.3.5.1" class="ltx_text ltx_font_bold">33.5</span></td>
<td id="S3.SS2.7.7.3.3.6" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;"><span id="S3.SS2.7.7.3.3.6.1" class="ltx_text ltx_font_bold">62.4</span></td>
<td id="S3.SS2.7.7.3.3.7" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;"><span id="S3.SS2.7.7.3.3.7.1" class="ltx_text ltx_font_bold">68.6</span></td>
<td id="S3.SS2.7.7.3.3.8" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;"><span id="S3.SS2.7.7.3.3.8.1" class="ltx_text ltx_font_bold">44.7</span></td>
<td id="S3.SS2.7.7.3.3.9" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;"><span id="S3.SS2.7.7.3.3.9.1" class="ltx_text ltx_font_bold">49.6</span></td>
<td id="S3.SS2.7.7.3.3.10" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;"><span id="S3.SS2.7.7.3.3.10.1" class="ltx_text ltx_font_bold">40.0</span></td>
<td id="S3.SS2.7.7.3.3.11" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;"><span id="S3.SS2.7.7.3.3.11.1" class="ltx_text ltx_font_bold">57.6</span></td>
<td id="S3.SS2.7.7.3.3.12" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;"><span id="S3.SS2.7.7.3.3.12.1" class="ltx_text ltx_font_bold">51.3</span></td>
</tr>
<tr id="S3.SS2.8.8.4.4" class="ltx_tr">
<th id="S3.SS2.8.8.4.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb" style="padding:-2pt 4.3pt;">
<span id="S3.SS2.8.8.4.4.1.1" class="ltx_text ltx_markedasmath ltx_font_typewriter">MobileLLM-LS</span>-350M</th>
<th id="S3.SS2.8.8.4.4.2" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_bb" style="padding:-2pt 4.3pt;">32</th>
<td id="S3.SS2.8.8.4.4.3" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 4.3pt;">345M</td>
<td id="S3.SS2.8.8.4.4.4" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 4.3pt;"><span id="S3.SS2.8.8.4.4.4.1" class="ltx_text ltx_font_bold">54.4</span></td>
<td id="S3.SS2.8.8.4.4.5" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 4.3pt;"><span id="S3.SS2.8.8.4.4.5.1" class="ltx_text ltx_font_bold">32.5</span></td>
<td id="S3.SS2.8.8.4.4.6" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 4.3pt;"><span id="S3.SS2.8.8.4.4.6.1" class="ltx_text ltx_font_bold">62.8</span></td>
<td id="S3.SS2.8.8.4.4.7" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 4.3pt;"><span id="S3.SS2.8.8.4.4.7.1" class="ltx_text ltx_font_bold">69.8</span></td>
<td id="S3.SS2.8.8.4.4.8" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 4.3pt;"><span id="S3.SS2.8.8.4.4.8.1" class="ltx_text ltx_font_bold">44.1</span></td>
<td id="S3.SS2.8.8.4.4.9" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 4.3pt;"><span id="S3.SS2.8.8.4.4.9.1" class="ltx_text ltx_font_bold">50.6</span></td>
<td id="S3.SS2.8.8.4.4.10" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 4.3pt;"><span id="S3.SS2.8.8.4.4.10.1" class="ltx_text ltx_font_bold">45.8</span></td>
<td id="S3.SS2.8.8.4.4.11" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 4.3pt;"><span id="S3.SS2.8.8.4.4.11.1" class="ltx_text ltx_font_bold">57.2</span></td>
<td id="S3.SS2.8.8.4.4.12" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 4.3pt;"><span id="S3.SS2.8.8.4.4.12.1" class="ltx_text ltx_font_bold">52.1</span></td>
</tr>
</tbody>
</table>
</span></div>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="S3.SS2.21.21" class="ltx_table ltx_figure_panel">

<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table"><span id="S3.SS2.21.21.14.1.1" class="ltx_text" style="font-size:90%;">Table 4</span>: </span><span id="S3.SS2.21.21.15.2" class="ltx_text" style="font-size:90%;">Performance on Trivia QA and RACE datasets for question answering and reading comprehension tasks.</span></figcaption><div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_1">
<div id="S3.SS2.12.12.4" class="ltx_inline-block ltx_figure_panel ltx_transformed_outer" style="width:216.8pt;height:182.8pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(-51.7pt,43.6pt) scale(0.677138969357551,0.677138969357551) ;">
<table id="S3.SS2.12.12.4.4" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S3.SS2.12.12.4.4.5.1" class="ltx_tr">
<th id="S3.SS2.12.12.4.4.5.1.1" class="ltx_td ltx_th ltx_th_row ltx_border_tt" style="padding:-2pt 2.8pt;"></th>
<th id="S3.SS2.12.12.4.4.5.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 2.8pt;" colspan="3">
<span id="S3.SS2.12.12.4.4.5.1.2.1" class="ltx_text ltx_font_bold">TQA</span> (F1 score)</th>
<th id="S3.SS2.12.12.4.4.5.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 2.8pt;" colspan="2">
<span id="S3.SS2.12.12.4.4.5.1.3.1" class="ltx_text ltx_font_bold">RACE</span> (Acc)</th>
</tr>
<tr id="S3.SS2.12.12.4.4.6.2" class="ltx_tr">
<th id="S3.SS2.12.12.4.4.6.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row" style="padding:-2pt 2.8pt;"><span id="S3.SS2.12.12.4.4.6.2.1.1" class="ltx_text ltx_font_bold">Model</span></th>
<th id="S3.SS2.12.12.4.4.6.2.2" class="ltx_td ltx_align_center ltx_th ltx_th_column" style="padding:-2pt 2.8pt;">1-shot</th>
<th id="S3.SS2.12.12.4.4.6.2.3" class="ltx_td ltx_align_center ltx_th ltx_th_column" style="padding:-2pt 2.8pt;">5-shot</th>
<th id="S3.SS2.12.12.4.4.6.2.4" class="ltx_td ltx_align_center ltx_th ltx_th_column" style="padding:-2pt 2.8pt;">64-shot</th>
<th id="S3.SS2.12.12.4.4.6.2.5" class="ltx_td ltx_align_center ltx_th ltx_th_column" style="padding:-2pt 2.8pt;">middle</th>
<th id="S3.SS2.12.12.4.4.6.2.6" class="ltx_td ltx_align_center ltx_th ltx_th_column" style="padding:-2pt 2.8pt;">high</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S3.SS2.12.12.4.4.7.1" class="ltx_tr">
<th id="S3.SS2.12.12.4.4.7.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t" style="padding:-2pt 2.8pt;">Cerebras-GPT-111M</th>
<td id="S3.SS2.12.12.4.4.7.1.2" class="ltx_td ltx_align_center ltx_border_t" style="padding:-2pt 2.8pt;">1.9</td>
<td id="S3.SS2.12.12.4.4.7.1.3" class="ltx_td ltx_align_center ltx_border_t" style="padding:-2pt 2.8pt;">3.8</td>
<td id="S3.SS2.12.12.4.4.7.1.4" class="ltx_td ltx_align_center ltx_border_t" style="padding:-2pt 2.8pt;">4.4</td>
<td id="S3.SS2.12.12.4.4.7.1.5" class="ltx_td ltx_align_center ltx_border_t" style="padding:-2pt 2.8pt;">29.2</td>
<td id="S3.SS2.12.12.4.4.7.1.6" class="ltx_td ltx_align_center ltx_border_t" style="padding:-2pt 2.8pt;">24.3</td>
</tr>
<tr id="S3.SS2.12.12.4.4.8.2" class="ltx_tr">
<th id="S3.SS2.12.12.4.4.8.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">OPT-125M</th>
<td id="S3.SS2.12.12.4.4.8.2.2" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">8.7</td>
<td id="S3.SS2.12.12.4.4.8.2.3" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">9.6</td>
<td id="S3.SS2.12.12.4.4.8.2.4" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">8.2</td>
<td id="S3.SS2.12.12.4.4.8.2.5" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">34.7</td>
<td id="S3.SS2.12.12.4.4.8.2.6" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">27.5</td>
</tr>
<tr id="S3.SS2.12.12.4.4.9.3" class="ltx_tr">
<th id="S3.SS2.12.12.4.4.9.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">GPT-Neo-125M</th>
<td id="S3.SS2.12.12.4.4.9.3.2" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">8.0</td>
<td id="S3.SS2.12.12.4.4.9.3.3" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">7.9</td>
<td id="S3.SS2.12.12.4.4.9.3.4" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">5.0</td>
<td id="S3.SS2.12.12.4.4.9.3.5" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">34.7</td>
<td id="S3.SS2.12.12.4.4.9.3.6" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">27.0</td>
</tr>
<tr id="S3.SS2.12.12.4.4.10.4" class="ltx_tr">
<th id="S3.SS2.12.12.4.4.10.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">Pythia-160M</th>
<td id="S3.SS2.12.12.4.4.10.4.2" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">2.1</td>
<td id="S3.SS2.12.12.4.4.10.4.3" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">1.4</td>
<td id="S3.SS2.12.12.4.4.10.4.4" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">2.1</td>
<td id="S3.SS2.12.12.4.4.10.4.5" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">30.2</td>
<td id="S3.SS2.12.12.4.4.10.4.6" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">25.1</td>
</tr>
<tr id="S3.SS2.9.9.1.1.1" class="ltx_tr">
<th id="S3.SS2.9.9.1.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">
<span id="S3.SS2.9.9.1.1.1.1.1" class="ltx_text ltx_markedasmath ltx_font_typewriter">MobileLLM</span>-125M</th>
<td id="S3.SS2.9.9.1.1.1.2" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;"><span id="S3.SS2.9.9.1.1.1.2.1" class="ltx_text ltx_font_bold">13.9</span></td>
<td id="S3.SS2.9.9.1.1.1.3" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;"><span id="S3.SS2.9.9.1.1.1.3.1" class="ltx_text ltx_font_bold">14.3</span></td>
<td id="S3.SS2.9.9.1.1.1.4" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;"><span id="S3.SS2.9.9.1.1.1.4.1" class="ltx_text ltx_font_bold">12.5</span></td>
<td id="S3.SS2.9.9.1.1.1.5" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;"><span id="S3.SS2.9.9.1.1.1.5.1" class="ltx_text ltx_font_bold">39.7</span></td>
<td id="S3.SS2.9.9.1.1.1.6" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;"><span id="S3.SS2.9.9.1.1.1.6.1" class="ltx_text ltx_font_bold">28.9</span></td>
</tr>
<tr id="S3.SS2.10.10.2.2.2" class="ltx_tr">
<th id="S3.SS2.10.10.2.2.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">
<span id="S3.SS2.10.10.2.2.2.1.1" class="ltx_text ltx_markedasmath ltx_font_typewriter">MobileLLM-LS</span>-125M</th>
<td id="S3.SS2.10.10.2.2.2.2" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;"><span id="S3.SS2.10.10.2.2.2.2.1" class="ltx_text ltx_font_bold">14.2</span></td>
<td id="S3.SS2.10.10.2.2.2.3" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;"><span id="S3.SS2.10.10.2.2.2.3.1" class="ltx_text ltx_font_bold">14.8</span></td>
<td id="S3.SS2.10.10.2.2.2.4" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;"><span id="S3.SS2.10.10.2.2.2.4.1" class="ltx_text ltx_font_bold">14.6</span></td>
<td id="S3.SS2.10.10.2.2.2.5" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;"><span id="S3.SS2.10.10.2.2.2.5.1" class="ltx_text ltx_font_bold">40.7</span></td>
<td id="S3.SS2.10.10.2.2.2.6" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;"><span id="S3.SS2.10.10.2.2.2.6.1" class="ltx_text ltx_font_bold">29.6</span></td>
</tr>
<tr id="S3.SS2.12.12.4.4.11.5" class="ltx_tr">
<th id="S3.SS2.12.12.4.4.11.5.1" class="ltx_td ltx_align_left ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">
<span id="S3.SS2.12.12.4.4.11.5.1.1" class="ltx_ERROR undefined">\hdashline</span>[0.8pt/1pt]
Cerebras-GPT-256M</th>
<td id="S3.SS2.12.12.4.4.11.5.2" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">5.2</td>
<td id="S3.SS2.12.12.4.4.11.5.3" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">6.8</td>
<td id="S3.SS2.12.12.4.4.11.5.4" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">3.3</td>
<td id="S3.SS2.12.12.4.4.11.5.5" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">31.7</td>
<td id="S3.SS2.12.12.4.4.11.5.6" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">26.2</td>
</tr>
<tr id="S3.SS2.12.12.4.4.12.6" class="ltx_tr">
<th id="S3.SS2.12.12.4.4.12.6.1" class="ltx_td ltx_align_left ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">OPT-350M</th>
<td id="S3.SS2.12.12.4.4.12.6.2" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">11.0</td>
<td id="S3.SS2.12.12.4.4.12.6.3" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">12.3</td>
<td id="S3.SS2.12.12.4.4.12.6.4" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">10.4</td>
<td id="S3.SS2.12.12.4.4.12.6.5" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">37.1</td>
<td id="S3.SS2.12.12.4.4.12.6.6" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">28.0</td>
</tr>
<tr id="S3.SS2.12.12.4.4.13.7" class="ltx_tr">
<th id="S3.SS2.12.12.4.4.13.7.1" class="ltx_td ltx_align_left ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">Pythia-410M</th>
<td id="S3.SS2.12.12.4.4.13.7.2" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">12.4</td>
<td id="S3.SS2.12.12.4.4.13.7.3" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">13.8</td>
<td id="S3.SS2.12.12.4.4.13.7.4" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">12.8</td>
<td id="S3.SS2.12.12.4.4.13.7.5" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">39.1</td>
<td id="S3.SS2.12.12.4.4.13.7.6" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">29.7</td>
</tr>
<tr id="S3.SS2.12.12.4.4.14.8" class="ltx_tr">
<th id="S3.SS2.12.12.4.4.14.8.1" class="ltx_td ltx_align_left ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">BLOOM-560M</th>
<td id="S3.SS2.12.12.4.4.14.8.2" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">8.8</td>
<td id="S3.SS2.12.12.4.4.14.8.3" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">8.9</td>
<td id="S3.SS2.12.12.4.4.14.8.4" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">5.3</td>
<td id="S3.SS2.12.12.4.4.14.8.5" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">37.6</td>
<td id="S3.SS2.12.12.4.4.14.8.6" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">28.2</td>
</tr>
<tr id="S3.SS2.12.12.4.4.15.9" class="ltx_tr">
<th id="S3.SS2.12.12.4.4.15.9.1" class="ltx_td ltx_align_left ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">Cerebras-GPT-590M</th>
<td id="S3.SS2.12.12.4.4.15.9.2" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">6.4</td>
<td id="S3.SS2.12.12.4.4.15.9.3" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">9.1</td>
<td id="S3.SS2.12.12.4.4.15.9.4" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">4.9</td>
<td id="S3.SS2.12.12.4.4.15.9.5" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">34.6</td>
<td id="S3.SS2.12.12.4.4.15.9.6" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">27.4</td>
</tr>
<tr id="S3.SS2.11.11.3.3.3" class="ltx_tr">
<th id="S3.SS2.11.11.3.3.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">
<span id="S3.SS2.11.11.3.3.3.1.1" class="ltx_text ltx_markedasmath ltx_font_typewriter">MobileLLM</span>-350M</th>
<td id="S3.SS2.11.11.3.3.3.2" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;"><span id="S3.SS2.11.11.3.3.3.2.1" class="ltx_text ltx_font_bold">22.0</span></td>
<td id="S3.SS2.11.11.3.3.3.3" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;"><span id="S3.SS2.11.11.3.3.3.3.1" class="ltx_text ltx_font_bold">23.9</span></td>
<td id="S3.SS2.11.11.3.3.3.4" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;"><span id="S3.SS2.11.11.3.3.3.4.1" class="ltx_text ltx_font_bold">24.2</span></td>
<td id="S3.SS2.11.11.3.3.3.5" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;"><span id="S3.SS2.11.11.3.3.3.5.1" class="ltx_text ltx_font_bold">45.6</span></td>
<td id="S3.SS2.11.11.3.3.3.6" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;"><span id="S3.SS2.11.11.3.3.3.6.1" class="ltx_text ltx_font_bold">33.8</span></td>
</tr>
<tr id="S3.SS2.12.12.4.4.4" class="ltx_tr">
<th id="S3.SS2.12.12.4.4.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb" style="padding:-2pt 2.8pt;">
<span id="S3.SS2.12.12.4.4.4.1.1" class="ltx_text ltx_markedasmath ltx_font_typewriter">MobileLLM-LS</span>-350M</th>
<td id="S3.SS2.12.12.4.4.4.2" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 2.8pt;"><span id="S3.SS2.12.12.4.4.4.2.1" class="ltx_text ltx_font_bold">21.4</span></td>
<td id="S3.SS2.12.12.4.4.4.3" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 2.8pt;"><span id="S3.SS2.12.12.4.4.4.3.1" class="ltx_text ltx_font_bold">22.5</span></td>
<td id="S3.SS2.12.12.4.4.4.4" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 2.8pt;"><span id="S3.SS2.12.12.4.4.4.4.1" class="ltx_text ltx_font_bold">22.6</span></td>
<td id="S3.SS2.12.12.4.4.4.5" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 2.8pt;"><span id="S3.SS2.12.12.4.4.4.5.1" class="ltx_text ltx_font_bold">47.3</span></td>
<td id="S3.SS2.12.12.4.4.4.6" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 2.8pt;"><span id="S3.SS2.12.12.4.4.4.6.1" class="ltx_text ltx_font_bold">33.7</span></td>
</tr>
</tbody>
</table>
</span></div>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<p id="S3.SS2.21.21.16" class="ltx_p ltx_figure_panel">We compare the final performance on zero-shot common sense reasoning tasks, question answering and reading comprehension tasks. The results of baseline methods were evaluated using their open-source Hugging Face models to ensure consistent evaluation procedures.</p>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<p id="S3.SS2.18.18.10" class="ltx_p ltx_figure_panel"><span id="S3.SS2.18.18.10.1" class="ltx_text ltx_font_bold">Zero-shot Common Sense Reasoning</span> Table <a href="#S3.SS2" title="3.2 Main Results ‣ 3 Experiments ‣ 2.3 Layer Sharing ‣ 2 Improving Sub-billion Scale LLM Design ‣ MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.2</span></a> presents a comparison between our proposed model, <span id="S3.SS2.18.18.10.2" class="ltx_text ltx_markedasmath ltx_font_typewriter">MobileLLM</span>, and state-of-the-art sub-billion parameter models, including the early open-sourced LLMs, OPT <cite class="ltx_cite ltx_citemacro_citep">(Zhang et al., <a href="#bib.bib57" title="" class="ltx_ref">2022</a>)</cite>, BLOOM <cite class="ltx_cite ltx_citemacro_citep">(Scao et al., <a href="#bib.bib39" title="" class="ltx_ref">2022</a>)</cite>, and recent releases such as Galactica <cite class="ltx_cite ltx_citemacro_citep">(Taylor et al., <a href="#bib.bib44" title="" class="ltx_ref">2022</a>)</cite>, Cerebras <cite class="ltx_cite ltx_citemacro_citep">(Dey et al., <a href="#bib.bib11" title="" class="ltx_ref">2023</a>)</cite>, GPT-neo <cite class="ltx_cite ltx_citemacro_citep">(Black et al., <a href="#bib.bib4" title="" class="ltx_ref">2022</a>)</cite> as well as the LLM analyzing suite Pythia <cite class="ltx_cite ltx_citemacro_citep">(Biderman et al., <a href="#bib.bib2" title="" class="ltx_ref">2023</a>)</cite> and transformer variants RWKV <cite class="ltx_cite ltx_citemacro_citep">(Peng et al., <a href="#bib.bib34" title="" class="ltx_ref">2023</a>)</cite> on zero-shot common sense reasoning tasks.
For the 125M model size, <span id="S3.SS2.18.18.10.3" class="ltx_text ltx_markedasmath ltx_font_typewriter">MobileLLM</span> favorably outperforms previous models such as OPT, GPT-Neo, and Calactica at the same model size by a significant margin. Additionally, <span id="S3.SS2.18.18.10.4" class="ltx_text ltx_markedasmath ltx_font_typewriter">MobileLLM</span>-125M achieves 3.8 points and 2.7 points higher accuracy than Pythia-160M and RWKV-169M while being 22% and 26% smaller, respectively. Furthermore, incorporating layer-sharing in <span id="S3.SS2.18.18.10.5" class="ltx_text ltx_markedasmath ltx_font_typewriter">MobileLLM-LS</span>-125M results in an additional 0.7 points improvement in accuracy. It is noteworthy that <span id="S3.SS2.18.18.10.6" class="ltx_text ltx_markedasmath ltx_font_typewriter">MobileLLM-LS</span>-125M achieves comparable or even higher results than most previous 350M models.
In the 350M model size category, <span id="S3.SS2.18.18.10.7" class="ltx_text ltx_markedasmath ltx_font_typewriter">MobileLLM</span> surpasses previous state-of-the-art models by more than 4 points with comparable or smaller model size.</p>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<p id="S3.SS2.21.21.13" class="ltx_p ltx_figure_panel"><span id="S3.SS2.21.21.13.1" class="ltx_text ltx_font_bold">Question Answering and Reading Comprehension</span> We evaluate pre-trained models on the TQA question answering benchmark <cite class="ltx_cite ltx_citemacro_citep">(Joshi et al., <a href="#bib.bib21" title="" class="ltx_ref">2017</a>)</cite> and RACE reading comprehension benchmark <cite class="ltx_cite ltx_citemacro_citep">(Lai et al., <a href="#bib.bib27" title="" class="ltx_ref">2017</a>)</cite>. We follow the evalutation setup from <cite class="ltx_cite ltx_citemacro_citep">(Touvron et al., <a href="#bib.bib46" title="" class="ltx_ref">2023a</a>)</cite> and report the results in Table <a href="#S3.SS2" title="3.2 Main Results ‣ 3 Experiments ‣ 2.3 Layer Sharing ‣ 2 Improving Sub-billion Scale LLM Design ‣ MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.2</span></a>. Comparing models of 125M size, <span id="S3.SS2.21.21.13.2" class="ltx_text ltx_markedasmath ltx_font_typewriter">MobileLLM</span>-125M demonstrates a noteworthy improvement of over 4.3 points on the TQA benchmark contrast to its predecessor. Moreover, the <span id="S3.SS2.21.21.13.3" class="ltx_text ltx_markedasmath ltx_font_typewriter">MobileLLM</span>-350M model exhibits a substantial performance increase of approximately 10 points compared to other 350M-sized models. For the reading comprehension tasks, <span id="S3.SS2.21.21.13.4" class="ltx_text ltx_markedasmath ltx_font_typewriter">MobileLLM</span> model family also exhibit significant higher scores than preceding sub-billion parameter models.</p>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<section id="S3.SS3" class="ltx_subsection ltx_figure_panel">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.3 </span>Downstream Tasks</h3>

<div id="S3.SS3.p1" class="ltx_para">
<p id="S3.SS3.p1.1" class="ltx_p">To validate the effectiveness of sub-billion scale models for on-device applications, we assess their performance in two crucial on-device tasks: Chat and API calling.</p>
</div>
<section id="S3.SS3.SSS1" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">3.3.1 </span>Chat</h4>

<div id="S3.SS3.SSS1.p1" class="ltx_para">
<p id="S3.SS3.SSS1.p1.5" class="ltx_p">We fine-tune <span id="S3.SS3.SSS1.p1.5.1" class="ltx_text ltx_markedasmath ltx_font_typewriter">MobileLLM</span> models, as well as previous state-of-the-art (SoTA) models sourced from HuggingFace checkpoints for chat-based tasks and evaluate them under identical settings to ensure consistency. We evaluate on two benchmarks: AlpacaEval <cite class="ltx_cite ltx_citemacro_citep">(Li et al., <a href="#bib.bib28" title="" class="ltx_ref">2023</a>)</cite>, a single-run chat benchmark, and MT-Bench <cite class="ltx_cite ltx_citemacro_citep">(Zheng et al., <a href="#bib.bib58" title="" class="ltx_ref">2023</a>)</cite>, a multi-run chat benchmark. The results presented in Table <a href="#S3.SS3.SSS1" title="3.3.1 Chat ‣ 3.3 Downstream Tasks ‣ 3.2 Main Results ‣ 3 Experiments ‣ 2.3 Layer Sharing ‣ 2 Improving Sub-billion Scale LLM Design ‣ MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.3.1</span></a> showcases that <span id="S3.SS3.SSS1.p1.5.2" class="ltx_text ltx_markedasmath ltx_font_typewriter">MobileLLM</span> models significantly outperforms previous state-of-the-art sub-billion scale models, surpassing even models with 1 billion parameters. Notably, <span id="S3.SS3.SSS1.p1.5.3" class="ltx_text ltx_markedasmath ltx_font_typewriter">MobileLLM-LS</span>-350M achieves a remarkable win rate of 48.2% when compared to the baseline GPT-3 model (text-davinci-001). Considering the self-win rate of GPT-3 is 50%, is noteworthy that <span id="S3.SS3.SSS1.p1.5.4" class="ltx_text ltx_markedasmath ltx_font_typewriter">MobileLLM-LS</span>-350M obtains comparable chat performance as this baseline model. Visualizations of chat examples in the appendix also underscore the impressive quality of responses generated by <span id="S3.SS3.SSS1.p1.5.5" class="ltx_text ltx_markedasmath ltx_font_typewriter">MobileLLM</span> models.</p>
</div>
<figure id="S3.SS3.SSS1.6" class="ltx_table">

<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table"><span id="S3.SS3.SSS1.6.7.1.1" class="ltx_text" style="font-size:90%;">Table 5</span>: </span><span id="S3.SS3.SSS1.6.8.2" class="ltx_text" style="font-size:90%;">Benchmark results on AlpacaEval (Evaluator: GPT-4; Reference model: text-davinci-001) and MT-Bench.</span></figcaption><div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_2">
<div id="S3.SS3.SSS1.6.6" class="ltx_inline-block ltx_figure_panel ltx_transformed_outer" style="width:208.1pt;height:254pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(-30.9pt,37.7pt) scale(0.771243426522247,0.771243426522247) ;">
<table id="S3.SS3.SSS1.6.6.6" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="S3.SS3.SSS1.2.2.2.2" class="ltx_tr">
<th id="S3.SS3.SSS1.2.2.2.2.3" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_tt" style="padding:-2pt 2.8pt;"><span id="S3.SS3.SSS1.2.2.2.2.3.1" class="ltx_text ltx_font_bold">Model</span></th>
<td id="S3.SS3.SSS1.1.1.1.1.1" class="ltx_td ltx_align_center ltx_border_tt" style="padding:-2pt 2.8pt;">
<table id="S3.SS3.SSS1.1.1.1.1.1.1" class="ltx_tabular ltx_align_middle">
<tr id="S3.SS3.SSS1.1.1.1.1.1.1.1" class="ltx_tr">
<td id="S3.SS3.SSS1.1.1.1.1.1.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center" style="padding:-2pt 2.8pt;"><span id="S3.SS3.SSS1.1.1.1.1.1.1.1.1.1" class="ltx_text ltx_font_bold">MT-Bench<math id="S3.SS3.SSS1.1.1.1.1.1.1.1.1.1.m1.1" class="ltx_Math" alttext="{}_{\text{(score)}}" display="inline"><semantics id="S3.SS3.SSS1.1.1.1.1.1.1.1.1.1.m1.1a"><msub id="S3.SS3.SSS1.1.1.1.1.1.1.1.1.1.m1.1.1" xref="S3.SS3.SSS1.1.1.1.1.1.1.1.1.1.m1.1.1.cmml"><mi id="S3.SS3.SSS1.1.1.1.1.1.1.1.1.1.m1.1.1a" xref="S3.SS3.SSS1.1.1.1.1.1.1.1.1.1.m1.1.1.cmml"></mi><mtext id="S3.SS3.SSS1.1.1.1.1.1.1.1.1.1.m1.1.1.1.1" xref="S3.SS3.SSS1.1.1.1.1.1.1.1.1.1.m1.1.1.1.1a.cmml">(score)</mtext></msub><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS1.1.1.1.1.1.1.1.1.1.m1.1b"><apply id="S3.SS3.SSS1.1.1.1.1.1.1.1.1.1.m1.1.1.cmml" xref="S3.SS3.SSS1.1.1.1.1.1.1.1.1.1.m1.1.1"><ci id="S3.SS3.SSS1.1.1.1.1.1.1.1.1.1.m1.1.1.1.1a.cmml" xref="S3.SS3.SSS1.1.1.1.1.1.1.1.1.1.m1.1.1.1.1"><mtext mathsize="70%" id="S3.SS3.SSS1.1.1.1.1.1.1.1.1.1.m1.1.1.1.1.cmml" xref="S3.SS3.SSS1.1.1.1.1.1.1.1.1.1.m1.1.1.1.1">(score)</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS1.1.1.1.1.1.1.1.1.1.m1.1c">{}_{\text{(score)}}</annotation></semantics></math></span></td>
</tr>
</table>
</td>
<td id="S3.SS3.SSS1.2.2.2.2.2" class="ltx_td ltx_align_center ltx_border_tt" style="padding:-2pt 2.8pt;">
<table id="S3.SS3.SSS1.2.2.2.2.2.1" class="ltx_tabular ltx_align_middle">
<tr id="S3.SS3.SSS1.2.2.2.2.2.1.1" class="ltx_tr">
<td id="S3.SS3.SSS1.2.2.2.2.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center" style="padding:-2pt 2.8pt;"><span id="S3.SS3.SSS1.2.2.2.2.2.1.1.1.1" class="ltx_text ltx_font_bold">Alpaca Eval<math id="S3.SS3.SSS1.2.2.2.2.2.1.1.1.1.m1.1" class="ltx_Math" alttext="{}_{\text{(win \%)}}" display="inline"><semantics id="S3.SS3.SSS1.2.2.2.2.2.1.1.1.1.m1.1a"><msub id="S3.SS3.SSS1.2.2.2.2.2.1.1.1.1.m1.1.1" xref="S3.SS3.SSS1.2.2.2.2.2.1.1.1.1.m1.1.1.cmml"><mi id="S3.SS3.SSS1.2.2.2.2.2.1.1.1.1.m1.1.1a" xref="S3.SS3.SSS1.2.2.2.2.2.1.1.1.1.m1.1.1.cmml"></mi><mtext id="S3.SS3.SSS1.2.2.2.2.2.1.1.1.1.m1.1.1.1.1" xref="S3.SS3.SSS1.2.2.2.2.2.1.1.1.1.m1.1.1.1.1a.cmml">(win %)</mtext></msub><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS1.2.2.2.2.2.1.1.1.1.m1.1b"><apply id="S3.SS3.SSS1.2.2.2.2.2.1.1.1.1.m1.1.1.cmml" xref="S3.SS3.SSS1.2.2.2.2.2.1.1.1.1.m1.1.1"><ci id="S3.SS3.SSS1.2.2.2.2.2.1.1.1.1.m1.1.1.1.1a.cmml" xref="S3.SS3.SSS1.2.2.2.2.2.1.1.1.1.m1.1.1.1.1"><mtext mathsize="70%" id="S3.SS3.SSS1.2.2.2.2.2.1.1.1.1.m1.1.1.1.1.cmml" xref="S3.SS3.SSS1.2.2.2.2.2.1.1.1.1.m1.1.1.1.1">(win %)</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS1.2.2.2.2.2.1.1.1.1.m1.1c">{}_{\text{(win \%)}}</annotation></semantics></math></span></td>
</tr>
</table>
</td>
</tr>
<tr id="S3.SS3.SSS1.6.6.6.7.1" class="ltx_tr">
<th id="S3.SS3.SSS1.6.6.6.7.1.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_t" style="padding:-2pt 2.8pt;" colspan="3"><span id="S3.SS3.SSS1.6.6.6.7.1.1.1" class="ltx_text ltx_font_italic">number of parameters &lt; 200M</span></th>
</tr>
<tr id="S3.SS3.SSS1.6.6.6.8.2" class="ltx_tr">
<th id="S3.SS3.SSS1.6.6.6.8.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">
<span id="S3.SS3.SSS1.6.6.6.8.2.1.1" class="ltx_ERROR undefined">\hdashline</span>[0.8pt/1pt]
OPT-125M</th>
<td id="S3.SS3.SSS1.6.6.6.8.2.2" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">1.21</td>
<td id="S3.SS3.SSS1.6.6.6.8.2.3" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">3.91</td>
</tr>
<tr id="S3.SS3.SSS1.6.6.6.9.3" class="ltx_tr">
<th id="S3.SS3.SSS1.6.6.6.9.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">GPT-Neo-125M</th>
<td id="S3.SS3.SSS1.6.6.6.9.3.2" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">1.06</td>
<td id="S3.SS3.SSS1.6.6.6.9.3.3" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">1.01</td>
</tr>
<tr id="S3.SS3.SSS1.6.6.6.10.4" class="ltx_tr">
<th id="S3.SS3.SSS1.6.6.6.10.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">Pythia-160M</th>
<td id="S3.SS3.SSS1.6.6.6.10.4.2" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">1.01</td>
<td id="S3.SS3.SSS1.6.6.6.10.4.3" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">0.63</td>
</tr>
<tr id="S3.SS3.SSS1.3.3.3.3" class="ltx_tr">
<th id="S3.SS3.SSS1.3.3.3.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">
<span id="S3.SS3.SSS1.3.3.3.3.1.1" class="ltx_text ltx_markedasmath ltx_font_typewriter">MobileLLM</span>-125M</th>
<td id="S3.SS3.SSS1.3.3.3.3.2" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;"><span id="S3.SS3.SSS1.3.3.3.3.2.1" class="ltx_text ltx_font_bold">2.33</span></td>
<td id="S3.SS3.SSS1.3.3.3.3.3" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;"><span id="S3.SS3.SSS1.3.3.3.3.3.1" class="ltx_text ltx_font_bold">24.07</span></td>
</tr>
<tr id="S3.SS3.SSS1.4.4.4.4" class="ltx_tr">
<th id="S3.SS3.SSS1.4.4.4.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">
<span id="S3.SS3.SSS1.4.4.4.4.1.1" class="ltx_text ltx_markedasmath ltx_font_typewriter">MobileLLM-LS</span>-125M</th>
<td id="S3.SS3.SSS1.4.4.4.4.2" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;"><span id="S3.SS3.SSS1.4.4.4.4.2.1" class="ltx_text ltx_font_bold">2.52</span></td>
<td id="S3.SS3.SSS1.4.4.4.4.3" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;"><span id="S3.SS3.SSS1.4.4.4.4.3.1" class="ltx_text ltx_font_bold">23.79</span></td>
</tr>
<tr id="S3.SS3.SSS1.6.6.6.11.5" class="ltx_tr">
<th id="S3.SS3.SSS1.6.6.6.11.5.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_t" style="padding:-2pt 2.8pt;" colspan="3"><span id="S3.SS3.SSS1.6.6.6.11.5.1.1" class="ltx_text ltx_font_italic">200M &lt; number of parameters &lt; 1B</span></th>
</tr>
<tr id="S3.SS3.SSS1.6.6.6.12.6" class="ltx_tr">
<th id="S3.SS3.SSS1.6.6.6.12.6.1" class="ltx_td ltx_align_left ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">
<span id="S3.SS3.SSS1.6.6.6.12.6.1.1" class="ltx_ERROR undefined">\hdashline</span>[0.8pt/1pt]
OPT-350M</th>
<td id="S3.SS3.SSS1.6.6.6.12.6.2" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">1.37</td>
<td id="S3.SS3.SSS1.6.6.6.12.6.3" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">6.80</td>
</tr>
<tr id="S3.SS3.SSS1.6.6.6.13.7" class="ltx_tr">
<th id="S3.SS3.SSS1.6.6.6.13.7.1" class="ltx_td ltx_align_left ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">Pythia-410M</th>
<td id="S3.SS3.SSS1.6.6.6.13.7.2" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">1.62</td>
<td id="S3.SS3.SSS1.6.6.6.13.7.3" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">13.87</td>
</tr>
<tr id="S3.SS3.SSS1.6.6.6.14.8" class="ltx_tr">
<th id="S3.SS3.SSS1.6.6.6.14.8.1" class="ltx_td ltx_align_left ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">BLOOM-560M</th>
<td id="S3.SS3.SSS1.6.6.6.14.8.2" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">1.73</td>
<td id="S3.SS3.SSS1.6.6.6.14.8.3" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">10.29</td>
</tr>
<tr id="S3.SS3.SSS1.5.5.5.5" class="ltx_tr">
<th id="S3.SS3.SSS1.5.5.5.5.1" class="ltx_td ltx_align_left ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">
<span id="S3.SS3.SSS1.5.5.5.5.1.1" class="ltx_text ltx_markedasmath ltx_font_typewriter">MobileLLM</span>-350M</th>
<td id="S3.SS3.SSS1.5.5.5.5.2" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;"><span id="S3.SS3.SSS1.5.5.5.5.2.1" class="ltx_text ltx_font_bold">3.28</span></td>
<td id="S3.SS3.SSS1.5.5.5.5.3" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;"><span id="S3.SS3.SSS1.5.5.5.5.3.1" class="ltx_text ltx_font_bold">47.08</span></td>
</tr>
<tr id="S3.SS3.SSS1.6.6.6.6" class="ltx_tr">
<th id="S3.SS3.SSS1.6.6.6.6.1" class="ltx_td ltx_align_left ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">
<span id="S3.SS3.SSS1.6.6.6.6.1.1" class="ltx_text ltx_markedasmath ltx_font_typewriter">MobileLLM-LS</span>-350M</th>
<td id="S3.SS3.SSS1.6.6.6.6.2" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;"><span id="S3.SS3.SSS1.6.6.6.6.2.1" class="ltx_text ltx_font_bold">3.16</span></td>
<td id="S3.SS3.SSS1.6.6.6.6.3" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;"><span id="S3.SS3.SSS1.6.6.6.6.3.1" class="ltx_text ltx_font_bold">48.20</span></td>
</tr>
<tr id="S3.SS3.SSS1.6.6.6.15.9" class="ltx_tr">
<th id="S3.SS3.SSS1.6.6.6.15.9.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_t" style="padding:-2pt 2.8pt;" colspan="3"><span id="S3.SS3.SSS1.6.6.6.15.9.1.1" class="ltx_text ltx_font_italic">number of parameters &gt; 1B</span></th>
</tr>
<tr id="S3.SS3.SSS1.6.6.6.16.10" class="ltx_tr">
<th id="S3.SS3.SSS1.6.6.6.16.10.1" class="ltx_td ltx_align_left ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">
<span id="S3.SS3.SSS1.6.6.6.16.10.1.1" class="ltx_ERROR undefined">\hdashline</span>[0.8pt/1pt]
Pythia-1B</th>
<td id="S3.SS3.SSS1.6.6.6.16.10.2" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">1.70</td>
<td id="S3.SS3.SSS1.6.6.6.16.10.3" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">16.62</td>
</tr>
<tr id="S3.SS3.SSS1.6.6.6.17.11" class="ltx_tr">
<th id="S3.SS3.SSS1.6.6.6.17.11.1" class="ltx_td ltx_align_left ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">BLOOM-1.1B</th>
<td id="S3.SS3.SSS1.6.6.6.17.11.2" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">2.37</td>
<td id="S3.SS3.SSS1.6.6.6.17.11.3" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">19.90</td>
</tr>
<tr id="S3.SS3.SSS1.6.6.6.18.12" class="ltx_tr">
<th id="S3.SS3.SSS1.6.6.6.18.12.1" class="ltx_td ltx_align_left ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">Falcon-1.3B</th>
<td id="S3.SS3.SSS1.6.6.6.18.12.2" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">2.54</td>
<td id="S3.SS3.SSS1.6.6.6.18.12.3" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">30.38</td>
</tr>
<tr id="S3.SS3.SSS1.6.6.6.19.13" class="ltx_tr">
<th id="S3.SS3.SSS1.6.6.6.19.13.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb" style="padding:-2pt 2.8pt;">OPT-1.3B</th>
<td id="S3.SS3.SSS1.6.6.6.19.13.2" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 2.8pt;">2.24</td>
<td id="S3.SS3.SSS1.6.6.6.19.13.3" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 2.8pt;">38.84</td>
</tr>
</tbody>
</table>
</span></div>
</div>
<div class="ltx_flex_cell ltx_flex_size_2">
<section id="S3.SS3.SSS2" class="ltx_subsubsection ltx_figure_panel">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">3.3.2 </span>API Calling</h4>

<div id="S3.SS3.SSS2.p1" class="ltx_para">
<p id="S3.SS3.SSS2.p1.1" class="ltx_p">API calling is a common on-device application, particularly in collaboration with audio-to-text models for assistant functionalities. Leveraging LLMs for API calling involves the conversion of natural language inputs into JSON configurations to invoke corresponding APIs<span id="footnote6" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">6</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">6</sup><span class="ltx_tag ltx_tag_note">6</span>https://platform.openai.com/docs/guides/function-calling</span></span></span>. For instance, given the input <span id="S3.SS3.SSS2.p1.1.1" class="ltx_text ltx_font_italic">"Help me set an alarm at 7:30 AM"</span> the model outputs {<span id="S3.SS3.SSS2.p1.1.2" class="ltx_text ltx_font_italic">API: "alarm(time="7:30 am")"</span>}. Additionally, the model generates an agent response: <span id="S3.SS3.SSS2.p1.1.3" class="ltx_text ltx_font_italic">"Sure! Your alarm is set to 7:30 AM."</span></p>
</div>
<div id="S3.SS3.SSS2.p2" class="ltx_para">
<p id="S3.SS3.SSS2.p2.1" class="ltx_p">To adapt LLMs for this task, we create a synthetic dataset with 5000 training samples and 2500 testing samples. Each sample involves 8 conversation turns on average. Detailed examples of this dataset are provided in appendix. The pre-trained models undergoes fine-tuning on training set for 4 epochs, utilizing the Adam optimizer with a linear-decay learning rate starting at 2e-5 and a weight decay of 0.01.</p>
</div>
<div id="S3.SS3.SSS2.p3" class="ltx_para">
<p id="S3.SS3.SSS2.p3.3" class="ltx_p">Table <a href="#S3.SS3.SSS2" title="3.3.2 API Calling ‣ 3.3.1 Chat ‣ 3.3 Downstream Tasks ‣ 3.2 Main Results ‣ 3 Experiments ‣ 2.3 Layer Sharing ‣ 2 Improving Sub-billion Scale LLM Design ‣ MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.3.2</span></a> shows that <span id="S3.SS3.SSS2.p3.3.1" class="ltx_text ltx_markedasmath ltx_font_typewriter">MobileLLM</span>-350M demonstrates comparable intent and structure exact match scores to LLaMA-v2 7B, where high intent scores indicate correct prediction of the API user intends to call, while structural exact match scores reflects the proficiency in predicting content within API functions. Despite lower Rouge scores in <span id="S3.SS3.SSS2.p3.3.2" class="ltx_text ltx_markedasmath ltx_font_typewriter">MobileLLM</span>-350M compared to 7B models, it is crucial to note that API calling prioritize correct API invocation. The results suggest that certain common scenarios in on-device applications are not particularly challenging, and smaller models like <span id="S3.SS3.SSS2.p3.3.3" class="ltx_text ltx_markedasmath ltx_font_typewriter">MobileLLM</span>-350M can adeptly handle it.</p>
</div>
<figure id="S3.SS3.SSS2.7" class="ltx_table">

<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table"><span id="S3.SS3.SSS2.7.8.3.1" class="ltx_text" style="font-size:90%;">Table 6</span>: </span><span id="S3.SS3.SSS2.4.4.2" class="ltx_text" style="font-size:90%;">API calling evaluation score. EM<math id="S3.SS3.SSS2.3.3.1.m1.1" class="ltx_Math" alttext="{}_{\text{intent}}" display="inline"><semantics id="S3.SS3.SSS2.3.3.1.m1.1b"><msub id="S3.SS3.SSS2.3.3.1.m1.1.1" xref="S3.SS3.SSS2.3.3.1.m1.1.1.cmml"><mi id="S3.SS3.SSS2.3.3.1.m1.1.1b" xref="S3.SS3.SSS2.3.3.1.m1.1.1.cmml"></mi><mtext id="S3.SS3.SSS2.3.3.1.m1.1.1.1.1" xref="S3.SS3.SSS2.3.3.1.m1.1.1.1.1a.cmml">intent</mtext></msub><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS2.3.3.1.m1.1c"><apply id="S3.SS3.SSS2.3.3.1.m1.1.1.cmml" xref="S3.SS3.SSS2.3.3.1.m1.1.1"><ci id="S3.SS3.SSS2.3.3.1.m1.1.1.1.1a.cmml" xref="S3.SS3.SSS2.3.3.1.m1.1.1.1.1"><mtext mathsize="70%" id="S3.SS3.SSS2.3.3.1.m1.1.1.1.1.cmml" xref="S3.SS3.SSS2.3.3.1.m1.1.1.1.1">intent</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS2.3.3.1.m1.1d">{}_{\text{intent}}</annotation></semantics></math>/EM<math id="S3.SS3.SSS2.4.4.2.m2.1" class="ltx_Math" alttext="{}_{\text{structure}}" display="inline"><semantics id="S3.SS3.SSS2.4.4.2.m2.1b"><msub id="S3.SS3.SSS2.4.4.2.m2.1.1" xref="S3.SS3.SSS2.4.4.2.m2.1.1.cmml"><mi id="S3.SS3.SSS2.4.4.2.m2.1.1b" xref="S3.SS3.SSS2.4.4.2.m2.1.1.cmml"></mi><mtext id="S3.SS3.SSS2.4.4.2.m2.1.1.1.1" xref="S3.SS3.SSS2.4.4.2.m2.1.1.1.1a.cmml">structure</mtext></msub><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS2.4.4.2.m2.1c"><apply id="S3.SS3.SSS2.4.4.2.m2.1.1.cmml" xref="S3.SS3.SSS2.4.4.2.m2.1.1"><ci id="S3.SS3.SSS2.4.4.2.m2.1.1.1.1a.cmml" xref="S3.SS3.SSS2.4.4.2.m2.1.1.1.1"><mtext mathsize="70%" id="S3.SS3.SSS2.4.4.2.m2.1.1.1.1.cmml" xref="S3.SS3.SSS2.4.4.2.m2.1.1.1.1">structure</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS2.4.4.2.m2.1d">{}_{\text{structure}}</annotation></semantics></math> measures the exact match in API calling. R1/RL refers to Rouge-1/-L score measuring the quality of agent response.</span></figcaption><div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_2">
<div id="S3.SS3.SSS2.7.7" class="ltx_inline-block ltx_figure_panel ltx_transformed_outer" style="width:173.4pt;height:74.9pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(-44.6pt,19.3pt) scale(0.660268206727142,0.660268206727142) ;">
<table id="S3.SS3.SSS2.7.7.3" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="S3.SS3.SSS2.6.6.2.2" class="ltx_tr">
<th id="S3.SS3.SSS2.6.6.2.2.3" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_tt" style="padding:-2pt 2.8pt;"><span id="S3.SS3.SSS2.6.6.2.2.3.1" class="ltx_text ltx_font_bold">Model</span></th>
<td id="S3.SS3.SSS2.5.5.1.1.1" class="ltx_td ltx_align_center ltx_border_tt" style="padding:-2pt 2.8pt;">
<table id="S3.SS3.SSS2.5.5.1.1.1.1" class="ltx_tabular ltx_align_middle">
<tr id="S3.SS3.SSS2.5.5.1.1.1.1.1" class="ltx_tr">
<td id="S3.SS3.SSS2.5.5.1.1.1.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center" style="padding:-2pt 2.8pt;">EM<math id="S3.SS3.SSS2.5.5.1.1.1.1.1.1.m1.1" class="ltx_Math" alttext="{}_{\text{intent}}" display="inline"><semantics id="S3.SS3.SSS2.5.5.1.1.1.1.1.1.m1.1a"><msub id="S3.SS3.SSS2.5.5.1.1.1.1.1.1.m1.1.1" xref="S3.SS3.SSS2.5.5.1.1.1.1.1.1.m1.1.1.cmml"><mi id="S3.SS3.SSS2.5.5.1.1.1.1.1.1.m1.1.1a" xref="S3.SS3.SSS2.5.5.1.1.1.1.1.1.m1.1.1.cmml"></mi><mtext id="S3.SS3.SSS2.5.5.1.1.1.1.1.1.m1.1.1.1.1" xref="S3.SS3.SSS2.5.5.1.1.1.1.1.1.m1.1.1.1.1a.cmml">intent</mtext></msub><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS2.5.5.1.1.1.1.1.1.m1.1b"><apply id="S3.SS3.SSS2.5.5.1.1.1.1.1.1.m1.1.1.cmml" xref="S3.SS3.SSS2.5.5.1.1.1.1.1.1.m1.1.1"><ci id="S3.SS3.SSS2.5.5.1.1.1.1.1.1.m1.1.1.1.1a.cmml" xref="S3.SS3.SSS2.5.5.1.1.1.1.1.1.m1.1.1.1.1"><mtext mathsize="70%" id="S3.SS3.SSS2.5.5.1.1.1.1.1.1.m1.1.1.1.1.cmml" xref="S3.SS3.SSS2.5.5.1.1.1.1.1.1.m1.1.1.1.1">intent</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS2.5.5.1.1.1.1.1.1.m1.1c">{}_{\text{intent}}</annotation></semantics></math>
</td>
</tr>
</table>
</td>
<td id="S3.SS3.SSS2.6.6.2.2.2" class="ltx_td ltx_align_center ltx_border_tt" style="padding:-2pt 2.8pt;">
<table id="S3.SS3.SSS2.6.6.2.2.2.1" class="ltx_tabular ltx_align_middle">
<tr id="S3.SS3.SSS2.6.6.2.2.2.1.1" class="ltx_tr">
<td id="S3.SS3.SSS2.6.6.2.2.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center" style="padding:-2pt 2.8pt;">EM<math id="S3.SS3.SSS2.6.6.2.2.2.1.1.1.m1.1" class="ltx_Math" alttext="{}_{\text{structure}}" display="inline"><semantics id="S3.SS3.SSS2.6.6.2.2.2.1.1.1.m1.1a"><msub id="S3.SS3.SSS2.6.6.2.2.2.1.1.1.m1.1.1" xref="S3.SS3.SSS2.6.6.2.2.2.1.1.1.m1.1.1.cmml"><mi id="S3.SS3.SSS2.6.6.2.2.2.1.1.1.m1.1.1a" xref="S3.SS3.SSS2.6.6.2.2.2.1.1.1.m1.1.1.cmml"></mi><mtext id="S3.SS3.SSS2.6.6.2.2.2.1.1.1.m1.1.1.1.1" xref="S3.SS3.SSS2.6.6.2.2.2.1.1.1.m1.1.1.1.1a.cmml">structure</mtext></msub><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS2.6.6.2.2.2.1.1.1.m1.1b"><apply id="S3.SS3.SSS2.6.6.2.2.2.1.1.1.m1.1.1.cmml" xref="S3.SS3.SSS2.6.6.2.2.2.1.1.1.m1.1.1"><ci id="S3.SS3.SSS2.6.6.2.2.2.1.1.1.m1.1.1.1.1a.cmml" xref="S3.SS3.SSS2.6.6.2.2.2.1.1.1.m1.1.1.1.1"><mtext mathsize="70%" id="S3.SS3.SSS2.6.6.2.2.2.1.1.1.m1.1.1.1.1.cmml" xref="S3.SS3.SSS2.6.6.2.2.2.1.1.1.m1.1.1.1.1">structure</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS2.6.6.2.2.2.1.1.1.m1.1c">{}_{\text{structure}}</annotation></semantics></math>
</td>
</tr>
</table>
</td>
<td id="S3.SS3.SSS2.6.6.2.2.4" class="ltx_td ltx_align_center ltx_border_tt" style="padding:-2pt 2.8pt;">R1</td>
<td id="S3.SS3.SSS2.6.6.2.2.5" class="ltx_td ltx_align_center ltx_border_tt" style="padding:-2pt 2.8pt;">RL</td>
</tr>
<tr id="S3.SS3.SSS2.7.7.3.4.1" class="ltx_tr">
<th id="S3.SS3.SSS2.7.7.3.4.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">
<span id="S3.SS3.SSS2.7.7.3.4.1.1.1" class="ltx_ERROR undefined">\hdashline</span>[0.8pt/1pt]
OPT-350M</th>
<td id="S3.SS3.SSS2.7.7.3.4.1.2" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">56.1</td>
<td id="S3.SS3.SSS2.7.7.3.4.1.3" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">38.6</td>
<td id="S3.SS3.SSS2.7.7.3.4.1.4" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">37.1</td>
<td id="S3.SS3.SSS2.7.7.3.4.1.5" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">35.3</td>
</tr>
<tr id="S3.SS3.SSS2.7.7.3.5.2" class="ltx_tr">
<th id="S3.SS3.SSS2.7.7.3.5.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">Pythia-410M</th>
<td id="S3.SS3.SSS2.7.7.3.5.2.2" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">62.2</td>
<td id="S3.SS3.SSS2.7.7.3.5.2.3" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">44.7</td>
<td id="S3.SS3.SSS2.7.7.3.5.2.4" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">43.1</td>
<td id="S3.SS3.SSS2.7.7.3.5.2.5" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">41.1</td>
</tr>
<tr id="S3.SS3.SSS2.7.7.3.6.3" class="ltx_tr">
<th id="S3.SS3.SSS2.7.7.3.6.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">BLOOM-560M</th>
<td id="S3.SS3.SSS2.7.7.3.6.3.2" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;"><span id="S3.SS3.SSS2.7.7.3.6.3.2.1" class="ltx_text ltx_font_bold">64.7</span></td>
<td id="S3.SS3.SSS2.7.7.3.6.3.3" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">37.9</td>
<td id="S3.SS3.SSS2.7.7.3.6.3.4" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">36.9</td>
<td id="S3.SS3.SSS2.7.7.3.6.3.5" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">34.6</td>
</tr>
<tr id="S3.SS3.SSS2.7.7.3.3" class="ltx_tr">
<th id="S3.SS3.SSS2.7.7.3.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">
<span id="S3.SS3.SSS2.7.7.3.3.1.1" class="ltx_text ltx_markedasmath ltx_font_typewriter">MobileLLM</span>-350M</th>
<td id="S3.SS3.SSS2.7.7.3.3.2" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;"><span id="S3.SS3.SSS2.7.7.3.3.2.1" class="ltx_text ltx_font_bold">65.3</span></td>
<td id="S3.SS3.SSS2.7.7.3.3.3" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;"><span id="S3.SS3.SSS2.7.7.3.3.3.1" class="ltx_text ltx_font_bold">48.8</span></td>
<td id="S3.SS3.SSS2.7.7.3.3.4" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;"><span id="S3.SS3.SSS2.7.7.3.3.4.1" class="ltx_text ltx_font_bold">46.8</span></td>
<td id="S3.SS3.SSS2.7.7.3.3.5" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;"><span id="S3.SS3.SSS2.7.7.3.3.5.1" class="ltx_text ltx_font_bold">44.6</span></td>
</tr>
<tr id="S3.SS3.SSS2.7.7.3.7.4" class="ltx_tr">
<th id="S3.SS3.SSS2.7.7.3.7.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb" style="padding:-2pt 2.8pt;">
<span id="S3.SS3.SSS2.7.7.3.7.4.1.1" class="ltx_ERROR undefined">\hdashline</span>[0.8pt/1pt]
LLaMA-v2 7B</th>
<td id="S3.SS3.SSS2.7.7.3.7.4.2" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 2.8pt;">62.8</td>
<td id="S3.SS3.SSS2.7.7.3.7.4.3" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 2.8pt;"><span id="S3.SS3.SSS2.7.7.3.7.4.3.1" class="ltx_text ltx_font_bold">50.9</span></td>
<td id="S3.SS3.SSS2.7.7.3.7.4.4" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 2.8pt;"><span id="S3.SS3.SSS2.7.7.3.7.4.4.1" class="ltx_text ltx_font_bold">56.5</span></td>
<td id="S3.SS3.SSS2.7.7.3.7.4.5" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 2.8pt;"><span id="S3.SS3.SSS2.7.7.3.7.4.5.1" class="ltx_text ltx_font_bold">54.3</span></td>
</tr>
</tbody>
</table>
</span></div>
</div>
<div class="ltx_flex_cell ltx_flex_size_2">
<section id="S3.SS4" class="ltx_subsection ltx_figure_panel">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.4 </span>Compatibility with Quantization</h3>

<figure id="S3.F7" class="ltx_figure"><img src="/html/2402.14905/assets/x7.png" id="S3.F7.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="368" height="246" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S3.F7.2.1.1" class="ltx_text" style="font-size:90%;">Figure 7</span>: </span><span id="S3.F7.3.2" class="ltx_text" style="font-size:90%;">Comparison between BFloat16 model and 8-bit weight 8-bit activation post-training quantized model.</span></figcaption>
</figure>
<div id="S3.SS4.p1" class="ltx_para">
<p id="S3.SS4.p1.2" class="ltx_p">We further conduct per-token min-max post-training quantization (PTQ) experiments on both <span id="S3.SS4.p1.2.1" class="ltx_text ltx_markedasmath ltx_font_typewriter">MobileLLM</span> and <span id="S3.SS4.p1.2.2" class="ltx_text ltx_markedasmath ltx_font_typewriter">MobileLLM-LS</span> models with 125M and 350M model size trained on 0.25T tokens. Figure <a href="#S3.F7" title="Figure 7 ‣ 3.4 Compatibility with Quantization ‣ 3.3.2 API Calling ‣ 3.3.1 Chat ‣ 3.3 Downstream Tasks ‣ 3.2 Main Results ‣ 3 Experiments ‣ 2.3 Layer Sharing ‣ 2 Improving Sub-billion Scale LLM Design ‣ MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases" class="ltx_ref"><span class="ltx_text ltx_ref_tag">7</span></a> shows that employing W8A8 PTQ yields a modest accuracy reduction of less than 0.5 points and remains compatible with layer-sharing.</p>
</div>
<section id="S3.SS5" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.5 </span>Knowledge Distillation</h3>

<div id="S3.SS5.p1" class="ltx_para">
<p id="S3.SS5.p1.1" class="ltx_p">So far, we trained compact models from scratch using next tokens as hard labels.
We explored Knowledge Distillation (KD) of 125M and 350M models with LLAMA-v2 7B as a teacher.
Unfortunately KD increases training time (slowdown of <math id="S3.SS5.p1.1.m1.1" class="ltx_Math" alttext="2.6-3.2\times" display="inline"><semantics id="S3.SS5.p1.1.m1.1a"><mrow id="S3.SS5.p1.1.m1.1b"><mn id="S3.SS5.p1.1.m1.1.1" xref="S3.SS5.p1.1.m1.1.1.cmml">2.6</mn><mo id="S3.SS5.p1.1.m1.1.2" xref="S3.SS5.p1.1.m1.1.2.cmml">−</mo><mn id="S3.SS5.p1.1.m1.1.3" xref="S3.SS5.p1.1.m1.1.3.cmml">3.2</mn><mo lspace="0.222em" id="S3.SS5.p1.1.m1.1.4" xref="S3.SS5.p1.1.m1.1.4.cmml">×</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.SS5.p1.1.m1.1c"><cerror id="S3.SS5.p1.1.m1.1d"><csymbol cd="ambiguous" id="S3.SS5.p1.1.m1.1e">fragments</csymbol><cn type="float" id="S3.SS5.p1.1.m1.1.1.cmml" xref="S3.SS5.p1.1.m1.1.1">2.6</cn><minus id="S3.SS5.p1.1.m1.1.2.cmml" xref="S3.SS5.p1.1.m1.1.2"></minus><cn type="float" id="S3.SS5.p1.1.m1.1.3.cmml" xref="S3.SS5.p1.1.m1.1.3">3.2</cn><times id="S3.SS5.p1.1.m1.1.4.cmml" xref="S3.SS5.p1.1.m1.1.4"></times></cerror></annotation-xml><annotation encoding="application/x-tex" id="S3.SS5.p1.1.m1.1f">2.6-3.2\times</annotation></semantics></math>) and exhibits comparable or inferior accuracy to label-based training (details in appendix).</p>
</div>
<section id="S3.SS6" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.6 </span>On-device Profiling</h3>

<figure id="S3.SS6.6" class="ltx_table">

<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table"><span id="S3.SS6.6.9.2.1" class="ltx_text" style="font-size:90%;">Table 7</span>: </span><span id="S3.SS6.2.2.1" class="ltx_text" style="font-size:90%;">
Latency analysis of <span id="S3.SS6.2.2.1.1" class="ltx_text ltx_font_typewriter">MobileLLM</span>-125M (30 layers), <span id="S3.SS6.2.2.1.2" class="ltx_text ltx_font_typewriter">MobileLLM-LS</span>-125M (2<math id="S3.SS6.2.2.1.m1.1" class="ltx_Math" alttext="\times" display="inline"><semantics id="S3.SS6.2.2.1.m1.1b"><mo id="S3.SS6.2.2.1.m1.1.1" xref="S3.SS6.2.2.1.m1.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="S3.SS6.2.2.1.m1.1c"><times id="S3.SS6.2.2.1.m1.1.1.cmml" xref="S3.SS6.2.2.1.m1.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="S3.SS6.2.2.1.m1.1d">\times</annotation></semantics></math>30 layers, adjacent blocks sharing weights), and a 60-layer non-shared weight model, with consistent configurations in all other aspects. </span></figcaption><div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_1">
<div id="S3.SS6.6.10" class="ltx_inline-block ltx_figure_panel ltx_transformed_outer" style="width:195.1pt;height:52pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(-37.4pt,10.0pt) scale(0.722851780611981,0.722851780611981) ;">
<table id="S3.SS6.6.10.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S3.SS6.6.10.1.1.1" class="ltx_tr">
<th id="S3.SS6.6.10.1.1.1.1" class="ltx_td ltx_th ltx_th_row ltx_border_tt" style="padding:-2pt 5.7pt;"></th>
<th id="S3.SS6.6.10.1.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 5.7pt;">Load</th>
<th id="S3.SS6.6.10.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 5.7pt;">Init</th>
<th id="S3.SS6.6.10.1.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 5.7pt;">Execute</th>
</tr>
<tr id="S3.SS6.6.10.1.2.2" class="ltx_tr">
<th id="S3.SS6.6.10.1.2.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row" style="padding:-2pt 5.7pt;">
<span id="S3.SS6.6.10.1.2.2.1.1" class="ltx_ERROR undefined">\hdashline</span>[0.8pt/1pt]
<span id="S3.SS6.6.10.1.2.2.1.2" class="ltx_text ltx_font_typewriter">MobileLLM</span>
</th>
<th id="S3.SS6.6.10.1.2.2.2" class="ltx_td ltx_align_center ltx_th ltx_th_column" style="padding:-2pt 5.7pt;">39.2 ms</th>
<th id="S3.SS6.6.10.1.2.2.3" class="ltx_td ltx_align_center ltx_th ltx_th_column" style="padding:-2pt 5.7pt;">1361.7 ms</th>
<th id="S3.SS6.6.10.1.2.2.4" class="ltx_td ltx_align_center ltx_th ltx_th_column" style="padding:-2pt 5.7pt;">15.6 ms</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S3.SS6.6.10.1.3.1" class="ltx_tr">
<th id="S3.SS6.6.10.1.3.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row" style="padding:-2pt 5.7pt;"><span id="S3.SS6.6.10.1.3.1.1.1" class="ltx_text ltx_font_typewriter">MobileLLM-LS</span></th>
<td id="S3.SS6.6.10.1.3.1.2" class="ltx_td ltx_align_center" style="padding:-2pt 5.7pt;">43.6 ms</td>
<td id="S3.SS6.6.10.1.3.1.3" class="ltx_td ltx_align_center" style="padding:-2pt 5.7pt;">1388.2 ms</td>
<td id="S3.SS6.6.10.1.3.1.4" class="ltx_td ltx_align_center" style="padding:-2pt 5.7pt;">16.0 ms</td>
</tr>
<tr id="S3.SS6.6.10.1.4.2" class="ltx_tr">
<th id="S3.SS6.6.10.1.4.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb" style="padding:-2pt 5.7pt;">60-layer non-shared</th>
<td id="S3.SS6.6.10.1.4.2.2" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 5.7pt;">68.6 ms</td>
<td id="S3.SS6.6.10.1.4.2.3" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 5.7pt;">3347.7 ms</td>
<td id="S3.SS6.6.10.1.4.2.4" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 5.7pt;">29.0 ms</td>
</tr>
</tbody>
</table>
</span></div>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<p id="S3.SS6.4.4" class="ltx_p ltx_figure_panel">We measure the latency for <span id="S3.SS6.4.4.1" class="ltx_text ltx_markedasmath ltx_font_typewriter">MobileLLM</span>-125M and <span id="S3.SS6.4.4.2" class="ltx_text ltx_markedasmath ltx_font_typewriter">MobileLLM-LS</span>-125M FP16 models via ExecuTorch<span id="footnote7" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">7</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">7</sup><span class="ltx_tag ltx_tag_note">7</span>https://pytorch.org/executorch-overview</span></span></span> on iPhone 13 (iOS 17.2.1), with Metal Performance Shaders (MPS) backend<span id="footnote8" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">8</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">8</sup><span class="ltx_tag ltx_tag_note">8</span>https://pytorch.org/executorch/stable/build-run-mps.html</span></span></span>. Model loading, initialization and execution time are reported in Table <a href="#S3.SS6" title="3.6 On-device Profiling ‣ 3.5 Knowledge Distillation ‣ 3.4 Compatibility with Quantization ‣ 3.3.2 API Calling ‣ 3.3.1 Chat ‣ 3.3 Downstream Tasks ‣ 3.2 Main Results ‣ 3 Experiments ‣ 2.3 Layer Sharing ‣ 2 Improving Sub-billion Scale LLM Design ‣ MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.6</span></a>. Specifically, execution time is averaged over 50 iterations.</p>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<p id="S3.SS6.6.6" class="ltx_p ltx_figure_panel">Results in Table <a href="#S3.SS6" title="3.6 On-device Profiling ‣ 3.5 Knowledge Distillation ‣ 3.4 Compatibility with Quantization ‣ 3.3.2 API Calling ‣ 3.3.1 Chat ‣ 3.3 Downstream Tasks ‣ 3.2 Main Results ‣ 3 Experiments ‣ 2.3 Layer Sharing ‣ 2 Improving Sub-billion Scale LLM Design ‣ MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.6</span></a> reflects that through weight sharing and doubling the number of layers, <span id="S3.SS6.6.6.1" class="ltx_text ltx_markedasmath ltx_font_typewriter">MobileLLM-LS</span> incurs only a 2.2% increase in loading and initialization time compared to <span id="S3.SS6.6.6.2" class="ltx_text ltx_markedasmath ltx_font_typewriter">MobileLLM</span>, attributable to their similar model sizes. Execution time also experiences a mere 2.6% overhead, benefitting from data locality. In contrast, a model with doubled number of layers without weight sharing exhibits a substantial 143% rise in loading and initialization time and an 86% increase in execution time.</p>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<section id="S4" class="ltx_section ltx_figure_panel">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">4 </span>Related Work</h2>

<div id="S4.p1" class="ltx_para">
<p id="S4.p1.1" class="ltx_p">The excellent performance of LLMs has fostered its wide applications.
Considering computational cost and energy consumption of LLMs, an new stream of research direction have emerged to downsize LLMs to enable on-device inference. These methods include:</p>
</div>
<div id="S4.p2" class="ltx_para">
<p id="S4.p2.1" class="ltx_p"><span id="S4.p2.1.1" class="ltx_text ltx_font_bold">Model Compression.</span> Numerous model compression methods are developed for LLMs, including pruning<cite class="ltx_cite ltx_citemacro_citep">(Xia et al., <a href="#bib.bib51" title="" class="ltx_ref">2023b</a>)</cite>, sparsity <cite class="ltx_cite ltx_citemacro_citep">(Sun et al., <a href="#bib.bib42" title="" class="ltx_ref">2023</a>; Xia et al., <a href="#bib.bib50" title="" class="ltx_ref">2023a</a>; Frantar &amp; Alistarh, <a href="#bib.bib12" title="" class="ltx_ref">2023</a>)</cite>, and quantization <cite class="ltx_cite ltx_citemacro_citep">(Liu et al., <a href="#bib.bib29" title="" class="ltx_ref">2023a</a>; Dettmers et al., <a href="#bib.bib10" title="" class="ltx_ref">2022</a>; Kim et al., <a href="#bib.bib24" title="" class="ltx_ref">2023</a>; Frantar et al., <a href="#bib.bib13" title="" class="ltx_ref">2022</a>; Xiao et al., <a href="#bib.bib52" title="" class="ltx_ref">2023</a>; Yao et al., <a href="#bib.bib55" title="" class="ltx_ref">2022</a>; Liu et al., <a href="#bib.bib31" title="" class="ltx_ref">2023c</a>, <a href="#bib.bib30" title="" class="ltx_ref">b</a>; Frantar et al., <a href="#bib.bib13" title="" class="ltx_ref">2022</a>)</cite>. Our research is complementary to these techniques. As also substantiated in Section <a href="#S3.SS4" title="3.4 Compatibility with Quantization ‣ 3.3.2 API Calling ‣ 3.3.1 Chat ‣ 3.3 Downstream Tasks ‣ 3.2 Main Results ‣ 3 Experiments ‣ 2.3 Layer Sharing ‣ 2 Improving Sub-billion Scale LLM Design ‣ MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.4</span></a>, our methodology is compatible with quantization.</p>
</div>
<div id="S4.p3" class="ltx_para">
<p id="S4.p3.1" class="ltx_p"><span id="S4.p3.1.1" class="ltx_text ltx_font_bold">Small Model Design.</span>
A limited number of studies have explored compact model architectures, such as TinyLLaMA <cite class="ltx_cite ltx_citemacro_citep">(Timiryasov &amp; Tastet, <a href="#bib.bib45" title="" class="ltx_ref">2023</a>)</cite>. However, even the smallest TinyLLaMA exceeds 1 billion parameters, making them still prohibitive for many on-device applications. Some research proposes large model architectures alongside their smaller LLM variants in a model family <cite class="ltx_cite ltx_citemacro_citep">(Zhang et al., <a href="#bib.bib57" title="" class="ltx_ref">2022</a>; Scao et al., <a href="#bib.bib39" title="" class="ltx_ref">2022</a>; Black et al., <a href="#bib.bib4" title="" class="ltx_ref">2022</a>; Dey et al., <a href="#bib.bib11" title="" class="ltx_ref">2023</a>)</cite> or a analytical suite containing small LLM variants <cite class="ltx_cite ltx_citemacro_citep">(Biderman et al., <a href="#bib.bib2" title="" class="ltx_ref">2023</a>)</cite>. However, these models are not optimized under the constraint of sub-billion parameters and therefore may not be optimal.</p>
</div>
<div id="S4.p4" class="ltx_para">
<p id="S4.p4.1" class="ltx_p"><span id="S4.p4.1.1" class="ltx_text ltx_font_bold">Neural Architecture Search.</span> NAS has garnered substantial attention in the realm of convolutional neural networks, particularly in the context of vision tasks <cite class="ltx_cite ltx_citemacro_citep">(Tan &amp; Le, <a href="#bib.bib43" title="" class="ltx_ref">2019</a>; Zoph &amp; Le, <a href="#bib.bib59" title="" class="ltx_ref">2016</a>; Wu et al., <a href="#bib.bib49" title="" class="ltx_ref">2019</a>; Guo et al., <a href="#bib.bib16" title="" class="ltx_ref">2020</a>)</cite>. In contrast, within the transformer domain, the prevailing consensus posits that the model architecture exerts minimal influence on accuracy, provided the total number of parameters remains consistent <cite class="ltx_cite ltx_citemacro_citep">(Kaplan et al., <a href="#bib.bib22" title="" class="ltx_ref">2020</a>)</cite>.
Only a limited number of studies have developed NAS algorithm for language transformer, targeting at BERT models <cite class="ltx_cite ltx_citemacro_citep">(Xu et al., <a href="#bib.bib54" title="" class="ltx_ref">2021</a>; Jawahar et al., <a href="#bib.bib20" title="" class="ltx_ref">2023</a>; Ganesan et al., <a href="#bib.bib14" title="" class="ltx_ref">2021</a>)</cite>. Our current investigation, focusing on the interplay between depth and width, can be conceptualized as a meticulous grid search within the depth space. The outcomes of that study challenge the prevalent orthodoxy surrounding scaling laws, proposing that deep and thin architectures demonstrate higher performance for compact LLMs.</p>
</div>
<div id="S4.p5" class="ltx_para">
<p id="S4.p5.1" class="ltx_p"><span id="S4.p5.1.1" class="ltx_text ltx_font_bold">Weight Sharing.</span> Weight sharing is an intuitive strategy for optimizing model weight utilization within fixed parameter constraints. While the OPT family <cite class="ltx_cite ltx_citemacro_citep">(Zhang et al., <a href="#bib.bib57" title="" class="ltx_ref">2022</a>)</cite> and subsequent works <cite class="ltx_cite ltx_citemacro_citep">(Black et al., <a href="#bib.bib4" title="" class="ltx_ref">2022</a>)</cite> leverage weight sharing between input and output embeddings, limited research has explored weight sharing for intermediate layers in transformers <cite class="ltx_cite ltx_citemacro_citep">(Shen et al., <a href="#bib.bib40" title="" class="ltx_ref">2022</a>; Reid et al., <a href="#bib.bib35" title="" class="ltx_ref">2021</a>)</cite>. Prior efforts often entail specialized designs for shared layers. In contrast, our contribution highlights a more straightforward yet effective way of simply repeating transformer blocks, yielding improved accuracy with a fixed model size and minimal latency increase.</p>
</div>
<div id="S4.p6" class="ltx_para">
<p id="S4.p6.1" class="ltx_p"><span id="S4.p6.1.1" class="ltx_text ltx_font_bold">Efficient Attention and Implementation.</span> In the realm of efficient transformer design, much research has focused on optimizing attention computation through methods like low-rank approximation <cite class="ltx_cite ltx_citemacro_citep">(Wang et al., <a href="#bib.bib48" title="" class="ltx_ref">2020</a>; Katharopoulos et al., <a href="#bib.bib23" title="" class="ltx_ref">2020</a>; Xiong et al., <a href="#bib.bib53" title="" class="ltx_ref">2021</a>)</cite> and sparse attention <cite class="ltx_cite ltx_citemacro_citep">(Kitaev et al., <a href="#bib.bib26" title="" class="ltx_ref">2020</a>; Roy et al., <a href="#bib.bib36" title="" class="ltx_ref">2021</a>)</cite>. Another line of work explores hardware scheduling and weight movement, exemplified by works such as FlashAttention <cite class="ltx_cite ltx_citemacro_citep">(Dao et al., <a href="#bib.bib8" title="" class="ltx_ref">2022</a>)</cite> and FlexGen <cite class="ltx_cite ltx_citemacro_citep">(Sheng et al., <a href="#bib.bib41" title="" class="ltx_ref">2023</a>)</cite>. In contrast, our primary goal is to optimize model size without introducing new attention computation or efficient hardware implementation methods.</p>
</div>
<section id="S5" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">5 </span>Conclusion</h2>

<div id="S5.p1" class="ltx_para">
<p id="S5.p1.2" class="ltx_p">This study focuses on optimizing sub-billion scale models for on-device applications. Our findings indicate that, for smaller models, prioritizing depth over width enhances model performance. Furthermore, by leveraging advanced weight-sharing techniques, including embedding sharing, grouped query attention, and block-wise weight sharing, we achieve significant enhancements in weight utilization within storage-constrained scenarios. The resulting models denoted as <span id="S5.p1.2.1" class="ltx_text ltx_markedasmath ltx_font_typewriter">MobileLLM</span> exhibit substantial advancements in zero-shot commonsense reasoning, question answering, and reading comprehension tasks compared to previous SoTA methods. Last but not least, we demonstrate the effectiveness of the fine-tuned <span id="S5.p1.2.2" class="ltx_text ltx_markedasmath ltx_font_typewriter">MobileLLM</span> models in two prevalent on-device use cases: chat and API calling, underscoring their adeptness in handling such tasks.</p>
</div>
<section id="Sx1" class="ltx_section">
<h2 class="ltx_title ltx_title_section">Acknowledgment</h2>

<div id="Sx1.p1" class="ltx_para">
<p id="Sx1.p1.1" class="ltx_p">We thank Hansong Zhang for his valuable contribution in setting up latency measurement environment on iOS, and the full support from PyTorch Edge team.</p>
</div>
<section id="Sx2" class="ltx_section">
<h2 class="ltx_title ltx_title_section">Broader Impact</h2>

<div id="Sx2.p1" class="ltx_para">
<p id="Sx2.p1.1" class="ltx_p">This paper advocates for the adoption of sub-billion scale large language models in on-device applications, aiming to mitigate energy consumption during LLM inference. The proposed approach is promising in alleviating computational costs associated with LLM deployment.</p>
</div>
<section id="bib" class="ltx_bibliography">
<h2 class="ltx_title ltx_title_bibliography">References</h2>

<ul class="ltx_biblist">
<li id="bib.bib1" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ainslie et al. (2023)</span>
<span class="ltx_bibblock">
Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y., Lebron, F., and Sanghai, S.

</span>
<span class="ltx_bibblock">GQA: Training generalized multi-query transformer models from multi-head checkpoints.

</span>
<span class="ltx_bibblock">In <em id="bib.bib1.1.1" class="ltx_emph ltx_font_italic">EMNLP</em>, 2023.

</span>
</li>
<li id="bib.bib2" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Biderman et al. (2023)</span>
<span class="ltx_bibblock">
Biderman, S., Schoelkopf, H., Anthony, Q. G., Bradley, H., O’Brien, K., Hallahan, E., Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., et al.

</span>
<span class="ltx_bibblock">Pythia: A suite for analyzing large language models across training and scaling.

</span>
<span class="ltx_bibblock">In <em id="bib.bib2.1.1" class="ltx_emph ltx_font_italic">International Conference on Machine Learning</em>, pp.  2397–2430. PMLR, 2023.

</span>
</li>
<li id="bib.bib3" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Bisk et al. (2020)</span>
<span class="ltx_bibblock">
Bisk, Y., Zellers, R., Gao, J., Choi, Y., et al.

</span>
<span class="ltx_bibblock">Piqa: Reasoning about physical commonsense in natural language.

</span>
<span class="ltx_bibblock">In <em id="bib.bib3.1.1" class="ltx_emph ltx_font_italic">Proceedings of the AAAI conference on artificial intelligence</em>, volume 34, pp.  7432–7439, 2020.

</span>
</li>
<li id="bib.bib4" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Black et al. (2022)</span>
<span class="ltx_bibblock">
Black, S., Biderman, S., Hallahan, E., Anthony, Q., Gao, L., Golding, L., He, H., Leahy, C., McDonell, K., Phang, J., et al.

</span>
<span class="ltx_bibblock">Gpt-neox-20b: An open-source autoregressive language model.

</span>
<span class="ltx_bibblock"><em id="bib.bib4.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2204.06745</em>, 2022.

</span>
</li>
<li id="bib.bib5" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Chowdhery et al. (2023)</span>
<span class="ltx_bibblock">
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al.

</span>
<span class="ltx_bibblock">Palm: Scaling language modeling with pathways.

</span>
<span class="ltx_bibblock"><em id="bib.bib5.1.1" class="ltx_emph ltx_font_italic">Journal of Machine Learning Research</em>, 24(240):1–113, 2023.

</span>
</li>
<li id="bib.bib6" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Clark et al. (2019)</span>
<span class="ltx_bibblock">
Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., and Toutanova, K.

</span>
<span class="ltx_bibblock">Boolq: Exploring the surprising difficulty of natural yes/no questions.

</span>
<span class="ltx_bibblock"><em id="bib.bib6.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:1905.10044</em>, 2019.

</span>
</li>
<li id="bib.bib7" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Clark et al. (2018)</span>
<span class="ltx_bibblock">
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O.

</span>
<span class="ltx_bibblock">Think you have solved question answering? try arc, the ai2 reasoning challenge.

</span>
<span class="ltx_bibblock"><em id="bib.bib7.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:1803.05457</em>, 2018.

</span>
</li>
<li id="bib.bib8" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Dao et al. (2022)</span>
<span class="ltx_bibblock">
Dao, T., Fu, D., Ermon, S., Rudra, A., and Ré, C.

</span>
<span class="ltx_bibblock">Flashattention: Fast and memory-efficient exact attention with io-awareness.

</span>
<span class="ltx_bibblock"><em id="bib.bib8.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>, 35:16344–16359, 2022.

</span>
</li>
<li id="bib.bib9" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Dauphin et al. (2017)</span>
<span class="ltx_bibblock">
Dauphin, Y. N., Fan, A., Auli, M., and Grangier, D.

</span>
<span class="ltx_bibblock">Language modeling with gated convolutional networks.

</span>
<span class="ltx_bibblock">In <em id="bib.bib9.1.1" class="ltx_emph ltx_font_italic">International conference on machine learning</em>, pp.  933–941. PMLR, 2017.

</span>
</li>
<li id="bib.bib10" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Dettmers et al. (2022)</span>
<span class="ltx_bibblock">
Dettmers, T., Lewis, M., Belkada, Y., and Zettlemoyer, L.

</span>
<span class="ltx_bibblock">Llm. int8 (): 8-bit matrix multiplication for transformers at scale.

</span>
<span class="ltx_bibblock"><em id="bib.bib10.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2208.07339</em>, 2022.

</span>
</li>
<li id="bib.bib11" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Dey et al. (2023)</span>
<span class="ltx_bibblock">
Dey, N., Gosal, G., Khachane, H., Marshall, W., Pathria, R., Tom, M., Hestness, J., et al.

</span>
<span class="ltx_bibblock">Cerebras-gpt: Open compute-optimal language models trained on the cerebras wafer-scale cluster.

</span>
<span class="ltx_bibblock"><em id="bib.bib11.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2304.03208</em>, 2023.

</span>
</li>
<li id="bib.bib12" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Frantar &amp; Alistarh (2023)</span>
<span class="ltx_bibblock">
Frantar, E. and Alistarh, D.

</span>
<span class="ltx_bibblock">Sparsegpt: Massive language models can be accurately pruned in one-shot.

</span>
<span class="ltx_bibblock">In <em id="bib.bib12.1.1" class="ltx_emph ltx_font_italic">International Conference on Machine Learning</em>, pp.  10323–10337. PMLR, 2023.

</span>
</li>
<li id="bib.bib13" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Frantar et al. (2022)</span>
<span class="ltx_bibblock">
Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D.

</span>
<span class="ltx_bibblock">Gptq: Accurate post-training quantization for generative pre-trained transformers.

</span>
<span class="ltx_bibblock"><em id="bib.bib13.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2210.17323</em>, 2022.

</span>
</li>
<li id="bib.bib14" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ganesan et al. (2021)</span>
<span class="ltx_bibblock">
Ganesan, V., Ramesh, G., and Kumar, P.

</span>
<span class="ltx_bibblock">Supershaper: Task-agnostic super pre-training of bert models with variable hidden dimensions.

</span>
<span class="ltx_bibblock"><em id="bib.bib14.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2110.04711</em>, 2021.

</span>
</li>
<li id="bib.bib15" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Google (2023)</span>
<span class="ltx_bibblock">
Google.

</span>
<span class="ltx_bibblock">Pixel 8 pro tech specs.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://store.google.com/gb/product/pixel_8_pro_specs" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://store.google.com/gb/product/pixel_8_pro_specs</a>, 2023.

</span>
</li>
<li id="bib.bib16" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Guo et al. (2020)</span>
<span class="ltx_bibblock">
Guo, Z., Zhang, X., Mu, H., Heng, W., Liu, Z., Wei, Y., and Sun, J.

</span>
<span class="ltx_bibblock">Single path one-shot neural architecture search with uniform sampling.

</span>
<span class="ltx_bibblock">In <em id="bib.bib16.1.1" class="ltx_emph ltx_font_italic">Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XVI 16</em>, pp.  544–560. Springer, 2020.

</span>
</li>
<li id="bib.bib17" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Han et al. (2016)</span>
<span class="ltx_bibblock">
Han, S., Liu, X., Mao, H., Pu, J., Pedram, A., Horowitz, M. A., and Dally, W. J.

</span>
<span class="ltx_bibblock">Eie: Efficient inference engine on compressed deep neural network.

</span>
<span class="ltx_bibblock"><em id="bib.bib17.1.1" class="ltx_emph ltx_font_italic">ACM SIGARCH Computer Architecture News</em>, 44(3):243–254, 2016.

</span>
</li>
<li id="bib.bib18" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hinton et al. (2015)</span>
<span class="ltx_bibblock">
Hinton, G., Vinyals, O., Dean, J., et al.

</span>
<span class="ltx_bibblock">Distilling the knowledge in a neural network.

</span>
<span class="ltx_bibblock"><em id="bib.bib18.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:1503.02531</em>, 2(7), 2015.

</span>
</li>
<li id="bib.bib19" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hristov (2022)</span>
<span class="ltx_bibblock">
Hristov, V.

</span>
<span class="ltx_bibblock">A16 bionic explained: what’s new in apple’s pro-grade mobile chip?

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://www.phonearena.com/news/A16-Bionic-explained-whats-new_id142438" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://www.phonearena.com/news/A16-Bionic-explained-whats-new_id142438</a>, 2022.

</span>
</li>
<li id="bib.bib20" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Jawahar et al. (2023)</span>
<span class="ltx_bibblock">
Jawahar, G., Yang, H., Xiong, Y., Liu, Z., Wang, D., Sun, F., Li, M., Pappu, A., Oguz, B., Abdul-Mageed, M., et al.

</span>
<span class="ltx_bibblock">Mixture-of-supernets: Improving weight-sharing supernet training with architecture-routed mixture-of-experts.

</span>
<span class="ltx_bibblock"><em id="bib.bib20.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2306.04845</em>, 2023.

</span>
</li>
<li id="bib.bib21" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Joshi et al. (2017)</span>
<span class="ltx_bibblock">
Joshi, M., Choi, E., Weld, D. S., and Zettlemoyer, L.

</span>
<span class="ltx_bibblock">Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension.

</span>
<span class="ltx_bibblock"><em id="bib.bib21.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:1705.03551</em>, 2017.

</span>
</li>
<li id="bib.bib22" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Kaplan et al. (2020)</span>
<span class="ltx_bibblock">
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D.

</span>
<span class="ltx_bibblock">Scaling laws for neural language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib22.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2001.08361</em>, 2020.

</span>
</li>
<li id="bib.bib23" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Katharopoulos et al. (2020)</span>
<span class="ltx_bibblock">
Katharopoulos, A., Vyas, A., Pappas, N., and Fleuret, F.

</span>
<span class="ltx_bibblock">Transformers are rnns: Fast autoregressive transformers with linear attention.

</span>
<span class="ltx_bibblock">In <em id="bib.bib23.1.1" class="ltx_emph ltx_font_italic">International conference on machine learning</em>, pp.  5156–5165. PMLR, 2020.

</span>
</li>
<li id="bib.bib24" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Kim et al. (2023)</span>
<span class="ltx_bibblock">
Kim, S., Hooper, C., Gholami, A., Dong, Z., Li, X., Shen, S., Mahoney, M. W., and Keutzer, K.

</span>
<span class="ltx_bibblock">Squeezellm: Dense-and-sparse quantization.

</span>
<span class="ltx_bibblock"><em id="bib.bib24.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2306.07629</em>, 2023.

</span>
</li>
<li id="bib.bib25" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Kingma &amp; Ba (2014)</span>
<span class="ltx_bibblock">
Kingma, D. P. and Ba, J.

</span>
<span class="ltx_bibblock">Adam: A method for stochastic optimization.

</span>
<span class="ltx_bibblock"><em id="bib.bib25.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:1412.6980</em>, 2014.

</span>
</li>
<li id="bib.bib26" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Kitaev et al. (2020)</span>
<span class="ltx_bibblock">
Kitaev, N., Kaiser, Ł., and Levskaya, A.

</span>
<span class="ltx_bibblock">Reformer: The efficient transformer.

</span>
<span class="ltx_bibblock"><em id="bib.bib26.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2001.04451</em>, 2020.

</span>
</li>
<li id="bib.bib27" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lai et al. (2017)</span>
<span class="ltx_bibblock">
Lai, G., Xie, Q., Liu, H., Yang, Y., and Hovy, E.

</span>
<span class="ltx_bibblock">Race: Large-scale reading comprehension dataset from examinations.

</span>
<span class="ltx_bibblock"><em id="bib.bib27.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:1704.04683</em>, 2017.

</span>
</li>
<li id="bib.bib28" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Li et al. (2023)</span>
<span class="ltx_bibblock">
Li, X., Zhang, T., Dubois, Y., Taori, R., Gulrajani, I., Guestrin, C., Liang, P., and Hashimoto, T. B.

</span>
<span class="ltx_bibblock">Alpacaeval: An automatic evaluator of instruction-following models.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://github.com/tatsu-lab/alpaca_eval" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/tatsu-lab/alpaca_eval</a>, 2023.

</span>
</li>
<li id="bib.bib29" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Liu et al. (2023a)</span>
<span class="ltx_bibblock">
Liu, J., Gong, R., Wei, X., Dong, Z., Cai, J., and Zhuang, B.

</span>
<span class="ltx_bibblock">Qllm: Accurate and efficient low-bitwidth quantization for large language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib29.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2310.08041</em>, 2023a.

</span>
</li>
<li id="bib.bib30" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Liu et al. (2023b)</span>
<span class="ltx_bibblock">
Liu, S.-y., Liu, Z., Huang, X., Dong, P., and Cheng, K.-T.

</span>
<span class="ltx_bibblock">Llm-fp4: 4-bit floating-point quantized transformers.

</span>
<span class="ltx_bibblock"><em id="bib.bib30.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2310.16836</em>, 2023b.

</span>
</li>
<li id="bib.bib31" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Liu et al. (2023c)</span>
<span class="ltx_bibblock">
Liu, Z., Oguz, B., Zhao, C., Chang, E., Stock, P., Mehdad, Y., Shi, Y., Krishnamoorthi, R., and Chandra, V.

</span>
<span class="ltx_bibblock">Llm-qat: Data-free quantization aware training for large language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib31.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2305.17888</em>, 2023c.

</span>
</li>
<li id="bib.bib32" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Malladi et al. (2012)</span>
<span class="ltx_bibblock">
Malladi, K. T., Lee, B. C., Nothaft, F. A., Kozyrakis, C., Periyathambi, K., and Horowitz, M.

</span>
<span class="ltx_bibblock">Towards energy-proportional datacenter memory with mobile dram.

</span>
<span class="ltx_bibblock"><em id="bib.bib32.1.1" class="ltx_emph ltx_font_italic">ACM SIGARCH Computer Architecture News</em>, 40(3):37–48, 2012.

</span>
</li>
<li id="bib.bib33" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Mihaylov et al. (2018)</span>
<span class="ltx_bibblock">
Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A.

</span>
<span class="ltx_bibblock">Can a suit of armor conduct electricity? a new dataset for open book question answering.

</span>
<span class="ltx_bibblock"><em id="bib.bib33.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:1809.02789</em>, 2018.

</span>
</li>
<li id="bib.bib34" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Peng et al. (2023)</span>
<span class="ltx_bibblock">
Peng, B., Alcaide, E., Anthony, Q., Albalak, A., Arcadinho, S., Cao, H., Cheng, X., Chung, M., Grella, M., GV, K. K., et al.

</span>
<span class="ltx_bibblock">Rwkv: Reinventing rnns for the transformer era.

</span>
<span class="ltx_bibblock"><em id="bib.bib34.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2305.13048</em>, 2023.

</span>
</li>
<li id="bib.bib35" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Reid et al. (2021)</span>
<span class="ltx_bibblock">
Reid, M., Marrese-Taylor, E., and Matsuo, Y.

</span>
<span class="ltx_bibblock">Subformer: Exploring weight sharing for parameter efficiency in generative transformers.

</span>
<span class="ltx_bibblock"><em id="bib.bib35.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2101.00234</em>, 2021.

</span>
</li>
<li id="bib.bib36" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Roy et al. (2021)</span>
<span class="ltx_bibblock">
Roy, A., Saffar, M., Vaswani, A., and Grangier, D.

</span>
<span class="ltx_bibblock">Efficient content-based sparse attention with routing transformers.

</span>
<span class="ltx_bibblock"><em id="bib.bib36.1.1" class="ltx_emph ltx_font_italic">Transactions of the Association for Computational Linguistics</em>, 9:53–68, 2021.

</span>
</li>
<li id="bib.bib37" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Sakaguchi et al. (2021)</span>
<span class="ltx_bibblock">
Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y.

</span>
<span class="ltx_bibblock">Winogrande: An adversarial winograd schema challenge at scale.

</span>
<span class="ltx_bibblock"><em id="bib.bib37.1.1" class="ltx_emph ltx_font_italic">Communications of the ACM</em>, 64(9):99–106, 2021.

</span>
</li>
<li id="bib.bib38" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Sap et al. (2019)</span>
<span class="ltx_bibblock">
Sap, M., Rashkin, H., Chen, D., LeBras, R., and Choi, Y.

</span>
<span class="ltx_bibblock">Socialiqa: Commonsense reasoning about social interactions.

</span>
<span class="ltx_bibblock"><em id="bib.bib38.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:1904.09728</em>, 2019.

</span>
</li>
<li id="bib.bib39" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Scao et al. (2022)</span>
<span class="ltx_bibblock">
Scao, T. L., Fan, A., Akiki, C., Pavlick, E., Ilić, S., Hesslow, D., Castagné, R., Luccioni, A. S., Yvon, F., Gallé, M., et al.

</span>
<span class="ltx_bibblock">Bloom: A 176b-parameter open-access multilingual language model.

</span>
<span class="ltx_bibblock"><em id="bib.bib39.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2211.05100</em>, 2022.

</span>
</li>
<li id="bib.bib40" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Shen et al. (2022)</span>
<span class="ltx_bibblock">
Shen, Z., Liu, Z., and Xing, E.

</span>
<span class="ltx_bibblock">Sliced recursive transformer.

</span>
<span class="ltx_bibblock">In <em id="bib.bib40.1.1" class="ltx_emph ltx_font_italic">European Conference on Computer Vision</em>, pp.  727–744. Springer, 2022.

</span>
</li>
<li id="bib.bib41" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Sheng et al. (2023)</span>
<span class="ltx_bibblock">
Sheng, Y., Zheng, L., Yuan, B., Li, Z., Ryabinin, M., Chen, B., Liang, P., Ré, C., Stoica, I., and Zhang, C.

</span>
<span class="ltx_bibblock">Flexgen: High-throughput generative inference of large language models with a single gpu.

</span>
<span class="ltx_bibblock">In <em id="bib.bib41.1.1" class="ltx_emph ltx_font_italic">International Conference on Machine Learning</em>, pp.  31094–31116. PMLR, 2023.

</span>
</li>
<li id="bib.bib42" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Sun et al. (2023)</span>
<span class="ltx_bibblock">
Sun, M., Liu, Z., Bair, A., and Kolter, J. Z.

</span>
<span class="ltx_bibblock">A simple and effective pruning approach for large language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib42.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2306.11695</em>, 2023.

</span>
</li>
<li id="bib.bib43" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Tan &amp; Le (2019)</span>
<span class="ltx_bibblock">
Tan, M. and Le, Q.

</span>
<span class="ltx_bibblock">Efficientnet: Rethinking model scaling for convolutional neural networks.

</span>
<span class="ltx_bibblock">In <em id="bib.bib43.1.1" class="ltx_emph ltx_font_italic">International conference on machine learning</em>, pp.  6105–6114. PMLR, 2019.

</span>
</li>
<li id="bib.bib44" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Taylor et al. (2022)</span>
<span class="ltx_bibblock">
Taylor, R., Kardas, M., Cucurull, G., Scialom, T., Hartshorn, A., Saravia, E., Poulton, A., Kerkez, V., and Stojnic, R.

</span>
<span class="ltx_bibblock">Galactica: A large language model for science.

</span>
<span class="ltx_bibblock"><em id="bib.bib44.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2211.09085</em>, 2022.

</span>
</li>
<li id="bib.bib45" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Timiryasov &amp; Tastet (2023)</span>
<span class="ltx_bibblock">
Timiryasov, I. and Tastet, J.-L.

</span>
<span class="ltx_bibblock">Baby llama: knowledge distillation from an ensemble of teachers trained on a small dataset with no performance penalty.

</span>
<span class="ltx_bibblock"><em id="bib.bib45.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2308.02019</em>, 2023.

</span>
</li>
<li id="bib.bib46" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Touvron et al. (2023a)</span>
<span class="ltx_bibblock">
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al.

</span>
<span class="ltx_bibblock">Llama: Open and efficient foundation language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib46.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2302.13971</em>, 2023a.

</span>
</li>
<li id="bib.bib47" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Touvron et al. (2023b)</span>
<span class="ltx_bibblock">
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Ferrer, C. C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshorn, A., Hosseini, S., Hou, R., Inan, H., Kardas, M., Kerkez, V., Khabsa, M., Kloumann, I., Korenev, A., Koura, P. S., Lachaux, M.-A., Lavril, T., Lee, J., Liskovich, D., Lu, Y., Mao, Y., Martinet, X., Mihaylov, T., Mishra, P., Molybog, I., Nie, Y., Poulton, A., Reizenstein, J., Rungta, R., Saladi, K., Schelten, A., Silva, R., Smith, E. M., Subramanian, R., Tan, X. E., Tang, B., Taylor, R., Williams, A., Kuan, J. X., Xu, P., Yan, Z., Zarov, I., Zhang, Y., Fan, A., Kambadur, M., Narang, S., Rodriguez, A., Stojnic, R., Edunov, S., and Scialom, T.

</span>
<span class="ltx_bibblock">Llama 2: Open foundation and fine-tuned chat models, 2023b.

</span>
</li>
<li id="bib.bib48" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wang et al. (2020)</span>
<span class="ltx_bibblock">
Wang, S., Li, B. Z., Khabsa, M., Fang, H., and Ma, H.

</span>
<span class="ltx_bibblock">Linformer: Self-attention with linear complexity.

</span>
<span class="ltx_bibblock"><em id="bib.bib48.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2006.04768</em>, 2020.

</span>
</li>
<li id="bib.bib49" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wu et al. (2019)</span>
<span class="ltx_bibblock">
Wu, B., Dai, X., Zhang, P., Wang, Y., Sun, F., Wu, Y., Tian, Y., Vajda, P., Jia, Y., and Keutzer, K.

</span>
<span class="ltx_bibblock">Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search.

</span>
<span class="ltx_bibblock">In <em id="bib.bib49.1.1" class="ltx_emph ltx_font_italic">Proceedings of the IEEE/CVF conference on computer vision and pattern recognition</em>, pp.  10734–10742, 2019.

</span>
</li>
<li id="bib.bib50" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Xia et al. (2023a)</span>
<span class="ltx_bibblock">
Xia, H., Zheng, Z., Li, Y., Zhuang, D., Zhou, Z., Qiu, X., Li, Y., Lin, W., and Song, S. L.

</span>
<span class="ltx_bibblock">Flash-llm: Enabling cost-effective and highly-efficient large generative model inference with unstructured sparsity.

</span>
<span class="ltx_bibblock"><em id="bib.bib50.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2309.10285</em>, 2023a.

</span>
</li>
<li id="bib.bib51" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Xia et al. (2023b)</span>
<span class="ltx_bibblock">
Xia, M., Gao, T., Zeng, Z., and Chen, D.

</span>
<span class="ltx_bibblock">Sheared llama: Accelerating language model pre-training via structured pruning.

</span>
<span class="ltx_bibblock"><em id="bib.bib51.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2310.06694</em>, 2023b.

</span>
</li>
<li id="bib.bib52" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Xiao et al. (2023)</span>
<span class="ltx_bibblock">
Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., and Han, S.

</span>
<span class="ltx_bibblock">Smoothquant: Accurate and efficient post-training quantization for large language models.

</span>
<span class="ltx_bibblock">In <em id="bib.bib52.1.1" class="ltx_emph ltx_font_italic">International Conference on Machine Learning</em>, pp.  38087–38099. PMLR, 2023.

</span>
</li>
<li id="bib.bib53" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Xiong et al. (2021)</span>
<span class="ltx_bibblock">
Xiong, Y., Zeng, Z., Chakraborty, R., Tan, M., Fung, G., Li, Y., and Singh, V.

</span>
<span class="ltx_bibblock">Nyströmformer: A nyström-based algorithm for approximating self-attention.

</span>
<span class="ltx_bibblock">In <em id="bib.bib53.1.1" class="ltx_emph ltx_font_italic">Proceedings of the AAAI Conference on Artificial Intelligence</em>, volume 35, pp.  14138–14148, 2021.

</span>
</li>
<li id="bib.bib54" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Xu et al. (2021)</span>
<span class="ltx_bibblock">
Xu, J., Tan, X., Luo, R., Song, K., Li, J., Qin, T., and Liu, T.-Y.

</span>
<span class="ltx_bibblock">Nas-bert: task-agnostic and adaptive-size bert compression with neural architecture search.

</span>
<span class="ltx_bibblock">In <em id="bib.bib54.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery &amp; Data Mining</em>, pp.  1933–1943, 2021.

</span>
</li>
<li id="bib.bib55" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Yao et al. (2022)</span>
<span class="ltx_bibblock">
Yao, Z., Yazdani Aminabadi, R., Zhang, M., Wu, X., Li, C., and He, Y.

</span>
<span class="ltx_bibblock">Zeroquant: Efficient and affordable post-training quantization for large-scale transformers.

</span>
<span class="ltx_bibblock"><em id="bib.bib55.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>, 35:27168–27183, 2022.

</span>
</li>
<li id="bib.bib56" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zellers et al. (2019)</span>
<span class="ltx_bibblock">
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y.

</span>
<span class="ltx_bibblock">Hellaswag: Can a machine really finish your sentence?

</span>
<span class="ltx_bibblock"><em id="bib.bib56.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:1905.07830</em>, 2019.

</span>
</li>
<li id="bib.bib57" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhang et al. (2022)</span>
<span class="ltx_bibblock">
Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al.

</span>
<span class="ltx_bibblock">Opt: Open pre-trained transformer language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib57.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2205.01068</em>, 2022.

</span>
</li>
<li id="bib.bib58" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zheng et al. (2023)</span>
<span class="ltx_bibblock">
Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., et al.

</span>
<span class="ltx_bibblock">Judging llm-as-a-judge with mt-bench and chatbot arena.

</span>
<span class="ltx_bibblock"><em id="bib.bib58.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2306.05685</em>, 2023.

</span>
</li>
<li id="bib.bib59" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zoph &amp; Le (2016)</span>
<span class="ltx_bibblock">
Zoph, B. and Le, Q. V.

</span>
<span class="ltx_bibblock">Neural architecture search with reinforcement learning.

</span>
<span class="ltx_bibblock"><em id="bib.bib59.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:1611.01578</em>, 2016.

</span>
</li>
</ul>
</section>
<div class="ltx_pagination ltx_role_newpage"></div>
<div class="ltx_pagination ltx_role_newpage"></div>
<section id="Ax1" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">Appendix</h2>

</section>
<section id="A1" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix A </span>Impact of Each Design Choice</h2>

<div id="A1.p1" class="ltx_para">
<p id="A1.p1.6" class="ltx_p">This section presents comprehensive tabulated results for the <span id="A1.p1.6.1" class="ltx_text ltx_font_italic">improving sub-billion scale LLM design</span> experiments, at the model sizes of 125M and 350M. Looking at the results in Table <a href="#A1" title="Appendix A Impact of Each Design Choice ‣ Broader Impact ‣ Acknowledgment ‣ 5 Conclusion ‣ 4 Related Work ‣ 3.6 On-device Profiling ‣ 3.5 Knowledge Distillation ‣ 3.4 Compatibility with Quantization ‣ 3.3.2 API Calling ‣ 3.3.1 Chat ‣ 3.3 Downstream Tasks ‣ 3.2 Main Results ‣ 3 Experiments ‣ 2.3 Layer Sharing ‣ 2 Improving Sub-billion Scale LLM Design ‣ MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases" class="ltx_ref"><span class="ltx_text ltx_ref_tag">A</span></a>, transitioning from the traditional Feedforward Network (<math id="A1.p1.1.m1.1" class="ltx_Math" alttext="FC\rightarrow ReLU\rightarrow FC" display="inline"><semantics id="A1.p1.1.m1.1a"><mrow id="A1.p1.1.m1.1b"><mi id="A1.p1.1.m1.1.1" xref="A1.p1.1.m1.1.1.cmml">F</mi><mi id="A1.p1.1.m1.1.2" xref="A1.p1.1.m1.1.2.cmml">C</mi><mo stretchy="false" id="A1.p1.1.m1.1.3" xref="A1.p1.1.m1.1.3.cmml">→</mo><mi id="A1.p1.1.m1.1.4" xref="A1.p1.1.m1.1.4.cmml">R</mi><mi id="A1.p1.1.m1.1.5" xref="A1.p1.1.m1.1.5.cmml">e</mi><mi id="A1.p1.1.m1.1.6" xref="A1.p1.1.m1.1.6.cmml">L</mi><mi id="A1.p1.1.m1.1.7" xref="A1.p1.1.m1.1.7.cmml">U</mi><mo stretchy="false" id="A1.p1.1.m1.1.8" xref="A1.p1.1.m1.1.8.cmml">→</mo><mi id="A1.p1.1.m1.1.9" xref="A1.p1.1.m1.1.9.cmml">F</mi><mi id="A1.p1.1.m1.1.10" xref="A1.p1.1.m1.1.10.cmml">C</mi></mrow><annotation-xml encoding="MathML-Content" id="A1.p1.1.m1.1c"><cerror id="A1.p1.1.m1.1d"><csymbol cd="ambiguous" id="A1.p1.1.m1.1e">fragments</csymbol><csymbol cd="unknown" id="A1.p1.1.m1.1.1.cmml" xref="A1.p1.1.m1.1.1">F</csymbol><csymbol cd="unknown" id="A1.p1.1.m1.1.2.cmml" xref="A1.p1.1.m1.1.2">C</csymbol><ci id="A1.p1.1.m1.1.3.cmml" xref="A1.p1.1.m1.1.3">→</ci><csymbol cd="unknown" id="A1.p1.1.m1.1.4.cmml" xref="A1.p1.1.m1.1.4">R</csymbol><csymbol cd="unknown" id="A1.p1.1.m1.1.5.cmml" xref="A1.p1.1.m1.1.5">e</csymbol><csymbol cd="unknown" id="A1.p1.1.m1.1.6.cmml" xref="A1.p1.1.m1.1.6">L</csymbol><csymbol cd="unknown" id="A1.p1.1.m1.1.7.cmml" xref="A1.p1.1.m1.1.7">U</csymbol><ci id="A1.p1.1.m1.1.8.cmml" xref="A1.p1.1.m1.1.8">→</ci><csymbol cd="unknown" id="A1.p1.1.m1.1.9.cmml" xref="A1.p1.1.m1.1.9">F</csymbol><csymbol cd="unknown" id="A1.p1.1.m1.1.10.cmml" xref="A1.p1.1.m1.1.10">C</csymbol></cerror></annotation-xml><annotation encoding="application/x-tex" id="A1.p1.1.m1.1f">FC\rightarrow ReLU\rightarrow FC</annotation></semantics></math>) to SwiGLU yields a accuracy improvement of 1.3% for both model sizes. Further increasing the model depth enhances accuracy by 0.9%/1.1% for 125M/350M models, respectively. Then, introducing input and output embedding sharing achieves a parameter reduction of approximately 10%, while with only marginal accuracy drops of 0.2% for the 125M and 0.6% for 350M models. Additionally, following in findings in Section <a href="#A3" title="Appendix C Number of Heads and Key-Value Heads ‣ Appendix B Depth vs Width ‣ Appendix A Impact of Each Design Choice ‣ Broader Impact ‣ Acknowledgment ‣ 5 Conclusion ‣ 4 Related Work ‣ 3.6 On-device Profiling ‣ 3.5 Knowledge Distillation ‣ 3.4 Compatibility with Quantization ‣ 3.3.2 API Calling ‣ 3.3.1 Chat ‣ 3.3 Downstream Tasks ‣ 3.2 Main Results ‣ 3 Experiments ‣ 2.3 Layer Sharing ‣ 2 Improving Sub-billion Scale LLM Design ‣ MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases" class="ltx_ref"><span class="ltx_text ltx_ref_tag">C</span></a>, we incorporate grouped query attention with head dimension equals to 64, and head number to be near 4<math id="A1.p1.2.m2.1" class="ltx_Math" alttext="\times" display="inline"><semantics id="A1.p1.2.m2.1a"><mo id="A1.p1.2.m2.1.1" xref="A1.p1.2.m2.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="A1.p1.2.m2.1b"><times id="A1.p1.2.m2.1.1.cmml" xref="A1.p1.2.m2.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="A1.p1.2.m2.1c">\times</annotation></semantics></math> to the kv head number, while increasing the embedding dimension to preserve model size. This modification further results in a performance boost of 0.4%/0.7% for 125M/350M models. Combining these techniques establishes a strong baseline network denoted as <span id="A1.p1.6.2" class="ltx_text ltx_markedasmath ltx_font_typewriter">MobileLLM</span>. Finally, the immediate block-wise weight-sharing technique contributes an additional accuracy gain of 1.1% for models trained on 0.25 trillion tokens, and resulting in the model <span id="A1.p1.6.3" class="ltx_text ltx_markedasmath ltx_font_typewriter">MobileLLM-LS</span>. Final models including <span id="A1.p1.6.4" class="ltx_text ltx_markedasmath ltx_font_typewriter">MobileLLM</span> and <span id="A1.p1.6.5" class="ltx_text ltx_markedasmath ltx_font_typewriter">MobileLLM-LS</span> are trained with 1 trillion tokens.</p>
</div>
<figure id="A1.3" class="ltx_table">

<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table"><span id="A1.3.4.2.1" class="ltx_text" style="font-size:90%;">Table 8</span>: </span><span id="A1.2.2.1" class="ltx_text" style="font-size:90%;">Ablation study on the impact of each design choice to the model accuracy on zero-shot common sense reasoning tasks. Corresponding to the bar chart in Figure <a href="#S2.F3" title="Figure 3 ‣ 2.1 Training Setup ‣ 2 Improving Sub-billion Scale LLM Design ‣ MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>. Here, L, H, <math id="A1.2.2.1.m1.1" class="ltx_Math" alttext="\text{H}_{\text{KV}}" display="inline"><semantics id="A1.2.2.1.m1.1b"><mrow id="A1.2.2.1.m1.1c"><mtext id="A1.2.2.1.m1.1.1" xref="A1.2.2.1.m1.1.1a.cmml">H</mtext><msub id="A1.2.2.1.m1.1.2" xref="A1.2.2.1.m1.1.2.cmml"><mi id="A1.2.2.1.m1.1.2b" xref="A1.2.2.1.m1.1.2.cmml"></mi><mtext id="A1.2.2.1.m1.1.2.1.1" xref="A1.2.2.1.m1.1.2.1.1a.cmml">KV</mtext></msub></mrow><annotation-xml encoding="MathML-Content" id="A1.2.2.1.m1.1d"><cerror id="A1.2.2.1.m1.1e"><csymbol cd="ambiguous" id="A1.2.2.1.m1.1f">fragments</csymbol><ci id="A1.2.2.1.m1.1.1a.cmml" xref="A1.2.2.1.m1.1.1"><mtext id="A1.2.2.1.m1.1.1.cmml" xref="A1.2.2.1.m1.1.1">H</mtext></ci><apply id="A1.2.2.1.m1.1.2.cmml" xref="A1.2.2.1.m1.1.2"><ci id="A1.2.2.1.m1.1.2.1.1a.cmml" xref="A1.2.2.1.m1.1.2.1.1"><mtext mathsize="70%" id="A1.2.2.1.m1.1.2.1.1.cmml" xref="A1.2.2.1.m1.1.2.1.1">KV</mtext></ci></apply></cerror></annotation-xml><annotation encoding="application/x-tex" id="A1.2.2.1.m1.1g">\text{H}_{\text{KV}}</annotation></semantics></math> denotes number of layers, heads, kv-heads, respectively and dim denotes embedding dimension. </span></figcaption><div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_1">
<div id="A1.3.3" class="ltx_inline-block ltx_figure_panel ltx_transformed_outer" style="width:429.3pt;height:243.5pt;vertical-align:-0.7pt;"><span class="ltx_transformed_inner" style="transform:translate(-87.8pt,49.6pt) scale(0.709806336897272,0.709806336897272) ;">
<table id="A1.3.3.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A1.3.3.1.1" class="ltx_tr">
<th id="A1.3.3.1.1.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 2.0pt;"><span id="A1.3.3.1.1.2.1" class="ltx_text ltx_font_bold">Techniques</span></th>
<th id="A1.3.3.1.1.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 2.0pt;"><span id="A1.3.3.1.1.3.1" class="ltx_text ltx_font_bold">L</span></th>
<th id="A1.3.3.1.1.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 2.0pt;"><span id="A1.3.3.1.1.4.1" class="ltx_text ltx_font_bold">H</span></th>
<th id="A1.3.3.1.1.1" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 2.0pt;"><math id="A1.3.3.1.1.1.m1.1" class="ltx_Math" alttext="\text{H}_{\text{KV}}" display="inline"><semantics id="A1.3.3.1.1.1.m1.1a"><mrow id="A1.3.3.1.1.1.m1.1b"><mtext class="ltx_mathvariant_bold" id="A1.3.3.1.1.1.m1.1.1" xref="A1.3.3.1.1.1.m1.1.1a.cmml">H</mtext><msub id="A1.3.3.1.1.1.m1.1.2" xref="A1.3.3.1.1.1.m1.1.2.cmml"><mi id="A1.3.3.1.1.1.m1.1.2a" xref="A1.3.3.1.1.1.m1.1.2.cmml"></mi><mtext class="ltx_mathvariant_bold" id="A1.3.3.1.1.1.m1.1.2.1.1" xref="A1.3.3.1.1.1.m1.1.2.1.1a.cmml">KV</mtext></msub></mrow><annotation-xml encoding="MathML-Content" id="A1.3.3.1.1.1.m1.1c"><cerror id="A1.3.3.1.1.1.m1.1d"><csymbol cd="ambiguous" id="A1.3.3.1.1.1.m1.1e">fragments</csymbol><ci id="A1.3.3.1.1.1.m1.1.1a.cmml" xref="A1.3.3.1.1.1.m1.1.1"><mtext class="ltx_mathvariant_bold" id="A1.3.3.1.1.1.m1.1.1.cmml" xref="A1.3.3.1.1.1.m1.1.1">H</mtext></ci><apply id="A1.3.3.1.1.1.m1.1.2.cmml" xref="A1.3.3.1.1.1.m1.1.2"><ci id="A1.3.3.1.1.1.m1.1.2.1.1a.cmml" xref="A1.3.3.1.1.1.m1.1.2.1.1"><mtext class="ltx_mathvariant_bold" mathsize="70%" id="A1.3.3.1.1.1.m1.1.2.1.1.cmml" xref="A1.3.3.1.1.1.m1.1.2.1.1">KV</mtext></ci></apply></cerror></annotation-xml><annotation encoding="application/x-tex" id="A1.3.3.1.1.1.m1.1f">\text{H}_{\text{KV}}</annotation></semantics></math></th>
<th id="A1.3.3.1.1.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 2.0pt;"><span id="A1.3.3.1.1.5.1" class="ltx_text ltx_font_bold">Dim</span></th>
<th id="A1.3.3.1.1.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 2.0pt;"><span id="A1.3.3.1.1.6.1" class="ltx_text ltx_font_bold">#Params(M)</span></th>
<th id="A1.3.3.1.1.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 2.0pt;"><span id="A1.3.3.1.1.7.1" class="ltx_text ltx_font_bold">ARC-e</span></th>
<th id="A1.3.3.1.1.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 2.0pt;"><span id="A1.3.3.1.1.8.1" class="ltx_text ltx_font_bold">ARC-c</span></th>
<th id="A1.3.3.1.1.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 2.0pt;"><span id="A1.3.3.1.1.9.1" class="ltx_text ltx_font_bold">BoolQ</span></th>
<th id="A1.3.3.1.1.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 2.0pt;"><span id="A1.3.3.1.1.10.1" class="ltx_text ltx_font_bold">PIQA</span></th>
<th id="A1.3.3.1.1.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 2.0pt;"><span id="A1.3.3.1.1.11.1" class="ltx_text ltx_font_bold">SIQA</span></th>
<th id="A1.3.3.1.1.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 2.0pt;"><span id="A1.3.3.1.1.12.1" class="ltx_text ltx_font_bold">HellaSwag</span></th>
<th id="A1.3.3.1.1.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 2.0pt;"><span id="A1.3.3.1.1.13.1" class="ltx_text ltx_font_bold">OBQA</span></th>
<th id="A1.3.3.1.1.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 2.0pt;"><span id="A1.3.3.1.1.14.1" class="ltx_text ltx_font_bold">WinoGrande</span></th>
<th id="A1.3.3.1.1.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 2.0pt;"><span id="A1.3.3.1.1.15.1" class="ltx_text ltx_font_bold">Avg.</span></th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A1.3.3.1.2.1" class="ltx_tr">
<td id="A1.3.3.1.2.1.1" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_border_t" style="padding:-2pt 2.0pt;"><span id="A1.3.3.1.2.1.1.1" class="ltx_text ltx_font_bold">125M</span></td>
<td id="A1.3.3.1.2.1.2" class="ltx_td ltx_border_t" style="padding:-2pt 2.0pt;"></td>
<td id="A1.3.3.1.2.1.3" class="ltx_td ltx_border_t" style="padding:-2pt 2.0pt;"></td>
<td id="A1.3.3.1.2.1.4" class="ltx_td ltx_border_t" style="padding:-2pt 2.0pt;"></td>
<td id="A1.3.3.1.2.1.5" class="ltx_td ltx_border_t" style="padding:-2pt 2.0pt;"></td>
<td id="A1.3.3.1.2.1.6" class="ltx_td ltx_border_t" style="padding:-2pt 2.0pt;"></td>
<td id="A1.3.3.1.2.1.7" class="ltx_td ltx_border_t" style="padding:-2pt 2.0pt;"></td>
<td id="A1.3.3.1.2.1.8" class="ltx_td ltx_border_t" style="padding:-2pt 2.0pt;"></td>
<td id="A1.3.3.1.2.1.9" class="ltx_td ltx_border_t" style="padding:-2pt 2.0pt;"></td>
<td id="A1.3.3.1.2.1.10" class="ltx_td ltx_border_t" style="padding:-2pt 2.0pt;"></td>
<td id="A1.3.3.1.2.1.11" class="ltx_td ltx_border_t" style="padding:-2pt 2.0pt;"></td>
<td id="A1.3.3.1.2.1.12" class="ltx_td ltx_border_t" style="padding:-2pt 2.0pt;"></td>
<td id="A1.3.3.1.2.1.13" class="ltx_td ltx_border_t" style="padding:-2pt 2.0pt;"></td>
<td id="A1.3.3.1.2.1.14" class="ltx_td ltx_border_t" style="padding:-2pt 2.0pt;"></td>
<td id="A1.3.3.1.2.1.15" class="ltx_td ltx_border_t" style="padding:-2pt 2.0pt;"></td>
</tr>
<tr id="A1.3.3.1.3.2" class="ltx_tr">
<td id="A1.3.3.1.3.2.1" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left" style="padding:-2pt 2.0pt;">
<span id="A1.3.3.1.3.2.1.1" class="ltx_ERROR undefined">\hdashline</span>[0.8pt/1pt]
Baseline model</td>
<td id="A1.3.3.1.3.2.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">12</td>
<td id="A1.3.3.1.3.2.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">12</td>
<td id="A1.3.3.1.3.2.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">12</td>
<td id="A1.3.3.1.3.2.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">768</td>
<td id="A1.3.3.1.3.2.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">134.1</td>
<td id="A1.3.3.1.3.2.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">41.3</td>
<td id="A1.3.3.1.3.2.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">25.2</td>
<td id="A1.3.3.1.3.2.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">57.5</td>
<td id="A1.3.3.1.3.2.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">62.0</td>
<td id="A1.3.3.1.3.2.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">41.9</td>
<td id="A1.3.3.1.3.2.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">31.1</td>
<td id="A1.3.3.1.3.2.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">31.2</td>
<td id="A1.3.3.1.3.2.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">50.8</td>
<td id="A1.3.3.1.3.2.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">42.6</td>
</tr>
<tr id="A1.3.3.1.4.3" class="ltx_tr">
<td id="A1.3.3.1.4.3.1" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left" style="padding:-2pt 2.0pt;">+ SwiGLU in FFN</td>
<td id="A1.3.3.1.4.3.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">12</td>
<td id="A1.3.3.1.4.3.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">12</td>
<td id="A1.3.3.1.4.3.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">12</td>
<td id="A1.3.3.1.4.3.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">768</td>
<td id="A1.3.3.1.4.3.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">134.1</td>
<td id="A1.3.3.1.4.3.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">43.1</td>
<td id="A1.3.3.1.4.3.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">28.9</td>
<td id="A1.3.3.1.4.3.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">58.1</td>
<td id="A1.3.3.1.4.3.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">62.3</td>
<td id="A1.3.3.1.4.3.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">42.3</td>
<td id="A1.3.3.1.4.3.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">34.6</td>
<td id="A1.3.3.1.4.3.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">31.5</td>
<td id="A1.3.3.1.4.3.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">50.1</td>
<td id="A1.3.3.1.4.3.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">43.9</td>
</tr>
<tr id="A1.3.3.1.5.4" class="ltx_tr">
<td id="A1.3.3.1.5.4.1" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left" style="padding:-2pt 2.0pt;">+ Use deep-thin structure</td>
<td id="A1.3.3.1.5.4.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">30</td>
<td id="A1.3.3.1.5.4.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">8</td>
<td id="A1.3.3.1.5.4.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">8</td>
<td id="A1.3.3.1.5.4.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">512</td>
<td id="A1.3.3.1.5.4.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">135.0</td>
<td id="A1.3.3.1.5.4.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">43.6</td>
<td id="A1.3.3.1.5.4.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">26.1</td>
<td id="A1.3.3.1.5.4.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">58.0</td>
<td id="A1.3.3.1.5.4.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">62.5</td>
<td id="A1.3.3.1.5.4.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">42.6</td>
<td id="A1.3.3.1.5.4.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">36.5</td>
<td id="A1.3.3.1.5.4.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">37.5</td>
<td id="A1.3.3.1.5.4.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">51.5</td>
<td id="A1.3.3.1.5.4.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">44.8</td>
</tr>
<tr id="A1.3.3.1.6.5" class="ltx_tr">
<td id="A1.3.3.1.6.5.1" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left" style="padding:-2pt 2.0pt;">+ Embedding share</td>
<td id="A1.3.3.1.6.5.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">30</td>
<td id="A1.3.3.1.6.5.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">8</td>
<td id="A1.3.3.1.6.5.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">8</td>
<td id="A1.3.3.1.6.5.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">512</td>
<td id="A1.3.3.1.6.5.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">118.6</td>
<td id="A1.3.3.1.6.5.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">44.4</td>
<td id="A1.3.3.1.6.5.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">26.0</td>
<td id="A1.3.3.1.6.5.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">56.2</td>
<td id="A1.3.3.1.6.5.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">62.8</td>
<td id="A1.3.3.1.6.5.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">43.1</td>
<td id="A1.3.3.1.6.5.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">35.9</td>
<td id="A1.3.3.1.6.5.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">36.0</td>
<td id="A1.3.3.1.6.5.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">52.6</td>
<td id="A1.3.3.1.6.5.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">44.6</td>
</tr>
<tr id="A1.3.3.1.7.6" class="ltx_tr">
<td id="A1.3.3.1.7.6.1" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left" style="padding:-2pt 2.0pt;">+ Multi-query attention</td>
<td id="A1.3.3.1.7.6.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">30</td>
<td id="A1.3.3.1.7.6.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">9</td>
<td id="A1.3.3.1.7.6.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">3</td>
<td id="A1.3.3.1.7.6.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">576</td>
<td id="A1.3.3.1.7.6.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">124.6</td>
<td id="A1.3.3.1.7.6.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">45.5</td>
<td id="A1.3.3.1.7.6.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">27.7</td>
<td id="A1.3.3.1.7.6.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">58.3</td>
<td id="A1.3.3.1.7.6.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">64.6</td>
<td id="A1.3.3.1.7.6.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">41.9</td>
<td id="A1.3.3.1.7.6.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">36.4</td>
<td id="A1.3.3.1.7.6.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">35.4</td>
<td id="A1.3.3.1.7.6.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">50.4</td>
<td id="A1.3.3.1.7.6.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">45.0</td>
</tr>
<tr id="A1.3.3.1.8.7" class="ltx_tr">
<td id="A1.3.3.1.8.7.1" class="ltx_td ltx_nopad_r ltx_align_left" style="padding:-2pt 2.0pt;">    (Train on 1T token)</td>
<td id="A1.3.3.1.8.7.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">30</td>
<td id="A1.3.3.1.8.7.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">9</td>
<td id="A1.3.3.1.8.7.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">3</td>
<td id="A1.3.3.1.8.7.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">576</td>
<td id="A1.3.3.1.8.7.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">124.6</td>
<td id="A1.3.3.1.8.7.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">43.9</td>
<td id="A1.3.3.1.8.7.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">27.1</td>
<td id="A1.3.3.1.8.7.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">60.2</td>
<td id="A1.3.3.1.8.7.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">65.3</td>
<td id="A1.3.3.1.8.7.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">42.4</td>
<td id="A1.3.3.1.8.7.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">38.9</td>
<td id="A1.3.3.1.8.7.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">39.5</td>
<td id="A1.3.3.1.8.7.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">53.1</td>
<td id="A1.3.3.1.8.7.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;"><span id="A1.3.3.1.8.7.15.1" class="ltx_text ltx_font_bold">46.3</span></td>
</tr>
<tr id="A1.3.3.1.9.8" class="ltx_tr">
<td id="A1.3.3.1.9.8.1" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left" style="padding:-2pt 2.0pt;">+ Layer sharing</td>
<td id="A1.3.3.1.9.8.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">30</td>
<td id="A1.3.3.1.9.8.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">9</td>
<td id="A1.3.3.1.9.8.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">3</td>
<td id="A1.3.3.1.9.8.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">576</td>
<td id="A1.3.3.1.9.8.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">124.6</td>
<td id="A1.3.3.1.9.8.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">44.4</td>
<td id="A1.3.3.1.9.8.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">27.0</td>
<td id="A1.3.3.1.9.8.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">61.5</td>
<td id="A1.3.3.1.9.8.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">65.1</td>
<td id="A1.3.3.1.9.8.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">43.0</td>
<td id="A1.3.3.1.9.8.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">37.6</td>
<td id="A1.3.3.1.9.8.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">37.8</td>
<td id="A1.3.3.1.9.8.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">52.0</td>
<td id="A1.3.3.1.9.8.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">46.1</td>
</tr>
<tr id="A1.3.3.1.10.9" class="ltx_tr">
<td id="A1.3.3.1.10.9.1" class="ltx_td ltx_nopad_r ltx_align_left" style="padding:-2pt 2.0pt;">    (Train on 1T token)</td>
<td id="A1.3.3.1.10.9.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">30</td>
<td id="A1.3.3.1.10.9.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">9</td>
<td id="A1.3.3.1.10.9.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">3</td>
<td id="A1.3.3.1.10.9.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">576</td>
<td id="A1.3.3.1.10.9.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">124.6</td>
<td id="A1.3.3.1.10.9.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">45.8</td>
<td id="A1.3.3.1.10.9.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">28.7</td>
<td id="A1.3.3.1.10.9.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">60.4</td>
<td id="A1.3.3.1.10.9.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">65.7</td>
<td id="A1.3.3.1.10.9.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">42.9</td>
<td id="A1.3.3.1.10.9.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">39.5</td>
<td id="A1.3.3.1.10.9.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">41.1</td>
<td id="A1.3.3.1.10.9.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">52.1</td>
<td id="A1.3.3.1.10.9.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;"><span id="A1.3.3.1.10.9.15.1" class="ltx_text ltx_font_bold">47.0</span></td>
</tr>
<tr id="A1.3.3.1.11.10" class="ltx_tr">
<td id="A1.3.3.1.11.10.1" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_border_t" style="padding:-2pt 2.0pt;"><span id="A1.3.3.1.11.10.1.1" class="ltx_text ltx_font_bold">350M</span></td>
<td id="A1.3.3.1.11.10.2" class="ltx_td ltx_border_t" style="padding:-2pt 2.0pt;"></td>
<td id="A1.3.3.1.11.10.3" class="ltx_td ltx_border_t" style="padding:-2pt 2.0pt;"></td>
<td id="A1.3.3.1.11.10.4" class="ltx_td ltx_border_t" style="padding:-2pt 2.0pt;"></td>
<td id="A1.3.3.1.11.10.5" class="ltx_td ltx_border_t" style="padding:-2pt 2.0pt;"></td>
<td id="A1.3.3.1.11.10.6" class="ltx_td ltx_border_t" style="padding:-2pt 2.0pt;"></td>
<td id="A1.3.3.1.11.10.7" class="ltx_td ltx_border_t" style="padding:-2pt 2.0pt;"></td>
<td id="A1.3.3.1.11.10.8" class="ltx_td ltx_border_t" style="padding:-2pt 2.0pt;"></td>
<td id="A1.3.3.1.11.10.9" class="ltx_td ltx_border_t" style="padding:-2pt 2.0pt;"></td>
<td id="A1.3.3.1.11.10.10" class="ltx_td ltx_border_t" style="padding:-2pt 2.0pt;"></td>
<td id="A1.3.3.1.11.10.11" class="ltx_td ltx_border_t" style="padding:-2pt 2.0pt;"></td>
<td id="A1.3.3.1.11.10.12" class="ltx_td ltx_border_t" style="padding:-2pt 2.0pt;"></td>
<td id="A1.3.3.1.11.10.13" class="ltx_td ltx_border_t" style="padding:-2pt 2.0pt;"></td>
<td id="A1.3.3.1.11.10.14" class="ltx_td ltx_border_t" style="padding:-2pt 2.0pt;"></td>
<td id="A1.3.3.1.11.10.15" class="ltx_td ltx_border_t" style="padding:-2pt 2.0pt;"></td>
</tr>
<tr id="A1.3.3.1.12.11" class="ltx_tr">
<td id="A1.3.3.1.12.11.1" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left" style="padding:-2pt 2.0pt;">
<span id="A1.3.3.1.12.11.1.1" class="ltx_ERROR undefined">\hdashline</span>[0.8pt/1pt]
Baseline model</td>
<td id="A1.3.3.1.12.11.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">15</td>
<td id="A1.3.3.1.12.11.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">20</td>
<td id="A1.3.3.1.12.11.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">20</td>
<td id="A1.3.3.1.12.11.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">1280</td>
<td id="A1.3.3.1.12.11.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">376.8</td>
<td id="A1.3.3.1.12.11.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">50.3</td>
<td id="A1.3.3.1.12.11.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">27.6</td>
<td id="A1.3.3.1.12.11.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">53.8</td>
<td id="A1.3.3.1.12.11.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">68.1</td>
<td id="A1.3.3.1.12.11.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">44.1</td>
<td id="A1.3.3.1.12.11.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">42.6</td>
<td id="A1.3.3.1.12.11.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">40.1</td>
<td id="A1.3.3.1.12.11.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">52.4</td>
<td id="A1.3.3.1.12.11.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">47.4</td>
</tr>
<tr id="A1.3.3.1.13.12" class="ltx_tr">
<td id="A1.3.3.1.13.12.1" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left" style="padding:-2pt 2.0pt;">+ SwiGLU in FFN</td>
<td id="A1.3.3.1.13.12.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">15</td>
<td id="A1.3.3.1.13.12.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">20</td>
<td id="A1.3.3.1.13.12.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">20</td>
<td id="A1.3.3.1.13.12.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">1280</td>
<td id="A1.3.3.1.13.12.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">386.7</td>
<td id="A1.3.3.1.13.12.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">49.2</td>
<td id="A1.3.3.1.13.12.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">30.6</td>
<td id="A1.3.3.1.13.12.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">59.1</td>
<td id="A1.3.3.1.13.12.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">67.7</td>
<td id="A1.3.3.1.13.12.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">44.3</td>
<td id="A1.3.3.1.13.12.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">43.2</td>
<td id="A1.3.3.1.13.12.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">41.0</td>
<td id="A1.3.3.1.13.12.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">54.2</td>
<td id="A1.3.3.1.13.12.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">48.7</td>
</tr>
<tr id="A1.3.3.1.14.13" class="ltx_tr">
<td id="A1.3.3.1.14.13.1" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left" style="padding:-2pt 2.0pt;">+ Use deep-thin structure</td>
<td id="A1.3.3.1.14.13.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">32</td>
<td id="A1.3.3.1.14.13.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">14</td>
<td id="A1.3.3.1.14.13.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">14</td>
<td id="A1.3.3.1.14.13.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">896</td>
<td id="A1.3.3.1.14.13.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">380.3</td>
<td id="A1.3.3.1.14.13.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">50.7</td>
<td id="A1.3.3.1.14.13.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">31.4</td>
<td id="A1.3.3.1.14.13.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">59.4</td>
<td id="A1.3.3.1.14.13.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">67.8</td>
<td id="A1.3.3.1.14.13.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">43.3</td>
<td id="A1.3.3.1.14.13.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">46.2</td>
<td id="A1.3.3.1.14.13.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">43.8</td>
<td id="A1.3.3.1.14.13.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">56.2</td>
<td id="A1.3.3.1.14.13.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">49.8</td>
</tr>
<tr id="A1.3.3.1.15.14" class="ltx_tr">
<td id="A1.3.3.1.15.14.1" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left" style="padding:-2pt 2.0pt;">+ Embedding share</td>
<td id="A1.3.3.1.15.14.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">32</td>
<td id="A1.3.3.1.15.14.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">14</td>
<td id="A1.3.3.1.15.14.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">14</td>
<td id="A1.3.3.1.15.14.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">896</td>
<td id="A1.3.3.1.15.14.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">351.6</td>
<td id="A1.3.3.1.15.14.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">49.9</td>
<td id="A1.3.3.1.15.14.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">32.0</td>
<td id="A1.3.3.1.15.14.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">60.3</td>
<td id="A1.3.3.1.15.14.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">67.9</td>
<td id="A1.3.3.1.15.14.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">43.2</td>
<td id="A1.3.3.1.15.14.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">47.0</td>
<td id="A1.3.3.1.15.14.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">38.9</td>
<td id="A1.3.3.1.15.14.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">54.8</td>
<td id="A1.3.3.1.15.14.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">49.2</td>
</tr>
<tr id="A1.3.3.1.16.15" class="ltx_tr">
<td id="A1.3.3.1.16.15.1" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left" style="padding:-2pt 2.0pt;">+ Multi-query attention</td>
<td id="A1.3.3.1.16.15.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">32</td>
<td id="A1.3.3.1.16.15.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">15</td>
<td id="A1.3.3.1.16.15.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">5</td>
<td id="A1.3.3.1.16.15.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">960</td>
<td id="A1.3.3.1.16.15.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">345.3</td>
<td id="A1.3.3.1.16.15.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">51.4</td>
<td id="A1.3.3.1.16.15.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">31.3</td>
<td id="A1.3.3.1.16.15.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">61.0</td>
<td id="A1.3.3.1.16.15.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">68.1</td>
<td id="A1.3.3.1.16.15.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">43.6</td>
<td id="A1.3.3.1.16.15.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">47.2</td>
<td id="A1.3.3.1.16.15.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">41.6</td>
<td id="A1.3.3.1.16.15.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">55.4</td>
<td id="A1.3.3.1.16.15.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">49.9</td>
</tr>
<tr id="A1.3.3.1.17.16" class="ltx_tr">
<td id="A1.3.3.1.17.16.1" class="ltx_td ltx_nopad_r ltx_align_left" style="padding:-2pt 2.0pt;">    (Train on 1T token)</td>
<td id="A1.3.3.1.17.16.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">32</td>
<td id="A1.3.3.1.17.16.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">15</td>
<td id="A1.3.3.1.17.16.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">5</td>
<td id="A1.3.3.1.17.16.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">960</td>
<td id="A1.3.3.1.17.16.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">345.3</td>
<td id="A1.3.3.1.17.16.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">53.8</td>
<td id="A1.3.3.1.17.16.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">33.5</td>
<td id="A1.3.3.1.17.16.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">62.4</td>
<td id="A1.3.3.1.17.16.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">68.6</td>
<td id="A1.3.3.1.17.16.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">44.7</td>
<td id="A1.3.3.1.17.16.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">49.6</td>
<td id="A1.3.3.1.17.16.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">40.0</td>
<td id="A1.3.3.1.17.16.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">57.6</td>
<td id="A1.3.3.1.17.16.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;"><span id="A1.3.3.1.17.16.15.1" class="ltx_text ltx_font_bold">51.3</span></td>
</tr>
<tr id="A1.3.3.1.18.17" class="ltx_tr">
<td id="A1.3.3.1.18.17.1" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left" style="padding:-2pt 2.0pt;">+ Layer sharing</td>
<td id="A1.3.3.1.18.17.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">32</td>
<td id="A1.3.3.1.18.17.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">15</td>
<td id="A1.3.3.1.18.17.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">5</td>
<td id="A1.3.3.1.18.17.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">960</td>
<td id="A1.3.3.1.18.17.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">345.3</td>
<td id="A1.3.3.1.18.17.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">51.9</td>
<td id="A1.3.3.1.18.17.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">35.2</td>
<td id="A1.3.3.1.18.17.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">59.6</td>
<td id="A1.3.3.1.18.17.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">68.9</td>
<td id="A1.3.3.1.18.17.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">43.4</td>
<td id="A1.3.3.1.18.17.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">47.2</td>
<td id="A1.3.3.1.18.17.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">43.3</td>
<td id="A1.3.3.1.18.17.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">58.4</td>
<td id="A1.3.3.1.18.17.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding:-2pt 2.0pt;">51.0</td>
</tr>
<tr id="A1.3.3.1.19.18" class="ltx_tr">
<td id="A1.3.3.1.19.18.1" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_bb" style="padding:-2pt 2.0pt;">    (Train on 1T token)</td>
<td id="A1.3.3.1.19.18.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb" style="padding:-2pt 2.0pt;">32</td>
<td id="A1.3.3.1.19.18.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb" style="padding:-2pt 2.0pt;">15</td>
<td id="A1.3.3.1.19.18.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb" style="padding:-2pt 2.0pt;">5</td>
<td id="A1.3.3.1.19.18.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb" style="padding:-2pt 2.0pt;">960</td>
<td id="A1.3.3.1.19.18.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb" style="padding:-2pt 2.0pt;">345.3</td>
<td id="A1.3.3.1.19.18.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb" style="padding:-2pt 2.0pt;">54.4</td>
<td id="A1.3.3.1.19.18.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb" style="padding:-2pt 2.0pt;">32.5</td>
<td id="A1.3.3.1.19.18.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb" style="padding:-2pt 2.0pt;">62.8</td>
<td id="A1.3.3.1.19.18.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb" style="padding:-2pt 2.0pt;">69.8</td>
<td id="A1.3.3.1.19.18.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb" style="padding:-2pt 2.0pt;">44.1</td>
<td id="A1.3.3.1.19.18.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb" style="padding:-2pt 2.0pt;">50.6</td>
<td id="A1.3.3.1.19.18.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb" style="padding:-2pt 2.0pt;">45.8</td>
<td id="A1.3.3.1.19.18.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb" style="padding:-2pt 2.0pt;">57.2</td>
<td id="A1.3.3.1.19.18.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb" style="padding:-2pt 2.0pt;"><span id="A1.3.3.1.19.18.15.1" class="ltx_text ltx_font_bold">52.1</span></td>
</tr>
</tbody>
</table>
</span></div>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<section id="A2" class="ltx_appendix ltx_figure_panel">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix B </span>Depth vs Width</h2>

<div id="A2.p1" class="ltx_para">
<p id="A2.p1.1" class="ltx_p">We provide network depth versus width exploration results on zero-shot reasoning tasks in Table <a href="#A2" title="Appendix B Depth vs Width ‣ Appendix A Impact of Each Design Choice ‣ Broader Impact ‣ Acknowledgment ‣ 5 Conclusion ‣ 4 Related Work ‣ 3.6 On-device Profiling ‣ 3.5 Knowledge Distillation ‣ 3.4 Compatibility with Quantization ‣ 3.3.2 API Calling ‣ 3.3.1 Chat ‣ 3.3 Downstream Tasks ‣ 3.2 Main Results ‣ 3 Experiments ‣ 2.3 Layer Sharing ‣ 2 Improving Sub-billion Scale LLM Design ‣ MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases" class="ltx_ref"><span class="ltx_text ltx_ref_tag">B</span></a>, as well as results on question answering and reading comprehension tasks in Table <a href="#A2" title="Appendix B Depth vs Width ‣ Appendix A Impact of Each Design Choice ‣ Broader Impact ‣ Acknowledgment ‣ 5 Conclusion ‣ 4 Related Work ‣ 3.6 On-device Profiling ‣ 3.5 Knowledge Distillation ‣ 3.4 Compatibility with Quantization ‣ 3.3.2 API Calling ‣ 3.3.1 Chat ‣ 3.3 Downstream Tasks ‣ 3.2 Main Results ‣ 3 Experiments ‣ 2.3 Layer Sharing ‣ 2 Improving Sub-billion Scale LLM Design ‣ MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases" class="ltx_ref"><span class="ltx_text ltx_ref_tag">B</span></a>. The findings indicate that shallow architectures with fewer than 10 layers perform poorly in reasoning or handling comprehension tasks. Models with 10-20 layers exhibit improved accuracy, while further increasing depth continues to provide significant benefits across all three tasks. Notably, the optimal depth is found to be near 30 layers for sub-billion scale models.</p>
</div>
<figure id="A2.1" class="ltx_table">

<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table"><span id="A2.1.1.1.1" class="ltx_text" style="font-size:90%;">Table 9</span>: </span><span id="A2.1.2.2" class="ltx_text" style="font-size:90%;">Ablation study on depth versus width in architecture design, illustrated in Figure <a href="#S2.F4" title="Figure 4 ‣ 2.2.2 Architecture Depth vs Width ‣ 2.2 Building a Strong Baseline ‣ 2 Improving Sub-billion Scale LLM Design ‣ MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a> (a)(b). For compact models, prioritizing depth over width yields superior performance, assessed through zero-shot common sense reasoning tasks.</span></figcaption><div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_1">
<div id="A2.1.3" class="ltx_inline-block ltx_figure_panel ltx_transformed_outer" style="width:390.3pt;height:253pt;vertical-align:-0.7pt;"><span class="ltx_transformed_inner" style="transform:translate(-83.3pt,53.8pt) scale(0.700905616645088,0.700905616645088) ;">
<table id="A2.1.3.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A2.1.3.1.1.1" class="ltx_tr">
<th id="A2.1.3.1.1.1.1" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_th_row ltx_border_tt" style="padding:-2pt 3.4pt;"><span id="A2.1.3.1.1.1.1.1" class="ltx_text ltx_font_bold">#Layer</span></th>
<th id="A2.1.3.1.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_th_row ltx_border_tt" style="padding:-2pt 3.4pt;"><span id="A2.1.3.1.1.1.2.1" class="ltx_text ltx_font_bold">#Heads</span></th>
<th id="A2.1.3.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_th_row ltx_border_tt" style="padding:-2pt 3.4pt;"><span id="A2.1.3.1.1.1.3.1" class="ltx_text ltx_font_bold">Dim</span></th>
<th id="A2.1.3.1.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 3.4pt;"><span id="A2.1.3.1.1.1.4.1" class="ltx_text ltx_font_bold">#Params(M)</span></th>
<th id="A2.1.3.1.1.1.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 3.4pt;"><span id="A2.1.3.1.1.1.5.1" class="ltx_text ltx_font_bold">ARC-e</span></th>
<th id="A2.1.3.1.1.1.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 3.4pt;"><span id="A2.1.3.1.1.1.6.1" class="ltx_text ltx_font_bold">ARC-c</span></th>
<th id="A2.1.3.1.1.1.7" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 3.4pt;"><span id="A2.1.3.1.1.1.7.1" class="ltx_text ltx_font_bold">BoolQ</span></th>
<th id="A2.1.3.1.1.1.8" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 3.4pt;"><span id="A2.1.3.1.1.1.8.1" class="ltx_text ltx_font_bold">PIQA</span></th>
<th id="A2.1.3.1.1.1.9" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 3.4pt;"><span id="A2.1.3.1.1.1.9.1" class="ltx_text ltx_font_bold">SIQA</span></th>
<th id="A2.1.3.1.1.1.10" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 3.4pt;"><span id="A2.1.3.1.1.1.10.1" class="ltx_text ltx_font_bold">HellaSwag</span></th>
<th id="A2.1.3.1.1.1.11" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 3.4pt;"><span id="A2.1.3.1.1.1.11.1" class="ltx_text ltx_font_bold">OBQA</span></th>
<th id="A2.1.3.1.1.1.12" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 3.4pt;"><span id="A2.1.3.1.1.1.12.1" class="ltx_text ltx_font_bold">WinoGrande</span></th>
<th id="A2.1.3.1.1.1.13" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 3.4pt;"><span id="A2.1.3.1.1.1.13.1" class="ltx_text ltx_font_bold">Avg.</span></th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A2.1.3.1.2.1" class="ltx_tr">
<th id="A2.1.3.1.2.1.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_t" style="padding:-2pt 3.4pt;">4</th>
<th id="A2.1.3.1.2.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_t" style="padding:-2pt 3.4pt;">20</th>
<th id="A2.1.3.1.2.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_t" style="padding:-2pt 3.4pt;">1280</th>
<td id="A2.1.3.1.2.1.4" class="ltx_td ltx_align_center ltx_border_t" style="padding:-2pt 3.4pt;">163.2</td>
<td id="A2.1.3.1.2.1.5" class="ltx_td ltx_align_center ltx_border_t" style="padding:-2pt 3.4pt;">42.0</td>
<td id="A2.1.3.1.2.1.6" class="ltx_td ltx_align_center ltx_border_t" style="padding:-2pt 3.4pt;">26.1</td>
<td id="A2.1.3.1.2.1.7" class="ltx_td ltx_align_center ltx_border_t" style="padding:-2pt 3.4pt;">61.0</td>
<td id="A2.1.3.1.2.1.8" class="ltx_td ltx_align_center ltx_border_t" style="padding:-2pt 3.4pt;">61.5</td>
<td id="A2.1.3.1.2.1.9" class="ltx_td ltx_align_center ltx_border_t" style="padding:-2pt 3.4pt;">41.3</td>
<td id="A2.1.3.1.2.1.10" class="ltx_td ltx_align_center ltx_border_t" style="padding:-2pt 3.4pt;">32.9</td>
<td id="A2.1.3.1.2.1.11" class="ltx_td ltx_align_center ltx_border_t" style="padding:-2pt 3.4pt;">31.4</td>
<td id="A2.1.3.1.2.1.12" class="ltx_td ltx_align_center ltx_border_t" style="padding:-2pt 3.4pt;">49.8</td>
<td id="A2.1.3.1.2.1.13" class="ltx_td ltx_align_center ltx_border_t" style="padding:-2pt 3.4pt;">43.3</td>
</tr>
<tr id="A2.1.3.1.3.2" class="ltx_tr">
<th id="A2.1.3.1.3.2.1" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 3.4pt;">6</th>
<th id="A2.1.3.1.3.2.2" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 3.4pt;">16</th>
<th id="A2.1.3.1.3.2.3" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 3.4pt;">1024</th>
<td id="A2.1.3.1.3.2.4" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">142.6</td>
<td id="A2.1.3.1.3.2.5" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">42.2</td>
<td id="A2.1.3.1.3.2.6" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">25.5</td>
<td id="A2.1.3.1.3.2.7" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">51.0</td>
<td id="A2.1.3.1.3.2.8" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">62.6</td>
<td id="A2.1.3.1.3.2.9" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">41.6</td>
<td id="A2.1.3.1.3.2.10" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">33.5</td>
<td id="A2.1.3.1.3.2.11" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">32.1</td>
<td id="A2.1.3.1.3.2.12" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">50.0</td>
<td id="A2.1.3.1.3.2.13" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">42.3</td>
</tr>
<tr id="A2.1.3.1.4.3" class="ltx_tr">
<th id="A2.1.3.1.4.3.1" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 3.4pt;">8</th>
<th id="A2.1.3.1.4.3.2" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 3.4pt;">14</th>
<th id="A2.1.3.1.4.3.3" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 3.4pt;">896</th>
<td id="A2.1.3.1.4.3.4" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">138.1</td>
<td id="A2.1.3.1.4.3.5" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">43.6</td>
<td id="A2.1.3.1.4.3.6" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">25.5</td>
<td id="A2.1.3.1.4.3.7" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">48.6</td>
<td id="A2.1.3.1.4.3.8" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">61.8</td>
<td id="A2.1.3.1.4.3.9" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">40.9</td>
<td id="A2.1.3.1.4.3.10" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">34.0</td>
<td id="A2.1.3.1.4.3.11" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">37.5</td>
<td id="A2.1.3.1.4.3.12" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">52.0</td>
<td id="A2.1.3.1.4.3.13" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">43.0</td>
</tr>
<tr id="A2.1.3.1.5.4" class="ltx_tr">
<th id="A2.1.3.1.5.4.1" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 3.4pt;">12</th>
<th id="A2.1.3.1.5.4.2" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 3.4pt;">12</th>
<th id="A2.1.3.1.5.4.3" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 3.4pt;">768</th>
<td id="A2.1.3.1.5.4.4" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">134.1</td>
<td id="A2.1.3.1.5.4.5" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">43.1</td>
<td id="A2.1.3.1.5.4.6" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">28.9</td>
<td id="A2.1.3.1.5.4.7" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">58.1</td>
<td id="A2.1.3.1.5.4.8" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">62.3</td>
<td id="A2.1.3.1.5.4.9" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">42.3</td>
<td id="A2.1.3.1.5.4.10" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">34.6</td>
<td id="A2.1.3.1.5.4.11" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">31.5</td>
<td id="A2.1.3.1.5.4.12" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">50.1</td>
<td id="A2.1.3.1.5.4.13" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">43.9</td>
</tr>
<tr id="A2.1.3.1.6.5" class="ltx_tr">
<th id="A2.1.3.1.6.5.1" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 3.4pt;">18</th>
<th id="A2.1.3.1.6.5.2" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 3.4pt;">10</th>
<th id="A2.1.3.1.6.5.3" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 3.4pt;">640</th>
<td id="A2.1.3.1.6.5.4" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">132.4</td>
<td id="A2.1.3.1.6.5.5" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">41.0</td>
<td id="A2.1.3.1.6.5.6" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">25.5</td>
<td id="A2.1.3.1.6.5.7" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">59.3</td>
<td id="A2.1.3.1.6.5.8" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">62.6</td>
<td id="A2.1.3.1.6.5.9" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">41.6</td>
<td id="A2.1.3.1.6.5.10" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">34.2</td>
<td id="A2.1.3.1.6.5.11" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">36.7</td>
<td id="A2.1.3.1.6.5.12" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">50.1</td>
<td id="A2.1.3.1.6.5.13" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">43.9</td>
</tr>
<tr id="A2.1.3.1.7.6" class="ltx_tr">
<th id="A2.1.3.1.7.6.1" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 3.4pt;">24</th>
<th id="A2.1.3.1.7.6.2" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 3.4pt;">9</th>
<th id="A2.1.3.1.7.6.3" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 3.4pt;">576</th>
<td id="A2.1.3.1.7.6.4" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">132.4</td>
<td id="A2.1.3.1.7.6.5" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">43.6</td>
<td id="A2.1.3.1.7.6.6" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">27.0</td>
<td id="A2.1.3.1.7.6.7" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">59.7</td>
<td id="A2.1.3.1.7.6.8" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">63.8</td>
<td id="A2.1.3.1.7.6.9" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">42.0</td>
<td id="A2.1.3.1.7.6.10" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">36.2</td>
<td id="A2.1.3.1.7.6.11" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">33.7</td>
<td id="A2.1.3.1.7.6.12" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">52.2</td>
<td id="A2.1.3.1.7.6.13" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;"><span id="A2.1.3.1.7.6.13.1" class="ltx_text ltx_font_bold">44.8</span></td>
</tr>
<tr id="A2.1.3.1.8.7" class="ltx_tr">
<th id="A2.1.3.1.8.7.1" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 3.4pt;">30</th>
<th id="A2.1.3.1.8.7.2" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 3.4pt;">8</th>
<th id="A2.1.3.1.8.7.3" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 3.4pt;">512</th>
<td id="A2.1.3.1.8.7.4" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">135.0</td>
<td id="A2.1.3.1.8.7.5" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">43.6</td>
<td id="A2.1.3.1.8.7.6" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">26.1</td>
<td id="A2.1.3.1.8.7.7" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">58.0</td>
<td id="A2.1.3.1.8.7.8" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">62.5</td>
<td id="A2.1.3.1.8.7.9" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">42.6</td>
<td id="A2.1.3.1.8.7.10" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">36.5</td>
<td id="A2.1.3.1.8.7.11" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">37.5</td>
<td id="A2.1.3.1.8.7.12" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">51.5</td>
<td id="A2.1.3.1.8.7.13" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;"><span id="A2.1.3.1.8.7.13.1" class="ltx_text ltx_font_bold">44.8</span></td>
</tr>
<tr id="A2.1.3.1.9.8" class="ltx_tr">
<th id="A2.1.3.1.9.8.1" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 3.4pt;">42</th>
<th id="A2.1.3.1.9.8.2" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 3.4pt;">7</th>
<th id="A2.1.3.1.9.8.3" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 3.4pt;">448</th>
<td id="A2.1.3.1.9.8.4" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">134.6</td>
<td id="A2.1.3.1.9.8.5" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">43.3</td>
<td id="A2.1.3.1.9.8.6" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">26.1</td>
<td id="A2.1.3.1.9.8.7" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">57.8</td>
<td id="A2.1.3.1.9.8.8" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">63.3</td>
<td id="A2.1.3.1.9.8.9" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">41.7</td>
<td id="A2.1.3.1.9.8.10" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">36.3</td>
<td id="A2.1.3.1.9.8.11" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">35.9</td>
<td id="A2.1.3.1.9.8.12" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">52.0</td>
<td id="A2.1.3.1.9.8.13" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">44.5</td>
</tr>
<tr id="A2.1.3.1.10.9" class="ltx_tr">
<th id="A2.1.3.1.10.9.1" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 3.4pt;">62</th>
<th id="A2.1.3.1.10.9.2" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 3.4pt;">6</th>
<th id="A2.1.3.1.10.9.3" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 3.4pt;">384</th>
<td id="A2.1.3.1.10.9.4" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">134.3</td>
<td id="A2.1.3.1.10.9.5" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">44.1</td>
<td id="A2.1.3.1.10.9.6" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">26.7</td>
<td id="A2.1.3.1.10.9.7" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">59.0</td>
<td id="A2.1.3.1.10.9.8" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">64.7</td>
<td id="A2.1.3.1.10.9.9" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">42.4</td>
<td id="A2.1.3.1.10.9.10" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">36.2</td>
<td id="A2.1.3.1.10.9.11" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">33.1</td>
<td id="A2.1.3.1.10.9.12" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">51.3</td>
<td id="A2.1.3.1.10.9.13" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">44.7</td>
</tr>
<tr id="A2.1.3.1.11.10" class="ltx_tr">
<th id="A2.1.3.1.11.10.1" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 3.4pt;">
<span id="A2.1.3.1.11.10.1.1" class="ltx_ERROR undefined">\hdashline</span>[0.8pt/1pt]
5</th>
<th id="A2.1.3.1.11.10.2" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 3.4pt;">32</th>
<th id="A2.1.3.1.11.10.3" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 3.4pt;">2048</th>
<td id="A2.1.3.1.11.10.4" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">388.0</td>
<td id="A2.1.3.1.11.10.5" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">47.9</td>
<td id="A2.1.3.1.11.10.6" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">28.8</td>
<td id="A2.1.3.1.11.10.7" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">62.2</td>
<td id="A2.1.3.1.11.10.8" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">65.2</td>
<td id="A2.1.3.1.11.10.9" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">42.8</td>
<td id="A2.1.3.1.11.10.10" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">38.1</td>
<td id="A2.1.3.1.11.10.11" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">39.0</td>
<td id="A2.1.3.1.11.10.12" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">52.6</td>
<td id="A2.1.3.1.11.10.13" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">47.1</td>
</tr>
<tr id="A2.1.3.1.12.11" class="ltx_tr">
<th id="A2.1.3.1.12.11.1" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 3.4pt;">10</th>
<th id="A2.1.3.1.12.11.2" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 3.4pt;">24</th>
<th id="A2.1.3.1.12.11.3" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 3.4pt;">1536</th>
<td id="A2.1.3.1.12.11.4" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">381.4</td>
<td id="A2.1.3.1.12.11.5" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">49.1</td>
<td id="A2.1.3.1.12.11.6" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">29.6</td>
<td id="A2.1.3.1.12.11.7" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">60.6</td>
<td id="A2.1.3.1.12.11.8" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">67.1</td>
<td id="A2.1.3.1.12.11.9" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">44.0</td>
<td id="A2.1.3.1.12.11.10" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">42.7</td>
<td id="A2.1.3.1.12.11.11" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">41.2</td>
<td id="A2.1.3.1.12.11.12" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">54.4</td>
<td id="A2.1.3.1.12.11.13" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">48.6</td>
</tr>
<tr id="A2.1.3.1.13.12" class="ltx_tr">
<th id="A2.1.3.1.13.12.1" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 3.4pt;">12</th>
<th id="A2.1.3.1.13.12.2" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 3.4pt;">22</th>
<th id="A2.1.3.1.13.12.3" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 3.4pt;">1408</th>
<td id="A2.1.3.1.13.12.4" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">379.9</td>
<td id="A2.1.3.1.13.12.5" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">49.9</td>
<td id="A2.1.3.1.13.12.6" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">31.6</td>
<td id="A2.1.3.1.13.12.7" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">59.1</td>
<td id="A2.1.3.1.13.12.8" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">68.2</td>
<td id="A2.1.3.1.13.12.9" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">42.6</td>
<td id="A2.1.3.1.13.12.10" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">44.0</td>
<td id="A2.1.3.1.13.12.11" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">43.7</td>
<td id="A2.1.3.1.13.12.12" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">53.8</td>
<td id="A2.1.3.1.13.12.13" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">49.1</td>
</tr>
<tr id="A2.1.3.1.14.13" class="ltx_tr">
<th id="A2.1.3.1.14.13.1" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 3.4pt;">15</th>
<th id="A2.1.3.1.14.13.2" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 3.4pt;">20</th>
<th id="A2.1.3.1.14.13.3" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 3.4pt;">1280</th>
<td id="A2.1.3.1.14.13.4" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">386.7</td>
<td id="A2.1.3.1.14.13.5" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">49.2</td>
<td id="A2.1.3.1.14.13.6" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">30.6</td>
<td id="A2.1.3.1.14.13.7" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">59.1</td>
<td id="A2.1.3.1.14.13.8" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">67.7</td>
<td id="A2.1.3.1.14.13.9" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">44.3</td>
<td id="A2.1.3.1.14.13.10" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">43.2</td>
<td id="A2.1.3.1.14.13.11" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">41.0</td>
<td id="A2.1.3.1.14.13.12" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">54.2</td>
<td id="A2.1.3.1.14.13.13" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">48.7</td>
</tr>
<tr id="A2.1.3.1.15.14" class="ltx_tr">
<th id="A2.1.3.1.15.14.1" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 3.4pt;">19</th>
<th id="A2.1.3.1.15.14.2" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 3.4pt;">18</th>
<th id="A2.1.3.1.15.14.3" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 3.4pt;">1152</th>
<td id="A2.1.3.1.15.14.4" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">376.3</td>
<td id="A2.1.3.1.15.14.5" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">50.8</td>
<td id="A2.1.3.1.15.14.6" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">30.9</td>
<td id="A2.1.3.1.15.14.7" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">56.3</td>
<td id="A2.1.3.1.15.14.8" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">69.1</td>
<td id="A2.1.3.1.15.14.9" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">43.8</td>
<td id="A2.1.3.1.15.14.10" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">45.2</td>
<td id="A2.1.3.1.15.14.11" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">39.6</td>
<td id="A2.1.3.1.15.14.12" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">54.5</td>
<td id="A2.1.3.1.15.14.13" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">48.8</td>
</tr>
<tr id="A2.1.3.1.16.15" class="ltx_tr">
<th id="A2.1.3.1.16.15.1" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 3.4pt;">24</th>
<th id="A2.1.3.1.16.15.2" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 3.4pt;">16</th>
<th id="A2.1.3.1.16.15.3" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 3.4pt;">1024</th>
<td id="A2.1.3.1.16.15.4" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">373.8</td>
<td id="A2.1.3.1.16.15.5" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">50.7</td>
<td id="A2.1.3.1.16.15.6" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">33.8</td>
<td id="A2.1.3.1.16.15.7" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">58.8</td>
<td id="A2.1.3.1.16.15.8" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">68.6</td>
<td id="A2.1.3.1.16.15.9" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">43.5</td>
<td id="A2.1.3.1.16.15.10" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">45.0</td>
<td id="A2.1.3.1.16.15.11" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">40.0</td>
<td id="A2.1.3.1.16.15.12" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">54.9</td>
<td id="A2.1.3.1.16.15.13" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">49.4</td>
</tr>
<tr id="A2.1.3.1.17.16" class="ltx_tr">
<th id="A2.1.3.1.17.16.1" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 3.4pt;">28</th>
<th id="A2.1.3.1.17.16.2" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 3.4pt;">15</th>
<th id="A2.1.3.1.17.16.3" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 3.4pt;">960</th>
<td id="A2.1.3.1.17.16.4" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">371.1</td>
<td id="A2.1.3.1.17.16.5" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">51.7</td>
<td id="A2.1.3.1.17.16.6" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">33.2</td>
<td id="A2.1.3.1.17.16.7" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">57.6</td>
<td id="A2.1.3.1.17.16.8" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">67.9</td>
<td id="A2.1.3.1.17.16.9" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">42.9</td>
<td id="A2.1.3.1.17.16.10" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">46.0</td>
<td id="A2.1.3.1.17.16.11" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">37.7</td>
<td id="A2.1.3.1.17.16.12" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">53.9</td>
<td id="A2.1.3.1.17.16.13" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">48.9</td>
</tr>
<tr id="A2.1.3.1.18.17" class="ltx_tr">
<th id="A2.1.3.1.18.17.1" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 3.4pt;">32</th>
<th id="A2.1.3.1.18.17.2" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 3.4pt;">14</th>
<th id="A2.1.3.1.18.17.3" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 3.4pt;">896</th>
<td id="A2.1.3.1.18.17.4" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">380.3</td>
<td id="A2.1.3.1.18.17.5" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">50.7</td>
<td id="A2.1.3.1.18.17.6" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">31.4</td>
<td id="A2.1.3.1.18.17.7" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">59.4</td>
<td id="A2.1.3.1.18.17.8" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">67.8</td>
<td id="A2.1.3.1.18.17.9" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">43.3</td>
<td id="A2.1.3.1.18.17.10" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">46.2</td>
<td id="A2.1.3.1.18.17.11" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">43.8</td>
<td id="A2.1.3.1.18.17.12" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">56.2</td>
<td id="A2.1.3.1.18.17.13" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;"><span id="A2.1.3.1.18.17.13.1" class="ltx_text ltx_font_bold">49.8</span></td>
</tr>
<tr id="A2.1.3.1.19.18" class="ltx_tr">
<th id="A2.1.3.1.19.18.1" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 3.4pt;">46</th>
<th id="A2.1.3.1.19.18.2" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 3.4pt;">12</th>
<th id="A2.1.3.1.19.18.3" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 3.4pt;">768</th>
<td id="A2.1.3.1.19.18.4" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">374.7</td>
<td id="A2.1.3.1.19.18.5" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">49.7</td>
<td id="A2.1.3.1.19.18.6" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">30.5</td>
<td id="A2.1.3.1.19.18.7" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">59.5</td>
<td id="A2.1.3.1.19.18.8" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">68.1</td>
<td id="A2.1.3.1.19.18.9" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">45.1</td>
<td id="A2.1.3.1.19.18.10" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">44.9</td>
<td id="A2.1.3.1.19.18.11" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">43.4</td>
<td id="A2.1.3.1.19.18.12" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">55.5</td>
<td id="A2.1.3.1.19.18.13" class="ltx_td ltx_align_center" style="padding:-2pt 3.4pt;">49.6</td>
</tr>
<tr id="A2.1.3.1.20.19" class="ltx_tr">
<th id="A2.1.3.1.20.19.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_bb" style="padding:-2pt 3.4pt;">66</th>
<th id="A2.1.3.1.20.19.2" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_bb" style="padding:-2pt 3.4pt;">10</th>
<th id="A2.1.3.1.20.19.3" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_bb" style="padding:-2pt 3.4pt;">640</th>
<td id="A2.1.3.1.20.19.4" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 3.4pt;">376.2</td>
<td id="A2.1.3.1.20.19.5" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 3.4pt;">50.5</td>
<td id="A2.1.3.1.20.19.6" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 3.4pt;">31.8</td>
<td id="A2.1.3.1.20.19.7" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 3.4pt;">61.0</td>
<td id="A2.1.3.1.20.19.8" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 3.4pt;">67.4</td>
<td id="A2.1.3.1.20.19.9" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 3.4pt;">43.8</td>
<td id="A2.1.3.1.20.19.10" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 3.4pt;">46.0</td>
<td id="A2.1.3.1.20.19.11" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 3.4pt;">40.1</td>
<td id="A2.1.3.1.20.19.12" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 3.4pt;">55.6</td>
<td id="A2.1.3.1.20.19.13" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 3.4pt;">49.5</td>
</tr>
</tbody>
</table>
</span></div>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="A2.1.tab1" class="ltx_table ltx_figure_panel">

<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table"><span id="A2.1.tab1.1.1.1" class="ltx_text" style="font-size:90%;">Table 10</span>: </span><span id="A2.1.tab1.2.2" class="ltx_text" style="font-size:90%;">Ablation study on depth vs. width in architecture design on TQA and RACE datasets, depicted in Figure <a href="#S2.F4" title="Figure 4 ‣ 2.2.2 Architecture Depth vs Width ‣ 2.2 Building a Strong Baseline ‣ 2 Improving Sub-billion Scale LLM Design ‣ MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a> (c-f).</span></figcaption><div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_2">
<div id="A2.1.tab1.3" class="ltx_inline-block ltx_figure_panel ltx_transformed_outer" style="width:238.5pt;height:247.7pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(-62.7pt,65.1pt) scale(0.65541094938869,0.65541094938869) ;">
<table id="A2.1.tab1.3.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A2.1.tab1.3.1.1.1" class="ltx_tr">
<th id="A2.1.tab1.3.1.1.1.1" class="ltx_td ltx_th ltx_th_row ltx_border_tt" style="padding:-2pt 2.8pt;"></th>
<th id="A2.1.tab1.3.1.1.1.2" class="ltx_td ltx_th ltx_th_column ltx_th_row ltx_border_tt" style="padding:-2pt 2.8pt;"></th>
<th id="A2.1.tab1.3.1.1.1.3" class="ltx_td ltx_th ltx_th_column ltx_th_row ltx_border_tt" style="padding:-2pt 2.8pt;"></th>
<th id="A2.1.tab1.3.1.1.1.4" class="ltx_td ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 2.8pt;"></th>
<th id="A2.1.tab1.3.1.1.1.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 2.8pt;" colspan="3">
<span id="A2.1.tab1.3.1.1.1.5.1" class="ltx_text ltx_font_bold">TQA</span> (F1 score)</th>
<th id="A2.1.tab1.3.1.1.1.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 2.8pt;" colspan="2">
<span id="A2.1.tab1.3.1.1.1.6.1" class="ltx_text ltx_font_bold">RACE</span> (Acc)</th>
</tr>
<tr id="A2.1.tab1.3.1.2.2" class="ltx_tr">
<th id="A2.1.tab1.3.1.2.2.1" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_th_row" style="padding:-2pt 2.8pt;"><span id="A2.1.tab1.3.1.2.2.1.1" class="ltx_text ltx_font_bold">#Layer</span></th>
<th id="A2.1.tab1.3.1.2.2.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_th_row" style="padding:-2pt 2.8pt;"><span id="A2.1.tab1.3.1.2.2.2.1" class="ltx_text ltx_font_bold">#Heads</span></th>
<th id="A2.1.tab1.3.1.2.2.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_th_row" style="padding:-2pt 2.8pt;"><span id="A2.1.tab1.3.1.2.2.3.1" class="ltx_text ltx_font_bold">Dim</span></th>
<th id="A2.1.tab1.3.1.2.2.4" class="ltx_td ltx_align_center ltx_th ltx_th_column" style="padding:-2pt 2.8pt;"><span id="A2.1.tab1.3.1.2.2.4.1" class="ltx_text ltx_font_bold">#Params(M)</span></th>
<th id="A2.1.tab1.3.1.2.2.5" class="ltx_td ltx_align_center ltx_th ltx_th_column" style="padding:-2pt 2.8pt;">1-shot</th>
<th id="A2.1.tab1.3.1.2.2.6" class="ltx_td ltx_align_center ltx_th ltx_th_column" style="padding:-2pt 2.8pt;">5-shot</th>
<th id="A2.1.tab1.3.1.2.2.7" class="ltx_td ltx_align_center ltx_th ltx_th_column" style="padding:-2pt 2.8pt;">64-shot</th>
<th id="A2.1.tab1.3.1.2.2.8" class="ltx_td ltx_align_center ltx_th ltx_th_column" style="padding:-2pt 2.8pt;">middle</th>
<th id="A2.1.tab1.3.1.2.2.9" class="ltx_td ltx_align_center ltx_th ltx_th_column" style="padding:-2pt 2.8pt;">high</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A2.1.tab1.3.1.3.1" class="ltx_tr">
<th id="A2.1.tab1.3.1.3.1.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_t" style="padding:-2pt 2.8pt;">4</th>
<th id="A2.1.tab1.3.1.3.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_t" style="padding:-2pt 2.8pt;">20</th>
<th id="A2.1.tab1.3.1.3.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_t" style="padding:-2pt 2.8pt;">1280</th>
<td id="A2.1.tab1.3.1.3.1.4" class="ltx_td ltx_align_center ltx_border_t" style="padding:-2pt 2.8pt;">163.2</td>
<td id="A2.1.tab1.3.1.3.1.5" class="ltx_td ltx_align_center ltx_border_t" style="padding:-2pt 2.8pt;">2.0</td>
<td id="A2.1.tab1.3.1.3.1.6" class="ltx_td ltx_align_center ltx_border_t" style="padding:-2pt 2.8pt;">3.9</td>
<td id="A2.1.tab1.3.1.3.1.7" class="ltx_td ltx_align_center ltx_border_t" style="padding:-2pt 2.8pt;">3.9</td>
<td id="A2.1.tab1.3.1.3.1.8" class="ltx_td ltx_align_center ltx_border_t" style="padding:-2pt 2.8pt;">33.2</td>
<td id="A2.1.tab1.3.1.3.1.9" class="ltx_td ltx_align_center ltx_border_t" style="padding:-2pt 2.8pt;">26.0</td>
</tr>
<tr id="A2.1.tab1.3.1.4.2" class="ltx_tr">
<th id="A2.1.tab1.3.1.4.2.1" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">6</th>
<th id="A2.1.tab1.3.1.4.2.2" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">16</th>
<th id="A2.1.tab1.3.1.4.2.3" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">1024</th>
<td id="A2.1.tab1.3.1.4.2.4" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">142.6</td>
<td id="A2.1.tab1.3.1.4.2.5" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">3.5</td>
<td id="A2.1.tab1.3.1.4.2.6" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">4.9</td>
<td id="A2.1.tab1.3.1.4.2.7" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">4.6</td>
<td id="A2.1.tab1.3.1.4.2.8" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">29.2</td>
<td id="A2.1.tab1.3.1.4.2.9" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">25.5</td>
</tr>
<tr id="A2.1.tab1.3.1.5.3" class="ltx_tr">
<th id="A2.1.tab1.3.1.5.3.1" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">8</th>
<th id="A2.1.tab1.3.1.5.3.2" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">14</th>
<th id="A2.1.tab1.3.1.5.3.3" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">896</th>
<td id="A2.1.tab1.3.1.5.3.4" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">138.1</td>
<td id="A2.1.tab1.3.1.5.3.5" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">4.2</td>
<td id="A2.1.tab1.3.1.5.3.6" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">5.2</td>
<td id="A2.1.tab1.3.1.5.3.7" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">4.7</td>
<td id="A2.1.tab1.3.1.5.3.8" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">29.2</td>
<td id="A2.1.tab1.3.1.5.3.9" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">24.3</td>
</tr>
<tr id="A2.1.tab1.3.1.6.4" class="ltx_tr">
<th id="A2.1.tab1.3.1.6.4.1" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">12</th>
<th id="A2.1.tab1.3.1.6.4.2" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">12</th>
<th id="A2.1.tab1.3.1.6.4.3" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">768</th>
<td id="A2.1.tab1.3.1.6.4.4" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">134.1</td>
<td id="A2.1.tab1.3.1.6.4.5" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">5.1</td>
<td id="A2.1.tab1.3.1.6.4.6" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">6.1</td>
<td id="A2.1.tab1.3.1.6.4.7" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">6.3</td>
<td id="A2.1.tab1.3.1.6.4.8" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">35.6</td>
<td id="A2.1.tab1.3.1.6.4.9" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">27.3</td>
</tr>
<tr id="A2.1.tab1.3.1.7.5" class="ltx_tr">
<th id="A2.1.tab1.3.1.7.5.1" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">18</th>
<th id="A2.1.tab1.3.1.7.5.2" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">10</th>
<th id="A2.1.tab1.3.1.7.5.3" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">640</th>
<td id="A2.1.tab1.3.1.7.5.4" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">132.4</td>
<td id="A2.1.tab1.3.1.7.5.5" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">4.7</td>
<td id="A2.1.tab1.3.1.7.5.6" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">6.8</td>
<td id="A2.1.tab1.3.1.7.5.7" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">5.2</td>
<td id="A2.1.tab1.3.1.7.5.8" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">34.3</td>
<td id="A2.1.tab1.3.1.7.5.9" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">27.4</td>
</tr>
<tr id="A2.1.tab1.3.1.8.6" class="ltx_tr">
<th id="A2.1.tab1.3.1.8.6.1" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">24</th>
<th id="A2.1.tab1.3.1.8.6.2" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">9</th>
<th id="A2.1.tab1.3.1.8.6.3" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">576</th>
<td id="A2.1.tab1.3.1.8.6.4" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">132.4</td>
<td id="A2.1.tab1.3.1.8.6.5" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">5.9</td>
<td id="A2.1.tab1.3.1.8.6.6" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">7.2</td>
<td id="A2.1.tab1.3.1.8.6.7" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">7.3</td>
<td id="A2.1.tab1.3.1.8.6.8" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">35.3</td>
<td id="A2.1.tab1.3.1.8.6.9" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">28.6</td>
</tr>
<tr id="A2.1.tab1.3.1.9.7" class="ltx_tr">
<th id="A2.1.tab1.3.1.9.7.1" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">30</th>
<th id="A2.1.tab1.3.1.9.7.2" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">8</th>
<th id="A2.1.tab1.3.1.9.7.3" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">512</th>
<td id="A2.1.tab1.3.1.9.7.4" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">135.0</td>
<td id="A2.1.tab1.3.1.9.7.5" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">6.2</td>
<td id="A2.1.tab1.3.1.9.7.6" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">6.8</td>
<td id="A2.1.tab1.3.1.9.7.7" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">6.9</td>
<td id="A2.1.tab1.3.1.9.7.8" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">34.2</td>
<td id="A2.1.tab1.3.1.9.7.9" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">27.9</td>
</tr>
<tr id="A2.1.tab1.3.1.10.8" class="ltx_tr">
<th id="A2.1.tab1.3.1.10.8.1" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">42</th>
<th id="A2.1.tab1.3.1.10.8.2" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">7</th>
<th id="A2.1.tab1.3.1.10.8.3" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">448</th>
<td id="A2.1.tab1.3.1.10.8.4" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">134.6</td>
<td id="A2.1.tab1.3.1.10.8.5" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">6.1</td>
<td id="A2.1.tab1.3.1.10.8.6" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">7.2</td>
<td id="A2.1.tab1.3.1.10.8.7" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">7.2</td>
<td id="A2.1.tab1.3.1.10.8.8" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">34.4</td>
<td id="A2.1.tab1.3.1.10.8.9" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">28.6</td>
</tr>
<tr id="A2.1.tab1.3.1.11.9" class="ltx_tr">
<th id="A2.1.tab1.3.1.11.9.1" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">62</th>
<th id="A2.1.tab1.3.1.11.9.2" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">6</th>
<th id="A2.1.tab1.3.1.11.9.3" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">384</th>
<td id="A2.1.tab1.3.1.11.9.4" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">134.3</td>
<td id="A2.1.tab1.3.1.11.9.5" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">6.0</td>
<td id="A2.1.tab1.3.1.11.9.6" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">7.0</td>
<td id="A2.1.tab1.3.1.11.9.7" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">7.3</td>
<td id="A2.1.tab1.3.1.11.9.8" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">38.9</td>
<td id="A2.1.tab1.3.1.11.9.9" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">28.9</td>
</tr>
<tr id="A2.1.tab1.3.1.12.10" class="ltx_tr">
<th id="A2.1.tab1.3.1.12.10.1" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">
<span id="A2.1.tab1.3.1.12.10.1.1" class="ltx_ERROR undefined">\hdashline</span>[0.8pt/1pt]
5</th>
<th id="A2.1.tab1.3.1.12.10.2" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">32</th>
<th id="A2.1.tab1.3.1.12.10.3" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">2048</th>
<td id="A2.1.tab1.3.1.12.10.4" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">388.0</td>
<td id="A2.1.tab1.3.1.12.10.5" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">5.6</td>
<td id="A2.1.tab1.3.1.12.10.6" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">7.8</td>
<td id="A2.1.tab1.3.1.12.10.7" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">8.6</td>
<td id="A2.1.tab1.3.1.12.10.8" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">38.1</td>
<td id="A2.1.tab1.3.1.12.10.9" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">29.0</td>
</tr>
<tr id="A2.1.tab1.3.1.13.11" class="ltx_tr">
<th id="A2.1.tab1.3.1.13.11.1" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">10</th>
<th id="A2.1.tab1.3.1.13.11.2" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">24</th>
<th id="A2.1.tab1.3.1.13.11.3" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">1536</th>
<td id="A2.1.tab1.3.1.13.11.4" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">381.4</td>
<td id="A2.1.tab1.3.1.13.11.5" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">9.0</td>
<td id="A2.1.tab1.3.1.13.11.6" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">11.8</td>
<td id="A2.1.tab1.3.1.13.11.7" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">12.7</td>
<td id="A2.1.tab1.3.1.13.11.8" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">40.6</td>
<td id="A2.1.tab1.3.1.13.11.9" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">30.0</td>
</tr>
<tr id="A2.1.tab1.3.1.14.12" class="ltx_tr">
<th id="A2.1.tab1.3.1.14.12.1" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">12</th>
<th id="A2.1.tab1.3.1.14.12.2" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">22</th>
<th id="A2.1.tab1.3.1.14.12.3" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">1408</th>
<td id="A2.1.tab1.3.1.14.12.4" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">379.9</td>
<td id="A2.1.tab1.3.1.14.12.5" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">11.7</td>
<td id="A2.1.tab1.3.1.14.12.6" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">12.9</td>
<td id="A2.1.tab1.3.1.14.12.7" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">13.3</td>
<td id="A2.1.tab1.3.1.14.12.8" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">42.4</td>
<td id="A2.1.tab1.3.1.14.12.9" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">31.5</td>
</tr>
<tr id="A2.1.tab1.3.1.15.13" class="ltx_tr">
<th id="A2.1.tab1.3.1.15.13.1" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">15</th>
<th id="A2.1.tab1.3.1.15.13.2" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">20</th>
<th id="A2.1.tab1.3.1.15.13.3" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">1280</th>
<td id="A2.1.tab1.3.1.15.13.4" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">386.7</td>
<td id="A2.1.tab1.3.1.15.13.5" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">10.7</td>
<td id="A2.1.tab1.3.1.15.13.6" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">13.1</td>
<td id="A2.1.tab1.3.1.15.13.7" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">14.2</td>
<td id="A2.1.tab1.3.1.15.13.8" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">42.6</td>
<td id="A2.1.tab1.3.1.15.13.9" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">30.6</td>
</tr>
<tr id="A2.1.tab1.3.1.16.14" class="ltx_tr">
<th id="A2.1.tab1.3.1.16.14.1" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">19</th>
<th id="A2.1.tab1.3.1.16.14.2" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">18</th>
<th id="A2.1.tab1.3.1.16.14.3" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">1152</th>
<td id="A2.1.tab1.3.1.16.14.4" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">376.3</td>
<td id="A2.1.tab1.3.1.16.14.5" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">11.3</td>
<td id="A2.1.tab1.3.1.16.14.6" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">13.3</td>
<td id="A2.1.tab1.3.1.16.14.7" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">13.5</td>
<td id="A2.1.tab1.3.1.16.14.8" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">42.6</td>
<td id="A2.1.tab1.3.1.16.14.9" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">32.4</td>
</tr>
<tr id="A2.1.tab1.3.1.17.15" class="ltx_tr">
<th id="A2.1.tab1.3.1.17.15.1" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">24</th>
<th id="A2.1.tab1.3.1.17.15.2" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">16</th>
<th id="A2.1.tab1.3.1.17.15.3" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">1024</th>
<td id="A2.1.tab1.3.1.17.15.4" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">373.8</td>
<td id="A2.1.tab1.3.1.17.15.5" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">11.8</td>
<td id="A2.1.tab1.3.1.17.15.6" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">13.5</td>
<td id="A2.1.tab1.3.1.17.15.7" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">14.4</td>
<td id="A2.1.tab1.3.1.17.15.8" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">43.0</td>
<td id="A2.1.tab1.3.1.17.15.9" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">31.9</td>
</tr>
<tr id="A2.1.tab1.3.1.18.16" class="ltx_tr">
<th id="A2.1.tab1.3.1.18.16.1" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">28</th>
<th id="A2.1.tab1.3.1.18.16.2" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">15</th>
<th id="A2.1.tab1.3.1.18.16.3" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">960</th>
<td id="A2.1.tab1.3.1.18.16.4" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">371.1</td>
<td id="A2.1.tab1.3.1.18.16.5" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">12.5</td>
<td id="A2.1.tab1.3.1.18.16.6" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">13.6</td>
<td id="A2.1.tab1.3.1.18.16.7" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">14.9</td>
<td id="A2.1.tab1.3.1.18.16.8" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">42.8</td>
<td id="A2.1.tab1.3.1.18.16.9" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">32.3</td>
</tr>
<tr id="A2.1.tab1.3.1.19.17" class="ltx_tr">
<th id="A2.1.tab1.3.1.19.17.1" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">32</th>
<th id="A2.1.tab1.3.1.19.17.2" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">14</th>
<th id="A2.1.tab1.3.1.19.17.3" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">896</th>
<td id="A2.1.tab1.3.1.19.17.4" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">380.3</td>
<td id="A2.1.tab1.3.1.19.17.5" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">12.6</td>
<td id="A2.1.tab1.3.1.19.17.6" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">14.5</td>
<td id="A2.1.tab1.3.1.19.17.7" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">15.3</td>
<td id="A2.1.tab1.3.1.19.17.8" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">44.2</td>
<td id="A2.1.tab1.3.1.19.17.9" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">32.2</td>
</tr>
<tr id="A2.1.tab1.3.1.20.18" class="ltx_tr">
<th id="A2.1.tab1.3.1.20.18.1" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">46</th>
<th id="A2.1.tab1.3.1.20.18.2" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">12</th>
<th id="A2.1.tab1.3.1.20.18.3" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">768</th>
<td id="A2.1.tab1.3.1.20.18.4" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">374.7</td>
<td id="A2.1.tab1.3.1.20.18.5" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">13.0</td>
<td id="A2.1.tab1.3.1.20.18.6" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">15.4</td>
<td id="A2.1.tab1.3.1.20.18.7" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">15.6</td>
<td id="A2.1.tab1.3.1.20.18.8" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">44.7</td>
<td id="A2.1.tab1.3.1.20.18.9" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">31.4</td>
</tr>
<tr id="A2.1.tab1.3.1.21.19" class="ltx_tr">
<th id="A2.1.tab1.3.1.21.19.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_bb" style="padding:-2pt 2.8pt;">66</th>
<th id="A2.1.tab1.3.1.21.19.2" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_bb" style="padding:-2pt 2.8pt;">10</th>
<th id="A2.1.tab1.3.1.21.19.3" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_bb" style="padding:-2pt 2.8pt;">640</th>
<td id="A2.1.tab1.3.1.21.19.4" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 2.8pt;">376.2</td>
<td id="A2.1.tab1.3.1.21.19.5" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 2.8pt;">12.5</td>
<td id="A2.1.tab1.3.1.21.19.6" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 2.8pt;">15.1</td>
<td id="A2.1.tab1.3.1.21.19.7" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 2.8pt;">15.7</td>
<td id="A2.1.tab1.3.1.21.19.8" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 2.8pt;">44.0</td>
<td id="A2.1.tab1.3.1.21.19.9" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 2.8pt;">32.5</td>
</tr>
</tbody>
</table>
</span></div>
</div>
<div class="ltx_flex_cell ltx_flex_size_2">
<section id="A3" class="ltx_appendix ltx_figure_panel">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix C </span>Number of Heads and Key-Value Heads</h2>

<div id="A3.p1" class="ltx_para">
<p id="A3.p1.1" class="ltx_p">We provide detailed experimental results assessing the impact of the number of attention heads and key-value heads on zero-shot reasoning accuracy in Table <a href="#A3" title="Appendix C Number of Heads and Key-Value Heads ‣ Appendix B Depth vs Width ‣ Appendix A Impact of Each Design Choice ‣ Broader Impact ‣ Acknowledgment ‣ 5 Conclusion ‣ 4 Related Work ‣ 3.6 On-device Profiling ‣ 3.5 Knowledge Distillation ‣ 3.4 Compatibility with Quantization ‣ 3.3.2 API Calling ‣ 3.3.1 Chat ‣ 3.3 Downstream Tasks ‣ 3.2 Main Results ‣ 3 Experiments ‣ 2.3 Layer Sharing ‣ 2 Improving Sub-billion Scale LLM Design ‣ MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases" class="ltx_ref"><span class="ltx_text ltx_ref_tag">C</span></a>. Our study involves two baseline architectures: a 8-layer 125M model with a embedding dimension of 896, and a 15-layer 350M model with a embedding dimension of 1280. We conduct head size sweeps in {8, 16, 32}. The findings shown in Table <a href="#A3" title="Appendix C Number of Heads and Key-Value Heads ‣ Appendix B Depth vs Width ‣ Appendix A Impact of Each Design Choice ‣ Broader Impact ‣ Acknowledgment ‣ 5 Conclusion ‣ 4 Related Work ‣ 3.6 On-device Profiling ‣ 3.5 Knowledge Distillation ‣ 3.4 Compatibility with Quantization ‣ 3.3.2 API Calling ‣ 3.3.1 Chat ‣ 3.3 Downstream Tasks ‣ 3.2 Main Results ‣ 3 Experiments ‣ 2.3 Layer Sharing ‣ 2 Improving Sub-billion Scale LLM Design ‣ MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases" class="ltx_ref"><span class="ltx_text ltx_ref_tag">C</span></a> indicate that using 16 heads, with a head dimension close to 64, and 4 key-value heads, yields the best accuracy and memory trade-off. This setting serves as a guiding principle in our model architecture design.</p>
</div>
<figure id="A3.tab1" class="ltx_table">

<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table"><span id="A3.tab1.1.1.1" class="ltx_text" style="font-size:90%;">Table 11</span>: </span><span id="A3.tab1.2.2" class="ltx_text" style="font-size:90%;">Ablation study investigating the impact of the number of attention heads and key-value heads.</span></figcaption><div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_1">
<div id="A3.tab1.3" class="ltx_inline-block ltx_figure_panel ltx_transformed_outer" style="width:429.3pt;height:423.2pt;vertical-align:-0.8pt;"><span class="ltx_transformed_inner" style="transform:translate(-68.9pt,67.8pt) scale(0.757055316516644,0.757055316516644) ;">
<table id="A3.tab1.3.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A3.tab1.3.1.1.1" class="ltx_tr">
<th id="A3.tab1.3.1.1.1.1" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_th_row ltx_border_tt" style="padding:-2pt 2.8pt;"><span id="A3.tab1.3.1.1.1.1.1" class="ltx_text ltx_font_bold">Model</span></th>
<th id="A3.tab1.3.1.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_th_row ltx_border_tt" style="padding:-2pt 2.8pt;"><span id="A3.tab1.3.1.1.1.2.1" class="ltx_text ltx_font_bold">#Heads</span></th>
<th id="A3.tab1.3.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_th_row ltx_border_tt" style="padding:-2pt 2.8pt;"><span id="A3.tab1.3.1.1.1.3.1" class="ltx_text ltx_font_bold">#KV-Heads</span></th>
<th id="A3.tab1.3.1.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 2.8pt;"><span id="A3.tab1.3.1.1.1.4.1" class="ltx_text ltx_font_bold">#Params(M)</span></th>
<th id="A3.tab1.3.1.1.1.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 2.8pt;"><span id="A3.tab1.3.1.1.1.5.1" class="ltx_text ltx_font_bold">ARC-e</span></th>
<th id="A3.tab1.3.1.1.1.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 2.8pt;"><span id="A3.tab1.3.1.1.1.6.1" class="ltx_text ltx_font_bold">ARC-c</span></th>
<th id="A3.tab1.3.1.1.1.7" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 2.8pt;"><span id="A3.tab1.3.1.1.1.7.1" class="ltx_text ltx_font_bold">BoolQ</span></th>
<th id="A3.tab1.3.1.1.1.8" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 2.8pt;"><span id="A3.tab1.3.1.1.1.8.1" class="ltx_text ltx_font_bold">PIQA</span></th>
<th id="A3.tab1.3.1.1.1.9" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 2.8pt;"><span id="A3.tab1.3.1.1.1.9.1" class="ltx_text ltx_font_bold">SIQA</span></th>
<th id="A3.tab1.3.1.1.1.10" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 2.8pt;"><span id="A3.tab1.3.1.1.1.10.1" class="ltx_text ltx_font_bold">HellaSwag</span></th>
<th id="A3.tab1.3.1.1.1.11" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 2.8pt;"><span id="A3.tab1.3.1.1.1.11.1" class="ltx_text ltx_font_bold">OBQA</span></th>
<th id="A3.tab1.3.1.1.1.12" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 2.8pt;"><span id="A3.tab1.3.1.1.1.12.1" class="ltx_text ltx_font_bold">WinoGrande</span></th>
<th id="A3.tab1.3.1.1.1.13" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 2.8pt;"><span id="A3.tab1.3.1.1.1.13.1" class="ltx_text ltx_font_bold">Avg.</span></th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A3.tab1.3.1.2.1" class="ltx_tr">
<th id="A3.tab1.3.1.2.1.1" class="ltx_td ltx_th ltx_th_row ltx_border_t" style="padding:-2pt 2.8pt;"></th>
<th id="A3.tab1.3.1.2.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_t" style="padding:-2pt 2.8pt;">32</th>
<th id="A3.tab1.3.1.2.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_t" style="padding:-2pt 2.8pt;">32</th>
<td id="A3.tab1.3.1.2.1.4" class="ltx_td ltx_align_center ltx_border_t" style="padding:-2pt 2.8pt;">138.1</td>
<td id="A3.tab1.3.1.2.1.5" class="ltx_td ltx_align_center ltx_border_t" style="padding:-2pt 2.8pt;">42.1</td>
<td id="A3.tab1.3.1.2.1.6" class="ltx_td ltx_align_center ltx_border_t" style="padding:-2pt 2.8pt;">26.9</td>
<td id="A3.tab1.3.1.2.1.7" class="ltx_td ltx_align_center ltx_border_t" style="padding:-2pt 2.8pt;">58.4</td>
<td id="A3.tab1.3.1.2.1.8" class="ltx_td ltx_align_center ltx_border_t" style="padding:-2pt 2.8pt;">62.2</td>
<td id="A3.tab1.3.1.2.1.9" class="ltx_td ltx_align_center ltx_border_t" style="padding:-2pt 2.8pt;">42.1</td>
<td id="A3.tab1.3.1.2.1.10" class="ltx_td ltx_align_center ltx_border_t" style="padding:-2pt 2.8pt;">33.8</td>
<td id="A3.tab1.3.1.2.1.11" class="ltx_td ltx_align_center ltx_border_t" style="padding:-2pt 2.8pt;">36.7</td>
<td id="A3.tab1.3.1.2.1.12" class="ltx_td ltx_align_center ltx_border_t" style="padding:-2pt 2.8pt;">52.4</td>
<td id="A3.tab1.3.1.2.1.13" class="ltx_td ltx_align_center ltx_border_t" style="padding:-2pt 2.8pt;">44.3</td>
</tr>
<tr id="A3.tab1.3.1.3.2" class="ltx_tr">
<th id="A3.tab1.3.1.3.2.1" class="ltx_td ltx_th ltx_th_row" style="padding:-2pt 2.8pt;"></th>
<th id="A3.tab1.3.1.3.2.2" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">32</th>
<th id="A3.tab1.3.1.3.2.3" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">16</th>
<td id="A3.tab1.3.1.3.2.4" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">131.7</td>
<td id="A3.tab1.3.1.3.2.5" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">41.2</td>
<td id="A3.tab1.3.1.3.2.6" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">26.4</td>
<td id="A3.tab1.3.1.3.2.7" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">57.7</td>
<td id="A3.tab1.3.1.3.2.8" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">62.5</td>
<td id="A3.tab1.3.1.3.2.9" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">42.3</td>
<td id="A3.tab1.3.1.3.2.10" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">33.3</td>
<td id="A3.tab1.3.1.3.2.11" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">34.2</td>
<td id="A3.tab1.3.1.3.2.12" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">52.9</td>
<td id="A3.tab1.3.1.3.2.13" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">43.8</td>
</tr>
<tr id="A3.tab1.3.1.4.3" class="ltx_tr">
<th id="A3.tab1.3.1.4.3.1" class="ltx_td ltx_th ltx_th_row" style="padding:-2pt 2.8pt;"></th>
<th id="A3.tab1.3.1.4.3.2" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">32</th>
<th id="A3.tab1.3.1.4.3.3" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">8</th>
<td id="A3.tab1.3.1.4.3.4" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">128.5</td>
<td id="A3.tab1.3.1.4.3.5" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">42.6</td>
<td id="A3.tab1.3.1.4.3.6" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">27.3</td>
<td id="A3.tab1.3.1.4.3.7" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">61.1</td>
<td id="A3.tab1.3.1.4.3.8" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">61.9</td>
<td id="A3.tab1.3.1.4.3.9" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">41.9</td>
<td id="A3.tab1.3.1.4.3.10" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">32.2</td>
<td id="A3.tab1.3.1.4.3.11" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">35.0</td>
<td id="A3.tab1.3.1.4.3.12" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">52.0</td>
<td id="A3.tab1.3.1.4.3.13" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">44.2</td>
</tr>
<tr id="A3.tab1.3.1.5.4" class="ltx_tr">
<th id="A3.tab1.3.1.5.4.1" class="ltx_td ltx_th ltx_th_row" style="padding:-2pt 2.8pt;"></th>
<th id="A3.tab1.3.1.5.4.2" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">32</th>
<th id="A3.tab1.3.1.5.4.3" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">4</th>
<td id="A3.tab1.3.1.5.4.4" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">126.8</td>
<td id="A3.tab1.3.1.5.4.5" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">43.1</td>
<td id="A3.tab1.3.1.5.4.6" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">26.8</td>
<td id="A3.tab1.3.1.5.4.7" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">59.8</td>
<td id="A3.tab1.3.1.5.4.8" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">62.7</td>
<td id="A3.tab1.3.1.5.4.9" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">41.4</td>
<td id="A3.tab1.3.1.5.4.10" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">32.5</td>
<td id="A3.tab1.3.1.5.4.11" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">34.4</td>
<td id="A3.tab1.3.1.5.4.12" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">51.1</td>
<td id="A3.tab1.3.1.5.4.13" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">44.0</td>
</tr>
<tr id="A3.tab1.3.1.6.5" class="ltx_tr">
<th id="A3.tab1.3.1.6.5.1" class="ltx_td ltx_th ltx_th_row" style="padding:-2pt 2.8pt;"></th>
<th id="A3.tab1.3.1.6.5.2" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">32</th>
<th id="A3.tab1.3.1.6.5.3" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">2</th>
<td id="A3.tab1.3.1.6.5.4" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">126.0</td>
<td id="A3.tab1.3.1.6.5.5" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">39.8</td>
<td id="A3.tab1.3.1.6.5.6" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">26.7</td>
<td id="A3.tab1.3.1.6.5.7" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">59.4</td>
<td id="A3.tab1.3.1.6.5.8" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">59.4</td>
<td id="A3.tab1.3.1.6.5.9" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">42.0</td>
<td id="A3.tab1.3.1.6.5.10" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">31.3</td>
<td id="A3.tab1.3.1.6.5.11" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">32.6</td>
<td id="A3.tab1.3.1.6.5.12" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">52.9</td>
<td id="A3.tab1.3.1.6.5.13" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">43.0</td>
</tr>
<tr id="A3.tab1.3.1.7.6" class="ltx_tr">
<th id="A3.tab1.3.1.7.6.1" class="ltx_td ltx_th ltx_th_row" style="padding:-2pt 2.8pt;"></th>
<th id="A3.tab1.3.1.7.6.2" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">32</th>
<th id="A3.tab1.3.1.7.6.3" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">1</th>
<td id="A3.tab1.3.1.7.6.4" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">125.6</td>
<td id="A3.tab1.3.1.7.6.5" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">41.0</td>
<td id="A3.tab1.3.1.7.6.6" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">24.3</td>
<td id="A3.tab1.3.1.7.6.7" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">59.1</td>
<td id="A3.tab1.3.1.7.6.8" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">60.8</td>
<td id="A3.tab1.3.1.7.6.9" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">41.2</td>
<td id="A3.tab1.3.1.7.6.10" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">31.4</td>
<td id="A3.tab1.3.1.7.6.11" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">35.4</td>
<td id="A3.tab1.3.1.7.6.12" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">52.0</td>
<td id="A3.tab1.3.1.7.6.13" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">43.1</td>
</tr>
<tr id="A3.tab1.3.1.8.7" class="ltx_tr">
<th id="A3.tab1.3.1.8.7.1" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;"><span id="A3.tab1.3.1.8.7.1.1" class="ltx_text ltx_font_bold">125M</span></th>
<th id="A3.tab1.3.1.8.7.2" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">16</th>
<th id="A3.tab1.3.1.8.7.3" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">16</th>
<td id="A3.tab1.3.1.8.7.4" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">138.1</td>
<td id="A3.tab1.3.1.8.7.5" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">41.6</td>
<td id="A3.tab1.3.1.8.7.6" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">25.7</td>
<td id="A3.tab1.3.1.8.7.7" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">61.1</td>
<td id="A3.tab1.3.1.8.7.8" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">62.4</td>
<td id="A3.tab1.3.1.8.7.9" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">43.1</td>
<td id="A3.tab1.3.1.8.7.10" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">34.4</td>
<td id="A3.tab1.3.1.8.7.11" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">36.9</td>
<td id="A3.tab1.3.1.8.7.12" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">51.6</td>
<td id="A3.tab1.3.1.8.7.13" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;"><span id="A3.tab1.3.1.8.7.13.1" class="ltx_text ltx_font_bold">44.6</span></td>
</tr>
<tr id="A3.tab1.3.1.9.8" class="ltx_tr">
<th id="A3.tab1.3.1.9.8.1" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;"># layers=8</th>
<th id="A3.tab1.3.1.9.8.2" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">16</th>
<th id="A3.tab1.3.1.9.8.3" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">8</th>
<td id="A3.tab1.3.1.9.8.4" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">131.7</td>
<td id="A3.tab1.3.1.9.8.5" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">42.4</td>
<td id="A3.tab1.3.1.9.8.6" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">26.4</td>
<td id="A3.tab1.3.1.9.8.7" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">60.7</td>
<td id="A3.tab1.3.1.9.8.8" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">63.4</td>
<td id="A3.tab1.3.1.9.8.9" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">41.9</td>
<td id="A3.tab1.3.1.9.8.10" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">33.5</td>
<td id="A3.tab1.3.1.9.8.11" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">34.7</td>
<td id="A3.tab1.3.1.9.8.12" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">51.5</td>
<td id="A3.tab1.3.1.9.8.13" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">44.3</td>
</tr>
<tr id="A3.tab1.3.1.10.9" class="ltx_tr">
<th id="A3.tab1.3.1.10.9.1" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">dim=896</th>
<th id="A3.tab1.3.1.10.9.2" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">16</th>
<th id="A3.tab1.3.1.10.9.3" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">4</th>
<td id="A3.tab1.3.1.10.9.4" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">128.5</td>
<td id="A3.tab1.3.1.10.9.5" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">42.5</td>
<td id="A3.tab1.3.1.10.9.6" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">25.6</td>
<td id="A3.tab1.3.1.10.9.7" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">62.3</td>
<td id="A3.tab1.3.1.10.9.8" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">62.4</td>
<td id="A3.tab1.3.1.10.9.9" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">41.8</td>
<td id="A3.tab1.3.1.10.9.10" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">33.0</td>
<td id="A3.tab1.3.1.10.9.11" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">35.9</td>
<td id="A3.tab1.3.1.10.9.12" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">54.5</td>
<td id="A3.tab1.3.1.10.9.13" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;"><span id="A3.tab1.3.1.10.9.13.1" class="ltx_text ltx_font_bold">44.7</span></td>
</tr>
<tr id="A3.tab1.3.1.11.10" class="ltx_tr">
<th id="A3.tab1.3.1.11.10.1" class="ltx_td ltx_th ltx_th_row" style="padding:-2pt 2.8pt;"></th>
<th id="A3.tab1.3.1.11.10.2" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">16</th>
<th id="A3.tab1.3.1.11.10.3" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">2</th>
<td id="A3.tab1.3.1.11.10.4" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">126.8</td>
<td id="A3.tab1.3.1.11.10.5" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">41.7</td>
<td id="A3.tab1.3.1.11.10.6" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">25.3</td>
<td id="A3.tab1.3.1.11.10.7" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">56.9</td>
<td id="A3.tab1.3.1.11.10.8" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">61.7</td>
<td id="A3.tab1.3.1.11.10.9" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">42.0</td>
<td id="A3.tab1.3.1.11.10.10" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">32.9</td>
<td id="A3.tab1.3.1.11.10.11" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">32.6</td>
<td id="A3.tab1.3.1.11.10.12" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">54.5</td>
<td id="A3.tab1.3.1.11.10.13" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">43.5</td>
</tr>
<tr id="A3.tab1.3.1.12.11" class="ltx_tr">
<th id="A3.tab1.3.1.12.11.1" class="ltx_td ltx_th ltx_th_row" style="padding:-2pt 2.8pt;"></th>
<th id="A3.tab1.3.1.12.11.2" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">16</th>
<th id="A3.tab1.3.1.12.11.3" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">1</th>
<td id="A3.tab1.3.1.12.11.4" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">126.0</td>
<td id="A3.tab1.3.1.12.11.5" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">40.4</td>
<td id="A3.tab1.3.1.12.11.6" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">26.3</td>
<td id="A3.tab1.3.1.12.11.7" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">61.8</td>
<td id="A3.tab1.3.1.12.11.8" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">63.2</td>
<td id="A3.tab1.3.1.12.11.9" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">41.7</td>
<td id="A3.tab1.3.1.12.11.10" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">32.0</td>
<td id="A3.tab1.3.1.12.11.11" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">34.0</td>
<td id="A3.tab1.3.1.12.11.12" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">50.4</td>
<td id="A3.tab1.3.1.12.11.13" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">43.7</td>
</tr>
<tr id="A3.tab1.3.1.13.12" class="ltx_tr">
<th id="A3.tab1.3.1.13.12.1" class="ltx_td ltx_th ltx_th_row" style="padding:-2pt 2.8pt;"></th>
<th id="A3.tab1.3.1.13.12.2" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">8</th>
<th id="A3.tab1.3.1.13.12.3" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">8</th>
<td id="A3.tab1.3.1.13.12.4" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">138.1</td>
<td id="A3.tab1.3.1.13.12.5" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">41.4</td>
<td id="A3.tab1.3.1.13.12.6" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">25.0</td>
<td id="A3.tab1.3.1.13.12.7" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">58.3</td>
<td id="A3.tab1.3.1.13.12.8" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">61.7</td>
<td id="A3.tab1.3.1.13.12.9" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">41.7</td>
<td id="A3.tab1.3.1.13.12.10" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">33.3</td>
<td id="A3.tab1.3.1.13.12.11" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">35.9</td>
<td id="A3.tab1.3.1.13.12.12" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">53.2</td>
<td id="A3.tab1.3.1.13.12.13" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">43.8</td>
</tr>
<tr id="A3.tab1.3.1.14.13" class="ltx_tr">
<th id="A3.tab1.3.1.14.13.1" class="ltx_td ltx_th ltx_th_row" style="padding:-2pt 2.8pt;"></th>
<th id="A3.tab1.3.1.14.13.2" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">8</th>
<th id="A3.tab1.3.1.14.13.3" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">4</th>
<td id="A3.tab1.3.1.14.13.4" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">131.7</td>
<td id="A3.tab1.3.1.14.13.5" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">43.3</td>
<td id="A3.tab1.3.1.14.13.6" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">28.2</td>
<td id="A3.tab1.3.1.14.13.7" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">58.3</td>
<td id="A3.tab1.3.1.14.13.8" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">61.8</td>
<td id="A3.tab1.3.1.14.13.9" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">42.8</td>
<td id="A3.tab1.3.1.14.13.10" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">33.8</td>
<td id="A3.tab1.3.1.14.13.11" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">30.9</td>
<td id="A3.tab1.3.1.14.13.12" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">53.0</td>
<td id="A3.tab1.3.1.14.13.13" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">44.0</td>
</tr>
<tr id="A3.tab1.3.1.15.14" class="ltx_tr">
<th id="A3.tab1.3.1.15.14.1" class="ltx_td ltx_th ltx_th_row" style="padding:-2pt 2.8pt;"></th>
<th id="A3.tab1.3.1.15.14.2" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">8</th>
<th id="A3.tab1.3.1.15.14.3" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">2</th>
<td id="A3.tab1.3.1.15.14.4" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">128.5</td>
<td id="A3.tab1.3.1.15.14.5" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">40.7</td>
<td id="A3.tab1.3.1.15.14.6" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">26.2</td>
<td id="A3.tab1.3.1.15.14.7" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">58.1</td>
<td id="A3.tab1.3.1.15.14.8" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">61.2</td>
<td id="A3.tab1.3.1.15.14.9" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">41.6</td>
<td id="A3.tab1.3.1.15.14.10" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">32.8</td>
<td id="A3.tab1.3.1.15.14.11" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">34.8</td>
<td id="A3.tab1.3.1.15.14.12" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">51.5</td>
<td id="A3.tab1.3.1.15.14.13" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">43.4</td>
</tr>
<tr id="A3.tab1.3.1.16.15" class="ltx_tr">
<th id="A3.tab1.3.1.16.15.1" class="ltx_td ltx_th ltx_th_row" style="padding:-2pt 2.8pt;"></th>
<th id="A3.tab1.3.1.16.15.2" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">8</th>
<th id="A3.tab1.3.1.16.15.3" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">1</th>
<td id="A3.tab1.3.1.16.15.4" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">126.8</td>
<td id="A3.tab1.3.1.16.15.5" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">42.5</td>
<td id="A3.tab1.3.1.16.15.6" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">24.8</td>
<td id="A3.tab1.3.1.16.15.7" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">59.4</td>
<td id="A3.tab1.3.1.16.15.8" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">62.3</td>
<td id="A3.tab1.3.1.16.15.9" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">42.0</td>
<td id="A3.tab1.3.1.16.15.10" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">32.0</td>
<td id="A3.tab1.3.1.16.15.11" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">36.3</td>
<td id="A3.tab1.3.1.16.15.12" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">51.3</td>
<td id="A3.tab1.3.1.16.15.13" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">43.8</td>
</tr>
<tr id="A3.tab1.3.1.17.16" class="ltx_tr">
<th id="A3.tab1.3.1.17.16.1" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">
<span id="A3.tab1.3.1.17.16.1.1" class="ltx_ERROR undefined">\hdashline</span>[0.8pt/1pt]</th>
<th id="A3.tab1.3.1.17.16.2" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">32</th>
<th id="A3.tab1.3.1.17.16.3" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">32</th>
<td id="A3.tab1.3.1.17.16.4" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">386.7</td>
<td id="A3.tab1.3.1.17.16.5" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">48.6</td>
<td id="A3.tab1.3.1.17.16.6" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">30.4</td>
<td id="A3.tab1.3.1.17.16.7" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">59.7</td>
<td id="A3.tab1.3.1.17.16.8" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">67.2</td>
<td id="A3.tab1.3.1.17.16.9" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">43.9</td>
<td id="A3.tab1.3.1.17.16.10" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">44.0</td>
<td id="A3.tab1.3.1.17.16.11" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">40.9</td>
<td id="A3.tab1.3.1.17.16.12" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">53.9</td>
<td id="A3.tab1.3.1.17.16.13" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">48.6</td>
</tr>
<tr id="A3.tab1.3.1.18.17" class="ltx_tr">
<th id="A3.tab1.3.1.18.17.1" class="ltx_td ltx_th ltx_th_row" style="padding:-2pt 2.8pt;"></th>
<th id="A3.tab1.3.1.18.17.2" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">32</th>
<th id="A3.tab1.3.1.18.17.3" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">16</th>
<td id="A3.tab1.3.1.18.17.4" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">362.1</td>
<td id="A3.tab1.3.1.18.17.5" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">48.9</td>
<td id="A3.tab1.3.1.18.17.6" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">31.6</td>
<td id="A3.tab1.3.1.18.17.7" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">57.6</td>
<td id="A3.tab1.3.1.18.17.8" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">68.4</td>
<td id="A3.tab1.3.1.18.17.9" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">43.4</td>
<td id="A3.tab1.3.1.18.17.10" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">43.8</td>
<td id="A3.tab1.3.1.18.17.11" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">38.6</td>
<td id="A3.tab1.3.1.18.17.12" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">54.9</td>
<td id="A3.tab1.3.1.18.17.13" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">48.4</td>
</tr>
<tr id="A3.tab1.3.1.19.18" class="ltx_tr">
<th id="A3.tab1.3.1.19.18.1" class="ltx_td ltx_th ltx_th_row" style="padding:-2pt 2.8pt;"></th>
<th id="A3.tab1.3.1.19.18.2" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">32</th>
<th id="A3.tab1.3.1.19.18.3" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">8</th>
<td id="A3.tab1.3.1.19.18.4" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">349.8</td>
<td id="A3.tab1.3.1.19.18.5" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">48.3</td>
<td id="A3.tab1.3.1.19.18.6" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">33.1</td>
<td id="A3.tab1.3.1.19.18.7" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">61.0</td>
<td id="A3.tab1.3.1.19.18.8" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">67.2</td>
<td id="A3.tab1.3.1.19.18.9" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">42.6</td>
<td id="A3.tab1.3.1.19.18.10" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">42.1</td>
<td id="A3.tab1.3.1.19.18.11" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">39.0</td>
<td id="A3.tab1.3.1.19.18.12" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">53.9</td>
<td id="A3.tab1.3.1.19.18.13" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">48.4</td>
</tr>
<tr id="A3.tab1.3.1.20.19" class="ltx_tr">
<th id="A3.tab1.3.1.20.19.1" class="ltx_td ltx_th ltx_th_row" style="padding:-2pt 2.8pt;"></th>
<th id="A3.tab1.3.1.20.19.2" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">32</th>
<th id="A3.tab1.3.1.20.19.3" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">4</th>
<td id="A3.tab1.3.1.20.19.4" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">343.7</td>
<td id="A3.tab1.3.1.20.19.5" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">47.2</td>
<td id="A3.tab1.3.1.20.19.6" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">29.8</td>
<td id="A3.tab1.3.1.20.19.7" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">59.4</td>
<td id="A3.tab1.3.1.20.19.8" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">67.2</td>
<td id="A3.tab1.3.1.20.19.9" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">43.5</td>
<td id="A3.tab1.3.1.20.19.10" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">42.5</td>
<td id="A3.tab1.3.1.20.19.11" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">42.5</td>
<td id="A3.tab1.3.1.20.19.12" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">54.1</td>
<td id="A3.tab1.3.1.20.19.13" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">48.3</td>
</tr>
<tr id="A3.tab1.3.1.21.20" class="ltx_tr">
<th id="A3.tab1.3.1.21.20.1" class="ltx_td ltx_th ltx_th_row" style="padding:-2pt 2.8pt;"></th>
<th id="A3.tab1.3.1.21.20.2" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">32</th>
<th id="A3.tab1.3.1.21.20.3" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">2</th>
<td id="A3.tab1.3.1.21.20.4" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">340.6</td>
<td id="A3.tab1.3.1.21.20.5" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">47.6</td>
<td id="A3.tab1.3.1.21.20.6" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">30.3</td>
<td id="A3.tab1.3.1.21.20.7" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">62.4</td>
<td id="A3.tab1.3.1.21.20.8" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">66.9</td>
<td id="A3.tab1.3.1.21.20.9" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">42.6</td>
<td id="A3.tab1.3.1.21.20.10" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">41.6</td>
<td id="A3.tab1.3.1.21.20.11" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">38.6</td>
<td id="A3.tab1.3.1.21.20.12" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">52.0</td>
<td id="A3.tab1.3.1.21.20.13" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">47.7</td>
</tr>
<tr id="A3.tab1.3.1.22.21" class="ltx_tr">
<th id="A3.tab1.3.1.22.21.1" class="ltx_td ltx_th ltx_th_row" style="padding:-2pt 2.8pt;"></th>
<th id="A3.tab1.3.1.22.21.2" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">32</th>
<th id="A3.tab1.3.1.22.21.3" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">1</th>
<td id="A3.tab1.3.1.22.21.4" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">339.0</td>
<td id="A3.tab1.3.1.22.21.5" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">48.5</td>
<td id="A3.tab1.3.1.22.21.6" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">27.3</td>
<td id="A3.tab1.3.1.22.21.7" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">56.3</td>
<td id="A3.tab1.3.1.22.21.8" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">67.1</td>
<td id="A3.tab1.3.1.22.21.9" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">42.9</td>
<td id="A3.tab1.3.1.22.21.10" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">40.9</td>
<td id="A3.tab1.3.1.22.21.11" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">36.7</td>
<td id="A3.tab1.3.1.22.21.12" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">53.3</td>
<td id="A3.tab1.3.1.22.21.13" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">46.6</td>
</tr>
<tr id="A3.tab1.3.1.23.22" class="ltx_tr">
<th id="A3.tab1.3.1.23.22.1" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;"><span id="A3.tab1.3.1.23.22.1.1" class="ltx_text ltx_font_bold">350M</span></th>
<th id="A3.tab1.3.1.23.22.2" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">16</th>
<th id="A3.tab1.3.1.23.22.3" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">16</th>
<td id="A3.tab1.3.1.23.22.4" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">386.7</td>
<td id="A3.tab1.3.1.23.22.5" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">50.8</td>
<td id="A3.tab1.3.1.23.22.6" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">30.6</td>
<td id="A3.tab1.3.1.23.22.7" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">62.3</td>
<td id="A3.tab1.3.1.23.22.8" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">68.6</td>
<td id="A3.tab1.3.1.23.22.9" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">43.5</td>
<td id="A3.tab1.3.1.23.22.10" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">45.1</td>
<td id="A3.tab1.3.1.23.22.11" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">43.8</td>
<td id="A3.tab1.3.1.23.22.12" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">52.4</td>
<td id="A3.tab1.3.1.23.22.13" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;"><span id="A3.tab1.3.1.23.22.13.1" class="ltx_text ltx_font_bold">49.6</span></td>
</tr>
<tr id="A3.tab1.3.1.24.23" class="ltx_tr">
<th id="A3.tab1.3.1.24.23.1" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;"># layers=15</th>
<th id="A3.tab1.3.1.24.23.2" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">16</th>
<th id="A3.tab1.3.1.24.23.3" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">8</th>
<td id="A3.tab1.3.1.24.23.4" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">362.1</td>
<td id="A3.tab1.3.1.24.23.5" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">48.5</td>
<td id="A3.tab1.3.1.24.23.6" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">30.7</td>
<td id="A3.tab1.3.1.24.23.7" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">59.4</td>
<td id="A3.tab1.3.1.24.23.8" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">67.3</td>
<td id="A3.tab1.3.1.24.23.9" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">43.8</td>
<td id="A3.tab1.3.1.24.23.10" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">43.8</td>
<td id="A3.tab1.3.1.24.23.11" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">41.3</td>
<td id="A3.tab1.3.1.24.23.12" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">53.3</td>
<td id="A3.tab1.3.1.24.23.13" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">48.5</td>
</tr>
<tr id="A3.tab1.3.1.25.24" class="ltx_tr">
<th id="A3.tab1.3.1.25.24.1" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">dim=1280</th>
<th id="A3.tab1.3.1.25.24.2" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">16</th>
<th id="A3.tab1.3.1.25.24.3" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">4</th>
<td id="A3.tab1.3.1.25.24.4" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">349.8</td>
<td id="A3.tab1.3.1.25.24.5" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">49.9</td>
<td id="A3.tab1.3.1.25.24.6" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">30.6</td>
<td id="A3.tab1.3.1.25.24.7" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">60.0</td>
<td id="A3.tab1.3.1.25.24.8" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">69.2</td>
<td id="A3.tab1.3.1.25.24.9" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">43.5</td>
<td id="A3.tab1.3.1.25.24.10" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">44.2</td>
<td id="A3.tab1.3.1.25.24.11" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">41.8</td>
<td id="A3.tab1.3.1.25.24.12" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">55.8</td>
<td id="A3.tab1.3.1.25.24.13" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;"><span id="A3.tab1.3.1.25.24.13.1" class="ltx_text ltx_font_bold">49.4</span></td>
</tr>
<tr id="A3.tab1.3.1.26.25" class="ltx_tr">
<th id="A3.tab1.3.1.26.25.1" class="ltx_td ltx_th ltx_th_row" style="padding:-2pt 2.8pt;"></th>
<th id="A3.tab1.3.1.26.25.2" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">16</th>
<th id="A3.tab1.3.1.26.25.3" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">2</th>
<td id="A3.tab1.3.1.26.25.4" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">343.7</td>
<td id="A3.tab1.3.1.26.25.5" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">49.3</td>
<td id="A3.tab1.3.1.26.25.6" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">28.4</td>
<td id="A3.tab1.3.1.26.25.7" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">55.0</td>
<td id="A3.tab1.3.1.26.25.8" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">67.3</td>
<td id="A3.tab1.3.1.26.25.9" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">42.7</td>
<td id="A3.tab1.3.1.26.25.10" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">42.6</td>
<td id="A3.tab1.3.1.26.25.11" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">40.3</td>
<td id="A3.tab1.3.1.26.25.12" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">54.5</td>
<td id="A3.tab1.3.1.26.25.13" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">47.5</td>
</tr>
<tr id="A3.tab1.3.1.27.26" class="ltx_tr">
<th id="A3.tab1.3.1.27.26.1" class="ltx_td ltx_th ltx_th_row" style="padding:-2pt 2.8pt;"></th>
<th id="A3.tab1.3.1.27.26.2" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">16</th>
<th id="A3.tab1.3.1.27.26.3" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">1</th>
<td id="A3.tab1.3.1.27.26.4" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">340.6</td>
<td id="A3.tab1.3.1.27.26.5" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">49.2</td>
<td id="A3.tab1.3.1.27.26.6" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">29.3</td>
<td id="A3.tab1.3.1.27.26.7" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">58.8</td>
<td id="A3.tab1.3.1.27.26.8" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">67.4</td>
<td id="A3.tab1.3.1.27.26.9" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">43.5</td>
<td id="A3.tab1.3.1.27.26.10" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">42.1</td>
<td id="A3.tab1.3.1.27.26.11" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">39.9</td>
<td id="A3.tab1.3.1.27.26.12" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">52.8</td>
<td id="A3.tab1.3.1.27.26.13" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">47.9</td>
</tr>
<tr id="A3.tab1.3.1.28.27" class="ltx_tr">
<th id="A3.tab1.3.1.28.27.1" class="ltx_td ltx_th ltx_th_row" style="padding:-2pt 2.8pt;"></th>
<th id="A3.tab1.3.1.28.27.2" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">8</th>
<th id="A3.tab1.3.1.28.27.3" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">8</th>
<td id="A3.tab1.3.1.28.27.4" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">386.7</td>
<td id="A3.tab1.3.1.28.27.5" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">51.0</td>
<td id="A3.tab1.3.1.28.27.6" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">33.2</td>
<td id="A3.tab1.3.1.28.27.7" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">58.2</td>
<td id="A3.tab1.3.1.28.27.8" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">67.0</td>
<td id="A3.tab1.3.1.28.27.9" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">43.9</td>
<td id="A3.tab1.3.1.28.27.10" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">43.9</td>
<td id="A3.tab1.3.1.28.27.11" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">42.6</td>
<td id="A3.tab1.3.1.28.27.12" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">54.7</td>
<td id="A3.tab1.3.1.28.27.13" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">49.3</td>
</tr>
<tr id="A3.tab1.3.1.29.28" class="ltx_tr">
<th id="A3.tab1.3.1.29.28.1" class="ltx_td ltx_th ltx_th_row" style="padding:-2pt 2.8pt;"></th>
<th id="A3.tab1.3.1.29.28.2" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">8</th>
<th id="A3.tab1.3.1.29.28.3" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">4</th>
<td id="A3.tab1.3.1.29.28.4" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">362.1</td>
<td id="A3.tab1.3.1.29.28.5" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">50.0</td>
<td id="A3.tab1.3.1.29.28.6" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">31.3</td>
<td id="A3.tab1.3.1.29.28.7" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">60.2</td>
<td id="A3.tab1.3.1.29.28.8" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">66.0</td>
<td id="A3.tab1.3.1.29.28.9" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">42.7</td>
<td id="A3.tab1.3.1.29.28.10" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">43.7</td>
<td id="A3.tab1.3.1.29.28.11" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">40.9</td>
<td id="A3.tab1.3.1.29.28.12" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">53.7</td>
<td id="A3.tab1.3.1.29.28.13" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">48.6</td>
</tr>
<tr id="A3.tab1.3.1.30.29" class="ltx_tr">
<th id="A3.tab1.3.1.30.29.1" class="ltx_td ltx_th ltx_th_row" style="padding:-2pt 2.8pt;"></th>
<th id="A3.tab1.3.1.30.29.2" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">8</th>
<th id="A3.tab1.3.1.30.29.3" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">2</th>
<td id="A3.tab1.3.1.30.29.4" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">349.8</td>
<td id="A3.tab1.3.1.30.29.5" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">49.3</td>
<td id="A3.tab1.3.1.30.29.6" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">30.4</td>
<td id="A3.tab1.3.1.30.29.7" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">59.9</td>
<td id="A3.tab1.3.1.30.29.8" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">67.9</td>
<td id="A3.tab1.3.1.30.29.9" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">42.8</td>
<td id="A3.tab1.3.1.30.29.10" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">43.5</td>
<td id="A3.tab1.3.1.30.29.11" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">38.9</td>
<td id="A3.tab1.3.1.30.29.12" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">53.8</td>
<td id="A3.tab1.3.1.30.29.13" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">48.3</td>
</tr>
<tr id="A3.tab1.3.1.31.30" class="ltx_tr">
<th id="A3.tab1.3.1.31.30.1" class="ltx_td ltx_th ltx_th_row ltx_border_bb" style="padding:-2pt 2.8pt;"></th>
<th id="A3.tab1.3.1.31.30.2" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_bb" style="padding:-2pt 2.8pt;">8</th>
<th id="A3.tab1.3.1.31.30.3" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_bb" style="padding:-2pt 2.8pt;">1</th>
<td id="A3.tab1.3.1.31.30.4" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 2.8pt;">343.7</td>
<td id="A3.tab1.3.1.31.30.5" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 2.8pt;">48.0</td>
<td id="A3.tab1.3.1.31.30.6" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 2.8pt;">27.7</td>
<td id="A3.tab1.3.1.31.30.7" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 2.8pt;">61.5</td>
<td id="A3.tab1.3.1.31.30.8" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 2.8pt;">67.1</td>
<td id="A3.tab1.3.1.31.30.9" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 2.8pt;">43.1</td>
<td id="A3.tab1.3.1.31.30.10" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 2.8pt;">42.5</td>
<td id="A3.tab1.3.1.31.30.11" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 2.8pt;">37.3</td>
<td id="A3.tab1.3.1.31.30.12" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 2.8pt;">54.7</td>
<td id="A3.tab1.3.1.31.30.13" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 2.8pt;">47.8</td>
</tr>
</tbody>
</table>
</span></div>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<section id="A4" class="ltx_appendix ltx_figure_panel">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix D </span>Layer Sharing Number Ablation</h2>

<figure id="A4.6" class="ltx_table">

<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table"><span id="A4.6.7.1.1" class="ltx_text" style="font-size:90%;">Table 12</span>: </span><span id="A4.6.8.2" class="ltx_text" style="font-size:90%;">Ablation study on the impact of varying layer repetition numbers.</span></figcaption><div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_1">
<div id="A4.6.6" class="ltx_inline-block ltx_figure_panel ltx_transformed_outer" style="width:429.3pt;height:119.3pt;vertical-align:-0.7pt;"><span class="ltx_transformed_inner" style="transform:translate(-78.5pt,21.7pt) scale(0.732267854035748,0.732267854035748) ;">
<table id="A4.6.6.6" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A4.6.6.6.7.1" class="ltx_tr">
<th id="A4.6.6.6.7.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_tt" style="padding:-2pt 4.3pt;"><span id="A4.6.6.6.7.1.1.1" class="ltx_text ltx_font_bold">Model Size</span></th>
<th id="A4.6.6.6.7.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 4.3pt;"><span id="A4.6.6.6.7.1.2.1" class="ltx_text ltx_font_bold">Repeat Times</span></th>
<th id="A4.6.6.6.7.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 4.3pt;"><span id="A4.6.6.6.7.1.3.1" class="ltx_text ltx_font_bold">ARC-e</span></th>
<th id="A4.6.6.6.7.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 4.3pt;"><span id="A4.6.6.6.7.1.4.1" class="ltx_text ltx_font_bold">ARC-c</span></th>
<th id="A4.6.6.6.7.1.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 4.3pt;"><span id="A4.6.6.6.7.1.5.1" class="ltx_text ltx_font_bold">BoolQ</span></th>
<th id="A4.6.6.6.7.1.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 4.3pt;"><span id="A4.6.6.6.7.1.6.1" class="ltx_text ltx_font_bold">PIQA</span></th>
<th id="A4.6.6.6.7.1.7" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 4.3pt;"><span id="A4.6.6.6.7.1.7.1" class="ltx_text ltx_font_bold">SIQA</span></th>
<th id="A4.6.6.6.7.1.8" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 4.3pt;"><span id="A4.6.6.6.7.1.8.1" class="ltx_text ltx_font_bold">HellaSwag</span></th>
<th id="A4.6.6.6.7.1.9" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 4.3pt;"><span id="A4.6.6.6.7.1.9.1" class="ltx_text ltx_font_bold">OBQA</span></th>
<th id="A4.6.6.6.7.1.10" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 4.3pt;"><span id="A4.6.6.6.7.1.10.1" class="ltx_text ltx_font_bold">WinoGrande</span></th>
<th id="A4.6.6.6.7.1.11" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 4.3pt;"><span id="A4.6.6.6.7.1.11.1" class="ltx_text ltx_font_bold">Avg.</span></th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A4.6.6.6.8.1" class="ltx_tr">
<th id="A4.6.6.6.8.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t" style="padding:-2pt 4.3pt;" rowspan="4"><span id="A4.6.6.6.8.1.1.1" class="ltx_text">125M</span></th>
<td id="A4.6.6.6.8.1.2" class="ltx_td ltx_align_center ltx_border_t" style="padding:-2pt 4.3pt;">baseline</td>
<td id="A4.6.6.6.8.1.3" class="ltx_td ltx_align_center ltx_border_t" style="padding:-2pt 4.3pt;">41.6</td>
<td id="A4.6.6.6.8.1.4" class="ltx_td ltx_align_center ltx_border_t" style="padding:-2pt 4.3pt;">25.7</td>
<td id="A4.6.6.6.8.1.5" class="ltx_td ltx_align_center ltx_border_t" style="padding:-2pt 4.3pt;">61.1</td>
<td id="A4.6.6.6.8.1.6" class="ltx_td ltx_align_center ltx_border_t" style="padding:-2pt 4.3pt;">62.4</td>
<td id="A4.6.6.6.8.1.7" class="ltx_td ltx_align_center ltx_border_t" style="padding:-2pt 4.3pt;">43.1</td>
<td id="A4.6.6.6.8.1.8" class="ltx_td ltx_align_center ltx_border_t" style="padding:-2pt 4.3pt;">34.4</td>
<td id="A4.6.6.6.8.1.9" class="ltx_td ltx_align_center ltx_border_t" style="padding:-2pt 4.3pt;">36.9</td>
<td id="A4.6.6.6.8.1.10" class="ltx_td ltx_align_center ltx_border_t" style="padding:-2pt 4.3pt;">51.6</td>
<td id="A4.6.6.6.8.1.11" class="ltx_td ltx_align_center ltx_border_t" style="padding:-2pt 4.3pt;">44.6</td>
</tr>
<tr id="A4.1.1.1.1" class="ltx_tr">
<td id="A4.1.1.1.1.1" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">block-wise share repeat <math id="A4.1.1.1.1.1.m1.1" class="ltx_Math" alttext="\times" display="inline"><semantics id="A4.1.1.1.1.1.m1.1a"><mo id="A4.1.1.1.1.1.m1.1.1" xref="A4.1.1.1.1.1.m1.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="A4.1.1.1.1.1.m1.1b"><times id="A4.1.1.1.1.1.m1.1.1.cmml" xref="A4.1.1.1.1.1.m1.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="A4.1.1.1.1.1.m1.1c">\times</annotation></semantics></math>2</td>
<td id="A4.1.1.1.1.2" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">43.9</td>
<td id="A4.1.1.1.1.3" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">27.9</td>
<td id="A4.1.1.1.1.4" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">61.5</td>
<td id="A4.1.1.1.1.5" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">64.3</td>
<td id="A4.1.1.1.1.6" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">41.5</td>
<td id="A4.1.1.1.1.7" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">35.5</td>
<td id="A4.1.1.1.1.8" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">35.1</td>
<td id="A4.1.1.1.1.9" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">50.2</td>
<td id="A4.1.1.1.1.10" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">45.0</td>
</tr>
<tr id="A4.2.2.2.2" class="ltx_tr">
<td id="A4.2.2.2.2.1" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">block-wise share repeat <math id="A4.2.2.2.2.1.m1.1" class="ltx_Math" alttext="\times" display="inline"><semantics id="A4.2.2.2.2.1.m1.1a"><mo id="A4.2.2.2.2.1.m1.1.1" xref="A4.2.2.2.2.1.m1.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="A4.2.2.2.2.1.m1.1b"><times id="A4.2.2.2.2.1.m1.1.1.cmml" xref="A4.2.2.2.2.1.m1.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="A4.2.2.2.2.1.m1.1c">\times</annotation></semantics></math>3</td>
<td id="A4.2.2.2.2.2" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">43.3</td>
<td id="A4.2.2.2.2.3" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">27.6</td>
<td id="A4.2.2.2.2.4" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">60.9</td>
<td id="A4.2.2.2.2.5" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">63.0</td>
<td id="A4.2.2.2.2.6" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">42.3</td>
<td id="A4.2.2.2.2.7" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">35.6</td>
<td id="A4.2.2.2.2.8" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">34.1</td>
<td id="A4.2.2.2.2.9" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">52.9</td>
<td id="A4.2.2.2.2.10" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">45.0</td>
</tr>
<tr id="A4.3.3.3.3" class="ltx_tr">
<td id="A4.3.3.3.3.1" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">block-wise share repeat <math id="A4.3.3.3.3.1.m1.1" class="ltx_Math" alttext="\times" display="inline"><semantics id="A4.3.3.3.3.1.m1.1a"><mo id="A4.3.3.3.3.1.m1.1.1" xref="A4.3.3.3.3.1.m1.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="A4.3.3.3.3.1.m1.1b"><times id="A4.3.3.3.3.1.m1.1.1.cmml" xref="A4.3.3.3.3.1.m1.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="A4.3.3.3.3.1.m1.1c">\times</annotation></semantics></math>4</td>
<td id="A4.3.3.3.3.2" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">42.0</td>
<td id="A4.3.3.3.3.3" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">26.6</td>
<td id="A4.3.3.3.3.4" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">62.3</td>
<td id="A4.3.3.3.3.5" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">64.4</td>
<td id="A4.3.3.3.3.6" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">41.5</td>
<td id="A4.3.3.3.3.7" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">36.2</td>
<td id="A4.3.3.3.3.8" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">35.8</td>
<td id="A4.3.3.3.3.9" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">53.9</td>
<td id="A4.3.3.3.3.10" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">45.3</td>
</tr>
<tr id="A4.6.6.6.9.2" class="ltx_tr">
<th id="A4.6.6.6.9.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb" style="padding:-2pt 4.3pt;" rowspan="4">
<span id="A4.6.6.6.9.2.1.1" class="ltx_ERROR undefined">\hdashline</span>[0.8pt/1pt]
<span id="A4.6.6.6.9.2.1.2" class="ltx_text">350M</span>
</th>
<td id="A4.6.6.6.9.2.2" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">baseline</td>
<td id="A4.6.6.6.9.2.3" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">50.8</td>
<td id="A4.6.6.6.9.2.4" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">30.6</td>
<td id="A4.6.6.6.9.2.5" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">62.3</td>
<td id="A4.6.6.6.9.2.6" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">68.6</td>
<td id="A4.6.6.6.9.2.7" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">43.5</td>
<td id="A4.6.6.6.9.2.8" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">45.1</td>
<td id="A4.6.6.6.9.2.9" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">43.8</td>
<td id="A4.6.6.6.9.2.10" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">52.4</td>
<td id="A4.6.6.6.9.2.11" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">49.6</td>
</tr>
<tr id="A4.4.4.4.4" class="ltx_tr">
<td id="A4.4.4.4.4.1" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">block-wise share repeat <math id="A4.4.4.4.4.1.m1.1" class="ltx_Math" alttext="\times" display="inline"><semantics id="A4.4.4.4.4.1.m1.1a"><mo id="A4.4.4.4.4.1.m1.1.1" xref="A4.4.4.4.4.1.m1.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="A4.4.4.4.4.1.m1.1b"><times id="A4.4.4.4.4.1.m1.1.1.cmml" xref="A4.4.4.4.4.1.m1.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="A4.4.4.4.4.1.m1.1c">\times</annotation></semantics></math>2</td>
<td id="A4.4.4.4.4.2" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">51.5</td>
<td id="A4.4.4.4.4.3" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">30.8</td>
<td id="A4.4.4.4.4.4" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">59.6</td>
<td id="A4.4.4.4.4.5" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">68.2</td>
<td id="A4.4.4.4.4.6" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">43.9</td>
<td id="A4.4.4.4.4.7" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">47.7</td>
<td id="A4.4.4.4.4.8" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">44.7</td>
<td id="A4.4.4.4.4.9" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">55.0</td>
<td id="A4.4.4.4.4.10" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">50.2</td>
</tr>
<tr id="A4.5.5.5.5" class="ltx_tr">
<td id="A4.5.5.5.5.1" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">block-wise share repeat <math id="A4.5.5.5.5.1.m1.1" class="ltx_Math" alttext="\times" display="inline"><semantics id="A4.5.5.5.5.1.m1.1a"><mo id="A4.5.5.5.5.1.m1.1.1" xref="A4.5.5.5.5.1.m1.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="A4.5.5.5.5.1.m1.1b"><times id="A4.5.5.5.5.1.m1.1.1.cmml" xref="A4.5.5.5.5.1.m1.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="A4.5.5.5.5.1.m1.1c">\times</annotation></semantics></math>3</td>
<td id="A4.5.5.5.5.2" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">49.6</td>
<td id="A4.5.5.5.5.3" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">32.0</td>
<td id="A4.5.5.5.5.4" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">57.3</td>
<td id="A4.5.5.5.5.5" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">68.4</td>
<td id="A4.5.5.5.5.6" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">43.9</td>
<td id="A4.5.5.5.5.7" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">47.0</td>
<td id="A4.5.5.5.5.8" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">40.6</td>
<td id="A4.5.5.5.5.9" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">56.2</td>
<td id="A4.5.5.5.5.10" class="ltx_td ltx_align_center" style="padding:-2pt 4.3pt;">49.4</td>
</tr>
<tr id="A4.6.6.6.6" class="ltx_tr">
<td id="A4.6.6.6.6.1" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 4.3pt;">block-wise share repeat <math id="A4.6.6.6.6.1.m1.1" class="ltx_Math" alttext="\times" display="inline"><semantics id="A4.6.6.6.6.1.m1.1a"><mo id="A4.6.6.6.6.1.m1.1.1" xref="A4.6.6.6.6.1.m1.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="A4.6.6.6.6.1.m1.1b"><times id="A4.6.6.6.6.1.m1.1.1.cmml" xref="A4.6.6.6.6.1.m1.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="A4.6.6.6.6.1.m1.1c">\times</annotation></semantics></math>4</td>
<td id="A4.6.6.6.6.2" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 4.3pt;">52.2</td>
<td id="A4.6.6.6.6.3" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 4.3pt;">32.6</td>
<td id="A4.6.6.6.6.4" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 4.3pt;">60.0</td>
<td id="A4.6.6.6.6.5" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 4.3pt;">69.0</td>
<td id="A4.6.6.6.6.6" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 4.3pt;">44.5</td>
<td id="A4.6.6.6.6.7" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 4.3pt;">47.6</td>
<td id="A4.6.6.6.6.8" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 4.3pt;">41.8</td>
<td id="A4.6.6.6.6.9" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 4.3pt;">55.2</td>
<td id="A4.6.6.6.6.10" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 4.3pt;">50.4</td>
</tr>
</tbody>
</table>
</span></div>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<p id="A4.6.9" class="ltx_p ltx_figure_panel">We extended our investigation to determine the optimal number of layer repetitions. The experiment involved the 8-layer 125M model with a embedding dimension of 896 and the 15-layer 350M model with a embedding dimension of 1280. The results in Table <a href="#A4" title="Appendix D Layer Sharing Number Ablation ‣ Appendix C Number of Heads and Key-Value Heads ‣ Appendix B Depth vs Width ‣ Appendix A Impact of Each Design Choice ‣ Broader Impact ‣ Acknowledgment ‣ 5 Conclusion ‣ 4 Related Work ‣ 3.6 On-device Profiling ‣ 3.5 Knowledge Distillation ‣ 3.4 Compatibility with Quantization ‣ 3.3.2 API Calling ‣ 3.3.1 Chat ‣ 3.3 Downstream Tasks ‣ 3.2 Main Results ‣ 3 Experiments ‣ 2.3 Layer Sharing ‣ 2 Improving Sub-billion Scale LLM Design ‣ MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases" class="ltx_ref"><span class="ltx_text ltx_ref_tag">D</span></a> demonstrate that when we double the layer number and with each two transformer blocks sharing weights, the accuracy is enhanced by 0.4-0.6%. However, as we further triple or quadruple the layer repetition times, this accuracy enhancement effect diminishes. Consequently, in our experiments, we adopt a configuration where every two blocks share weights, effectively doubling the total number of layers.</p>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<section id="A5" class="ltx_appendix ltx_figure_panel">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix E </span>Compatibility with Quantization</h2>

<div id="A5.p1" class="ltx_para">
<p id="A5.p1.2" class="ltx_p">This section explores the compatibility of quantization with the proposed model architecture and layer sharing. We employ a straightforward per-token min-max quantization, quantizing both weight and activation to 8-bit using post-training quantization (PTQ). Experiments are conducted on both <span id="A5.p1.2.1" class="ltx_text ltx_markedasmath ltx_font_typewriter">MobileLLM</span> and <span id="A5.p1.2.2" class="ltx_text ltx_markedasmath ltx_font_typewriter">MobileLLM-LS</span> with model size of 125M and 350M, trained on 0.25T tokens. Results in Table <a href="#A5" title="Appendix E Compatibility with Quantization ‣ Appendix D Layer Sharing Number Ablation ‣ Appendix C Number of Heads and Key-Value Heads ‣ Appendix B Depth vs Width ‣ Appendix A Impact of Each Design Choice ‣ Broader Impact ‣ Acknowledgment ‣ 5 Conclusion ‣ 4 Related Work ‣ 3.6 On-device Profiling ‣ 3.5 Knowledge Distillation ‣ 3.4 Compatibility with Quantization ‣ 3.3.2 API Calling ‣ 3.3.1 Chat ‣ 3.3 Downstream Tasks ‣ 3.2 Main Results ‣ 3 Experiments ‣ 2.3 Layer Sharing ‣ 2 Improving Sub-billion Scale LLM Design ‣ MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases" class="ltx_ref"><span class="ltx_text ltx_ref_tag">E</span></a> demonstrate that W8A8 PTQ results in a modest accuracy drop of within 0.5% and is compatible with the proposed layer sharing method.</p>
</div>
<figure id="A5.8" class="ltx_table">

<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table"><span id="A5.8.9.1.1" class="ltx_text" style="font-size:90%;">Table 13</span>: </span><span id="A5.8.10.2" class="ltx_text" style="font-size:90%;">Ablation study: 8-bit weight, 8-bit activation post-training quantization results on zero-shot common sense reasoning tasks. Quantized models achieve accuracy gap of within 0.5% compared to the full-precision BF16 counterpart.</span></figcaption><div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_1">
<div id="A5.8.8" class="ltx_inline-block ltx_figure_panel ltx_transformed_outer" style="width:429.3pt;height:122.7pt;vertical-align:-0.8pt;"><span class="ltx_transformed_inner" style="transform:translate(-70.7pt,20.1pt) scale(0.752314453382222,0.752314453382222) ;">
<table id="A5.8.8.8" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A5.8.8.8.9.1" class="ltx_tr">
<th id="A5.8.8.8.9.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 2.8pt;"><span id="A5.8.8.8.9.1.1.1" class="ltx_text ltx_font_bold">Model</span></th>
<th id="A5.8.8.8.9.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 2.8pt;"><span id="A5.8.8.8.9.1.2.1" class="ltx_text ltx_font_bold">Precision</span></th>
<th id="A5.8.8.8.9.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 2.8pt;"><span id="A5.8.8.8.9.1.3.1" class="ltx_text ltx_font_bold">ARC-e</span></th>
<th id="A5.8.8.8.9.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 2.8pt;"><span id="A5.8.8.8.9.1.4.1" class="ltx_text ltx_font_bold">ARC-c</span></th>
<th id="A5.8.8.8.9.1.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 2.8pt;"><span id="A5.8.8.8.9.1.5.1" class="ltx_text ltx_font_bold">BoolQ</span></th>
<th id="A5.8.8.8.9.1.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 2.8pt;"><span id="A5.8.8.8.9.1.6.1" class="ltx_text ltx_font_bold">PIQA</span></th>
<th id="A5.8.8.8.9.1.7" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 2.8pt;"><span id="A5.8.8.8.9.1.7.1" class="ltx_text ltx_font_bold">SIQA</span></th>
<th id="A5.8.8.8.9.1.8" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 2.8pt;"><span id="A5.8.8.8.9.1.8.1" class="ltx_text ltx_font_bold">HellaSwag</span></th>
<th id="A5.8.8.8.9.1.9" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 2.8pt;"><span id="A5.8.8.8.9.1.9.1" class="ltx_text ltx_font_bold">OBQA</span></th>
<th id="A5.8.8.8.9.1.10" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 2.8pt;"><span id="A5.8.8.8.9.1.10.1" class="ltx_text ltx_font_bold">WinoGrande</span></th>
<th id="A5.8.8.8.9.1.11" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 2.8pt;"><span id="A5.8.8.8.9.1.11.1" class="ltx_text ltx_font_bold">Avg.</span></th>
<th id="A5.8.8.8.9.1.12" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 2.8pt;"><span id="A5.8.8.8.9.1.12.1" class="ltx_text ltx_font_bold">Gap</span></th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A5.1.1.1.1" class="ltx_tr">
<td id="A5.1.1.1.1.1" class="ltx_td ltx_align_left ltx_border_t" style="padding:-2pt 2.8pt;">
<span id="A5.1.1.1.1.1.1" class="ltx_text ltx_markedasmath ltx_font_typewriter">MobileLLM</span>-125M</td>
<td id="A5.1.1.1.1.2" class="ltx_td ltx_align_center ltx_border_t" style="padding:-2pt 2.8pt;">BF16</td>
<td id="A5.1.1.1.1.3" class="ltx_td ltx_align_center ltx_border_t" style="padding:-2pt 2.8pt;">45.5</td>
<td id="A5.1.1.1.1.4" class="ltx_td ltx_align_center ltx_border_t" style="padding:-2pt 2.8pt;">27.7</td>
<td id="A5.1.1.1.1.5" class="ltx_td ltx_align_center ltx_border_t" style="padding:-2pt 2.8pt;">58.3</td>
<td id="A5.1.1.1.1.6" class="ltx_td ltx_align_center ltx_border_t" style="padding:-2pt 2.8pt;">64.6</td>
<td id="A5.1.1.1.1.7" class="ltx_td ltx_align_center ltx_border_t" style="padding:-2pt 2.8pt;">41.9</td>
<td id="A5.1.1.1.1.8" class="ltx_td ltx_align_center ltx_border_t" style="padding:-2pt 2.8pt;">36.4</td>
<td id="A5.1.1.1.1.9" class="ltx_td ltx_align_center ltx_border_t" style="padding:-2pt 2.8pt;">35.4</td>
<td id="A5.1.1.1.1.10" class="ltx_td ltx_align_center ltx_border_t" style="padding:-2pt 2.8pt;">50.4</td>
<td id="A5.1.1.1.1.11" class="ltx_td ltx_align_center ltx_border_t" style="padding:-2pt 2.8pt;">45.0</td>
<td id="A5.1.1.1.1.12" class="ltx_td ltx_align_center ltx_border_t" style="padding:-2pt 2.8pt;">–</td>
</tr>
<tr id="A5.2.2.2.2" class="ltx_tr">
<td id="A5.2.2.2.2.1" class="ltx_td ltx_align_left" style="padding:-2pt 2.8pt;">
<span id="A5.2.2.2.2.1.1" class="ltx_text ltx_markedasmath ltx_font_typewriter">MobileLLM</span>-125M</td>
<td id="A5.2.2.2.2.2" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">W8A8</td>
<td id="A5.2.2.2.2.3" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">45.2</td>
<td id="A5.2.2.2.2.4" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">27.1</td>
<td id="A5.2.2.2.2.5" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">58.3</td>
<td id="A5.2.2.2.2.6" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">65.0</td>
<td id="A5.2.2.2.2.7" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">41.7</td>
<td id="A5.2.2.2.2.8" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">36.2</td>
<td id="A5.2.2.2.2.9" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">33.6</td>
<td id="A5.2.2.2.2.10" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">51.0</td>
<td id="A5.2.2.2.2.11" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">44.8</td>
<td id="A5.2.2.2.2.12" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">0.2</td>
</tr>
<tr id="A5.3.3.3.3" class="ltx_tr">
<td id="A5.3.3.3.3.1" class="ltx_td ltx_align_left" style="padding:-2pt 2.8pt;">
<span id="A5.3.3.3.3.1.1" class="ltx_ERROR undefined">\hdashline</span>[0.8pt/1pt]
<span id="A5.3.3.3.3.1.2" class="ltx_text ltx_markedasmath ltx_font_typewriter">MobileLLM-LS</span>-125M</td>
<td id="A5.3.3.3.3.2" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">BF16</td>
<td id="A5.3.3.3.3.3" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">44.4</td>
<td id="A5.3.3.3.3.4" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">27.0</td>
<td id="A5.3.3.3.3.5" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">61.5</td>
<td id="A5.3.3.3.3.6" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">65.1</td>
<td id="A5.3.3.3.3.7" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">43.0</td>
<td id="A5.3.3.3.3.8" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">37.6</td>
<td id="A5.3.3.3.3.9" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">37.8</td>
<td id="A5.3.3.3.3.10" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">52.0</td>
<td id="A5.3.3.3.3.11" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">46.1</td>
<td id="A5.3.3.3.3.12" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">–</td>
</tr>
<tr id="A5.4.4.4.4" class="ltx_tr">
<td id="A5.4.4.4.4.1" class="ltx_td ltx_align_left" style="padding:-2pt 2.8pt;">
<span id="A5.4.4.4.4.1.1" class="ltx_text ltx_markedasmath ltx_font_typewriter">MobileLLM-LS</span>-125M</td>
<td id="A5.4.4.4.4.2" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">W8A8</td>
<td id="A5.4.4.4.4.3" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">44.0</td>
<td id="A5.4.4.4.4.4" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">27.5</td>
<td id="A5.4.4.4.4.5" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">60.9</td>
<td id="A5.4.4.4.4.6" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">64.6</td>
<td id="A5.4.4.4.4.7" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">43.1</td>
<td id="A5.4.4.4.4.8" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">37.7</td>
<td id="A5.4.4.4.4.9" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">37.7</td>
<td id="A5.4.4.4.4.10" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">51.0</td>
<td id="A5.4.4.4.4.11" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">45.8</td>
<td id="A5.4.4.4.4.12" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">0.3</td>
</tr>
<tr id="A5.5.5.5.5" class="ltx_tr">
<td id="A5.5.5.5.5.1" class="ltx_td ltx_align_left" style="padding:-2pt 2.8pt;">
<span id="A5.5.5.5.5.1.1" class="ltx_ERROR undefined">\hdashline</span>[0.8pt/1pt]
<span id="A5.5.5.5.5.1.2" class="ltx_text ltx_markedasmath ltx_font_typewriter">MobileLLM</span>-350M</td>
<td id="A5.5.5.5.5.2" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">BF16</td>
<td id="A5.5.5.5.5.3" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">51.4</td>
<td id="A5.5.5.5.5.4" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">31.3</td>
<td id="A5.5.5.5.5.5" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">61.0</td>
<td id="A5.5.5.5.5.6" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">68.1</td>
<td id="A5.5.5.5.5.7" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">43.6</td>
<td id="A5.5.5.5.5.8" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">47.2</td>
<td id="A5.5.5.5.5.9" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">41.6</td>
<td id="A5.5.5.5.5.10" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">55.4</td>
<td id="A5.5.5.5.5.11" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">49.9</td>
<td id="A5.5.5.5.5.12" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">–</td>
</tr>
<tr id="A5.6.6.6.6" class="ltx_tr">
<td id="A5.6.6.6.6.1" class="ltx_td ltx_align_left" style="padding:-2pt 2.8pt;">
<span id="A5.6.6.6.6.1.1" class="ltx_text ltx_markedasmath ltx_font_typewriter">MobileLLM</span>-350M</td>
<td id="A5.6.6.6.6.2" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">W8A8</td>
<td id="A5.6.6.6.6.3" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">51.4</td>
<td id="A5.6.6.6.6.4" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">32.1</td>
<td id="A5.6.6.6.6.5" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">61.1</td>
<td id="A5.6.6.6.6.6" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">68.8</td>
<td id="A5.6.6.6.6.7" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">43.1</td>
<td id="A5.6.6.6.6.8" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">47.1</td>
<td id="A5.6.6.6.6.9" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">40.6</td>
<td id="A5.6.6.6.6.10" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">55.1</td>
<td id="A5.6.6.6.6.11" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">49.9</td>
<td id="A5.6.6.6.6.12" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">0.0</td>
</tr>
<tr id="A5.7.7.7.7" class="ltx_tr">
<td id="A5.7.7.7.7.1" class="ltx_td ltx_align_left" style="padding:-2pt 2.8pt;">
<span id="A5.7.7.7.7.1.1" class="ltx_ERROR undefined">\hdashline</span>[0.8pt/1pt]
<span id="A5.7.7.7.7.1.2" class="ltx_text ltx_markedasmath ltx_font_typewriter">MobileLLM-LS</span>-350M</td>
<td id="A5.7.7.7.7.2" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">BF16</td>
<td id="A5.7.7.7.7.3" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">51.9</td>
<td id="A5.7.7.7.7.4" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">35.2</td>
<td id="A5.7.7.7.7.5" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">59.6</td>
<td id="A5.7.7.7.7.6" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">68.9</td>
<td id="A5.7.7.7.7.7" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">43.4</td>
<td id="A5.7.7.7.7.8" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">47.2</td>
<td id="A5.7.7.7.7.9" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">43.3</td>
<td id="A5.7.7.7.7.10" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">58.4</td>
<td id="A5.7.7.7.7.11" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">51.0</td>
<td id="A5.7.7.7.7.12" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">–</td>
</tr>
<tr id="A5.8.8.8.8" class="ltx_tr">
<td id="A5.8.8.8.8.1" class="ltx_td ltx_align_left ltx_border_bb" style="padding:-2pt 2.8pt;">
<span id="A5.8.8.8.8.1.1" class="ltx_text ltx_markedasmath ltx_font_typewriter">MobileLLM-LS</span>-350M</td>
<td id="A5.8.8.8.8.2" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 2.8pt;">W8A8</td>
<td id="A5.8.8.8.8.3" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 2.8pt;">51.3</td>
<td id="A5.8.8.8.8.4" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 2.8pt;">33.8</td>
<td id="A5.8.8.8.8.5" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 2.8pt;">59.5</td>
<td id="A5.8.8.8.8.6" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 2.8pt;">69.1</td>
<td id="A5.8.8.8.8.7" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 2.8pt;">43.7</td>
<td id="A5.8.8.8.8.8" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 2.8pt;">47.2</td>
<td id="A5.8.8.8.8.9" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 2.8pt;">43.0</td>
<td id="A5.8.8.8.8.10" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 2.8pt;">57.0</td>
<td id="A5.8.8.8.8.11" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 2.8pt;">50.6</td>
<td id="A5.8.8.8.8.12" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 2.8pt;">0.4</td>
</tr>
</tbody>
</table>
</span></div>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<section id="A6" class="ltx_appendix ltx_figure_panel">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix F </span>Knowledge Distillation</h2>

<div id="A6.p1" class="ltx_para">
<p id="A6.p1.7" class="ltx_p">The results of integrating knowledge distillation (KD) <cite class="ltx_cite ltx_citemacro_citep">(Hinton et al., <a href="#bib.bib18" title="" class="ltx_ref">2015</a>)</cite> into small model pre-training are presented in Table <a href="#A6" title="Appendix F Knowledge Distillation ‣ Appendix E Compatibility with Quantization ‣ Appendix D Layer Sharing Number Ablation ‣ Appendix C Number of Heads and Key-Value Heads ‣ Appendix B Depth vs Width ‣ Appendix A Impact of Each Design Choice ‣ Broader Impact ‣ Acknowledgment ‣ 5 Conclusion ‣ 4 Related Work ‣ 3.6 On-device Profiling ‣ 3.5 Knowledge Distillation ‣ 3.4 Compatibility with Quantization ‣ 3.3.2 API Calling ‣ 3.3.1 Chat ‣ 3.3 Downstream Tasks ‣ 3.2 Main Results ‣ 3 Experiments ‣ 2.3 Layer Sharing ‣ 2 Improving Sub-billion Scale LLM Design ‣ MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases" class="ltx_ref"><span class="ltx_text ltx_ref_tag">F</span></a>. LLaMA-v2 7B models serve as the teacher, and the KD loss is computed using cross-entropy between the logits from the large pre-trained teacher model (<span id="A6.p1.7.1" class="ltx_text ltx_font_italic">i.e.</span>, LLaMA-v2 7B) and the small student network (<span id="A6.p1.7.2" class="ltx_text ltx_font_italic">i.e.</span>, 125M or 350M models):</p>
<table id="A9.EGx1" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="A6.E1"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="A6.E1.m1.1" class="ltx_Math" alttext="\displaystyle\mathcal{L}_{CE}=-\frac{1}{n}\sum_{c}\sum^{n}_{i=1}p_{c}^{\mathcal{T}}(X_{i})\log(p_{c}^{\mathcal{S}}(X_{i}))," display="inline"><semantics id="A6.E1.m1.1a"><mrow id="A6.E1.m1.1b"><mi class="ltx_font_mathcaligraphic" id="A6.E1.m1.1.1" xref="A6.E1.m1.1.1.cmml">ℒ</mi><msub id="A6.E1.m1.1.2" xref="A6.E1.m1.1.2.cmml"><mi id="A6.E1.m1.1.2a" xref="A6.E1.m1.1.2.cmml"></mi><mrow id="A6.E1.m1.1.2.1" xref="A6.E1.m1.1.2.1.cmml"><mi id="A6.E1.m1.1.2.1.1" xref="A6.E1.m1.1.2.1.1.cmml">C</mi><mi id="A6.E1.m1.1.2.1.2" xref="A6.E1.m1.1.2.1.2.cmml">E</mi></mrow></msub><mo rspace="0em" id="A6.E1.m1.1.3" xref="A6.E1.m1.1.3.cmml">=</mo><mo lspace="0em" id="A6.E1.m1.1.4" xref="A6.E1.m1.1.4.cmml">−</mo><mstyle displaystyle="true" id="A6.E1.m1.1.5" xref="A6.E1.m1.1.5.cmml"><mfrac id="A6.E1.m1.1.5a" xref="A6.E1.m1.1.5.cmml"><mn id="A6.E1.m1.1.5.2.1" xref="A6.E1.m1.1.5.2.1.cmml">1</mn><mi id="A6.E1.m1.1.5.3.1" xref="A6.E1.m1.1.5.3.1.cmml">n</mi></mfrac></mstyle><mstyle displaystyle="true" id="A6.E1.m1.1.6" xref="A6.E1.m1.1.6.cmml"><mo movablelimits="false" id="A6.E1.m1.1.6a" xref="A6.E1.m1.1.6.cmml">∑</mo></mstyle><msub id="A6.E1.m1.1.7" xref="A6.E1.m1.1.7.cmml"><mi id="A6.E1.m1.1.7a" xref="A6.E1.m1.1.7.cmml"></mi><mi id="A6.E1.m1.1.7.1.1" xref="A6.E1.m1.1.7.1.1.cmml">c</mi></msub><mstyle displaystyle="true" id="A6.E1.m1.1.8" xref="A6.E1.m1.1.8.cmml"><mo movablelimits="false" id="A6.E1.m1.1.8a" xref="A6.E1.m1.1.8.cmml">∑</mo></mstyle><msup id="A6.E1.m1.1.9" xref="A6.E1.m1.1.9.cmml"><mi id="A6.E1.m1.1.9a" xref="A6.E1.m1.1.9.cmml"></mi><mi id="A6.E1.m1.1.9.1.1" xref="A6.E1.m1.1.9.1.1.cmml">n</mi></msup><msub id="A6.E1.m1.1.10" xref="A6.E1.m1.1.10.cmml"><mi id="A6.E1.m1.1.10a" xref="A6.E1.m1.1.10.cmml"></mi><mrow id="A6.E1.m1.1.10.1" xref="A6.E1.m1.1.10.1.cmml"><mi id="A6.E1.m1.1.10.1.1" xref="A6.E1.m1.1.10.1.1.cmml">i</mi><mo id="A6.E1.m1.1.10.1.2" xref="A6.E1.m1.1.10.1.2.cmml">=</mo><mn id="A6.E1.m1.1.10.1.3" xref="A6.E1.m1.1.10.1.3.cmml">1</mn></mrow></msub><mi id="A6.E1.m1.1.11" xref="A6.E1.m1.1.11.cmml">p</mi><msub id="A6.E1.m1.1.12" xref="A6.E1.m1.1.12.cmml"><mi id="A6.E1.m1.1.12a" xref="A6.E1.m1.1.12.cmml"></mi><mi id="A6.E1.m1.1.12.1.1" xref="A6.E1.m1.1.12.1.1.cmml">c</mi></msub><msup id="A6.E1.m1.1.13" xref="A6.E1.m1.1.13.cmml"><mi id="A6.E1.m1.1.13a" xref="A6.E1.m1.1.13.cmml"></mi><mi class="ltx_font_mathcaligraphic" id="A6.E1.m1.1.13.1.1" xref="A6.E1.m1.1.13.1.1.cmml">𝒯</mi></msup><mo stretchy="false" id="A6.E1.m1.1.14" xref="A6.E1.m1.1.14.cmml">(</mo><mi id="A6.E1.m1.1.15" xref="A6.E1.m1.1.15.cmml">X</mi><msub id="A6.E1.m1.1.16" xref="A6.E1.m1.1.16.cmml"><mi id="A6.E1.m1.1.16a" xref="A6.E1.m1.1.16.cmml"></mi><mi id="A6.E1.m1.1.16.1.1" xref="A6.E1.m1.1.16.1.1.cmml">i</mi></msub><mo rspace="0.167em" stretchy="false" id="A6.E1.m1.1.17" xref="A6.E1.m1.1.17.cmml">)</mo><mi id="A6.E1.m1.1.18" xref="A6.E1.m1.1.18.cmml">log</mi><mo stretchy="false" id="A6.E1.m1.1.19" xref="A6.E1.m1.1.19.cmml">(</mo><mi id="A6.E1.m1.1.20" xref="A6.E1.m1.1.20.cmml">p</mi><msub id="A6.E1.m1.1.21" xref="A6.E1.m1.1.21.cmml"><mi id="A6.E1.m1.1.21a" xref="A6.E1.m1.1.21.cmml"></mi><mi id="A6.E1.m1.1.21.1.1" xref="A6.E1.m1.1.21.1.1.cmml">c</mi></msub><msup id="A6.E1.m1.1.22" xref="A6.E1.m1.1.22.cmml"><mi id="A6.E1.m1.1.22a" xref="A6.E1.m1.1.22.cmml"></mi><mi class="ltx_font_mathcaligraphic" id="A6.E1.m1.1.22.1.1" xref="A6.E1.m1.1.22.1.1.cmml">𝒮</mi></msup><mo stretchy="false" id="A6.E1.m1.1.23" xref="A6.E1.m1.1.23.cmml">(</mo><mi id="A6.E1.m1.1.24" xref="A6.E1.m1.1.24.cmml">X</mi><msub id="A6.E1.m1.1.25" xref="A6.E1.m1.1.25.cmml"><mi id="A6.E1.m1.1.25a" xref="A6.E1.m1.1.25.cmml"></mi><mi id="A6.E1.m1.1.25.1.1" xref="A6.E1.m1.1.25.1.1.cmml">i</mi></msub><mo stretchy="false" id="A6.E1.m1.1.26" xref="A6.E1.m1.1.26.cmml">)</mo><mo stretchy="false" id="A6.E1.m1.1.27" xref="A6.E1.m1.1.27.cmml">)</mo><mo id="A6.E1.m1.1.28" xref="A6.E1.m1.1.28.cmml">,</mo></mrow><annotation-xml encoding="MathML-Content" id="A6.E1.m1.1c"><cerror id="A6.E1.m1.1d"><csymbol cd="ambiguous" id="A6.E1.m1.1e">fragments</csymbol><csymbol cd="unknown" id="A6.E1.m1.1.1.cmml" xref="A6.E1.m1.1.1">L</csymbol><apply id="A6.E1.m1.1.2.cmml" xref="A6.E1.m1.1.2"><cerror id="A6.E1.m1.1.2.1.cmml" xref="A6.E1.m1.1.2.1"><csymbol cd="ambiguous" id="A6.E1.m1.1.2.1a.cmml" xref="A6.E1.m1.1.2.1">fragments</csymbol><csymbol cd="unknown" id="A6.E1.m1.1.2.1.1.cmml" xref="A6.E1.m1.1.2.1.1">C</csymbol><csymbol cd="unknown" id="A6.E1.m1.1.2.1.2.cmml" xref="A6.E1.m1.1.2.1.2">E</csymbol></cerror></apply><eq id="A6.E1.m1.1.3.cmml" xref="A6.E1.m1.1.3"></eq><minus id="A6.E1.m1.1.4.cmml" xref="A6.E1.m1.1.4"></minus><apply id="A6.E1.m1.1.5.cmml" xref="A6.E1.m1.1.5"><divide id="A6.E1.m1.1.5.1.cmml" xref="A6.E1.m1.1.5"></divide><cn type="integer" id="A6.E1.m1.1.5.2.1.cmml" xref="A6.E1.m1.1.5.2.1">1</cn><ci id="A6.E1.m1.1.5.3.1.cmml" xref="A6.E1.m1.1.5.3.1">𝑛</ci></apply><sum id="A6.E1.m1.1.6.cmml" xref="A6.E1.m1.1.6"></sum><apply id="A6.E1.m1.1.7.cmml" xref="A6.E1.m1.1.7"><ci id="A6.E1.m1.1.7.1.1.cmml" xref="A6.E1.m1.1.7.1.1">𝑐</ci></apply><sum id="A6.E1.m1.1.8.cmml" xref="A6.E1.m1.1.8"></sum><apply id="A6.E1.m1.1.9.cmml" xref="A6.E1.m1.1.9"><ci id="A6.E1.m1.1.9.1.1.cmml" xref="A6.E1.m1.1.9.1.1">𝑛</ci></apply><apply id="A6.E1.m1.1.10.cmml" xref="A6.E1.m1.1.10"><cerror id="A6.E1.m1.1.10.1.cmml" xref="A6.E1.m1.1.10.1"><csymbol cd="ambiguous" id="A6.E1.m1.1.10.1a.cmml" xref="A6.E1.m1.1.10.1">fragments</csymbol><csymbol cd="unknown" id="A6.E1.m1.1.10.1.1.cmml" xref="A6.E1.m1.1.10.1.1">i</csymbol><eq id="A6.E1.m1.1.10.1.2.cmml" xref="A6.E1.m1.1.10.1.2"></eq><cn type="integer" id="A6.E1.m1.1.10.1.3.cmml" xref="A6.E1.m1.1.10.1.3">1</cn></cerror></apply><csymbol cd="unknown" id="A6.E1.m1.1.11.cmml" xref="A6.E1.m1.1.11">p</csymbol><apply id="A6.E1.m1.1.12.cmml" xref="A6.E1.m1.1.12"><ci id="A6.E1.m1.1.12.1.1.cmml" xref="A6.E1.m1.1.12.1.1">𝑐</ci></apply><apply id="A6.E1.m1.1.13.cmml" xref="A6.E1.m1.1.13"><ci id="A6.E1.m1.1.13.1.1.cmml" xref="A6.E1.m1.1.13.1.1">𝒯</ci></apply><ci id="A6.E1.m1.1.14.cmml" xref="A6.E1.m1.1.14">(</ci><csymbol cd="unknown" id="A6.E1.m1.1.15.cmml" xref="A6.E1.m1.1.15">X</csymbol><apply id="A6.E1.m1.1.16.cmml" xref="A6.E1.m1.1.16"><ci id="A6.E1.m1.1.16.1.1.cmml" xref="A6.E1.m1.1.16.1.1">𝑖</ci></apply><ci id="A6.E1.m1.1.17.cmml" xref="A6.E1.m1.1.17">)</ci><log id="A6.E1.m1.1.18.cmml" xref="A6.E1.m1.1.18"></log><ci id="A6.E1.m1.1.19.cmml" xref="A6.E1.m1.1.19">(</ci><csymbol cd="unknown" id="A6.E1.m1.1.20.cmml" xref="A6.E1.m1.1.20">p</csymbol><apply id="A6.E1.m1.1.21.cmml" xref="A6.E1.m1.1.21"><ci id="A6.E1.m1.1.21.1.1.cmml" xref="A6.E1.m1.1.21.1.1">𝑐</ci></apply><apply id="A6.E1.m1.1.22.cmml" xref="A6.E1.m1.1.22"><ci id="A6.E1.m1.1.22.1.1.cmml" xref="A6.E1.m1.1.22.1.1">𝒮</ci></apply><ci id="A6.E1.m1.1.23.cmml" xref="A6.E1.m1.1.23">(</ci><csymbol cd="unknown" id="A6.E1.m1.1.24.cmml" xref="A6.E1.m1.1.24">X</csymbol><apply id="A6.E1.m1.1.25.cmml" xref="A6.E1.m1.1.25"><ci id="A6.E1.m1.1.25.1.1.cmml" xref="A6.E1.m1.1.25.1.1">𝑖</ci></apply><ci id="A6.E1.m1.1.26.cmml" xref="A6.E1.m1.1.26">)</ci><ci id="A6.E1.m1.1.27.cmml" xref="A6.E1.m1.1.27">)</ci><ci id="A6.E1.m1.1.28.cmml" xref="A6.E1.m1.1.28">,</ci></cerror></annotation-xml><annotation encoding="application/x-tex" id="A6.E1.m1.1f">\displaystyle\mathcal{L}_{CE}=-\frac{1}{n}\sum_{c}\sum^{n}_{i=1}p_{c}^{\mathcal{T}}(X_{i})\log(p_{c}^{\mathcal{S}}(X_{i})),</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(1)</span></td>
</tr></tbody>
</table>
<p id="A6.p1.6" class="ltx_p">Here, <math id="A6.p1.1.m1.1" class="ltx_Math" alttext="i" display="inline"><semantics id="A6.p1.1.m1.1a"><mi id="A6.p1.1.m1.1.1" xref="A6.p1.1.m1.1.1.cmml">i</mi><annotation-xml encoding="MathML-Content" id="A6.p1.1.m1.1b"><ci id="A6.p1.1.m1.1.1.cmml" xref="A6.p1.1.m1.1.1">𝑖</ci></annotation-xml><annotation encoding="application/x-tex" id="A6.p1.1.m1.1c">i</annotation></semantics></math> denotes the <math id="A6.p1.2.m2.1" class="ltx_Math" alttext="i^{th}" display="inline"><semantics id="A6.p1.2.m2.1a"><mrow id="A6.p1.2.m2.1b"><mi id="A6.p1.2.m2.1.1" xref="A6.p1.2.m2.1.1.cmml">i</mi><msup id="A6.p1.2.m2.1.2" xref="A6.p1.2.m2.1.2.cmml"><mi id="A6.p1.2.m2.1.2a" xref="A6.p1.2.m2.1.2.cmml"></mi><mrow id="A6.p1.2.m2.1.2.1" xref="A6.p1.2.m2.1.2.1.cmml"><mi id="A6.p1.2.m2.1.2.1.1" xref="A6.p1.2.m2.1.2.1.1.cmml">t</mi><mi id="A6.p1.2.m2.1.2.1.2" xref="A6.p1.2.m2.1.2.1.2.cmml">h</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="A6.p1.2.m2.1c"><cerror id="A6.p1.2.m2.1d"><csymbol cd="ambiguous" id="A6.p1.2.m2.1e">fragments</csymbol><csymbol cd="unknown" id="A6.p1.2.m2.1.1.cmml" xref="A6.p1.2.m2.1.1">i</csymbol><apply id="A6.p1.2.m2.1.2.cmml" xref="A6.p1.2.m2.1.2"><cerror id="A6.p1.2.m2.1.2.1.cmml" xref="A6.p1.2.m2.1.2.1"><csymbol cd="ambiguous" id="A6.p1.2.m2.1.2.1a.cmml" xref="A6.p1.2.m2.1.2.1">fragments</csymbol><csymbol cd="unknown" id="A6.p1.2.m2.1.2.1.1.cmml" xref="A6.p1.2.m2.1.2.1.1">t</csymbol><csymbol cd="unknown" id="A6.p1.2.m2.1.2.1.2.cmml" xref="A6.p1.2.m2.1.2.1.2">h</csymbol></cerror></apply></cerror></annotation-xml><annotation encoding="application/x-tex" id="A6.p1.2.m2.1f">i^{th}</annotation></semantics></math> sample in the current batch with <math id="A6.p1.3.m3.1" class="ltx_Math" alttext="n" display="inline"><semantics id="A6.p1.3.m3.1a"><mi id="A6.p1.3.m3.1.1" xref="A6.p1.3.m3.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="A6.p1.3.m3.1b"><ci id="A6.p1.3.m3.1.1.cmml" xref="A6.p1.3.m3.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="A6.p1.3.m3.1c">n</annotation></semantics></math> total samples in the batch, and <math id="A6.p1.4.m4.1" class="ltx_Math" alttext="c" display="inline"><semantics id="A6.p1.4.m4.1a"><mi id="A6.p1.4.m4.1.1" xref="A6.p1.4.m4.1.1.cmml">c</mi><annotation-xml encoding="MathML-Content" id="A6.p1.4.m4.1b"><ci id="A6.p1.4.m4.1.1.cmml" xref="A6.p1.4.m4.1.1">𝑐</ci></annotation-xml><annotation encoding="application/x-tex" id="A6.p1.4.m4.1c">c</annotation></semantics></math> represents the number of classes, which, in our case, equals the size of the vocabulary. <math id="A6.p1.5.m5.1" class="ltx_Math" alttext="\mathcal{T}" display="inline"><semantics id="A6.p1.5.m5.1a"><mi class="ltx_font_mathcaligraphic" id="A6.p1.5.m5.1.1" xref="A6.p1.5.m5.1.1.cmml">𝒯</mi><annotation-xml encoding="MathML-Content" id="A6.p1.5.m5.1b"><ci id="A6.p1.5.m5.1.1.cmml" xref="A6.p1.5.m5.1.1">𝒯</ci></annotation-xml><annotation encoding="application/x-tex" id="A6.p1.5.m5.1c">\mathcal{T}</annotation></semantics></math> and <math id="A6.p1.6.m6.1" class="ltx_Math" alttext="\mathcal{S}" display="inline"><semantics id="A6.p1.6.m6.1a"><mi class="ltx_font_mathcaligraphic" id="A6.p1.6.m6.1.1" xref="A6.p1.6.m6.1.1.cmml">𝒮</mi><annotation-xml encoding="MathML-Content" id="A6.p1.6.m6.1b"><ci id="A6.p1.6.m6.1.1.cmml" xref="A6.p1.6.m6.1.1">𝒮</ci></annotation-xml><annotation encoding="application/x-tex" id="A6.p1.6.m6.1c">\mathcal{S}</annotation></semantics></math> are the teacher and student networks, respectively.</p>
</div>
<div id="A6.p2" class="ltx_para">
<p id="A6.p2.1" class="ltx_p">The results in Table <a href="#A6" title="Appendix F Knowledge Distillation ‣ Appendix E Compatibility with Quantization ‣ Appendix D Layer Sharing Number Ablation ‣ Appendix C Number of Heads and Key-Value Heads ‣ Appendix B Depth vs Width ‣ Appendix A Impact of Each Design Choice ‣ Broader Impact ‣ Acknowledgment ‣ 5 Conclusion ‣ 4 Related Work ‣ 3.6 On-device Profiling ‣ 3.5 Knowledge Distillation ‣ 3.4 Compatibility with Quantization ‣ 3.3.2 API Calling ‣ 3.3.1 Chat ‣ 3.3 Downstream Tasks ‣ 3.2 Main Results ‣ 3 Experiments ‣ 2.3 Layer Sharing ‣ 2 Improving Sub-billion Scale LLM Design ‣ MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases" class="ltx_ref"><span class="ltx_text ltx_ref_tag">F</span></a> indicate that adding KD loss is comparable or even lower than solely using the next token as labels. However, it’s noteworthy that the training time using KD is <math id="A6.p2.1.m1.1" class="ltx_Math" alttext="2.6-3.2\times" display="inline"><semantics id="A6.p2.1.m1.1a"><mrow id="A6.p2.1.m1.1b"><mn id="A6.p2.1.m1.1.1" xref="A6.p2.1.m1.1.1.cmml">2.6</mn><mo id="A6.p2.1.m1.1.2" xref="A6.p2.1.m1.1.2.cmml">−</mo><mn id="A6.p2.1.m1.1.3" xref="A6.p2.1.m1.1.3.cmml">3.2</mn><mo lspace="0.222em" id="A6.p2.1.m1.1.4" xref="A6.p2.1.m1.1.4.cmml">×</mo></mrow><annotation-xml encoding="MathML-Content" id="A6.p2.1.m1.1c"><cerror id="A6.p2.1.m1.1d"><csymbol cd="ambiguous" id="A6.p2.1.m1.1e">fragments</csymbol><cn type="float" id="A6.p2.1.m1.1.1.cmml" xref="A6.p2.1.m1.1.1">2.6</cn><minus id="A6.p2.1.m1.1.2.cmml" xref="A6.p2.1.m1.1.2"></minus><cn type="float" id="A6.p2.1.m1.1.3.cmml" xref="A6.p2.1.m1.1.3">3.2</cn><times id="A6.p2.1.m1.1.4.cmml" xref="A6.p2.1.m1.1.4"></times></cerror></annotation-xml><annotation encoding="application/x-tex" id="A6.p2.1.m1.1f">2.6-3.2\times</annotation></semantics></math> slower than training from scratch using labels. All models are trained on 32 A100 80G GPUs with a batch size of 32 for 120k iterations. Consequently, we opt to use labels in our experiments.</p>
</div>
<figure id="A6.tab1" class="ltx_table">

<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table"><span id="A6.tab1.1.1.1" class="ltx_text" style="font-size:90%;">Table 14</span>: </span><span id="A6.tab1.2.2" class="ltx_text" style="font-size:90%;">Ablation study on employing LLaMA-v2 7B teacher’s output as soft labels for knowledge distillation (KD). Results indicate a slight degradation in performance when incorporating KD loss compared to only using hard labels. </span></figcaption><div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_1">
<div id="A6.tab1.3" class="ltx_inline-block ltx_figure_panel ltx_transformed_outer" style="width:429.3pt;height:65.6pt;vertical-align:-0.7pt;"><span class="ltx_transformed_inner" style="transform:translate(-82.8pt,12.5pt) scale(0.721561943182504,0.721561943182504) ;">
<table id="A6.tab1.3.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A6.tab1.3.1.1.1" class="ltx_tr">
<th id="A6.tab1.3.1.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_tt" style="padding:-2pt 2.8pt;"><span id="A6.tab1.3.1.1.1.1.1" class="ltx_text ltx_font_bold">Model</span></th>
<th id="A6.tab1.3.1.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 2.8pt;"><span id="A6.tab1.3.1.1.1.2.1" class="ltx_text ltx_font_bold">Training Loss</span></th>
<th id="A6.tab1.3.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 2.8pt;"><span id="A6.tab1.3.1.1.1.3.1" class="ltx_text ltx_font_bold">Training Time</span></th>
<th id="A6.tab1.3.1.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 2.8pt;"><span id="A6.tab1.3.1.1.1.4.1" class="ltx_text ltx_font_bold">ARC-e</span></th>
<th id="A6.tab1.3.1.1.1.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 2.8pt;"><span id="A6.tab1.3.1.1.1.5.1" class="ltx_text ltx_font_bold">ARC-c</span></th>
<th id="A6.tab1.3.1.1.1.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 2.8pt;"><span id="A6.tab1.3.1.1.1.6.1" class="ltx_text ltx_font_bold">BoolQ</span></th>
<th id="A6.tab1.3.1.1.1.7" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 2.8pt;"><span id="A6.tab1.3.1.1.1.7.1" class="ltx_text ltx_font_bold">PIQA</span></th>
<th id="A6.tab1.3.1.1.1.8" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 2.8pt;"><span id="A6.tab1.3.1.1.1.8.1" class="ltx_text ltx_font_bold">SIQA</span></th>
<th id="A6.tab1.3.1.1.1.9" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 2.8pt;"><span id="A6.tab1.3.1.1.1.9.1" class="ltx_text ltx_font_bold">HellaSwag</span></th>
<th id="A6.tab1.3.1.1.1.10" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 2.8pt;"><span id="A6.tab1.3.1.1.1.10.1" class="ltx_text ltx_font_bold">OBQA</span></th>
<th id="A6.tab1.3.1.1.1.11" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 2.8pt;"><span id="A6.tab1.3.1.1.1.11.1" class="ltx_text ltx_font_bold">WinoGrande</span></th>
<th id="A6.tab1.3.1.1.1.12" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:-2pt 2.8pt;"><span id="A6.tab1.3.1.1.1.12.1" class="ltx_text ltx_font_bold">Avg.</span></th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A6.tab1.3.1.2.1" class="ltx_tr">
<th id="A6.tab1.3.1.2.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t" style="padding:-2pt 2.8pt;">125M model</th>
<td id="A6.tab1.3.1.2.1.2" class="ltx_td ltx_align_center ltx_border_t" style="padding:-2pt 2.8pt;">Label</td>
<td id="A6.tab1.3.1.2.1.3" class="ltx_td ltx_align_center ltx_border_t" style="padding:-2pt 2.8pt;">29h</td>
<td id="A6.tab1.3.1.2.1.4" class="ltx_td ltx_align_center ltx_border_t" style="padding:-2pt 2.8pt;">43.1</td>
<td id="A6.tab1.3.1.2.1.5" class="ltx_td ltx_align_center ltx_border_t" style="padding:-2pt 2.8pt;">28.9</td>
<td id="A6.tab1.3.1.2.1.6" class="ltx_td ltx_align_center ltx_border_t" style="padding:-2pt 2.8pt;">58.1</td>
<td id="A6.tab1.3.1.2.1.7" class="ltx_td ltx_align_center ltx_border_t" style="padding:-2pt 2.8pt;">62.3</td>
<td id="A6.tab1.3.1.2.1.8" class="ltx_td ltx_align_center ltx_border_t" style="padding:-2pt 2.8pt;">42.3</td>
<td id="A6.tab1.3.1.2.1.9" class="ltx_td ltx_align_center ltx_border_t" style="padding:-2pt 2.8pt;">34.6</td>
<td id="A6.tab1.3.1.2.1.10" class="ltx_td ltx_align_center ltx_border_t" style="padding:-2pt 2.8pt;">31.5</td>
<td id="A6.tab1.3.1.2.1.11" class="ltx_td ltx_align_center ltx_border_t" style="padding:-2pt 2.8pt;">50.1</td>
<td id="A6.tab1.3.1.2.1.12" class="ltx_td ltx_align_center ltx_border_t" style="padding:-2pt 2.8pt;">43.9</td>
</tr>
<tr id="A6.tab1.3.1.3.2" class="ltx_tr">
<th id="A6.tab1.3.1.3.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">125M model</th>
<td id="A6.tab1.3.1.3.2.2" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">Label + KD</td>
<td id="A6.tab1.3.1.3.2.3" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">93h</td>
<td id="A6.tab1.3.1.3.2.4" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">41.8</td>
<td id="A6.tab1.3.1.3.2.5" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">28.5</td>
<td id="A6.tab1.3.1.3.2.6" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">58.5</td>
<td id="A6.tab1.3.1.3.2.7" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">61.6</td>
<td id="A6.tab1.3.1.3.2.8" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">41.1</td>
<td id="A6.tab1.3.1.3.2.9" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">34.5</td>
<td id="A6.tab1.3.1.3.2.10" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">32.7</td>
<td id="A6.tab1.3.1.3.2.11" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">51.6</td>
<td id="A6.tab1.3.1.3.2.12" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">43.8</td>
</tr>
<tr id="A6.tab1.3.1.4.3" class="ltx_tr">
<th id="A6.tab1.3.1.4.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row" style="padding:-2pt 2.8pt;">
<span id="A6.tab1.3.1.4.3.1.1" class="ltx_ERROR undefined">\hdashline</span>[0.8pt/1pt]
350M model</th>
<td id="A6.tab1.3.1.4.3.2" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">Label</td>
<td id="A6.tab1.3.1.4.3.3" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">42h</td>
<td id="A6.tab1.3.1.4.3.4" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">50.2</td>
<td id="A6.tab1.3.1.4.3.5" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">31.8</td>
<td id="A6.tab1.3.1.4.3.6" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">56.9</td>
<td id="A6.tab1.3.1.4.3.7" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">67.7</td>
<td id="A6.tab1.3.1.4.3.8" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">44.3</td>
<td id="A6.tab1.3.1.4.3.9" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">45.8</td>
<td id="A6.tab1.3.1.4.3.10" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">40.8</td>
<td id="A6.tab1.3.1.4.3.11" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">55.5</td>
<td id="A6.tab1.3.1.4.3.12" class="ltx_td ltx_align_center" style="padding:-2pt 2.8pt;">49.1</td>
</tr>
<tr id="A6.tab1.3.1.5.4" class="ltx_tr">
<th id="A6.tab1.3.1.5.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb" style="padding:-2pt 2.8pt;">350M model</th>
<td id="A6.tab1.3.1.5.4.2" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 2.8pt;">Label + KD</td>
<td id="A6.tab1.3.1.5.4.3" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 2.8pt;">109h</td>
<td id="A6.tab1.3.1.5.4.4" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 2.8pt;">48.7</td>
<td id="A6.tab1.3.1.5.4.5" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 2.8pt;">31.8</td>
<td id="A6.tab1.3.1.5.4.6" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 2.8pt;">60.7</td>
<td id="A6.tab1.3.1.5.4.7" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 2.8pt;">67.4</td>
<td id="A6.tab1.3.1.5.4.8" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 2.8pt;">43.2</td>
<td id="A6.tab1.3.1.5.4.9" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 2.8pt;">45.9</td>
<td id="A6.tab1.3.1.5.4.10" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 2.8pt;">38.9</td>
<td id="A6.tab1.3.1.5.4.11" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 2.8pt;">53.7</td>
<td id="A6.tab1.3.1.5.4.12" class="ltx_td ltx_align_center ltx_border_bb" style="padding:-2pt 2.8pt;">48.8</td>
</tr>
</tbody>
</table>
</span></div>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<section id="A7" class="ltx_appendix ltx_figure_panel">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix G </span>Datasets and Benchmarks</h2>

<div id="A7.p1" class="ltx_para">
<p id="A7.p1.1" class="ltx_p"><span id="A7.p1.1.1" class="ltx_text ltx_markedasmath ltx_font_typewriter">MobileLLM</span> is assessed on zero-shot common sense reasoning (BoolQ, PIQA, SIQA, HellaSwag, Winogrande, ARC, OBQA), question answering (TriviaQA), and reading comprehension (RACE) tasks. Additionally, we evaluate our chat models on MT-Bench and AlpacaEval benchmarks. We also generated a API calling dataset to fine-tune and evaluate models for this particular task.</p>
</div>
<section id="A7.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">G.1 </span>Zero-shot Common Sense Reasoning tasks</h3>

<div id="A7.SS1.p1" class="ltx_para">
<p id="A7.SS1.p1.1" class="ltx_p"><span id="A7.SS1.p1.1.1" class="ltx_text ltx_font_bold">BoolQ</span> <cite class="ltx_cite ltx_citemacro_citep">(Clark et al., <a href="#bib.bib6" title="" class="ltx_ref">2019</a>)</cite> is a reading comprehension dataset focused on naturally occurring yes/no questions. Each instance includes a question (Q), an excerpt from a passage (P), and an answer (A), with an added explanation for enhanced clarity.</p>
</div>
<div id="A7.SS1.p2" class="ltx_para">
<p id="A7.SS1.p2.1" class="ltx_p"><span id="A7.SS1.p2.1.1" class="ltx_text ltx_font_bold">PIQA</span> <cite class="ltx_cite ltx_citemacro_citep">(Bisk et al., <a href="#bib.bib3" title="" class="ltx_ref">2020</a>)</cite>, abbreviated for Physical Interaction: Question Answering, serves as a benchmark for evaluating and studying the capacity of natural language models in comprehending physical commonsense understanding.</p>
</div>
<div id="A7.SS1.p3" class="ltx_para">
<p id="A7.SS1.p3.1" class="ltx_p"><span id="A7.SS1.p3.1.1" class="ltx_text ltx_font_bold">SIQA</span> <cite class="ltx_cite ltx_citemacro_citep">(Sap et al., <a href="#bib.bib38" title="" class="ltx_ref">2019</a>)</cite>, abbreviated for Social Interaction Question Answering, is designed to measure the social and emotional intelligence of computational models through multiple-choice question answering.</p>
</div>
<div id="A7.SS1.p4" class="ltx_para">
<p id="A7.SS1.p4.1" class="ltx_p"><span id="A7.SS1.p4.1.1" class="ltx_text ltx_font_bold">HellaSwag</span> <cite class="ltx_cite ltx_citemacro_citep">(Zellers et al., <a href="#bib.bib56" title="" class="ltx_ref">2019</a>)</cite> serves as a benchmark for physically situated commonsense natural language inference. It comprises four-way multiple-choice problems that are considered trivial for humans (&gt;95% accuracy) but pose a challenge for language models.</p>
</div>
<div id="A7.SS1.p5" class="ltx_para">
<p id="A7.SS1.p5.1" class="ltx_p"><span id="A7.SS1.p5.1.1" class="ltx_text ltx_font_bold">WinoGrande</span> <cite class="ltx_cite ltx_citemacro_citep">(Sakaguchi et al., <a href="#bib.bib37" title="" class="ltx_ref">2021</a>)</cite> is a benchmark for commonsense reasoning. It consists of a set of 273 expert-crafted pronoun resolution problems, deliberately designed to be unsolvable for statistical models relying on selectional preferences or word associations.</p>
</div>
<div id="A7.SS1.p6" class="ltx_para">
<p id="A7.SS1.p6.1" class="ltx_p"><span id="A7.SS1.p6.1.1" class="ltx_text ltx_font_bold">ARC</span> <cite class="ltx_cite ltx_citemacro_citep">(Clark et al., <a href="#bib.bib7" title="" class="ltx_ref">2018</a>)</cite>, the AI2 Reasoning Challenge, is a compilation of 7787 natural science questions. It is divided into a Challenge Set and an Easy Set, with the Challenge Set exclusively comprising questions that were answered incorrectly by both a retrieval-based algorithm and a word co-occurrence algorithm.</p>
</div>
<div id="A7.SS1.p7" class="ltx_para">
<p id="A7.SS1.p7.1" class="ltx_p"><span id="A7.SS1.p7.1.1" class="ltx_text ltx_font_bold">OBQA</span> <cite class="ltx_cite ltx_citemacro_citep">(Mihaylov et al., <a href="#bib.bib33" title="" class="ltx_ref">2018</a>)</cite> is a dataset consisting of approximately 6000 questions designed for open book question answering. The task involves integrating a corpus of provided science facts (open book) with external broad common knowledge. Providing correct answers necessitates leveraging simple common knowledge beyond the core facts provided.</p>
</div>
</section>
<section id="A7.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">G.2 </span>Question Answering Tasks</h3>

<div id="A7.SS2.p1" class="ltx_para">
<p id="A7.SS2.p1.1" class="ltx_p"><span id="A7.SS2.p1.1.1" class="ltx_text ltx_font_bold">TriviaQA</span> <cite class="ltx_cite ltx_citemacro_citep">(Joshi et al., <a href="#bib.bib21" title="" class="ltx_ref">2017</a>)</cite> is a closed-book question answering benchmark. It encompasses over 650,000 question-answer evidence triples derived by combining 95,000 question-answer pairs authored by Trivia enthusiasts. Each question is supported by an average of six evidence documents.</p>
</div>
</section>
<section id="A7.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">G.3 </span>Reading Comprehension Tasks</h3>

<div id="A7.SS3.p1" class="ltx_para">
<p id="A7.SS3.p1.1" class="ltx_p"><span id="A7.SS3.p1.1.1" class="ltx_text ltx_font_bold">RACE</span> <cite class="ltx_cite ltx_citemacro_citep">(Lai et al., <a href="#bib.bib27" title="" class="ltx_ref">2017</a>)</cite> is a large-scale reading comprehension dataset with more than 28,000 passages and nearly 100,000 questions. The dataset is collected from English examinations in China, which are designed for middle school and high school students.</p>
</div>
</section>
<section id="A7.SS4" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">G.4 </span>Chat Benchmarks</h3>

<div id="A7.SS4.p1" class="ltx_para">
<p id="A7.SS4.p1.1" class="ltx_p"><span id="A7.SS4.p1.1.1" class="ltx_text ltx_font_bold">AlpacaEval</span> <cite class="ltx_cite ltx_citemacro_citep">(Li et al., <a href="#bib.bib28" title="" class="ltx_ref">2023</a>)</cite> is a single-turn benchmark featuring 805 questions across various topics. In the evaluation of this paper, models generate responses and are scored by GPT-4, with the ultimate metric being the pairwise win-rate against a baseline model (text-davinci-001).</p>
</div>
<div id="A7.SS4.p2" class="ltx_para">
<p id="A7.SS4.p2.1" class="ltx_p"><span id="A7.SS4.p2.1.1" class="ltx_text ltx_font_bold">MT-Bench</span> <cite class="ltx_cite ltx_citemacro_citep">(Zheng et al., <a href="#bib.bib58" title="" class="ltx_ref">2023</a>)</cite> is a multi-turn benchmark comprising 160 questions spanning eight diverse knowledge domains. In this evaluation, models are tasked with answering an initial question and subsequently providing a second response to a predefined follow-up question. Each model response is rated by GPT-4 on a scale from 1 to 10, and the final score is determined by the mean over the two turns.</p>
</div>
</section>
<section id="A7.SS5" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">G.5 </span>API Calling Dataset</h3>

<div id="A7.SS5.p1" class="ltx_para">
<p id="A7.SS5.p1.1" class="ltx_p">The API calling dataset is generated by instructing a language model to simulate a conversation between a human and an agent. This process instructs the language model to explicitly mention tasks that should be performed using designated APIs throughout the simulated conversation.
Examples of a conversation turn in generated dataset are shown in the following boxes.</p>
</div>
<div id="A7.SS5.p2" class="ltx_para ltx_framed ltx_framed_rectangle" style="border-color: #000000;padding:9pt;border-width:0.4pt;">
<div id="A7.SS5.p2.1" class="ltx_block ltx_minipage ltx_align_middle" style="width:390.3pt;">
<p id="A7.SS5.p2.1.1" class="ltx_p">{"role": "USER", "message": "Can you help me send a SMS message: We are looking forward to seeing you"},</p>
<p id="A7.SS5.p2.1.2" class="ltx_p">{"role": "PARSER", "message": "style_transfer(text = "We are looking forward to seeing you", style = ["sms"], contact = None)"},</p>
<p id="A7.SS5.p2.1.3" class="ltx_p">{"role": "AGENT", "message": "Sure! Here’s a possible SMS message: Hey [name], we are looking forward to seeing you! Wishing you all well."},</p>
</div>
</div>
<div id="A7.SS5.p3" class="ltx_para ltx_framed ltx_framed_rectangle" style="border-color: #000000;padding:9pt;border-width:0.4pt;">
<div id="A7.SS5.p3.1" class="ltx_block ltx_minipage ltx_align_middle" style="width:390.3pt;">
<p id="A7.SS5.p3.1.1" class="ltx_p">{"role": "USER", "message": "How did the S&amp;P 500 perform last month?"},</p>
<p id="A7.SS5.p3.1.2" class="ltx_p">{"role": "PARSER", "message": "get_stock_information(company_name = "S&amp;P 500", ticker_symbol = "GSPC", date_time = "last month", financial_metrics = "price")"},</p>
<p id="A7.SS5.p3.1.3" class="ltx_p">{"role": "AGENT", "message": "The S&amp;P 500 closed at $2,977.94 last month, with a monthly return of 2.5%."},</p>
</div>
</div>
<div id="A7.SS5.p4" class="ltx_para ltx_framed ltx_framed_rectangle" style="border-color: #000000;padding:9pt;border-width:0.4pt;">
<div id="A7.SS5.p4.1" class="ltx_block ltx_minipage ltx_align_middle" style="width:390.3pt;">
<p id="A7.SS5.p4.1.1" class="ltx_p">{"role": "USER", "message": "Is the San Diego Zoo open during the winter months?"},</p>
<p id="A7.SS5.p4.1.2" class="ltx_p">{"role": "PARSER", "message": "get_local_information(point_of_interest = "San Diego Zoo", attributes = "hours of operation")"},</p>
<p id="A7.SS5.p4.1.3" class="ltx_p">{"role": "AGENT", "message": "The San Diego Zoo is open year-round, but the hours of operation vary depending on the time of year. During the winter months, the zoo is open from 9:00 AM to 5:00 PM."},</p>
</div>
</div>
<div id="A7.SS5.p5" class="ltx_para ltx_framed ltx_framed_rectangle" style="border-color: #000000;padding:9pt;border-width:0.4pt;">
<div id="A7.SS5.p5.1" class="ltx_block ltx_minipage ltx_align_middle" style="width:390.3pt;">
<p id="A7.SS5.p5.1.1" class="ltx_p">{"role": "USER", "message": "What is the most recent news concerning aliens?"},</p>
<p id="A7.SS5.p5.1.2" class="ltx_p">{"role": "PARSER", "message": "news(query = "latest alien news")"},</p>
<p id="A7.SS5.p5.1.3" class="ltx_p">{"role": "AGENT", "message": "According to recent news articles, NASA has announced the discovery of a new planet outside of our solar system that has the potential to support alien life. The planet, named Kepler-186f, is located in the constellation Cygnus and is the first known Earth-sized planet to be found in the habitable zone of a star. While this news does not necessarily prove the existence of aliens, it does provide hope for the possibility of finding life beyond Earth. Additionally, a recent study published in the journal Nature has suggested that the universe may be teeming with alien civilizations, with estimates suggesting that there could be tens of billions of intelligent civilizations in the observable universe. However, it’s important to note that these findings are still being researched and debated in the scientific community."},</p>
</div>
</div>
<div id="A7.SS5.p6" class="ltx_para ltx_framed ltx_framed_rectangle" style="border-color: #000000;padding:9pt;border-width:0.4pt;">
<div id="A7.SS5.p6.1" class="ltx_block ltx_minipage ltx_align_middle" style="width:390.3pt;">
<p id="A7.SS5.p6.1.1" class="ltx_p">{"role": "USER", "message": "Who won the last Super Bowl?"},</p>
<p id="A7.SS5.p6.1.2" class="ltx_p">{"role": "PARSER", "message": "get_sports_information(match = "", sport_type = "NFL", metric = "winner")"},</p>
<p id="A7.SS5.p6.1.3" class="ltx_p">{"role": "AGENT", "message": "The last Super Bowl was won by the Kansas City Chiefs."},</p>
</div>
</div>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
<section id="A8" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix H </span>Approximate calculation of envisioned future GPU consumption</h2>

<div id="A8.p1" class="ltx_para">
<p id="A8.p1.11" class="ltx_p">#GPUs = <math id="A8.p1.1.m1.1" class="ltx_Math" alttext="7.88\!\!\times\!\!10^{9}" display="inline"><semantics id="A8.p1.1.m1.1a"><mrow id="A8.p1.1.m1.1b"><mn id="A8.p1.1.m1.1.1" xref="A8.p1.1.m1.1.1.cmml">7.88</mn><mspace width="-0.166666666666667em" id="A8.p1.1.m1.1.2"></mspace><mspace width="0.055em" id="A8.p1.1.m1.1.3"></mspace><mo id="A8.p1.1.m1.1.4" xref="A8.p1.1.m1.1.4.cmml">×</mo><mspace width="0.055em" id="A8.p1.1.m1.1.5"></mspace><mspace width="-0.166666666666667em" id="A8.p1.1.m1.1.6"></mspace><mn id="A8.p1.1.m1.1.7" xref="A8.p1.1.m1.1.7.cmml">10</mn><msup id="A8.p1.1.m1.1.8" xref="A8.p1.1.m1.1.8.cmml"><mi id="A8.p1.1.m1.1.8a" xref="A8.p1.1.m1.1.8.cmml"></mi><mn id="A8.p1.1.m1.1.8.1.1" xref="A8.p1.1.m1.1.8.1.1.cmml">9</mn></msup></mrow><annotation-xml encoding="MathML-Content" id="A8.p1.1.m1.1c"><cerror id="A8.p1.1.m1.1d"><csymbol cd="ambiguous" id="A8.p1.1.m1.1e">fragments</csymbol><cn type="float" id="A8.p1.1.m1.1.1.cmml" xref="A8.p1.1.m1.1.1">7.88</cn><times id="A8.p1.1.m1.1.4.cmml" xref="A8.p1.1.m1.1.4"></times><cn type="integer" id="A8.p1.1.m1.1.7.cmml" xref="A8.p1.1.m1.1.7">10</cn><apply id="A8.p1.1.m1.1.8.cmml" xref="A8.p1.1.m1.1.8"><cn type="integer" id="A8.p1.1.m1.1.8.1.1.cmml" xref="A8.p1.1.m1.1.8.1.1">9</cn></apply></cerror></annotation-xml><annotation encoding="application/x-tex" id="A8.p1.1.m1.1f">7.88\!\!\times\!\!10^{9}</annotation></semantics></math> (population size) <math id="A8.p1.2.m2.1" class="ltx_Math" alttext="\times 5\%" display="inline"><semantics id="A8.p1.2.m2.1a"><mrow id="A8.p1.2.m2.1b"><mo rspace="0.222em" id="A8.p1.2.m2.1.1" xref="A8.p1.2.m2.1.1.cmml">×</mo><mn id="A8.p1.2.m2.1.2" xref="A8.p1.2.m2.1.2.cmml">5</mn><mo id="A8.p1.2.m2.1.3" xref="A8.p1.2.m2.1.3.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="A8.p1.2.m2.1c"><cerror id="A8.p1.2.m2.1d"><csymbol cd="ambiguous" id="A8.p1.2.m2.1e">fragments</csymbol><times id="A8.p1.2.m2.1.1.cmml" xref="A8.p1.2.m2.1.1"></times><cn type="integer" id="A8.p1.2.m2.1.2.cmml" xref="A8.p1.2.m2.1.2">5</cn><csymbol cd="latexml" id="A8.p1.2.m2.1.3.cmml" xref="A8.p1.2.m2.1.3">percent</csymbol></cerror></annotation-xml><annotation encoding="application/x-tex" id="A8.p1.2.m2.1f">\times 5\%</annotation></semantics></math> (individual time<span id="footnote9" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">9</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">9</sup><span class="ltx_tag ltx_tag_note">9</span>According to the statistics that YouTube users spending an average of 23 hours per month (<a target="_blank" href="https://www.mixbloom.com/resources/average-time-spent-on-social-media-2022" title="" class="ltx_ref ltx_href">source</a>) and Instagram users dedicating 12 hours monthly to the app (<a target="_blank" href="https://blog.hootsuite.com/instagram-statistics/" title="" class="ltx_ref ltx_href">source</a>), we have reason to believe the interaction with LLM can easily surpass the cumulative engagement of these platforms in the future.</span></span></span>) <math id="A8.p1.3.m3.1" class="ltx_Math" alttext="\times\ 220\!\!\times\!\!10^{9}" display="inline"><semantics id="A8.p1.3.m3.1a"><mrow id="A8.p1.3.m3.1b"><mo rspace="0.222em" id="A8.p1.3.m3.1.1" xref="A8.p1.3.m3.1.1.cmml">×</mo><mn id="A8.p1.3.m3.1.2" xref="A8.p1.3.m3.1.2.cmml"> 220</mn><mspace width="-0.166666666666667em" id="A8.p1.3.m3.1.3"></mspace><mspace width="0.055em" id="A8.p1.3.m3.1.4"></mspace><mo id="A8.p1.3.m3.1.5" xref="A8.p1.3.m3.1.5.cmml">×</mo><mspace width="0.055em" id="A8.p1.3.m3.1.6"></mspace><mspace width="-0.166666666666667em" id="A8.p1.3.m3.1.7"></mspace><mn id="A8.p1.3.m3.1.8" xref="A8.p1.3.m3.1.8.cmml">10</mn><msup id="A8.p1.3.m3.1.9" xref="A8.p1.3.m3.1.9.cmml"><mi id="A8.p1.3.m3.1.9a" xref="A8.p1.3.m3.1.9.cmml"></mi><mn id="A8.p1.3.m3.1.9.1.1" xref="A8.p1.3.m3.1.9.1.1.cmml">9</mn></msup></mrow><annotation-xml encoding="MathML-Content" id="A8.p1.3.m3.1c"><cerror id="A8.p1.3.m3.1d"><csymbol cd="ambiguous" id="A8.p1.3.m3.1e">fragments</csymbol><times id="A8.p1.3.m3.1.1.cmml" xref="A8.p1.3.m3.1.1"></times><cn type="integer" id="A8.p1.3.m3.1.2.cmml" xref="A8.p1.3.m3.1.2">220</cn><times id="A8.p1.3.m3.1.5.cmml" xref="A8.p1.3.m3.1.5"></times><cn type="integer" id="A8.p1.3.m3.1.8.cmml" xref="A8.p1.3.m3.1.8">10</cn><apply id="A8.p1.3.m3.1.9.cmml" xref="A8.p1.3.m3.1.9"><cn type="integer" id="A8.p1.3.m3.1.9.1.1.cmml" xref="A8.p1.3.m3.1.9.1.1">9</cn></apply></cerror></annotation-xml><annotation encoding="application/x-tex" id="A8.p1.3.m3.1f">\times\ 220\!\!\times\!\!10^{9}</annotation></semantics></math> (approximating FLOPs/token for 220B GPT-4 model with one expert activated<span id="footnote10" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">10</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">10</sup><span class="ltx_tag ltx_tag_note">10</span>https://the-decoder.com/gpt-4-has-a-trillion-parameters/</span></span></span>) <math id="A8.p1.4.m4.1" class="ltx_Math" alttext="\times 50" display="inline"><semantics id="A8.p1.4.m4.1a"><mrow id="A8.p1.4.m4.1b"><mo rspace="0.222em" id="A8.p1.4.m4.1.1" xref="A8.p1.4.m4.1.1.cmml">×</mo><mn id="A8.p1.4.m4.1.2" xref="A8.p1.4.m4.1.2.cmml">50</mn></mrow><annotation-xml encoding="MathML-Content" id="A8.p1.4.m4.1c"><cerror id="A8.p1.4.m4.1d"><csymbol cd="ambiguous" id="A8.p1.4.m4.1e">fragments</csymbol><times id="A8.p1.4.m4.1.1.cmml" xref="A8.p1.4.m4.1.1"></times><cn type="integer" id="A8.p1.4.m4.1.2.cmml" xref="A8.p1.4.m4.1.2">50</cn></cerror></annotation-xml><annotation encoding="application/x-tex" id="A8.p1.4.m4.1f">\times 50</annotation></semantics></math> (token/s) <math id="A8.p1.5.m5.1" class="ltx_Math" alttext="\times 24" display="inline"><semantics id="A8.p1.5.m5.1a"><mrow id="A8.p1.5.m5.1b"><mo rspace="0.222em" id="A8.p1.5.m5.1.1" xref="A8.p1.5.m5.1.1.cmml">×</mo><mn id="A8.p1.5.m5.1.2" xref="A8.p1.5.m5.1.2.cmml">24</mn></mrow><annotation-xml encoding="MathML-Content" id="A8.p1.5.m5.1c"><cerror id="A8.p1.5.m5.1d"><csymbol cd="ambiguous" id="A8.p1.5.m5.1e">fragments</csymbol><times id="A8.p1.5.m5.1.1.cmml" xref="A8.p1.5.m5.1.1"></times><cn type="integer" id="A8.p1.5.m5.1.2.cmml" xref="A8.p1.5.m5.1.2">24</cn></cerror></annotation-xml><annotation encoding="application/x-tex" id="A8.p1.5.m5.1f">\times 24</annotation></semantics></math> (h) <math id="A8.p1.6.m6.1" class="ltx_Math" alttext="\times 3600" display="inline"><semantics id="A8.p1.6.m6.1a"><mrow id="A8.p1.6.m6.1b"><mo rspace="0.222em" id="A8.p1.6.m6.1.1" xref="A8.p1.6.m6.1.1.cmml">×</mo><mn id="A8.p1.6.m6.1.2" xref="A8.p1.6.m6.1.2.cmml">3600</mn></mrow><annotation-xml encoding="MathML-Content" id="A8.p1.6.m6.1c"><cerror id="A8.p1.6.m6.1d"><csymbol cd="ambiguous" id="A8.p1.6.m6.1e">fragments</csymbol><times id="A8.p1.6.m6.1.1.cmml" xref="A8.p1.6.m6.1.1"></times><cn type="integer" id="A8.p1.6.m6.1.2.cmml" xref="A8.p1.6.m6.1.2">3600</cn></cerror></annotation-xml><annotation encoding="application/x-tex" id="A8.p1.6.m6.1f">\times 3600</annotation></semantics></math> (s) / (<math id="A8.p1.7.m7.1" class="ltx_Math" alttext="60\!\!\times\!\!10^{12}" display="inline"><semantics id="A8.p1.7.m7.1a"><mrow id="A8.p1.7.m7.1b"><mn id="A8.p1.7.m7.1.1" xref="A8.p1.7.m7.1.1.cmml">60</mn><mspace width="-0.166666666666667em" id="A8.p1.7.m7.1.2"></mspace><mspace width="0.055em" id="A8.p1.7.m7.1.3"></mspace><mo id="A8.p1.7.m7.1.4" xref="A8.p1.7.m7.1.4.cmml">×</mo><mspace width="0.055em" id="A8.p1.7.m7.1.5"></mspace><mspace width="-0.166666666666667em" id="A8.p1.7.m7.1.6"></mspace><mn id="A8.p1.7.m7.1.7" xref="A8.p1.7.m7.1.7.cmml">10</mn><msup id="A8.p1.7.m7.1.8" xref="A8.p1.7.m7.1.8.cmml"><mi id="A8.p1.7.m7.1.8a" xref="A8.p1.7.m7.1.8.cmml"></mi><mn id="A8.p1.7.m7.1.8.1.1" xref="A8.p1.7.m7.1.8.1.1.cmml">12</mn></msup></mrow><annotation-xml encoding="MathML-Content" id="A8.p1.7.m7.1c"><cerror id="A8.p1.7.m7.1d"><csymbol cd="ambiguous" id="A8.p1.7.m7.1e">fragments</csymbol><cn type="integer" id="A8.p1.7.m7.1.1.cmml" xref="A8.p1.7.m7.1.1">60</cn><times id="A8.p1.7.m7.1.4.cmml" xref="A8.p1.7.m7.1.4"></times><cn type="integer" id="A8.p1.7.m7.1.7.cmml" xref="A8.p1.7.m7.1.7">10</cn><apply id="A8.p1.7.m7.1.8.cmml" xref="A8.p1.7.m7.1.8"><cn type="integer" id="A8.p1.7.m7.1.8.1.1.cmml" xref="A8.p1.7.m7.1.8.1.1">12</cn></apply></cerror></annotation-xml><annotation encoding="application/x-tex" id="A8.p1.7.m7.1f">60\!\!\times\!\!10^{12}</annotation></semantics></math> (FLOPs/s, H100 computation capacity ) <math id="A8.p1.8.m8.1" class="ltx_Math" alttext="\times 24" display="inline"><semantics id="A8.p1.8.m8.1a"><mrow id="A8.p1.8.m8.1b"><mo rspace="0.222em" id="A8.p1.8.m8.1.1" xref="A8.p1.8.m8.1.1.cmml">×</mo><mn id="A8.p1.8.m8.1.2" xref="A8.p1.8.m8.1.2.cmml">24</mn></mrow><annotation-xml encoding="MathML-Content" id="A8.p1.8.m8.1c"><cerror id="A8.p1.8.m8.1d"><csymbol cd="ambiguous" id="A8.p1.8.m8.1e">fragments</csymbol><times id="A8.p1.8.m8.1.1.cmml" xref="A8.p1.8.m8.1.1"></times><cn type="integer" id="A8.p1.8.m8.1.2.cmml" xref="A8.p1.8.m8.1.2">24</cn></cerror></annotation-xml><annotation encoding="application/x-tex" id="A8.p1.8.m8.1f">\times 24</annotation></semantics></math> (h) <math id="A8.p1.9.m9.1" class="ltx_Math" alttext="\times 3600" display="inline"><semantics id="A8.p1.9.m9.1a"><mrow id="A8.p1.9.m9.1b"><mo rspace="0.222em" id="A8.p1.9.m9.1.1" xref="A8.p1.9.m9.1.1.cmml">×</mo><mn id="A8.p1.9.m9.1.2" xref="A8.p1.9.m9.1.2.cmml">3600</mn></mrow><annotation-xml encoding="MathML-Content" id="A8.p1.9.m9.1c"><cerror id="A8.p1.9.m9.1d"><csymbol cd="ambiguous" id="A8.p1.9.m9.1e">fragments</csymbol><times id="A8.p1.9.m9.1.1.cmml" xref="A8.p1.9.m9.1.1"></times><cn type="integer" id="A8.p1.9.m9.1.2.cmml" xref="A8.p1.9.m9.1.2">3600</cn></cerror></annotation-xml><annotation encoding="application/x-tex" id="A8.p1.9.m9.1f">\times 3600</annotation></semantics></math> (s)) <math id="A8.p1.10.m10.1" class="ltx_Math" alttext="\approx" display="inline"><semantics id="A8.p1.10.m10.1a"><mo id="A8.p1.10.m10.1.1" xref="A8.p1.10.m10.1.1.cmml">≈</mo><annotation-xml encoding="MathML-Content" id="A8.p1.10.m10.1b"><approx id="A8.p1.10.m10.1.1.cmml" xref="A8.p1.10.m10.1.1"></approx></annotation-xml><annotation encoding="application/x-tex" id="A8.p1.10.m10.1c">\approx</annotation></semantics></math> <math id="A8.p1.11.m11.1" class="ltx_Math" alttext="1\times 10^{8}" display="inline"><semantics id="A8.p1.11.m11.1a"><mrow id="A8.p1.11.m11.1b"><mn id="A8.p1.11.m11.1.1" xref="A8.p1.11.m11.1.1.cmml">1</mn><mo lspace="0.222em" rspace="0.222em" id="A8.p1.11.m11.1.2" xref="A8.p1.11.m11.1.2.cmml">×</mo><mn id="A8.p1.11.m11.1.3" xref="A8.p1.11.m11.1.3.cmml">10</mn><msup id="A8.p1.11.m11.1.4" xref="A8.p1.11.m11.1.4.cmml"><mi id="A8.p1.11.m11.1.4a" xref="A8.p1.11.m11.1.4.cmml"></mi><mn id="A8.p1.11.m11.1.4.1.1" xref="A8.p1.11.m11.1.4.1.1.cmml">8</mn></msup></mrow><annotation-xml encoding="MathML-Content" id="A8.p1.11.m11.1c"><cerror id="A8.p1.11.m11.1d"><csymbol cd="ambiguous" id="A8.p1.11.m11.1e">fragments</csymbol><cn type="integer" id="A8.p1.11.m11.1.1.cmml" xref="A8.p1.11.m11.1.1">1</cn><times id="A8.p1.11.m11.1.2.cmml" xref="A8.p1.11.m11.1.2"></times><cn type="integer" id="A8.p1.11.m11.1.3.cmml" xref="A8.p1.11.m11.1.3">10</cn><apply id="A8.p1.11.m11.1.4.cmml" xref="A8.p1.11.m11.1.4"><cn type="integer" id="A8.p1.11.m11.1.4.1.1.cmml" xref="A8.p1.11.m11.1.4.1.1">8</cn></apply></cerror></annotation-xml><annotation encoding="application/x-tex" id="A8.p1.11.m11.1f">1\times 10^{8}</annotation></semantics></math></p>
</div>
<div id="A8.p2" class="ltx_para">
<p id="A8.p2.1" class="ltx_p">This assumes short sequence lengths (<span id="A8.p2.1.1" class="ltx_text ltx_font_italic">i.e.</span> hundreds of tokens, not tens of thousands of tokens), where the compute is dominated by linear layers.
For very long sequences, the self-attention computation would dominate, and this would necessitate a more complicated formula.</p>
</div>
<section id="A9" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix I </span>Chat Generations from <span id="A9.7.1" class="ltx_text ltx_font_typewriter">MobileLLM</span>
</h2>

<div id="A9.p1" class="ltx_para">
<p id="A9.p1.1" class="ltx_p">We show a several examples of chat output generated with <span id="A9.p1.1.1" class="ltx_text ltx_markedasmath ltx_font_typewriter">MobileLLM</span> while responding to various questions.</p>
</div>
<div id="A9.p2" class="ltx_para ltx_minipage ltx_align_middle" style="width:411.9pt;">
<p id="A9.p2.1" class="ltx_p ltx_framed ltx_framed_rectangle" style="border-color: #000000;padding:9pt;border-width:0.4pt;"><span id="A9.p2.1.1" class="ltx_text ltx_font_bold">Input:
<br class="ltx_break"></span>Are there any particular physical benefits to mindful walking, such as improved posture or increased physical fitness?</p>
</div>
<div id="A9.1" class="ltx_para ltx_minipage ltx_align_middle" style="width:411.9pt;">
<div id="A9.1.1" class="ltx_block ltx_framed ltx_framed_rectangle" style="border-color: #000000;padding:9pt;border-width:0.4pt;">
<p id="A9.1.1.1" class="ltx_p"><span id="A9.1.1.1.1" class="ltx_text ltx_font_bold">Output</span> (<span id="A9.1.1.1.2" class="ltx_text ltx_markedasmath ltx_font_typewriter">MobileLLM</span>-125M):</p>
<p id="A9.1.1.2" class="ltx_p">Yes, there are several physical benefits to mindful walking. Here are a few:</p>
<p id="A9.1.1.3" class="ltx_p">1. Improved Posture: Mindful walking can help improve your posture by promoting good alignment and reducing stress on the joints. This can lead to better posture and reduced back pain.</p>
<p id="A9.1.1.4" class="ltx_p">2. Increased Flexibility: Mindful walking can also help increase flexibility in your body, which can reduce the risk of injury and improve overall health.</p>
<p id="A9.1.1.5" class="ltx_p">3. Better Sleep: Mindful walking can help you fall asleep more easily and stay asleep longer, which can improve your sleep quality.</p>
<p id="A9.1.1.6" class="ltx_p">4. Reduced Stress: Mindful walking can help reduce stress levels by promoting relaxation and reducing tension in the body.</p>
<p id="A9.1.1.7" class="ltx_p">5. Enhanced Mental Health: Mindful walking has been shown to have positive effects on mental health, including reducing symptoms of depression and anxiety.</p>
<p id="A9.1.1.8" class="ltx_p">Overall, mindful walking is an excellent way to improve your physical health and well-being. It’s a simple and effective way to get moving and enjoy the benefits it offers. If you’re interested in trying mindful walking, I would recommend starting with shorter walks and gradually increasing the duration over time. You may also want to consider incorporating mindfulness into your daily routine to further enhance your experience.</p>
<p id="A9.1.1.9" class="ltx_p">In conclusion, mindful walking is a great way to improve your physical health and well-being. With its numerous physical benefits, it’s easy to see why it’s becoming increasingly popular among people looking for a fun and sustainable way to get moving. So go ahead and give it a try!</p>
</div>
</div>
<div id="A9.2" class="ltx_para ltx_minipage ltx_align_middle" style="width:411.9pt;">
<div id="A9.2.1" class="ltx_block ltx_framed ltx_framed_rectangle" style="border-color: #000000;padding:9pt;border-width:0.4pt;">
<p id="A9.2.1.1" class="ltx_p"><span id="A9.2.1.1.1" class="ltx_text ltx_font_bold">Output</span> (<span id="A9.2.1.1.2" class="ltx_text ltx_markedasmath ltx_font_typewriter">MobileLLM</span>-350M):</p>
<p id="A9.2.1.2" class="ltx_p">Yes, there are several physical benefits of mindful walking. Firstly, it can improve your posture by reducing the amount of tension in your muscles and joints. Secondly, it can increase your physical fitness by improving your cardiovascular health, strengthening your core muscles, and increasing your flexibility. Additionally, mindful walking can help you to stay more focused on the present moment, which can lead to better sleep quality and reduced stress levels. Finally, it can also help you to develop a greater sense of awareness and appreciation for the world around you, which can enhance your overall well-being.</p>
<p id="A9.2.1.3" class="ltx_p">Overall, mindful walking is a great way to improve your physical health and well-being, while also promoting relaxation and mindfulness. Whether you’re looking to reduce stress, improve your posture, or simply enjoy a peaceful walk, mindful walking can be a rewarding and fulfilling experience.</p>
<p id="A9.2.1.4" class="ltx_p">If you’re interested in trying out mindful walking, there are many resources available that can guide you through the process. Some popular options include guided meditations, apps that offer mindful walking exercises, and classes that teach you how to practice mindful walking. With a little bit of effort and dedication, anyone can start incorporating mindful walking into their daily routine and reap the many benefits it offers.</p>
<p id="A9.2.1.5" class="ltx_p">In conclusion, mindful walking is a simple yet powerful tool that can help you to improve your physical health, reduce stress, and cultivate a deeper sense of awareness and appreciation for the world around you. So why not give it a try and see what kind of benefits you can achieve? Your body and mind will thank you!</p>
</div>
</div>
<div id="A9.p3" class="ltx_para ltx_minipage ltx_align_middle" style="width:411.9pt;">
<div id="A9.p3.1" class="ltx_block ltx_framed ltx_framed_rectangle" style="border-color: #000000;padding:9pt;border-width:0.4pt;">
<p id="A9.p3.1.1" class="ltx_p"><span id="A9.p3.1.1.1" class="ltx_text ltx_font_bold">Output</span> (llama-v2-7b-chat):</p>
<p id="A9.p3.1.2" class="ltx_p">Mindful walking can have a range of physical benefits, including:</p>
<p id="A9.p3.1.3" class="ltx_p">1. Improved posture: By focusing on the sensation of each step and the movement of the body, mindful walking can help improve posture and reduce slouching or hunching.</p>
<p id="A9.p3.1.4" class="ltx_p">2. Increased physical fitness: Mindful walking can help improve cardiovascular fitness, increase muscle strength and endurance, and boost overall physical fitness.</p>
<p id="A9.p3.1.5" class="ltx_p">3. Better balance and coordination: By paying attention to the body’s position in space and the movement of the legs and feet, mindful walking can help improve balance and coordination.</p>
<p id="A9.p3.1.6" class="ltx_p">4. Reduced stress and tension: Mindful walking can help reduce stress and tension in the body, which can lead to improved physical relaxation and well-being.</p>
<p id="A9.p3.1.7" class="ltx_p">5. Improved flexibility: By focusing on the movement of the body and the sensation of each step, mindful walking can help improve flexibility and range of motion.</p>
<p id="A9.p3.1.8" class="ltx_p">6. Boosted immune system: Studies have shown that mindfulness practices, including mindful walking, can help boost the immune system and reduce inflammation.</p>
<p id="A9.p3.1.9" class="ltx_p">7. Improved sleep: Mindful walking can help improve sleep quality and duration by reducing stress and promoting relaxation.
8. Reduced pain: Mindful walking can help reduce pain by focusing on the sensation of each step and the movement of the body, rather than the pain itself.</p>
<p id="A9.p3.1.10" class="ltx_p">9. Improved mood: Mindful walking can help improve mood and reduce symptoms of depression and anxiety by promoting relaxation and reducing stress.</p>
<p id="A9.p3.1.11" class="ltx_p">It’s important to note that the physical benefits of mindful walking may vary depending on the individual and the frequency and duration of the practice. However, with regular practice, mindful walking can have a range of positive effects on physical health and well-being.</p>
</div>
</div>
<div id="A9.p4" class="ltx_para ltx_minipage ltx_align_middle" style="width:411.9pt;">
<p id="A9.p4.1" class="ltx_p ltx_framed ltx_framed_rectangle" style="border-color: #000000;padding:9pt;border-width:0.4pt;"><span id="A9.p4.1.1" class="ltx_text ltx_font_bold">Input:
<br class="ltx_break"></span>How do I wrap a present neatly?</p>
</div>
<div id="A9.3" class="ltx_para ltx_minipage ltx_align_middle" style="width:411.9pt;">
<div id="A9.3.1" class="ltx_block ltx_framed ltx_framed_rectangle" style="border-color: #000000;padding:9pt;border-width:0.4pt;">
<p id="A9.3.1.1" class="ltx_p"><span id="A9.3.1.1.1" class="ltx_text ltx_font_bold">Output</span> (<span id="A9.3.1.1.2" class="ltx_text ltx_markedasmath ltx_font_typewriter">MobileLLM</span>-125M):</p>
<p id="A9.3.1.2" class="ltx_p">1. Choose the right size gift: The size of the present you want to give will depend on your budget and personal preference. If you are looking for a small gift, consider buying a small box or card with a single-use plastic bag.</p>
<p id="A9.3.1.3" class="ltx_p">2. Prepare the present: Make sure all the items in the present are clean and tidy. You can use a cleaning kit or a tissue paper to remove any dirt or debris from the present.</p>
<p id="A9.3.1.4" class="ltx_p">3. Arrange the presents: Place the present in a place where it is easy to see and touch. Use a pencil to draw a line around the present and arrange it in a way that makes sense to you.</p>
<p id="A9.3.1.5" class="ltx_p">4. Add decorations: Decorate the present with stickers, ribbons, or other decorative items. This will add some personality to the present and make it more visually appealing.</p>
<p id="A9.3.1.6" class="ltx_p">5. Enjoy the present: Take time to enjoy the present and let yourself feel the warmth and comfort of the present. Smile at yourself and take a moment to appreciate the present.</p>
<p id="A9.3.1.7" class="ltx_p">Remember, the most important thing is to have fun and enjoy the present! Happy gift-giving!</p>
<p id="A9.3.1.8" class="ltx_p">If you’re feeling adventurous, try making a surprise gift for someone special by wrapping the present in a cute and colorful gift basket. Or, if you’re feeling adventurous, try making a gift for a friend who needs a little extra help with their homework. Whatever you choose, remember to be creative and have fun with your present!</p>
<p id="A9.3.1.9" class="ltx_p">With these tips, you’ll be able to create a beautiful and thoughtful present that will last for years to come. Happy gift-giving!</p>
</div>
</div>
<div id="A9.4" class="ltx_para ltx_minipage ltx_align_middle" style="width:411.9pt;">
<div id="A9.4.1" class="ltx_block ltx_framed ltx_framed_rectangle" style="border-color: #000000;padding:9pt;border-width:0.4pt;">
<p id="A9.4.1.1" class="ltx_p"><span id="A9.4.1.1.1" class="ltx_text ltx_font_bold">Output</span> (<span id="A9.4.1.1.2" class="ltx_text ltx_markedasmath ltx_font_typewriter">MobileLLM</span>-350M):</p>
<p id="A9.4.1.2" class="ltx_p">1. Start by wrapping the present in a decorative gift wrap or ribbon.</p>
<p id="A9.4.1.3" class="ltx_p">2. Use a small amount of glue or tape to secure the gift wrap or ribbon to the surface of the present.</p>
<p id="A9.4.1.4" class="ltx_p">3. Place the gift wrap or ribbon around the present, making sure it is centered and evenly distributed.</p>
<p id="A9.4.1.5" class="ltx_p">4. Use a small amount of glue or tape to secure the gift wrap or ribbon to the surface of the present.</p>
<p id="A9.4.1.6" class="ltx_p">5. Once the gift wrap or ribbon is securely attached, you can add any additional decorations or embellishments to the present.</p>
<p id="A9.4.1.7" class="ltx_p">6. If you want to add a personal touch, you can write a message or a special message on the gift wrap or ribbon.</p>
<p id="A9.4.1.8" class="ltx_p">7. Finally, enjoy your beautifully wrapped present!</p>
<p id="A9.4.1.9" class="ltx_p">Remember, wrapping a gift is a fun and creative way to show your loved one how much you care. Have fun and enjoy the process!</p>
</div>
</div>
<div id="A9.p5" class="ltx_para ltx_minipage ltx_align_middle" style="width:411.9pt;">
<div id="A9.p5.1" class="ltx_block ltx_framed ltx_framed_rectangle" style="border-color: #000000;padding:9pt;border-width:0.4pt;">
<p id="A9.p5.1.1" class="ltx_p"><span id="A9.p5.1.1.1" class="ltx_text ltx_font_bold">Output</span> (llama-v2-7b-chat):</p>
<p id="A9.p5.1.2" class="ltx_p">Wrapping a present neatly can be a bit of a challenge, but with a few simple tips and tricks, you can create a beautifully wrapped gift that your recipient will love. Here are some tips to help you wrap a present neatly:</p>
<p id="A9.p5.1.3" class="ltx_p">1. Use the right wrapping paper: Choose a wrapping paper that complements the gift and is the right size for the present. You can also use a patterned paper or a solid color, depending on your preference.</p>
<p id="A9.p5.1.4" class="ltx_p">2. Measure the gift: Before you start wrapping, measure the gift to make sure you have enough wrapping paper. You can also use a piece of scrap paper to measure the gift and mark the edges with a pencil.</p>
<p id="A9.p5.1.5" class="ltx_p">3. Use a flat surface: Make sure you have a flat surface to work on, such as a table or countertop. This will help you keep the wrapping paper flat and prevent it from wrinkling.</p>
<p id="A9.p5.1.6" class="ltx_p">4. Start with the edges: Begin by folding the edges of the wrapping paper over the gift, making sure they are even and smooth. You can use tape to secure the edges if needed.</p>
<p id="A9.p5.1.7" class="ltx_p">5. Roll the gift: Once you have the edges in place, roll the gift into the center of the wrapping paper. Make sure the gift is securely in place before moving on to the next step.</p>
<p id="A9.p5.1.8" class="ltx_p">6. Tuck in the edges: As you wrap the gift, tuck the edges of the wrapping paper under the gift, smoothing them out as you go. Make sure the edges are even and smooth, and use tape to secure them if needed.</p>
<p id="A9.p5.1.9" class="ltx_p">7. Add a ribbon or bow: Once you have the gift wrapped, add a ribbon or bow to the top. This will not only make the gift look more festive, but it will also help to keep the wrapping paper in place.</p>
<p id="A9.p5.1.10" class="ltx_p">8. Practice makes perfect: Wrapping a present neatly takes a bit of practice, so don’t be discouraged if it doesn’t come out perfectly the first time. Keep practicing, and you’ll get the hang of it in no time!</p>
<p id="A9.p5.1.11" class="ltx_p">By following these tips, you can create a beautifully wrapped present that your recipient will love. Happy wrapping!</p>
</div>
</div>
<div id="A9.p6" class="ltx_para ltx_minipage ltx_align_middle" style="width:411.9pt;">
<p id="A9.p6.1" class="ltx_p ltx_framed ltx_framed_rectangle" style="border-color: #000000;padding:9pt;border-width:0.4pt;"><span id="A9.p6.1.1" class="ltx_text ltx_font_bold">Input:
<br class="ltx_break"></span>What are the best exercises for beginners?</p>
</div>
<div id="A9.5" class="ltx_para ltx_minipage ltx_align_middle" style="width:411.9pt;">
<div id="A9.5.1" class="ltx_block ltx_framed ltx_framed_rectangle" style="border-color: #000000;padding:9pt;border-width:0.4pt;">
<p id="A9.5.1.1" class="ltx_p"><span id="A9.5.1.1.1" class="ltx_text ltx_font_bold">Output</span> (<span id="A9.5.1.1.2" class="ltx_text ltx_markedasmath ltx_font_typewriter">MobileLLM</span>-125M):</p>
<p id="A9.5.1.2" class="ltx_p">1. Push-ups: This is a classic exercise that can be done anywhere, anytime. It’s easy to do and requires no equipment.</p>
<p id="A9.5.1.3" class="ltx_p">2. Dumbbell Rows: These rows are great for building strength in your legs and core.</p>
<p id="A9.5.1.4" class="ltx_p">3. Squats: Squats are a great way to build strength in your legs and core.</p>
<p id="A9.5.1.5" class="ltx_p">4. Lunges: Lunges are a great way to improve balance and stability.</p>
<p id="A9.5.1.6" class="ltx_p">5. Plank: A plank is a great way to build strength in your back and core.</p>
<p id="A9.5.1.7" class="ltx_p">6. Jumping Jacks: Jumping jacks are a fun and effective way to build strength in your legs and core.</p>
<p id="A9.5.1.8" class="ltx_p">7. Burpees: Burpees are a great way to build strength in your legs and core.</p>
<p id="A9.5.1.9" class="ltx_p">8. Mountain Climbers: Mountain climbers are a great way to build strength in your legs and core.</p>
<p id="A9.5.1.10" class="ltx_p">9. Deadlifts: Deadlifts are a great way to build strength in your legs and core.</p>
<p id="A9.5.1.11" class="ltx_p">10. Tricep Dips: Tricep dips are a great way to build strength in your triceps.</p>
<p id="A9.5.1.12" class="ltx_p">Remember, it’s important to start with simple exercises and gradually increase the intensity as you get stronger. Don’t push yourself too hard or you may find it difficult to maintain proper form. With consistent practice, you will see improvements in your strength and overall fitness.</p>
<p id="A9.5.1.13" class="ltx_p">If you have any questions about exercises or healthy living, don’t hesitate to talk to a doctor or physical therapist. They can help you develop a personalized plan that works best for you.</p>
<p id="A9.5.1.14" class="ltx_p">In conclusion, there are many exercises that can be done at home, but it’s important to start slowly and gradually increase the intensity over time. Remember to listen to your body and take breaks when needed. And most importantly, enjoy the process!</p>
</div>
</div>
<div id="A9.6" class="ltx_para ltx_minipage ltx_align_middle" style="width:411.9pt;">
<div id="A9.6.1" class="ltx_block ltx_framed ltx_framed_rectangle" style="border-color: #000000;padding:9pt;border-width:0.4pt;">
<p id="A9.6.1.1" class="ltx_p"><span id="A9.6.1.1.1" class="ltx_text ltx_font_bold">Output</span> (<span id="A9.6.1.1.2" class="ltx_text ltx_markedasmath ltx_font_typewriter">MobileLLM</span>-350M):</p>
<p id="A9.6.1.2" class="ltx_p">Beginners should start with exercises that are easy to do and don’t require a lot of strength or endurance. Here are some exercises that are great for beginners:</p>
<p id="A9.6.1.3" class="ltx_p">1. Push-ups: This is a great exercise for beginners because it’s a simple exercise that can be done anywhere. Start with a push-up on your knees and then lower yourself down to the ground.</p>
<p id="A9.6.1.4" class="ltx_p">2. Squats: Squats are a great exercise for building leg strength and endurance. Start with a squat on your knees and then lower yourself down to the ground.</p>
<p id="A9.6.1.5" class="ltx_p">3. Lunges: Lunges are a great exercise for building leg strength and endurance. Start with a lunge on your front foot and then lower yourself down to the ground.</p>
<p id="A9.6.1.6" class="ltx_p">4. Plank: Planks are a great exercise for building core strength and stability. Start with a plank on your forearms and then lower yourself down to the ground.</p>
<p id="A9.6.1.7" class="ltx_p">5. Bicycle crunches: Bicycle crunches are a great exercise for building upper body strength and stability. Start with a bicycle crunch on your upper back and then lower yourself down to the ground.</p>
<p id="A9.6.1.8" class="ltx_p">6. Plank with leg lifts: This exercise is great for building leg strength and stability. Start with a plank on your forearms and then lift your legs off the ground and hold for a few seconds before lowering them back down.</p>
<p id="A9.6.1.9" class="ltx_p">7. Jumping jacks: Jumping jacks are a great exercise for building cardiovascular endurance and strength. Start with a jumping jack on your feet and then jump up and spread your legs.</p>
<p id="A9.6.1.10" class="ltx_p">Remember to start with a weight that is comfortable for you and gradually increase the weight as you get stronger. It’s also important to warm up before exercising and to stretch after your workout to prevent injury.</p>
<p id="A9.6.1.11" class="ltx_p">In conclusion, beginners should start with exercises that are easy to do and don’t require a lot of strength or endurance. These exercises are great for building leg strength and endurance, and can be done anywhere. Remember to start with a weight that is comfortable for you and gradually increase the weight as you get stronger. Warming up and stretching after your workout are also important to prevent injury.</p>
<p id="A9.6.1.12" class="ltx_p">Following these exercises can help you build strength, endurance, and improve your overall fitness level. Remember to always consult with a healthcare professional before starting any new exercise program.</p>
</div>
</div>
<div id="A9.p7" class="ltx_para ltx_minipage ltx_align_middle" style="width:411.9pt;">
<div id="A9.p7.1" class="ltx_block ltx_framed ltx_framed_rectangle" style="border-color: #000000;padding:9pt;border-width:0.4pt;">
<p id="A9.p7.1.1" class="ltx_p"><span id="A9.p7.1.1.1" class="ltx_text ltx_font_bold">Output</span> (llama-v2-7b-chat):</p>
<p id="A9.p7.1.2" class="ltx_p">Exercise is an essential part of a healthy lifestyle, but for beginners, it can be intimidating and overwhelming. Here are some of the best exercises for beginners:</p>
<p id="A9.p7.1.3" class="ltx_p">1. Bodyweight exercises: These exercises are great for beginners because they don’t require any equipment. Examples include push-ups, squats, lunges, and planks.</p>
<p id="A9.p7.1.4" class="ltx_p">2. Yoga: Yoga is a low-impact exercise that can help improve flexibility, balance, and strength. It’s also great for reducing stress and improving overall well-being.</p>
<p id="A9.p7.1.5" class="ltx_p">3. Brisk walking: Walking is a simple and accessible form of exercise that can be done anywhere. It’s a great way to get your heart rate up and improve cardiovascular health.</p>
<p id="A9.p7.1.6" class="ltx_p">4. Swimming: Swimming is a low-impact exercise that can help improve cardiovascular health and strengthen muscles. It’s also easy on the joints, making it a great option for beginners.</p>
<p id="A9.p7.1.7" class="ltx_p">5. Cycling: Cycling is another low-impact exercise that can be done indoors or outdoors. It’s a great way to improve cardiovascular health and build leg strength.</p>
<p id="A9.p7.1.8" class="ltx_p">6. Resistance band exercises: Resistance bands are lightweight and portable, making them easy to use at home or on the go. They can help improve strength and flexibility in the upper body, lower body, and core.</p>
<p id="A9.p7.1.9" class="ltx_p">7. Pilates: Pilates is a form of exercise that focuses on core strength, flexibility, and body control. It’s a great option for beginners because it’s low-impact and can help improve posture and balance.</p>
<p id="A9.p7.1.10" class="ltx_p">8. Dance-based workouts: Dance-based workouts, such as Zumba or Hip Hop Abs, are fun and energetic ways to get in shape. They often involve aerobic exercise and strength training, making them a great option for beginners.</p>
<p id="A9.p7.1.11" class="ltx_p">9. Jumping rope: Jumping rope is a high-intensity exercise that can help improve cardiovascular health and burn calories. It’s also easy to do at home with a simple jump rope.</p>
<p id="A9.p7.1.12" class="ltx_p">10. Kettlebell exercises: Kettlebells are small weights that can be used to perform a variety of exercises, including swings, squats, and presses. They’re a great way to improve strength and cardiovascular health.</p>
<p id="A9.p7.1.13" class="ltx_p">Remember to always consult with a doctor before starting any new exercise program, especially if you’re a beginner. It’s also important to start slowly and gradually increase the intensity and duration of your workouts as you become more comfortable.</p>
</div>
</div>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
</section>
</section>
</div>
</div>
</figure>
</section>
</div>
</div>
</figure>
</section>
</div>
</div>
</figure>
</section>
</div>
</div>
</figure>
</section>
</div>
</div>
</figure>
</div>
</div>
</figure>
</section>
</div>
</div>
</figure>
</section>
</section>
</section>
</section>
</section>
</div>
</div>
</figure>
</section>
</section>
</section>
</div>
</div>
</figure>
</section>
</div>
</div>
</figure>
</section>
</section>
</div>
</div>
</figure>
</div>
</div>
</figure>
</section>
</section>
</div>
</div>
</figure>
</section>
</section>
</article>
</div>

<div class="ltx_document"><div class="ltx_para"><div class="ltx_p"><span class="ltx_ERROR">
Conversion to HTML had a Fatal error and exited abruptly. This document may be truncated or damaged.
</span></div></div></div>
</article>
<div class="ar5iv-footer"><a href="/html/2402.14904" class="ar5iv-nav-button ar5iv-nav-button-prev">◄</a>
    <a class="ar5iv-home-button" href="/"><img height="40" alt="ar5iv homepage" src="/assets/ar5iv.png"></a>
    <a href="/feeling_lucky" class="ar5iv-text-button">Feeling<br>lucky?</a>
    <a href="/log/2402.14905" class="ar5iv-text-button ar5iv-severity-fatal">Conversion<br>report</a>
    <a class="ar5iv-text-button" target="_blank" href="https://github.com/dginev/ar5iv/issues/new?template=improve-article--arxiv-id-.md&title=Improve+article+2402.14905">Report<br>an issue</a>
    <a href="https://arxiv.org/abs/2402.14905" class="ar5iv-text-button arxiv-ui-theme">View&nbsp;original<br>on&nbsp;arXiv</a><a href="/html/2402.14906" class="ar5iv-nav-button ar5iv-nav-button-next">►</a>
</div><footer class="ltx_page_footer">
<a class="ar5iv-toggle-color-scheme" href="javascript:toggleColorScheme()" title="Toggle ar5iv color scheme"><span class="color-scheme-icon"></span></a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/license" target="_blank">Copyright</a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/policies/privacy_policy" target="_blank">Privacy Policy</a>

<div class="ltx_page_logo">Generated  on Tue Mar  5 18:33:22 2024 by <a target="_blank" href="http://dlmf.nist.gov/LaTeXML/" class="ltx_LaTeXML_logo"><span style="letter-spacing:-0.2em; margin-right:0.1em;">L<span class="ltx_font_smallcaps" style="position:relative; bottom:2.2pt;">a</span>T<span class="ltx_font_smallcaps" style="font-size:120%;position:relative; bottom:-0.2ex;">e</span></span><span style="font-size:90%; position:relative; bottom:-0.2ex;">XML</span><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAsAAAAOCAYAAAD5YeaVAAAAAXNSR0IArs4c6QAAAAZiS0dEAP8A/wD/oL2nkwAAAAlwSFlzAAALEwAACxMBAJqcGAAAAAd0SU1FB9wKExQZLWTEaOUAAAAddEVYdENvbW1lbnQAQ3JlYXRlZCB3aXRoIFRoZSBHSU1Q72QlbgAAAdpJREFUKM9tkL+L2nAARz9fPZNCKFapUn8kyI0e4iRHSR1Kb8ng0lJw6FYHFwv2LwhOpcWxTjeUunYqOmqd6hEoRDhtDWdA8ApRYsSUCDHNt5ul13vz4w0vWCgUnnEc975arX6ORqN3VqtVZbfbTQC4uEHANM3jSqXymFI6yWazP2KxWAXAL9zCUa1Wy2tXVxheKA9YNoR8Pt+aTqe4FVVVvz05O6MBhqUIBGk8Hn8HAOVy+T+XLJfLS4ZhTiRJgqIoVBRFIoric47jPnmeB1mW/9rr9ZpSSn3Lsmir1fJZlqWlUonKsvwWwD8ymc/nXwVBeLjf7xEKhdBut9Hr9WgmkyGEkJwsy5eHG5vN5g0AKIoCAEgkEkin0wQAfN9/cXPdheu6P33fBwB4ngcAcByHJpPJl+fn54mD3Gg0NrquXxeLRQAAwzAYj8cwTZPwPH9/sVg8PXweDAauqqr2cDjEer1GJBLBZDJBs9mE4zjwfZ85lAGg2+06hmGgXq+j3+/DsixYlgVN03a9Xu8jgCNCyIegIAgx13Vfd7vdu+FweG8YRkjXdWy329+dTgeSJD3ieZ7RNO0VAXAPwDEAO5VKndi2fWrb9jWl9Esul6PZbDY9Go1OZ7PZ9z/lyuD3OozU2wAAAABJRU5ErkJggg==" alt="Mascot Sammy"></a>
</div></footer>
</div>

    <script>
      var canMathML = typeof(MathMLElement) == "function";
      if (!canMathML) {
        var body = document.querySelector("body");
        body.firstElementChild.setAttribute('style', 'opacity: 0;');
        var loading = document.createElement("div");
        loading.setAttribute("id", "mathjax-loading-spinner");
        var message = document.createElement("div");
        message.setAttribute("id", "mathjax-loading-message");
        message.innerText = "Typesetting Equations...";
        body.prepend(loading);
        body.prepend(message);

        var el = document.createElement("script");
        el.src = "https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js";
        document.querySelector("head").appendChild(el);

        window.MathJax = {
          startup: {
            pageReady: () => {
              return MathJax.startup.defaultPageReady().then(() => {
                body.removeChild(loading);
                body.removeChild(message);
                body.firstElementChild.removeAttribute('style');
              }); } } };
      }
    </script>
    <script>
    // Auxiliary function, building the preview feature when
    // an inline citation is clicked
    function clicked_cite(e) {
      e.preventDefault();
      let cite = this.closest('.ltx_cite');
      let next = cite.nextSibling;
      if (next && next.nodeType == Node.ELEMENT_NODE && next.getAttribute('class') == "ar5iv-bibitem-preview") {
        next.remove();
        return; }
      // Before adding a preview modal,
      // cleanup older previews, in case they're still open
      document.querySelectorAll('span.ar5iv-bibitem-preview').forEach(function(node) {
        node.remove();
      })

      // Create the preview
      preview = document.createElement('span');
      preview.setAttribute('class','ar5iv-bibitem-preview');
      let target = document.getElementById(this.getAttribute('href').slice(1));
      target.childNodes.forEach(function (child) {
        preview.append(child.cloneNode(true));
      });
      let close_x = document.createElement('button');
      close_x.setAttribute("aria-label","Close modal for bibliography item preview");
      close_x.textContent = "×";
      close_x.setAttribute('class', 'ar5iv-button-close-preview');
      close_x.setAttribute('onclick','this.parentNode.remove()');
      preview.append(close_x);
      preview.querySelectorAll('.ltx_tag_bibitem').forEach(function(node) {
        node.remove();
      });
      cite.parentNode.insertBefore(preview, cite.nextSibling);
      return;
    }
    // Global Document initialization:
    // - assign the preview feature to all inline citation links
    document.querySelectorAll(".ltx_cite .ltx_ref").forEach(function (link) {
      link.addEventListener("click", clicked_cite);
    });
    </script>
    </body>
</html>

2402.14905
This paper addresses the growing need for efficient large language models (LLMs)  on mobile devices, driven by increasing cloud costs and latency concerns. We focus on designing top-quality LLMs with fewer than a billion parameters, a practical choice for mobile deployment.
Contrary to prevailing belief emphasizing the pivotal role of data and parameter quantity in determining model quality, our investigation underscores the significance of model architecture for sub-billion scale LLMs. Leveraging deep and thin architectures, coupled with embedding sharing and grouped-query attention mechanisms, we establish a strong baseline network denoted asMobileLLM, which attains a remarkable 2.7%/4.3% accuracy boost over preceding 125M/350M state-of-the-art models. Additionally, we propose an immediate block-wise weight sharing approach with no increase in model size and only marginal latency overhead. The resultant models, denoted asMobileLLM-LS, demonstrate a further accuracy enhancement of 0.7%/0.8% thanMobileLLM125M/350M.
Moreover,MobileLLMmodel family shows significant improvements compared to previous sub-billion models on chat benchmarks, and demonstrates close correctness to LLaMA-v2 7B in API calling tasks, highlighting the capability of small models for common on-device use cases.
abstract
MobileLLM: Optimizing Sub-billion Parameter Language Modelsfor On-Device Use Cases
Abstract
2402.14905
1IntroductionFigure 1:Average score on zero-shot common sense tasks for LLMs smaller than 1B parameters. Each bubble’s area is proportional to the model size of a variant in a model family. The results of previous methods were evaluated using open-source Hugging Face models to ensure consistent evaluation procedures. The full list of tasks is in Table3.2.Large language models (LLMs)  are permeating various facets of human life, influencing not only the way people communicate and work but also shaping everyday entertainment experiences. Prominent examples of contemporary LLM products, such as ChatGPT and Perplexity AI, primarily operate in cloud environments. Leading models such as ChatGPT4 exceed 1 trillion parameters.111https://the-decoder.com/gpt-4-has-a-trillion-parametersHowever, envisioning a future scenario characterized by widespread human reliance of LLMs in both front-end conversational interfaces and back-end operations like recommendation system, equating to5% of individuals’ daily time. In this hypothetical scenario, employing GPT-4 at processing rate of 50 tokens/s entails the deployment of around one hundred million H100 GPUs222Detailed calculation can be found in appendix., each capable of 60 TFLOPs/s333https://www.nvidia.com/en-us/data-center/h100/. This computation scale, excluding communication and data transmission overhead, is on par with 160 Meta-scale companies444https://twitter.com/soumithchintala/status/1748074223187173724. The ensuing energy consumption and carbon dioxide emissions would present staggering environmental challenges. Consequently, it is imperative that we downsize LLMs.Figure 2:Memory hierarchy in prevalent mobile devices. Despite adequate Flash storage, the operational memory for executing high-speed applications predominantly resides in DRAM, typically constrained to 6-12 GB.Furthermore, considerations ofportabilityandcomputational costpropel the necessity to deploy LLMs on smartphones and mobile devices. In the current landscape of mobile technology, integrating a LLM like the LLaMA-v2 7Bwith 8-bit weights proves prohibitively expensive due to limitations in main-memory (DRAM)  capacity source. A prevalent memory hierarchy in mobile devices is depicted in Figure2.
With DRAM capacities ranging from 6 GB for the iPhone 15 and 12 GB for the Google Pixel 8 Pro, a mobile app should not exceed 10% of the DRAM, since DRAM is shared with the operating system and other applications. This motivates deploying sub-billion parameter LLMs. Additionally, factoring in LLM energy consumption (0.1 J/token per billion in model parameters) , a 7B-parameter LLM consumes 0.7 J/token. A fully charged iPhone, with approximately 50kJ of energy, can sustain this model in conversation for less than 2 hours at a rate of 10 tokens/s, with every 64 tokens draining 0.2% of the battery.These demands converge on a singular imperative: the adoption of compact models for on-device execution.
By utilizing a sub-billion model, such as a 350M 8-bit model consuming only 0.035 J/token, an iPhone can support conversational use an entire day.
Moreover, the decoding speed can be significantly enhanced, as exemplified by the benchmark results of the 125M model, capable of operating at of 50 tokens/s, comparing to the state-of-the-art iPhone App MLC Chat utilizing the LLaMA 7B model at 36 tokens/second555https://llm.mlc.ai. In light of these considerations, our paper is motivated by the design and implementation of LLMs with parameters less than 1 billion.We make following contributions to build the most accurate LLMs to date under 1 billion parameters.•Contradictory to the scaling law, we demonstrate that depth is more important than width for small LLMs. A deep-and-thin model structure excels in capturing abstract concepts, resulting in superior final performance.•We revisit embedding sharing methodsand implement grouped query attentionin small LLMs to maximize weight utilization.•We propose immediate block-wise weight sharing. In scenarios where memory movement is the latency bottleneck, weight sharing between two adjacent blocks avoids weight movement, requiring only computing the block twice and incurring minimal latency overhead.•We propose a new family of models,MobileLLM, showcasing SOTA performance. In a suite of zero-shot tasks,MobileLLMoutperforms the previous SOTA 125M/350M models by 2.7%/4.3%.•In downstream tasks, such as Chat and API calling,MobileLLMmodel family significantly outperforms equivalently-sized models. In an API calling task,MobileLLMeven achieves comparable exact-match score as considerably larger LLaMA-v2 7B model.
introduction
MobileLLM: Optimizing Sub-billion Parameter Language Modelsfor On-Device Use Cases
1Introduction
2402.14905
2Improving Sub-billion Scale LLM DesignIn this section, we present the evolutionary path from a baseline sub-billion parameter model to the new state-of-the-art models (Figure3) . We explore both 125M and 350M models and demonstrate consistent improvements in both cases. For on-device use cases where model size is a major constraint, how to allocate the limited weight parameters effectively becomes more critical than ever. We first propose a strong baseline model namedMobileLLMby testing out four model design techniques that are beneficial for sub-billion scale LLMs, including (1)  adopting SwiGLU FFN; (2)  forcinglanky(deep and thin)  architectures (3)  revisiting embedding sharing method(4)  utilizing grouped query attention. Then we develop an immediate block-wise layer sharing method that further boosts accuracy without incurring any additional memory overhead and with only slight latency overhead in the memory-bounded LM decoding process. We denote our model with layer sharing asMobileLLM-LS.2.1Training SetupOur experiments are conducted on 32 A100 GPUs, with each GPU having a batch size of 32. We performed exploratory experiments with 120k iterations on 0.25T tokens. Subsequently, the top models reported in Table3.2and Table3.2, are trained with 480k iterations on 1T tokens.We evaluate the pre-trained model on zero-shot common sense reasoning tasks, including ARC-easy, ARC-challenge, BoolQ, PIQA, SIQA, HellaSwag, OBQA, WinoGrande, as well as question answering and reading comprehension tasks using TQAand RACE dataset.Figure 3:Design roadmap of sub-billion sized transformer models. The foreground and background bars represents the averaged accuracy on zero-shot common sense reasoning tasks for 125M and 350M models, respectively. The 125M model, initially a 12-layer 768-dimension structure, is enhanced through improving feed-forward network design, network depth adjustments, and weight sharing strategies. The detailed accuracy on each modification can be found in the appendix.2.2Building a Strong Baseline2.2.1Feed-forward Network ChoiceWe first investigate activation functions commonly used in feed-forward networks (FFNs)  and find that the state-of-the-art SwiGLUis also beneficial for small models. By changing the vanilla FFN ()  to SwiGLU, The average performance on zero-shot reasoning tasks is boost from 42.6 to 43.9 for the 125M model. Therefore, we use SwiGLU in FFN for the experiments afterwards.2.2.2Architecture Depth vs WidthFigure 4:Under comparable model sizes, deeper and thinner models generally outperform their wider and shallower counterparts across various tasks such as zero-shot common sense reasoning, question answering, and reading comprehension.A prevalent beliefin the field suggests that the performance of transformer models is primarily determined by number of parameters, the size of the training dataset, and the number of training iterations. This belief posits that architectural designs have negligible impact on the transformer model’s performance. However, our findings indicate that this may not hold true for smaller models.Our experimental results, specifically for small models with limited model capacity, reveals that going deeper is more crucial than going wider for performance improvement.
We conducted an extensive study involving the training of 19 models, including 9 models with125M parameters and 10 models with350M parameters. Each model is designed with similar size but varied in terms of depth and width. We experiment on eight zero-shot common sense reasoning tasks, as well as question answering and reading comprehension benchmarks. Our findings consistently demonstrate that deeper and thinner models outperform their shallower and wider counterparts. Figure4(a)  and (b)  illustrate the superior performance of deeper networks across most zero-shot reasoning tasks, including ARC-easy, ARC-challenge, PIQA, HellaSwag, OBQA, WinoGrande. Particularly, this trend is even more pronounced on the TQA and RACE datasets, as shown in Figure4(c) -(f) . Detailed model configurations and results can be seen in appendix. Our findings suggest that models with 30 or even 42 layers perform significantly better than those with 12 layers for transformer models sized around 125M. This finding is surprising considering the number of layers in most previous 125M modelsis limited to 12.2.2.3Embedding SharingTable 1:Ablation study on input-output embedding sharing with a 30-layer model with 512 embedding dimension, on zero-shot common-sense reasoning tasks. Increased depth (depth)  model has 32 layers.Model# ParamsARC-eARC-cBoolQPIQASIQAHSOBQAWinoGrandeAvg.Without emb-share135M43.626.158.062.542.636.537.551.544.8+ emb-share119M44.426.056.262.843.135.936.052.644.6+ emb-share,depth125M43.326.454.464.743.536.938.552.645.0In sub-billion scale language models, the embedding layers constitute a significant portion of the parameter count. For instance, with an embedding dimension of 512 and a vocabulary size of 32k, the input and output embedding layers each comprise 16 million parameters. Together, these embedding layers account for more than 20% the total parameters of a 125M-parameter model.
Contrastingly, this proportion is considerably lower in larger language models. For example, the input and output embeddings only account for 3.7% of the total number of parameters in the LLaMA-7B modeland a mere 0.7% in the LLaMA-70B model. This disparity might elucidate why embedding sharing was initially proposed and implemented in OPT modelsbut was subsequently disregarded in recent designs of LLMs.In the development of sub-billion scale language models, we revisit the concept of input-output embedding sharing. The input embedding in LLM models maps the token id in the vocabulary to the corresponding token embedding and has a dimension of. Conversely, the output fully-connected layer maps the embedding dimension back to the logits prediction across the vocabulary, with weight size of. By sharing the embedding, we reuse the input embedding weights as the output fully connected layer weights, resulting in a more efficient and compact model architecture.We experiment on a 30-layer 125M model. In Table1, we demonstrate that sharing the input and output embeddings reduces the number of parameters by 16M, approximately 11.8% of total parameters with a 0.2 points drop in average accuracy. The marginal accuracy drop can be readily restored by reallocating the saved parameters to add more layers. Increasing the depth to 32 layers produces a 0.4 points accuracy gain while still maintaining 10M fewer parameters compared to the original 135M model. Similar results are also observed in 350M models.
These findings further suggest that embedding sharing is a valuable technique for maximizing weight utilization and optimizing model performance given limited model storage budget.2.2.4Number of Heads and KV HeadsFigure 5:Ablation study on number of heads and kv-heads. Here, ratio denotes the number of heads divided by number of kv-heads. Averaged accuracy on zero-shot reasoning tasks is reported.We now investigate the optimal head size for small transformer models. The trade-off between more semantics per head dimension and more non-linear combinations of multiple heads is a key consideration in choosing the head size. In addition, most previous studies have typically used an identical number of key-value heads to query heads in sub-billion parameter language models. Instead, we found that grouped query attention, which is initially designed for reducing key-value cache size in LLMs, that can also effectively reduce redundancy in key-value heads in small LMs. Grouped query attention can be viewed as another form of weight-sharing for weight re-utilization, where the number of key-value heads isthat of query heads, and the kv head is repeatedtimes in computing attention scores and output together with the query. Here,denotes a positive integer that the number of query heads are divisible by.To establish a solid foundation for a state-of-the-art small transformer model, we conducted experiments to determine the desirable head size on 125M and 350M models. Results in Figure5show that using 16 query heads produces the best results. Additionally, reducing the number of kv-heads from 16 to 4 resulted in comparable accuracy for the 125M model and only 0.2 points accuracy drop in the 350M model with almost 10% model size reduction. These results serve as a guideline in our model architecture design. By adopting the grouped query attention (GQA)  and meanwhile increasing the embedding dimension to maintain the model size, the accuracy of 125M further increases 0.4 points, indicating GQA as a favorable method to further squeeze out small model’s potential.In summary, we tested out four state-of-the-art techniques beneficial to small model designs, including FFN with SwiGLU, deep and thin architecture, embedding sharing, and grouped query attention. Combining these techniques, we build a strong baseline small LLM and we name itMobileLLM.2.3Layer SharingTable 2:Ablation study of layer-sharing strategy on zero-shot common sense reasoning tasks.ModelSharing methodARC-eARC-cBoolQPIQASIQAHellaSwagOBQAWinoGrandeAvg.125Mbaseline41.625.761.162.443.134.436.951.644.6Immediate block-wise share43.927.961.564.341.535.535.150.245.0Repeat-all-over share43.627.160.763.442.635.536.951.745.2Reverse share43.826.058.962.942.235.236.852.244.8\hdashline[0.8pt/1pt]350Mbaseline50.830.662.368.643.545.143.852.449.6Immediate block-wise share51.530.859.668.243.947.744.755.050.2Repeat-all-over share53.533.061.269.443.248.342.254.650.7Reverse share50.732.261.068.843.847.443.153.850.1The findings in Section2.2.2on the impact of layer depth versus width suggest deeper layers are favorable for small transformer models. This motivates us to investigate layer sharing as a strategy to increase the number of hidden layers without additional model storage cost. This approach is particularly helpful in on-device scenarios where model size is a major constraint.Surprisingly, the experimental findings show that accuracy enhancement can be achieved by simply replicating transformer blocks, without necessitating architectural modifications or an enlargement of the model size.
We further examined three different weight sharing strategies, illustrated in Figure6. Results in Table2.3indicate that the repeat-over layer-sharing strategy produces the best performance among immediate block-wise repeat, repeat all-over, and reverse sharing strategies. However, considering the hardware memory hierarchy (Figure2) , the SRAM for computing is typically limited to around 20MB. This capacity is usually only sufficient to hold a single transformer block. Therefore, placing shared weights in the cache and computing twice immediately can avoid the need to transfer weights between the SRAM and DRAM, resulting in improved overall execution speed for auto-regressive inference. Consequently, we have opted for the immediate block-wise sharing strategy in our model design. We denote the proposed model with layer sharing asMobileLLM-LS.Figure 6:(a)  Baseline model without layer sharing; (b)  Immediate block-wise sharing; (c)  Repeat-all-over sharing; (d)  Reverse sharing. A transformer block contains the multi-head self-attention (MHSA)  and the feed-forward network (FFN) . While repeat-all-over sharing has slightly higher performance, immediate block-wise sharing best utilize the cache because the shared weights can stay in the cache and be immediately computed twice.3Experiments3.1Experimental SettingsWe trainMobileLLMfrom scratch using Adam optimizerwith weight decay of 0.1. The experiments are conducted using 32 A100 GPUs, with a batch size of 32 on each GPU. The initial learning rate is set to 2e-3 and follows a cosine learning-rate decay strategy. We perform quick exploration experiments with 120k iterations on 0.25T tokens and train the best models reported in Tables3.2and3.2with 480k iterations on 1T tokens.3.2Main ResultsTable 3:Zero-shot performance on Common Sense Reasoning tasks.MobileLLMdenotes the proposed baseline model andMobileLLM-LSis integrated with layer sharing with the #layer counting layers with distinct weights.Model#Layers#ParamsARC-eARC-cBoolQPIQASIQAHellaSwagOBQAWinoGrandeAvg.Cerebras-GPT-111M10111M35.820.262.058.039.826.729.048.840.0LaMini-GPT-124M12124M43.626.051.862.742.130.229.649.241.9Galactica-125M12125M44.026.254.955.438.929.628.249.640.9OPT-125M12125M41.325.257.562.041.931.131.250.842.6GPT-neo-125M12125M40.724.861.362.541.929.731.650.742.9Pythia-160M12162M40.025.359.562.041.529.931.250.942.5RWKV-169M12169M42.525.359.163.940.731.933.851.543.6MobileLLM-125M30125M43.927.160.265.342.438.939.553.146.3MobileLLM-LS-125M30125M45.828.760.465.742.939.541.152.147.0\hdashline[0.8pt/1pt]
Cerebras-GPT-256M14256M37.923.260.361.440.628.331.850.541.8OPT-350M24331M41.925.754.064.842.636.233.352.443.9RWKV-430M24430M48.932.053.468.143.640.637.851.647.0Pythia-410M24405M47.130.355.367.243.140.136.253.446.6BLOOM-560M24559M43.727.553.765.142.536.532.652.244.2Cerebras-GPT-590M18590M42.624.957.762.840.932.033.249.743.0MobileLLM-350M32345M53.833.562.468.644.749.640.057.651.3MobileLLM-LS-350M32345M54.432.562.869.844.150.645.857.252.1Table 4:Performance on Trivia QA and RACE datasets for question answering and reading comprehension tasks.TQA(F1 score) RACE(Acc) Model1-shot5-shot64-shotmiddlehighCerebras-GPT-111M1.93.84.429.224.3OPT-125M8.79.68.234.727.5GPT-Neo-125M8.07.95.034.727.0Pythia-160M2.11.42.130.225.1MobileLLM-125M13.914.312.539.728.9MobileLLM-LS-125M14.214.814.640.729.6\hdashline[0.8pt/1pt]
Cerebras-GPT-256M5.26.83.331.726.2OPT-350M11.012.310.437.128.0Pythia-410M12.413.812.839.129.7BLOOM-560M8.88.95.337.628.2Cerebras-GPT-590M6.49.14.934.627.4MobileLLM-350M22.023.924.245.633.8MobileLLM-LS-350M21.422.522.647.333.7We compare the final performance on zero-shot common sense reasoning tasks, question answering and reading comprehension tasks. The results of baseline methods were evaluated using their open-source Hugging Face models to ensure consistent evaluation procedures.Zero-shot Common Sense ReasoningTable3.2presents a comparison between our proposed model,MobileLLM, and state-of-the-art sub-billion parameter models, including the early open-sourced LLMs, OPT, BLOOM, and recent releases such as Galactica, Cerebras, GPT-neoas well as the LLM analyzing suite Pythiaand transformer variants RWKVon zero-shot common sense reasoning tasks.
For the 125M model size,MobileLLMfavorably outperforms previous models such as OPT, GPT-Neo, and Calactica at the same model size by a significant margin. Additionally,MobileLLM-125M achieves 3.8 points and 2.7 points higher accuracy than Pythia-160M and RWKV-169M while being 22% and 26% smaller, respectively. Furthermore, incorporating layer-sharing inMobileLLM-LS-125M results in an additional 0.7 points improvement in accuracy. It is noteworthy thatMobileLLM-LS-125M achieves comparable or even higher results than most previous 350M models.
In the 350M model size category,MobileLLMsurpasses previous state-of-the-art models by more than 4 points with comparable or smaller model size.Question Answering and Reading ComprehensionWe evaluate pre-trained models on the TQA question answering benchmarkand RACE reading comprehension benchmark. We follow the evalutation setup fromand report the results in Table3.2. Comparing models of 125M size,MobileLLM-125M demonstrates a noteworthy improvement of over 4.3 points on the TQA benchmark contrast to its predecessor. Moreover, theMobileLLM-350M model exhibits a substantial performance increase of approximately 10 points compared to other 350M-sized models. For the reading comprehension tasks,MobileLLMmodel family also exhibit significant higher scores than preceding sub-billion parameter models.3.3Downstream TasksTo validate the effectiveness of sub-billion scale models for on-device applications, we assess their performance in two crucial on-device tasks: Chat and API calling.3.3.1ChatWe fine-tuneMobileLLMmodels, as well as previous state-of-the-art (SoTA)  models sourced from HuggingFace checkpoints for chat-based tasks and evaluate them under identical settings to ensure consistency. We evaluate on two benchmarks: AlpacaEval, a single-run chat benchmark, and MT-Bench, a multi-run chat benchmark. The results presented in Table3.3.1showcases thatMobileLLMmodels significantly outperforms previous state-of-the-art sub-billion scale models, surpassing even models with 1 billion parameters. Notably,MobileLLM-LS-350M achieves a remarkable win rate of 48.2% when compared to the baseline GPT-3 model (text-davinci-001) . Considering the self-win rate of GPT-3 is 50%, is noteworthy thatMobileLLM-LS-350M obtains comparable chat performance as this baseline model. Visualizations of chat examples in the appendix also underscore the impressive quality of responses generated byMobileLLMmodels.Table 5:Benchmark results on AlpacaEval (Evaluator: GPT-4; Reference model: text-davinci-001)  and MT-Bench.ModelMT-BenchAlpaca Evalnumber of parameters < 200M\hdashline[0.8pt/1pt]
OPT-125M1.213.91GPT-Neo-125M1.061.01Pythia-160M1.010.63MobileLLM-125M2.3324.07MobileLLM-LS-125M2.5223.79200M < number of parameters < 1B\hdashline[0.8pt/1pt]
OPT-350M1.376.80Pythia-410M1.6213.87BLOOM-560M1.7310.29MobileLLM-350M3.2847.08MobileLLM-LS-350M3.1648.20number of parameters > 1B\hdashline[0.8pt/1pt]
Pythia-1B1.7016.62BLOOM-1.1B2.3719.90Falcon-1.3B2.5430.38OPT-1.3B2.2438.843.3.2API CallingAPI calling is a common on-device application, particularly in collaboration with audio-to-text models for assistant functionalities. Leveraging LLMs for API calling involves the conversion of natural language inputs into JSON configurations to invoke corresponding APIs666https://platform.openai.com/docs/guides/function-calling. For instance, given the input"Help me set an alarm at 7:30 AM"the model outputs {API: "alarm(time="7:30 am") "}. Additionally, the model generates an agent response:"Sure! Your alarm is set to 7:30 AM."To adapt LLMs for this task, we create a synthetic dataset with 5000 training samples and 2500 testing samples. Each sample involves 8 conversation turns on average. Detailed examples of this dataset are provided in appendix. The pre-trained models undergoes fine-tuning on training set for 4 epochs, utilizing the Adam optimizer with a linear-decay learning rate starting at 2e-5 and a weight decay of 0.01.Table3.3.2shows thatMobileLLM-350M demonstrates comparable intent and structure exact match scores to LLaMA-v2 7B, where high intent scores indicate correct prediction of the API user intends to call, while structural exact match scores reflects the proficiency in predicting content within API functions. Despite lower Rouge scores inMobileLLM-350M compared to 7B models, it is crucial to note that API calling prioritize correct API invocation. The results suggest that certain common scenarios in on-device applications are not particularly challenging, and smaller models likeMobileLLM-350M can adeptly handle it.Table 6:API calling evaluation score. EM/EMmeasures the exact match in API calling. R1/RL refers to Rouge-1/-L score measuring the quality of agent response.ModelEMEMR1RL\hdashline[0.8pt/1pt]
OPT-350M56.138.637.135.3Pythia-410M62.244.743.141.1BLOOM-560M64.737.936.934.6MobileLLM-350M65.348.846.844.6\hdashline[0.8pt/1pt]
LLaMA-v2 7B62.850.956.554.33.4Compatibility with QuantizationFigure 7:Comparison between BFloat16 model and 8-bit weight 8-bit activation post-training quantized model.We further conduct per-token min-max post-training quantization (PTQ)  experiments on bothMobileLLMandMobileLLM-LSmodels with 125M and 350M model size trained on 0.25T tokens. Figure7shows that employing W8A8 PTQ yields a modest accuracy reduction of less than 0.5 points and remains compatible with layer-sharing.3.5Knowledge DistillationSo far, we trained compact models from scratch using next tokens as hard labels.
We explored Knowledge Distillation (KD)  of 125M and 350M models with LLAMA-v2 7B as a teacher.
Unfortunately KD increases training time (slowdown of)  and exhibits comparable or inferior accuracy to label-based training (details in appendix) .3.6On-device ProfilingTable 7:Latency analysis ofMobileLLM-125M (30 layers) ,MobileLLM-LS-125M (230 layers, adjacent blocks sharing weights) , and a 60-layer non-shared weight model, with consistent configurations in all other aspects.LoadInitExecute\hdashline[0.8pt/1pt]MobileLLM39.2 ms1361.7 ms15.6 msMobileLLM-LS43.6 ms1388.2 ms16.0 ms60-layer non-shared68.6 ms3347.7 ms29.0 msWe measure the latency forMobileLLM-125M andMobileLLM-LS-125M FP16 models via ExecuTorch777https://pytorch.org/executorch-overviewon iPhone 13 (iOS 17.2.1) , with Metal Performance Shaders (MPS)  backend888https://pytorch.org/executorch/stable/build-run-mps.html. Model loading, initialization and execution time are reported in Table3.6. Specifically, execution time is averaged over 50 iterations.Results in Table3.6reflects that through weight sharing and doubling the number of layers,MobileLLM-LSincurs only a 2.2% increase in loading and initialization time compared toMobileLLM, attributable to their similar model sizes. Execution time also experiences a mere 2.6% overhead, benefitting from data locality. In contrast, a model with doubled number of layers without weight sharing exhibits a substantial 143% rise in loading and initialization time and an 86% increase in execution time.4Related WorkThe excellent performance of LLMs has fostered its wide applications.
Considering computational cost and energy consumption of LLMs, an new stream of research direction have emerged to downsize LLMs to enable on-device inference. These methods include:Model Compression.Numerous model compression methods are developed for LLMs, including pruning, sparsity, and quantization. Our research is complementary to these techniques. As also substantiated in Section3.4, our methodology is compatible with quantization.Small Model Design.A limited number of studies have explored compact model architectures, such as TinyLLaMA. However, even the smallest TinyLLaMA exceeds 1 billion parameters, making them still prohibitive for many on-device applications. Some research proposes large model architectures alongside their smaller LLM variants in a model familyor a analytical suite containing small LLM variants. However, these models are not optimized under the constraint of sub-billion parameters and therefore may not be optimal.Neural Architecture Search.NAS has garnered substantial attention in the realm of convolutional neural networks, particularly in the context of vision tasks. In contrast, within the transformer domain, the prevailing consensus posits that the model architecture exerts minimal influence on accuracy, provided the total number of parameters remains consistent.
Only a limited number of studies have developed NAS algorithm for language transformer, targeting at BERT models. Our current investigation, focusing on the interplay between depth and width, can be conceptualized as a meticulous grid search within the depth space. The outcomes of that study challenge the prevalent orthodoxy surrounding scaling laws, proposing that deep and thin architectures demonstrate higher performance for compact LLMs.Weight Sharing.Weight sharing is an intuitive strategy for optimizing model weight utilization within fixed parameter constraints. While the OPT familyand subsequent worksleverage weight sharing between input and output embeddings, limited research has explored weight sharing for intermediate layers in transformers. Prior efforts often entail specialized designs for shared layers. In contrast, our contribution highlights a more straightforward yet effective way of simply repeating transformer blocks, yielding improved accuracy with a fixed model size and minimal latency increase.Efficient Attention and Implementation.In the realm of efficient transformer design, much research has focused on optimizing attention computation through methods like low-rank approximationand sparse attention. Another line of work explores hardware scheduling and weight movement, exemplified by works such as FlashAttentionand FlexGen. In contrast, our primary goal is to optimize model size without introducing new attention computation or efficient hardware implementation methods.5ConclusionThis study focuses on optimizing sub-billion scale models for on-device applications. Our findings indicate that, for smaller models, prioritizing depth over width enhances model performance. Furthermore, by leveraging advanced weight-sharing techniques, including embedding sharing, grouped query attention, and block-wise weight sharing, we achieve significant enhancements in weight utilization within storage-constrained scenarios. The resulting models denoted asMobileLLMexhibit substantial advancements in zero-shot commonsense reasoning, question answering, and reading comprehension tasks compared to previous SoTA methods. Last but not least, we demonstrate the effectiveness of the fine-tunedMobileLLMmodels in two prevalent on-device use cases: chat and API calling, underscoring their adeptness in handling such tasks.AcknowledgmentWe thank Hansong Zhang for his valuable contribution in setting up latency measurement environment on iOS, and the full support from PyTorch Edge team.Broader ImpactThis paper advocates for the adoption of sub-billion scale large language models in on-device applications, aiming to mitigate energy consumption during LLM inference. The proposed approach is promising in alleviating computational costs associated with LLM deployment.ReferencesAinslie et al. (2023) Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y., Lebron, F., and Sanghai, S.GQA: Training generalized multi-query transformer models from multi-head checkpoints.InEMNLP, 2023.Biderman et al. (2023) Biderman, S., Schoelkopf, H., Anthony, Q. G., Bradley, H., O’Brien, K., Hallahan, E., Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., et al.Pythia: A suite for analyzing large language models across training and scaling.InInternational Conference on Machine Learning, pp.  2397–2430. PMLR, 2023.Bisk et al. (2020) Bisk, Y., Zellers, R., Gao, J., Choi, Y., et al.Piqa: Reasoning about physical commonsense in natural language.InProceedings of the AAAI conference on artificial intelligence, volume 34, pp.  7432–7439, 2020.Black et al. (2022) Black, S., Biderman, S., Hallahan, E., Anthony, Q., Gao, L., Golding, L., He, H., Leahy, C., McDonell, K., Phang, J., et al.Gpt-neox-20b: An open-source autoregressive language model.arXiv preprint arXiv:2204.06745, 2022.Chowdhery et al. (2023) Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al.Palm: Scaling language modeling with pathways.Journal of Machine Learning Research, 24(240) :1–113, 2023.Clark et al. (2019) Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., and Toutanova, K.Boolq: Exploring the surprising difficulty of natural yes/no questions.arXiv preprint arXiv:1905.10044, 2019.Clark et al. (2018) Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O.Think you have solved question answering? try arc, the ai2 reasoning challenge.arXiv preprint arXiv:1803.05457, 2018.Dao et al. (2022) Dao, T., Fu, D., Ermon, S., Rudra, A., and Ré, C.Flashattention: Fast and memory-efficient exact attention with io-awareness.Advances in Neural Information Processing Systems, 35:16344–16359, 2022.Dauphin et al. (2017) Dauphin, Y. N., Fan, A., Auli, M., and Grangier, D.Language modeling with gated convolutional networks.InInternational conference on machine learning, pp.  933–941. PMLR, 2017.Dettmers et al. (2022) Dettmers, T., Lewis, M., Belkada, Y., and Zettlemoyer, L.Llm. int8 () : 8-bit matrix multiplication for transformers at scale.arXiv preprint arXiv:2208.07339, 2022.Dey et al. (2023) Dey, N., Gosal, G., Khachane, H., Marshall, W., Pathria, R., Tom, M., Hestness, J., et al.Cerebras-gpt: Open compute-optimal language models trained on the cerebras wafer-scale cluster.arXiv preprint arXiv:2304.03208, 2023.Frantar & Alistarh (2023) Frantar, E. and Alistarh, D.Sparsegpt: Massive language models can be accurately pruned in one-shot.InInternational Conference on Machine Learning, pp.  10323–10337. PMLR, 2023.Frantar et al. (2022) Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D.Gptq: Accurate post-training quantization for generative pre-trained transformers.arXiv preprint arXiv:2210.17323, 2022.Ganesan et al. (2021) Ganesan, V., Ramesh, G., and Kumar, P.Supershaper: Task-agnostic super pre-training of bert models with variable hidden dimensions.arXiv preprint arXiv:2110.04711, 2021.Google (2023) Google.Pixel 8 pro tech specs.https://store.google.com/gb/product/pixel_8_pro_specs, 2023.Guo et al. (2020) Guo, Z., Zhang, X., Mu, H., Heng, W., Liu, Z., Wei, Y., and Sun, J.Single path one-shot neural architecture search with uniform sampling.InComputer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XVI 16, pp.  544–560. Springer, 2020.Han et al. (2016) Han, S., Liu, X., Mao, H., Pu, J., Pedram, A., Horowitz, M. A., and Dally, W. J.Eie: Efficient inference engine on compressed deep neural network.ACM SIGARCH Computer Architecture News, 44(3) :243–254, 2016.Hinton et al. (2015) Hinton, G., Vinyals, O., Dean, J., et al.Distilling the knowledge in a neural network.arXiv preprint arXiv:1503.02531, 2(7) , 2015.Hristov (2022) Hristov, V.A16 bionic explained: what’s new in apple’s pro-grade mobile chip?https://www.phonearena.com/news/A16-Bionic-explained-whats-new_id142438, 2022.Jawahar et al. (2023) Jawahar, G., Yang, H., Xiong, Y., Liu, Z., Wang, D., Sun, F., Li, M., Pappu, A., Oguz, B., Abdul-Mageed, M., et al.Mixture-of-supernets: Improving weight-sharing supernet training with architecture-routed mixture-of-experts.arXiv preprint arXiv:2306.04845, 2023.Joshi et al. (2017) Joshi, M., Choi, E., Weld, D. S., and Zettlemoyer, L.Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension.arXiv preprint arXiv:1705.03551, 2017.Kaplan et al. (2020) Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D.Scaling laws for neural language models.arXiv preprint arXiv:2001.08361, 2020.Katharopoulos et al. (2020) Katharopoulos, A., Vyas, A., Pappas, N., and Fleuret, F.Transformers are rnns: Fast autoregressive transformers with linear attention.InInternational conference on machine learning, pp.  5156–5165. PMLR, 2020.Kim et al. (2023) Kim, S., Hooper, C., Gholami, A., Dong, Z., Li, X., Shen, S., Mahoney, M. W., and Keutzer, K.Squeezellm: Dense-and-sparse quantization.arXiv preprint arXiv:2306.07629, 2023.Kingma & Ba (2014) Kingma, D. P. and Ba, J.Adam: A method for stochastic optimization.arXiv preprint arXiv:1412.6980, 2014.Kitaev et al. (2020) Kitaev, N., Kaiser, Ł., and Levskaya, A.Reformer: The efficient transformer.arXiv preprint arXiv:2001.04451, 2020.Lai et al. (2017) Lai, G., Xie, Q., Liu, H., Yang, Y., and Hovy, E.Race: Large-scale reading comprehension dataset from examinations.arXiv preprint arXiv:1704.04683, 2017.Li et al. (2023) Li, X., Zhang, T., Dubois, Y., Taori, R., Gulrajani, I., Guestrin, C., Liang, P., and Hashimoto, T. B.Alpacaeval: An automatic evaluator of instruction-following models.https://github.com/tatsu-lab/alpaca_eval, 2023.Liu et al. (2023a) Liu, J., Gong, R., Wei, X., Dong, Z., Cai, J., and Zhuang, B.Qllm: Accurate and efficient low-bitwidth quantization for large language models.arXiv preprint arXiv:2310.08041, 2023a.Liu et al. (2023b) Liu, S.-y., Liu, Z., Huang, X., Dong, P., and Cheng, K.-T.Llm-fp4: 4-bit floating-point quantized transformers.arXiv preprint arXiv:2310.16836, 2023b.Liu et al. (2023c) Liu, Z., Oguz, B., Zhao, C., Chang, E., Stock, P., Mehdad, Y., Shi, Y., Krishnamoorthi, R., and Chandra, V.Llm-qat: Data-free quantization aware training for large language models.arXiv preprint arXiv:2305.17888, 2023c.Malladi et al. (2012) Malladi, K. T., Lee, B. C., Nothaft, F. A., Kozyrakis, C., Periyathambi, K., and Horowitz, M.Towards energy-proportional datacenter memory with mobile dram.ACM SIGARCH Computer Architecture News, 40(3) :37–48, 2012.Mihaylov et al. (2018) Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A.Can a suit of armor conduct electricity? a new dataset for open book question answering.arXiv preprint arXiv:1809.02789, 2018.Peng et al. (2023) Peng, B., Alcaide, E., Anthony, Q., Albalak, A., Arcadinho, S., Cao, H., Cheng, X., Chung, M., Grella, M., GV, K. K., et al.Rwkv: Reinventing rnns for the transformer era.arXiv preprint arXiv:2305.13048, 2023.Reid et al. (2021) Reid, M., Marrese-Taylor, E., and Matsuo, Y.Subformer: Exploring weight sharing for parameter efficiency in generative transformers.arXiv preprint arXiv:2101.00234, 2021.Roy et al. (2021) Roy, A., Saffar, M., Vaswani, A., and Grangier, D.Efficient content-based sparse attention with routing transformers.Transactions of the Association for Computational Linguistics, 9:53–68, 2021.Sakaguchi et al. (2021) Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y.Winogrande: An adversarial winograd schema challenge at scale.Communications of the ACM, 64(9) :99–106, 2021.Sap et al. (2019) Sap, M., Rashkin, H., Chen, D., LeBras, R., and Choi, Y.Socialiqa: Commonsense reasoning about social interactions.arXiv preprint arXiv:1904.09728, 2019.Scao et al. (2022) Scao, T. L., Fan, A., Akiki, C., Pavlick, E., Ilić, S., Hesslow, D., Castagné, R., Luccioni, A. S., Yvon, F., Gallé, M., et al.Bloom: A 176b-parameter open-access multilingual language model.arXiv preprint arXiv:2211.05100, 2022.Shen et al. (2022) Shen, Z., Liu, Z., and Xing, E.Sliced recursive transformer.InEuropean Conference on Computer Vision, pp.  727–744. Springer, 2022.Sheng et al. (2023) Sheng, Y., Zheng, L., Yuan, B., Li, Z., Ryabinin, M., Chen, B., Liang, P., Ré, C., Stoica, I., and Zhang, C.Flexgen: High-throughput generative inference of large language models with a single gpu.InInternational Conference on Machine Learning, pp.  31094–31116. PMLR, 2023.Sun et al. (2023) Sun, M., Liu, Z., Bair, A., and Kolter, J. Z.A simple and effective pruning approach for large language models.arXiv preprint arXiv:2306.11695, 2023.Tan & Le (2019) Tan, M. and Le, Q.Efficientnet: Rethinking model scaling for convolutional neural networks.InInternational conference on machine learning, pp.  6105–6114. PMLR, 2019.Taylor et al. (2022) Taylor, R., Kardas, M., Cucurull, G., Scialom, T., Hartshorn, A., Saravia, E., Poulton, A., Kerkez, V., and Stojnic, R.Galactica: A large language model for science.arXiv preprint arXiv:2211.09085, 2022.Timiryasov & Tastet (2023) Timiryasov, I. and Tastet, J.-L.Baby llama: knowledge distillation from an ensemble of teachers trained on a small dataset with no performance penalty.arXiv preprint arXiv:2308.02019, 2023.Touvron et al. (2023a) Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al.Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023a.Touvron et al. (2023b) Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Ferrer, C. C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshorn, A., Hosseini, S., Hou, R., Inan, H., Kardas, M., Kerkez, V., Khabsa, M., Kloumann, I., Korenev, A., Koura, P. S., Lachaux, M.-A., Lavril, T., Lee, J., Liskovich, D., Lu, Y., Mao, Y., Martinet, X., Mihaylov, T., Mishra, P., Molybog, I., Nie, Y., Poulton, A., Reizenstein, J., Rungta, R., Saladi, K., Schelten, A., Silva, R., Smith, E. M., Subramanian, R., Tan, X. E., Tang, B., Taylor, R., Williams, A., Kuan, J. X., Xu, P., Yan, Z., Zarov, I., Zhang, Y., Fan, A., Kambadur, M., Narang, S., Rodriguez, A., Stojnic, R., Edunov, S., and Scialom, T.Llama 2: Open foundation and fine-tuned chat models, 2023b.Wang et al. (2020) Wang, S., Li, B. Z., Khabsa, M., Fang, H., and Ma, H.Linformer: Self-attention with linear complexity.arXiv preprint arXiv:2006.04768, 2020.Wu et al. (2019) Wu, B., Dai, X., Zhang, P., Wang, Y., Sun, F., Wu, Y., Tian, Y., Vajda, P., Jia, Y., and Keutzer, K.Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search.InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.  10734–10742, 2019.Xia et al. (2023a) Xia, H., Zheng, Z., Li, Y., Zhuang, D., Zhou, Z., Qiu, X., Li, Y., Lin, W., and Song, S. L.Flash-llm: Enabling cost-effective and highly-efficient large generative model inference with unstructured sparsity.arXiv preprint arXiv:2309.10285, 2023a.Xia et al. (2023b) Xia, M., Gao, T., Zeng, Z., and Chen, D.Sheared llama: Accelerating language model pre-training via structured pruning.arXiv preprint arXiv:2310.06694, 2023b.Xiao et al. (2023) Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., and Han, S.Smoothquant: Accurate and efficient post-training quantization for large language models.InInternational Conference on Machine Learning, pp.  38087–38099. PMLR, 2023.Xiong et al. (2021) Xiong, Y., Zeng, Z., Chakraborty, R., Tan, M., Fung, G., Li, Y., and Singh, V.Nyströmformer: A nyström-based algorithm for approximating self-attention.InProceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp.  14138–14148, 2021.Xu et al. (2021) Xu, J., Tan, X., Luo, R., Song, K., Li, J., Qin, T., and Liu, T.-Y.Nas-bert: task-agnostic and adaptive-size bert compression with neural architecture search.InProceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp.  1933–1943, 2021.Yao et al. (2022) Yao, Z., Yazdani Aminabadi, R., Zhang, M., Wu, X., Li, C., and He, Y.Zeroquant: Efficient and affordable post-training quantization for large-scale transformers.Advances in Neural Information Processing Systems, 35:27168–27183, 2022.Zellers et al. (2019) Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y.Hellaswag: Can a machine really finish your sentence?arXiv preprint arXiv:1905.07830, 2019.Zhang et al. (2022) Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al.Opt: Open pre-trained transformer language models.arXiv preprint arXiv:2205.01068, 2022.Zheng et al. (2023) Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., et al.Judging llm-as-a-judge with mt-bench and chatbot arena.arXiv preprint arXiv:2306.05685, 2023.Zoph & Le (2016) Zoph, B. and Le, Q. V.Neural architecture search with reinforcement learning.arXiv preprint arXiv:1611.01578, 2016.AppendixAppendix AImpact of Each Design ChoiceThis section presents comprehensive tabulated results for theimproving sub-billion scale LLM designexperiments, at the model sizes of 125M and 350M. Looking at the results in TableA, transitioning from the traditional Feedforward Network ()  to SwiGLU yields a accuracy improvement of 1.3% for both model sizes. Further increasing the model depth enhances accuracy by 0.9%/1.1% for 125M/350M models, respectively. Then, introducing input and output embedding sharing achieves a parameter reduction of approximately 10%, while with only marginal accuracy drops of 0.2% for the 125M and 0.6% for 350M models. Additionally, following in findings in SectionC, we incorporate grouped query attention with head dimension equals to 64, and head number to be near 4to the kv head number, while increasing the embedding dimension to preserve model size. This modification further results in a performance boost of 0.4%/0.7% for 125M/350M models. Combining these techniques establishes a strong baseline network denoted asMobileLLM. Finally, the immediate block-wise weight-sharing technique contributes an additional accuracy gain of 1.1% for models trained on 0.25 trillion tokens, and resulting in the modelMobileLLM-LS. Final models includingMobileLLMandMobileLLM-LSare trained with 1 trillion tokens.Table 8:Ablation study on the impact of each design choice to the model accuracy on zero-shot common sense reasoning tasks. Corresponding to the bar chart in Figure3. Here, L, H,denotes number of layers, heads, kv-heads, respectively and dim denotes embedding dimension.TechniquesLHDim#Params(M) ARC-eARC-cBoolQPIQASIQAHellaSwagOBQAWinoGrandeAvg.125M\hdashline[0.8pt/1pt]
Baseline model121212768134.141.325.257.562.041.931.131.250.842.6+ SwiGLU in FFN121212768134.143.128.958.162.342.334.631.550.143.9+ Use deep-thin structure3088512135.043.626.158.062.542.636.537.551.544.8+ Embedding share3088512118.644.426.056.262.843.135.936.052.644.6+ Multi-query attention3093576124.645.527.758.364.641.936.435.450.445.0(Train on 1T token) 3093576124.643.927.160.265.342.438.939.553.146.3+ Layer sharing3093576124.644.427.061.565.143.037.637.852.046.1(Train on 1T token) 3093576124.645.828.760.465.742.939.541.152.147.0350M\hdashline[0.8pt/1pt]
Baseline model1520201280376.850.327.653.868.144.142.640.152.447.4+ SwiGLU in FFN1520201280386.749.230.659.167.744.343.241.054.248.7+ Use deep-thin structure321414896380.350.731.459.467.843.346.243.856.249.8+ Embedding share321414896351.649.932.060.367.943.247.038.954.849.2+ Multi-query attention32155960345.351.431.361.068.143.647.241.655.449.9(Train on 1T token) 32155960345.353.833.562.468.644.749.640.057.651.3+ Layer sharing32155960345.351.935.259.668.943.447.243.358.451.0(Train on 1T token) 32155960345.354.432.562.869.844.150.645.857.252.1Appendix BDepth vs WidthWe provide network depth versus width exploration results on zero-shot reasoning tasks in TableB, as well as results on question answering and reading comprehension tasks in TableB. The findings indicate that shallow architectures with fewer than 10 layers perform poorly in reasoning or handling comprehension tasks. Models with 10-20 layers exhibit improved accuracy, while further increasing depth continues to provide significant benefits across all three tasks. Notably, the optimal depth is found to be near 30 layers for sub-billion scale models.Table 9:Ablation study on depth versus width in architecture design, illustrated in Figure4(a) (b) . For compact models, prioritizing depth over width yields superior performance, assessed through zero-shot common sense reasoning tasks.#Layer#HeadsDim#Params(M) ARC-eARC-cBoolQPIQASIQAHellaSwagOBQAWinoGrandeAvg.4201280163.242.026.161.061.541.332.931.449.843.36161024142.642.225.551.062.641.633.532.150.042.3814896138.143.625.548.661.840.934.037.552.043.01212768134.143.128.958.162.342.334.631.550.143.91810640132.441.025.559.362.641.634.236.750.143.9249576132.443.627.059.763.842.036.233.752.244.8308512135.043.626.158.062.542.636.537.551.544.8427448134.643.326.157.863.341.736.335.952.044.5626384134.344.126.759.064.742.436.233.151.344.7\hdashline[0.8pt/1pt]
5322048388.047.928.862.265.242.838.139.052.647.110241536381.449.129.660.667.144.042.741.254.448.612221408379.949.931.659.168.242.644.043.753.849.115201280386.749.230.659.167.744.343.241.054.248.719181152376.350.830.956.369.143.845.239.654.548.824161024373.850.733.858.868.643.545.040.054.949.42815960371.151.733.257.667.942.946.037.753.948.93214896380.350.731.459.467.843.346.243.856.249.84612768374.749.730.559.568.145.144.943.455.549.66610640376.250.531.861.067.443.846.040.155.649.5Table 10:Ablation study on depth vs. width in architecture design on TQA and RACE datasets, depicted in Figure4(c-f) .TQA(F1 score) RACE(Acc) #Layer#HeadsDim#Params(M) 1-shot5-shot64-shotmiddlehigh4201280163.22.03.93.933.226.06161024142.63.54.94.629.225.5814896138.14.25.24.729.224.31212768134.15.16.16.335.627.31810640132.44.76.85.234.327.4249576132.45.97.27.335.328.6308512135.06.26.86.934.227.9427448134.66.17.27.234.428.6626384134.36.07.07.338.928.9\hdashline[0.8pt/1pt]
5322048388.05.67.88.638.129.010241536381.49.011.812.740.630.012221408379.911.712.913.342.431.515201280386.710.713.114.242.630.619181152376.311.313.313.542.632.424161024373.811.813.514.443.031.92815960371.112.513.614.942.832.33214896380.312.614.515.344.232.24612768374.713.015.415.644.731.46610640376.212.515.115.744.032.5Appendix CNumber of Heads and Key-Value HeadsWe provide detailed experimental results assessing the impact of the number of attention heads and key-value heads on zero-shot reasoning accuracy in TableC. Our study involves two baseline architectures: a 8-layer 125M model with a embedding dimension of 896, and a 15-layer 350M model with a embedding dimension of 1280. We conduct head size sweeps in {8, 16, 32}. The findings shown in TableCindicate that using 16 heads, with a head dimension close to 64, and 4 key-value heads, yields the best accuracy and memory trade-off. This setting serves as a guiding principle in our model architecture design.Table 11:Ablation study investigating the impact of the number of attention heads and key-value heads.Model#Heads#KV-Heads#Params(M) ARC-eARC-cBoolQPIQASIQAHellaSwagOBQAWinoGrandeAvg.3232138.142.126.958.462.242.133.836.752.444.33216131.741.226.457.762.542.333.334.252.943.8328128.542.627.361.161.941.932.235.052.044.2324126.843.126.859.862.741.432.534.451.144.0322126.039.826.759.459.442.031.332.652.943.0321125.641.024.359.160.841.231.435.452.043.1125M1616138.141.625.761.162.443.134.436.951.644.6# layers=8168131.742.426.460.763.441.933.534.751.544.3dim=896164128.542.525.662.362.441.833.035.954.544.7162126.841.725.356.961.742.032.932.654.543.5161126.040.426.361.863.241.732.034.050.443.788138.141.425.058.361.741.733.335.953.243.884131.743.328.258.361.842.833.830.953.044.082128.540.726.258.161.241.632.834.851.543.481126.842.524.859.462.342.032.036.351.343.8\hdashline[0.8pt/1pt]3232386.748.630.459.767.243.944.040.953.948.63216362.148.931.657.668.443.443.838.654.948.4328349.848.333.161.067.242.642.139.053.948.4324343.747.229.859.467.243.542.542.554.148.3322340.647.630.362.466.942.641.638.652.047.7321339.048.527.356.367.142.940.936.753.346.6350M1616386.750.830.662.368.643.545.143.852.449.6# layers=15168362.148.530.759.467.343.843.841.353.348.5dim=1280164349.849.930.660.069.243.544.241.855.849.4162343.749.328.455.067.342.742.640.354.547.5161340.649.229.358.867.443.542.139.952.847.988386.751.033.258.267.043.943.942.654.749.384362.150.031.360.266.042.743.740.953.748.682349.849.330.459.967.942.843.538.953.848.381343.748.027.761.567.143.142.537.354.747.8Appendix DLayer Sharing Number AblationTable 12:Ablation study on the impact of varying layer repetition numbers.Model SizeRepeat TimesARC-eARC-cBoolQPIQASIQAHellaSwagOBQAWinoGrandeAvg.125Mbaseline41.625.761.162.443.134.436.951.644.6block-wise share repeat243.927.961.564.341.535.535.150.245.0block-wise share repeat343.327.660.963.042.335.634.152.945.0block-wise share repeat442.026.662.364.441.536.235.853.945.3\hdashline[0.8pt/1pt]350Mbaseline50.830.662.368.643.545.143.852.449.6block-wise share repeat251.530.859.668.243.947.744.755.050.2block-wise share repeat349.632.057.368.443.947.040.656.249.4block-wise share repeat452.232.660.069.044.547.641.855.250.4We extended our investigation to determine the optimal number of layer repetitions. The experiment involved the 8-layer 125M model with a embedding dimension of 896 and the 15-layer 350M model with a embedding dimension of 1280. The results in TableDdemonstrate that when we double the layer number and with each two transformer blocks sharing weights, the accuracy is enhanced by 0.4-0.6%. However, as we further triple or quadruple the layer repetition times, this accuracy enhancement effect diminishes. Consequently, in our experiments, we adopt a configuration where every two blocks share weights, effectively doubling the total number of layers.Appendix ECompatibility with QuantizationThis section explores the compatibility of quantization with the proposed model architecture and layer sharing. We employ a straightforward per-token min-max quantization, quantizing both weight and activation to 8-bit using post-training quantization (PTQ) . Experiments are conducted on bothMobileLLMandMobileLLM-LSwith model size of 125M and 350M, trained on 0.25T tokens. Results in TableEdemonstrate that W8A8 PTQ results in a modest accuracy drop of within 0.5% and is compatible with the proposed layer sharing method.Table 13:Ablation study: 8-bit weight, 8-bit activation post-training quantization results on zero-shot common sense reasoning tasks. Quantized models achieve accuracy gap of within 0.5% compared to the full-precision BF16 counterpart.ModelPrecisionARC-eARC-cBoolQPIQASIQAHellaSwagOBQAWinoGrandeAvg.GapMobileLLM-125MBF1645.527.758.364.641.936.435.450.445.0–MobileLLM-125MW8A845.227.158.365.041.736.233.651.044.80.2\hdashline[0.8pt/1pt]MobileLLM-LS-125MBF1644.427.061.565.143.037.637.852.046.1–MobileLLM-LS-125MW8A844.027.560.964.643.137.737.751.045.80.3\hdashline[0.8pt/1pt]MobileLLM-350MBF1651.431.361.068.143.647.241.655.449.9–MobileLLM-350MW8A851.432.161.168.843.147.140.655.149.90.0\hdashline[0.8pt/1pt]MobileLLM-LS-350MBF1651.935.259.668.943.447.243.358.451.0–MobileLLM-LS-350MW8A851.333.859.569.143.747.243.057.050.60.4Appendix FKnowledge DistillationThe results of integrating knowledge distillation (KD) into small model pre-training are presented in TableF. LLaMA-v2 7B models serve as the teacher, and the KD loss is computed using cross-entropy between the logits from the large pre-trained teacher model (i.e., LLaMA-v2 7B)  and the small student network (i.e., 125M or 350M models) :(1) Here,denotes thesample in the current batch withtotal samples in the batch, andrepresents the number of classes, which, in our case, equals the size of the vocabulary.andare the teacher and student networks, respectively.The results in TableFindicate that adding KD loss is comparable or even lower than solely using the next token as labels. However, it’s noteworthy that the training time using KD isslower than training from scratch using labels. All models are trained on 32 A100 80G GPUs with a batch size of 32 for 120k iterations. Consequently, we opt to use labels in our experiments.Table 14:Ablation study on employing LLaMA-v2 7B teacher’s output as soft labels for knowledge distillation (KD) . Results indicate a slight degradation in performance when incorporating KD loss compared to only using hard labels.ModelTraining LossTraining TimeARC-eARC-cBoolQPIQASIQAHellaSwagOBQAWinoGrandeAvg.125M modelLabel29h43.128.958.162.342.334.631.550.143.9125M modelLabel + KD93h41.828.558.561.641.134.532.751.643.8\hdashline[0.8pt/1pt]
350M modelLabel42h50.231.856.967.744.345.840.855.549.1350M modelLabel + KD109h48.731.860.767.443.245.938.953.748.8Appendix GDatasets and BenchmarksMobileLLMis assessed on zero-shot common sense reasoning (BoolQ, PIQA, SIQA, HellaSwag, Winogrande, ARC, OBQA) , question answering (TriviaQA) , and reading comprehension (RACE)  tasks. Additionally, we evaluate our chat models on MT-Bench and AlpacaEval benchmarks. We also generated a API calling dataset to fine-tune and evaluate models for this particular task.G.1Zero-shot Common Sense Reasoning tasksBoolQis a reading comprehension dataset focused on naturally occurring yes/no questions. Each instance includes a question (Q) , an excerpt from a passage (P) , and an answer (A) , with an added explanation for enhanced clarity.PIQA, abbreviated for Physical Interaction: Question Answering, serves as a benchmark for evaluating and studying the capacity of natural language models in comprehending physical commonsense understanding.SIQA, abbreviated for Social Interaction Question Answering, is designed to measure the social and emotional intelligence of computational models through multiple-choice question answering.HellaSwagserves as a benchmark for physically situated commonsense natural language inference. It comprises four-way multiple-choice problems that are considered trivial for humans (>95% accuracy)  but pose a challenge for language models.WinoGrandeis a benchmark for commonsense reasoning. It consists of a set of 273 expert-crafted pronoun resolution problems, deliberately designed to be unsolvable for statistical models relying on selectional preferences or word associations.ARC, the AI2 Reasoning Challenge, is a compilation of 7787 natural science questions. It is divided into a Challenge Set and an Easy Set, with the Challenge Set exclusively comprising questions that were answered incorrectly by both a retrieval-based algorithm and a word co-occurrence algorithm.OBQAis a dataset consisting of approximately 6000 questions designed for open book question answering. The task involves integrating a corpus of provided science facts (open book)  with external broad common knowledge. Providing correct answers necessitates leveraging simple common knowledge beyond the core facts provided.G.2Question Answering TasksTriviaQAis a closed-book question answering benchmark. It encompasses over 650,000 question-answer evidence triples derived by combining 95,000 question-answer pairs authored by Trivia enthusiasts. Each question is supported by an average of six evidence documents.G.3Reading Comprehension TasksRACEis a large-scale reading comprehension dataset with more than 28,000 passages and nearly 100,000 questions. The dataset is collected from English examinations in China, which are designed for middle school and high school students.G.4Chat BenchmarksAlpacaEvalis a single-turn benchmark featuring 805 questions across various topics. In the evaluation of this paper, models generate responses and are scored by GPT-4, with the ultimate metric being the pairwise win-rate against a baseline model (text-davinci-001) .MT-Benchis a multi-turn benchmark comprising 160 questions spanning eight diverse knowledge domains. In this evaluation, models are tasked with answering an initial question and subsequently providing a second response to a predefined follow-up question. Each model response is rated by GPT-4 on a scale from 1 to 10, and the final score is determined by the mean over the two turns.G.5API Calling DatasetThe API calling dataset is generated by instructing a language model to simulate a conversation between a human and an agent. This process instructs the language model to explicitly mention tasks that should be performed using designated APIs throughout the simulated conversation.
Examples of a conversation turn in generated dataset are shown in the following boxes.{"role": "USER", "message": "Can you help me send a SMS message: We are looking forward to seeing you"},{"role": "PARSER", "message": "style_transfer(text = "We are looking forward to seeing you", style = ["sms"], contact = None) "},{"role": "AGENT", "message": "Sure! Here’s a possible SMS message: Hey [name], we are looking forward to seeing you! Wishing you all well."},{"role": "USER", "message": "How did the S&P 500 perform last month?"},{"role": "PARSER", "message": "get_stock_information(company_name = "S&P 500", ticker_symbol = "GSPC", date_time = "last month", financial_metrics = "price") "},{"role": "AGENT", "message": "The S&P 500 closed at $2,977.94 last month, with a monthly return of 2.5%."},{"role": "USER", "message": "Is the San Diego Zoo open during the winter months?"},{"role": "PARSER", "message": "get_local_information(point_of_interest = "San Diego Zoo", attributes = "hours of operation") "},{"role": "AGENT", "message": "The San Diego Zoo is open year-round, but the hours of operation vary depending on the time of year. During the winter months, the zoo is open from 9:00 AM to 5:00 PM."},{"role": "USER", "message": "What is the most recent news concerning aliens?"},{"role": "PARSER", "message": "news(query = "latest alien news") "},{"role": "AGENT", "message": "According to recent news articles, NASA has announced the discovery of a new planet outside of our solar system that has the potential to support alien life. The planet, named Kepler-186f, is located in the constellation Cygnus and is the first known Earth-sized planet to be found in the habitable zone of a star. While this news does not necessarily prove the existence of aliens, it does provide hope for the possibility of finding life beyond Earth. Additionally, a recent study published in the journal Nature has suggested that the universe may be teeming with alien civilizations, with estimates suggesting that there could be tens of billions of intelligent civilizations in the observable universe. However, it’s important to note that these findings are still being researched and debated in the scientific community."},{"role": "USER", "message": "Who won the last Super Bowl?"},{"role": "PARSER", "message": "get_sports_information(match = "", sport_type = "NFL", metric = "winner") "},{"role": "AGENT", "message": "The last Super Bowl was won by the Kansas City Chiefs."},Appendix HApproximate calculation of envisioned future GPU consumption#GPUs =(population size) (individual time999According to the statistics that YouTube users spending an average of 23 hours per month (source)  and Instagram users dedicating 12 hours monthly to the app (source) , we have reason to believe the interaction with LLM can easily surpass the cumulative engagement of these platforms in the future.) (approximating FLOPs/token for 220B GPT-4 model with one expert activated101010https://the-decoder.com/gpt-4-has-a-trillion-parameters/) (token/s) (h) (s)  / ((FLOPs/s, H100 computation capacity ) (h) (s) ) This assumes short sequence lengths (i.e.hundreds of tokens, not tens of thousands of tokens) , where the compute is dominated by linear layers.
For very long sequences, the self-attention computation would dominate, and this would necessitate a more complicated formula.Appendix IChat Generations fromMobileLLMWe show a several examples of chat output generated withMobileLLMwhile responding to various questions.Input:Are there any particular physical benefits to mindful walking, such as improved posture or increased physical fitness?Output(MobileLLM-125M) :Yes, there are several physical benefits to mindful walking. Here are a few:1. Improved Posture: Mindful walking can help improve your posture by promoting good alignment and reducing stress on the joints. This can lead to better posture and reduced back pain.2. Increased Flexibility: Mindful walking can also help increase flexibility in your body, which can reduce the risk of injury and improve overall health.3. Better Sleep: Mindful walking can help you fall asleep more easily and stay asleep longer, which can improve your sleep quality.4. Reduced Stress: Mindful walking can help reduce stress levels by promoting relaxation and reducing tension in the body.5. Enhanced Mental Health: Mindful walking has been shown to have positive effects on mental health, including reducing symptoms of depression and anxiety.Overall, mindful walking is an excellent way to improve your physical health and well-being. It’s a simple and effective way to get moving and enjoy the benefits it offers. If you’re interested in trying mindful walking, I would recommend starting with shorter walks and gradually increasing the duration over time. You may also want to consider incorporating mindfulness into your daily routine to further enhance your experience.In conclusion, mindful walking is a great way to improve your physical health and well-being. With its numerous physical benefits, it’s easy to see why it’s becoming increasingly popular among people looking for a fun and sustainable way to get moving. So go ahead and give it a try!Output(MobileLLM-350M) :Yes, there are several physical benefits of mindful walking. Firstly, it can improve your posture by reducing the amount of tension in your muscles and joints. Secondly, it can increase your physical fitness by improving your cardiovascular health, strengthening your core muscles, and increasing your flexibility. Additionally, mindful walking can help you to stay more focused on the present moment, which can lead to better sleep quality and reduced stress levels. Finally, it can also help you to develop a greater sense of awareness and appreciation for the world around you, which can enhance your overall well-being.Overall, mindful walking is a great way to improve your physical health and well-being, while also promoting relaxation and mindfulness. Whether you’re looking to reduce stress, improve your posture, or simply enjoy a peaceful walk, mindful walking can be a rewarding and fulfilling experience.If you’re interested in trying out mindful walking, there are many resources available that can guide you through the process. Some popular options include guided meditations, apps that offer mindful walking exercises, and classes that teach you how to practice mindful walking. With a little bit of effort and dedication, anyone can start incorporating mindful walking into their daily routine and reap the many benefits it offers.In conclusion, mindful walking is a simple yet powerful tool that can help you to improve your physical health, reduce stress, and cultivate a deeper sense of awareness and appreciation for the world around you. So why not give it a try and see what kind of benefits you can achieve? Your body and mind will thank you!Output(llama-v2-7b-chat) :Mindful walking can have a range of physical benefits, including:1. Improved posture: By focusing on the sensation of each step and the movement of the body, mindful walking can help improve posture and reduce slouching or hunching.2. Increased physical fitness: Mindful walking can help improve cardiovascular fitness, increase muscle strength and endurance, and boost overall physical fitness.3. Better balance and coordination: By paying attention to the body’s position in space and the movement of the legs and feet, mindful walking can help improve balance and coordination.4. Reduced stress and tension: Mindful walking can help reduce stress and tension in the body, which can lead to improved physical relaxation and well-being.5. Improved flexibility: By focusing on the movement of the body and the sensation of each step, mindful walking can help improve flexibility and range of motion.6. Boosted immune system: Studies have shown that mindfulness practices, including mindful walking, can help boost the immune system and reduce inflammation.7. Improved sleep: Mindful walking can help improve sleep quality and duration by reducing stress and promoting relaxation.
8. Reduced pain: Mindful walking can help reduce pain by focusing on the sensation of each step and the movement of the body, rather than the pain itself.9. Improved mood: Mindful walking can help improve mood and reduce symptoms of depression and anxiety by promoting relaxation and reducing stress.It’s important to note that the physical benefits of mindful walking may vary depending on the individual and the frequency and duration of the practice. However, with regular practice, mindful walking can have a range of positive effects on physical health and well-being.Input:How do I wrap a present neatly?Output(MobileLLM-125M) :1. Choose the right size gift: The size of the present you want to give will depend on your budget and personal preference. If you are looking for a small gift, consider buying a small box or card with a single-use plastic bag.2. Prepare the present: Make sure all the items in the present are clean and tidy. You can use a cleaning kit or a tissue paper to remove any dirt or debris from the present.3. Arrange the presents: Place the present in a place where it is easy to see and touch. Use a pencil to draw a line around the present and arrange it in a way that makes sense to you.4. Add decorations: Decorate the present with stickers, ribbons, or other decorative items. This will add some personality to the present and make it more visually appealing.5. Enjoy the present: Take time to enjoy the present and let yourself feel the warmth and comfort of the present. Smile at yourself and take a moment to appreciate the present.Remember, the most important thing is to have fun and enjoy the present! Happy gift-giving!If you’re feeling adventurous, try making a surprise gift for someone special by wrapping the present in a cute and colorful gift basket. Or, if you’re feeling adventurous, try making a gift for a friend who needs a little extra help with their homework. Whatever you choose, remember to be creative and have fun with your present!With these tips, you’ll be able to create a beautiful and thoughtful present that will last for years to come. Happy gift-giving!Output(MobileLLM-350M) :1. Start by wrapping the present in a decorative gift wrap or ribbon.2. Use a small amount of glue or tape to secure the gift wrap or ribbon to the surface of the present.3. Place the gift wrap or ribbon around the present, making sure it is centered and evenly distributed.4. Use a small amount of glue or tape to secure the gift wrap or ribbon to the surface of the present.5. Once the gift wrap or ribbon is securely attached, you can add any additional decorations or embellishments to the present.6. If you want to add a personal touch, you can write a message or a special message on the gift wrap or ribbon.7. Finally, enjoy your beautifully wrapped present!Remember, wrapping a gift is a fun and creative way to show your loved one how much you care. Have fun and enjoy the process!Output(llama-v2-7b-chat) :Wrapping a present neatly can be a bit of a challenge, but with a few simple tips and tricks, you can create a beautifully wrapped gift that your recipient will love. Here are some tips to help you wrap a present neatly:1. Use the right wrapping paper: Choose a wrapping paper that complements the gift and is the right size for the present. You can also use a patterned paper or a solid color, depending on your preference.2. Measure the gift: Before you start wrapping, measure the gift to make sure you have enough wrapping paper. You can also use a piece of scrap paper to measure the gift and mark the edges with a pencil.3. Use a flat surface: Make sure you have a flat surface to work on, such as a table or countertop. This will help you keep the wrapping paper flat and prevent it from wrinkling.4. Start with the edges: Begin by folding the edges of the wrapping paper over the gift, making sure they are even and smooth. You can use tape to secure the edges if needed.5. Roll the gift: Once you have the edges in place, roll the gift into the center of the wrapping paper. Make sure the gift is securely in place before moving on to the next step.6. Tuck in the edges: As you wrap the gift, tuck the edges of the wrapping paper under the gift, smoothing them out as you go. Make sure the edges are even and smooth, and use tape to secure them if needed.7. Add a ribbon or bow: Once you have the gift wrapped, add a ribbon or bow to the top. This will not only make the gift look more festive, but it will also help to keep the wrapping paper in place.8. Practice makes perfect: Wrapping a present neatly takes a bit of practice, so don’t be discouraged if it doesn’t come out perfectly the first time. Keep practicing, and you’ll get the hang of it in no time!By following these tips, you can create a beautifully wrapped present that your recipient will love. Happy wrapping!Input:What are the best exercises for beginners?Output(MobileLLM-125M) :1. Push-ups: This is a classic exercise that can be done anywhere, anytime. It’s easy to do and requires no equipment.2. Dumbbell Rows: These rows are great for building strength in your legs and core.3. Squats: Squats are a great way to build strength in your legs and core.4. Lunges: Lunges are a great way to improve balance and stability.5. Plank: A plank is a great way to build strength in your back and core.6. Jumping Jacks: Jumping jacks are a fun and effective way to build strength in your legs and core.7. Burpees: Burpees are a great way to build strength in your legs and core.8. Mountain Climbers: Mountain climbers are a great way to build strength in your legs and core.9. Deadlifts: Deadlifts are a great way to build strength in your legs and core.10. Tricep Dips: Tricep dips are a great way to build strength in your triceps.Remember, it’s important to start with simple exercises and gradually increase the intensity as you get stronger. Don’t push yourself too hard or you may find it difficult to maintain proper form. With consistent practice, you will see improvements in your strength and overall fitness.If you have any questions about exercises or healthy living, don’t hesitate to talk to a doctor or physical therapist. They can help you develop a personalized plan that works best for you.In conclusion, there are many exercises that can be done at home, but it’s important to start slowly and gradually increase the intensity over time. Remember to listen to your body and take breaks when needed. And most importantly, enjoy the process!Output(MobileLLM-350M) :Beginners should start with exercises that are easy to do and don’t require a lot of strength or endurance. Here are some exercises that are great for beginners:1. Push-ups: This is a great exercise for beginners because it’s a simple exercise that can be done anywhere. Start with a push-up on your knees and then lower yourself down to the ground.2. Squats: Squats are a great exercise for building leg strength and endurance. Start with a squat on your knees and then lower yourself down to the ground.3. Lunges: Lunges are a great exercise for building leg strength and endurance. Start with a lunge on your front foot and then lower yourself down to the ground.4. Plank: Planks are a great exercise for building core strength and stability. Start with a plank on your forearms and then lower yourself down to the ground.5. Bicycle crunches: Bicycle crunches are a great exercise for building upper body strength and stability. Start with a bicycle crunch on your upper back and then lower yourself down to the ground.6. Plank with leg lifts: This exercise is great for building leg strength and stability. Start with a plank on your forearms and then lift your legs off the ground and hold for a few seconds before lowering them back down.7. Jumping jacks: Jumping jacks are a great exercise for building cardiovascular endurance and strength. Start with a jumping jack on your feet and then jump up and spread your legs.Remember to start with a weight that is comfortable for you and gradually increase the weight as you get stronger. It’s also important to warm up before exercising and to stretch after your workout to prevent injury.In conclusion, beginners should start with exercises that are easy to do and don’t require a lot of strength or endurance. These exercises are great for building leg strength and endurance, and can be done anywhere. Remember to start with a weight that is comfortable for you and gradually increase the weight as you get stronger. Warming up and stretching after your workout are also important to prevent injury.Following these exercises can help you build strength, endurance, and improve your overall fitness level. Remember to always consult with a healthcare professional before starting any new exercise program.Output(llama-v2-7b-chat) :Exercise is an essential part of a healthy lifestyle, but for beginners, it can be intimidating and overwhelming. Here are some of the best exercises for beginners:1. Bodyweight exercises: These exercises are great for beginners because they don’t require any equipment. Examples include push-ups, squats, lunges, and planks.2. Yoga: Yoga is a low-impact exercise that can help improve flexibility, balance, and strength. It’s also great for reducing stress and improving overall well-being.3. Brisk walking: Walking is a simple and accessible form of exercise that can be done anywhere. It’s a great way to get your heart rate up and improve cardiovascular health.4. Swimming: Swimming is a low-impact exercise that can help improve cardiovascular health and strengthen muscles. It’s also easy on the joints, making it a great option for beginners.5. Cycling: Cycling is another low-impact exercise that can be done indoors or outdoors. It’s a great way to improve cardiovascular health and build leg strength.6. Resistance band exercises: Resistance bands are lightweight and portable, making them easy to use at home or on the go. They can help improve strength and flexibility in the upper body, lower body, and core.7. Pilates: Pilates is a form of exercise that focuses on core strength, flexibility, and body control. It’s a great option for beginners because it’s low-impact and can help improve posture and balance.8. Dance-based workouts: Dance-based workouts, such as Zumba or Hip Hop Abs, are fun and energetic ways to get in shape. They often involve aerobic exercise and strength training, making them a great option for beginners.9. Jumping rope: Jumping rope is a high-intensity exercise that can help improve cardiovascular health and burn calories. It’s also easy to do at home with a simple jump rope.10. Kettlebell exercises: Kettlebells are small weights that can be used to perform a variety of exercises, including swings, squats, and presses. They’re a great way to improve strength and cardiovascular health.Remember to always consult with a doctor before starting any new exercise program, especially if you’re a beginner. It’s also important to start slowly and gradually increase the intensity and duration of your workouts as you become more comfortable.
body
MobileLLM: Optimizing Sub-billion Parameter Language Modelsfor On-Device Use Cases
2Improving Sub-billion Scale LLM Design
2402.14905
3Experiments3.1Experimental SettingsWe trainMobileLLMfrom scratch using Adam optimizerwith weight decay of 0.1. The experiments are conducted using 32 A100 GPUs, with a batch size of 32 on each GPU. The initial learning rate is set to 2e-3 and follows a cosine learning-rate decay strategy. We perform quick exploration experiments with 120k iterations on 0.25T tokens and train the best models reported in Tables3.2and3.2with 480k iterations on 1T tokens.3.2Main ResultsTable 3:Zero-shot performance on Common Sense Reasoning tasks.MobileLLMdenotes the proposed baseline model andMobileLLM-LSis integrated with layer sharing with the #layer counting layers with distinct weights.Model#Layers#ParamsARC-eARC-cBoolQPIQASIQAHellaSwagOBQAWinoGrandeAvg.Cerebras-GPT-111M10111M35.820.262.058.039.826.729.048.840.0LaMini-GPT-124M12124M43.626.051.862.742.130.229.649.241.9Galactica-125M12125M44.026.254.955.438.929.628.249.640.9OPT-125M12125M41.325.257.562.041.931.131.250.842.6GPT-neo-125M12125M40.724.861.362.541.929.731.650.742.9Pythia-160M12162M40.025.359.562.041.529.931.250.942.5RWKV-169M12169M42.525.359.163.940.731.933.851.543.6MobileLLM-125M30125M43.927.160.265.342.438.939.553.146.3MobileLLM-LS-125M30125M45.828.760.465.742.939.541.152.147.0\hdashline[0.8pt/1pt]
Cerebras-GPT-256M14256M37.923.260.361.440.628.331.850.541.8OPT-350M24331M41.925.754.064.842.636.233.352.443.9RWKV-430M24430M48.932.053.468.143.640.637.851.647.0Pythia-410M24405M47.130.355.367.243.140.136.253.446.6BLOOM-560M24559M43.727.553.765.142.536.532.652.244.2Cerebras-GPT-590M18590M42.624.957.762.840.932.033.249.743.0MobileLLM-350M32345M53.833.562.468.644.749.640.057.651.3MobileLLM-LS-350M32345M54.432.562.869.844.150.645.857.252.1Table 4:Performance on Trivia QA and RACE datasets for question answering and reading comprehension tasks.TQA(F1 score) RACE(Acc) Model1-shot5-shot64-shotmiddlehighCerebras-GPT-111M1.93.84.429.224.3OPT-125M8.79.68.234.727.5GPT-Neo-125M8.07.95.034.727.0Pythia-160M2.11.42.130.225.1MobileLLM-125M13.914.312.539.728.9MobileLLM-LS-125M14.214.814.640.729.6\hdashline[0.8pt/1pt]
Cerebras-GPT-256M5.26.83.331.726.2OPT-350M11.012.310.437.128.0Pythia-410M12.413.812.839.129.7BLOOM-560M8.88.95.337.628.2Cerebras-GPT-590M6.49.14.934.627.4MobileLLM-350M22.023.924.245.633.8MobileLLM-LS-350M21.422.522.647.333.7We compare the final performance on zero-shot common sense reasoning tasks, question answering and reading comprehension tasks. The results of baseline methods were evaluated using their open-source Hugging Face models to ensure consistent evaluation procedures.Zero-shot Common Sense ReasoningTable3.2presents a comparison between our proposed model,MobileLLM, and state-of-the-art sub-billion parameter models, including the early open-sourced LLMs, OPT, BLOOM, and recent releases such as Galactica, Cerebras, GPT-neoas well as the LLM analyzing suite Pythiaand transformer variants RWKVon zero-shot common sense reasoning tasks.
For the 125M model size,MobileLLMfavorably outperforms previous models such as OPT, GPT-Neo, and Calactica at the same model size by a significant margin. Additionally,MobileLLM-125M achieves 3.8 points and 2.7 points higher accuracy than Pythia-160M and RWKV-169M while being 22% and 26% smaller, respectively. Furthermore, incorporating layer-sharing inMobileLLM-LS-125M results in an additional 0.7 points improvement in accuracy. It is noteworthy thatMobileLLM-LS-125M achieves comparable or even higher results than most previous 350M models.
In the 350M model size category,MobileLLMsurpasses previous state-of-the-art models by more than 4 points with comparable or smaller model size.Question Answering and Reading ComprehensionWe evaluate pre-trained models on the TQA question answering benchmarkand RACE reading comprehension benchmark. We follow the evalutation setup fromand report the results in Table3.2. Comparing models of 125M size,MobileLLM-125M demonstrates a noteworthy improvement of over 4.3 points on the TQA benchmark contrast to its predecessor. Moreover, theMobileLLM-350M model exhibits a substantial performance increase of approximately 10 points compared to other 350M-sized models. For the reading comprehension tasks,MobileLLMmodel family also exhibit significant higher scores than preceding sub-billion parameter models.3.3Downstream TasksTo validate the effectiveness of sub-billion scale models for on-device applications, we assess their performance in two crucial on-device tasks: Chat and API calling.3.3.1ChatWe fine-tuneMobileLLMmodels, as well as previous state-of-the-art (SoTA)  models sourced from HuggingFace checkpoints for chat-based tasks and evaluate them under identical settings to ensure consistency. We evaluate on two benchmarks: AlpacaEval, a single-run chat benchmark, and MT-Bench, a multi-run chat benchmark. The results presented in Table3.3.1showcases thatMobileLLMmodels significantly outperforms previous state-of-the-art sub-billion scale models, surpassing even models with 1 billion parameters. Notably,MobileLLM-LS-350M achieves a remarkable win rate of 48.2% when compared to the baseline GPT-3 model (text-davinci-001) . Considering the self-win rate of GPT-3 is 50%, is noteworthy thatMobileLLM-LS-350M obtains comparable chat performance as this baseline model. Visualizations of chat examples in the appendix also underscore the impressive quality of responses generated byMobileLLMmodels.Table 5:Benchmark results on AlpacaEval (Evaluator: GPT-4; Reference model: text-davinci-001)  and MT-Bench.ModelMT-BenchAlpaca Evalnumber of parameters < 200M\hdashline[0.8pt/1pt]
OPT-125M1.213.91GPT-Neo-125M1.061.01Pythia-160M1.010.63MobileLLM-125M2.3324.07MobileLLM-LS-125M2.5223.79200M < number of parameters < 1B\hdashline[0.8pt/1pt]
OPT-350M1.376.80Pythia-410M1.6213.87BLOOM-560M1.7310.29MobileLLM-350M3.2847.08MobileLLM-LS-350M3.1648.20number of parameters > 1B\hdashline[0.8pt/1pt]
Pythia-1B1.7016.62BLOOM-1.1B2.3719.90Falcon-1.3B2.5430.38OPT-1.3B2.2438.843.3.2API CallingAPI calling is a common on-device application, particularly in collaboration with audio-to-text models for assistant functionalities. Leveraging LLMs for API calling involves the conversion of natural language inputs into JSON configurations to invoke corresponding APIs666https://platform.openai.com/docs/guides/function-calling. For instance, given the input"Help me set an alarm at 7:30 AM"the model outputs {API: "alarm(time="7:30 am") "}. Additionally, the model generates an agent response:"Sure! Your alarm is set to 7:30 AM."To adapt LLMs for this task, we create a synthetic dataset with 5000 training samples and 2500 testing samples. Each sample involves 8 conversation turns on average. Detailed examples of this dataset are provided in appendix. The pre-trained models undergoes fine-tuning on training set for 4 epochs, utilizing the Adam optimizer with a linear-decay learning rate starting at 2e-5 and a weight decay of 0.01.Table3.3.2shows thatMobileLLM-350M demonstrates comparable intent and structure exact match scores to LLaMA-v2 7B, where high intent scores indicate correct prediction of the API user intends to call, while structural exact match scores reflects the proficiency in predicting content within API functions. Despite lower Rouge scores inMobileLLM-350M compared to 7B models, it is crucial to note that API calling prioritize correct API invocation. The results suggest that certain common scenarios in on-device applications are not particularly challenging, and smaller models likeMobileLLM-350M can adeptly handle it.Table 6:API calling evaluation score. EM/EMmeasures the exact match in API calling. R1/RL refers to Rouge-1/-L score measuring the quality of agent response.ModelEMEMR1RL\hdashline[0.8pt/1pt]
OPT-350M56.138.637.135.3Pythia-410M62.244.743.141.1BLOOM-560M64.737.936.934.6MobileLLM-350M65.348.846.844.6\hdashline[0.8pt/1pt]
LLaMA-v2 7B62.850.956.554.33.4Compatibility with QuantizationFigure 7:Comparison between BFloat16 model and 8-bit weight 8-bit activation post-training quantized model.We further conduct per-token min-max post-training quantization (PTQ)  experiments on bothMobileLLMandMobileLLM-LSmodels with 125M and 350M model size trained on 0.25T tokens. Figure7shows that employing W8A8 PTQ yields a modest accuracy reduction of less than 0.5 points and remains compatible with layer-sharing.3.5Knowledge DistillationSo far, we trained compact models from scratch using next tokens as hard labels.
We explored Knowledge Distillation (KD)  of 125M and 350M models with LLAMA-v2 7B as a teacher.
Unfortunately KD increases training time (slowdown of)  and exhibits comparable or inferior accuracy to label-based training (details in appendix) .3.6On-device ProfilingTable 7:Latency analysis ofMobileLLM-125M (30 layers) ,MobileLLM-LS-125M (230 layers, adjacent blocks sharing weights) , and a 60-layer non-shared weight model, with consistent configurations in all other aspects.LoadInitExecute\hdashline[0.8pt/1pt]MobileLLM39.2 ms1361.7 ms15.6 msMobileLLM-LS43.6 ms1388.2 ms16.0 ms60-layer non-shared68.6 ms3347.7 ms29.0 msWe measure the latency forMobileLLM-125M andMobileLLM-LS-125M FP16 models via ExecuTorch777https://pytorch.org/executorch-overviewon iPhone 13 (iOS 17.2.1) , with Metal Performance Shaders (MPS)  backend888https://pytorch.org/executorch/stable/build-run-mps.html. Model loading, initialization and execution time are reported in Table3.6. Specifically, execution time is averaged over 50 iterations.Results in Table3.6reflects that through weight sharing and doubling the number of layers,MobileLLM-LSincurs only a 2.2% increase in loading and initialization time compared toMobileLLM, attributable to their similar model sizes. Execution time also experiences a mere 2.6% overhead, benefitting from data locality. In contrast, a model with doubled number of layers without weight sharing exhibits a substantial 143% rise in loading and initialization time and an 86% increase in execution time.4Related WorkThe excellent performance of LLMs has fostered its wide applications.
Considering computational cost and energy consumption of LLMs, an new stream of research direction have emerged to downsize LLMs to enable on-device inference. These methods include:Model Compression.Numerous model compression methods are developed for LLMs, including pruning, sparsity, and quantization. Our research is complementary to these techniques. As also substantiated in Section3.4, our methodology is compatible with quantization.Small Model Design.A limited number of studies have explored compact model architectures, such as TinyLLaMA. However, even the smallest TinyLLaMA exceeds 1 billion parameters, making them still prohibitive for many on-device applications. Some research proposes large model architectures alongside their smaller LLM variants in a model familyor a analytical suite containing small LLM variants. However, these models are not optimized under the constraint of sub-billion parameters and therefore may not be optimal.Neural Architecture Search.NAS has garnered substantial attention in the realm of convolutional neural networks, particularly in the context of vision tasks. In contrast, within the transformer domain, the prevailing consensus posits that the model architecture exerts minimal influence on accuracy, provided the total number of parameters remains consistent.
Only a limited number of studies have developed NAS algorithm for language transformer, targeting at BERT models. Our current investigation, focusing on the interplay between depth and width, can be conceptualized as a meticulous grid search within the depth space. The outcomes of that study challenge the prevalent orthodoxy surrounding scaling laws, proposing that deep and thin architectures demonstrate higher performance for compact LLMs.Weight Sharing.Weight sharing is an intuitive strategy for optimizing model weight utilization within fixed parameter constraints. While the OPT familyand subsequent worksleverage weight sharing between input and output embeddings, limited research has explored weight sharing for intermediate layers in transformers. Prior efforts often entail specialized designs for shared layers. In contrast, our contribution highlights a more straightforward yet effective way of simply repeating transformer blocks, yielding improved accuracy with a fixed model size and minimal latency increase.Efficient Attention and Implementation.In the realm of efficient transformer design, much research has focused on optimizing attention computation through methods like low-rank approximationand sparse attention. Another line of work explores hardware scheduling and weight movement, exemplified by works such as FlashAttentionand FlexGen. In contrast, our primary goal is to optimize model size without introducing new attention computation or efficient hardware implementation methods.5ConclusionThis study focuses on optimizing sub-billion scale models for on-device applications. Our findings indicate that, for smaller models, prioritizing depth over width enhances model performance. Furthermore, by leveraging advanced weight-sharing techniques, including embedding sharing, grouped query attention, and block-wise weight sharing, we achieve significant enhancements in weight utilization within storage-constrained scenarios. The resulting models denoted asMobileLLMexhibit substantial advancements in zero-shot commonsense reasoning, question answering, and reading comprehension tasks compared to previous SoTA methods. Last but not least, we demonstrate the effectiveness of the fine-tunedMobileLLMmodels in two prevalent on-device use cases: chat and API calling, underscoring their adeptness in handling such tasks.AcknowledgmentWe thank Hansong Zhang for his valuable contribution in setting up latency measurement environment on iOS, and the full support from PyTorch Edge team.Broader ImpactThis paper advocates for the adoption of sub-billion scale large language models in on-device applications, aiming to mitigate energy consumption during LLM inference. The proposed approach is promising in alleviating computational costs associated with LLM deployment.ReferencesAinslie et al. (2023) Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y., Lebron, F., and Sanghai, S.GQA: Training generalized multi-query transformer models from multi-head checkpoints.InEMNLP, 2023.Biderman et al. (2023) Biderman, S., Schoelkopf, H., Anthony, Q. G., Bradley, H., O’Brien, K., Hallahan, E., Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., et al.Pythia: A suite for analyzing large language models across training and scaling.InInternational Conference on Machine Learning, pp.  2397–2430. PMLR, 2023.Bisk et al. (2020) Bisk, Y., Zellers, R., Gao, J., Choi, Y., et al.Piqa: Reasoning about physical commonsense in natural language.InProceedings of the AAAI conference on artificial intelligence, volume 34, pp.  7432–7439, 2020.Black et al. (2022) Black, S., Biderman, S., Hallahan, E., Anthony, Q., Gao, L., Golding, L., He, H., Leahy, C., McDonell, K., Phang, J., et al.Gpt-neox-20b: An open-source autoregressive language model.arXiv preprint arXiv:2204.06745, 2022.Chowdhery et al. (2023) Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al.Palm: Scaling language modeling with pathways.Journal of Machine Learning Research, 24(240) :1–113, 2023.Clark et al. (2019) Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., and Toutanova, K.Boolq: Exploring the surprising difficulty of natural yes/no questions.arXiv preprint arXiv:1905.10044, 2019.Clark et al. (2018) Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O.Think you have solved question answering? try arc, the ai2 reasoning challenge.arXiv preprint arXiv:1803.05457, 2018.Dao et al. (2022) Dao, T., Fu, D., Ermon, S., Rudra, A., and Ré, C.Flashattention: Fast and memory-efficient exact attention with io-awareness.Advances in Neural Information Processing Systems, 35:16344–16359, 2022.Dauphin et al. (2017) Dauphin, Y. N., Fan, A., Auli, M., and Grangier, D.Language modeling with gated convolutional networks.InInternational conference on machine learning, pp.  933–941. PMLR, 2017.Dettmers et al. (2022) Dettmers, T., Lewis, M., Belkada, Y., and Zettlemoyer, L.Llm. int8 () : 8-bit matrix multiplication for transformers at scale.arXiv preprint arXiv:2208.07339, 2022.Dey et al. (2023) Dey, N., Gosal, G., Khachane, H., Marshall, W., Pathria, R., Tom, M., Hestness, J., et al.Cerebras-gpt: Open compute-optimal language models trained on the cerebras wafer-scale cluster.arXiv preprint arXiv:2304.03208, 2023.Frantar & Alistarh (2023) Frantar, E. and Alistarh, D.Sparsegpt: Massive language models can be accurately pruned in one-shot.InInternational Conference on Machine Learning, pp.  10323–10337. PMLR, 2023.Frantar et al. (2022) Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D.Gptq: Accurate post-training quantization for generative pre-trained transformers.arXiv preprint arXiv:2210.17323, 2022.Ganesan et al. (2021) Ganesan, V., Ramesh, G., and Kumar, P.Supershaper: Task-agnostic super pre-training of bert models with variable hidden dimensions.arXiv preprint arXiv:2110.04711, 2021.Google (2023) Google.Pixel 8 pro tech specs.https://store.google.com/gb/product/pixel_8_pro_specs, 2023.Guo et al. (2020) Guo, Z., Zhang, X., Mu, H., Heng, W., Liu, Z., Wei, Y., and Sun, J.Single path one-shot neural architecture search with uniform sampling.InComputer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XVI 16, pp.  544–560. Springer, 2020.Han et al. (2016) Han, S., Liu, X., Mao, H., Pu, J., Pedram, A., Horowitz, M. A., and Dally, W. J.Eie: Efficient inference engine on compressed deep neural network.ACM SIGARCH Computer Architecture News, 44(3) :243–254, 2016.Hinton et al. (2015) Hinton, G., Vinyals, O., Dean, J., et al.Distilling the knowledge in a neural network.arXiv preprint arXiv:1503.02531, 2(7) , 2015.Hristov (2022) Hristov, V.A16 bionic explained: what’s new in apple’s pro-grade mobile chip?https://www.phonearena.com/news/A16-Bionic-explained-whats-new_id142438, 2022.Jawahar et al. (2023) Jawahar, G., Yang, H., Xiong, Y., Liu, Z., Wang, D., Sun, F., Li, M., Pappu, A., Oguz, B., Abdul-Mageed, M., et al.Mixture-of-supernets: Improving weight-sharing supernet training with architecture-routed mixture-of-experts.arXiv preprint arXiv:2306.04845, 2023.Joshi et al. (2017) Joshi, M., Choi, E., Weld, D. S., and Zettlemoyer, L.Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension.arXiv preprint arXiv:1705.03551, 2017.Kaplan et al. (2020) Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D.Scaling laws for neural language models.arXiv preprint arXiv:2001.08361, 2020.Katharopoulos et al. (2020) Katharopoulos, A., Vyas, A., Pappas, N., and Fleuret, F.Transformers are rnns: Fast autoregressive transformers with linear attention.InInternational conference on machine learning, pp.  5156–5165. PMLR, 2020.Kim et al. (2023) Kim, S., Hooper, C., Gholami, A., Dong, Z., Li, X., Shen, S., Mahoney, M. W., and Keutzer, K.Squeezellm: Dense-and-sparse quantization.arXiv preprint arXiv:2306.07629, 2023.Kingma & Ba (2014) Kingma, D. P. and Ba, J.Adam: A method for stochastic optimization.arXiv preprint arXiv:1412.6980, 2014.Kitaev et al. (2020) Kitaev, N., Kaiser, Ł., and Levskaya, A.Reformer: The efficient transformer.arXiv preprint arXiv:2001.04451, 2020.Lai et al. (2017) Lai, G., Xie, Q., Liu, H., Yang, Y., and Hovy, E.Race: Large-scale reading comprehension dataset from examinations.arXiv preprint arXiv:1704.04683, 2017.Li et al. (2023) Li, X., Zhang, T., Dubois, Y., Taori, R., Gulrajani, I., Guestrin, C., Liang, P., and Hashimoto, T. B.Alpacaeval: An automatic evaluator of instruction-following models.https://github.com/tatsu-lab/alpaca_eval, 2023.Liu et al. (2023a) Liu, J., Gong, R., Wei, X., Dong, Z., Cai, J., and Zhuang, B.Qllm: Accurate and efficient low-bitwidth quantization for large language models.arXiv preprint arXiv:2310.08041, 2023a.Liu et al. (2023b) Liu, S.-y., Liu, Z., Huang, X., Dong, P., and Cheng, K.-T.Llm-fp4: 4-bit floating-point quantized transformers.arXiv preprint arXiv:2310.16836, 2023b.Liu et al. (2023c) Liu, Z., Oguz, B., Zhao, C., Chang, E., Stock, P., Mehdad, Y., Shi, Y., Krishnamoorthi, R., and Chandra, V.Llm-qat: Data-free quantization aware training for large language models.arXiv preprint arXiv:2305.17888, 2023c.Malladi et al. (2012) Malladi, K. T., Lee, B. C., Nothaft, F. A., Kozyrakis, C., Periyathambi, K., and Horowitz, M.Towards energy-proportional datacenter memory with mobile dram.ACM SIGARCH Computer Architecture News, 40(3) :37–48, 2012.Mihaylov et al. (2018) Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A.Can a suit of armor conduct electricity? a new dataset for open book question answering.arXiv preprint arXiv:1809.02789, 2018.Peng et al. (2023) Peng, B., Alcaide, E., Anthony, Q., Albalak, A., Arcadinho, S., Cao, H., Cheng, X., Chung, M., Grella, M., GV, K. K., et al.Rwkv: Reinventing rnns for the transformer era.arXiv preprint arXiv:2305.13048, 2023.Reid et al. (2021) Reid, M., Marrese-Taylor, E., and Matsuo, Y.Subformer: Exploring weight sharing for parameter efficiency in generative transformers.arXiv preprint arXiv:2101.00234, 2021.Roy et al. (2021) Roy, A., Saffar, M., Vaswani, A., and Grangier, D.Efficient content-based sparse attention with routing transformers.Transactions of the Association for Computational Linguistics, 9:53–68, 2021.Sakaguchi et al. (2021) Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y.Winogrande: An adversarial winograd schema challenge at scale.Communications of the ACM, 64(9) :99–106, 2021.Sap et al. (2019) Sap, M., Rashkin, H., Chen, D., LeBras, R., and Choi, Y.Socialiqa: Commonsense reasoning about social interactions.arXiv preprint arXiv:1904.09728, 2019.Scao et al. (2022) Scao, T. L., Fan, A., Akiki, C., Pavlick, E., Ilić, S., Hesslow, D., Castagné, R., Luccioni, A. S., Yvon, F., Gallé, M., et al.Bloom: A 176b-parameter open-access multilingual language model.arXiv preprint arXiv:2211.05100, 2022.Shen et al. (2022) Shen, Z., Liu, Z., and Xing, E.Sliced recursive transformer.InEuropean Conference on Computer Vision, pp.  727–744. Springer, 2022.Sheng et al. (2023) Sheng, Y., Zheng, L., Yuan, B., Li, Z., Ryabinin, M., Chen, B., Liang, P., Ré, C., Stoica, I., and Zhang, C.Flexgen: High-throughput generative inference of large language models with a single gpu.InInternational Conference on Machine Learning, pp.  31094–31116. PMLR, 2023.Sun et al. (2023) Sun, M., Liu, Z., Bair, A., and Kolter, J. Z.A simple and effective pruning approach for large language models.arXiv preprint arXiv:2306.11695, 2023.Tan & Le (2019) Tan, M. and Le, Q.Efficientnet: Rethinking model scaling for convolutional neural networks.InInternational conference on machine learning, pp.  6105–6114. PMLR, 2019.Taylor et al. (2022) Taylor, R., Kardas, M., Cucurull, G., Scialom, T., Hartshorn, A., Saravia, E., Poulton, A., Kerkez, V., and Stojnic, R.Galactica: A large language model for science.arXiv preprint arXiv:2211.09085, 2022.Timiryasov & Tastet (2023) Timiryasov, I. and Tastet, J.-L.Baby llama: knowledge distillation from an ensemble of teachers trained on a small dataset with no performance penalty.arXiv preprint arXiv:2308.02019, 2023.Touvron et al. (2023a) Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al.Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023a.Touvron et al. (2023b) Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Ferrer, C. C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshorn, A., Hosseini, S., Hou, R., Inan, H., Kardas, M., Kerkez, V., Khabsa, M., Kloumann, I., Korenev, A., Koura, P. S., Lachaux, M.-A., Lavril, T., Lee, J., Liskovich, D., Lu, Y., Mao, Y., Martinet, X., Mihaylov, T., Mishra, P., Molybog, I., Nie, Y., Poulton, A., Reizenstein, J., Rungta, R., Saladi, K., Schelten, A., Silva, R., Smith, E. M., Subramanian, R., Tan, X. E., Tang, B., Taylor, R., Williams, A., Kuan, J. X., Xu, P., Yan, Z., Zarov, I., Zhang, Y., Fan, A., Kambadur, M., Narang, S., Rodriguez, A., Stojnic, R., Edunov, S., and Scialom, T.Llama 2: Open foundation and fine-tuned chat models, 2023b.Wang et al. (2020) Wang, S., Li, B. Z., Khabsa, M., Fang, H., and Ma, H.Linformer: Self-attention with linear complexity.arXiv preprint arXiv:2006.04768, 2020.Wu et al. (2019) Wu, B., Dai, X., Zhang, P., Wang, Y., Sun, F., Wu, Y., Tian, Y., Vajda, P., Jia, Y., and Keutzer, K.Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search.InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.  10734–10742, 2019.Xia et al. (2023a) Xia, H., Zheng, Z., Li, Y., Zhuang, D., Zhou, Z., Qiu, X., Li, Y., Lin, W., and Song, S. L.Flash-llm: Enabling cost-effective and highly-efficient large generative model inference with unstructured sparsity.arXiv preprint arXiv:2309.10285, 2023a.Xia et al. (2023b) Xia, M., Gao, T., Zeng, Z., and Chen, D.Sheared llama: Accelerating language model pre-training via structured pruning.arXiv preprint arXiv:2310.06694, 2023b.Xiao et al. (2023) Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., and Han, S.Smoothquant: Accurate and efficient post-training quantization for large language models.InInternational Conference on Machine Learning, pp.  38087–38099. PMLR, 2023.Xiong et al. (2021) Xiong, Y., Zeng, Z., Chakraborty, R., Tan, M., Fung, G., Li, Y., and Singh, V.Nyströmformer: A nyström-based algorithm for approximating self-attention.InProceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp.  14138–14148, 2021.Xu et al. (2021) Xu, J., Tan, X., Luo, R., Song, K., Li, J., Qin, T., and Liu, T.-Y.Nas-bert: task-agnostic and adaptive-size bert compression with neural architecture search.InProceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp.  1933–1943, 2021.Yao et al. (2022) Yao, Z., Yazdani Aminabadi, R., Zhang, M., Wu, X., Li, C., and He, Y.Zeroquant: Efficient and affordable post-training quantization for large-scale transformers.Advances in Neural Information Processing Systems, 35:27168–27183, 2022.Zellers et al. (2019) Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y.Hellaswag: Can a machine really finish your sentence?arXiv preprint arXiv:1905.07830, 2019.Zhang et al. (2022) Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al.Opt: Open pre-trained transformer language models.arXiv preprint arXiv:2205.01068, 2022.Zheng et al. (2023) Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., et al.Judging llm-as-a-judge with mt-bench and chatbot arena.arXiv preprint arXiv:2306.05685, 2023.Zoph & Le (2016) Zoph, B. and Le, Q. V.Neural architecture search with reinforcement learning.arXiv preprint arXiv:1611.01578, 2016.AppendixAppendix AImpact of Each Design ChoiceThis section presents comprehensive tabulated results for theimproving sub-billion scale LLM designexperiments, at the model sizes of 125M and 350M. Looking at the results in TableA, transitioning from the traditional Feedforward Network ()  to SwiGLU yields a accuracy improvement of 1.3% for both model sizes. Further increasing the model depth enhances accuracy by 0.9%/1.1% for 125M/350M models, respectively. Then, introducing input and output embedding sharing achieves a parameter reduction of approximately 10%, while with only marginal accuracy drops of 0.2% for the 125M and 0.6% for 350M models. Additionally, following in findings in SectionC, we incorporate grouped query attention with head dimension equals to 64, and head number to be near 4to the kv head number, while increasing the embedding dimension to preserve model size. This modification further results in a performance boost of 0.4%/0.7% for 125M/350M models. Combining these techniques establishes a strong baseline network denoted asMobileLLM. Finally, the immediate block-wise weight-sharing technique contributes an additional accuracy gain of 1.1% for models trained on 0.25 trillion tokens, and resulting in the modelMobileLLM-LS. Final models includingMobileLLMandMobileLLM-LSare trained with 1 trillion tokens.Table 8:Ablation study on the impact of each design choice to the model accuracy on zero-shot common sense reasoning tasks. Corresponding to the bar chart in Figure3. Here, L, H,denotes number of layers, heads, kv-heads, respectively and dim denotes embedding dimension.TechniquesLHDim#Params(M) ARC-eARC-cBoolQPIQASIQAHellaSwagOBQAWinoGrandeAvg.125M\hdashline[0.8pt/1pt]
Baseline model121212768134.141.325.257.562.041.931.131.250.842.6+ SwiGLU in FFN121212768134.143.128.958.162.342.334.631.550.143.9+ Use deep-thin structure3088512135.043.626.158.062.542.636.537.551.544.8+ Embedding share3088512118.644.426.056.262.843.135.936.052.644.6+ Multi-query attention3093576124.645.527.758.364.641.936.435.450.445.0(Train on 1T token) 3093576124.643.927.160.265.342.438.939.553.146.3+ Layer sharing3093576124.644.427.061.565.143.037.637.852.046.1(Train on 1T token) 3093576124.645.828.760.465.742.939.541.152.147.0350M\hdashline[0.8pt/1pt]
Baseline model1520201280376.850.327.653.868.144.142.640.152.447.4+ SwiGLU in FFN1520201280386.749.230.659.167.744.343.241.054.248.7+ Use deep-thin structure321414896380.350.731.459.467.843.346.243.856.249.8+ Embedding share321414896351.649.932.060.367.943.247.038.954.849.2+ Multi-query attention32155960345.351.431.361.068.143.647.241.655.449.9(Train on 1T token) 32155960345.353.833.562.468.644.749.640.057.651.3+ Layer sharing32155960345.351.935.259.668.943.447.243.358.451.0(Train on 1T token) 32155960345.354.432.562.869.844.150.645.857.252.1Appendix BDepth vs WidthWe provide network depth versus width exploration results on zero-shot reasoning tasks in TableB, as well as results on question answering and reading comprehension tasks in TableB. The findings indicate that shallow architectures with fewer than 10 layers perform poorly in reasoning or handling comprehension tasks. Models with 10-20 layers exhibit improved accuracy, while further increasing depth continues to provide significant benefits across all three tasks. Notably, the optimal depth is found to be near 30 layers for sub-billion scale models.Table 9:Ablation study on depth versus width in architecture design, illustrated in Figure4(a) (b) . For compact models, prioritizing depth over width yields superior performance, assessed through zero-shot common sense reasoning tasks.#Layer#HeadsDim#Params(M) ARC-eARC-cBoolQPIQASIQAHellaSwagOBQAWinoGrandeAvg.4201280163.242.026.161.061.541.332.931.449.843.36161024142.642.225.551.062.641.633.532.150.042.3814896138.143.625.548.661.840.934.037.552.043.01212768134.143.128.958.162.342.334.631.550.143.91810640132.441.025.559.362.641.634.236.750.143.9249576132.443.627.059.763.842.036.233.752.244.8308512135.043.626.158.062.542.636.537.551.544.8427448134.643.326.157.863.341.736.335.952.044.5626384134.344.126.759.064.742.436.233.151.344.7\hdashline[0.8pt/1pt]
5322048388.047.928.862.265.242.838.139.052.647.110241536381.449.129.660.667.144.042.741.254.448.612221408379.949.931.659.168.242.644.043.753.849.115201280386.749.230.659.167.744.343.241.054.248.719181152376.350.830.956.369.143.845.239.654.548.824161024373.850.733.858.868.643.545.040.054.949.42815960371.151.733.257.667.942.946.037.753.948.93214896380.350.731.459.467.843.346.243.856.249.84612768374.749.730.559.568.145.144.943.455.549.66610640376.250.531.861.067.443.846.040.155.649.5Table 10:Ablation study on depth vs. width in architecture design on TQA and RACE datasets, depicted in Figure4(c-f) .TQA(F1 score) RACE(Acc) #Layer#HeadsDim#Params(M) 1-shot5-shot64-shotmiddlehigh4201280163.22.03.93.933.226.06161024142.63.54.94.629.225.5814896138.14.25.24.729.224.31212768134.15.16.16.335.627.31810640132.44.76.85.234.327.4249576132.45.97.27.335.328.6308512135.06.26.86.934.227.9427448134.66.17.27.234.428.6626384134.36.07.07.338.928.9\hdashline[0.8pt/1pt]
5322048388.05.67.88.638.129.010241536381.49.011.812.740.630.012221408379.911.712.913.342.431.515201280386.710.713.114.242.630.619181152376.311.313.313.542.632.424161024373.811.813.514.443.031.92815960371.112.513.614.942.832.33214896380.312.614.515.344.232.24612768374.713.015.415.644.731.46610640376.212.515.115.744.032.5Appendix CNumber of Heads and Key-Value HeadsWe provide detailed experimental results assessing the impact of the number of attention heads and key-value heads on zero-shot reasoning accuracy in TableC. Our study involves two baseline architectures: a 8-layer 125M model with a embedding dimension of 896, and a 15-layer 350M model with a embedding dimension of 1280. We conduct head size sweeps in {8, 16, 32}. The findings shown in TableCindicate that using 16 heads, with a head dimension close to 64, and 4 key-value heads, yields the best accuracy and memory trade-off. This setting serves as a guiding principle in our model architecture design.Table 11:Ablation study investigating the impact of the number of attention heads and key-value heads.Model#Heads#KV-Heads#Params(M) ARC-eARC-cBoolQPIQASIQAHellaSwagOBQAWinoGrandeAvg.3232138.142.126.958.462.242.133.836.752.444.33216131.741.226.457.762.542.333.334.252.943.8328128.542.627.361.161.941.932.235.052.044.2324126.843.126.859.862.741.432.534.451.144.0322126.039.826.759.459.442.031.332.652.943.0321125.641.024.359.160.841.231.435.452.043.1125M1616138.141.625.761.162.443.134.436.951.644.6# layers=8168131.742.426.460.763.441.933.534.751.544.3dim=896164128.542.525.662.362.441.833.035.954.544.7162126.841.725.356.961.742.032.932.654.543.5161126.040.426.361.863.241.732.034.050.443.788138.141.425.058.361.741.733.335.953.243.884131.743.328.258.361.842.833.830.953.044.082128.540.726.258.161.241.632.834.851.543.481126.842.524.859.462.342.032.036.351.343.8\hdashline[0.8pt/1pt]3232386.748.630.459.767.243.944.040.953.948.63216362.148.931.657.668.443.443.838.654.948.4328349.848.333.161.067.242.642.139.053.948.4324343.747.229.859.467.243.542.542.554.148.3322340.647.630.362.466.942.641.638.652.047.7321339.048.527.356.367.142.940.936.753.346.6350M1616386.750.830.662.368.643.545.143.852.449.6# layers=15168362.148.530.759.467.343.843.841.353.348.5dim=1280164349.849.930.660.069.243.544.241.855.849.4162343.749.328.455.067.342.742.640.354.547.5161340.649.229.358.867.443.542.139.952.847.988386.751.033.258.267.043.943.942.654.749.384362.150.031.360.266.042.743.740.953.748.682349.849.330.459.967.942.843.538.953.848.381343.748.027.761.567.143.142.537.354.747.8Appendix DLayer Sharing Number AblationTable 12:Ablation study on the impact of varying layer repetition numbers.Model SizeRepeat TimesARC-eARC-cBoolQPIQASIQAHellaSwagOBQAWinoGrandeAvg.125Mbaseline41.625.761.162.443.134.436.951.644.6block-wise share repeat243.927.961.564.341.535.535.150.245.0block-wise share repeat343.327.660.963.042.335.634.152.945.0block-wise share repeat442.026.662.364.441.536.235.853.945.3\hdashline[0.8pt/1pt]350Mbaseline50.830.662.368.643.545.143.852.449.6block-wise share repeat251.530.859.668.243.947.744.755.050.2block-wise share repeat349.632.057.368.443.947.040.656.249.4block-wise share repeat452.232.660.069.044.547.641.855.250.4We extended our investigation to determine the optimal number of layer repetitions. The experiment involved the 8-layer 125M model with a embedding dimension of 896 and the 15-layer 350M model with a embedding dimension of 1280. The results in TableDdemonstrate that when we double the layer number and with each two transformer blocks sharing weights, the accuracy is enhanced by 0.4-0.6%. However, as we further triple or quadruple the layer repetition times, this accuracy enhancement effect diminishes. Consequently, in our experiments, we adopt a configuration where every two blocks share weights, effectively doubling the total number of layers.Appendix ECompatibility with QuantizationThis section explores the compatibility of quantization with the proposed model architecture and layer sharing. We employ a straightforward per-token min-max quantization, quantizing both weight and activation to 8-bit using post-training quantization (PTQ) . Experiments are conducted on bothMobileLLMandMobileLLM-LSwith model size of 125M and 350M, trained on 0.25T tokens. Results in TableEdemonstrate that W8A8 PTQ results in a modest accuracy drop of within 0.5% and is compatible with the proposed layer sharing method.Table 13:Ablation study: 8-bit weight, 8-bit activation post-training quantization results on zero-shot common sense reasoning tasks. Quantized models achieve accuracy gap of within 0.5% compared to the full-precision BF16 counterpart.ModelPrecisionARC-eARC-cBoolQPIQASIQAHellaSwagOBQAWinoGrandeAvg.GapMobileLLM-125MBF1645.527.758.364.641.936.435.450.445.0–MobileLLM-125MW8A845.227.158.365.041.736.233.651.044.80.2\hdashline[0.8pt/1pt]MobileLLM-LS-125MBF1644.427.061.565.143.037.637.852.046.1–MobileLLM-LS-125MW8A844.027.560.964.643.137.737.751.045.80.3\hdashline[0.8pt/1pt]MobileLLM-350MBF1651.431.361.068.143.647.241.655.449.9–MobileLLM-350MW8A851.432.161.168.843.147.140.655.149.90.0\hdashline[0.8pt/1pt]MobileLLM-LS-350MBF1651.935.259.668.943.447.243.358.451.0–MobileLLM-LS-350MW8A851.333.859.569.143.747.243.057.050.60.4Appendix FKnowledge DistillationThe results of integrating knowledge distillation (KD) into small model pre-training are presented in TableF. LLaMA-v2 7B models serve as the teacher, and the KD loss is computed using cross-entropy between the logits from the large pre-trained teacher model (i.e., LLaMA-v2 7B)  and the small student network (i.e., 125M or 350M models) :(1) Here,denotes thesample in the current batch withtotal samples in the batch, andrepresents the number of classes, which, in our case, equals the size of the vocabulary.andare the teacher and student networks, respectively.The results in TableFindicate that adding KD loss is comparable or even lower than solely using the next token as labels. However, it’s noteworthy that the training time using KD isslower than training from scratch using labels. All models are trained on 32 A100 80G GPUs with a batch size of 32 for 120k iterations. Consequently, we opt to use labels in our experiments.Table 14:Ablation study on employing LLaMA-v2 7B teacher’s output as soft labels for knowledge distillation (KD) . Results indicate a slight degradation in performance when incorporating KD loss compared to only using hard labels.ModelTraining LossTraining TimeARC-eARC-cBoolQPIQASIQAHellaSwagOBQAWinoGrandeAvg.125M modelLabel29h43.128.958.162.342.334.631.550.143.9125M modelLabel + KD93h41.828.558.561.641.134.532.751.643.8\hdashline[0.8pt/1pt]
350M modelLabel42h50.231.856.967.744.345.840.855.549.1350M modelLabel + KD109h48.731.860.767.443.245.938.953.748.8Appendix GDatasets and BenchmarksMobileLLMis assessed on zero-shot common sense reasoning (BoolQ, PIQA, SIQA, HellaSwag, Winogrande, ARC, OBQA) , question answering (TriviaQA) , and reading comprehension (RACE)  tasks. Additionally, we evaluate our chat models on MT-Bench and AlpacaEval benchmarks. We also generated a API calling dataset to fine-tune and evaluate models for this particular task.G.1Zero-shot Common Sense Reasoning tasksBoolQis a reading comprehension dataset focused on naturally occurring yes/no questions. Each instance includes a question (Q) , an excerpt from a passage (P) , and an answer (A) , with an added explanation for enhanced clarity.PIQA, abbreviated for Physical Interaction: Question Answering, serves as a benchmark for evaluating and studying the capacity of natural language models in comprehending physical commonsense understanding.SIQA, abbreviated for Social Interaction Question Answering, is designed to measure the social and emotional intelligence of computational models through multiple-choice question answering.HellaSwagserves as a benchmark for physically situated commonsense natural language inference. It comprises four-way multiple-choice problems that are considered trivial for humans (>95% accuracy)  but pose a challenge for language models.WinoGrandeis a benchmark for commonsense reasoning. It consists of a set of 273 expert-crafted pronoun resolution problems, deliberately designed to be unsolvable for statistical models relying on selectional preferences or word associations.ARC, the AI2 Reasoning Challenge, is a compilation of 7787 natural science questions. It is divided into a Challenge Set and an Easy Set, with the Challenge Set exclusively comprising questions that were answered incorrectly by both a retrieval-based algorithm and a word co-occurrence algorithm.OBQAis a dataset consisting of approximately 6000 questions designed for open book question answering. The task involves integrating a corpus of provided science facts (open book)  with external broad common knowledge. Providing correct answers necessitates leveraging simple common knowledge beyond the core facts provided.G.2Question Answering TasksTriviaQAis a closed-book question answering benchmark. It encompasses over 650,000 question-answer evidence triples derived by combining 95,000 question-answer pairs authored by Trivia enthusiasts. Each question is supported by an average of six evidence documents.G.3Reading Comprehension TasksRACEis a large-scale reading comprehension dataset with more than 28,000 passages and nearly 100,000 questions. The dataset is collected from English examinations in China, which are designed for middle school and high school students.G.4Chat BenchmarksAlpacaEvalis a single-turn benchmark featuring 805 questions across various topics. In the evaluation of this paper, models generate responses and are scored by GPT-4, with the ultimate metric being the pairwise win-rate against a baseline model (text-davinci-001) .MT-Benchis a multi-turn benchmark comprising 160 questions spanning eight diverse knowledge domains. In this evaluation, models are tasked with answering an initial question and subsequently providing a second response to a predefined follow-up question. Each model response is rated by GPT-4 on a scale from 1 to 10, and the final score is determined by the mean over the two turns.G.5API Calling DatasetThe API calling dataset is generated by instructing a language model to simulate a conversation between a human and an agent. This process instructs the language model to explicitly mention tasks that should be performed using designated APIs throughout the simulated conversation.
Examples of a conversation turn in generated dataset are shown in the following boxes.{"role": "USER", "message": "Can you help me send a SMS message: We are looking forward to seeing you"},{"role": "PARSER", "message": "style_transfer(text = "We are looking forward to seeing you", style = ["sms"], contact = None) "},{"role": "AGENT", "message": "Sure! Here’s a possible SMS message: Hey [name], we are looking forward to seeing you! Wishing you all well."},{"role": "USER", "message": "How did the S&P 500 perform last month?"},{"role": "PARSER", "message": "get_stock_information(company_name = "S&P 500", ticker_symbol = "GSPC", date_time = "last month", financial_metrics = "price") "},{"role": "AGENT", "message": "The S&P 500 closed at $2,977.94 last month, with a monthly return of 2.5%."},{"role": "USER", "message": "Is the San Diego Zoo open during the winter months?"},{"role": "PARSER", "message": "get_local_information(point_of_interest = "San Diego Zoo", attributes = "hours of operation") "},{"role": "AGENT", "message": "The San Diego Zoo is open year-round, but the hours of operation vary depending on the time of year. During the winter months, the zoo is open from 9:00 AM to 5:00 PM."},{"role": "USER", "message": "What is the most recent news concerning aliens?"},{"role": "PARSER", "message": "news(query = "latest alien news") "},{"role": "AGENT", "message": "According to recent news articles, NASA has announced the discovery of a new planet outside of our solar system that has the potential to support alien life. The planet, named Kepler-186f, is located in the constellation Cygnus and is the first known Earth-sized planet to be found in the habitable zone of a star. While this news does not necessarily prove the existence of aliens, it does provide hope for the possibility of finding life beyond Earth. Additionally, a recent study published in the journal Nature has suggested that the universe may be teeming with alien civilizations, with estimates suggesting that there could be tens of billions of intelligent civilizations in the observable universe. However, it’s important to note that these findings are still being researched and debated in the scientific community."},{"role": "USER", "message": "Who won the last Super Bowl?"},{"role": "PARSER", "message": "get_sports_information(match = "", sport_type = "NFL", metric = "winner") "},{"role": "AGENT", "message": "The last Super Bowl was won by the Kansas City Chiefs."},Appendix HApproximate calculation of envisioned future GPU consumption#GPUs =(population size) (individual time999According to the statistics that YouTube users spending an average of 23 hours per month (source)  and Instagram users dedicating 12 hours monthly to the app (source) , we have reason to believe the interaction with LLM can easily surpass the cumulative engagement of these platforms in the future.) (approximating FLOPs/token for 220B GPT-4 model with one expert activated101010https://the-decoder.com/gpt-4-has-a-trillion-parameters/) (token/s) (h) (s)  / ((FLOPs/s, H100 computation capacity ) (h) (s) ) This assumes short sequence lengths (i.e.hundreds of tokens, not tens of thousands of tokens) , where the compute is dominated by linear layers.
For very long sequences, the self-attention computation would dominate, and this would necessitate a more complicated formula.Appendix IChat Generations fromMobileLLMWe show a several examples of chat output generated withMobileLLMwhile responding to various questions.Input:Are there any particular physical benefits to mindful walking, such as improved posture or increased physical fitness?Output(MobileLLM-125M) :Yes, there are several physical benefits to mindful walking. Here are a few:1. Improved Posture: Mindful walking can help improve your posture by promoting good alignment and reducing stress on the joints. This can lead to better posture and reduced back pain.2. Increased Flexibility: Mindful walking can also help increase flexibility in your body, which can reduce the risk of injury and improve overall health.3. Better Sleep: Mindful walking can help you fall asleep more easily and stay asleep longer, which can improve your sleep quality.4. Reduced Stress: Mindful walking can help reduce stress levels by promoting relaxation and reducing tension in the body.5. Enhanced Mental Health: Mindful walking has been shown to have positive effects on mental health, including reducing symptoms of depression and anxiety.Overall, mindful walking is an excellent way to improve your physical health and well-being. It’s a simple and effective way to get moving and enjoy the benefits it offers. If you’re interested in trying mindful walking, I would recommend starting with shorter walks and gradually increasing the duration over time. You may also want to consider incorporating mindfulness into your daily routine to further enhance your experience.In conclusion, mindful walking is a great way to improve your physical health and well-being. With its numerous physical benefits, it’s easy to see why it’s becoming increasingly popular among people looking for a fun and sustainable way to get moving. So go ahead and give it a try!Output(MobileLLM-350M) :Yes, there are several physical benefits of mindful walking. Firstly, it can improve your posture by reducing the amount of tension in your muscles and joints. Secondly, it can increase your physical fitness by improving your cardiovascular health, strengthening your core muscles, and increasing your flexibility. Additionally, mindful walking can help you to stay more focused on the present moment, which can lead to better sleep quality and reduced stress levels. Finally, it can also help you to develop a greater sense of awareness and appreciation for the world around you, which can enhance your overall well-being.Overall, mindful walking is a great way to improve your physical health and well-being, while also promoting relaxation and mindfulness. Whether you’re looking to reduce stress, improve your posture, or simply enjoy a peaceful walk, mindful walking can be a rewarding and fulfilling experience.If you’re interested in trying out mindful walking, there are many resources available that can guide you through the process. Some popular options include guided meditations, apps that offer mindful walking exercises, and classes that teach you how to practice mindful walking. With a little bit of effort and dedication, anyone can start incorporating mindful walking into their daily routine and reap the many benefits it offers.In conclusion, mindful walking is a simple yet powerful tool that can help you to improve your physical health, reduce stress, and cultivate a deeper sense of awareness and appreciation for the world around you. So why not give it a try and see what kind of benefits you can achieve? Your body and mind will thank you!Output(llama-v2-7b-chat) :Mindful walking can have a range of physical benefits, including:1. Improved posture: By focusing on the sensation of each step and the movement of the body, mindful walking can help improve posture and reduce slouching or hunching.2. Increased physical fitness: Mindful walking can help improve cardiovascular fitness, increase muscle strength and endurance, and boost overall physical fitness.3. Better balance and coordination: By paying attention to the body’s position in space and the movement of the legs and feet, mindful walking can help improve balance and coordination.4. Reduced stress and tension: Mindful walking can help reduce stress and tension in the body, which can lead to improved physical relaxation and well-being.5. Improved flexibility: By focusing on the movement of the body and the sensation of each step, mindful walking can help improve flexibility and range of motion.6. Boosted immune system: Studies have shown that mindfulness practices, including mindful walking, can help boost the immune system and reduce inflammation.7. Improved sleep: Mindful walking can help improve sleep quality and duration by reducing stress and promoting relaxation.
8. Reduced pain: Mindful walking can help reduce pain by focusing on the sensation of each step and the movement of the body, rather than the pain itself.9. Improved mood: Mindful walking can help improve mood and reduce symptoms of depression and anxiety by promoting relaxation and reducing stress.It’s important to note that the physical benefits of mindful walking may vary depending on the individual and the frequency and duration of the practice. However, with regular practice, mindful walking can have a range of positive effects on physical health and well-being.Input:How do I wrap a present neatly?Output(MobileLLM-125M) :1. Choose the right size gift: The size of the present you want to give will depend on your budget and personal preference. If you are looking for a small gift, consider buying a small box or card with a single-use plastic bag.2. Prepare the present: Make sure all the items in the present are clean and tidy. You can use a cleaning kit or a tissue paper to remove any dirt or debris from the present.3. Arrange the presents: Place the present in a place where it is easy to see and touch. Use a pencil to draw a line around the present and arrange it in a way that makes sense to you.4. Add decorations: Decorate the present with stickers, ribbons, or other decorative items. This will add some personality to the present and make it more visually appealing.5. Enjoy the present: Take time to enjoy the present and let yourself feel the warmth and comfort of the present. Smile at yourself and take a moment to appreciate the present.Remember, the most important thing is to have fun and enjoy the present! Happy gift-giving!If you’re feeling adventurous, try making a surprise gift for someone special by wrapping the present in a cute and colorful gift basket. Or, if you’re feeling adventurous, try making a gift for a friend who needs a little extra help with their homework. Whatever you choose, remember to be creative and have fun with your present!With these tips, you’ll be able to create a beautiful and thoughtful present that will last for years to come. Happy gift-giving!Output(MobileLLM-350M) :1. Start by wrapping the present in a decorative gift wrap or ribbon.2. Use a small amount of glue or tape to secure the gift wrap or ribbon to the surface of the present.3. Place the gift wrap or ribbon around the present, making sure it is centered and evenly distributed.4. Use a small amount of glue or tape to secure the gift wrap or ribbon to the surface of the present.5. Once the gift wrap or ribbon is securely attached, you can add any additional decorations or embellishments to the present.6. If you want to add a personal touch, you can write a message or a special message on the gift wrap or ribbon.7. Finally, enjoy your beautifully wrapped present!Remember, wrapping a gift is a fun and creative way to show your loved one how much you care. Have fun and enjoy the process!Output(llama-v2-7b-chat) :Wrapping a present neatly can be a bit of a challenge, but with a few simple tips and tricks, you can create a beautifully wrapped gift that your recipient will love. Here are some tips to help you wrap a present neatly:1. Use the right wrapping paper: Choose a wrapping paper that complements the gift and is the right size for the present. You can also use a patterned paper or a solid color, depending on your preference.2. Measure the gift: Before you start wrapping, measure the gift to make sure you have enough wrapping paper. You can also use a piece of scrap paper to measure the gift and mark the edges with a pencil.3. Use a flat surface: Make sure you have a flat surface to work on, such as a table or countertop. This will help you keep the wrapping paper flat and prevent it from wrinkling.4. Start with the edges: Begin by folding the edges of the wrapping paper over the gift, making sure they are even and smooth. You can use tape to secure the edges if needed.5. Roll the gift: Once you have the edges in place, roll the gift into the center of the wrapping paper. Make sure the gift is securely in place before moving on to the next step.6. Tuck in the edges: As you wrap the gift, tuck the edges of the wrapping paper under the gift, smoothing them out as you go. Make sure the edges are even and smooth, and use tape to secure them if needed.7. Add a ribbon or bow: Once you have the gift wrapped, add a ribbon or bow to the top. This will not only make the gift look more festive, but it will also help to keep the wrapping paper in place.8. Practice makes perfect: Wrapping a present neatly takes a bit of practice, so don’t be discouraged if it doesn’t come out perfectly the first time. Keep practicing, and you’ll get the hang of it in no time!By following these tips, you can create a beautifully wrapped present that your recipient will love. Happy wrapping!Input:What are the best exercises for beginners?Output(MobileLLM-125M) :1. Push-ups: This is a classic exercise that can be done anywhere, anytime. It’s easy to do and requires no equipment.2. Dumbbell Rows: These rows are great for building strength in your legs and core.3. Squats: Squats are a great way to build strength in your legs and core.4. Lunges: Lunges are a great way to improve balance and stability.5. Plank: A plank is a great way to build strength in your back and core.6. Jumping Jacks: Jumping jacks are a fun and effective way to build strength in your legs and core.7. Burpees: Burpees are a great way to build strength in your legs and core.8. Mountain Climbers: Mountain climbers are a great way to build strength in your legs and core.9. Deadlifts: Deadlifts are a great way to build strength in your legs and core.10. Tricep Dips: Tricep dips are a great way to build strength in your triceps.Remember, it’s important to start with simple exercises and gradually increase the intensity as you get stronger. Don’t push yourself too hard or you may find it difficult to maintain proper form. With consistent practice, you will see improvements in your strength and overall fitness.If you have any questions about exercises or healthy living, don’t hesitate to talk to a doctor or physical therapist. They can help you develop a personalized plan that works best for you.In conclusion, there are many exercises that can be done at home, but it’s important to start slowly and gradually increase the intensity over time. Remember to listen to your body and take breaks when needed. And most importantly, enjoy the process!Output(MobileLLM-350M) :Beginners should start with exercises that are easy to do and don’t require a lot of strength or endurance. Here are some exercises that are great for beginners:1. Push-ups: This is a great exercise for beginners because it’s a simple exercise that can be done anywhere. Start with a push-up on your knees and then lower yourself down to the ground.2. Squats: Squats are a great exercise for building leg strength and endurance. Start with a squat on your knees and then lower yourself down to the ground.3. Lunges: Lunges are a great exercise for building leg strength and endurance. Start with a lunge on your front foot and then lower yourself down to the ground.4. Plank: Planks are a great exercise for building core strength and stability. Start with a plank on your forearms and then lower yourself down to the ground.5. Bicycle crunches: Bicycle crunches are a great exercise for building upper body strength and stability. Start with a bicycle crunch on your upper back and then lower yourself down to the ground.6. Plank with leg lifts: This exercise is great for building leg strength and stability. Start with a plank on your forearms and then lift your legs off the ground and hold for a few seconds before lowering them back down.7. Jumping jacks: Jumping jacks are a great exercise for building cardiovascular endurance and strength. Start with a jumping jack on your feet and then jump up and spread your legs.Remember to start with a weight that is comfortable for you and gradually increase the weight as you get stronger. It’s also important to warm up before exercising and to stretch after your workout to prevent injury.In conclusion, beginners should start with exercises that are easy to do and don’t require a lot of strength or endurance. These exercises are great for building leg strength and endurance, and can be done anywhere. Remember to start with a weight that is comfortable for you and gradually increase the weight as you get stronger. Warming up and stretching after your workout are also important to prevent injury.Following these exercises can help you build strength, endurance, and improve your overall fitness level. Remember to always consult with a healthcare professional before starting any new exercise program.Output(llama-v2-7b-chat) :Exercise is an essential part of a healthy lifestyle, but for beginners, it can be intimidating and overwhelming. Here are some of the best exercises for beginners:1. Bodyweight exercises: These exercises are great for beginners because they don’t require any equipment. Examples include push-ups, squats, lunges, and planks.2. Yoga: Yoga is a low-impact exercise that can help improve flexibility, balance, and strength. It’s also great for reducing stress and improving overall well-being.3. Brisk walking: Walking is a simple and accessible form of exercise that can be done anywhere. It’s a great way to get your heart rate up and improve cardiovascular health.4. Swimming: Swimming is a low-impact exercise that can help improve cardiovascular health and strengthen muscles. It’s also easy on the joints, making it a great option for beginners.5. Cycling: Cycling is another low-impact exercise that can be done indoors or outdoors. It’s a great way to improve cardiovascular health and build leg strength.6. Resistance band exercises: Resistance bands are lightweight and portable, making them easy to use at home or on the go. They can help improve strength and flexibility in the upper body, lower body, and core.7. Pilates: Pilates is a form of exercise that focuses on core strength, flexibility, and body control. It’s a great option for beginners because it’s low-impact and can help improve posture and balance.8. Dance-based workouts: Dance-based workouts, such as Zumba or Hip Hop Abs, are fun and energetic ways to get in shape. They often involve aerobic exercise and strength training, making them a great option for beginners.9. Jumping rope: Jumping rope is a high-intensity exercise that can help improve cardiovascular health and burn calories. It’s also easy to do at home with a simple jump rope.10. Kettlebell exercises: Kettlebells are small weights that can be used to perform a variety of exercises, including swings, squats, and presses. They’re a great way to improve strength and cardiovascular health.Remember to always consult with a doctor before starting any new exercise program, especially if you’re a beginner. It’s also important to start slowly and gradually increase the intensity and duration of your workouts as you become more comfortable.
body
MobileLLM: Optimizing Sub-billion Parameter Language Modelsfor On-Device Use Cases
3Experiments
2402.14905
4Related WorkThe excellent performance of LLMs has fostered its wide applications.
Considering computational cost and energy consumption of LLMs, an new stream of research direction have emerged to downsize LLMs to enable on-device inference. These methods include:Model Compression.Numerous model compression methods are developed for LLMs, including pruning, sparsity, and quantization. Our research is complementary to these techniques. As also substantiated in Section3.4, our methodology is compatible with quantization.Small Model Design.A limited number of studies have explored compact model architectures, such as TinyLLaMA. However, even the smallest TinyLLaMA exceeds 1 billion parameters, making them still prohibitive for many on-device applications. Some research proposes large model architectures alongside their smaller LLM variants in a model familyor a analytical suite containing small LLM variants. However, these models are not optimized under the constraint of sub-billion parameters and therefore may not be optimal.Neural Architecture Search.NAS has garnered substantial attention in the realm of convolutional neural networks, particularly in the context of vision tasks. In contrast, within the transformer domain, the prevailing consensus posits that the model architecture exerts minimal influence on accuracy, provided the total number of parameters remains consistent.
Only a limited number of studies have developed NAS algorithm for language transformer, targeting at BERT models. Our current investigation, focusing on the interplay between depth and width, can be conceptualized as a meticulous grid search within the depth space. The outcomes of that study challenge the prevalent orthodoxy surrounding scaling laws, proposing that deep and thin architectures demonstrate higher performance for compact LLMs.Weight Sharing.Weight sharing is an intuitive strategy for optimizing model weight utilization within fixed parameter constraints. While the OPT familyand subsequent worksleverage weight sharing between input and output embeddings, limited research has explored weight sharing for intermediate layers in transformers. Prior efforts often entail specialized designs for shared layers. In contrast, our contribution highlights a more straightforward yet effective way of simply repeating transformer blocks, yielding improved accuracy with a fixed model size and minimal latency increase.Efficient Attention and Implementation.In the realm of efficient transformer design, much research has focused on optimizing attention computation through methods like low-rank approximationand sparse attention. Another line of work explores hardware scheduling and weight movement, exemplified by works such as FlashAttentionand FlexGen. In contrast, our primary goal is to optimize model size without introducing new attention computation or efficient hardware implementation methods.5ConclusionThis study focuses on optimizing sub-billion scale models for on-device applications. Our findings indicate that, for smaller models, prioritizing depth over width enhances model performance. Furthermore, by leveraging advanced weight-sharing techniques, including embedding sharing, grouped query attention, and block-wise weight sharing, we achieve significant enhancements in weight utilization within storage-constrained scenarios. The resulting models denoted asMobileLLMexhibit substantial advancements in zero-shot commonsense reasoning, question answering, and reading comprehension tasks compared to previous SoTA methods. Last but not least, we demonstrate the effectiveness of the fine-tunedMobileLLMmodels in two prevalent on-device use cases: chat and API calling, underscoring their adeptness in handling such tasks.AcknowledgmentWe thank Hansong Zhang for his valuable contribution in setting up latency measurement environment on iOS, and the full support from PyTorch Edge team.Broader ImpactThis paper advocates for the adoption of sub-billion scale large language models in on-device applications, aiming to mitigate energy consumption during LLM inference. The proposed approach is promising in alleviating computational costs associated with LLM deployment.ReferencesAinslie et al. (2023) Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y., Lebron, F., and Sanghai, S.GQA: Training generalized multi-query transformer models from multi-head checkpoints.InEMNLP, 2023.Biderman et al. (2023) Biderman, S., Schoelkopf, H., Anthony, Q. G., Bradley, H., O’Brien, K., Hallahan, E., Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., et al.Pythia: A suite for analyzing large language models across training and scaling.InInternational Conference on Machine Learning, pp.  2397–2430. PMLR, 2023.Bisk et al. (2020) Bisk, Y., Zellers, R., Gao, J., Choi, Y., et al.Piqa: Reasoning about physical commonsense in natural language.InProceedings of the AAAI conference on artificial intelligence, volume 34, pp.  7432–7439, 2020.Black et al. (2022) Black, S., Biderman, S., Hallahan, E., Anthony, Q., Gao, L., Golding, L., He, H., Leahy, C., McDonell, K., Phang, J., et al.Gpt-neox-20b: An open-source autoregressive language model.arXiv preprint arXiv:2204.06745, 2022.Chowdhery et al. (2023) Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al.Palm: Scaling language modeling with pathways.Journal of Machine Learning Research, 24(240) :1–113, 2023.Clark et al. (2019) Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., and Toutanova, K.Boolq: Exploring the surprising difficulty of natural yes/no questions.arXiv preprint arXiv:1905.10044, 2019.Clark et al. (2018) Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O.Think you have solved question answering? try arc, the ai2 reasoning challenge.arXiv preprint arXiv:1803.05457, 2018.Dao et al. (2022) Dao, T., Fu, D., Ermon, S., Rudra, A., and Ré, C.Flashattention: Fast and memory-efficient exact attention with io-awareness.Advances in Neural Information Processing Systems, 35:16344–16359, 2022.Dauphin et al. (2017) Dauphin, Y. N., Fan, A., Auli, M., and Grangier, D.Language modeling with gated convolutional networks.InInternational conference on machine learning, pp.  933–941. PMLR, 2017.Dettmers et al. (2022) Dettmers, T., Lewis, M., Belkada, Y., and Zettlemoyer, L.Llm. int8 () : 8-bit matrix multiplication for transformers at scale.arXiv preprint arXiv:2208.07339, 2022.Dey et al. (2023) Dey, N., Gosal, G., Khachane, H., Marshall, W., Pathria, R., Tom, M., Hestness, J., et al.Cerebras-gpt: Open compute-optimal language models trained on the cerebras wafer-scale cluster.arXiv preprint arXiv:2304.03208, 2023.Frantar & Alistarh (2023) Frantar, E. and Alistarh, D.Sparsegpt: Massive language models can be accurately pruned in one-shot.InInternational Conference on Machine Learning, pp.  10323–10337. PMLR, 2023.Frantar et al. (2022) Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D.Gptq: Accurate post-training quantization for generative pre-trained transformers.arXiv preprint arXiv:2210.17323, 2022.Ganesan et al. (2021) Ganesan, V., Ramesh, G., and Kumar, P.Supershaper: Task-agnostic super pre-training of bert models with variable hidden dimensions.arXiv preprint arXiv:2110.04711, 2021.Google (2023) Google.Pixel 8 pro tech specs.https://store.google.com/gb/product/pixel_8_pro_specs, 2023.Guo et al. (2020) Guo, Z., Zhang, X., Mu, H., Heng, W., Liu, Z., Wei, Y., and Sun, J.Single path one-shot neural architecture search with uniform sampling.InComputer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XVI 16, pp.  544–560. Springer, 2020.Han et al. (2016) Han, S., Liu, X., Mao, H., Pu, J., Pedram, A., Horowitz, M. A., and Dally, W. J.Eie: Efficient inference engine on compressed deep neural network.ACM SIGARCH Computer Architecture News, 44(3) :243–254, 2016.Hinton et al. (2015) Hinton, G., Vinyals, O., Dean, J., et al.Distilling the knowledge in a neural network.arXiv preprint arXiv:1503.02531, 2(7) , 2015.Hristov (2022) Hristov, V.A16 bionic explained: what’s new in apple’s pro-grade mobile chip?https://www.phonearena.com/news/A16-Bionic-explained-whats-new_id142438, 2022.Jawahar et al. (2023) Jawahar, G., Yang, H., Xiong, Y., Liu, Z., Wang, D., Sun, F., Li, M., Pappu, A., Oguz, B., Abdul-Mageed, M., et al.Mixture-of-supernets: Improving weight-sharing supernet training with architecture-routed mixture-of-experts.arXiv preprint arXiv:2306.04845, 2023.Joshi et al. (2017) Joshi, M., Choi, E., Weld, D. S., and Zettlemoyer, L.Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension.arXiv preprint arXiv:1705.03551, 2017.Kaplan et al. (2020) Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D.Scaling laws for neural language models.arXiv preprint arXiv:2001.08361, 2020.Katharopoulos et al. (2020) Katharopoulos, A., Vyas, A., Pappas, N., and Fleuret, F.Transformers are rnns: Fast autoregressive transformers with linear attention.InInternational conference on machine learning, pp.  5156–5165. PMLR, 2020.Kim et al. (2023) Kim, S., Hooper, C., Gholami, A., Dong, Z., Li, X., Shen, S., Mahoney, M. W., and Keutzer, K.Squeezellm: Dense-and-sparse quantization.arXiv preprint arXiv:2306.07629, 2023.Kingma & Ba (2014) Kingma, D. P. and Ba, J.Adam: A method for stochastic optimization.arXiv preprint arXiv:1412.6980, 2014.Kitaev et al. (2020) Kitaev, N., Kaiser, Ł., and Levskaya, A.Reformer: The efficient transformer.arXiv preprint arXiv:2001.04451, 2020.Lai et al. (2017) Lai, G., Xie, Q., Liu, H., Yang, Y., and Hovy, E.Race: Large-scale reading comprehension dataset from examinations.arXiv preprint arXiv:1704.04683, 2017.Li et al. (2023) Li, X., Zhang, T., Dubois, Y., Taori, R., Gulrajani, I., Guestrin, C., Liang, P., and Hashimoto, T. B.Alpacaeval: An automatic evaluator of instruction-following models.https://github.com/tatsu-lab/alpaca_eval, 2023.Liu et al. (2023a) Liu, J., Gong, R., Wei, X., Dong, Z., Cai, J., and Zhuang, B.Qllm: Accurate and efficient low-bitwidth quantization for large language models.arXiv preprint arXiv:2310.08041, 2023a.Liu et al. (2023b) Liu, S.-y., Liu, Z., Huang, X., Dong, P., and Cheng, K.-T.Llm-fp4: 4-bit floating-point quantized transformers.arXiv preprint arXiv:2310.16836, 2023b.Liu et al. (2023c) Liu, Z., Oguz, B., Zhao, C., Chang, E., Stock, P., Mehdad, Y., Shi, Y., Krishnamoorthi, R., and Chandra, V.Llm-qat: Data-free quantization aware training for large language models.arXiv preprint arXiv:2305.17888, 2023c.Malladi et al. (2012) Malladi, K. T., Lee, B. C., Nothaft, F. A., Kozyrakis, C., Periyathambi, K., and Horowitz, M.Towards energy-proportional datacenter memory with mobile dram.ACM SIGARCH Computer Architecture News, 40(3) :37–48, 2012.Mihaylov et al. (2018) Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A.Can a suit of armor conduct electricity? a new dataset for open book question answering.arXiv preprint arXiv:1809.02789, 2018.Peng et al. (2023) Peng, B., Alcaide, E., Anthony, Q., Albalak, A., Arcadinho, S., Cao, H., Cheng, X., Chung, M., Grella, M., GV, K. K., et al.Rwkv: Reinventing rnns for the transformer era.arXiv preprint arXiv:2305.13048, 2023.Reid et al. (2021) Reid, M., Marrese-Taylor, E., and Matsuo, Y.Subformer: Exploring weight sharing for parameter efficiency in generative transformers.arXiv preprint arXiv:2101.00234, 2021.Roy et al. (2021) Roy, A., Saffar, M., Vaswani, A., and Grangier, D.Efficient content-based sparse attention with routing transformers.Transactions of the Association for Computational Linguistics, 9:53–68, 2021.Sakaguchi et al. (2021) Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y.Winogrande: An adversarial winograd schema challenge at scale.Communications of the ACM, 64(9) :99–106, 2021.Sap et al. (2019) Sap, M., Rashkin, H., Chen, D., LeBras, R., and Choi, Y.Socialiqa: Commonsense reasoning about social interactions.arXiv preprint arXiv:1904.09728, 2019.Scao et al. (2022) Scao, T. L., Fan, A., Akiki, C., Pavlick, E., Ilić, S., Hesslow, D., Castagné, R., Luccioni, A. S., Yvon, F., Gallé, M., et al.Bloom: A 176b-parameter open-access multilingual language model.arXiv preprint arXiv:2211.05100, 2022.Shen et al. (2022) Shen, Z., Liu, Z., and Xing, E.Sliced recursive transformer.InEuropean Conference on Computer Vision, pp.  727–744. Springer, 2022.Sheng et al. (2023) Sheng, Y., Zheng, L., Yuan, B., Li, Z., Ryabinin, M., Chen, B., Liang, P., Ré, C., Stoica, I., and Zhang, C.Flexgen: High-throughput generative inference of large language models with a single gpu.InInternational Conference on Machine Learning, pp.  31094–31116. PMLR, 2023.Sun et al. (2023) Sun, M., Liu, Z., Bair, A., and Kolter, J. Z.A simple and effective pruning approach for large language models.arXiv preprint arXiv:2306.11695, 2023.Tan & Le (2019) Tan, M. and Le, Q.Efficientnet: Rethinking model scaling for convolutional neural networks.InInternational conference on machine learning, pp.  6105–6114. PMLR, 2019.Taylor et al. (2022) Taylor, R., Kardas, M., Cucurull, G., Scialom, T., Hartshorn, A., Saravia, E., Poulton, A., Kerkez, V., and Stojnic, R.Galactica: A large language model for science.arXiv preprint arXiv:2211.09085, 2022.Timiryasov & Tastet (2023) Timiryasov, I. and Tastet, J.-L.Baby llama: knowledge distillation from an ensemble of teachers trained on a small dataset with no performance penalty.arXiv preprint arXiv:2308.02019, 2023.Touvron et al. (2023a) Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al.Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023a.Touvron et al. (2023b) Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Ferrer, C. C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshorn, A., Hosseini, S., Hou, R., Inan, H., Kardas, M., Kerkez, V., Khabsa, M., Kloumann, I., Korenev, A., Koura, P. S., Lachaux, M.-A., Lavril, T., Lee, J., Liskovich, D., Lu, Y., Mao, Y., Martinet, X., Mihaylov, T., Mishra, P., Molybog, I., Nie, Y., Poulton, A., Reizenstein, J., Rungta, R., Saladi, K., Schelten, A., Silva, R., Smith, E. M., Subramanian, R., Tan, X. E., Tang, B., Taylor, R., Williams, A., Kuan, J. X., Xu, P., Yan, Z., Zarov, I., Zhang, Y., Fan, A., Kambadur, M., Narang, S., Rodriguez, A., Stojnic, R., Edunov, S., and Scialom, T.Llama 2: Open foundation and fine-tuned chat models, 2023b.Wang et al. (2020) Wang, S., Li, B. Z., Khabsa, M., Fang, H., and Ma, H.Linformer: Self-attention with linear complexity.arXiv preprint arXiv:2006.04768, 2020.Wu et al. (2019) Wu, B., Dai, X., Zhang, P., Wang, Y., Sun, F., Wu, Y., Tian, Y., Vajda, P., Jia, Y., and Keutzer, K.Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search.InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.  10734–10742, 2019.Xia et al. (2023a) Xia, H., Zheng, Z., Li, Y., Zhuang, D., Zhou, Z., Qiu, X., Li, Y., Lin, W., and Song, S. L.Flash-llm: Enabling cost-effective and highly-efficient large generative model inference with unstructured sparsity.arXiv preprint arXiv:2309.10285, 2023a.Xia et al. (2023b) Xia, M., Gao, T., Zeng, Z., and Chen, D.Sheared llama: Accelerating language model pre-training via structured pruning.arXiv preprint arXiv:2310.06694, 2023b.Xiao et al. (2023) Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., and Han, S.Smoothquant: Accurate and efficient post-training quantization for large language models.InInternational Conference on Machine Learning, pp.  38087–38099. PMLR, 2023.Xiong et al. (2021) Xiong, Y., Zeng, Z., Chakraborty, R., Tan, M., Fung, G., Li, Y., and Singh, V.Nyströmformer: A nyström-based algorithm for approximating self-attention.InProceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp.  14138–14148, 2021.Xu et al. (2021) Xu, J., Tan, X., Luo, R., Song, K., Li, J., Qin, T., and Liu, T.-Y.Nas-bert: task-agnostic and adaptive-size bert compression with neural architecture search.InProceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp.  1933–1943, 2021.Yao et al. (2022) Yao, Z., Yazdani Aminabadi, R., Zhang, M., Wu, X., Li, C., and He, Y.Zeroquant: Efficient and affordable post-training quantization for large-scale transformers.Advances in Neural Information Processing Systems, 35:27168–27183, 2022.Zellers et al. (2019) Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y.Hellaswag: Can a machine really finish your sentence?arXiv preprint arXiv:1905.07830, 2019.Zhang et al. (2022) Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al.Opt: Open pre-trained transformer language models.arXiv preprint arXiv:2205.01068, 2022.Zheng et al. (2023) Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., et al.Judging llm-as-a-judge with mt-bench and chatbot arena.arXiv preprint arXiv:2306.05685, 2023.Zoph & Le (2016) Zoph, B. and Le, Q. V.Neural architecture search with reinforcement learning.arXiv preprint arXiv:1611.01578, 2016.AppendixAppendix AImpact of Each Design ChoiceThis section presents comprehensive tabulated results for theimproving sub-billion scale LLM designexperiments, at the model sizes of 125M and 350M. Looking at the results in TableA, transitioning from the traditional Feedforward Network ()  to SwiGLU yields a accuracy improvement of 1.3% for both model sizes. Further increasing the model depth enhances accuracy by 0.9%/1.1% for 125M/350M models, respectively. Then, introducing input and output embedding sharing achieves a parameter reduction of approximately 10%, while with only marginal accuracy drops of 0.2% for the 125M and 0.6% for 350M models. Additionally, following in findings in SectionC, we incorporate grouped query attention with head dimension equals to 64, and head number to be near 4to the kv head number, while increasing the embedding dimension to preserve model size. This modification further results in a performance boost of 0.4%/0.7% for 125M/350M models. Combining these techniques establishes a strong baseline network denoted asMobileLLM. Finally, the immediate block-wise weight-sharing technique contributes an additional accuracy gain of 1.1% for models trained on 0.25 trillion tokens, and resulting in the modelMobileLLM-LS. Final models includingMobileLLMandMobileLLM-LSare trained with 1 trillion tokens.Table 8:Ablation study on the impact of each design choice to the model accuracy on zero-shot common sense reasoning tasks. Corresponding to the bar chart in Figure3. Here, L, H,denotes number of layers, heads, kv-heads, respectively and dim denotes embedding dimension.TechniquesLHDim#Params(M) ARC-eARC-cBoolQPIQASIQAHellaSwagOBQAWinoGrandeAvg.125M\hdashline[0.8pt/1pt]
Baseline model121212768134.141.325.257.562.041.931.131.250.842.6+ SwiGLU in FFN121212768134.143.128.958.162.342.334.631.550.143.9+ Use deep-thin structure3088512135.043.626.158.062.542.636.537.551.544.8+ Embedding share3088512118.644.426.056.262.843.135.936.052.644.6+ Multi-query attention3093576124.645.527.758.364.641.936.435.450.445.0(Train on 1T token) 3093576124.643.927.160.265.342.438.939.553.146.3+ Layer sharing3093576124.644.427.061.565.143.037.637.852.046.1(Train on 1T token) 3093576124.645.828.760.465.742.939.541.152.147.0350M\hdashline[0.8pt/1pt]
Baseline model1520201280376.850.327.653.868.144.142.640.152.447.4+ SwiGLU in FFN1520201280386.749.230.659.167.744.343.241.054.248.7+ Use deep-thin structure321414896380.350.731.459.467.843.346.243.856.249.8+ Embedding share321414896351.649.932.060.367.943.247.038.954.849.2+ Multi-query attention32155960345.351.431.361.068.143.647.241.655.449.9(Train on 1T token) 32155960345.353.833.562.468.644.749.640.057.651.3+ Layer sharing32155960345.351.935.259.668.943.447.243.358.451.0(Train on 1T token) 32155960345.354.432.562.869.844.150.645.857.252.1Appendix BDepth vs WidthWe provide network depth versus width exploration results on zero-shot reasoning tasks in TableB, as well as results on question answering and reading comprehension tasks in TableB. The findings indicate that shallow architectures with fewer than 10 layers perform poorly in reasoning or handling comprehension tasks. Models with 10-20 layers exhibit improved accuracy, while further increasing depth continues to provide significant benefits across all three tasks. Notably, the optimal depth is found to be near 30 layers for sub-billion scale models.Table 9:Ablation study on depth versus width in architecture design, illustrated in Figure4(a) (b) . For compact models, prioritizing depth over width yields superior performance, assessed through zero-shot common sense reasoning tasks.#Layer#HeadsDim#Params(M) ARC-eARC-cBoolQPIQASIQAHellaSwagOBQAWinoGrandeAvg.4201280163.242.026.161.061.541.332.931.449.843.36161024142.642.225.551.062.641.633.532.150.042.3814896138.143.625.548.661.840.934.037.552.043.01212768134.143.128.958.162.342.334.631.550.143.91810640132.441.025.559.362.641.634.236.750.143.9249576132.443.627.059.763.842.036.233.752.244.8308512135.043.626.158.062.542.636.537.551.544.8427448134.643.326.157.863.341.736.335.952.044.5626384134.344.126.759.064.742.436.233.151.344.7\hdashline[0.8pt/1pt]
5322048388.047.928.862.265.242.838.139.052.647.110241536381.449.129.660.667.144.042.741.254.448.612221408379.949.931.659.168.242.644.043.753.849.115201280386.749.230.659.167.744.343.241.054.248.719181152376.350.830.956.369.143.845.239.654.548.824161024373.850.733.858.868.643.545.040.054.949.42815960371.151.733.257.667.942.946.037.753.948.93214896380.350.731.459.467.843.346.243.856.249.84612768374.749.730.559.568.145.144.943.455.549.66610640376.250.531.861.067.443.846.040.155.649.5Table 10:Ablation study on depth vs. width in architecture design on TQA and RACE datasets, depicted in Figure4(c-f) .TQA(F1 score) RACE(Acc) #Layer#HeadsDim#Params(M) 1-shot5-shot64-shotmiddlehigh4201280163.22.03.93.933.226.06161024142.63.54.94.629.225.5814896138.14.25.24.729.224.31212768134.15.16.16.335.627.31810640132.44.76.85.234.327.4249576132.45.97.27.335.328.6308512135.06.26.86.934.227.9427448134.66.17.27.234.428.6626384134.36.07.07.338.928.9\hdashline[0.8pt/1pt]
5322048388.05.67.88.638.129.010241536381.49.011.812.740.630.012221408379.911.712.913.342.431.515201280386.710.713.114.242.630.619181152376.311.313.313.542.632.424161024373.811.813.514.443.031.92815960371.112.513.614.942.832.33214896380.312.614.515.344.232.24612768374.713.015.415.644.731.46610640376.212.515.115.744.032.5Appendix CNumber of Heads and Key-Value HeadsWe provide detailed experimental results assessing the impact of the number of attention heads and key-value heads on zero-shot reasoning accuracy in TableC. Our study involves two baseline architectures: a 8-layer 125M model with a embedding dimension of 896, and a 15-layer 350M model with a embedding dimension of 1280. We conduct head size sweeps in {8, 16, 32}. The findings shown in TableCindicate that using 16 heads, with a head dimension close to 64, and 4 key-value heads, yields the best accuracy and memory trade-off. This setting serves as a guiding principle in our model architecture design.Table 11:Ablation study investigating the impact of the number of attention heads and key-value heads.Model#Heads#KV-Heads#Params(M) ARC-eARC-cBoolQPIQASIQAHellaSwagOBQAWinoGrandeAvg.3232138.142.126.958.462.242.133.836.752.444.33216131.741.226.457.762.542.333.334.252.943.8328128.542.627.361.161.941.932.235.052.044.2324126.843.126.859.862.741.432.534.451.144.0322126.039.826.759.459.442.031.332.652.943.0321125.641.024.359.160.841.231.435.452.043.1125M1616138.141.625.761.162.443.134.436.951.644.6# layers=8168131.742.426.460.763.441.933.534.751.544.3dim=896164128.542.525.662.362.441.833.035.954.544.7162126.841.725.356.961.742.032.932.654.543.5161126.040.426.361.863.241.732.034.050.443.788138.141.425.058.361.741.733.335.953.243.884131.743.328.258.361.842.833.830.953.044.082128.540.726.258.161.241.632.834.851.543.481126.842.524.859.462.342.032.036.351.343.8\hdashline[0.8pt/1pt]3232386.748.630.459.767.243.944.040.953.948.63216362.148.931.657.668.443.443.838.654.948.4328349.848.333.161.067.242.642.139.053.948.4324343.747.229.859.467.243.542.542.554.148.3322340.647.630.362.466.942.641.638.652.047.7321339.048.527.356.367.142.940.936.753.346.6350M1616386.750.830.662.368.643.545.143.852.449.6# layers=15168362.148.530.759.467.343.843.841.353.348.5dim=1280164349.849.930.660.069.243.544.241.855.849.4162343.749.328.455.067.342.742.640.354.547.5161340.649.229.358.867.443.542.139.952.847.988386.751.033.258.267.043.943.942.654.749.384362.150.031.360.266.042.743.740.953.748.682349.849.330.459.967.942.843.538.953.848.381343.748.027.761.567.143.142.537.354.747.8Appendix DLayer Sharing Number AblationTable 12:Ablation study on the impact of varying layer repetition numbers.Model SizeRepeat TimesARC-eARC-cBoolQPIQASIQAHellaSwagOBQAWinoGrandeAvg.125Mbaseline41.625.761.162.443.134.436.951.644.6block-wise share repeat243.927.961.564.341.535.535.150.245.0block-wise share repeat343.327.660.963.042.335.634.152.945.0block-wise share repeat442.026.662.364.441.536.235.853.945.3\hdashline[0.8pt/1pt]350Mbaseline50.830.662.368.643.545.143.852.449.6block-wise share repeat251.530.859.668.243.947.744.755.050.2block-wise share repeat349.632.057.368.443.947.040.656.249.4block-wise share repeat452.232.660.069.044.547.641.855.250.4We extended our investigation to determine the optimal number of layer repetitions. The experiment involved the 8-layer 125M model with a embedding dimension of 896 and the 15-layer 350M model with a embedding dimension of 1280. The results in TableDdemonstrate that when we double the layer number and with each two transformer blocks sharing weights, the accuracy is enhanced by 0.4-0.6%. However, as we further triple or quadruple the layer repetition times, this accuracy enhancement effect diminishes. Consequently, in our experiments, we adopt a configuration where every two blocks share weights, effectively doubling the total number of layers.Appendix ECompatibility with QuantizationThis section explores the compatibility of quantization with the proposed model architecture and layer sharing. We employ a straightforward per-token min-max quantization, quantizing both weight and activation to 8-bit using post-training quantization (PTQ) . Experiments are conducted on bothMobileLLMandMobileLLM-LSwith model size of 125M and 350M, trained on 0.25T tokens. Results in TableEdemonstrate that W8A8 PTQ results in a modest accuracy drop of within 0.5% and is compatible with the proposed layer sharing method.Table 13:Ablation study: 8-bit weight, 8-bit activation post-training quantization results on zero-shot common sense reasoning tasks. Quantized models achieve accuracy gap of within 0.5% compared to the full-precision BF16 counterpart.ModelPrecisionARC-eARC-cBoolQPIQASIQAHellaSwagOBQAWinoGrandeAvg.GapMobileLLM-125MBF1645.527.758.364.641.936.435.450.445.0–MobileLLM-125MW8A845.227.158.365.041.736.233.651.044.80.2\hdashline[0.8pt/1pt]MobileLLM-LS-125MBF1644.427.061.565.143.037.637.852.046.1–MobileLLM-LS-125MW8A844.027.560.964.643.137.737.751.045.80.3\hdashline[0.8pt/1pt]MobileLLM-350MBF1651.431.361.068.143.647.241.655.449.9–MobileLLM-350MW8A851.432.161.168.843.147.140.655.149.90.0\hdashline[0.8pt/1pt]MobileLLM-LS-350MBF1651.935.259.668.943.447.243.358.451.0–MobileLLM-LS-350MW8A851.333.859.569.143.747.243.057.050.60.4Appendix FKnowledge DistillationThe results of integrating knowledge distillation (KD) into small model pre-training are presented in TableF. LLaMA-v2 7B models serve as the teacher, and the KD loss is computed using cross-entropy between the logits from the large pre-trained teacher model (i.e., LLaMA-v2 7B)  and the small student network (i.e., 125M or 350M models) :(1) Here,denotes thesample in the current batch withtotal samples in the batch, andrepresents the number of classes, which, in our case, equals the size of the vocabulary.andare the teacher and student networks, respectively.The results in TableFindicate that adding KD loss is comparable or even lower than solely using the next token as labels. However, it’s noteworthy that the training time using KD isslower than training from scratch using labels. All models are trained on 32 A100 80G GPUs with a batch size of 32 for 120k iterations. Consequently, we opt to use labels in our experiments.Table 14:Ablation study on employing LLaMA-v2 7B teacher’s output as soft labels for knowledge distillation (KD) . Results indicate a slight degradation in performance when incorporating KD loss compared to only using hard labels.ModelTraining LossTraining TimeARC-eARC-cBoolQPIQASIQAHellaSwagOBQAWinoGrandeAvg.125M modelLabel29h43.128.958.162.342.334.631.550.143.9125M modelLabel + KD93h41.828.558.561.641.134.532.751.643.8\hdashline[0.8pt/1pt]
350M modelLabel42h50.231.856.967.744.345.840.855.549.1350M modelLabel + KD109h48.731.860.767.443.245.938.953.748.8Appendix GDatasets and BenchmarksMobileLLMis assessed on zero-shot common sense reasoning (BoolQ, PIQA, SIQA, HellaSwag, Winogrande, ARC, OBQA) , question answering (TriviaQA) , and reading comprehension (RACE)  tasks. Additionally, we evaluate our chat models on MT-Bench and AlpacaEval benchmarks. We also generated a API calling dataset to fine-tune and evaluate models for this particular task.G.1Zero-shot Common Sense Reasoning tasksBoolQis a reading comprehension dataset focused on naturally occurring yes/no questions. Each instance includes a question (Q) , an excerpt from a passage (P) , and an answer (A) , with an added explanation for enhanced clarity.PIQA, abbreviated for Physical Interaction: Question Answering, serves as a benchmark for evaluating and studying the capacity of natural language models in comprehending physical commonsense understanding.SIQA, abbreviated for Social Interaction Question Answering, is designed to measure the social and emotional intelligence of computational models through multiple-choice question answering.HellaSwagserves as a benchmark for physically situated commonsense natural language inference. It comprises four-way multiple-choice problems that are considered trivial for humans (>95% accuracy)  but pose a challenge for language models.WinoGrandeis a benchmark for commonsense reasoning. It consists of a set of 273 expert-crafted pronoun resolution problems, deliberately designed to be unsolvable for statistical models relying on selectional preferences or word associations.ARC, the AI2 Reasoning Challenge, is a compilation of 7787 natural science questions. It is divided into a Challenge Set and an Easy Set, with the Challenge Set exclusively comprising questions that were answered incorrectly by both a retrieval-based algorithm and a word co-occurrence algorithm.OBQAis a dataset consisting of approximately 6000 questions designed for open book question answering. The task involves integrating a corpus of provided science facts (open book)  with external broad common knowledge. Providing correct answers necessitates leveraging simple common knowledge beyond the core facts provided.G.2Question Answering TasksTriviaQAis a closed-book question answering benchmark. It encompasses over 650,000 question-answer evidence triples derived by combining 95,000 question-answer pairs authored by Trivia enthusiasts. Each question is supported by an average of six evidence documents.G.3Reading Comprehension TasksRACEis a large-scale reading comprehension dataset with more than 28,000 passages and nearly 100,000 questions. The dataset is collected from English examinations in China, which are designed for middle school and high school students.G.4Chat BenchmarksAlpacaEvalis a single-turn benchmark featuring 805 questions across various topics. In the evaluation of this paper, models generate responses and are scored by GPT-4, with the ultimate metric being the pairwise win-rate against a baseline model (text-davinci-001) .MT-Benchis a multi-turn benchmark comprising 160 questions spanning eight diverse knowledge domains. In this evaluation, models are tasked with answering an initial question and subsequently providing a second response to a predefined follow-up question. Each model response is rated by GPT-4 on a scale from 1 to 10, and the final score is determined by the mean over the two turns.G.5API Calling DatasetThe API calling dataset is generated by instructing a language model to simulate a conversation between a human and an agent. This process instructs the language model to explicitly mention tasks that should be performed using designated APIs throughout the simulated conversation.
Examples of a conversation turn in generated dataset are shown in the following boxes.{"role": "USER", "message": "Can you help me send a SMS message: We are looking forward to seeing you"},{"role": "PARSER", "message": "style_transfer(text = "We are looking forward to seeing you", style = ["sms"], contact = None) "},{"role": "AGENT", "message": "Sure! Here’s a possible SMS message: Hey [name], we are looking forward to seeing you! Wishing you all well."},{"role": "USER", "message": "How did the S&P 500 perform last month?"},{"role": "PARSER", "message": "get_stock_information(company_name = "S&P 500", ticker_symbol = "GSPC", date_time = "last month", financial_metrics = "price") "},{"role": "AGENT", "message": "The S&P 500 closed at $2,977.94 last month, with a monthly return of 2.5%."},{"role": "USER", "message": "Is the San Diego Zoo open during the winter months?"},{"role": "PARSER", "message": "get_local_information(point_of_interest = "San Diego Zoo", attributes = "hours of operation") "},{"role": "AGENT", "message": "The San Diego Zoo is open year-round, but the hours of operation vary depending on the time of year. During the winter months, the zoo is open from 9:00 AM to 5:00 PM."},{"role": "USER", "message": "What is the most recent news concerning aliens?"},{"role": "PARSER", "message": "news(query = "latest alien news") "},{"role": "AGENT", "message": "According to recent news articles, NASA has announced the discovery of a new planet outside of our solar system that has the potential to support alien life. The planet, named Kepler-186f, is located in the constellation Cygnus and is the first known Earth-sized planet to be found in the habitable zone of a star. While this news does not necessarily prove the existence of aliens, it does provide hope for the possibility of finding life beyond Earth. Additionally, a recent study published in the journal Nature has suggested that the universe may be teeming with alien civilizations, with estimates suggesting that there could be tens of billions of intelligent civilizations in the observable universe. However, it’s important to note that these findings are still being researched and debated in the scientific community."},{"role": "USER", "message": "Who won the last Super Bowl?"},{"role": "PARSER", "message": "get_sports_information(match = "", sport_type = "NFL", metric = "winner") "},{"role": "AGENT", "message": "The last Super Bowl was won by the Kansas City Chiefs."},Appendix HApproximate calculation of envisioned future GPU consumption#GPUs =(population size) (individual time999According to the statistics that YouTube users spending an average of 23 hours per month (source)  and Instagram users dedicating 12 hours monthly to the app (source) , we have reason to believe the interaction with LLM can easily surpass the cumulative engagement of these platforms in the future.) (approximating FLOPs/token for 220B GPT-4 model with one expert activated101010https://the-decoder.com/gpt-4-has-a-trillion-parameters/) (token/s) (h) (s)  / ((FLOPs/s, H100 computation capacity ) (h) (s) ) This assumes short sequence lengths (i.e.hundreds of tokens, not tens of thousands of tokens) , where the compute is dominated by linear layers.
For very long sequences, the self-attention computation would dominate, and this would necessitate a more complicated formula.Appendix IChat Generations fromMobileLLMWe show a several examples of chat output generated withMobileLLMwhile responding to various questions.Input:Are there any particular physical benefits to mindful walking, such as improved posture or increased physical fitness?Output(MobileLLM-125M) :Yes, there are several physical benefits to mindful walking. Here are a few:1. Improved Posture: Mindful walking can help improve your posture by promoting good alignment and reducing stress on the joints. This can lead to better posture and reduced back pain.2. Increased Flexibility: Mindful walking can also help increase flexibility in your body, which can reduce the risk of injury and improve overall health.3. Better Sleep: Mindful walking can help you fall asleep more easily and stay asleep longer, which can improve your sleep quality.4. Reduced Stress: Mindful walking can help reduce stress levels by promoting relaxation and reducing tension in the body.5. Enhanced Mental Health: Mindful walking has been shown to have positive effects on mental health, including reducing symptoms of depression and anxiety.Overall, mindful walking is an excellent way to improve your physical health and well-being. It’s a simple and effective way to get moving and enjoy the benefits it offers. If you’re interested in trying mindful walking, I would recommend starting with shorter walks and gradually increasing the duration over time. You may also want to consider incorporating mindfulness into your daily routine to further enhance your experience.In conclusion, mindful walking is a great way to improve your physical health and well-being. With its numerous physical benefits, it’s easy to see why it’s becoming increasingly popular among people looking for a fun and sustainable way to get moving. So go ahead and give it a try!Output(MobileLLM-350M) :Yes, there are several physical benefits of mindful walking. Firstly, it can improve your posture by reducing the amount of tension in your muscles and joints. Secondly, it can increase your physical fitness by improving your cardiovascular health, strengthening your core muscles, and increasing your flexibility. Additionally, mindful walking can help you to stay more focused on the present moment, which can lead to better sleep quality and reduced stress levels. Finally, it can also help you to develop a greater sense of awareness and appreciation for the world around you, which can enhance your overall well-being.Overall, mindful walking is a great way to improve your physical health and well-being, while also promoting relaxation and mindfulness. Whether you’re looking to reduce stress, improve your posture, or simply enjoy a peaceful walk, mindful walking can be a rewarding and fulfilling experience.If you’re interested in trying out mindful walking, there are many resources available that can guide you through the process. Some popular options include guided meditations, apps that offer mindful walking exercises, and classes that teach you how to practice mindful walking. With a little bit of effort and dedication, anyone can start incorporating mindful walking into their daily routine and reap the many benefits it offers.In conclusion, mindful walking is a simple yet powerful tool that can help you to improve your physical health, reduce stress, and cultivate a deeper sense of awareness and appreciation for the world around you. So why not give it a try and see what kind of benefits you can achieve? Your body and mind will thank you!Output(llama-v2-7b-chat) :Mindful walking can have a range of physical benefits, including:1. Improved posture: By focusing on the sensation of each step and the movement of the body, mindful walking can help improve posture and reduce slouching or hunching.2. Increased physical fitness: Mindful walking can help improve cardiovascular fitness, increase muscle strength and endurance, and boost overall physical fitness.3. Better balance and coordination: By paying attention to the body’s position in space and the movement of the legs and feet, mindful walking can help improve balance and coordination.4. Reduced stress and tension: Mindful walking can help reduce stress and tension in the body, which can lead to improved physical relaxation and well-being.5. Improved flexibility: By focusing on the movement of the body and the sensation of each step, mindful walking can help improve flexibility and range of motion.6. Boosted immune system: Studies have shown that mindfulness practices, including mindful walking, can help boost the immune system and reduce inflammation.7. Improved sleep: Mindful walking can help improve sleep quality and duration by reducing stress and promoting relaxation.
8. Reduced pain: Mindful walking can help reduce pain by focusing on the sensation of each step and the movement of the body, rather than the pain itself.9. Improved mood: Mindful walking can help improve mood and reduce symptoms of depression and anxiety by promoting relaxation and reducing stress.It’s important to note that the physical benefits of mindful walking may vary depending on the individual and the frequency and duration of the practice. However, with regular practice, mindful walking can have a range of positive effects on physical health and well-being.Input:How do I wrap a present neatly?Output(MobileLLM-125M) :1. Choose the right size gift: The size of the present you want to give will depend on your budget and personal preference. If you are looking for a small gift, consider buying a small box or card with a single-use plastic bag.2. Prepare the present: Make sure all the items in the present are clean and tidy. You can use a cleaning kit or a tissue paper to remove any dirt or debris from the present.3. Arrange the presents: Place the present in a place where it is easy to see and touch. Use a pencil to draw a line around the present and arrange it in a way that makes sense to you.4. Add decorations: Decorate the present with stickers, ribbons, or other decorative items. This will add some personality to the present and make it more visually appealing.5. Enjoy the present: Take time to enjoy the present and let yourself feel the warmth and comfort of the present. Smile at yourself and take a moment to appreciate the present.Remember, the most important thing is to have fun and enjoy the present! Happy gift-giving!If you’re feeling adventurous, try making a surprise gift for someone special by wrapping the present in a cute and colorful gift basket. Or, if you’re feeling adventurous, try making a gift for a friend who needs a little extra help with their homework. Whatever you choose, remember to be creative and have fun with your present!With these tips, you’ll be able to create a beautiful and thoughtful present that will last for years to come. Happy gift-giving!Output(MobileLLM-350M) :1. Start by wrapping the present in a decorative gift wrap or ribbon.2. Use a small amount of glue or tape to secure the gift wrap or ribbon to the surface of the present.3. Place the gift wrap or ribbon around the present, making sure it is centered and evenly distributed.4. Use a small amount of glue or tape to secure the gift wrap or ribbon to the surface of the present.5. Once the gift wrap or ribbon is securely attached, you can add any additional decorations or embellishments to the present.6. If you want to add a personal touch, you can write a message or a special message on the gift wrap or ribbon.7. Finally, enjoy your beautifully wrapped present!Remember, wrapping a gift is a fun and creative way to show your loved one how much you care. Have fun and enjoy the process!Output(llama-v2-7b-chat) :Wrapping a present neatly can be a bit of a challenge, but with a few simple tips and tricks, you can create a beautifully wrapped gift that your recipient will love. Here are some tips to help you wrap a present neatly:1. Use the right wrapping paper: Choose a wrapping paper that complements the gift and is the right size for the present. You can also use a patterned paper or a solid color, depending on your preference.2. Measure the gift: Before you start wrapping, measure the gift to make sure you have enough wrapping paper. You can also use a piece of scrap paper to measure the gift and mark the edges with a pencil.3. Use a flat surface: Make sure you have a flat surface to work on, such as a table or countertop. This will help you keep the wrapping paper flat and prevent it from wrinkling.4. Start with the edges: Begin by folding the edges of the wrapping paper over the gift, making sure they are even and smooth. You can use tape to secure the edges if needed.5. Roll the gift: Once you have the edges in place, roll the gift into the center of the wrapping paper. Make sure the gift is securely in place before moving on to the next step.6. Tuck in the edges: As you wrap the gift, tuck the edges of the wrapping paper under the gift, smoothing them out as you go. Make sure the edges are even and smooth, and use tape to secure them if needed.7. Add a ribbon or bow: Once you have the gift wrapped, add a ribbon or bow to the top. This will not only make the gift look more festive, but it will also help to keep the wrapping paper in place.8. Practice makes perfect: Wrapping a present neatly takes a bit of practice, so don’t be discouraged if it doesn’t come out perfectly the first time. Keep practicing, and you’ll get the hang of it in no time!By following these tips, you can create a beautifully wrapped present that your recipient will love. Happy wrapping!Input:What are the best exercises for beginners?Output(MobileLLM-125M) :1. Push-ups: This is a classic exercise that can be done anywhere, anytime. It’s easy to do and requires no equipment.2. Dumbbell Rows: These rows are great for building strength in your legs and core.3. Squats: Squats are a great way to build strength in your legs and core.4. Lunges: Lunges are a great way to improve balance and stability.5. Plank: A plank is a great way to build strength in your back and core.6. Jumping Jacks: Jumping jacks are a fun and effective way to build strength in your legs and core.7. Burpees: Burpees are a great way to build strength in your legs and core.8. Mountain Climbers: Mountain climbers are a great way to build strength in your legs and core.9. Deadlifts: Deadlifts are a great way to build strength in your legs and core.10. Tricep Dips: Tricep dips are a great way to build strength in your triceps.Remember, it’s important to start with simple exercises and gradually increase the intensity as you get stronger. Don’t push yourself too hard or you may find it difficult to maintain proper form. With consistent practice, you will see improvements in your strength and overall fitness.If you have any questions about exercises or healthy living, don’t hesitate to talk to a doctor or physical therapist. They can help you develop a personalized plan that works best for you.In conclusion, there are many exercises that can be done at home, but it’s important to start slowly and gradually increase the intensity over time. Remember to listen to your body and take breaks when needed. And most importantly, enjoy the process!Output(MobileLLM-350M) :Beginners should start with exercises that are easy to do and don’t require a lot of strength or endurance. Here are some exercises that are great for beginners:1. Push-ups: This is a great exercise for beginners because it’s a simple exercise that can be done anywhere. Start with a push-up on your knees and then lower yourself down to the ground.2. Squats: Squats are a great exercise for building leg strength and endurance. Start with a squat on your knees and then lower yourself down to the ground.3. Lunges: Lunges are a great exercise for building leg strength and endurance. Start with a lunge on your front foot and then lower yourself down to the ground.4. Plank: Planks are a great exercise for building core strength and stability. Start with a plank on your forearms and then lower yourself down to the ground.5. Bicycle crunches: Bicycle crunches are a great exercise for building upper body strength and stability. Start with a bicycle crunch on your upper back and then lower yourself down to the ground.6. Plank with leg lifts: This exercise is great for building leg strength and stability. Start with a plank on your forearms and then lift your legs off the ground and hold for a few seconds before lowering them back down.7. Jumping jacks: Jumping jacks are a great exercise for building cardiovascular endurance and strength. Start with a jumping jack on your feet and then jump up and spread your legs.Remember to start with a weight that is comfortable for you and gradually increase the weight as you get stronger. It’s also important to warm up before exercising and to stretch after your workout to prevent injury.In conclusion, beginners should start with exercises that are easy to do and don’t require a lot of strength or endurance. These exercises are great for building leg strength and endurance, and can be done anywhere. Remember to start with a weight that is comfortable for you and gradually increase the weight as you get stronger. Warming up and stretching after your workout are also important to prevent injury.Following these exercises can help you build strength, endurance, and improve your overall fitness level. Remember to always consult with a healthcare professional before starting any new exercise program.Output(llama-v2-7b-chat) :Exercise is an essential part of a healthy lifestyle, but for beginners, it can be intimidating and overwhelming. Here are some of the best exercises for beginners:1. Bodyweight exercises: These exercises are great for beginners because they don’t require any equipment. Examples include push-ups, squats, lunges, and planks.2. Yoga: Yoga is a low-impact exercise that can help improve flexibility, balance, and strength. It’s also great for reducing stress and improving overall well-being.3. Brisk walking: Walking is a simple and accessible form of exercise that can be done anywhere. It’s a great way to get your heart rate up and improve cardiovascular health.4. Swimming: Swimming is a low-impact exercise that can help improve cardiovascular health and strengthen muscles. It’s also easy on the joints, making it a great option for beginners.5. Cycling: Cycling is another low-impact exercise that can be done indoors or outdoors. It’s a great way to improve cardiovascular health and build leg strength.6. Resistance band exercises: Resistance bands are lightweight and portable, making them easy to use at home or on the go. They can help improve strength and flexibility in the upper body, lower body, and core.7. Pilates: Pilates is a form of exercise that focuses on core strength, flexibility, and body control. It’s a great option for beginners because it’s low-impact and can help improve posture and balance.8. Dance-based workouts: Dance-based workouts, such as Zumba or Hip Hop Abs, are fun and energetic ways to get in shape. They often involve aerobic exercise and strength training, making them a great option for beginners.9. Jumping rope: Jumping rope is a high-intensity exercise that can help improve cardiovascular health and burn calories. It’s also easy to do at home with a simple jump rope.10. Kettlebell exercises: Kettlebells are small weights that can be used to perform a variety of exercises, including swings, squats, and presses. They’re a great way to improve strength and cardiovascular health.Remember to always consult with a doctor before starting any new exercise program, especially if you’re a beginner. It’s also important to start slowly and gradually increase the intensity and duration of your workouts as you become more comfortable.
body
MobileLLM: Optimizing Sub-billion Parameter Language Modelsfor On-Device Use Cases
4Related Work
2402.14905
5ConclusionThis study focuses on optimizing sub-billion scale models for on-device applications. Our findings indicate that, for smaller models, prioritizing depth over width enhances model performance. Furthermore, by leveraging advanced weight-sharing techniques, including embedding sharing, grouped query attention, and block-wise weight sharing, we achieve significant enhancements in weight utilization within storage-constrained scenarios. The resulting models denoted asMobileLLMexhibit substantial advancements in zero-shot commonsense reasoning, question answering, and reading comprehension tasks compared to previous SoTA methods. Last but not least, we demonstrate the effectiveness of the fine-tunedMobileLLMmodels in two prevalent on-device use cases: chat and API calling, underscoring their adeptness in handling such tasks.AcknowledgmentWe thank Hansong Zhang for his valuable contribution in setting up latency measurement environment on iOS, and the full support from PyTorch Edge team.Broader ImpactThis paper advocates for the adoption of sub-billion scale large language models in on-device applications, aiming to mitigate energy consumption during LLM inference. The proposed approach is promising in alleviating computational costs associated with LLM deployment.ReferencesAinslie et al. (2023) Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y., Lebron, F., and Sanghai, S.GQA: Training generalized multi-query transformer models from multi-head checkpoints.InEMNLP, 2023.Biderman et al. (2023) Biderman, S., Schoelkopf, H., Anthony, Q. G., Bradley, H., O’Brien, K., Hallahan, E., Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., et al.Pythia: A suite for analyzing large language models across training and scaling.InInternational Conference on Machine Learning, pp.  2397–2430. PMLR, 2023.Bisk et al. (2020) Bisk, Y., Zellers, R., Gao, J., Choi, Y., et al.Piqa: Reasoning about physical commonsense in natural language.InProceedings of the AAAI conference on artificial intelligence, volume 34, pp.  7432–7439, 2020.Black et al. (2022) Black, S., Biderman, S., Hallahan, E., Anthony, Q., Gao, L., Golding, L., He, H., Leahy, C., McDonell, K., Phang, J., et al.Gpt-neox-20b: An open-source autoregressive language model.arXiv preprint arXiv:2204.06745, 2022.Chowdhery et al. (2023) Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al.Palm: Scaling language modeling with pathways.Journal of Machine Learning Research, 24(240) :1–113, 2023.Clark et al. (2019) Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., and Toutanova, K.Boolq: Exploring the surprising difficulty of natural yes/no questions.arXiv preprint arXiv:1905.10044, 2019.Clark et al. (2018) Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O.Think you have solved question answering? try arc, the ai2 reasoning challenge.arXiv preprint arXiv:1803.05457, 2018.Dao et al. (2022) Dao, T., Fu, D., Ermon, S., Rudra, A., and Ré, C.Flashattention: Fast and memory-efficient exact attention with io-awareness.Advances in Neural Information Processing Systems, 35:16344–16359, 2022.Dauphin et al. (2017) Dauphin, Y. N., Fan, A., Auli, M., and Grangier, D.Language modeling with gated convolutional networks.InInternational conference on machine learning, pp.  933–941. PMLR, 2017.Dettmers et al. (2022) Dettmers, T., Lewis, M., Belkada, Y., and Zettlemoyer, L.Llm. int8 () : 8-bit matrix multiplication for transformers at scale.arXiv preprint arXiv:2208.07339, 2022.Dey et al. (2023) Dey, N., Gosal, G., Khachane, H., Marshall, W., Pathria, R., Tom, M., Hestness, J., et al.Cerebras-gpt: Open compute-optimal language models trained on the cerebras wafer-scale cluster.arXiv preprint arXiv:2304.03208, 2023.Frantar & Alistarh (2023) Frantar, E. and Alistarh, D.Sparsegpt: Massive language models can be accurately pruned in one-shot.InInternational Conference on Machine Learning, pp.  10323–10337. PMLR, 2023.Frantar et al. (2022) Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D.Gptq: Accurate post-training quantization for generative pre-trained transformers.arXiv preprint arXiv:2210.17323, 2022.Ganesan et al. (2021) Ganesan, V., Ramesh, G., and Kumar, P.Supershaper: Task-agnostic super pre-training of bert models with variable hidden dimensions.arXiv preprint arXiv:2110.04711, 2021.Google (2023) Google.Pixel 8 pro tech specs.https://store.google.com/gb/product/pixel_8_pro_specs, 2023.Guo et al. (2020) Guo, Z., Zhang, X., Mu, H., Heng, W., Liu, Z., Wei, Y., and Sun, J.Single path one-shot neural architecture search with uniform sampling.InComputer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XVI 16, pp.  544–560. Springer, 2020.Han et al. (2016) Han, S., Liu, X., Mao, H., Pu, J., Pedram, A., Horowitz, M. A., and Dally, W. J.Eie: Efficient inference engine on compressed deep neural network.ACM SIGARCH Computer Architecture News, 44(3) :243–254, 2016.Hinton et al. (2015) Hinton, G., Vinyals, O., Dean, J., et al.Distilling the knowledge in a neural network.arXiv preprint arXiv:1503.02531, 2(7) , 2015.Hristov (2022) Hristov, V.A16 bionic explained: what’s new in apple’s pro-grade mobile chip?https://www.phonearena.com/news/A16-Bionic-explained-whats-new_id142438, 2022.Jawahar et al. (2023) Jawahar, G., Yang, H., Xiong, Y., Liu, Z., Wang, D., Sun, F., Li, M., Pappu, A., Oguz, B., Abdul-Mageed, M., et al.Mixture-of-supernets: Improving weight-sharing supernet training with architecture-routed mixture-of-experts.arXiv preprint arXiv:2306.04845, 2023.Joshi et al. (2017) Joshi, M., Choi, E., Weld, D. S., and Zettlemoyer, L.Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension.arXiv preprint arXiv:1705.03551, 2017.Kaplan et al. (2020) Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D.Scaling laws for neural language models.arXiv preprint arXiv:2001.08361, 2020.Katharopoulos et al. (2020) Katharopoulos, A., Vyas, A., Pappas, N., and Fleuret, F.Transformers are rnns: Fast autoregressive transformers with linear attention.InInternational conference on machine learning, pp.  5156–5165. PMLR, 2020.Kim et al. (2023) Kim, S., Hooper, C., Gholami, A., Dong, Z., Li, X., Shen, S., Mahoney, M. W., and Keutzer, K.Squeezellm: Dense-and-sparse quantization.arXiv preprint arXiv:2306.07629, 2023.Kingma & Ba (2014) Kingma, D. P. and Ba, J.Adam: A method for stochastic optimization.arXiv preprint arXiv:1412.6980, 2014.Kitaev et al. (2020) Kitaev, N., Kaiser, Ł., and Levskaya, A.Reformer: The efficient transformer.arXiv preprint arXiv:2001.04451, 2020.Lai et al. (2017) Lai, G., Xie, Q., Liu, H., Yang, Y., and Hovy, E.Race: Large-scale reading comprehension dataset from examinations.arXiv preprint arXiv:1704.04683, 2017.Li et al. (2023) Li, X., Zhang, T., Dubois, Y., Taori, R., Gulrajani, I., Guestrin, C., Liang, P., and Hashimoto, T. B.Alpacaeval: An automatic evaluator of instruction-following models.https://github.com/tatsu-lab/alpaca_eval, 2023.Liu et al. (2023a) Liu, J., Gong, R., Wei, X., Dong, Z., Cai, J., and Zhuang, B.Qllm: Accurate and efficient low-bitwidth quantization for large language models.arXiv preprint arXiv:2310.08041, 2023a.Liu et al. (2023b) Liu, S.-y., Liu, Z., Huang, X., Dong, P., and Cheng, K.-T.Llm-fp4: 4-bit floating-point quantized transformers.arXiv preprint arXiv:2310.16836, 2023b.Liu et al. (2023c) Liu, Z., Oguz, B., Zhao, C., Chang, E., Stock, P., Mehdad, Y., Shi, Y., Krishnamoorthi, R., and Chandra, V.Llm-qat: Data-free quantization aware training for large language models.arXiv preprint arXiv:2305.17888, 2023c.Malladi et al. (2012) Malladi, K. T., Lee, B. C., Nothaft, F. A., Kozyrakis, C., Periyathambi, K., and Horowitz, M.Towards energy-proportional datacenter memory with mobile dram.ACM SIGARCH Computer Architecture News, 40(3) :37–48, 2012.Mihaylov et al. (2018) Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A.Can a suit of armor conduct electricity? a new dataset for open book question answering.arXiv preprint arXiv:1809.02789, 2018.Peng et al. (2023) Peng, B., Alcaide, E., Anthony, Q., Albalak, A., Arcadinho, S., Cao, H., Cheng, X., Chung, M., Grella, M., GV, K. K., et al.Rwkv: Reinventing rnns for the transformer era.arXiv preprint arXiv:2305.13048, 2023.Reid et al. (2021) Reid, M., Marrese-Taylor, E., and Matsuo, Y.Subformer: Exploring weight sharing for parameter efficiency in generative transformers.arXiv preprint arXiv:2101.00234, 2021.Roy et al. (2021) Roy, A., Saffar, M., Vaswani, A., and Grangier, D.Efficient content-based sparse attention with routing transformers.Transactions of the Association for Computational Linguistics, 9:53–68, 2021.Sakaguchi et al. (2021) Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y.Winogrande: An adversarial winograd schema challenge at scale.Communications of the ACM, 64(9) :99–106, 2021.Sap et al. (2019) Sap, M., Rashkin, H., Chen, D., LeBras, R., and Choi, Y.Socialiqa: Commonsense reasoning about social interactions.arXiv preprint arXiv:1904.09728, 2019.Scao et al. (2022) Scao, T. L., Fan, A., Akiki, C., Pavlick, E., Ilić, S., Hesslow, D., Castagné, R., Luccioni, A. S., Yvon, F., Gallé, M., et al.Bloom: A 176b-parameter open-access multilingual language model.arXiv preprint arXiv:2211.05100, 2022.Shen et al. (2022) Shen, Z., Liu, Z., and Xing, E.Sliced recursive transformer.InEuropean Conference on Computer Vision, pp.  727–744. Springer, 2022.Sheng et al. (2023) Sheng, Y., Zheng, L., Yuan, B., Li, Z., Ryabinin, M., Chen, B., Liang, P., Ré, C., Stoica, I., and Zhang, C.Flexgen: High-throughput generative inference of large language models with a single gpu.InInternational Conference on Machine Learning, pp.  31094–31116. PMLR, 2023.Sun et al. (2023) Sun, M., Liu, Z., Bair, A., and Kolter, J. Z.A simple and effective pruning approach for large language models.arXiv preprint arXiv:2306.11695, 2023.Tan & Le (2019) Tan, M. and Le, Q.Efficientnet: Rethinking model scaling for convolutional neural networks.InInternational conference on machine learning, pp.  6105–6114. PMLR, 2019.Taylor et al. (2022) Taylor, R., Kardas, M., Cucurull, G., Scialom, T., Hartshorn, A., Saravia, E., Poulton, A., Kerkez, V., and Stojnic, R.Galactica: A large language model for science.arXiv preprint arXiv:2211.09085, 2022.Timiryasov & Tastet (2023) Timiryasov, I. and Tastet, J.-L.Baby llama: knowledge distillation from an ensemble of teachers trained on a small dataset with no performance penalty.arXiv preprint arXiv:2308.02019, 2023.Touvron et al. (2023a) Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al.Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023a.Touvron et al. (2023b) Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Ferrer, C. C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshorn, A., Hosseini, S., Hou, R., Inan, H., Kardas, M., Kerkez, V., Khabsa, M., Kloumann, I., Korenev, A., Koura, P. S., Lachaux, M.-A., Lavril, T., Lee, J., Liskovich, D., Lu, Y., Mao, Y., Martinet, X., Mihaylov, T., Mishra, P., Molybog, I., Nie, Y., Poulton, A., Reizenstein, J., Rungta, R., Saladi, K., Schelten, A., Silva, R., Smith, E. M., Subramanian, R., Tan, X. E., Tang, B., Taylor, R., Williams, A., Kuan, J. X., Xu, P., Yan, Z., Zarov, I., Zhang, Y., Fan, A., Kambadur, M., Narang, S., Rodriguez, A., Stojnic, R., Edunov, S., and Scialom, T.Llama 2: Open foundation and fine-tuned chat models, 2023b.Wang et al. (2020) Wang, S., Li, B. Z., Khabsa, M., Fang, H., and Ma, H.Linformer: Self-attention with linear complexity.arXiv preprint arXiv:2006.04768, 2020.Wu et al. (2019) Wu, B., Dai, X., Zhang, P., Wang, Y., Sun, F., Wu, Y., Tian, Y., Vajda, P., Jia, Y., and Keutzer, K.Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search.InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.  10734–10742, 2019.Xia et al. (2023a) Xia, H., Zheng, Z., Li, Y., Zhuang, D., Zhou, Z., Qiu, X., Li, Y., Lin, W., and Song, S. L.Flash-llm: Enabling cost-effective and highly-efficient large generative model inference with unstructured sparsity.arXiv preprint arXiv:2309.10285, 2023a.Xia et al. (2023b) Xia, M., Gao, T., Zeng, Z., and Chen, D.Sheared llama: Accelerating language model pre-training via structured pruning.arXiv preprint arXiv:2310.06694, 2023b.Xiao et al. (2023) Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., and Han, S.Smoothquant: Accurate and efficient post-training quantization for large language models.InInternational Conference on Machine Learning, pp.  38087–38099. PMLR, 2023.Xiong et al. (2021) Xiong, Y., Zeng, Z., Chakraborty, R., Tan, M., Fung, G., Li, Y., and Singh, V.Nyströmformer: A nyström-based algorithm for approximating self-attention.InProceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp.  14138–14148, 2021.Xu et al. (2021) Xu, J., Tan, X., Luo, R., Song, K., Li, J., Qin, T., and Liu, T.-Y.Nas-bert: task-agnostic and adaptive-size bert compression with neural architecture search.InProceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp.  1933–1943, 2021.Yao et al. (2022) Yao, Z., Yazdani Aminabadi, R., Zhang, M., Wu, X., Li, C., and He, Y.Zeroquant: Efficient and affordable post-training quantization for large-scale transformers.Advances in Neural Information Processing Systems, 35:27168–27183, 2022.Zellers et al. (2019) Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y.Hellaswag: Can a machine really finish your sentence?arXiv preprint arXiv:1905.07830, 2019.Zhang et al. (2022) Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al.Opt: Open pre-trained transformer language models.arXiv preprint arXiv:2205.01068, 2022.Zheng et al. (2023) Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., et al.Judging llm-as-a-judge with mt-bench and chatbot arena.arXiv preprint arXiv:2306.05685, 2023.Zoph & Le (2016) Zoph, B. and Le, Q. V.Neural architecture search with reinforcement learning.arXiv preprint arXiv:1611.01578, 2016.AppendixAppendix AImpact of Each Design ChoiceThis section presents comprehensive tabulated results for theimproving sub-billion scale LLM designexperiments, at the model sizes of 125M and 350M. Looking at the results in TableA, transitioning from the traditional Feedforward Network ()  to SwiGLU yields a accuracy improvement of 1.3% for both model sizes. Further increasing the model depth enhances accuracy by 0.9%/1.1% for 125M/350M models, respectively. Then, introducing input and output embedding sharing achieves a parameter reduction of approximately 10%, while with only marginal accuracy drops of 0.2% for the 125M and 0.6% for 350M models. Additionally, following in findings in SectionC, we incorporate grouped query attention with head dimension equals to 64, and head number to be near 4to the kv head number, while increasing the embedding dimension to preserve model size. This modification further results in a performance boost of 0.4%/0.7% for 125M/350M models. Combining these techniques establishes a strong baseline network denoted asMobileLLM. Finally, the immediate block-wise weight-sharing technique contributes an additional accuracy gain of 1.1% for models trained on 0.25 trillion tokens, and resulting in the modelMobileLLM-LS. Final models includingMobileLLMandMobileLLM-LSare trained with 1 trillion tokens.Table 8:Ablation study on the impact of each design choice to the model accuracy on zero-shot common sense reasoning tasks. Corresponding to the bar chart in Figure3. Here, L, H,denotes number of layers, heads, kv-heads, respectively and dim denotes embedding dimension.TechniquesLHDim#Params(M) ARC-eARC-cBoolQPIQASIQAHellaSwagOBQAWinoGrandeAvg.125M\hdashline[0.8pt/1pt]
Baseline model121212768134.141.325.257.562.041.931.131.250.842.6+ SwiGLU in FFN121212768134.143.128.958.162.342.334.631.550.143.9+ Use deep-thin structure3088512135.043.626.158.062.542.636.537.551.544.8+ Embedding share3088512118.644.426.056.262.843.135.936.052.644.6+ Multi-query attention3093576124.645.527.758.364.641.936.435.450.445.0(Train on 1T token) 3093576124.643.927.160.265.342.438.939.553.146.3+ Layer sharing3093576124.644.427.061.565.143.037.637.852.046.1(Train on 1T token) 3093576124.645.828.760.465.742.939.541.152.147.0350M\hdashline[0.8pt/1pt]
Baseline model1520201280376.850.327.653.868.144.142.640.152.447.4+ SwiGLU in FFN1520201280386.749.230.659.167.744.343.241.054.248.7+ Use deep-thin structure321414896380.350.731.459.467.843.346.243.856.249.8+ Embedding share321414896351.649.932.060.367.943.247.038.954.849.2+ Multi-query attention32155960345.351.431.361.068.143.647.241.655.449.9(Train on 1T token) 32155960345.353.833.562.468.644.749.640.057.651.3+ Layer sharing32155960345.351.935.259.668.943.447.243.358.451.0(Train on 1T token) 32155960345.354.432.562.869.844.150.645.857.252.1Appendix BDepth vs WidthWe provide network depth versus width exploration results on zero-shot reasoning tasks in TableB, as well as results on question answering and reading comprehension tasks in TableB. The findings indicate that shallow architectures with fewer than 10 layers perform poorly in reasoning or handling comprehension tasks. Models with 10-20 layers exhibit improved accuracy, while further increasing depth continues to provide significant benefits across all three tasks. Notably, the optimal depth is found to be near 30 layers for sub-billion scale models.Table 9:Ablation study on depth versus width in architecture design, illustrated in Figure4(a) (b) . For compact models, prioritizing depth over width yields superior performance, assessed through zero-shot common sense reasoning tasks.#Layer#HeadsDim#Params(M) ARC-eARC-cBoolQPIQASIQAHellaSwagOBQAWinoGrandeAvg.4201280163.242.026.161.061.541.332.931.449.843.36161024142.642.225.551.062.641.633.532.150.042.3814896138.143.625.548.661.840.934.037.552.043.01212768134.143.128.958.162.342.334.631.550.143.91810640132.441.025.559.362.641.634.236.750.143.9249576132.443.627.059.763.842.036.233.752.244.8308512135.043.626.158.062.542.636.537.551.544.8427448134.643.326.157.863.341.736.335.952.044.5626384134.344.126.759.064.742.436.233.151.344.7\hdashline[0.8pt/1pt]
5322048388.047.928.862.265.242.838.139.052.647.110241536381.449.129.660.667.144.042.741.254.448.612221408379.949.931.659.168.242.644.043.753.849.115201280386.749.230.659.167.744.343.241.054.248.719181152376.350.830.956.369.143.845.239.654.548.824161024373.850.733.858.868.643.545.040.054.949.42815960371.151.733.257.667.942.946.037.753.948.93214896380.350.731.459.467.843.346.243.856.249.84612768374.749.730.559.568.145.144.943.455.549.66610640376.250.531.861.067.443.846.040.155.649.5Table 10:Ablation study on depth vs. width in architecture design on TQA and RACE datasets, depicted in Figure4(c-f) .TQA(F1 score) RACE(Acc) #Layer#HeadsDim#Params(M) 1-shot5-shot64-shotmiddlehigh4201280163.22.03.93.933.226.06161024142.63.54.94.629.225.5814896138.14.25.24.729.224.31212768134.15.16.16.335.627.31810640132.44.76.85.234.327.4249576132.45.97.27.335.328.6308512135.06.26.86.934.227.9427448134.66.17.27.234.428.6626384134.36.07.07.338.928.9\hdashline[0.8pt/1pt]
5322048388.05.67.88.638.129.010241536381.49.011.812.740.630.012221408379.911.712.913.342.431.515201280386.710.713.114.242.630.619181152376.311.313.313.542.632.424161024373.811.813.514.443.031.92815960371.112.513.614.942.832.33214896380.312.614.515.344.232.24612768374.713.015.415.644.731.46610640376.212.515.115.744.032.5Appendix CNumber of Heads and Key-Value HeadsWe provide detailed experimental results assessing the impact of the number of attention heads and key-value heads on zero-shot reasoning accuracy in TableC. Our study involves two baseline architectures: a 8-layer 125M model with a embedding dimension of 896, and a 15-layer 350M model with a embedding dimension of 1280. We conduct head size sweeps in {8, 16, 32}. The findings shown in TableCindicate that using 16 heads, with a head dimension close to 64, and 4 key-value heads, yields the best accuracy and memory trade-off. This setting serves as a guiding principle in our model architecture design.Table 11:Ablation study investigating the impact of the number of attention heads and key-value heads.Model#Heads#KV-Heads#Params(M) ARC-eARC-cBoolQPIQASIQAHellaSwagOBQAWinoGrandeAvg.3232138.142.126.958.462.242.133.836.752.444.33216131.741.226.457.762.542.333.334.252.943.8328128.542.627.361.161.941.932.235.052.044.2324126.843.126.859.862.741.432.534.451.144.0322126.039.826.759.459.442.031.332.652.943.0321125.641.024.359.160.841.231.435.452.043.1125M1616138.141.625.761.162.443.134.436.951.644.6# layers=8168131.742.426.460.763.441.933.534.751.544.3dim=896164128.542.525.662.362.441.833.035.954.544.7162126.841.725.356.961.742.032.932.654.543.5161126.040.426.361.863.241.732.034.050.443.788138.141.425.058.361.741.733.335.953.243.884131.743.328.258.361.842.833.830.953.044.082128.540.726.258.161.241.632.834.851.543.481126.842.524.859.462.342.032.036.351.343.8\hdashline[0.8pt/1pt]3232386.748.630.459.767.243.944.040.953.948.63216362.148.931.657.668.443.443.838.654.948.4328349.848.333.161.067.242.642.139.053.948.4324343.747.229.859.467.243.542.542.554.148.3322340.647.630.362.466.942.641.638.652.047.7321339.048.527.356.367.142.940.936.753.346.6350M1616386.750.830.662.368.643.545.143.852.449.6# layers=15168362.148.530.759.467.343.843.841.353.348.5dim=1280164349.849.930.660.069.243.544.241.855.849.4162343.749.328.455.067.342.742.640.354.547.5161340.649.229.358.867.443.542.139.952.847.988386.751.033.258.267.043.943.942.654.749.384362.150.031.360.266.042.743.740.953.748.682349.849.330.459.967.942.843.538.953.848.381343.748.027.761.567.143.142.537.354.747.8Appendix DLayer Sharing Number AblationTable 12:Ablation study on the impact of varying layer repetition numbers.Model SizeRepeat TimesARC-eARC-cBoolQPIQASIQAHellaSwagOBQAWinoGrandeAvg.125Mbaseline41.625.761.162.443.134.436.951.644.6block-wise share repeat243.927.961.564.341.535.535.150.245.0block-wise share repeat343.327.660.963.042.335.634.152.945.0block-wise share repeat442.026.662.364.441.536.235.853.945.3\hdashline[0.8pt/1pt]350Mbaseline50.830.662.368.643.545.143.852.449.6block-wise share repeat251.530.859.668.243.947.744.755.050.2block-wise share repeat349.632.057.368.443.947.040.656.249.4block-wise share repeat452.232.660.069.044.547.641.855.250.4We extended our investigation to determine the optimal number of layer repetitions. The experiment involved the 8-layer 125M model with a embedding dimension of 896 and the 15-layer 350M model with a embedding dimension of 1280. The results in TableDdemonstrate that when we double the layer number and with each two transformer blocks sharing weights, the accuracy is enhanced by 0.4-0.6%. However, as we further triple or quadruple the layer repetition times, this accuracy enhancement effect diminishes. Consequently, in our experiments, we adopt a configuration where every two blocks share weights, effectively doubling the total number of layers.Appendix ECompatibility with QuantizationThis section explores the compatibility of quantization with the proposed model architecture and layer sharing. We employ a straightforward per-token min-max quantization, quantizing both weight and activation to 8-bit using post-training quantization (PTQ) . Experiments are conducted on bothMobileLLMandMobileLLM-LSwith model size of 125M and 350M, trained on 0.25T tokens. Results in TableEdemonstrate that W8A8 PTQ results in a modest accuracy drop of within 0.5% and is compatible with the proposed layer sharing method.Table 13:Ablation study: 8-bit weight, 8-bit activation post-training quantization results on zero-shot common sense reasoning tasks. Quantized models achieve accuracy gap of within 0.5% compared to the full-precision BF16 counterpart.ModelPrecisionARC-eARC-cBoolQPIQASIQAHellaSwagOBQAWinoGrandeAvg.GapMobileLLM-125MBF1645.527.758.364.641.936.435.450.445.0–MobileLLM-125MW8A845.227.158.365.041.736.233.651.044.80.2\hdashline[0.8pt/1pt]MobileLLM-LS-125MBF1644.427.061.565.143.037.637.852.046.1–MobileLLM-LS-125MW8A844.027.560.964.643.137.737.751.045.80.3\hdashline[0.8pt/1pt]MobileLLM-350MBF1651.431.361.068.143.647.241.655.449.9–MobileLLM-350MW8A851.432.161.168.843.147.140.655.149.90.0\hdashline[0.8pt/1pt]MobileLLM-LS-350MBF1651.935.259.668.943.447.243.358.451.0–MobileLLM-LS-350MW8A851.333.859.569.143.747.243.057.050.60.4Appendix FKnowledge DistillationThe results of integrating knowledge distillation (KD) into small model pre-training are presented in TableF. LLaMA-v2 7B models serve as the teacher, and the KD loss is computed using cross-entropy between the logits from the large pre-trained teacher model (i.e., LLaMA-v2 7B)  and the small student network (i.e., 125M or 350M models) :(1) Here,denotes thesample in the current batch withtotal samples in the batch, andrepresents the number of classes, which, in our case, equals the size of the vocabulary.andare the teacher and student networks, respectively.The results in TableFindicate that adding KD loss is comparable or even lower than solely using the next token as labels. However, it’s noteworthy that the training time using KD isslower than training from scratch using labels. All models are trained on 32 A100 80G GPUs with a batch size of 32 for 120k iterations. Consequently, we opt to use labels in our experiments.Table 14:Ablation study on employing LLaMA-v2 7B teacher’s output as soft labels for knowledge distillation (KD) . Results indicate a slight degradation in performance when incorporating KD loss compared to only using hard labels.ModelTraining LossTraining TimeARC-eARC-cBoolQPIQASIQAHellaSwagOBQAWinoGrandeAvg.125M modelLabel29h43.128.958.162.342.334.631.550.143.9125M modelLabel + KD93h41.828.558.561.641.134.532.751.643.8\hdashline[0.8pt/1pt]
350M modelLabel42h50.231.856.967.744.345.840.855.549.1350M modelLabel + KD109h48.731.860.767.443.245.938.953.748.8Appendix GDatasets and BenchmarksMobileLLMis assessed on zero-shot common sense reasoning (BoolQ, PIQA, SIQA, HellaSwag, Winogrande, ARC, OBQA) , question answering (TriviaQA) , and reading comprehension (RACE)  tasks. Additionally, we evaluate our chat models on MT-Bench and AlpacaEval benchmarks. We also generated a API calling dataset to fine-tune and evaluate models for this particular task.G.1Zero-shot Common Sense Reasoning tasksBoolQis a reading comprehension dataset focused on naturally occurring yes/no questions. Each instance includes a question (Q) , an excerpt from a passage (P) , and an answer (A) , with an added explanation for enhanced clarity.PIQA, abbreviated for Physical Interaction: Question Answering, serves as a benchmark for evaluating and studying the capacity of natural language models in comprehending physical commonsense understanding.SIQA, abbreviated for Social Interaction Question Answering, is designed to measure the social and emotional intelligence of computational models through multiple-choice question answering.HellaSwagserves as a benchmark for physically situated commonsense natural language inference. It comprises four-way multiple-choice problems that are considered trivial for humans (>95% accuracy)  but pose a challenge for language models.WinoGrandeis a benchmark for commonsense reasoning. It consists of a set of 273 expert-crafted pronoun resolution problems, deliberately designed to be unsolvable for statistical models relying on selectional preferences or word associations.ARC, the AI2 Reasoning Challenge, is a compilation of 7787 natural science questions. It is divided into a Challenge Set and an Easy Set, with the Challenge Set exclusively comprising questions that were answered incorrectly by both a retrieval-based algorithm and a word co-occurrence algorithm.OBQAis a dataset consisting of approximately 6000 questions designed for open book question answering. The task involves integrating a corpus of provided science facts (open book)  with external broad common knowledge. Providing correct answers necessitates leveraging simple common knowledge beyond the core facts provided.G.2Question Answering TasksTriviaQAis a closed-book question answering benchmark. It encompasses over 650,000 question-answer evidence triples derived by combining 95,000 question-answer pairs authored by Trivia enthusiasts. Each question is supported by an average of six evidence documents.G.3Reading Comprehension TasksRACEis a large-scale reading comprehension dataset with more than 28,000 passages and nearly 100,000 questions. The dataset is collected from English examinations in China, which are designed for middle school and high school students.G.4Chat BenchmarksAlpacaEvalis a single-turn benchmark featuring 805 questions across various topics. In the evaluation of this paper, models generate responses and are scored by GPT-4, with the ultimate metric being the pairwise win-rate against a baseline model (text-davinci-001) .MT-Benchis a multi-turn benchmark comprising 160 questions spanning eight diverse knowledge domains. In this evaluation, models are tasked with answering an initial question and subsequently providing a second response to a predefined follow-up question. Each model response is rated by GPT-4 on a scale from 1 to 10, and the final score is determined by the mean over the two turns.G.5API Calling DatasetThe API calling dataset is generated by instructing a language model to simulate a conversation between a human and an agent. This process instructs the language model to explicitly mention tasks that should be performed using designated APIs throughout the simulated conversation.
Examples of a conversation turn in generated dataset are shown in the following boxes.{"role": "USER", "message": "Can you help me send a SMS message: We are looking forward to seeing you"},{"role": "PARSER", "message": "style_transfer(text = "We are looking forward to seeing you", style = ["sms"], contact = None) "},{"role": "AGENT", "message": "Sure! Here’s a possible SMS message: Hey [name], we are looking forward to seeing you! Wishing you all well."},{"role": "USER", "message": "How did the S&P 500 perform last month?"},{"role": "PARSER", "message": "get_stock_information(company_name = "S&P 500", ticker_symbol = "GSPC", date_time = "last month", financial_metrics = "price") "},{"role": "AGENT", "message": "The S&P 500 closed at $2,977.94 last month, with a monthly return of 2.5%."},{"role": "USER", "message": "Is the San Diego Zoo open during the winter months?"},{"role": "PARSER", "message": "get_local_information(point_of_interest = "San Diego Zoo", attributes = "hours of operation") "},{"role": "AGENT", "message": "The San Diego Zoo is open year-round, but the hours of operation vary depending on the time of year. During the winter months, the zoo is open from 9:00 AM to 5:00 PM."},{"role": "USER", "message": "What is the most recent news concerning aliens?"},{"role": "PARSER", "message": "news(query = "latest alien news") "},{"role": "AGENT", "message": "According to recent news articles, NASA has announced the discovery of a new planet outside of our solar system that has the potential to support alien life. The planet, named Kepler-186f, is located in the constellation Cygnus and is the first known Earth-sized planet to be found in the habitable zone of a star. While this news does not necessarily prove the existence of aliens, it does provide hope for the possibility of finding life beyond Earth. Additionally, a recent study published in the journal Nature has suggested that the universe may be teeming with alien civilizations, with estimates suggesting that there could be tens of billions of intelligent civilizations in the observable universe. However, it’s important to note that these findings are still being researched and debated in the scientific community."},{"role": "USER", "message": "Who won the last Super Bowl?"},{"role": "PARSER", "message": "get_sports_information(match = "", sport_type = "NFL", metric = "winner") "},{"role": "AGENT", "message": "The last Super Bowl was won by the Kansas City Chiefs."},Appendix HApproximate calculation of envisioned future GPU consumption#GPUs =(population size) (individual time999According to the statistics that YouTube users spending an average of 23 hours per month (source)  and Instagram users dedicating 12 hours monthly to the app (source) , we have reason to believe the interaction with LLM can easily surpass the cumulative engagement of these platforms in the future.) (approximating FLOPs/token for 220B GPT-4 model with one expert activated101010https://the-decoder.com/gpt-4-has-a-trillion-parameters/) (token/s) (h) (s)  / ((FLOPs/s, H100 computation capacity ) (h) (s) ) This assumes short sequence lengths (i.e.hundreds of tokens, not tens of thousands of tokens) , where the compute is dominated by linear layers.
For very long sequences, the self-attention computation would dominate, and this would necessitate a more complicated formula.Appendix IChat Generations fromMobileLLMWe show a several examples of chat output generated withMobileLLMwhile responding to various questions.Input:Are there any particular physical benefits to mindful walking, such as improved posture or increased physical fitness?Output(MobileLLM-125M) :Yes, there are several physical benefits to mindful walking. Here are a few:1. Improved Posture: Mindful walking can help improve your posture by promoting good alignment and reducing stress on the joints. This can lead to better posture and reduced back pain.2. Increased Flexibility: Mindful walking can also help increase flexibility in your body, which can reduce the risk of injury and improve overall health.3. Better Sleep: Mindful walking can help you fall asleep more easily and stay asleep longer, which can improve your sleep quality.4. Reduced Stress: Mindful walking can help reduce stress levels by promoting relaxation and reducing tension in the body.5. Enhanced Mental Health: Mindful walking has been shown to have positive effects on mental health, including reducing symptoms of depression and anxiety.Overall, mindful walking is an excellent way to improve your physical health and well-being. It’s a simple and effective way to get moving and enjoy the benefits it offers. If you’re interested in trying mindful walking, I would recommend starting with shorter walks and gradually increasing the duration over time. You may also want to consider incorporating mindfulness into your daily routine to further enhance your experience.In conclusion, mindful walking is a great way to improve your physical health and well-being. With its numerous physical benefits, it’s easy to see why it’s becoming increasingly popular among people looking for a fun and sustainable way to get moving. So go ahead and give it a try!Output(MobileLLM-350M) :Yes, there are several physical benefits of mindful walking. Firstly, it can improve your posture by reducing the amount of tension in your muscles and joints. Secondly, it can increase your physical fitness by improving your cardiovascular health, strengthening your core muscles, and increasing your flexibility. Additionally, mindful walking can help you to stay more focused on the present moment, which can lead to better sleep quality and reduced stress levels. Finally, it can also help you to develop a greater sense of awareness and appreciation for the world around you, which can enhance your overall well-being.Overall, mindful walking is a great way to improve your physical health and well-being, while also promoting relaxation and mindfulness. Whether you’re looking to reduce stress, improve your posture, or simply enjoy a peaceful walk, mindful walking can be a rewarding and fulfilling experience.If you’re interested in trying out mindful walking, there are many resources available that can guide you through the process. Some popular options include guided meditations, apps that offer mindful walking exercises, and classes that teach you how to practice mindful walking. With a little bit of effort and dedication, anyone can start incorporating mindful walking into their daily routine and reap the many benefits it offers.In conclusion, mindful walking is a simple yet powerful tool that can help you to improve your physical health, reduce stress, and cultivate a deeper sense of awareness and appreciation for the world around you. So why not give it a try and see what kind of benefits you can achieve? Your body and mind will thank you!Output(llama-v2-7b-chat) :Mindful walking can have a range of physical benefits, including:1. Improved posture: By focusing on the sensation of each step and the movement of the body, mindful walking can help improve posture and reduce slouching or hunching.2. Increased physical fitness: Mindful walking can help improve cardiovascular fitness, increase muscle strength and endurance, and boost overall physical fitness.3. Better balance and coordination: By paying attention to the body’s position in space and the movement of the legs and feet, mindful walking can help improve balance and coordination.4. Reduced stress and tension: Mindful walking can help reduce stress and tension in the body, which can lead to improved physical relaxation and well-being.5. Improved flexibility: By focusing on the movement of the body and the sensation of each step, mindful walking can help improve flexibility and range of motion.6. Boosted immune system: Studies have shown that mindfulness practices, including mindful walking, can help boost the immune system and reduce inflammation.7. Improved sleep: Mindful walking can help improve sleep quality and duration by reducing stress and promoting relaxation.
8. Reduced pain: Mindful walking can help reduce pain by focusing on the sensation of each step and the movement of the body, rather than the pain itself.9. Improved mood: Mindful walking can help improve mood and reduce symptoms of depression and anxiety by promoting relaxation and reducing stress.It’s important to note that the physical benefits of mindful walking may vary depending on the individual and the frequency and duration of the practice. However, with regular practice, mindful walking can have a range of positive effects on physical health and well-being.Input:How do I wrap a present neatly?Output(MobileLLM-125M) :1. Choose the right size gift: The size of the present you want to give will depend on your budget and personal preference. If you are looking for a small gift, consider buying a small box or card with a single-use plastic bag.2. Prepare the present: Make sure all the items in the present are clean and tidy. You can use a cleaning kit or a tissue paper to remove any dirt or debris from the present.3. Arrange the presents: Place the present in a place where it is easy to see and touch. Use a pencil to draw a line around the present and arrange it in a way that makes sense to you.4. Add decorations: Decorate the present with stickers, ribbons, or other decorative items. This will add some personality to the present and make it more visually appealing.5. Enjoy the present: Take time to enjoy the present and let yourself feel the warmth and comfort of the present. Smile at yourself and take a moment to appreciate the present.Remember, the most important thing is to have fun and enjoy the present! Happy gift-giving!If you’re feeling adventurous, try making a surprise gift for someone special by wrapping the present in a cute and colorful gift basket. Or, if you’re feeling adventurous, try making a gift for a friend who needs a little extra help with their homework. Whatever you choose, remember to be creative and have fun with your present!With these tips, you’ll be able to create a beautiful and thoughtful present that will last for years to come. Happy gift-giving!Output(MobileLLM-350M) :1. Start by wrapping the present in a decorative gift wrap or ribbon.2. Use a small amount of glue or tape to secure the gift wrap or ribbon to the surface of the present.3. Place the gift wrap or ribbon around the present, making sure it is centered and evenly distributed.4. Use a small amount of glue or tape to secure the gift wrap or ribbon to the surface of the present.5. Once the gift wrap or ribbon is securely attached, you can add any additional decorations or embellishments to the present.6. If you want to add a personal touch, you can write a message or a special message on the gift wrap or ribbon.7. Finally, enjoy your beautifully wrapped present!Remember, wrapping a gift is a fun and creative way to show your loved one how much you care. Have fun and enjoy the process!Output(llama-v2-7b-chat) :Wrapping a present neatly can be a bit of a challenge, but with a few simple tips and tricks, you can create a beautifully wrapped gift that your recipient will love. Here are some tips to help you wrap a present neatly:1. Use the right wrapping paper: Choose a wrapping paper that complements the gift and is the right size for the present. You can also use a patterned paper or a solid color, depending on your preference.2. Measure the gift: Before you start wrapping, measure the gift to make sure you have enough wrapping paper. You can also use a piece of scrap paper to measure the gift and mark the edges with a pencil.3. Use a flat surface: Make sure you have a flat surface to work on, such as a table or countertop. This will help you keep the wrapping paper flat and prevent it from wrinkling.4. Start with the edges: Begin by folding the edges of the wrapping paper over the gift, making sure they are even and smooth. You can use tape to secure the edges if needed.5. Roll the gift: Once you have the edges in place, roll the gift into the center of the wrapping paper. Make sure the gift is securely in place before moving on to the next step.6. Tuck in the edges: As you wrap the gift, tuck the edges of the wrapping paper under the gift, smoothing them out as you go. Make sure the edges are even and smooth, and use tape to secure them if needed.7. Add a ribbon or bow: Once you have the gift wrapped, add a ribbon or bow to the top. This will not only make the gift look more festive, but it will also help to keep the wrapping paper in place.8. Practice makes perfect: Wrapping a present neatly takes a bit of practice, so don’t be discouraged if it doesn’t come out perfectly the first time. Keep practicing, and you’ll get the hang of it in no time!By following these tips, you can create a beautifully wrapped present that your recipient will love. Happy wrapping!Input:What are the best exercises for beginners?Output(MobileLLM-125M) :1. Push-ups: This is a classic exercise that can be done anywhere, anytime. It’s easy to do and requires no equipment.2. Dumbbell Rows: These rows are great for building strength in your legs and core.3. Squats: Squats are a great way to build strength in your legs and core.4. Lunges: Lunges are a great way to improve balance and stability.5. Plank: A plank is a great way to build strength in your back and core.6. Jumping Jacks: Jumping jacks are a fun and effective way to build strength in your legs and core.7. Burpees: Burpees are a great way to build strength in your legs and core.8. Mountain Climbers: Mountain climbers are a great way to build strength in your legs and core.9. Deadlifts: Deadlifts are a great way to build strength in your legs and core.10. Tricep Dips: Tricep dips are a great way to build strength in your triceps.Remember, it’s important to start with simple exercises and gradually increase the intensity as you get stronger. Don’t push yourself too hard or you may find it difficult to maintain proper form. With consistent practice, you will see improvements in your strength and overall fitness.If you have any questions about exercises or healthy living, don’t hesitate to talk to a doctor or physical therapist. They can help you develop a personalized plan that works best for you.In conclusion, there are many exercises that can be done at home, but it’s important to start slowly and gradually increase the intensity over time. Remember to listen to your body and take breaks when needed. And most importantly, enjoy the process!Output(MobileLLM-350M) :Beginners should start with exercises that are easy to do and don’t require a lot of strength or endurance. Here are some exercises that are great for beginners:1. Push-ups: This is a great exercise for beginners because it’s a simple exercise that can be done anywhere. Start with a push-up on your knees and then lower yourself down to the ground.2. Squats: Squats are a great exercise for building leg strength and endurance. Start with a squat on your knees and then lower yourself down to the ground.3. Lunges: Lunges are a great exercise for building leg strength and endurance. Start with a lunge on your front foot and then lower yourself down to the ground.4. Plank: Planks are a great exercise for building core strength and stability. Start with a plank on your forearms and then lower yourself down to the ground.5. Bicycle crunches: Bicycle crunches are a great exercise for building upper body strength and stability. Start with a bicycle crunch on your upper back and then lower yourself down to the ground.6. Plank with leg lifts: This exercise is great for building leg strength and stability. Start with a plank on your forearms and then lift your legs off the ground and hold for a few seconds before lowering them back down.7. Jumping jacks: Jumping jacks are a great exercise for building cardiovascular endurance and strength. Start with a jumping jack on your feet and then jump up and spread your legs.Remember to start with a weight that is comfortable for you and gradually increase the weight as you get stronger. It’s also important to warm up before exercising and to stretch after your workout to prevent injury.In conclusion, beginners should start with exercises that are easy to do and don’t require a lot of strength or endurance. These exercises are great for building leg strength and endurance, and can be done anywhere. Remember to start with a weight that is comfortable for you and gradually increase the weight as you get stronger. Warming up and stretching after your workout are also important to prevent injury.Following these exercises can help you build strength, endurance, and improve your overall fitness level. Remember to always consult with a healthcare professional before starting any new exercise program.Output(llama-v2-7b-chat) :Exercise is an essential part of a healthy lifestyle, but for beginners, it can be intimidating and overwhelming. Here are some of the best exercises for beginners:1. Bodyweight exercises: These exercises are great for beginners because they don’t require any equipment. Examples include push-ups, squats, lunges, and planks.2. Yoga: Yoga is a low-impact exercise that can help improve flexibility, balance, and strength. It’s also great for reducing stress and improving overall well-being.3. Brisk walking: Walking is a simple and accessible form of exercise that can be done anywhere. It’s a great way to get your heart rate up and improve cardiovascular health.4. Swimming: Swimming is a low-impact exercise that can help improve cardiovascular health and strengthen muscles. It’s also easy on the joints, making it a great option for beginners.5. Cycling: Cycling is another low-impact exercise that can be done indoors or outdoors. It’s a great way to improve cardiovascular health and build leg strength.6. Resistance band exercises: Resistance bands are lightweight and portable, making them easy to use at home or on the go. They can help improve strength and flexibility in the upper body, lower body, and core.7. Pilates: Pilates is a form of exercise that focuses on core strength, flexibility, and body control. It’s a great option for beginners because it’s low-impact and can help improve posture and balance.8. Dance-based workouts: Dance-based workouts, such as Zumba or Hip Hop Abs, are fun and energetic ways to get in shape. They often involve aerobic exercise and strength training, making them a great option for beginners.9. Jumping rope: Jumping rope is a high-intensity exercise that can help improve cardiovascular health and burn calories. It’s also easy to do at home with a simple jump rope.10. Kettlebell exercises: Kettlebells are small weights that can be used to perform a variety of exercises, including swings, squats, and presses. They’re a great way to improve strength and cardiovascular health.Remember to always consult with a doctor before starting any new exercise program, especially if you’re a beginner. It’s also important to start slowly and gradually increase the intensity and duration of your workouts as you become more comfortable.
body
MobileLLM: Optimizing Sub-billion Parameter Language Modelsfor On-Device Use Cases
5Conclusion
2402.14905
AcknowledgmentWe thank Hansong Zhang for his valuable contribution in setting up latency measurement environment on iOS, and the full support from PyTorch Edge team.Broader ImpactThis paper advocates for the adoption of sub-billion scale large language models in on-device applications, aiming to mitigate energy consumption during LLM inference. The proposed approach is promising in alleviating computational costs associated with LLM deployment.ReferencesAinslie et al. (2023) Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y., Lebron, F., and Sanghai, S.GQA: Training generalized multi-query transformer models from multi-head checkpoints.InEMNLP, 2023.Biderman et al. (2023) Biderman, S., Schoelkopf, H., Anthony, Q. G., Bradley, H., O’Brien, K., Hallahan, E., Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., et al.Pythia: A suite for analyzing large language models across training and scaling.InInternational Conference on Machine Learning, pp.  2397–2430. PMLR, 2023.Bisk et al. (2020) Bisk, Y., Zellers, R., Gao, J., Choi, Y., et al.Piqa: Reasoning about physical commonsense in natural language.InProceedings of the AAAI conference on artificial intelligence, volume 34, pp.  7432–7439, 2020.Black et al. (2022) Black, S., Biderman, S., Hallahan, E., Anthony, Q., Gao, L., Golding, L., He, H., Leahy, C., McDonell, K., Phang, J., et al.Gpt-neox-20b: An open-source autoregressive language model.arXiv preprint arXiv:2204.06745, 2022.Chowdhery et al. (2023) Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al.Palm: Scaling language modeling with pathways.Journal of Machine Learning Research, 24(240) :1–113, 2023.Clark et al. (2019) Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., and Toutanova, K.Boolq: Exploring the surprising difficulty of natural yes/no questions.arXiv preprint arXiv:1905.10044, 2019.Clark et al. (2018) Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O.Think you have solved question answering? try arc, the ai2 reasoning challenge.arXiv preprint arXiv:1803.05457, 2018.Dao et al. (2022) Dao, T., Fu, D., Ermon, S., Rudra, A., and Ré, C.Flashattention: Fast and memory-efficient exact attention with io-awareness.Advances in Neural Information Processing Systems, 35:16344–16359, 2022.Dauphin et al. (2017) Dauphin, Y. N., Fan, A., Auli, M., and Grangier, D.Language modeling with gated convolutional networks.InInternational conference on machine learning, pp.  933–941. PMLR, 2017.Dettmers et al. (2022) Dettmers, T., Lewis, M., Belkada, Y., and Zettlemoyer, L.Llm. int8 () : 8-bit matrix multiplication for transformers at scale.arXiv preprint arXiv:2208.07339, 2022.Dey et al. (2023) Dey, N., Gosal, G., Khachane, H., Marshall, W., Pathria, R., Tom, M., Hestness, J., et al.Cerebras-gpt: Open compute-optimal language models trained on the cerebras wafer-scale cluster.arXiv preprint arXiv:2304.03208, 2023.Frantar & Alistarh (2023) Frantar, E. and Alistarh, D.Sparsegpt: Massive language models can be accurately pruned in one-shot.InInternational Conference on Machine Learning, pp.  10323–10337. PMLR, 2023.Frantar et al. (2022) Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D.Gptq: Accurate post-training quantization for generative pre-trained transformers.arXiv preprint arXiv:2210.17323, 2022.Ganesan et al. (2021) Ganesan, V., Ramesh, G., and Kumar, P.Supershaper: Task-agnostic super pre-training of bert models with variable hidden dimensions.arXiv preprint arXiv:2110.04711, 2021.Google (2023) Google.Pixel 8 pro tech specs.https://store.google.com/gb/product/pixel_8_pro_specs, 2023.Guo et al. (2020) Guo, Z., Zhang, X., Mu, H., Heng, W., Liu, Z., Wei, Y., and Sun, J.Single path one-shot neural architecture search with uniform sampling.InComputer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XVI 16, pp.  544–560. Springer, 2020.Han et al. (2016) Han, S., Liu, X., Mao, H., Pu, J., Pedram, A., Horowitz, M. A., and Dally, W. J.Eie: Efficient inference engine on compressed deep neural network.ACM SIGARCH Computer Architecture News, 44(3) :243–254, 2016.Hinton et al. (2015) Hinton, G., Vinyals, O., Dean, J., et al.Distilling the knowledge in a neural network.arXiv preprint arXiv:1503.02531, 2(7) , 2015.Hristov (2022) Hristov, V.A16 bionic explained: what’s new in apple’s pro-grade mobile chip?https://www.phonearena.com/news/A16-Bionic-explained-whats-new_id142438, 2022.Jawahar et al. (2023) Jawahar, G., Yang, H., Xiong, Y., Liu, Z., Wang, D., Sun, F., Li, M., Pappu, A., Oguz, B., Abdul-Mageed, M., et al.Mixture-of-supernets: Improving weight-sharing supernet training with architecture-routed mixture-of-experts.arXiv preprint arXiv:2306.04845, 2023.Joshi et al. (2017) Joshi, M., Choi, E., Weld, D. S., and Zettlemoyer, L.Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension.arXiv preprint arXiv:1705.03551, 2017.Kaplan et al. (2020) Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D.Scaling laws for neural language models.arXiv preprint arXiv:2001.08361, 2020.Katharopoulos et al. (2020) Katharopoulos, A., Vyas, A., Pappas, N., and Fleuret, F.Transformers are rnns: Fast autoregressive transformers with linear attention.InInternational conference on machine learning, pp.  5156–5165. PMLR, 2020.Kim et al. (2023) Kim, S., Hooper, C., Gholami, A., Dong, Z., Li, X., Shen, S., Mahoney, M. W., and Keutzer, K.Squeezellm: Dense-and-sparse quantization.arXiv preprint arXiv:2306.07629, 2023.Kingma & Ba (2014) Kingma, D. P. and Ba, J.Adam: A method for stochastic optimization.arXiv preprint arXiv:1412.6980, 2014.Kitaev et al. (2020) Kitaev, N., Kaiser, Ł., and Levskaya, A.Reformer: The efficient transformer.arXiv preprint arXiv:2001.04451, 2020.Lai et al. (2017) Lai, G., Xie, Q., Liu, H., Yang, Y., and Hovy, E.Race: Large-scale reading comprehension dataset from examinations.arXiv preprint arXiv:1704.04683, 2017.Li et al. (2023) Li, X., Zhang, T., Dubois, Y., Taori, R., Gulrajani, I., Guestrin, C., Liang, P., and Hashimoto, T. B.Alpacaeval: An automatic evaluator of instruction-following models.https://github.com/tatsu-lab/alpaca_eval, 2023.Liu et al. (2023a) Liu, J., Gong, R., Wei, X., Dong, Z., Cai, J., and Zhuang, B.Qllm: Accurate and efficient low-bitwidth quantization for large language models.arXiv preprint arXiv:2310.08041, 2023a.Liu et al. (2023b) Liu, S.-y., Liu, Z., Huang, X., Dong, P., and Cheng, K.-T.Llm-fp4: 4-bit floating-point quantized transformers.arXiv preprint arXiv:2310.16836, 2023b.Liu et al. (2023c) Liu, Z., Oguz, B., Zhao, C., Chang, E., Stock, P., Mehdad, Y., Shi, Y., Krishnamoorthi, R., and Chandra, V.Llm-qat: Data-free quantization aware training for large language models.arXiv preprint arXiv:2305.17888, 2023c.Malladi et al. (2012) Malladi, K. T., Lee, B. C., Nothaft, F. A., Kozyrakis, C., Periyathambi, K., and Horowitz, M.Towards energy-proportional datacenter memory with mobile dram.ACM SIGARCH Computer Architecture News, 40(3) :37–48, 2012.Mihaylov et al. (2018) Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A.Can a suit of armor conduct electricity? a new dataset for open book question answering.arXiv preprint arXiv:1809.02789, 2018.Peng et al. (2023) Peng, B., Alcaide, E., Anthony, Q., Albalak, A., Arcadinho, S., Cao, H., Cheng, X., Chung, M., Grella, M., GV, K. K., et al.Rwkv: Reinventing rnns for the transformer era.arXiv preprint arXiv:2305.13048, 2023.Reid et al. (2021) Reid, M., Marrese-Taylor, E., and Matsuo, Y.Subformer: Exploring weight sharing for parameter efficiency in generative transformers.arXiv preprint arXiv:2101.00234, 2021.Roy et al. (2021) Roy, A., Saffar, M., Vaswani, A., and Grangier, D.Efficient content-based sparse attention with routing transformers.Transactions of the Association for Computational Linguistics, 9:53–68, 2021.Sakaguchi et al. (2021) Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y.Winogrande: An adversarial winograd schema challenge at scale.Communications of the ACM, 64(9) :99–106, 2021.Sap et al. (2019) Sap, M., Rashkin, H., Chen, D., LeBras, R., and Choi, Y.Socialiqa: Commonsense reasoning about social interactions.arXiv preprint arXiv:1904.09728, 2019.Scao et al. (2022) Scao, T. L., Fan, A., Akiki, C., Pavlick, E., Ilić, S., Hesslow, D., Castagné, R., Luccioni, A. S., Yvon, F., Gallé, M., et al.Bloom: A 176b-parameter open-access multilingual language model.arXiv preprint arXiv:2211.05100, 2022.Shen et al. (2022) Shen, Z., Liu, Z., and Xing, E.Sliced recursive transformer.InEuropean Conference on Computer Vision, pp.  727–744. Springer, 2022.Sheng et al. (2023) Sheng, Y., Zheng, L., Yuan, B., Li, Z., Ryabinin, M., Chen, B., Liang, P., Ré, C., Stoica, I., and Zhang, C.Flexgen: High-throughput generative inference of large language models with a single gpu.InInternational Conference on Machine Learning, pp.  31094–31116. PMLR, 2023.Sun et al. (2023) Sun, M., Liu, Z., Bair, A., and Kolter, J. Z.A simple and effective pruning approach for large language models.arXiv preprint arXiv:2306.11695, 2023.Tan & Le (2019) Tan, M. and Le, Q.Efficientnet: Rethinking model scaling for convolutional neural networks.InInternational conference on machine learning, pp.  6105–6114. PMLR, 2019.Taylor et al. (2022) Taylor, R., Kardas, M., Cucurull, G., Scialom, T., Hartshorn, A., Saravia, E., Poulton, A., Kerkez, V., and Stojnic, R.Galactica: A large language model for science.arXiv preprint arXiv:2211.09085, 2022.Timiryasov & Tastet (2023) Timiryasov, I. and Tastet, J.-L.Baby llama: knowledge distillation from an ensemble of teachers trained on a small dataset with no performance penalty.arXiv preprint arXiv:2308.02019, 2023.Touvron et al. (2023a) Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al.Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023a.Touvron et al. (2023b) Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Ferrer, C. C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshorn, A., Hosseini, S., Hou, R., Inan, H., Kardas, M., Kerkez, V., Khabsa, M., Kloumann, I., Korenev, A., Koura, P. S., Lachaux, M.-A., Lavril, T., Lee, J., Liskovich, D., Lu, Y., Mao, Y., Martinet, X., Mihaylov, T., Mishra, P., Molybog, I., Nie, Y., Poulton, A., Reizenstein, J., Rungta, R., Saladi, K., Schelten, A., Silva, R., Smith, E. M., Subramanian, R., Tan, X. E., Tang, B., Taylor, R., Williams, A., Kuan, J. X., Xu, P., Yan, Z., Zarov, I., Zhang, Y., Fan, A., Kambadur, M., Narang, S., Rodriguez, A., Stojnic, R., Edunov, S., and Scialom, T.Llama 2: Open foundation and fine-tuned chat models, 2023b.Wang et al. (2020) Wang, S., Li, B. Z., Khabsa, M., Fang, H., and Ma, H.Linformer: Self-attention with linear complexity.arXiv preprint arXiv:2006.04768, 2020.Wu et al. (2019) Wu, B., Dai, X., Zhang, P., Wang, Y., Sun, F., Wu, Y., Tian, Y., Vajda, P., Jia, Y., and Keutzer, K.Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search.InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.  10734–10742, 2019.Xia et al. (2023a) Xia, H., Zheng, Z., Li, Y., Zhuang, D., Zhou, Z., Qiu, X., Li, Y., Lin, W., and Song, S. L.Flash-llm: Enabling cost-effective and highly-efficient large generative model inference with unstructured sparsity.arXiv preprint arXiv:2309.10285, 2023a.Xia et al. (2023b) Xia, M., Gao, T., Zeng, Z., and Chen, D.Sheared llama: Accelerating language model pre-training via structured pruning.arXiv preprint arXiv:2310.06694, 2023b.Xiao et al. (2023) Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., and Han, S.Smoothquant: Accurate and efficient post-training quantization for large language models.InInternational Conference on Machine Learning, pp.  38087–38099. PMLR, 2023.Xiong et al. (2021) Xiong, Y., Zeng, Z., Chakraborty, R., Tan, M., Fung, G., Li, Y., and Singh, V.Nyströmformer: A nyström-based algorithm for approximating self-attention.InProceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp.  14138–14148, 2021.Xu et al. (2021) Xu, J., Tan, X., Luo, R., Song, K., Li, J., Qin, T., and Liu, T.-Y.Nas-bert: task-agnostic and adaptive-size bert compression with neural architecture search.InProceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp.  1933–1943, 2021.Yao et al. (2022) Yao, Z., Yazdani Aminabadi, R., Zhang, M., Wu, X., Li, C., and He, Y.Zeroquant: Efficient and affordable post-training quantization for large-scale transformers.Advances in Neural Information Processing Systems, 35:27168–27183, 2022.Zellers et al. (2019) Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y.Hellaswag: Can a machine really finish your sentence?arXiv preprint arXiv:1905.07830, 2019.Zhang et al. (2022) Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al.Opt: Open pre-trained transformer language models.arXiv preprint arXiv:2205.01068, 2022.Zheng et al. (2023) Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., et al.Judging llm-as-a-judge with mt-bench and chatbot arena.arXiv preprint arXiv:2306.05685, 2023.Zoph & Le (2016) Zoph, B. and Le, Q. V.Neural architecture search with reinforcement learning.arXiv preprint arXiv:1611.01578, 2016.AppendixAppendix AImpact of Each Design ChoiceThis section presents comprehensive tabulated results for theimproving sub-billion scale LLM designexperiments, at the model sizes of 125M and 350M. Looking at the results in TableA, transitioning from the traditional Feedforward Network ()  to SwiGLU yields a accuracy improvement of 1.3% for both model sizes. Further increasing the model depth enhances accuracy by 0.9%/1.1% for 125M/350M models, respectively. Then, introducing input and output embedding sharing achieves a parameter reduction of approximately 10%, while with only marginal accuracy drops of 0.2% for the 125M and 0.6% for 350M models. Additionally, following in findings in SectionC, we incorporate grouped query attention with head dimension equals to 64, and head number to be near 4to the kv head number, while increasing the embedding dimension to preserve model size. This modification further results in a performance boost of 0.4%/0.7% for 125M/350M models. Combining these techniques establishes a strong baseline network denoted asMobileLLM. Finally, the immediate block-wise weight-sharing technique contributes an additional accuracy gain of 1.1% for models trained on 0.25 trillion tokens, and resulting in the modelMobileLLM-LS. Final models includingMobileLLMandMobileLLM-LSare trained with 1 trillion tokens.Table 8:Ablation study on the impact of each design choice to the model accuracy on zero-shot common sense reasoning tasks. Corresponding to the bar chart in Figure3. Here, L, H,denotes number of layers, heads, kv-heads, respectively and dim denotes embedding dimension.TechniquesLHDim#Params(M) ARC-eARC-cBoolQPIQASIQAHellaSwagOBQAWinoGrandeAvg.125M\hdashline[0.8pt/1pt]
Baseline model121212768134.141.325.257.562.041.931.131.250.842.6+ SwiGLU in FFN121212768134.143.128.958.162.342.334.631.550.143.9+ Use deep-thin structure3088512135.043.626.158.062.542.636.537.551.544.8+ Embedding share3088512118.644.426.056.262.843.135.936.052.644.6+ Multi-query attention3093576124.645.527.758.364.641.936.435.450.445.0(Train on 1T token) 3093576124.643.927.160.265.342.438.939.553.146.3+ Layer sharing3093576124.644.427.061.565.143.037.637.852.046.1(Train on 1T token) 3093576124.645.828.760.465.742.939.541.152.147.0350M\hdashline[0.8pt/1pt]
Baseline model1520201280376.850.327.653.868.144.142.640.152.447.4+ SwiGLU in FFN1520201280386.749.230.659.167.744.343.241.054.248.7+ Use deep-thin structure321414896380.350.731.459.467.843.346.243.856.249.8+ Embedding share321414896351.649.932.060.367.943.247.038.954.849.2+ Multi-query attention32155960345.351.431.361.068.143.647.241.655.449.9(Train on 1T token) 32155960345.353.833.562.468.644.749.640.057.651.3+ Layer sharing32155960345.351.935.259.668.943.447.243.358.451.0(Train on 1T token) 32155960345.354.432.562.869.844.150.645.857.252.1Appendix BDepth vs WidthWe provide network depth versus width exploration results on zero-shot reasoning tasks in TableB, as well as results on question answering and reading comprehension tasks in TableB. The findings indicate that shallow architectures with fewer than 10 layers perform poorly in reasoning or handling comprehension tasks. Models with 10-20 layers exhibit improved accuracy, while further increasing depth continues to provide significant benefits across all three tasks. Notably, the optimal depth is found to be near 30 layers for sub-billion scale models.Table 9:Ablation study on depth versus width in architecture design, illustrated in Figure4(a) (b) . For compact models, prioritizing depth over width yields superior performance, assessed through zero-shot common sense reasoning tasks.#Layer#HeadsDim#Params(M) ARC-eARC-cBoolQPIQASIQAHellaSwagOBQAWinoGrandeAvg.4201280163.242.026.161.061.541.332.931.449.843.36161024142.642.225.551.062.641.633.532.150.042.3814896138.143.625.548.661.840.934.037.552.043.01212768134.143.128.958.162.342.334.631.550.143.91810640132.441.025.559.362.641.634.236.750.143.9249576132.443.627.059.763.842.036.233.752.244.8308512135.043.626.158.062.542.636.537.551.544.8427448134.643.326.157.863.341.736.335.952.044.5626384134.344.126.759.064.742.436.233.151.344.7\hdashline[0.8pt/1pt]
5322048388.047.928.862.265.242.838.139.052.647.110241536381.449.129.660.667.144.042.741.254.448.612221408379.949.931.659.168.242.644.043.753.849.115201280386.749.230.659.167.744.343.241.054.248.719181152376.350.830.956.369.143.845.239.654.548.824161024373.850.733.858.868.643.545.040.054.949.42815960371.151.733.257.667.942.946.037.753.948.93214896380.350.731.459.467.843.346.243.856.249.84612768374.749.730.559.568.145.144.943.455.549.66610640376.250.531.861.067.443.846.040.155.649.5Table 10:Ablation study on depth vs. width in architecture design on TQA and RACE datasets, depicted in Figure4(c-f) .TQA(F1 score) RACE(Acc) #Layer#HeadsDim#Params(M) 1-shot5-shot64-shotmiddlehigh4201280163.22.03.93.933.226.06161024142.63.54.94.629.225.5814896138.14.25.24.729.224.31212768134.15.16.16.335.627.31810640132.44.76.85.234.327.4249576132.45.97.27.335.328.6308512135.06.26.86.934.227.9427448134.66.17.27.234.428.6626384134.36.07.07.338.928.9\hdashline[0.8pt/1pt]
5322048388.05.67.88.638.129.010241536381.49.011.812.740.630.012221408379.911.712.913.342.431.515201280386.710.713.114.242.630.619181152376.311.313.313.542.632.424161024373.811.813.514.443.031.92815960371.112.513.614.942.832.33214896380.312.614.515.344.232.24612768374.713.015.415.644.731.46610640376.212.515.115.744.032.5Appendix CNumber of Heads and Key-Value HeadsWe provide detailed experimental results assessing the impact of the number of attention heads and key-value heads on zero-shot reasoning accuracy in TableC. Our study involves two baseline architectures: a 8-layer 125M model with a embedding dimension of 896, and a 15-layer 350M model with a embedding dimension of 1280. We conduct head size sweeps in {8, 16, 32}. The findings shown in TableCindicate that using 16 heads, with a head dimension close to 64, and 4 key-value heads, yields the best accuracy and memory trade-off. This setting serves as a guiding principle in our model architecture design.Table 11:Ablation study investigating the impact of the number of attention heads and key-value heads.Model#Heads#KV-Heads#Params(M) ARC-eARC-cBoolQPIQASIQAHellaSwagOBQAWinoGrandeAvg.3232138.142.126.958.462.242.133.836.752.444.33216131.741.226.457.762.542.333.334.252.943.8328128.542.627.361.161.941.932.235.052.044.2324126.843.126.859.862.741.432.534.451.144.0322126.039.826.759.459.442.031.332.652.943.0321125.641.024.359.160.841.231.435.452.043.1125M1616138.141.625.761.162.443.134.436.951.644.6# layers=8168131.742.426.460.763.441.933.534.751.544.3dim=896164128.542.525.662.362.441.833.035.954.544.7162126.841.725.356.961.742.032.932.654.543.5161126.040.426.361.863.241.732.034.050.443.788138.141.425.058.361.741.733.335.953.243.884131.743.328.258.361.842.833.830.953.044.082128.540.726.258.161.241.632.834.851.543.481126.842.524.859.462.342.032.036.351.343.8\hdashline[0.8pt/1pt]3232386.748.630.459.767.243.944.040.953.948.63216362.148.931.657.668.443.443.838.654.948.4328349.848.333.161.067.242.642.139.053.948.4324343.747.229.859.467.243.542.542.554.148.3322340.647.630.362.466.942.641.638.652.047.7321339.048.527.356.367.142.940.936.753.346.6350M1616386.750.830.662.368.643.545.143.852.449.6# layers=15168362.148.530.759.467.343.843.841.353.348.5dim=1280164349.849.930.660.069.243.544.241.855.849.4162343.749.328.455.067.342.742.640.354.547.5161340.649.229.358.867.443.542.139.952.847.988386.751.033.258.267.043.943.942.654.749.384362.150.031.360.266.042.743.740.953.748.682349.849.330.459.967.942.843.538.953.848.381343.748.027.761.567.143.142.537.354.747.8Appendix DLayer Sharing Number AblationTable 12:Ablation study on the impact of varying layer repetition numbers.Model SizeRepeat TimesARC-eARC-cBoolQPIQASIQAHellaSwagOBQAWinoGrandeAvg.125Mbaseline41.625.761.162.443.134.436.951.644.6block-wise share repeat243.927.961.564.341.535.535.150.245.0block-wise share repeat343.327.660.963.042.335.634.152.945.0block-wise share repeat442.026.662.364.441.536.235.853.945.3\hdashline[0.8pt/1pt]350Mbaseline50.830.662.368.643.545.143.852.449.6block-wise share repeat251.530.859.668.243.947.744.755.050.2block-wise share repeat349.632.057.368.443.947.040.656.249.4block-wise share repeat452.232.660.069.044.547.641.855.250.4We extended our investigation to determine the optimal number of layer repetitions. The experiment involved the 8-layer 125M model with a embedding dimension of 896 and the 15-layer 350M model with a embedding dimension of 1280. The results in TableDdemonstrate that when we double the layer number and with each two transformer blocks sharing weights, the accuracy is enhanced by 0.4-0.6%. However, as we further triple or quadruple the layer repetition times, this accuracy enhancement effect diminishes. Consequently, in our experiments, we adopt a configuration where every two blocks share weights, effectively doubling the total number of layers.Appendix ECompatibility with QuantizationThis section explores the compatibility of quantization with the proposed model architecture and layer sharing. We employ a straightforward per-token min-max quantization, quantizing both weight and activation to 8-bit using post-training quantization (PTQ) . Experiments are conducted on bothMobileLLMandMobileLLM-LSwith model size of 125M and 350M, trained on 0.25T tokens. Results in TableEdemonstrate that W8A8 PTQ results in a modest accuracy drop of within 0.5% and is compatible with the proposed layer sharing method.Table 13:Ablation study: 8-bit weight, 8-bit activation post-training quantization results on zero-shot common sense reasoning tasks. Quantized models achieve accuracy gap of within 0.5% compared to the full-precision BF16 counterpart.ModelPrecisionARC-eARC-cBoolQPIQASIQAHellaSwagOBQAWinoGrandeAvg.GapMobileLLM-125MBF1645.527.758.364.641.936.435.450.445.0–MobileLLM-125MW8A845.227.158.365.041.736.233.651.044.80.2\hdashline[0.8pt/1pt]MobileLLM-LS-125MBF1644.427.061.565.143.037.637.852.046.1–MobileLLM-LS-125MW8A844.027.560.964.643.137.737.751.045.80.3\hdashline[0.8pt/1pt]MobileLLM-350MBF1651.431.361.068.143.647.241.655.449.9–MobileLLM-350MW8A851.432.161.168.843.147.140.655.149.90.0\hdashline[0.8pt/1pt]MobileLLM-LS-350MBF1651.935.259.668.943.447.243.358.451.0–MobileLLM-LS-350MW8A851.333.859.569.143.747.243.057.050.60.4Appendix FKnowledge DistillationThe results of integrating knowledge distillation (KD) into small model pre-training are presented in TableF. LLaMA-v2 7B models serve as the teacher, and the KD loss is computed using cross-entropy between the logits from the large pre-trained teacher model (i.e., LLaMA-v2 7B)  and the small student network (i.e., 125M or 350M models) :(1) Here,denotes thesample in the current batch withtotal samples in the batch, andrepresents the number of classes, which, in our case, equals the size of the vocabulary.andare the teacher and student networks, respectively.The results in TableFindicate that adding KD loss is comparable or even lower than solely using the next token as labels. However, it’s noteworthy that the training time using KD isslower than training from scratch using labels. All models are trained on 32 A100 80G GPUs with a batch size of 32 for 120k iterations. Consequently, we opt to use labels in our experiments.Table 14:Ablation study on employing LLaMA-v2 7B teacher’s output as soft labels for knowledge distillation (KD) . Results indicate a slight degradation in performance when incorporating KD loss compared to only using hard labels.ModelTraining LossTraining TimeARC-eARC-cBoolQPIQASIQAHellaSwagOBQAWinoGrandeAvg.125M modelLabel29h43.128.958.162.342.334.631.550.143.9125M modelLabel + KD93h41.828.558.561.641.134.532.751.643.8\hdashline[0.8pt/1pt]
350M modelLabel42h50.231.856.967.744.345.840.855.549.1350M modelLabel + KD109h48.731.860.767.443.245.938.953.748.8Appendix GDatasets and BenchmarksMobileLLMis assessed on zero-shot common sense reasoning (BoolQ, PIQA, SIQA, HellaSwag, Winogrande, ARC, OBQA) , question answering (TriviaQA) , and reading comprehension (RACE)  tasks. Additionally, we evaluate our chat models on MT-Bench and AlpacaEval benchmarks. We also generated a API calling dataset to fine-tune and evaluate models for this particular task.G.1Zero-shot Common Sense Reasoning tasksBoolQis a reading comprehension dataset focused on naturally occurring yes/no questions. Each instance includes a question (Q) , an excerpt from a passage (P) , and an answer (A) , with an added explanation for enhanced clarity.PIQA, abbreviated for Physical Interaction: Question Answering, serves as a benchmark for evaluating and studying the capacity of natural language models in comprehending physical commonsense understanding.SIQA, abbreviated for Social Interaction Question Answering, is designed to measure the social and emotional intelligence of computational models through multiple-choice question answering.HellaSwagserves as a benchmark for physically situated commonsense natural language inference. It comprises four-way multiple-choice problems that are considered trivial for humans (>95% accuracy)  but pose a challenge for language models.WinoGrandeis a benchmark for commonsense reasoning. It consists of a set of 273 expert-crafted pronoun resolution problems, deliberately designed to be unsolvable for statistical models relying on selectional preferences or word associations.ARC, the AI2 Reasoning Challenge, is a compilation of 7787 natural science questions. It is divided into a Challenge Set and an Easy Set, with the Challenge Set exclusively comprising questions that were answered incorrectly by both a retrieval-based algorithm and a word co-occurrence algorithm.OBQAis a dataset consisting of approximately 6000 questions designed for open book question answering. The task involves integrating a corpus of provided science facts (open book)  with external broad common knowledge. Providing correct answers necessitates leveraging simple common knowledge beyond the core facts provided.G.2Question Answering TasksTriviaQAis a closed-book question answering benchmark. It encompasses over 650,000 question-answer evidence triples derived by combining 95,000 question-answer pairs authored by Trivia enthusiasts. Each question is supported by an average of six evidence documents.G.3Reading Comprehension TasksRACEis a large-scale reading comprehension dataset with more than 28,000 passages and nearly 100,000 questions. The dataset is collected from English examinations in China, which are designed for middle school and high school students.G.4Chat BenchmarksAlpacaEvalis a single-turn benchmark featuring 805 questions across various topics. In the evaluation of this paper, models generate responses and are scored by GPT-4, with the ultimate metric being the pairwise win-rate against a baseline model (text-davinci-001) .MT-Benchis a multi-turn benchmark comprising 160 questions spanning eight diverse knowledge domains. In this evaluation, models are tasked with answering an initial question and subsequently providing a second response to a predefined follow-up question. Each model response is rated by GPT-4 on a scale from 1 to 10, and the final score is determined by the mean over the two turns.G.5API Calling DatasetThe API calling dataset is generated by instructing a language model to simulate a conversation between a human and an agent. This process instructs the language model to explicitly mention tasks that should be performed using designated APIs throughout the simulated conversation.
Examples of a conversation turn in generated dataset are shown in the following boxes.{"role": "USER", "message": "Can you help me send a SMS message: We are looking forward to seeing you"},{"role": "PARSER", "message": "style_transfer(text = "We are looking forward to seeing you", style = ["sms"], contact = None) "},{"role": "AGENT", "message": "Sure! Here’s a possible SMS message: Hey [name], we are looking forward to seeing you! Wishing you all well."},{"role": "USER", "message": "How did the S&P 500 perform last month?"},{"role": "PARSER", "message": "get_stock_information(company_name = "S&P 500", ticker_symbol = "GSPC", date_time = "last month", financial_metrics = "price") "},{"role": "AGENT", "message": "The S&P 500 closed at $2,977.94 last month, with a monthly return of 2.5%."},{"role": "USER", "message": "Is the San Diego Zoo open during the winter months?"},{"role": "PARSER", "message": "get_local_information(point_of_interest = "San Diego Zoo", attributes = "hours of operation") "},{"role": "AGENT", "message": "The San Diego Zoo is open year-round, but the hours of operation vary depending on the time of year. During the winter months, the zoo is open from 9:00 AM to 5:00 PM."},{"role": "USER", "message": "What is the most recent news concerning aliens?"},{"role": "PARSER", "message": "news(query = "latest alien news") "},{"role": "AGENT", "message": "According to recent news articles, NASA has announced the discovery of a new planet outside of our solar system that has the potential to support alien life. The planet, named Kepler-186f, is located in the constellation Cygnus and is the first known Earth-sized planet to be found in the habitable zone of a star. While this news does not necessarily prove the existence of aliens, it does provide hope for the possibility of finding life beyond Earth. Additionally, a recent study published in the journal Nature has suggested that the universe may be teeming with alien civilizations, with estimates suggesting that there could be tens of billions of intelligent civilizations in the observable universe. However, it’s important to note that these findings are still being researched and debated in the scientific community."},{"role": "USER", "message": "Who won the last Super Bowl?"},{"role": "PARSER", "message": "get_sports_information(match = "", sport_type = "NFL", metric = "winner") "},{"role": "AGENT", "message": "The last Super Bowl was won by the Kansas City Chiefs."},Appendix HApproximate calculation of envisioned future GPU consumption#GPUs =(population size) (individual time999According to the statistics that YouTube users spending an average of 23 hours per month (source)  and Instagram users dedicating 12 hours monthly to the app (source) , we have reason to believe the interaction with LLM can easily surpass the cumulative engagement of these platforms in the future.) (approximating FLOPs/token for 220B GPT-4 model with one expert activated101010https://the-decoder.com/gpt-4-has-a-trillion-parameters/) (token/s) (h) (s)  / ((FLOPs/s, H100 computation capacity ) (h) (s) ) This assumes short sequence lengths (i.e.hundreds of tokens, not tens of thousands of tokens) , where the compute is dominated by linear layers.
For very long sequences, the self-attention computation would dominate, and this would necessitate a more complicated formula.Appendix IChat Generations fromMobileLLMWe show a several examples of chat output generated withMobileLLMwhile responding to various questions.Input:Are there any particular physical benefits to mindful walking, such as improved posture or increased physical fitness?Output(MobileLLM-125M) :Yes, there are several physical benefits to mindful walking. Here are a few:1. Improved Posture: Mindful walking can help improve your posture by promoting good alignment and reducing stress on the joints. This can lead to better posture and reduced back pain.2. Increased Flexibility: Mindful walking can also help increase flexibility in your body, which can reduce the risk of injury and improve overall health.3. Better Sleep: Mindful walking can help you fall asleep more easily and stay asleep longer, which can improve your sleep quality.4. Reduced Stress: Mindful walking can help reduce stress levels by promoting relaxation and reducing tension in the body.5. Enhanced Mental Health: Mindful walking has been shown to have positive effects on mental health, including reducing symptoms of depression and anxiety.Overall, mindful walking is an excellent way to improve your physical health and well-being. It’s a simple and effective way to get moving and enjoy the benefits it offers. If you’re interested in trying mindful walking, I would recommend starting with shorter walks and gradually increasing the duration over time. You may also want to consider incorporating mindfulness into your daily routine to further enhance your experience.In conclusion, mindful walking is a great way to improve your physical health and well-being. With its numerous physical benefits, it’s easy to see why it’s becoming increasingly popular among people looking for a fun and sustainable way to get moving. So go ahead and give it a try!Output(MobileLLM-350M) :Yes, there are several physical benefits of mindful walking. Firstly, it can improve your posture by reducing the amount of tension in your muscles and joints. Secondly, it can increase your physical fitness by improving your cardiovascular health, strengthening your core muscles, and increasing your flexibility. Additionally, mindful walking can help you to stay more focused on the present moment, which can lead to better sleep quality and reduced stress levels. Finally, it can also help you to develop a greater sense of awareness and appreciation for the world around you, which can enhance your overall well-being.Overall, mindful walking is a great way to improve your physical health and well-being, while also promoting relaxation and mindfulness. Whether you’re looking to reduce stress, improve your posture, or simply enjoy a peaceful walk, mindful walking can be a rewarding and fulfilling experience.If you’re interested in trying out mindful walking, there are many resources available that can guide you through the process. Some popular options include guided meditations, apps that offer mindful walking exercises, and classes that teach you how to practice mindful walking. With a little bit of effort and dedication, anyone can start incorporating mindful walking into their daily routine and reap the many benefits it offers.In conclusion, mindful walking is a simple yet powerful tool that can help you to improve your physical health, reduce stress, and cultivate a deeper sense of awareness and appreciation for the world around you. So why not give it a try and see what kind of benefits you can achieve? Your body and mind will thank you!Output(llama-v2-7b-chat) :Mindful walking can have a range of physical benefits, including:1. Improved posture: By focusing on the sensation of each step and the movement of the body, mindful walking can help improve posture and reduce slouching or hunching.2. Increased physical fitness: Mindful walking can help improve cardiovascular fitness, increase muscle strength and endurance, and boost overall physical fitness.3. Better balance and coordination: By paying attention to the body’s position in space and the movement of the legs and feet, mindful walking can help improve balance and coordination.4. Reduced stress and tension: Mindful walking can help reduce stress and tension in the body, which can lead to improved physical relaxation and well-being.5. Improved flexibility: By focusing on the movement of the body and the sensation of each step, mindful walking can help improve flexibility and range of motion.6. Boosted immune system: Studies have shown that mindfulness practices, including mindful walking, can help boost the immune system and reduce inflammation.7. Improved sleep: Mindful walking can help improve sleep quality and duration by reducing stress and promoting relaxation.
8. Reduced pain: Mindful walking can help reduce pain by focusing on the sensation of each step and the movement of the body, rather than the pain itself.9. Improved mood: Mindful walking can help improve mood and reduce symptoms of depression and anxiety by promoting relaxation and reducing stress.It’s important to note that the physical benefits of mindful walking may vary depending on the individual and the frequency and duration of the practice. However, with regular practice, mindful walking can have a range of positive effects on physical health and well-being.Input:How do I wrap a present neatly?Output(MobileLLM-125M) :1. Choose the right size gift: The size of the present you want to give will depend on your budget and personal preference. If you are looking for a small gift, consider buying a small box or card with a single-use plastic bag.2. Prepare the present: Make sure all the items in the present are clean and tidy. You can use a cleaning kit or a tissue paper to remove any dirt or debris from the present.3. Arrange the presents: Place the present in a place where it is easy to see and touch. Use a pencil to draw a line around the present and arrange it in a way that makes sense to you.4. Add decorations: Decorate the present with stickers, ribbons, or other decorative items. This will add some personality to the present and make it more visually appealing.5. Enjoy the present: Take time to enjoy the present and let yourself feel the warmth and comfort of the present. Smile at yourself and take a moment to appreciate the present.Remember, the most important thing is to have fun and enjoy the present! Happy gift-giving!If you’re feeling adventurous, try making a surprise gift for someone special by wrapping the present in a cute and colorful gift basket. Or, if you’re feeling adventurous, try making a gift for a friend who needs a little extra help with their homework. Whatever you choose, remember to be creative and have fun with your present!With these tips, you’ll be able to create a beautiful and thoughtful present that will last for years to come. Happy gift-giving!Output(MobileLLM-350M) :1. Start by wrapping the present in a decorative gift wrap or ribbon.2. Use a small amount of glue or tape to secure the gift wrap or ribbon to the surface of the present.3. Place the gift wrap or ribbon around the present, making sure it is centered and evenly distributed.4. Use a small amount of glue or tape to secure the gift wrap or ribbon to the surface of the present.5. Once the gift wrap or ribbon is securely attached, you can add any additional decorations or embellishments to the present.6. If you want to add a personal touch, you can write a message or a special message on the gift wrap or ribbon.7. Finally, enjoy your beautifully wrapped present!Remember, wrapping a gift is a fun and creative way to show your loved one how much you care. Have fun and enjoy the process!Output(llama-v2-7b-chat) :Wrapping a present neatly can be a bit of a challenge, but with a few simple tips and tricks, you can create a beautifully wrapped gift that your recipient will love. Here are some tips to help you wrap a present neatly:1. Use the right wrapping paper: Choose a wrapping paper that complements the gift and is the right size for the present. You can also use a patterned paper or a solid color, depending on your preference.2. Measure the gift: Before you start wrapping, measure the gift to make sure you have enough wrapping paper. You can also use a piece of scrap paper to measure the gift and mark the edges with a pencil.3. Use a flat surface: Make sure you have a flat surface to work on, such as a table or countertop. This will help you keep the wrapping paper flat and prevent it from wrinkling.4. Start with the edges: Begin by folding the edges of the wrapping paper over the gift, making sure they are even and smooth. You can use tape to secure the edges if needed.5. Roll the gift: Once you have the edges in place, roll the gift into the center of the wrapping paper. Make sure the gift is securely in place before moving on to the next step.6. Tuck in the edges: As you wrap the gift, tuck the edges of the wrapping paper under the gift, smoothing them out as you go. Make sure the edges are even and smooth, and use tape to secure them if needed.7. Add a ribbon or bow: Once you have the gift wrapped, add a ribbon or bow to the top. This will not only make the gift look more festive, but it will also help to keep the wrapping paper in place.8. Practice makes perfect: Wrapping a present neatly takes a bit of practice, so don’t be discouraged if it doesn’t come out perfectly the first time. Keep practicing, and you’ll get the hang of it in no time!By following these tips, you can create a beautifully wrapped present that your recipient will love. Happy wrapping!Input:What are the best exercises for beginners?Output(MobileLLM-125M) :1. Push-ups: This is a classic exercise that can be done anywhere, anytime. It’s easy to do and requires no equipment.2. Dumbbell Rows: These rows are great for building strength in your legs and core.3. Squats: Squats are a great way to build strength in your legs and core.4. Lunges: Lunges are a great way to improve balance and stability.5. Plank: A plank is a great way to build strength in your back and core.6. Jumping Jacks: Jumping jacks are a fun and effective way to build strength in your legs and core.7. Burpees: Burpees are a great way to build strength in your legs and core.8. Mountain Climbers: Mountain climbers are a great way to build strength in your legs and core.9. Deadlifts: Deadlifts are a great way to build strength in your legs and core.10. Tricep Dips: Tricep dips are a great way to build strength in your triceps.Remember, it’s important to start with simple exercises and gradually increase the intensity as you get stronger. Don’t push yourself too hard or you may find it difficult to maintain proper form. With consistent practice, you will see improvements in your strength and overall fitness.If you have any questions about exercises or healthy living, don’t hesitate to talk to a doctor or physical therapist. They can help you develop a personalized plan that works best for you.In conclusion, there are many exercises that can be done at home, but it’s important to start slowly and gradually increase the intensity over time. Remember to listen to your body and take breaks when needed. And most importantly, enjoy the process!Output(MobileLLM-350M) :Beginners should start with exercises that are easy to do and don’t require a lot of strength or endurance. Here are some exercises that are great for beginners:1. Push-ups: This is a great exercise for beginners because it’s a simple exercise that can be done anywhere. Start with a push-up on your knees and then lower yourself down to the ground.2. Squats: Squats are a great exercise for building leg strength and endurance. Start with a squat on your knees and then lower yourself down to the ground.3. Lunges: Lunges are a great exercise for building leg strength and endurance. Start with a lunge on your front foot and then lower yourself down to the ground.4. Plank: Planks are a great exercise for building core strength and stability. Start with a plank on your forearms and then lower yourself down to the ground.5. Bicycle crunches: Bicycle crunches are a great exercise for building upper body strength and stability. Start with a bicycle crunch on your upper back and then lower yourself down to the ground.6. Plank with leg lifts: This exercise is great for building leg strength and stability. Start with a plank on your forearms and then lift your legs off the ground and hold for a few seconds before lowering them back down.7. Jumping jacks: Jumping jacks are a great exercise for building cardiovascular endurance and strength. Start with a jumping jack on your feet and then jump up and spread your legs.Remember to start with a weight that is comfortable for you and gradually increase the weight as you get stronger. It’s also important to warm up before exercising and to stretch after your workout to prevent injury.In conclusion, beginners should start with exercises that are easy to do and don’t require a lot of strength or endurance. These exercises are great for building leg strength and endurance, and can be done anywhere. Remember to start with a weight that is comfortable for you and gradually increase the weight as you get stronger. Warming up and stretching after your workout are also important to prevent injury.Following these exercises can help you build strength, endurance, and improve your overall fitness level. Remember to always consult with a healthcare professional before starting any new exercise program.Output(llama-v2-7b-chat) :Exercise is an essential part of a healthy lifestyle, but for beginners, it can be intimidating and overwhelming. Here are some of the best exercises for beginners:1. Bodyweight exercises: These exercises are great for beginners because they don’t require any equipment. Examples include push-ups, squats, lunges, and planks.2. Yoga: Yoga is a low-impact exercise that can help improve flexibility, balance, and strength. It’s also great for reducing stress and improving overall well-being.3. Brisk walking: Walking is a simple and accessible form of exercise that can be done anywhere. It’s a great way to get your heart rate up and improve cardiovascular health.4. Swimming: Swimming is a low-impact exercise that can help improve cardiovascular health and strengthen muscles. It’s also easy on the joints, making it a great option for beginners.5. Cycling: Cycling is another low-impact exercise that can be done indoors or outdoors. It’s a great way to improve cardiovascular health and build leg strength.6. Resistance band exercises: Resistance bands are lightweight and portable, making them easy to use at home or on the go. They can help improve strength and flexibility in the upper body, lower body, and core.7. Pilates: Pilates is a form of exercise that focuses on core strength, flexibility, and body control. It’s a great option for beginners because it’s low-impact and can help improve posture and balance.8. Dance-based workouts: Dance-based workouts, such as Zumba or Hip Hop Abs, are fun and energetic ways to get in shape. They often involve aerobic exercise and strength training, making them a great option for beginners.9. Jumping rope: Jumping rope is a high-intensity exercise that can help improve cardiovascular health and burn calories. It’s also easy to do at home with a simple jump rope.10. Kettlebell exercises: Kettlebells are small weights that can be used to perform a variety of exercises, including swings, squats, and presses. They’re a great way to improve strength and cardiovascular health.Remember to always consult with a doctor before starting any new exercise program, especially if you’re a beginner. It’s also important to start slowly and gradually increase the intensity and duration of your workouts as you become more comfortable.
body
MobileLLM: Optimizing Sub-billion Parameter Language Modelsfor On-Device Use Cases
Acknowledgment
2402.14905
Broader ImpactThis paper advocates for the adoption of sub-billion scale large language models in on-device applications, aiming to mitigate energy consumption during LLM inference. The proposed approach is promising in alleviating computational costs associated with LLM deployment.ReferencesAinslie et al. (2023) Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y., Lebron, F., and Sanghai, S.GQA: Training generalized multi-query transformer models from multi-head checkpoints.InEMNLP, 2023.Biderman et al. (2023) Biderman, S., Schoelkopf, H., Anthony, Q. G., Bradley, H., O’Brien, K., Hallahan, E., Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., et al.Pythia: A suite for analyzing large language models across training and scaling.InInternational Conference on Machine Learning, pp.  2397–2430. PMLR, 2023.Bisk et al. (2020) Bisk, Y., Zellers, R., Gao, J., Choi, Y., et al.Piqa: Reasoning about physical commonsense in natural language.InProceedings of the AAAI conference on artificial intelligence, volume 34, pp.  7432–7439, 2020.Black et al. (2022) Black, S., Biderman, S., Hallahan, E., Anthony, Q., Gao, L., Golding, L., He, H., Leahy, C., McDonell, K., Phang, J., et al.Gpt-neox-20b: An open-source autoregressive language model.arXiv preprint arXiv:2204.06745, 2022.Chowdhery et al. (2023) Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al.Palm: Scaling language modeling with pathways.Journal of Machine Learning Research, 24(240) :1–113, 2023.Clark et al. (2019) Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., and Toutanova, K.Boolq: Exploring the surprising difficulty of natural yes/no questions.arXiv preprint arXiv:1905.10044, 2019.Clark et al. (2018) Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O.Think you have solved question answering? try arc, the ai2 reasoning challenge.arXiv preprint arXiv:1803.05457, 2018.Dao et al. (2022) Dao, T., Fu, D., Ermon, S., Rudra, A., and Ré, C.Flashattention: Fast and memory-efficient exact attention with io-awareness.Advances in Neural Information Processing Systems, 35:16344–16359, 2022.Dauphin et al. (2017) Dauphin, Y. N., Fan, A., Auli, M., and Grangier, D.Language modeling with gated convolutional networks.InInternational conference on machine learning, pp.  933–941. PMLR, 2017.Dettmers et al. (2022) Dettmers, T., Lewis, M., Belkada, Y., and Zettlemoyer, L.Llm. int8 () : 8-bit matrix multiplication for transformers at scale.arXiv preprint arXiv:2208.07339, 2022.Dey et al. (2023) Dey, N., Gosal, G., Khachane, H., Marshall, W., Pathria, R., Tom, M., Hestness, J., et al.Cerebras-gpt: Open compute-optimal language models trained on the cerebras wafer-scale cluster.arXiv preprint arXiv:2304.03208, 2023.Frantar & Alistarh (2023) Frantar, E. and Alistarh, D.Sparsegpt: Massive language models can be accurately pruned in one-shot.InInternational Conference on Machine Learning, pp.  10323–10337. PMLR, 2023.Frantar et al. (2022) Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D.Gptq: Accurate post-training quantization for generative pre-trained transformers.arXiv preprint arXiv:2210.17323, 2022.Ganesan et al. (2021) Ganesan, V., Ramesh, G., and Kumar, P.Supershaper: Task-agnostic super pre-training of bert models with variable hidden dimensions.arXiv preprint arXiv:2110.04711, 2021.Google (2023) Google.Pixel 8 pro tech specs.https://store.google.com/gb/product/pixel_8_pro_specs, 2023.Guo et al. (2020) Guo, Z., Zhang, X., Mu, H., Heng, W., Liu, Z., Wei, Y., and Sun, J.Single path one-shot neural architecture search with uniform sampling.InComputer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XVI 16, pp.  544–560. Springer, 2020.Han et al. (2016) Han, S., Liu, X., Mao, H., Pu, J., Pedram, A., Horowitz, M. A., and Dally, W. J.Eie: Efficient inference engine on compressed deep neural network.ACM SIGARCH Computer Architecture News, 44(3) :243–254, 2016.Hinton et al. (2015) Hinton, G., Vinyals, O., Dean, J., et al.Distilling the knowledge in a neural network.arXiv preprint arXiv:1503.02531, 2(7) , 2015.Hristov (2022) Hristov, V.A16 bionic explained: what’s new in apple’s pro-grade mobile chip?https://www.phonearena.com/news/A16-Bionic-explained-whats-new_id142438, 2022.Jawahar et al. (2023) Jawahar, G., Yang, H., Xiong, Y., Liu, Z., Wang, D., Sun, F., Li, M., Pappu, A., Oguz, B., Abdul-Mageed, M., et al.Mixture-of-supernets: Improving weight-sharing supernet training with architecture-routed mixture-of-experts.arXiv preprint arXiv:2306.04845, 2023.Joshi et al. (2017) Joshi, M., Choi, E., Weld, D. S., and Zettlemoyer, L.Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension.arXiv preprint arXiv:1705.03551, 2017.Kaplan et al. (2020) Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D.Scaling laws for neural language models.arXiv preprint arXiv:2001.08361, 2020.Katharopoulos et al. (2020) Katharopoulos, A., Vyas, A., Pappas, N., and Fleuret, F.Transformers are rnns: Fast autoregressive transformers with linear attention.InInternational conference on machine learning, pp.  5156–5165. PMLR, 2020.Kim et al. (2023) Kim, S., Hooper, C., Gholami, A., Dong, Z., Li, X., Shen, S., Mahoney, M. W., and Keutzer, K.Squeezellm: Dense-and-sparse quantization.arXiv preprint arXiv:2306.07629, 2023.Kingma & Ba (2014) Kingma, D. P. and Ba, J.Adam: A method for stochastic optimization.arXiv preprint arXiv:1412.6980, 2014.Kitaev et al. (2020) Kitaev, N., Kaiser, Ł., and Levskaya, A.Reformer: The efficient transformer.arXiv preprint arXiv:2001.04451, 2020.Lai et al. (2017) Lai, G., Xie, Q., Liu, H., Yang, Y., and Hovy, E.Race: Large-scale reading comprehension dataset from examinations.arXiv preprint arXiv:1704.04683, 2017.Li et al. (2023) Li, X., Zhang, T., Dubois, Y., Taori, R., Gulrajani, I., Guestrin, C., Liang, P., and Hashimoto, T. B.Alpacaeval: An automatic evaluator of instruction-following models.https://github.com/tatsu-lab/alpaca_eval, 2023.Liu et al. (2023a) Liu, J., Gong, R., Wei, X., Dong, Z., Cai, J., and Zhuang, B.Qllm: Accurate and efficient low-bitwidth quantization for large language models.arXiv preprint arXiv:2310.08041, 2023a.Liu et al. (2023b) Liu, S.-y., Liu, Z., Huang, X., Dong, P., and Cheng, K.-T.Llm-fp4: 4-bit floating-point quantized transformers.arXiv preprint arXiv:2310.16836, 2023b.Liu et al. (2023c) Liu, Z., Oguz, B., Zhao, C., Chang, E., Stock, P., Mehdad, Y., Shi, Y., Krishnamoorthi, R., and Chandra, V.Llm-qat: Data-free quantization aware training for large language models.arXiv preprint arXiv:2305.17888, 2023c.Malladi et al. (2012) Malladi, K. T., Lee, B. C., Nothaft, F. A., Kozyrakis, C., Periyathambi, K., and Horowitz, M.Towards energy-proportional datacenter memory with mobile dram.ACM SIGARCH Computer Architecture News, 40(3) :37–48, 2012.Mihaylov et al. (2018) Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A.Can a suit of armor conduct electricity? a new dataset for open book question answering.arXiv preprint arXiv:1809.02789, 2018.Peng et al. (2023) Peng, B., Alcaide, E., Anthony, Q., Albalak, A., Arcadinho, S., Cao, H., Cheng, X., Chung, M., Grella, M., GV, K. K., et al.Rwkv: Reinventing rnns for the transformer era.arXiv preprint arXiv:2305.13048, 2023.Reid et al. (2021) Reid, M., Marrese-Taylor, E., and Matsuo, Y.Subformer: Exploring weight sharing for parameter efficiency in generative transformers.arXiv preprint arXiv:2101.00234, 2021.Roy et al. (2021) Roy, A., Saffar, M., Vaswani, A., and Grangier, D.Efficient content-based sparse attention with routing transformers.Transactions of the Association for Computational Linguistics, 9:53–68, 2021.Sakaguchi et al. (2021) Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y.Winogrande: An adversarial winograd schema challenge at scale.Communications of the ACM, 64(9) :99–106, 2021.Sap et al. (2019) Sap, M., Rashkin, H., Chen, D., LeBras, R., and Choi, Y.Socialiqa: Commonsense reasoning about social interactions.arXiv preprint arXiv:1904.09728, 2019.Scao et al. (2022) Scao, T. L., Fan, A., Akiki, C., Pavlick, E., Ilić, S., Hesslow, D., Castagné, R., Luccioni, A. S., Yvon, F., Gallé, M., et al.Bloom: A 176b-parameter open-access multilingual language model.arXiv preprint arXiv:2211.05100, 2022.Shen et al. (2022) Shen, Z., Liu, Z., and Xing, E.Sliced recursive transformer.InEuropean Conference on Computer Vision, pp.  727–744. Springer, 2022.Sheng et al. (2023) Sheng, Y., Zheng, L., Yuan, B., Li, Z., Ryabinin, M., Chen, B., Liang, P., Ré, C., Stoica, I., and Zhang, C.Flexgen: High-throughput generative inference of large language models with a single gpu.InInternational Conference on Machine Learning, pp.  31094–31116. PMLR, 2023.Sun et al. (2023) Sun, M., Liu, Z., Bair, A., and Kolter, J. Z.A simple and effective pruning approach for large language models.arXiv preprint arXiv:2306.11695, 2023.Tan & Le (2019) Tan, M. and Le, Q.Efficientnet: Rethinking model scaling for convolutional neural networks.InInternational conference on machine learning, pp.  6105–6114. PMLR, 2019.Taylor et al. (2022) Taylor, R., Kardas, M., Cucurull, G., Scialom, T., Hartshorn, A., Saravia, E., Poulton, A., Kerkez, V., and Stojnic, R.Galactica: A large language model for science.arXiv preprint arXiv:2211.09085, 2022.Timiryasov & Tastet (2023) Timiryasov, I. and Tastet, J.-L.Baby llama: knowledge distillation from an ensemble of teachers trained on a small dataset with no performance penalty.arXiv preprint arXiv:2308.02019, 2023.Touvron et al. (2023a) Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al.Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023a.Touvron et al. (2023b) Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Ferrer, C. C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshorn, A., Hosseini, S., Hou, R., Inan, H., Kardas, M., Kerkez, V., Khabsa, M., Kloumann, I., Korenev, A., Koura, P. S., Lachaux, M.-A., Lavril, T., Lee, J., Liskovich, D., Lu, Y., Mao, Y., Martinet, X., Mihaylov, T., Mishra, P., Molybog, I., Nie, Y., Poulton, A., Reizenstein, J., Rungta, R., Saladi, K., Schelten, A., Silva, R., Smith, E. M., Subramanian, R., Tan, X. E., Tang, B., Taylor, R., Williams, A., Kuan, J. X., Xu, P., Yan, Z., Zarov, I., Zhang, Y., Fan, A., Kambadur, M., Narang, S., Rodriguez, A., Stojnic, R., Edunov, S., and Scialom, T.Llama 2: Open foundation and fine-tuned chat models, 2023b.Wang et al. (2020) Wang, S., Li, B. Z., Khabsa, M., Fang, H., and Ma, H.Linformer: Self-attention with linear complexity.arXiv preprint arXiv:2006.04768, 2020.Wu et al. (2019) Wu, B., Dai, X., Zhang, P., Wang, Y., Sun, F., Wu, Y., Tian, Y., Vajda, P., Jia, Y., and Keutzer, K.Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search.InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.  10734–10742, 2019.Xia et al. (2023a) Xia, H., Zheng, Z., Li, Y., Zhuang, D., Zhou, Z., Qiu, X., Li, Y., Lin, W., and Song, S. L.Flash-llm: Enabling cost-effective and highly-efficient large generative model inference with unstructured sparsity.arXiv preprint arXiv:2309.10285, 2023a.Xia et al. (2023b) Xia, M., Gao, T., Zeng, Z., and Chen, D.Sheared llama: Accelerating language model pre-training via structured pruning.arXiv preprint arXiv:2310.06694, 2023b.Xiao et al. (2023) Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., and Han, S.Smoothquant: Accurate and efficient post-training quantization for large language models.InInternational Conference on Machine Learning, pp.  38087–38099. PMLR, 2023.Xiong et al. (2021) Xiong, Y., Zeng, Z., Chakraborty, R., Tan, M., Fung, G., Li, Y., and Singh, V.Nyströmformer: A nyström-based algorithm for approximating self-attention.InProceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp.  14138–14148, 2021.Xu et al. (2021) Xu, J., Tan, X., Luo, R., Song, K., Li, J., Qin, T., and Liu, T.-Y.Nas-bert: task-agnostic and adaptive-size bert compression with neural architecture search.InProceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp.  1933–1943, 2021.Yao et al. (2022) Yao, Z., Yazdani Aminabadi, R., Zhang, M., Wu, X., Li, C., and He, Y.Zeroquant: Efficient and affordable post-training quantization for large-scale transformers.Advances in Neural Information Processing Systems, 35:27168–27183, 2022.Zellers et al. (2019) Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y.Hellaswag: Can a machine really finish your sentence?arXiv preprint arXiv:1905.07830, 2019.Zhang et al. (2022) Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al.Opt: Open pre-trained transformer language models.arXiv preprint arXiv:2205.01068, 2022.Zheng et al. (2023) Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., et al.Judging llm-as-a-judge with mt-bench and chatbot arena.arXiv preprint arXiv:2306.05685, 2023.Zoph & Le (2016) Zoph, B. and Le, Q. V.Neural architecture search with reinforcement learning.arXiv preprint arXiv:1611.01578, 2016.AppendixAppendix AImpact of Each Design ChoiceThis section presents comprehensive tabulated results for theimproving sub-billion scale LLM designexperiments, at the model sizes of 125M and 350M. Looking at the results in TableA, transitioning from the traditional Feedforward Network ()  to SwiGLU yields a accuracy improvement of 1.3% for both model sizes. Further increasing the model depth enhances accuracy by 0.9%/1.1% for 125M/350M models, respectively. Then, introducing input and output embedding sharing achieves a parameter reduction of approximately 10%, while with only marginal accuracy drops of 0.2% for the 125M and 0.6% for 350M models. Additionally, following in findings in SectionC, we incorporate grouped query attention with head dimension equals to 64, and head number to be near 4to the kv head number, while increasing the embedding dimension to preserve model size. This modification further results in a performance boost of 0.4%/0.7% for 125M/350M models. Combining these techniques establishes a strong baseline network denoted asMobileLLM. Finally, the immediate block-wise weight-sharing technique contributes an additional accuracy gain of 1.1% for models trained on 0.25 trillion tokens, and resulting in the modelMobileLLM-LS. Final models includingMobileLLMandMobileLLM-LSare trained with 1 trillion tokens.Table 8:Ablation study on the impact of each design choice to the model accuracy on zero-shot common sense reasoning tasks. Corresponding to the bar chart in Figure3. Here, L, H,denotes number of layers, heads, kv-heads, respectively and dim denotes embedding dimension.TechniquesLHDim#Params(M) ARC-eARC-cBoolQPIQASIQAHellaSwagOBQAWinoGrandeAvg.125M\hdashline[0.8pt/1pt]
Baseline model121212768134.141.325.257.562.041.931.131.250.842.6+ SwiGLU in FFN121212768134.143.128.958.162.342.334.631.550.143.9+ Use deep-thin structure3088512135.043.626.158.062.542.636.537.551.544.8+ Embedding share3088512118.644.426.056.262.843.135.936.052.644.6+ Multi-query attention3093576124.645.527.758.364.641.936.435.450.445.0(Train on 1T token) 3093576124.643.927.160.265.342.438.939.553.146.3+ Layer sharing3093576124.644.427.061.565.143.037.637.852.046.1(Train on 1T token) 3093576124.645.828.760.465.742.939.541.152.147.0350M\hdashline[0.8pt/1pt]
Baseline model1520201280376.850.327.653.868.144.142.640.152.447.4+ SwiGLU in FFN1520201280386.749.230.659.167.744.343.241.054.248.7+ Use deep-thin structure321414896380.350.731.459.467.843.346.243.856.249.8+ Embedding share321414896351.649.932.060.367.943.247.038.954.849.2+ Multi-query attention32155960345.351.431.361.068.143.647.241.655.449.9(Train on 1T token) 32155960345.353.833.562.468.644.749.640.057.651.3+ Layer sharing32155960345.351.935.259.668.943.447.243.358.451.0(Train on 1T token) 32155960345.354.432.562.869.844.150.645.857.252.1Appendix BDepth vs WidthWe provide network depth versus width exploration results on zero-shot reasoning tasks in TableB, as well as results on question answering and reading comprehension tasks in TableB. The findings indicate that shallow architectures with fewer than 10 layers perform poorly in reasoning or handling comprehension tasks. Models with 10-20 layers exhibit improved accuracy, while further increasing depth continues to provide significant benefits across all three tasks. Notably, the optimal depth is found to be near 30 layers for sub-billion scale models.Table 9:Ablation study on depth versus width in architecture design, illustrated in Figure4(a) (b) . For compact models, prioritizing depth over width yields superior performance, assessed through zero-shot common sense reasoning tasks.#Layer#HeadsDim#Params(M) ARC-eARC-cBoolQPIQASIQAHellaSwagOBQAWinoGrandeAvg.4201280163.242.026.161.061.541.332.931.449.843.36161024142.642.225.551.062.641.633.532.150.042.3814896138.143.625.548.661.840.934.037.552.043.01212768134.143.128.958.162.342.334.631.550.143.91810640132.441.025.559.362.641.634.236.750.143.9249576132.443.627.059.763.842.036.233.752.244.8308512135.043.626.158.062.542.636.537.551.544.8427448134.643.326.157.863.341.736.335.952.044.5626384134.344.126.759.064.742.436.233.151.344.7\hdashline[0.8pt/1pt]
5322048388.047.928.862.265.242.838.139.052.647.110241536381.449.129.660.667.144.042.741.254.448.612221408379.949.931.659.168.242.644.043.753.849.115201280386.749.230.659.167.744.343.241.054.248.719181152376.350.830.956.369.143.845.239.654.548.824161024373.850.733.858.868.643.545.040.054.949.42815960371.151.733.257.667.942.946.037.753.948.93214896380.350.731.459.467.843.346.243.856.249.84612768374.749.730.559.568.145.144.943.455.549.66610640376.250.531.861.067.443.846.040.155.649.5Table 10:Ablation study on depth vs. width in architecture design on TQA and RACE datasets, depicted in Figure4(c-f) .TQA(F1 score) RACE(Acc) #Layer#HeadsDim#Params(M) 1-shot5-shot64-shotmiddlehigh4201280163.22.03.93.933.226.06161024142.63.54.94.629.225.5814896138.14.25.24.729.224.31212768134.15.16.16.335.627.31810640132.44.76.85.234.327.4249576132.45.97.27.335.328.6308512135.06.26.86.934.227.9427448134.66.17.27.234.428.6626384134.36.07.07.338.928.9\hdashline[0.8pt/1pt]
5322048388.05.67.88.638.129.010241536381.49.011.812.740.630.012221408379.911.712.913.342.431.515201280386.710.713.114.242.630.619181152376.311.313.313.542.632.424161024373.811.813.514.443.031.92815960371.112.513.614.942.832.33214896380.312.614.515.344.232.24612768374.713.015.415.644.731.46610640376.212.515.115.744.032.5Appendix CNumber of Heads and Key-Value HeadsWe provide detailed experimental results assessing the impact of the number of attention heads and key-value heads on zero-shot reasoning accuracy in TableC. Our study involves two baseline architectures: a 8-layer 125M model with a embedding dimension of 896, and a 15-layer 350M model with a embedding dimension of 1280. We conduct head size sweeps in {8, 16, 32}. The findings shown in TableCindicate that using 16 heads, with a head dimension close to 64, and 4 key-value heads, yields the best accuracy and memory trade-off. This setting serves as a guiding principle in our model architecture design.Table 11:Ablation study investigating the impact of the number of attention heads and key-value heads.Model#Heads#KV-Heads#Params(M) ARC-eARC-cBoolQPIQASIQAHellaSwagOBQAWinoGrandeAvg.3232138.142.126.958.462.242.133.836.752.444.33216131.741.226.457.762.542.333.334.252.943.8328128.542.627.361.161.941.932.235.052.044.2324126.843.126.859.862.741.432.534.451.144.0322126.039.826.759.459.442.031.332.652.943.0321125.641.024.359.160.841.231.435.452.043.1125M1616138.141.625.761.162.443.134.436.951.644.6# layers=8168131.742.426.460.763.441.933.534.751.544.3dim=896164128.542.525.662.362.441.833.035.954.544.7162126.841.725.356.961.742.032.932.654.543.5161126.040.426.361.863.241.732.034.050.443.788138.141.425.058.361.741.733.335.953.243.884131.743.328.258.361.842.833.830.953.044.082128.540.726.258.161.241.632.834.851.543.481126.842.524.859.462.342.032.036.351.343.8\hdashline[0.8pt/1pt]3232386.748.630.459.767.243.944.040.953.948.63216362.148.931.657.668.443.443.838.654.948.4328349.848.333.161.067.242.642.139.053.948.4324343.747.229.859.467.243.542.542.554.148.3322340.647.630.362.466.942.641.638.652.047.7321339.048.527.356.367.142.940.936.753.346.6350M1616386.750.830.662.368.643.545.143.852.449.6# layers=15168362.148.530.759.467.343.843.841.353.348.5dim=1280164349.849.930.660.069.243.544.241.855.849.4162343.749.328.455.067.342.742.640.354.547.5161340.649.229.358.867.443.542.139.952.847.988386.751.033.258.267.043.943.942.654.749.384362.150.031.360.266.042.743.740.953.748.682349.849.330.459.967.942.843.538.953.848.381343.748.027.761.567.143.142.537.354.747.8Appendix DLayer Sharing Number AblationTable 12:Ablation study on the impact of varying layer repetition numbers.Model SizeRepeat TimesARC-eARC-cBoolQPIQASIQAHellaSwagOBQAWinoGrandeAvg.125Mbaseline41.625.761.162.443.134.436.951.644.6block-wise share repeat243.927.961.564.341.535.535.150.245.0block-wise share repeat343.327.660.963.042.335.634.152.945.0block-wise share repeat442.026.662.364.441.536.235.853.945.3\hdashline[0.8pt/1pt]350Mbaseline50.830.662.368.643.545.143.852.449.6block-wise share repeat251.530.859.668.243.947.744.755.050.2block-wise share repeat349.632.057.368.443.947.040.656.249.4block-wise share repeat452.232.660.069.044.547.641.855.250.4We extended our investigation to determine the optimal number of layer repetitions. The experiment involved the 8-layer 125M model with a embedding dimension of 896 and the 15-layer 350M model with a embedding dimension of 1280. The results in TableDdemonstrate that when we double the layer number and with each two transformer blocks sharing weights, the accuracy is enhanced by 0.4-0.6%. However, as we further triple or quadruple the layer repetition times, this accuracy enhancement effect diminishes. Consequently, in our experiments, we adopt a configuration where every two blocks share weights, effectively doubling the total number of layers.Appendix ECompatibility with QuantizationThis section explores the compatibility of quantization with the proposed model architecture and layer sharing. We employ a straightforward per-token min-max quantization, quantizing both weight and activation to 8-bit using post-training quantization (PTQ) . Experiments are conducted on bothMobileLLMandMobileLLM-LSwith model size of 125M and 350M, trained on 0.25T tokens. Results in TableEdemonstrate that W8A8 PTQ results in a modest accuracy drop of within 0.5% and is compatible with the proposed layer sharing method.Table 13:Ablation study: 8-bit weight, 8-bit activation post-training quantization results on zero-shot common sense reasoning tasks. Quantized models achieve accuracy gap of within 0.5% compared to the full-precision BF16 counterpart.ModelPrecisionARC-eARC-cBoolQPIQASIQAHellaSwagOBQAWinoGrandeAvg.GapMobileLLM-125MBF1645.527.758.364.641.936.435.450.445.0–MobileLLM-125MW8A845.227.158.365.041.736.233.651.044.80.2\hdashline[0.8pt/1pt]MobileLLM-LS-125MBF1644.427.061.565.143.037.637.852.046.1–MobileLLM-LS-125MW8A844.027.560.964.643.137.737.751.045.80.3\hdashline[0.8pt/1pt]MobileLLM-350MBF1651.431.361.068.143.647.241.655.449.9–MobileLLM-350MW8A851.432.161.168.843.147.140.655.149.90.0\hdashline[0.8pt/1pt]MobileLLM-LS-350MBF1651.935.259.668.943.447.243.358.451.0–MobileLLM-LS-350MW8A851.333.859.569.143.747.243.057.050.60.4Appendix FKnowledge DistillationThe results of integrating knowledge distillation (KD) into small model pre-training are presented in TableF. LLaMA-v2 7B models serve as the teacher, and the KD loss is computed using cross-entropy between the logits from the large pre-trained teacher model (i.e., LLaMA-v2 7B)  and the small student network (i.e., 125M or 350M models) :(1) Here,denotes thesample in the current batch withtotal samples in the batch, andrepresents the number of classes, which, in our case, equals the size of the vocabulary.andare the teacher and student networks, respectively.The results in TableFindicate that adding KD loss is comparable or even lower than solely using the next token as labels. However, it’s noteworthy that the training time using KD isslower than training from scratch using labels. All models are trained on 32 A100 80G GPUs with a batch size of 32 for 120k iterations. Consequently, we opt to use labels in our experiments.Table 14:Ablation study on employing LLaMA-v2 7B teacher’s output as soft labels for knowledge distillation (KD) . Results indicate a slight degradation in performance when incorporating KD loss compared to only using hard labels.ModelTraining LossTraining TimeARC-eARC-cBoolQPIQASIQAHellaSwagOBQAWinoGrandeAvg.125M modelLabel29h43.128.958.162.342.334.631.550.143.9125M modelLabel + KD93h41.828.558.561.641.134.532.751.643.8\hdashline[0.8pt/1pt]
350M modelLabel42h50.231.856.967.744.345.840.855.549.1350M modelLabel + KD109h48.731.860.767.443.245.938.953.748.8Appendix GDatasets and BenchmarksMobileLLMis assessed on zero-shot common sense reasoning (BoolQ, PIQA, SIQA, HellaSwag, Winogrande, ARC, OBQA) , question answering (TriviaQA) , and reading comprehension (RACE)  tasks. Additionally, we evaluate our chat models on MT-Bench and AlpacaEval benchmarks. We also generated a API calling dataset to fine-tune and evaluate models for this particular task.G.1Zero-shot Common Sense Reasoning tasksBoolQis a reading comprehension dataset focused on naturally occurring yes/no questions. Each instance includes a question (Q) , an excerpt from a passage (P) , and an answer (A) , with an added explanation for enhanced clarity.PIQA, abbreviated for Physical Interaction: Question Answering, serves as a benchmark for evaluating and studying the capacity of natural language models in comprehending physical commonsense understanding.SIQA, abbreviated for Social Interaction Question Answering, is designed to measure the social and emotional intelligence of computational models through multiple-choice question answering.HellaSwagserves as a benchmark for physically situated commonsense natural language inference. It comprises four-way multiple-choice problems that are considered trivial for humans (>95% accuracy)  but pose a challenge for language models.WinoGrandeis a benchmark for commonsense reasoning. It consists of a set of 273 expert-crafted pronoun resolution problems, deliberately designed to be unsolvable for statistical models relying on selectional preferences or word associations.ARC, the AI2 Reasoning Challenge, is a compilation of 7787 natural science questions. It is divided into a Challenge Set and an Easy Set, with the Challenge Set exclusively comprising questions that were answered incorrectly by both a retrieval-based algorithm and a word co-occurrence algorithm.OBQAis a dataset consisting of approximately 6000 questions designed for open book question answering. The task involves integrating a corpus of provided science facts (open book)  with external broad common knowledge. Providing correct answers necessitates leveraging simple common knowledge beyond the core facts provided.G.2Question Answering TasksTriviaQAis a closed-book question answering benchmark. It encompasses over 650,000 question-answer evidence triples derived by combining 95,000 question-answer pairs authored by Trivia enthusiasts. Each question is supported by an average of six evidence documents.G.3Reading Comprehension TasksRACEis a large-scale reading comprehension dataset with more than 28,000 passages and nearly 100,000 questions. The dataset is collected from English examinations in China, which are designed for middle school and high school students.G.4Chat BenchmarksAlpacaEvalis a single-turn benchmark featuring 805 questions across various topics. In the evaluation of this paper, models generate responses and are scored by GPT-4, with the ultimate metric being the pairwise win-rate against a baseline model (text-davinci-001) .MT-Benchis a multi-turn benchmark comprising 160 questions spanning eight diverse knowledge domains. In this evaluation, models are tasked with answering an initial question and subsequently providing a second response to a predefined follow-up question. Each model response is rated by GPT-4 on a scale from 1 to 10, and the final score is determined by the mean over the two turns.G.5API Calling DatasetThe API calling dataset is generated by instructing a language model to simulate a conversation between a human and an agent. This process instructs the language model to explicitly mention tasks that should be performed using designated APIs throughout the simulated conversation.
Examples of a conversation turn in generated dataset are shown in the following boxes.{"role": "USER", "message": "Can you help me send a SMS message: We are looking forward to seeing you"},{"role": "PARSER", "message": "style_transfer(text = "We are looking forward to seeing you", style = ["sms"], contact = None) "},{"role": "AGENT", "message": "Sure! Here’s a possible SMS message: Hey [name], we are looking forward to seeing you! Wishing you all well."},{"role": "USER", "message": "How did the S&P 500 perform last month?"},{"role": "PARSER", "message": "get_stock_information(company_name = "S&P 500", ticker_symbol = "GSPC", date_time = "last month", financial_metrics = "price") "},{"role": "AGENT", "message": "The S&P 500 closed at $2,977.94 last month, with a monthly return of 2.5%."},{"role": "USER", "message": "Is the San Diego Zoo open during the winter months?"},{"role": "PARSER", "message": "get_local_information(point_of_interest = "San Diego Zoo", attributes = "hours of operation") "},{"role": "AGENT", "message": "The San Diego Zoo is open year-round, but the hours of operation vary depending on the time of year. During the winter months, the zoo is open from 9:00 AM to 5:00 PM."},{"role": "USER", "message": "What is the most recent news concerning aliens?"},{"role": "PARSER", "message": "news(query = "latest alien news") "},{"role": "AGENT", "message": "According to recent news articles, NASA has announced the discovery of a new planet outside of our solar system that has the potential to support alien life. The planet, named Kepler-186f, is located in the constellation Cygnus and is the first known Earth-sized planet to be found in the habitable zone of a star. While this news does not necessarily prove the existence of aliens, it does provide hope for the possibility of finding life beyond Earth. Additionally, a recent study published in the journal Nature has suggested that the universe may be teeming with alien civilizations, with estimates suggesting that there could be tens of billions of intelligent civilizations in the observable universe. However, it’s important to note that these findings are still being researched and debated in the scientific community."},{"role": "USER", "message": "Who won the last Super Bowl?"},{"role": "PARSER", "message": "get_sports_information(match = "", sport_type = "NFL", metric = "winner") "},{"role": "AGENT", "message": "The last Super Bowl was won by the Kansas City Chiefs."},Appendix HApproximate calculation of envisioned future GPU consumption#GPUs =(population size) (individual time999According to the statistics that YouTube users spending an average of 23 hours per month (source)  and Instagram users dedicating 12 hours monthly to the app (source) , we have reason to believe the interaction with LLM can easily surpass the cumulative engagement of these platforms in the future.) (approximating FLOPs/token for 220B GPT-4 model with one expert activated101010https://the-decoder.com/gpt-4-has-a-trillion-parameters/) (token/s) (h) (s)  / ((FLOPs/s, H100 computation capacity ) (h) (s) ) This assumes short sequence lengths (i.e.hundreds of tokens, not tens of thousands of tokens) , where the compute is dominated by linear layers.
For very long sequences, the self-attention computation would dominate, and this would necessitate a more complicated formula.Appendix IChat Generations fromMobileLLMWe show a several examples of chat output generated withMobileLLMwhile responding to various questions.Input:Are there any particular physical benefits to mindful walking, such as improved posture or increased physical fitness?Output(MobileLLM-125M) :Yes, there are several physical benefits to mindful walking. Here are a few:1. Improved Posture: Mindful walking can help improve your posture by promoting good alignment and reducing stress on the joints. This can lead to better posture and reduced back pain.2. Increased Flexibility: Mindful walking can also help increase flexibility in your body, which can reduce the risk of injury and improve overall health.3. Better Sleep: Mindful walking can help you fall asleep more easily and stay asleep longer, which can improve your sleep quality.4. Reduced Stress: Mindful walking can help reduce stress levels by promoting relaxation and reducing tension in the body.5. Enhanced Mental Health: Mindful walking has been shown to have positive effects on mental health, including reducing symptoms of depression and anxiety.Overall, mindful walking is an excellent way to improve your physical health and well-being. It’s a simple and effective way to get moving and enjoy the benefits it offers. If you’re interested in trying mindful walking, I would recommend starting with shorter walks and gradually increasing the duration over time. You may also want to consider incorporating mindfulness into your daily routine to further enhance your experience.In conclusion, mindful walking is a great way to improve your physical health and well-being. With its numerous physical benefits, it’s easy to see why it’s becoming increasingly popular among people looking for a fun and sustainable way to get moving. So go ahead and give it a try!Output(MobileLLM-350M) :Yes, there are several physical benefits of mindful walking. Firstly, it can improve your posture by reducing the amount of tension in your muscles and joints. Secondly, it can increase your physical fitness by improving your cardiovascular health, strengthening your core muscles, and increasing your flexibility. Additionally, mindful walking can help you to stay more focused on the present moment, which can lead to better sleep quality and reduced stress levels. Finally, it can also help you to develop a greater sense of awareness and appreciation for the world around you, which can enhance your overall well-being.Overall, mindful walking is a great way to improve your physical health and well-being, while also promoting relaxation and mindfulness. Whether you’re looking to reduce stress, improve your posture, or simply enjoy a peaceful walk, mindful walking can be a rewarding and fulfilling experience.If you’re interested in trying out mindful walking, there are many resources available that can guide you through the process. Some popular options include guided meditations, apps that offer mindful walking exercises, and classes that teach you how to practice mindful walking. With a little bit of effort and dedication, anyone can start incorporating mindful walking into their daily routine and reap the many benefits it offers.In conclusion, mindful walking is a simple yet powerful tool that can help you to improve your physical health, reduce stress, and cultivate a deeper sense of awareness and appreciation for the world around you. So why not give it a try and see what kind of benefits you can achieve? Your body and mind will thank you!Output(llama-v2-7b-chat) :Mindful walking can have a range of physical benefits, including:1. Improved posture: By focusing on the sensation of each step and the movement of the body, mindful walking can help improve posture and reduce slouching or hunching.2. Increased physical fitness: Mindful walking can help improve cardiovascular fitness, increase muscle strength and endurance, and boost overall physical fitness.3. Better balance and coordination: By paying attention to the body’s position in space and the movement of the legs and feet, mindful walking can help improve balance and coordination.4. Reduced stress and tension: Mindful walking can help reduce stress and tension in the body, which can lead to improved physical relaxation and well-being.5. Improved flexibility: By focusing on the movement of the body and the sensation of each step, mindful walking can help improve flexibility and range of motion.6. Boosted immune system: Studies have shown that mindfulness practices, including mindful walking, can help boost the immune system and reduce inflammation.7. Improved sleep: Mindful walking can help improve sleep quality and duration by reducing stress and promoting relaxation.
8. Reduced pain: Mindful walking can help reduce pain by focusing on the sensation of each step and the movement of the body, rather than the pain itself.9. Improved mood: Mindful walking can help improve mood and reduce symptoms of depression and anxiety by promoting relaxation and reducing stress.It’s important to note that the physical benefits of mindful walking may vary depending on the individual and the frequency and duration of the practice. However, with regular practice, mindful walking can have a range of positive effects on physical health and well-being.Input:How do I wrap a present neatly?Output(MobileLLM-125M) :1. Choose the right size gift: The size of the present you want to give will depend on your budget and personal preference. If you are looking for a small gift, consider buying a small box or card with a single-use plastic bag.2. Prepare the present: Make sure all the items in the present are clean and tidy. You can use a cleaning kit or a tissue paper to remove any dirt or debris from the present.3. Arrange the presents: Place the present in a place where it is easy to see and touch. Use a pencil to draw a line around the present and arrange it in a way that makes sense to you.4. Add decorations: Decorate the present with stickers, ribbons, or other decorative items. This will add some personality to the present and make it more visually appealing.5. Enjoy the present: Take time to enjoy the present and let yourself feel the warmth and comfort of the present. Smile at yourself and take a moment to appreciate the present.Remember, the most important thing is to have fun and enjoy the present! Happy gift-giving!If you’re feeling adventurous, try making a surprise gift for someone special by wrapping the present in a cute and colorful gift basket. Or, if you’re feeling adventurous, try making a gift for a friend who needs a little extra help with their homework. Whatever you choose, remember to be creative and have fun with your present!With these tips, you’ll be able to create a beautiful and thoughtful present that will last for years to come. Happy gift-giving!Output(MobileLLM-350M) :1. Start by wrapping the present in a decorative gift wrap or ribbon.2. Use a small amount of glue or tape to secure the gift wrap or ribbon to the surface of the present.3. Place the gift wrap or ribbon around the present, making sure it is centered and evenly distributed.4. Use a small amount of glue or tape to secure the gift wrap or ribbon to the surface of the present.5. Once the gift wrap or ribbon is securely attached, you can add any additional decorations or embellishments to the present.6. If you want to add a personal touch, you can write a message or a special message on the gift wrap or ribbon.7. Finally, enjoy your beautifully wrapped present!Remember, wrapping a gift is a fun and creative way to show your loved one how much you care. Have fun and enjoy the process!Output(llama-v2-7b-chat) :Wrapping a present neatly can be a bit of a challenge, but with a few simple tips and tricks, you can create a beautifully wrapped gift that your recipient will love. Here are some tips to help you wrap a present neatly:1. Use the right wrapping paper: Choose a wrapping paper that complements the gift and is the right size for the present. You can also use a patterned paper or a solid color, depending on your preference.2. Measure the gift: Before you start wrapping, measure the gift to make sure you have enough wrapping paper. You can also use a piece of scrap paper to measure the gift and mark the edges with a pencil.3. Use a flat surface: Make sure you have a flat surface to work on, such as a table or countertop. This will help you keep the wrapping paper flat and prevent it from wrinkling.4. Start with the edges: Begin by folding the edges of the wrapping paper over the gift, making sure they are even and smooth. You can use tape to secure the edges if needed.5. Roll the gift: Once you have the edges in place, roll the gift into the center of the wrapping paper. Make sure the gift is securely in place before moving on to the next step.6. Tuck in the edges: As you wrap the gift, tuck the edges of the wrapping paper under the gift, smoothing them out as you go. Make sure the edges are even and smooth, and use tape to secure them if needed.7. Add a ribbon or bow: Once you have the gift wrapped, add a ribbon or bow to the top. This will not only make the gift look more festive, but it will also help to keep the wrapping paper in place.8. Practice makes perfect: Wrapping a present neatly takes a bit of practice, so don’t be discouraged if it doesn’t come out perfectly the first time. Keep practicing, and you’ll get the hang of it in no time!By following these tips, you can create a beautifully wrapped present that your recipient will love. Happy wrapping!Input:What are the best exercises for beginners?Output(MobileLLM-125M) :1. Push-ups: This is a classic exercise that can be done anywhere, anytime. It’s easy to do and requires no equipment.2. Dumbbell Rows: These rows are great for building strength in your legs and core.3. Squats: Squats are a great way to build strength in your legs and core.4. Lunges: Lunges are a great way to improve balance and stability.5. Plank: A plank is a great way to build strength in your back and core.6. Jumping Jacks: Jumping jacks are a fun and effective way to build strength in your legs and core.7. Burpees: Burpees are a great way to build strength in your legs and core.8. Mountain Climbers: Mountain climbers are a great way to build strength in your legs and core.9. Deadlifts: Deadlifts are a great way to build strength in your legs and core.10. Tricep Dips: Tricep dips are a great way to build strength in your triceps.Remember, it’s important to start with simple exercises and gradually increase the intensity as you get stronger. Don’t push yourself too hard or you may find it difficult to maintain proper form. With consistent practice, you will see improvements in your strength and overall fitness.If you have any questions about exercises or healthy living, don’t hesitate to talk to a doctor or physical therapist. They can help you develop a personalized plan that works best for you.In conclusion, there are many exercises that can be done at home, but it’s important to start slowly and gradually increase the intensity over time. Remember to listen to your body and take breaks when needed. And most importantly, enjoy the process!Output(MobileLLM-350M) :Beginners should start with exercises that are easy to do and don’t require a lot of strength or endurance. Here are some exercises that are great for beginners:1. Push-ups: This is a great exercise for beginners because it’s a simple exercise that can be done anywhere. Start with a push-up on your knees and then lower yourself down to the ground.2. Squats: Squats are a great exercise for building leg strength and endurance. Start with a squat on your knees and then lower yourself down to the ground.3. Lunges: Lunges are a great exercise for building leg strength and endurance. Start with a lunge on your front foot and then lower yourself down to the ground.4. Plank: Planks are a great exercise for building core strength and stability. Start with a plank on your forearms and then lower yourself down to the ground.5. Bicycle crunches: Bicycle crunches are a great exercise for building upper body strength and stability. Start with a bicycle crunch on your upper back and then lower yourself down to the ground.6. Plank with leg lifts: This exercise is great for building leg strength and stability. Start with a plank on your forearms and then lift your legs off the ground and hold for a few seconds before lowering them back down.7. Jumping jacks: Jumping jacks are a great exercise for building cardiovascular endurance and strength. Start with a jumping jack on your feet and then jump up and spread your legs.Remember to start with a weight that is comfortable for you and gradually increase the weight as you get stronger. It’s also important to warm up before exercising and to stretch after your workout to prevent injury.In conclusion, beginners should start with exercises that are easy to do and don’t require a lot of strength or endurance. These exercises are great for building leg strength and endurance, and can be done anywhere. Remember to start with a weight that is comfortable for you and gradually increase the weight as you get stronger. Warming up and stretching after your workout are also important to prevent injury.Following these exercises can help you build strength, endurance, and improve your overall fitness level. Remember to always consult with a healthcare professional before starting any new exercise program.Output(llama-v2-7b-chat) :Exercise is an essential part of a healthy lifestyle, but for beginners, it can be intimidating and overwhelming. Here are some of the best exercises for beginners:1. Bodyweight exercises: These exercises are great for beginners because they don’t require any equipment. Examples include push-ups, squats, lunges, and planks.2. Yoga: Yoga is a low-impact exercise that can help improve flexibility, balance, and strength. It’s also great for reducing stress and improving overall well-being.3. Brisk walking: Walking is a simple and accessible form of exercise that can be done anywhere. It’s a great way to get your heart rate up and improve cardiovascular health.4. Swimming: Swimming is a low-impact exercise that can help improve cardiovascular health and strengthen muscles. It’s also easy on the joints, making it a great option for beginners.5. Cycling: Cycling is another low-impact exercise that can be done indoors or outdoors. It’s a great way to improve cardiovascular health and build leg strength.6. Resistance band exercises: Resistance bands are lightweight and portable, making them easy to use at home or on the go. They can help improve strength and flexibility in the upper body, lower body, and core.7. Pilates: Pilates is a form of exercise that focuses on core strength, flexibility, and body control. It’s a great option for beginners because it’s low-impact and can help improve posture and balance.8. Dance-based workouts: Dance-based workouts, such as Zumba or Hip Hop Abs, are fun and energetic ways to get in shape. They often involve aerobic exercise and strength training, making them a great option for beginners.9. Jumping rope: Jumping rope is a high-intensity exercise that can help improve cardiovascular health and burn calories. It’s also easy to do at home with a simple jump rope.10. Kettlebell exercises: Kettlebells are small weights that can be used to perform a variety of exercises, including swings, squats, and presses. They’re a great way to improve strength and cardiovascular health.Remember to always consult with a doctor before starting any new exercise program, especially if you’re a beginner. It’s also important to start slowly and gradually increase the intensity and duration of your workouts as you become more comfortable.
conclusion
MobileLLM: Optimizing Sub-billion Parameter Language Modelsfor On-Device Use Cases
Broader Impact