<!DOCTYPE html><html lang="en">
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<title>[2403.03853] ShortGPT: Layers in Large Language Models are More Redundant Than You Expect</title><meta property="og:description" content="As Large Language Models (LLMs) continue to advance in performance, their size has escalated significantly, with current LLMs containing billions or even trillions of parameters. However, in this study, we discovered t…">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="twitter:card" content="summary">
<meta name="twitter:title" content="ShortGPT: Layers in Large Language Models are More Redundant Than You Expect">
<meta name="twitter:image:src" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta name="twitter:image:alt" content="ar5iv logo">
<meta property="og:title" content="ShortGPT: Layers in Large Language Models are More Redundant Than You Expect">
<meta property="og:site_name" content="ar5iv">
<meta property="og:image" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta property="og:type" content="article">
<meta property="og:url" content="https://ar5iv.labs.arxiv.org/html/2403.03853">

<!--Generated on Fri Apr  5 17:35:43 2024 by LaTeXML (version 0.8.8) http://dlmf.nist.gov/LaTeXML/.-->
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">

<script>
  function detectColorScheme(){
    var theme="light";
    var current_theme = localStorage.getItem("ar5iv_theme");
    if(current_theme){
      if(current_theme == "dark"){
        theme = "dark";
      } }
    else if(!window.matchMedia) { return false; }
    else if(window.matchMedia("(prefers-color-scheme: dark)").matches) {
      theme = "dark"; }
    if (theme=="dark") {
      document.documentElement.setAttribute("data-theme", "dark");
    } else {
      document.documentElement.setAttribute("data-theme", "light"); } }

  detectColorScheme();

  function toggleColorScheme(){
    var current_theme = localStorage.getItem("ar5iv_theme");
    if (current_theme) {
      if (current_theme == "light") {
        localStorage.setItem("ar5iv_theme", "dark"); }
      else {
        localStorage.setItem("ar5iv_theme", "light"); } }
    else {
        localStorage.setItem("ar5iv_theme", "dark"); }
    detectColorScheme(); }
</script>
<link media="all" rel="stylesheet" href="/assets/ar5iv-fonts.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv-site.0.2.2.css">
</head>
<body>
<div class="ltx_page_main">
<div class="ltx_page_content">
<article class="ltx_document ltx_authors_1line">
<h1 class="ltx_title ltx_title_document">ShortGPT: Layers in Large Language Models are More Redundant Than You Expect </h1>
<div class="ltx_authors">
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">
Xin Men<sup id="id11.11.id1" class="ltx_sup"><span id="id11.11.id1.1" class="ltx_text ltx_font_italic">1</span></sup>, Mingyu Xu<sup id="id12.12.id2" class="ltx_sup"><span id="id12.12.id2.1" class="ltx_text ltx_font_italic">1</span></sup>, Qingyu Zhang<sup id="id13.13.id3" class="ltx_sup"><span id="id13.13.id3.1" class="ltx_text ltx_font_italic">2</span></sup>, Bingning Wang<sup id="id14.14.id4" class="ltx_sup"><span id="id14.14.id4.1" class="ltx_text ltx_font_italic">1</span></sup> 
<br class="ltx_break"><span id="id9.9.5" class="ltx_text ltx_font_bold">Hongyu Lin<sup id="id9.9.5.1" class="ltx_sup"><span id="id9.9.5.1.1" class="ltx_text ltx_font_medium ltx_font_italic">2</span></sup>, Yaojie Lu<sup id="id9.9.5.2" class="ltx_sup"><span id="id9.9.5.2.1" class="ltx_text ltx_font_medium ltx_font_italic">2</span></sup>, Xianpei Han<sup id="id9.9.5.3" class="ltx_sup"><span id="id9.9.5.3.1" class="ltx_text ltx_font_medium ltx_font_italic">2</span></sup> and Weipeng Chen <sup id="id9.9.5.4" class="ltx_sup"><span id="id9.9.5.4.1" class="ltx_text ltx_font_medium ltx_font_italic">1</span></sup> 
<br class="ltx_break"><sup id="id9.9.5.5" class="ltx_sup"><span id="id9.9.5.5.1" class="ltx_text ltx_font_medium">1</span></sup></span> Baichuan Inc. 
<br class="ltx_break"><sup id="id15.15.id5" class="ltx_sup">2</sup> Chinese Information Processing Laboratory
<br class="ltx_break">Institute of Software, Chinese Academy of Sciences
<br class="ltx_break"><span id="id16.16.id6" class="ltx_text ltx_font_typewriter">daniel@baichuan-inc.com</span> 
<br class="ltx_break"><span id="id17.17.id7" class="ltx_text ltx_font_typewriter">{zhangqingyu2024,hongyu,yaojie,xianpei}@iscas.ac.cn</span> 
<br class="ltx_break">
</span><span class="ltx_author_notes">Corresponding author</span></span>
</div>

<div class="ltx_abstract">
<h6 class="ltx_title ltx_title_abstract">Abstract</h6>
<p id="id18.id1" class="ltx_p">As Large Language Models (LLMs) continue to advance in performance, their size has escalated significantly, with current LLMs containing billions or even trillions of parameters. However, in this study, we discovered that many layers of LLMs exhibit high similarity, and some layers play a negligible role in network functionality. Based on this observation, we define a metric called Block Influence (BI) to gauge the significance of each layer in LLMs. We then propose a straightforward pruning approach: layer removal, in which we directly delete the redundant layers in LLMs based on their BI scores. Experiments demonstrate that our method, which we call ShortGPT, significantly outperforms previous state-of-the-art (SOTA) methods in model pruning. Moreover, ShortGPT is orthogonal to quantization-like methods, enabling further reduction in parameters and computation. The ability to achieve better results through simple layer removal, as opposed to more complex pruning techniques, suggests a high degree of redundancy in the model architecture.</p>
</div>
<figure id="S0.F1" class="ltx_figure"><img src="/html/2403.03853/assets/x1.png" id="S0.F1.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="337" height="229" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 1: </span>The results on MMLU by pruning 25% parameters with different methods.</figcaption>
</figure>
<section id="S1" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">1 </span>Introduction</h2>

<div id="S1.p1" class="ltx_para">
<p id="S1.p1.1" class="ltx_p">The field of large language models (LLMs) has witnessed rapid development recently, with LLMs achieving impressive performance across various domains. Guided by the scaling laws identified in prior work <cite class="ltx_cite ltx_citemacro_citep">(Kaplan et al., <a href="#bib.bib25" title="" class="ltx_ref">2020</a>; Hoffmann et al., <a href="#bib.bib23" title="" class="ltx_ref">2022</a>)</cite>, current LLM research has tended to increase model parameters consistently to boost performance. However, the resulting massive models containing billions or even trillions of parameters pose stringent hardware requirements, hindering their practical deployment and use.</p>
</div>
<div id="S1.p2" class="ltx_para">
<p id="S1.p2.1" class="ltx_p">To alleviate the demanding hardware requirements for deploying massive trained models, many researchers have focused on model compression techniques <cite class="ltx_cite ltx_citemacro_citep">(Zhu et al., <a href="#bib.bib54" title="" class="ltx_ref">2023</a>)</cite> that reduce the inference cost. Model compression methods can be broadly categorized into two categories: quantization <cite class="ltx_cite ltx_citemacro_citep">(Liu et al., <a href="#bib.bib31" title="" class="ltx_ref">2021</a>; Gholami et al., <a href="#bib.bib16" title="" class="ltx_ref">2022</a>; Dettmers et al., <a href="#bib.bib11" title="" class="ltx_ref">2022</a>, <a href="#bib.bib12" title="" class="ltx_ref">2024</a>)</cite> and pruning <cite class="ltx_cite ltx_citemacro_citep">(LeCun et al., <a href="#bib.bib28" title="" class="ltx_ref">1989</a>; Han et al., <a href="#bib.bib19" title="" class="ltx_ref">2015</a>)</cite>. Quantization methods quantize the weights and activations to lower precision. However, the acceleration benefits of quantization are dependent on hardware support and sometimes require additional fine-tuning to maintain good performance. In contrast, pruning methods remove redundant model parameters to reduce the overall parameter count. Pruning can be directly applied to a trained model without retraining and is generally more hardware-friendly than quantization approaches.</p>
</div>
<div id="S1.p3" class="ltx_para">
<p id="S1.p3.1" class="ltx_p">While recently there has been significant progress in large language model compression methods based on model pruning, existing methods are often designed to be relatively complex. Some require the use of gradient information <cite class="ltx_cite ltx_citemacro_citep">(Zhang et al., <a href="#bib.bib52" title="" class="ltx_ref">2023</a>)</cite> or only focus on compression in width <cite class="ltx_cite ltx_citemacro_citep">(Ashkboos et al., <a href="#bib.bib1" title="" class="ltx_ref">2024</a>)</cite>, rendering them overly sophisticated for practical applications. Therefore, there is a need to explore simple and efficient model pruning approaches tailored specifically for large language models.</p>
</div>
<div id="S1.p4" class="ltx_para">
<p id="S1.p4.1" class="ltx_p">Identifying effective model pruning techniques necessitates studying model redundancy <cite class="ltx_cite ltx_citemacro_citep">(Huang et al., <a href="#bib.bib24" title="" class="ltx_ref">2021</a>; Dalvi et al., <a href="#bib.bib10" title="" class="ltx_ref">2020</a>)</cite>. Previous research on model redundancy has typically focused on relatively smaller models, such as convolutional neural networks (CNNs) or small-sized Transformers. For model pruning in LLMs, most previous works have concentrated on tensor-wise redundancy analysis, investigating the redundancy within each parameter tensor. However, a key finding in this paper is that LLMs exhibit significant redundancy at the layer level, enabling the simple removal of entire layers without substantially impacting downstream task performance. For instance, when deleting the last 10 layers (25% of the total 40 layers) from the LLaMA 2-13B model, the results on the MMLU benchmark <cite class="ltx_cite ltx_citemacro_citep">(Hendrycks et al., <a href="#bib.bib21" title="" class="ltx_ref">2020</a>)</cite> only drop from 55.0 to 52.2 (a 95% retention). Furthermore, by removing the last 22 layers (55% of the total 40 layers), resulting in a 5.6 billion parameter model, we can still obtain a score of 47.2 on MMLU without any fine-tuning, even outperforming the LLaMA 2-7B model.</p>
</div>
<div id="S1.p5" class="ltx_para">
<p id="S1.p5.1" class="ltx_p">In this paper, we propose to analyze the layer-wise redundancy through the lens of Block Influence (BI), which measures the hidden states transformations during the modeling process of LLMs. We find BI is a more relevant indicator of the layer’s importance in LLMs, and we can perform model pruning by simply deleting redundant layers with BI. This straightforward act of removing specific layers significantly outperforms previous more complex pruning methods. Our findings highlight substantial redundancy in current LLM architectures and shed light on opportunities for more efficient LLM training in the future.</p>
</div>
<div id="S1.p6" class="ltx_para">
<p id="S1.p6.1" class="ltx_p">The main contributions of our paper are summarized as follows:</p>
<ul id="S1.I1" class="ltx_itemize">
<li id="S1.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S1.I1.i1.p1" class="ltx_para">
<p id="S1.I1.i1.p1.1" class="ltx_p">We analyze the redundancy in large language models (LLMs) and find that they exhibit significant redundancy at the layer level. This insight inspires us to prune LLMs by simply removing redundant layers.</p>
</div>
</li>
<li id="S1.I1.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S1.I1.i2.p1" class="ltx_para">
<p id="S1.I1.i2.p1.1" class="ltx_p">We propose a metric called Block Influence (BI) as an effective indicator of layer importance. Through a quantitative analysis, we demonstrate that LLMs possess redundancy both in depth (layers) and width (parameters within layers).</p>
</div>
</li>
<li id="S1.I1.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S1.I1.i3.p1" class="ltx_para">
<p id="S1.I1.i3.p1.1" class="ltx_p">Based on the BI metric, we propose a simple yet effective pruning strategy by removing layers with low BI scores. Experimental results show that our method maintains 92% performance while reducing approximately 25% of the parameters and computation, outperforming previous state-of-the-art methods.</p>
</div>
</li>
<li id="S1.I1.i4" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S1.I1.i4.p1" class="ltx_para">
<p id="S1.I1.i4.p1.1" class="ltx_p">Furthermore, we demonstrate that our layer pruning approach is orthogonal to quantization methods, meaning it can be combined with quantization techniques to further reduce the deployment overhead of LLMs.</p>
</div>
</li>
</ul>
</div>
<figure id="S1.F2" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="S1.F2.sf1" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2403.03853/assets/x2.png" id="S1.F2.sf1.g1" class="ltx_graphics ltx_img_landscape" width="213" height="156" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure"><span id="S1.F2.sf1.2.1.1" class="ltx_text" style="font-size:80%;">(a)</span> </span><span id="S1.F2.sf1.3.2" class="ltx_text" style="font-size:80%;">Layer pruning order. We first calculate the BI score of each layer in LLM, and then prune the redundant layer in ascending BI score.</span></figcaption>
</figure>
</div>
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="S1.F2.sf2" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2403.03853/assets/x3.png" id="S1.F2.sf2.g1" class="ltx_graphics ltx_img_landscape" width="208" height="106" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure"><span id="S1.F2.sf2.2.1.1" class="ltx_text" style="font-size:80%;">(b)</span> </span><span id="S1.F2.sf2.3.2" class="ltx_text" style="font-size:80%;">Performance of ShortGPT with increasing pruning ratio.</span></figcaption>
</figure>
</div>
</div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 2: </span>Our Method: firstly, we sort layers by their BI score (introduced in Sec. <span class="ltx_ref ltx_missing_label ltx_ref_self">LABEL:layerimportacne</span>), and then delete layers in ascending order. All results are based on Llama2-7B-Base. </figcaption>
</figure>
</section>
<section id="S2" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">2 </span>Methodology</h2>

<div id="S2.p1" class="ltx_para">
<p id="S2.p1.1" class="ltx_p">In this section, we present the methodological framework of our layer deletion approach for LLMs, elucidating the underlying principles and techniques employed. We first quantify the layer redundancy problem present in current prominent LLMs, such as LLaMA 2 and Baichuan 2. Then, we introduce a metric called Block Influence (BI), designed to assess the transformation of hidden states by each layer during LLM inference. Building upon BI, we apply layer deletion to LLMs, thereby reducing their inference cost without compromising their predictive accuracy or linguistic capabilities.</p>
</div>
<section id="S2.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.1 </span>Layer redundancy</h3>

<div id="S2.SS1.p1" class="ltx_para">
<p id="S2.SS1.p1.6" class="ltx_p">The predominant LLMs in current are primarily based on the Transformer <cite class="ltx_cite ltx_citemacro_citep">(Vaswani et al., <a href="#bib.bib44" title="" class="ltx_ref">2017</a>)</cite>. The transformer architecture is based on an attention mechanism, typically consisting of several residual layers stacked upon each other. Transformer is a sequence to sequence mapping, which can be defined as <math id="S2.SS1.p1.1.m1.2" class="ltx_Math" alttext="y=f(X,\theta)" display="inline"><semantics id="S2.SS1.p1.1.m1.2a"><mrow id="S2.SS1.p1.1.m1.2.3" xref="S2.SS1.p1.1.m1.2.3.cmml"><mi id="S2.SS1.p1.1.m1.2.3.2" xref="S2.SS1.p1.1.m1.2.3.2.cmml">y</mi><mo id="S2.SS1.p1.1.m1.2.3.1" xref="S2.SS1.p1.1.m1.2.3.1.cmml">=</mo><mrow id="S2.SS1.p1.1.m1.2.3.3" xref="S2.SS1.p1.1.m1.2.3.3.cmml"><mi id="S2.SS1.p1.1.m1.2.3.3.2" xref="S2.SS1.p1.1.m1.2.3.3.2.cmml">f</mi><mo lspace="0em" rspace="0em" id="S2.SS1.p1.1.m1.2.3.3.1" xref="S2.SS1.p1.1.m1.2.3.3.1.cmml">​</mo><mrow id="S2.SS1.p1.1.m1.2.3.3.3.2" xref="S2.SS1.p1.1.m1.2.3.3.3.1.cmml"><mo stretchy="false" id="S2.SS1.p1.1.m1.2.3.3.3.2.1" xref="S2.SS1.p1.1.m1.2.3.3.3.1.cmml">(</mo><mi id="S2.SS1.p1.1.m1.1.1" xref="S2.SS1.p1.1.m1.1.1.cmml">X</mi><mo id="S2.SS1.p1.1.m1.2.3.3.3.2.2" xref="S2.SS1.p1.1.m1.2.3.3.3.1.cmml">,</mo><mi id="S2.SS1.p1.1.m1.2.2" xref="S2.SS1.p1.1.m1.2.2.cmml">θ</mi><mo stretchy="false" id="S2.SS1.p1.1.m1.2.3.3.3.2.3" xref="S2.SS1.p1.1.m1.2.3.3.3.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS1.p1.1.m1.2b"><apply id="S2.SS1.p1.1.m1.2.3.cmml" xref="S2.SS1.p1.1.m1.2.3"><eq id="S2.SS1.p1.1.m1.2.3.1.cmml" xref="S2.SS1.p1.1.m1.2.3.1"></eq><ci id="S2.SS1.p1.1.m1.2.3.2.cmml" xref="S2.SS1.p1.1.m1.2.3.2">𝑦</ci><apply id="S2.SS1.p1.1.m1.2.3.3.cmml" xref="S2.SS1.p1.1.m1.2.3.3"><times id="S2.SS1.p1.1.m1.2.3.3.1.cmml" xref="S2.SS1.p1.1.m1.2.3.3.1"></times><ci id="S2.SS1.p1.1.m1.2.3.3.2.cmml" xref="S2.SS1.p1.1.m1.2.3.3.2">𝑓</ci><interval closure="open" id="S2.SS1.p1.1.m1.2.3.3.3.1.cmml" xref="S2.SS1.p1.1.m1.2.3.3.3.2"><ci id="S2.SS1.p1.1.m1.1.1.cmml" xref="S2.SS1.p1.1.m1.1.1">𝑋</ci><ci id="S2.SS1.p1.1.m1.2.2.cmml" xref="S2.SS1.p1.1.m1.2.2">𝜃</ci></interval></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p1.1.m1.2c">y=f(X,\theta)</annotation></semantics></math>, where <math id="S2.SS1.p1.2.m2.1" class="ltx_Math" alttext="X\in R^{T\times n}" display="inline"><semantics id="S2.SS1.p1.2.m2.1a"><mrow id="S2.SS1.p1.2.m2.1.1" xref="S2.SS1.p1.2.m2.1.1.cmml"><mi id="S2.SS1.p1.2.m2.1.1.2" xref="S2.SS1.p1.2.m2.1.1.2.cmml">X</mi><mo id="S2.SS1.p1.2.m2.1.1.1" xref="S2.SS1.p1.2.m2.1.1.1.cmml">∈</mo><msup id="S2.SS1.p1.2.m2.1.1.3" xref="S2.SS1.p1.2.m2.1.1.3.cmml"><mi id="S2.SS1.p1.2.m2.1.1.3.2" xref="S2.SS1.p1.2.m2.1.1.3.2.cmml">R</mi><mrow id="S2.SS1.p1.2.m2.1.1.3.3" xref="S2.SS1.p1.2.m2.1.1.3.3.cmml"><mi id="S2.SS1.p1.2.m2.1.1.3.3.2" xref="S2.SS1.p1.2.m2.1.1.3.3.2.cmml">T</mi><mo lspace="0.222em" rspace="0.222em" id="S2.SS1.p1.2.m2.1.1.3.3.1" xref="S2.SS1.p1.2.m2.1.1.3.3.1.cmml">×</mo><mi id="S2.SS1.p1.2.m2.1.1.3.3.3" xref="S2.SS1.p1.2.m2.1.1.3.3.3.cmml">n</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S2.SS1.p1.2.m2.1b"><apply id="S2.SS1.p1.2.m2.1.1.cmml" xref="S2.SS1.p1.2.m2.1.1"><in id="S2.SS1.p1.2.m2.1.1.1.cmml" xref="S2.SS1.p1.2.m2.1.1.1"></in><ci id="S2.SS1.p1.2.m2.1.1.2.cmml" xref="S2.SS1.p1.2.m2.1.1.2">𝑋</ci><apply id="S2.SS1.p1.2.m2.1.1.3.cmml" xref="S2.SS1.p1.2.m2.1.1.3"><csymbol cd="ambiguous" id="S2.SS1.p1.2.m2.1.1.3.1.cmml" xref="S2.SS1.p1.2.m2.1.1.3">superscript</csymbol><ci id="S2.SS1.p1.2.m2.1.1.3.2.cmml" xref="S2.SS1.p1.2.m2.1.1.3.2">𝑅</ci><apply id="S2.SS1.p1.2.m2.1.1.3.3.cmml" xref="S2.SS1.p1.2.m2.1.1.3.3"><times id="S2.SS1.p1.2.m2.1.1.3.3.1.cmml" xref="S2.SS1.p1.2.m2.1.1.3.3.1"></times><ci id="S2.SS1.p1.2.m2.1.1.3.3.2.cmml" xref="S2.SS1.p1.2.m2.1.1.3.3.2">𝑇</ci><ci id="S2.SS1.p1.2.m2.1.1.3.3.3.cmml" xref="S2.SS1.p1.2.m2.1.1.3.3.3">𝑛</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p1.2.m2.1c">X\in R^{T\times n}</annotation></semantics></math>, <math id="S2.SS1.p1.3.m3.1" class="ltx_Math" alttext="Y\in R^{T\times n}" display="inline"><semantics id="S2.SS1.p1.3.m3.1a"><mrow id="S2.SS1.p1.3.m3.1.1" xref="S2.SS1.p1.3.m3.1.1.cmml"><mi id="S2.SS1.p1.3.m3.1.1.2" xref="S2.SS1.p1.3.m3.1.1.2.cmml">Y</mi><mo id="S2.SS1.p1.3.m3.1.1.1" xref="S2.SS1.p1.3.m3.1.1.1.cmml">∈</mo><msup id="S2.SS1.p1.3.m3.1.1.3" xref="S2.SS1.p1.3.m3.1.1.3.cmml"><mi id="S2.SS1.p1.3.m3.1.1.3.2" xref="S2.SS1.p1.3.m3.1.1.3.2.cmml">R</mi><mrow id="S2.SS1.p1.3.m3.1.1.3.3" xref="S2.SS1.p1.3.m3.1.1.3.3.cmml"><mi id="S2.SS1.p1.3.m3.1.1.3.3.2" xref="S2.SS1.p1.3.m3.1.1.3.3.2.cmml">T</mi><mo lspace="0.222em" rspace="0.222em" id="S2.SS1.p1.3.m3.1.1.3.3.1" xref="S2.SS1.p1.3.m3.1.1.3.3.1.cmml">×</mo><mi id="S2.SS1.p1.3.m3.1.1.3.3.3" xref="S2.SS1.p1.3.m3.1.1.3.3.3.cmml">n</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S2.SS1.p1.3.m3.1b"><apply id="S2.SS1.p1.3.m3.1.1.cmml" xref="S2.SS1.p1.3.m3.1.1"><in id="S2.SS1.p1.3.m3.1.1.1.cmml" xref="S2.SS1.p1.3.m3.1.1.1"></in><ci id="S2.SS1.p1.3.m3.1.1.2.cmml" xref="S2.SS1.p1.3.m3.1.1.2">𝑌</ci><apply id="S2.SS1.p1.3.m3.1.1.3.cmml" xref="S2.SS1.p1.3.m3.1.1.3"><csymbol cd="ambiguous" id="S2.SS1.p1.3.m3.1.1.3.1.cmml" xref="S2.SS1.p1.3.m3.1.1.3">superscript</csymbol><ci id="S2.SS1.p1.3.m3.1.1.3.2.cmml" xref="S2.SS1.p1.3.m3.1.1.3.2">𝑅</ci><apply id="S2.SS1.p1.3.m3.1.1.3.3.cmml" xref="S2.SS1.p1.3.m3.1.1.3.3"><times id="S2.SS1.p1.3.m3.1.1.3.3.1.cmml" xref="S2.SS1.p1.3.m3.1.1.3.3.1"></times><ci id="S2.SS1.p1.3.m3.1.1.3.3.2.cmml" xref="S2.SS1.p1.3.m3.1.1.3.3.2">𝑇</ci><ci id="S2.SS1.p1.3.m3.1.1.3.3.3.cmml" xref="S2.SS1.p1.3.m3.1.1.3.3.3">𝑛</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p1.3.m3.1c">Y\in R^{T\times n}</annotation></semantics></math>, <math id="S2.SS1.p1.4.m4.1" class="ltx_Math" alttext="T" display="inline"><semantics id="S2.SS1.p1.4.m4.1a"><mi id="S2.SS1.p1.4.m4.1.1" xref="S2.SS1.p1.4.m4.1.1.cmml">T</mi><annotation-xml encoding="MathML-Content" id="S2.SS1.p1.4.m4.1b"><ci id="S2.SS1.p1.4.m4.1.1.cmml" xref="S2.SS1.p1.4.m4.1.1">𝑇</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p1.4.m4.1c">T</annotation></semantics></math> is the length of the sequence, <math id="S2.SS1.p1.5.m5.1" class="ltx_Math" alttext="n" display="inline"><semantics id="S2.SS1.p1.5.m5.1a"><mi id="S2.SS1.p1.5.m5.1.1" xref="S2.SS1.p1.5.m5.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="S2.SS1.p1.5.m5.1b"><ci id="S2.SS1.p1.5.m5.1.1.cmml" xref="S2.SS1.p1.5.m5.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p1.5.m5.1c">n</annotation></semantics></math> is the vocabulary size, <math id="S2.SS1.p1.6.m6.1" class="ltx_Math" alttext="\theta" display="inline"><semantics id="S2.SS1.p1.6.m6.1a"><mi id="S2.SS1.p1.6.m6.1.1" xref="S2.SS1.p1.6.m6.1.1.cmml">θ</mi><annotation-xml encoding="MathML-Content" id="S2.SS1.p1.6.m6.1b"><ci id="S2.SS1.p1.6.m6.1.1.cmml" xref="S2.SS1.p1.6.m6.1.1">𝜃</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p1.6.m6.1c">\theta</annotation></semantics></math> is the learnable parameters. The formal expression of an L-layer transformer is as follows:</p>
<table id="A4.EGx1" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="S2.Ex1"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="S2.Ex1.m1.1" class="ltx_Math" alttext="\displaystyle\textbf{\footnotesize{X}}_{0}" display="inline"><semantics id="S2.Ex1.m1.1a"><msub id="S2.Ex1.m1.1.1" xref="S2.Ex1.m1.1.1.cmml"><mtext class="ltx_mathvariant_bold" mathsize="80%" id="S2.Ex1.m1.1.1.2" xref="S2.Ex1.m1.1.1.2a.cmml">X</mtext><mn id="S2.Ex1.m1.1.1.3" xref="S2.Ex1.m1.1.1.3.cmml">0</mn></msub><annotation-xml encoding="MathML-Content" id="S2.Ex1.m1.1b"><apply id="S2.Ex1.m1.1.1.cmml" xref="S2.Ex1.m1.1.1"><csymbol cd="ambiguous" id="S2.Ex1.m1.1.1.1.cmml" xref="S2.Ex1.m1.1.1">subscript</csymbol><ci id="S2.Ex1.m1.1.1.2a.cmml" xref="S2.Ex1.m1.1.1.2"><mtext class="ltx_mathvariant_bold" mathsize="80%" id="S2.Ex1.m1.1.1.2.cmml" xref="S2.Ex1.m1.1.1.2">X</mtext></ci><cn type="integer" id="S2.Ex1.m1.1.1.3.cmml" xref="S2.Ex1.m1.1.1.3">0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.Ex1.m1.1c">\displaystyle\textbf{\footnotesize{X}}_{0}</annotation></semantics></math></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="S2.Ex1.m2.1" class="ltx_Math" alttext="\displaystyle=\textbf{\footnotesize{X}}\textbf{W}_{\text{emb}}" display="inline"><semantics id="S2.Ex1.m2.1a"><mrow id="S2.Ex1.m2.1.1" xref="S2.Ex1.m2.1.1.cmml"><mi id="S2.Ex1.m2.1.1.2" xref="S2.Ex1.m2.1.1.2.cmml"></mi><mo id="S2.Ex1.m2.1.1.1" xref="S2.Ex1.m2.1.1.1.cmml">=</mo><msub id="S2.Ex1.m2.1.1.3" xref="S2.Ex1.m2.1.1.3.cmml"><mrow id="S2.Ex1.m2.1.1.3.2" xref="S2.Ex1.m2.1.1.3.2c.cmml"><mtext class="ltx_mathvariant_bold" mathsize="80%" id="S2.Ex1.m2.1.1.3.2a" xref="S2.Ex1.m2.1.1.3.2c.cmml">X</mtext><mtext class="ltx_mathvariant_bold" id="S2.Ex1.m2.1.1.3.2b" xref="S2.Ex1.m2.1.1.3.2c.cmml">W</mtext></mrow><mtext id="S2.Ex1.m2.1.1.3.3" xref="S2.Ex1.m2.1.1.3.3a.cmml">emb</mtext></msub></mrow><annotation-xml encoding="MathML-Content" id="S2.Ex1.m2.1b"><apply id="S2.Ex1.m2.1.1.cmml" xref="S2.Ex1.m2.1.1"><eq id="S2.Ex1.m2.1.1.1.cmml" xref="S2.Ex1.m2.1.1.1"></eq><csymbol cd="latexml" id="S2.Ex1.m2.1.1.2.cmml" xref="S2.Ex1.m2.1.1.2">absent</csymbol><apply id="S2.Ex1.m2.1.1.3.cmml" xref="S2.Ex1.m2.1.1.3"><csymbol cd="ambiguous" id="S2.Ex1.m2.1.1.3.1.cmml" xref="S2.Ex1.m2.1.1.3">subscript</csymbol><ci id="S2.Ex1.m2.1.1.3.2c.cmml" xref="S2.Ex1.m2.1.1.3.2"><mrow id="S2.Ex1.m2.1.1.3.2.cmml" xref="S2.Ex1.m2.1.1.3.2"><mtext class="ltx_mathvariant_bold" mathsize="80%" id="S2.Ex1.m2.1.1.3.2a.cmml" xref="S2.Ex1.m2.1.1.3.2">X</mtext><mtext class="ltx_mathvariant_bold" id="S2.Ex1.m2.1.1.3.2b.cmml" xref="S2.Ex1.m2.1.1.3.2">W</mtext></mrow></ci><ci id="S2.Ex1.m2.1.1.3.3a.cmml" xref="S2.Ex1.m2.1.1.3.3"><mtext mathsize="70%" id="S2.Ex1.m2.1.1.3.3.cmml" xref="S2.Ex1.m2.1.1.3.3">emb</mtext></ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.Ex1.m2.1c">\displaystyle=\textbf{\footnotesize{X}}\textbf{W}_{\text{emb}}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
<tbody id="S2.Ex2"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="S2.Ex2.m1.1" class="ltx_Math" alttext="\displaystyle\textbf{\footnotesize{A}}_{i+1}" display="inline"><semantics id="S2.Ex2.m1.1a"><msub id="S2.Ex2.m1.1.1" xref="S2.Ex2.m1.1.1.cmml"><mtext class="ltx_mathvariant_bold" mathsize="80%" id="S2.Ex2.m1.1.1.2" xref="S2.Ex2.m1.1.1.2a.cmml">A</mtext><mrow id="S2.Ex2.m1.1.1.3" xref="S2.Ex2.m1.1.1.3.cmml"><mi id="S2.Ex2.m1.1.1.3.2" xref="S2.Ex2.m1.1.1.3.2.cmml">i</mi><mo id="S2.Ex2.m1.1.1.3.1" xref="S2.Ex2.m1.1.1.3.1.cmml">+</mo><mn id="S2.Ex2.m1.1.1.3.3" xref="S2.Ex2.m1.1.1.3.3.cmml">1</mn></mrow></msub><annotation-xml encoding="MathML-Content" id="S2.Ex2.m1.1b"><apply id="S2.Ex2.m1.1.1.cmml" xref="S2.Ex2.m1.1.1"><csymbol cd="ambiguous" id="S2.Ex2.m1.1.1.1.cmml" xref="S2.Ex2.m1.1.1">subscript</csymbol><ci id="S2.Ex2.m1.1.1.2a.cmml" xref="S2.Ex2.m1.1.1.2"><mtext class="ltx_mathvariant_bold" mathsize="80%" id="S2.Ex2.m1.1.1.2.cmml" xref="S2.Ex2.m1.1.1.2">A</mtext></ci><apply id="S2.Ex2.m1.1.1.3.cmml" xref="S2.Ex2.m1.1.1.3"><plus id="S2.Ex2.m1.1.1.3.1.cmml" xref="S2.Ex2.m1.1.1.3.1"></plus><ci id="S2.Ex2.m1.1.1.3.2.cmml" xref="S2.Ex2.m1.1.1.3.2">𝑖</ci><cn type="integer" id="S2.Ex2.m1.1.1.3.3.cmml" xref="S2.Ex2.m1.1.1.3.3">1</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.Ex2.m1.1c">\displaystyle\textbf{\footnotesize{A}}_{i+1}</annotation></semantics></math></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="S2.Ex2.m2.1" class="ltx_Math" alttext="\displaystyle=\text{ATTN}(\text{LN}(\textbf{\footnotesize{X}}_{i}))+\textbf{\footnotesize{X}}_{i}" display="inline"><semantics id="S2.Ex2.m2.1a"><mrow id="S2.Ex2.m2.1.1" xref="S2.Ex2.m2.1.1.cmml"><mi id="S2.Ex2.m2.1.1.3" xref="S2.Ex2.m2.1.1.3.cmml"></mi><mo id="S2.Ex2.m2.1.1.2" xref="S2.Ex2.m2.1.1.2.cmml">=</mo><mrow id="S2.Ex2.m2.1.1.1" xref="S2.Ex2.m2.1.1.1.cmml"><mrow id="S2.Ex2.m2.1.1.1.1" xref="S2.Ex2.m2.1.1.1.1.cmml"><mtext id="S2.Ex2.m2.1.1.1.1.3" xref="S2.Ex2.m2.1.1.1.1.3a.cmml">ATTN</mtext><mo lspace="0em" rspace="0em" id="S2.Ex2.m2.1.1.1.1.2" xref="S2.Ex2.m2.1.1.1.1.2.cmml">​</mo><mrow id="S2.Ex2.m2.1.1.1.1.1.1" xref="S2.Ex2.m2.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S2.Ex2.m2.1.1.1.1.1.1.2" xref="S2.Ex2.m2.1.1.1.1.1.1.1.cmml">(</mo><mrow id="S2.Ex2.m2.1.1.1.1.1.1.1" xref="S2.Ex2.m2.1.1.1.1.1.1.1.cmml"><mtext id="S2.Ex2.m2.1.1.1.1.1.1.1.3" xref="S2.Ex2.m2.1.1.1.1.1.1.1.3a.cmml">LN</mtext><mo lspace="0em" rspace="0em" id="S2.Ex2.m2.1.1.1.1.1.1.1.2" xref="S2.Ex2.m2.1.1.1.1.1.1.1.2.cmml">​</mo><mrow id="S2.Ex2.m2.1.1.1.1.1.1.1.1.1" xref="S2.Ex2.m2.1.1.1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S2.Ex2.m2.1.1.1.1.1.1.1.1.1.2" xref="S2.Ex2.m2.1.1.1.1.1.1.1.1.1.1.cmml">(</mo><msub id="S2.Ex2.m2.1.1.1.1.1.1.1.1.1.1" xref="S2.Ex2.m2.1.1.1.1.1.1.1.1.1.1.cmml"><mtext class="ltx_mathvariant_bold" mathsize="80%" id="S2.Ex2.m2.1.1.1.1.1.1.1.1.1.1.2" xref="S2.Ex2.m2.1.1.1.1.1.1.1.1.1.1.2a.cmml">X</mtext><mi id="S2.Ex2.m2.1.1.1.1.1.1.1.1.1.1.3" xref="S2.Ex2.m2.1.1.1.1.1.1.1.1.1.1.3.cmml">i</mi></msub><mo stretchy="false" id="S2.Ex2.m2.1.1.1.1.1.1.1.1.1.3" xref="S2.Ex2.m2.1.1.1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo stretchy="false" id="S2.Ex2.m2.1.1.1.1.1.1.3" xref="S2.Ex2.m2.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="S2.Ex2.m2.1.1.1.2" xref="S2.Ex2.m2.1.1.1.2.cmml">+</mo><msub id="S2.Ex2.m2.1.1.1.3" xref="S2.Ex2.m2.1.1.1.3.cmml"><mtext class="ltx_mathvariant_bold" mathsize="80%" id="S2.Ex2.m2.1.1.1.3.2" xref="S2.Ex2.m2.1.1.1.3.2a.cmml">X</mtext><mi id="S2.Ex2.m2.1.1.1.3.3" xref="S2.Ex2.m2.1.1.1.3.3.cmml">i</mi></msub></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.Ex2.m2.1b"><apply id="S2.Ex2.m2.1.1.cmml" xref="S2.Ex2.m2.1.1"><eq id="S2.Ex2.m2.1.1.2.cmml" xref="S2.Ex2.m2.1.1.2"></eq><csymbol cd="latexml" id="S2.Ex2.m2.1.1.3.cmml" xref="S2.Ex2.m2.1.1.3">absent</csymbol><apply id="S2.Ex2.m2.1.1.1.cmml" xref="S2.Ex2.m2.1.1.1"><plus id="S2.Ex2.m2.1.1.1.2.cmml" xref="S2.Ex2.m2.1.1.1.2"></plus><apply id="S2.Ex2.m2.1.1.1.1.cmml" xref="S2.Ex2.m2.1.1.1.1"><times id="S2.Ex2.m2.1.1.1.1.2.cmml" xref="S2.Ex2.m2.1.1.1.1.2"></times><ci id="S2.Ex2.m2.1.1.1.1.3a.cmml" xref="S2.Ex2.m2.1.1.1.1.3"><mtext id="S2.Ex2.m2.1.1.1.1.3.cmml" xref="S2.Ex2.m2.1.1.1.1.3">ATTN</mtext></ci><apply id="S2.Ex2.m2.1.1.1.1.1.1.1.cmml" xref="S2.Ex2.m2.1.1.1.1.1.1"><times id="S2.Ex2.m2.1.1.1.1.1.1.1.2.cmml" xref="S2.Ex2.m2.1.1.1.1.1.1.1.2"></times><ci id="S2.Ex2.m2.1.1.1.1.1.1.1.3a.cmml" xref="S2.Ex2.m2.1.1.1.1.1.1.1.3"><mtext id="S2.Ex2.m2.1.1.1.1.1.1.1.3.cmml" xref="S2.Ex2.m2.1.1.1.1.1.1.1.3">LN</mtext></ci><apply id="S2.Ex2.m2.1.1.1.1.1.1.1.1.1.1.cmml" xref="S2.Ex2.m2.1.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.Ex2.m2.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S2.Ex2.m2.1.1.1.1.1.1.1.1.1">subscript</csymbol><ci id="S2.Ex2.m2.1.1.1.1.1.1.1.1.1.1.2a.cmml" xref="S2.Ex2.m2.1.1.1.1.1.1.1.1.1.1.2"><mtext class="ltx_mathvariant_bold" mathsize="80%" id="S2.Ex2.m2.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="S2.Ex2.m2.1.1.1.1.1.1.1.1.1.1.2">X</mtext></ci><ci id="S2.Ex2.m2.1.1.1.1.1.1.1.1.1.1.3.cmml" xref="S2.Ex2.m2.1.1.1.1.1.1.1.1.1.1.3">𝑖</ci></apply></apply></apply><apply id="S2.Ex2.m2.1.1.1.3.cmml" xref="S2.Ex2.m2.1.1.1.3"><csymbol cd="ambiguous" id="S2.Ex2.m2.1.1.1.3.1.cmml" xref="S2.Ex2.m2.1.1.1.3">subscript</csymbol><ci id="S2.Ex2.m2.1.1.1.3.2a.cmml" xref="S2.Ex2.m2.1.1.1.3.2"><mtext class="ltx_mathvariant_bold" mathsize="80%" id="S2.Ex2.m2.1.1.1.3.2.cmml" xref="S2.Ex2.m2.1.1.1.3.2">X</mtext></ci><ci id="S2.Ex2.m2.1.1.1.3.3.cmml" xref="S2.Ex2.m2.1.1.1.3.3">𝑖</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.Ex2.m2.1c">\displaystyle=\text{ATTN}(\text{LN}(\textbf{\footnotesize{X}}_{i}))+\textbf{\footnotesize{X}}_{i}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
<tbody id="S2.Ex3"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="S2.Ex3.m1.1" class="ltx_Math" alttext="\displaystyle\textbf{\footnotesize{X}}_{i+1}" display="inline"><semantics id="S2.Ex3.m1.1a"><msub id="S2.Ex3.m1.1.1" xref="S2.Ex3.m1.1.1.cmml"><mtext class="ltx_mathvariant_bold" mathsize="80%" id="S2.Ex3.m1.1.1.2" xref="S2.Ex3.m1.1.1.2a.cmml">X</mtext><mrow id="S2.Ex3.m1.1.1.3" xref="S2.Ex3.m1.1.1.3.cmml"><mi id="S2.Ex3.m1.1.1.3.2" xref="S2.Ex3.m1.1.1.3.2.cmml">i</mi><mo id="S2.Ex3.m1.1.1.3.1" xref="S2.Ex3.m1.1.1.3.1.cmml">+</mo><mn id="S2.Ex3.m1.1.1.3.3" xref="S2.Ex3.m1.1.1.3.3.cmml">1</mn></mrow></msub><annotation-xml encoding="MathML-Content" id="S2.Ex3.m1.1b"><apply id="S2.Ex3.m1.1.1.cmml" xref="S2.Ex3.m1.1.1"><csymbol cd="ambiguous" id="S2.Ex3.m1.1.1.1.cmml" xref="S2.Ex3.m1.1.1">subscript</csymbol><ci id="S2.Ex3.m1.1.1.2a.cmml" xref="S2.Ex3.m1.1.1.2"><mtext class="ltx_mathvariant_bold" mathsize="80%" id="S2.Ex3.m1.1.1.2.cmml" xref="S2.Ex3.m1.1.1.2">X</mtext></ci><apply id="S2.Ex3.m1.1.1.3.cmml" xref="S2.Ex3.m1.1.1.3"><plus id="S2.Ex3.m1.1.1.3.1.cmml" xref="S2.Ex3.m1.1.1.3.1"></plus><ci id="S2.Ex3.m1.1.1.3.2.cmml" xref="S2.Ex3.m1.1.1.3.2">𝑖</ci><cn type="integer" id="S2.Ex3.m1.1.1.3.3.cmml" xref="S2.Ex3.m1.1.1.3.3">1</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.Ex3.m1.1c">\displaystyle\textbf{\footnotesize{X}}_{i+1}</annotation></semantics></math></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="S2.Ex3.m2.1" class="ltx_Math" alttext="\displaystyle=\text{FFN}(\text{LN}(\textbf{\footnotesize{A}}_{i+1}))+\textbf{\footnotesize{A}}_{i+1}" display="inline"><semantics id="S2.Ex3.m2.1a"><mrow id="S2.Ex3.m2.1.1" xref="S2.Ex3.m2.1.1.cmml"><mi id="S2.Ex3.m2.1.1.3" xref="S2.Ex3.m2.1.1.3.cmml"></mi><mo id="S2.Ex3.m2.1.1.2" xref="S2.Ex3.m2.1.1.2.cmml">=</mo><mrow id="S2.Ex3.m2.1.1.1" xref="S2.Ex3.m2.1.1.1.cmml"><mrow id="S2.Ex3.m2.1.1.1.1" xref="S2.Ex3.m2.1.1.1.1.cmml"><mtext id="S2.Ex3.m2.1.1.1.1.3" xref="S2.Ex3.m2.1.1.1.1.3a.cmml">FFN</mtext><mo lspace="0em" rspace="0em" id="S2.Ex3.m2.1.1.1.1.2" xref="S2.Ex3.m2.1.1.1.1.2.cmml">​</mo><mrow id="S2.Ex3.m2.1.1.1.1.1.1" xref="S2.Ex3.m2.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S2.Ex3.m2.1.1.1.1.1.1.2" xref="S2.Ex3.m2.1.1.1.1.1.1.1.cmml">(</mo><mrow id="S2.Ex3.m2.1.1.1.1.1.1.1" xref="S2.Ex3.m2.1.1.1.1.1.1.1.cmml"><mtext id="S2.Ex3.m2.1.1.1.1.1.1.1.3" xref="S2.Ex3.m2.1.1.1.1.1.1.1.3a.cmml">LN</mtext><mo lspace="0em" rspace="0em" id="S2.Ex3.m2.1.1.1.1.1.1.1.2" xref="S2.Ex3.m2.1.1.1.1.1.1.1.2.cmml">​</mo><mrow id="S2.Ex3.m2.1.1.1.1.1.1.1.1.1" xref="S2.Ex3.m2.1.1.1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S2.Ex3.m2.1.1.1.1.1.1.1.1.1.2" xref="S2.Ex3.m2.1.1.1.1.1.1.1.1.1.1.cmml">(</mo><msub id="S2.Ex3.m2.1.1.1.1.1.1.1.1.1.1" xref="S2.Ex3.m2.1.1.1.1.1.1.1.1.1.1.cmml"><mtext class="ltx_mathvariant_bold" mathsize="80%" id="S2.Ex3.m2.1.1.1.1.1.1.1.1.1.1.2" xref="S2.Ex3.m2.1.1.1.1.1.1.1.1.1.1.2a.cmml">A</mtext><mrow id="S2.Ex3.m2.1.1.1.1.1.1.1.1.1.1.3" xref="S2.Ex3.m2.1.1.1.1.1.1.1.1.1.1.3.cmml"><mi id="S2.Ex3.m2.1.1.1.1.1.1.1.1.1.1.3.2" xref="S2.Ex3.m2.1.1.1.1.1.1.1.1.1.1.3.2.cmml">i</mi><mo id="S2.Ex3.m2.1.1.1.1.1.1.1.1.1.1.3.1" xref="S2.Ex3.m2.1.1.1.1.1.1.1.1.1.1.3.1.cmml">+</mo><mn id="S2.Ex3.m2.1.1.1.1.1.1.1.1.1.1.3.3" xref="S2.Ex3.m2.1.1.1.1.1.1.1.1.1.1.3.3.cmml">1</mn></mrow></msub><mo stretchy="false" id="S2.Ex3.m2.1.1.1.1.1.1.1.1.1.3" xref="S2.Ex3.m2.1.1.1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo stretchy="false" id="S2.Ex3.m2.1.1.1.1.1.1.3" xref="S2.Ex3.m2.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="S2.Ex3.m2.1.1.1.2" xref="S2.Ex3.m2.1.1.1.2.cmml">+</mo><msub id="S2.Ex3.m2.1.1.1.3" xref="S2.Ex3.m2.1.1.1.3.cmml"><mtext class="ltx_mathvariant_bold" mathsize="80%" id="S2.Ex3.m2.1.1.1.3.2" xref="S2.Ex3.m2.1.1.1.3.2a.cmml">A</mtext><mrow id="S2.Ex3.m2.1.1.1.3.3" xref="S2.Ex3.m2.1.1.1.3.3.cmml"><mi id="S2.Ex3.m2.1.1.1.3.3.2" xref="S2.Ex3.m2.1.1.1.3.3.2.cmml">i</mi><mo id="S2.Ex3.m2.1.1.1.3.3.1" xref="S2.Ex3.m2.1.1.1.3.3.1.cmml">+</mo><mn id="S2.Ex3.m2.1.1.1.3.3.3" xref="S2.Ex3.m2.1.1.1.3.3.3.cmml">1</mn></mrow></msub></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.Ex3.m2.1b"><apply id="S2.Ex3.m2.1.1.cmml" xref="S2.Ex3.m2.1.1"><eq id="S2.Ex3.m2.1.1.2.cmml" xref="S2.Ex3.m2.1.1.2"></eq><csymbol cd="latexml" id="S2.Ex3.m2.1.1.3.cmml" xref="S2.Ex3.m2.1.1.3">absent</csymbol><apply id="S2.Ex3.m2.1.1.1.cmml" xref="S2.Ex3.m2.1.1.1"><plus id="S2.Ex3.m2.1.1.1.2.cmml" xref="S2.Ex3.m2.1.1.1.2"></plus><apply id="S2.Ex3.m2.1.1.1.1.cmml" xref="S2.Ex3.m2.1.1.1.1"><times id="S2.Ex3.m2.1.1.1.1.2.cmml" xref="S2.Ex3.m2.1.1.1.1.2"></times><ci id="S2.Ex3.m2.1.1.1.1.3a.cmml" xref="S2.Ex3.m2.1.1.1.1.3"><mtext id="S2.Ex3.m2.1.1.1.1.3.cmml" xref="S2.Ex3.m2.1.1.1.1.3">FFN</mtext></ci><apply id="S2.Ex3.m2.1.1.1.1.1.1.1.cmml" xref="S2.Ex3.m2.1.1.1.1.1.1"><times id="S2.Ex3.m2.1.1.1.1.1.1.1.2.cmml" xref="S2.Ex3.m2.1.1.1.1.1.1.1.2"></times><ci id="S2.Ex3.m2.1.1.1.1.1.1.1.3a.cmml" xref="S2.Ex3.m2.1.1.1.1.1.1.1.3"><mtext id="S2.Ex3.m2.1.1.1.1.1.1.1.3.cmml" xref="S2.Ex3.m2.1.1.1.1.1.1.1.3">LN</mtext></ci><apply id="S2.Ex3.m2.1.1.1.1.1.1.1.1.1.1.cmml" xref="S2.Ex3.m2.1.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.Ex3.m2.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S2.Ex3.m2.1.1.1.1.1.1.1.1.1">subscript</csymbol><ci id="S2.Ex3.m2.1.1.1.1.1.1.1.1.1.1.2a.cmml" xref="S2.Ex3.m2.1.1.1.1.1.1.1.1.1.1.2"><mtext class="ltx_mathvariant_bold" mathsize="80%" id="S2.Ex3.m2.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="S2.Ex3.m2.1.1.1.1.1.1.1.1.1.1.2">A</mtext></ci><apply id="S2.Ex3.m2.1.1.1.1.1.1.1.1.1.1.3.cmml" xref="S2.Ex3.m2.1.1.1.1.1.1.1.1.1.1.3"><plus id="S2.Ex3.m2.1.1.1.1.1.1.1.1.1.1.3.1.cmml" xref="S2.Ex3.m2.1.1.1.1.1.1.1.1.1.1.3.1"></plus><ci id="S2.Ex3.m2.1.1.1.1.1.1.1.1.1.1.3.2.cmml" xref="S2.Ex3.m2.1.1.1.1.1.1.1.1.1.1.3.2">𝑖</ci><cn type="integer" id="S2.Ex3.m2.1.1.1.1.1.1.1.1.1.1.3.3.cmml" xref="S2.Ex3.m2.1.1.1.1.1.1.1.1.1.1.3.3">1</cn></apply></apply></apply></apply><apply id="S2.Ex3.m2.1.1.1.3.cmml" xref="S2.Ex3.m2.1.1.1.3"><csymbol cd="ambiguous" id="S2.Ex3.m2.1.1.1.3.1.cmml" xref="S2.Ex3.m2.1.1.1.3">subscript</csymbol><ci id="S2.Ex3.m2.1.1.1.3.2a.cmml" xref="S2.Ex3.m2.1.1.1.3.2"><mtext class="ltx_mathvariant_bold" mathsize="80%" id="S2.Ex3.m2.1.1.1.3.2.cmml" xref="S2.Ex3.m2.1.1.1.3.2">A</mtext></ci><apply id="S2.Ex3.m2.1.1.1.3.3.cmml" xref="S2.Ex3.m2.1.1.1.3.3"><plus id="S2.Ex3.m2.1.1.1.3.3.1.cmml" xref="S2.Ex3.m2.1.1.1.3.3.1"></plus><ci id="S2.Ex3.m2.1.1.1.3.3.2.cmml" xref="S2.Ex3.m2.1.1.1.3.3.2">𝑖</ci><cn type="integer" id="S2.Ex3.m2.1.1.1.3.3.3.cmml" xref="S2.Ex3.m2.1.1.1.3.3.3">1</cn></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.Ex3.m2.1c">\displaystyle=\text{FFN}(\text{LN}(\textbf{\footnotesize{A}}_{i+1}))+\textbf{\footnotesize{A}}_{i+1}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
<tbody id="S2.E1"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><span id="S2.E1.2.1.1.1" class="ltx_text ltx_markedasmath ltx_font_bold" style="font-size:80%;">Y</span></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="S2.E1.m2.1" class="ltx_Math" alttext="\displaystyle=\textbf{\footnotesize{X}}_{L}\textbf{W}_{\text{head}}" display="inline"><semantics id="S2.E1.m2.1a"><mrow id="S2.E1.m2.1.1" xref="S2.E1.m2.1.1.cmml"><mi id="S2.E1.m2.1.1.2" xref="S2.E1.m2.1.1.2.cmml"></mi><mo id="S2.E1.m2.1.1.1" xref="S2.E1.m2.1.1.1.cmml">=</mo><mrow id="S2.E1.m2.1.1.3" xref="S2.E1.m2.1.1.3.cmml"><msub id="S2.E1.m2.1.1.3.2" xref="S2.E1.m2.1.1.3.2.cmml"><mtext class="ltx_mathvariant_bold" mathsize="80%" id="S2.E1.m2.1.1.3.2.2" xref="S2.E1.m2.1.1.3.2.2a.cmml">X</mtext><mi id="S2.E1.m2.1.1.3.2.3" xref="S2.E1.m2.1.1.3.2.3.cmml">L</mi></msub><mo lspace="0em" rspace="0em" id="S2.E1.m2.1.1.3.1" xref="S2.E1.m2.1.1.3.1.cmml">​</mo><msub id="S2.E1.m2.1.1.3.3" xref="S2.E1.m2.1.1.3.3.cmml"><mtext class="ltx_mathvariant_bold" id="S2.E1.m2.1.1.3.3.2" xref="S2.E1.m2.1.1.3.3.2a.cmml">W</mtext><mtext id="S2.E1.m2.1.1.3.3.3" xref="S2.E1.m2.1.1.3.3.3a.cmml">head</mtext></msub></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.E1.m2.1b"><apply id="S2.E1.m2.1.1.cmml" xref="S2.E1.m2.1.1"><eq id="S2.E1.m2.1.1.1.cmml" xref="S2.E1.m2.1.1.1"></eq><csymbol cd="latexml" id="S2.E1.m2.1.1.2.cmml" xref="S2.E1.m2.1.1.2">absent</csymbol><apply id="S2.E1.m2.1.1.3.cmml" xref="S2.E1.m2.1.1.3"><times id="S2.E1.m2.1.1.3.1.cmml" xref="S2.E1.m2.1.1.3.1"></times><apply id="S2.E1.m2.1.1.3.2.cmml" xref="S2.E1.m2.1.1.3.2"><csymbol cd="ambiguous" id="S2.E1.m2.1.1.3.2.1.cmml" xref="S2.E1.m2.1.1.3.2">subscript</csymbol><ci id="S2.E1.m2.1.1.3.2.2a.cmml" xref="S2.E1.m2.1.1.3.2.2"><mtext class="ltx_mathvariant_bold" mathsize="80%" id="S2.E1.m2.1.1.3.2.2.cmml" xref="S2.E1.m2.1.1.3.2.2">X</mtext></ci><ci id="S2.E1.m2.1.1.3.2.3.cmml" xref="S2.E1.m2.1.1.3.2.3">𝐿</ci></apply><apply id="S2.E1.m2.1.1.3.3.cmml" xref="S2.E1.m2.1.1.3.3"><csymbol cd="ambiguous" id="S2.E1.m2.1.1.3.3.1.cmml" xref="S2.E1.m2.1.1.3.3">subscript</csymbol><ci id="S2.E1.m2.1.1.3.3.2a.cmml" xref="S2.E1.m2.1.1.3.3.2"><mtext class="ltx_mathvariant_bold" id="S2.E1.m2.1.1.3.3.2.cmml" xref="S2.E1.m2.1.1.3.3.2">W</mtext></ci><ci id="S2.E1.m2.1.1.3.3.3a.cmml" xref="S2.E1.m2.1.1.3.3.3"><mtext mathsize="70%" id="S2.E1.m2.1.1.3.3.3.cmml" xref="S2.E1.m2.1.1.3.3.3">head</mtext></ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E1.m2.1c">\displaystyle=\textbf{\footnotesize{X}}_{L}\textbf{W}_{\text{head}}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(1)</span></td>
</tr></tbody>
</table>
</div>
<div id="S2.SS1.p2" class="ltx_para">
<p id="S2.SS1.p2.8" class="ltx_p">where <math id="S2.SS1.p2.1.m1.1" class="ltx_Math" alttext="\textbf{W}_{\text{emb}}\in R^{n\times d}" display="inline"><semantics id="S2.SS1.p2.1.m1.1a"><mrow id="S2.SS1.p2.1.m1.1.1" xref="S2.SS1.p2.1.m1.1.1.cmml"><msub id="S2.SS1.p2.1.m1.1.1.2" xref="S2.SS1.p2.1.m1.1.1.2.cmml"><mtext class="ltx_mathvariant_bold" id="S2.SS1.p2.1.m1.1.1.2.2" xref="S2.SS1.p2.1.m1.1.1.2.2a.cmml">W</mtext><mtext id="S2.SS1.p2.1.m1.1.1.2.3" xref="S2.SS1.p2.1.m1.1.1.2.3a.cmml">emb</mtext></msub><mo id="S2.SS1.p2.1.m1.1.1.1" xref="S2.SS1.p2.1.m1.1.1.1.cmml">∈</mo><msup id="S2.SS1.p2.1.m1.1.1.3" xref="S2.SS1.p2.1.m1.1.1.3.cmml"><mi id="S2.SS1.p2.1.m1.1.1.3.2" xref="S2.SS1.p2.1.m1.1.1.3.2.cmml">R</mi><mrow id="S2.SS1.p2.1.m1.1.1.3.3" xref="S2.SS1.p2.1.m1.1.1.3.3.cmml"><mi id="S2.SS1.p2.1.m1.1.1.3.3.2" xref="S2.SS1.p2.1.m1.1.1.3.3.2.cmml">n</mi><mo lspace="0.222em" rspace="0.222em" id="S2.SS1.p2.1.m1.1.1.3.3.1" xref="S2.SS1.p2.1.m1.1.1.3.3.1.cmml">×</mo><mi id="S2.SS1.p2.1.m1.1.1.3.3.3" xref="S2.SS1.p2.1.m1.1.1.3.3.3.cmml">d</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S2.SS1.p2.1.m1.1b"><apply id="S2.SS1.p2.1.m1.1.1.cmml" xref="S2.SS1.p2.1.m1.1.1"><in id="S2.SS1.p2.1.m1.1.1.1.cmml" xref="S2.SS1.p2.1.m1.1.1.1"></in><apply id="S2.SS1.p2.1.m1.1.1.2.cmml" xref="S2.SS1.p2.1.m1.1.1.2"><csymbol cd="ambiguous" id="S2.SS1.p2.1.m1.1.1.2.1.cmml" xref="S2.SS1.p2.1.m1.1.1.2">subscript</csymbol><ci id="S2.SS1.p2.1.m1.1.1.2.2a.cmml" xref="S2.SS1.p2.1.m1.1.1.2.2"><mtext class="ltx_mathvariant_bold" id="S2.SS1.p2.1.m1.1.1.2.2.cmml" xref="S2.SS1.p2.1.m1.1.1.2.2">W</mtext></ci><ci id="S2.SS1.p2.1.m1.1.1.2.3a.cmml" xref="S2.SS1.p2.1.m1.1.1.2.3"><mtext mathsize="70%" id="S2.SS1.p2.1.m1.1.1.2.3.cmml" xref="S2.SS1.p2.1.m1.1.1.2.3">emb</mtext></ci></apply><apply id="S2.SS1.p2.1.m1.1.1.3.cmml" xref="S2.SS1.p2.1.m1.1.1.3"><csymbol cd="ambiguous" id="S2.SS1.p2.1.m1.1.1.3.1.cmml" xref="S2.SS1.p2.1.m1.1.1.3">superscript</csymbol><ci id="S2.SS1.p2.1.m1.1.1.3.2.cmml" xref="S2.SS1.p2.1.m1.1.1.3.2">𝑅</ci><apply id="S2.SS1.p2.1.m1.1.1.3.3.cmml" xref="S2.SS1.p2.1.m1.1.1.3.3"><times id="S2.SS1.p2.1.m1.1.1.3.3.1.cmml" xref="S2.SS1.p2.1.m1.1.1.3.3.1"></times><ci id="S2.SS1.p2.1.m1.1.1.3.3.2.cmml" xref="S2.SS1.p2.1.m1.1.1.3.3.2">𝑛</ci><ci id="S2.SS1.p2.1.m1.1.1.3.3.3.cmml" xref="S2.SS1.p2.1.m1.1.1.3.3.3">𝑑</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p2.1.m1.1c">\textbf{W}_{\text{emb}}\in R^{n\times d}</annotation></semantics></math> is the word embedding matrix, <math id="S2.SS1.p2.2.m2.1" class="ltx_Math" alttext="\textbf{W}_{\text{head}}\in R^{d\times n}" display="inline"><semantics id="S2.SS1.p2.2.m2.1a"><mrow id="S2.SS1.p2.2.m2.1.1" xref="S2.SS1.p2.2.m2.1.1.cmml"><msub id="S2.SS1.p2.2.m2.1.1.2" xref="S2.SS1.p2.2.m2.1.1.2.cmml"><mtext class="ltx_mathvariant_bold" id="S2.SS1.p2.2.m2.1.1.2.2" xref="S2.SS1.p2.2.m2.1.1.2.2a.cmml">W</mtext><mtext id="S2.SS1.p2.2.m2.1.1.2.3" xref="S2.SS1.p2.2.m2.1.1.2.3a.cmml">head</mtext></msub><mo id="S2.SS1.p2.2.m2.1.1.1" xref="S2.SS1.p2.2.m2.1.1.1.cmml">∈</mo><msup id="S2.SS1.p2.2.m2.1.1.3" xref="S2.SS1.p2.2.m2.1.1.3.cmml"><mi id="S2.SS1.p2.2.m2.1.1.3.2" xref="S2.SS1.p2.2.m2.1.1.3.2.cmml">R</mi><mrow id="S2.SS1.p2.2.m2.1.1.3.3" xref="S2.SS1.p2.2.m2.1.1.3.3.cmml"><mi id="S2.SS1.p2.2.m2.1.1.3.3.2" xref="S2.SS1.p2.2.m2.1.1.3.3.2.cmml">d</mi><mo lspace="0.222em" rspace="0.222em" id="S2.SS1.p2.2.m2.1.1.3.3.1" xref="S2.SS1.p2.2.m2.1.1.3.3.1.cmml">×</mo><mi id="S2.SS1.p2.2.m2.1.1.3.3.3" xref="S2.SS1.p2.2.m2.1.1.3.3.3.cmml">n</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S2.SS1.p2.2.m2.1b"><apply id="S2.SS1.p2.2.m2.1.1.cmml" xref="S2.SS1.p2.2.m2.1.1"><in id="S2.SS1.p2.2.m2.1.1.1.cmml" xref="S2.SS1.p2.2.m2.1.1.1"></in><apply id="S2.SS1.p2.2.m2.1.1.2.cmml" xref="S2.SS1.p2.2.m2.1.1.2"><csymbol cd="ambiguous" id="S2.SS1.p2.2.m2.1.1.2.1.cmml" xref="S2.SS1.p2.2.m2.1.1.2">subscript</csymbol><ci id="S2.SS1.p2.2.m2.1.1.2.2a.cmml" xref="S2.SS1.p2.2.m2.1.1.2.2"><mtext class="ltx_mathvariant_bold" id="S2.SS1.p2.2.m2.1.1.2.2.cmml" xref="S2.SS1.p2.2.m2.1.1.2.2">W</mtext></ci><ci id="S2.SS1.p2.2.m2.1.1.2.3a.cmml" xref="S2.SS1.p2.2.m2.1.1.2.3"><mtext mathsize="70%" id="S2.SS1.p2.2.m2.1.1.2.3.cmml" xref="S2.SS1.p2.2.m2.1.1.2.3">head</mtext></ci></apply><apply id="S2.SS1.p2.2.m2.1.1.3.cmml" xref="S2.SS1.p2.2.m2.1.1.3"><csymbol cd="ambiguous" id="S2.SS1.p2.2.m2.1.1.3.1.cmml" xref="S2.SS1.p2.2.m2.1.1.3">superscript</csymbol><ci id="S2.SS1.p2.2.m2.1.1.3.2.cmml" xref="S2.SS1.p2.2.m2.1.1.3.2">𝑅</ci><apply id="S2.SS1.p2.2.m2.1.1.3.3.cmml" xref="S2.SS1.p2.2.m2.1.1.3.3"><times id="S2.SS1.p2.2.m2.1.1.3.3.1.cmml" xref="S2.SS1.p2.2.m2.1.1.3.3.1"></times><ci id="S2.SS1.p2.2.m2.1.1.3.3.2.cmml" xref="S2.SS1.p2.2.m2.1.1.3.3.2">𝑑</ci><ci id="S2.SS1.p2.2.m2.1.1.3.3.3.cmml" xref="S2.SS1.p2.2.m2.1.1.3.3.3">𝑛</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p2.2.m2.1c">\textbf{W}_{\text{head}}\in R^{d\times n}</annotation></semantics></math> is the output projection matrix of the transformer, which are sometimes tied with the <math id="S2.SS1.p2.3.m3.1" class="ltx_Math" alttext="\textbf{W}_{\text{emb}}" display="inline"><semantics id="S2.SS1.p2.3.m3.1a"><msub id="S2.SS1.p2.3.m3.1.1" xref="S2.SS1.p2.3.m3.1.1.cmml"><mtext class="ltx_mathvariant_bold" id="S2.SS1.p2.3.m3.1.1.2" xref="S2.SS1.p2.3.m3.1.1.2a.cmml">W</mtext><mtext id="S2.SS1.p2.3.m3.1.1.3" xref="S2.SS1.p2.3.m3.1.1.3a.cmml">emb</mtext></msub><annotation-xml encoding="MathML-Content" id="S2.SS1.p2.3.m3.1b"><apply id="S2.SS1.p2.3.m3.1.1.cmml" xref="S2.SS1.p2.3.m3.1.1"><csymbol cd="ambiguous" id="S2.SS1.p2.3.m3.1.1.1.cmml" xref="S2.SS1.p2.3.m3.1.1">subscript</csymbol><ci id="S2.SS1.p2.3.m3.1.1.2a.cmml" xref="S2.SS1.p2.3.m3.1.1.2"><mtext class="ltx_mathvariant_bold" id="S2.SS1.p2.3.m3.1.1.2.cmml" xref="S2.SS1.p2.3.m3.1.1.2">W</mtext></ci><ci id="S2.SS1.p2.3.m3.1.1.3a.cmml" xref="S2.SS1.p2.3.m3.1.1.3"><mtext mathsize="70%" id="S2.SS1.p2.3.m3.1.1.3.cmml" xref="S2.SS1.p2.3.m3.1.1.3">emb</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p2.3.m3.1c">\textbf{W}_{\text{emb}}</annotation></semantics></math> <cite class="ltx_cite ltx_citemacro_citep">(Chowdhery et al., <a href="#bib.bib7" title="" class="ltx_ref">2023</a>)</cite>, <math id="S2.SS1.p2.4.m4.1" class="ltx_Math" alttext="d" display="inline"><semantics id="S2.SS1.p2.4.m4.1a"><mi id="S2.SS1.p2.4.m4.1.1" xref="S2.SS1.p2.4.m4.1.1.cmml">d</mi><annotation-xml encoding="MathML-Content" id="S2.SS1.p2.4.m4.1b"><ci id="S2.SS1.p2.4.m4.1.1.cmml" xref="S2.SS1.p2.4.m4.1.1">𝑑</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p2.4.m4.1c">d</annotation></semantics></math> is the hidden dim of the transformer. <span id="S2.SS1.p2.8.1" class="ltx_text ltx_markedasmath ltx_font_bold">ATTN</span> refers to the attention layer and <span id="S2.SS1.p2.8.2" class="ltx_text ltx_markedasmath ltx_font_bold">FFN</span> means the feed-forward layers, <math id="S2.SS1.p2.7.m7.1" class="ltx_Math" alttext="\textbf{X}_{i}\in R^{T\times d}" display="inline"><semantics id="S2.SS1.p2.7.m7.1a"><mrow id="S2.SS1.p2.7.m7.1.1" xref="S2.SS1.p2.7.m7.1.1.cmml"><msub id="S2.SS1.p2.7.m7.1.1.2" xref="S2.SS1.p2.7.m7.1.1.2.cmml"><mtext class="ltx_mathvariant_bold" id="S2.SS1.p2.7.m7.1.1.2.2" xref="S2.SS1.p2.7.m7.1.1.2.2a.cmml">X</mtext><mi id="S2.SS1.p2.7.m7.1.1.2.3" xref="S2.SS1.p2.7.m7.1.1.2.3.cmml">i</mi></msub><mo id="S2.SS1.p2.7.m7.1.1.1" xref="S2.SS1.p2.7.m7.1.1.1.cmml">∈</mo><msup id="S2.SS1.p2.7.m7.1.1.3" xref="S2.SS1.p2.7.m7.1.1.3.cmml"><mi id="S2.SS1.p2.7.m7.1.1.3.2" xref="S2.SS1.p2.7.m7.1.1.3.2.cmml">R</mi><mrow id="S2.SS1.p2.7.m7.1.1.3.3" xref="S2.SS1.p2.7.m7.1.1.3.3.cmml"><mi id="S2.SS1.p2.7.m7.1.1.3.3.2" xref="S2.SS1.p2.7.m7.1.1.3.3.2.cmml">T</mi><mo lspace="0.222em" rspace="0.222em" id="S2.SS1.p2.7.m7.1.1.3.3.1" xref="S2.SS1.p2.7.m7.1.1.3.3.1.cmml">×</mo><mi id="S2.SS1.p2.7.m7.1.1.3.3.3" xref="S2.SS1.p2.7.m7.1.1.3.3.3.cmml">d</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S2.SS1.p2.7.m7.1b"><apply id="S2.SS1.p2.7.m7.1.1.cmml" xref="S2.SS1.p2.7.m7.1.1"><in id="S2.SS1.p2.7.m7.1.1.1.cmml" xref="S2.SS1.p2.7.m7.1.1.1"></in><apply id="S2.SS1.p2.7.m7.1.1.2.cmml" xref="S2.SS1.p2.7.m7.1.1.2"><csymbol cd="ambiguous" id="S2.SS1.p2.7.m7.1.1.2.1.cmml" xref="S2.SS1.p2.7.m7.1.1.2">subscript</csymbol><ci id="S2.SS1.p2.7.m7.1.1.2.2a.cmml" xref="S2.SS1.p2.7.m7.1.1.2.2"><mtext class="ltx_mathvariant_bold" id="S2.SS1.p2.7.m7.1.1.2.2.cmml" xref="S2.SS1.p2.7.m7.1.1.2.2">X</mtext></ci><ci id="S2.SS1.p2.7.m7.1.1.2.3.cmml" xref="S2.SS1.p2.7.m7.1.1.2.3">𝑖</ci></apply><apply id="S2.SS1.p2.7.m7.1.1.3.cmml" xref="S2.SS1.p2.7.m7.1.1.3"><csymbol cd="ambiguous" id="S2.SS1.p2.7.m7.1.1.3.1.cmml" xref="S2.SS1.p2.7.m7.1.1.3">superscript</csymbol><ci id="S2.SS1.p2.7.m7.1.1.3.2.cmml" xref="S2.SS1.p2.7.m7.1.1.3.2">𝑅</ci><apply id="S2.SS1.p2.7.m7.1.1.3.3.cmml" xref="S2.SS1.p2.7.m7.1.1.3.3"><times id="S2.SS1.p2.7.m7.1.1.3.3.1.cmml" xref="S2.SS1.p2.7.m7.1.1.3.3.1"></times><ci id="S2.SS1.p2.7.m7.1.1.3.3.2.cmml" xref="S2.SS1.p2.7.m7.1.1.3.3.2">𝑇</ci><ci id="S2.SS1.p2.7.m7.1.1.3.3.3.cmml" xref="S2.SS1.p2.7.m7.1.1.3.3.3">𝑑</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p2.7.m7.1c">\textbf{X}_{i}\in R^{T\times d}</annotation></semantics></math> is the hidden states of the <math id="S2.SS1.p2.8.m8.1" class="ltx_Math" alttext="i^{th}" display="inline"><semantics id="S2.SS1.p2.8.m8.1a"><msup id="S2.SS1.p2.8.m8.1.1" xref="S2.SS1.p2.8.m8.1.1.cmml"><mi id="S2.SS1.p2.8.m8.1.1.2" xref="S2.SS1.p2.8.m8.1.1.2.cmml">i</mi><mrow id="S2.SS1.p2.8.m8.1.1.3" xref="S2.SS1.p2.8.m8.1.1.3.cmml"><mi id="S2.SS1.p2.8.m8.1.1.3.2" xref="S2.SS1.p2.8.m8.1.1.3.2.cmml">t</mi><mo lspace="0em" rspace="0em" id="S2.SS1.p2.8.m8.1.1.3.1" xref="S2.SS1.p2.8.m8.1.1.3.1.cmml">​</mo><mi id="S2.SS1.p2.8.m8.1.1.3.3" xref="S2.SS1.p2.8.m8.1.1.3.3.cmml">h</mi></mrow></msup><annotation-xml encoding="MathML-Content" id="S2.SS1.p2.8.m8.1b"><apply id="S2.SS1.p2.8.m8.1.1.cmml" xref="S2.SS1.p2.8.m8.1.1"><csymbol cd="ambiguous" id="S2.SS1.p2.8.m8.1.1.1.cmml" xref="S2.SS1.p2.8.m8.1.1">superscript</csymbol><ci id="S2.SS1.p2.8.m8.1.1.2.cmml" xref="S2.SS1.p2.8.m8.1.1.2">𝑖</ci><apply id="S2.SS1.p2.8.m8.1.1.3.cmml" xref="S2.SS1.p2.8.m8.1.1.3"><times id="S2.SS1.p2.8.m8.1.1.3.1.cmml" xref="S2.SS1.p2.8.m8.1.1.3.1"></times><ci id="S2.SS1.p2.8.m8.1.1.3.2.cmml" xref="S2.SS1.p2.8.m8.1.1.3.2">𝑡</ci><ci id="S2.SS1.p2.8.m8.1.1.3.3.cmml" xref="S2.SS1.p2.8.m8.1.1.3.3">ℎ</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p2.8.m8.1c">i^{th}</annotation></semantics></math> layers.
Given that the transformer structure is composed of several identical layers, a natural question is what the differences and connections between the functionalities and roles of these identical layers are. Previous works found that the Transformer possesses certain semantic capabilities in earlier layers <cite class="ltx_cite ltx_citemacro_citep">(Hasan et al., <a href="#bib.bib20" title="" class="ltx_ref">2021</a>)</cite>. In this work, we uncover a significant level of redundancy between the layers of the transformer.</p>
</div>
<div id="S2.SS1.p3" class="ltx_para">
<p id="S2.SS1.p3.1" class="ltx_p"><span id="S2.SS1.p3.1.1" class="ltx_text ltx_font_bold">Layer Redundancy</span>: A network with high redundancy should contain some redundant layers, which have a minimal impact on the network’s final performance. This may be due to these layers having homogenized functionalities compared to other layers in the network. We have found a high degree of redundancy in the current LLMs by omitting specific layers.</p>
</div>
<figure id="S2.F3" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="S2.F3.sf1" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2403.03853/assets/x4.png" id="S2.F3.sf1.g1" class="ltx_graphics ltx_img_landscape" width="147" height="73" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure"><span id="S2.F3.sf1.2.1.1" class="ltx_text" style="font-size:80%;">(a)</span> </span><span id="S2.F3.sf1.3.2" class="ltx_text" style="font-size:80%;">Llama2-7B-Base</span></figcaption>
</figure>
</div>
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="S2.F3.sf2" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2403.03853/assets/x5.png" id="S2.F3.sf2.g1" class="ltx_graphics ltx_img_landscape" width="144" height="73" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure"><span id="S2.F3.sf2.2.1.1" class="ltx_text" style="font-size:80%;">(b)</span> </span><span id="S2.F3.sf2.3.2" class="ltx_text" style="font-size:80%;">Baichuan2-7B-Base</span></figcaption>
</figure>
</div>
</div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 3: </span>Performance of removing single layer.</figcaption>
</figure>
<div id="S2.SS1.p4" class="ltx_para">
<p id="S2.SS1.p4.1" class="ltx_p">We observed that in many cases, the network’s prediction of <math id="S2.SS1.p4.1.m1.1" class="ltx_Math" alttext="Y" display="inline"><semantics id="S2.SS1.p4.1.m1.1a"><mi id="S2.SS1.p4.1.m1.1.1" xref="S2.SS1.p4.1.m1.1.1.cmml">Y</mi><annotation-xml encoding="MathML-Content" id="S2.SS1.p4.1.m1.1b"><ci id="S2.SS1.p4.1.m1.1.1.cmml" xref="S2.SS1.p4.1.m1.1.1">𝑌</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p4.1.m1.1c">Y</annotation></semantics></math> remains largely unchanged. This phenomenon demonstrates a significant level of redundancy across the layers of the transformer. Figure <a href="#S2.F3" title="Figure 3 ‣ 2.1 Layer redundancy ‣ 2 Methodology ‣ ShortGPT: Layers in Large Language Models are More Redundant Than You Expect" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a> shows the perplexity and MMLU <cite class="ltx_cite ltx_citemacro_citep">(Hendrycks et al., <a href="#bib.bib21" title="" class="ltx_ref">2020</a>)</cite> score of omitting a certain layer in the Llama2-7B-Base <cite class="ltx_cite ltx_citemacro_citep">(Touvron et al., <a href="#bib.bib43" title="" class="ltx_ref">2023</a>)</cite>, an English based LLMs, and Baichuan2-7B-Base <cite class="ltx_cite ltx_citemacro_citep">(Yang et al., <a href="#bib.bib48" title="" class="ltx_ref">2023</a>)</cite> which is mainly focused on Chinese. For more benchmark results, please refer to the section <a href="#S3" title="3 Experiments ‣ ShortGPT: Layers in Large Language Models are More Redundant Than You Expect" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>. Figure <a href="#S2.F3" title="Figure 3 ‣ 2.1 Layer redundancy ‣ 2 Methodology ‣ ShortGPT: Layers in Large Language Models are More Redundant Than You Expect" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a> reveals that some layers do not play a crucial role in LLMs. Dropping a certain layer from the LLMs may have a minimal effect on the final results, and the same holds for the perplexity. Moreover, this redundancy is primarily manifested in the middle to later layers of the network, with the initial layers and the final layer often being more critical.</p>
</div>
<figure id="S2.F4" class="ltx_figure"><img src="/html/2403.03853/assets/x6.png" id="S2.F4.g1" class="ltx_graphics ltx_centering ltx_img_square" width="276" height="237" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 4: </span> Illustration of our Block Influence(BI) metric. We omit LayerNorm for visual conciseness. </figcaption>
</figure>
</section>
<section id="S2.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.2 </span>Layer importance</h3>

<div id="S2.SS2.p1" class="ltx_para">
<p id="S2.SS2.p1.2" class="ltx_p">As outlined in the preceding section, the layers of LLMs exhibit redundancy, with varying degrees of redundancy across different layers. Notably, the deeper layers demonstrate a higher level of redundancy. To remove these redundant layers, we need a model-intrinsic metric to measure the importance of the layer. An intuitive method is to use the magnitude of each layer’s output as a measure of its importance, as a larger magnitude implies more activated neurons. In <cite class="ltx_cite ltx_citemacro_citep">(Samragh et al., <a href="#bib.bib39" title="" class="ltx_ref">2023</a>)</cite>, they proposed to use relative magnitude <math id="S2.SS2.p1.1.m1.2" class="ltx_Math" alttext="||\frac{f(x)}{x+f(x)}||" display="inline"><semantics id="S2.SS2.p1.1.m1.2a"><mrow id="S2.SS2.p1.1.m1.2.3.2" xref="S2.SS2.p1.1.m1.2.3.1.cmml"><mo stretchy="false" id="S2.SS2.p1.1.m1.2.3.2.1" xref="S2.SS2.p1.1.m1.2.3.1.1.cmml">‖</mo><mfrac id="S2.SS2.p1.1.m1.2.2" xref="S2.SS2.p1.1.m1.2.2.cmml"><mrow id="S2.SS2.p1.1.m1.1.1.1" xref="S2.SS2.p1.1.m1.1.1.1.cmml"><mi id="S2.SS2.p1.1.m1.1.1.1.3" xref="S2.SS2.p1.1.m1.1.1.1.3.cmml">f</mi><mo lspace="0em" rspace="0em" id="S2.SS2.p1.1.m1.1.1.1.2" xref="S2.SS2.p1.1.m1.1.1.1.2.cmml">​</mo><mrow id="S2.SS2.p1.1.m1.1.1.1.4.2" xref="S2.SS2.p1.1.m1.1.1.1.cmml"><mo stretchy="false" id="S2.SS2.p1.1.m1.1.1.1.4.2.1" xref="S2.SS2.p1.1.m1.1.1.1.cmml">(</mo><mi id="S2.SS2.p1.1.m1.1.1.1.1" xref="S2.SS2.p1.1.m1.1.1.1.1.cmml">x</mi><mo stretchy="false" id="S2.SS2.p1.1.m1.1.1.1.4.2.2" xref="S2.SS2.p1.1.m1.1.1.1.cmml">)</mo></mrow></mrow><mrow id="S2.SS2.p1.1.m1.2.2.2" xref="S2.SS2.p1.1.m1.2.2.2.cmml"><mi id="S2.SS2.p1.1.m1.2.2.2.3" xref="S2.SS2.p1.1.m1.2.2.2.3.cmml">x</mi><mo id="S2.SS2.p1.1.m1.2.2.2.2" xref="S2.SS2.p1.1.m1.2.2.2.2.cmml">+</mo><mrow id="S2.SS2.p1.1.m1.2.2.2.4" xref="S2.SS2.p1.1.m1.2.2.2.4.cmml"><mi id="S2.SS2.p1.1.m1.2.2.2.4.2" xref="S2.SS2.p1.1.m1.2.2.2.4.2.cmml">f</mi><mo lspace="0em" rspace="0em" id="S2.SS2.p1.1.m1.2.2.2.4.1" xref="S2.SS2.p1.1.m1.2.2.2.4.1.cmml">​</mo><mrow id="S2.SS2.p1.1.m1.2.2.2.4.3.2" xref="S2.SS2.p1.1.m1.2.2.2.4.cmml"><mo stretchy="false" id="S2.SS2.p1.1.m1.2.2.2.4.3.2.1" xref="S2.SS2.p1.1.m1.2.2.2.4.cmml">(</mo><mi id="S2.SS2.p1.1.m1.2.2.2.1" xref="S2.SS2.p1.1.m1.2.2.2.1.cmml">x</mi><mo stretchy="false" id="S2.SS2.p1.1.m1.2.2.2.4.3.2.2" xref="S2.SS2.p1.1.m1.2.2.2.4.cmml">)</mo></mrow></mrow></mrow></mfrac><mo stretchy="false" id="S2.SS2.p1.1.m1.2.3.2.2" xref="S2.SS2.p1.1.m1.2.3.1.1.cmml">‖</mo></mrow><annotation-xml encoding="MathML-Content" id="S2.SS2.p1.1.m1.2b"><apply id="S2.SS2.p1.1.m1.2.3.1.cmml" xref="S2.SS2.p1.1.m1.2.3.2"><csymbol cd="latexml" id="S2.SS2.p1.1.m1.2.3.1.1.cmml" xref="S2.SS2.p1.1.m1.2.3.2.1">norm</csymbol><apply id="S2.SS2.p1.1.m1.2.2.cmml" xref="S2.SS2.p1.1.m1.2.2"><divide id="S2.SS2.p1.1.m1.2.2.3.cmml" xref="S2.SS2.p1.1.m1.2.2"></divide><apply id="S2.SS2.p1.1.m1.1.1.1.cmml" xref="S2.SS2.p1.1.m1.1.1.1"><times id="S2.SS2.p1.1.m1.1.1.1.2.cmml" xref="S2.SS2.p1.1.m1.1.1.1.2"></times><ci id="S2.SS2.p1.1.m1.1.1.1.3.cmml" xref="S2.SS2.p1.1.m1.1.1.1.3">𝑓</ci><ci id="S2.SS2.p1.1.m1.1.1.1.1.cmml" xref="S2.SS2.p1.1.m1.1.1.1.1">𝑥</ci></apply><apply id="S2.SS2.p1.1.m1.2.2.2.cmml" xref="S2.SS2.p1.1.m1.2.2.2"><plus id="S2.SS2.p1.1.m1.2.2.2.2.cmml" xref="S2.SS2.p1.1.m1.2.2.2.2"></plus><ci id="S2.SS2.p1.1.m1.2.2.2.3.cmml" xref="S2.SS2.p1.1.m1.2.2.2.3">𝑥</ci><apply id="S2.SS2.p1.1.m1.2.2.2.4.cmml" xref="S2.SS2.p1.1.m1.2.2.2.4"><times id="S2.SS2.p1.1.m1.2.2.2.4.1.cmml" xref="S2.SS2.p1.1.m1.2.2.2.4.1"></times><ci id="S2.SS2.p1.1.m1.2.2.2.4.2.cmml" xref="S2.SS2.p1.1.m1.2.2.2.4.2">𝑓</ci><ci id="S2.SS2.p1.1.m1.2.2.2.1.cmml" xref="S2.SS2.p1.1.m1.2.2.2.1">𝑥</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p1.1.m1.2c">||\frac{f(x)}{x+f(x)}||</annotation></semantics></math> to measure the importance of layers. To characterize the influence of a certain layer. We defined a new metric, Block Influence (BI), under the assumption that the more a transformer block changes the hidden states, the more influential this layer is. As Figure <a href="#S2.F4" title="Figure 4 ‣ 2.1 Layer redundancy ‣ 2 Methodology ‣ ShortGPT: Layers in Large Language Models are More Redundant Than You Expect" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a> shows, the BI score of <math id="S2.SS2.p1.2.m2.1" class="ltx_Math" alttext="i^{th}" display="inline"><semantics id="S2.SS2.p1.2.m2.1a"><msup id="S2.SS2.p1.2.m2.1.1" xref="S2.SS2.p1.2.m2.1.1.cmml"><mi id="S2.SS2.p1.2.m2.1.1.2" xref="S2.SS2.p1.2.m2.1.1.2.cmml">i</mi><mrow id="S2.SS2.p1.2.m2.1.1.3" xref="S2.SS2.p1.2.m2.1.1.3.cmml"><mi id="S2.SS2.p1.2.m2.1.1.3.2" xref="S2.SS2.p1.2.m2.1.1.3.2.cmml">t</mi><mo lspace="0em" rspace="0em" id="S2.SS2.p1.2.m2.1.1.3.1" xref="S2.SS2.p1.2.m2.1.1.3.1.cmml">​</mo><mi id="S2.SS2.p1.2.m2.1.1.3.3" xref="S2.SS2.p1.2.m2.1.1.3.3.cmml">h</mi></mrow></msup><annotation-xml encoding="MathML-Content" id="S2.SS2.p1.2.m2.1b"><apply id="S2.SS2.p1.2.m2.1.1.cmml" xref="S2.SS2.p1.2.m2.1.1"><csymbol cd="ambiguous" id="S2.SS2.p1.2.m2.1.1.1.cmml" xref="S2.SS2.p1.2.m2.1.1">superscript</csymbol><ci id="S2.SS2.p1.2.m2.1.1.2.cmml" xref="S2.SS2.p1.2.m2.1.1.2">𝑖</ci><apply id="S2.SS2.p1.2.m2.1.1.3.cmml" xref="S2.SS2.p1.2.m2.1.1.3"><times id="S2.SS2.p1.2.m2.1.1.3.1.cmml" xref="S2.SS2.p1.2.m2.1.1.3.1"></times><ci id="S2.SS2.p1.2.m2.1.1.3.2.cmml" xref="S2.SS2.p1.2.m2.1.1.3.2">𝑡</ci><ci id="S2.SS2.p1.2.m2.1.1.3.3.cmml" xref="S2.SS2.p1.2.m2.1.1.3.3">ℎ</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p1.2.m2.1c">i^{th}</annotation></semantics></math> blocks can be calculated as follows:</p>
<table id="A4.EGx2" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="S2.E2"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="S2.E2.m1.13" class="ltx_Math" alttext="\displaystyle\text{BI}_{i}=1-\mathbb{E}_{X,t}\frac{X_{i,t}^{T}X_{i+1,t}}{||X_{i,t}||_{2}||X_{i+1,t}||_{2}}," display="inline"><semantics id="S2.E2.m1.13a"><mrow id="S2.E2.m1.13.13.1" xref="S2.E2.m1.13.13.1.1.cmml"><mrow id="S2.E2.m1.13.13.1.1" xref="S2.E2.m1.13.13.1.1.cmml"><msub id="S2.E2.m1.13.13.1.1.2" xref="S2.E2.m1.13.13.1.1.2.cmml"><mtext id="S2.E2.m1.13.13.1.1.2.2" xref="S2.E2.m1.13.13.1.1.2.2a.cmml">BI</mtext><mi id="S2.E2.m1.13.13.1.1.2.3" xref="S2.E2.m1.13.13.1.1.2.3.cmml">i</mi></msub><mo id="S2.E2.m1.13.13.1.1.1" xref="S2.E2.m1.13.13.1.1.1.cmml">=</mo><mrow id="S2.E2.m1.13.13.1.1.3" xref="S2.E2.m1.13.13.1.1.3.cmml"><mn id="S2.E2.m1.13.13.1.1.3.2" xref="S2.E2.m1.13.13.1.1.3.2.cmml">1</mn><mo id="S2.E2.m1.13.13.1.1.3.1" xref="S2.E2.m1.13.13.1.1.3.1.cmml">−</mo><mrow id="S2.E2.m1.13.13.1.1.3.3" xref="S2.E2.m1.13.13.1.1.3.3.cmml"><msub id="S2.E2.m1.13.13.1.1.3.3.2" xref="S2.E2.m1.13.13.1.1.3.3.2.cmml"><mi id="S2.E2.m1.13.13.1.1.3.3.2.2" xref="S2.E2.m1.13.13.1.1.3.3.2.2.cmml">𝔼</mi><mrow id="S2.E2.m1.2.2.2.4" xref="S2.E2.m1.2.2.2.3.cmml"><mi id="S2.E2.m1.1.1.1.1" xref="S2.E2.m1.1.1.1.1.cmml">X</mi><mo id="S2.E2.m1.2.2.2.4.1" xref="S2.E2.m1.2.2.2.3.cmml">,</mo><mi id="S2.E2.m1.2.2.2.2" xref="S2.E2.m1.2.2.2.2.cmml">t</mi></mrow></msub><mo lspace="0em" rspace="0em" id="S2.E2.m1.13.13.1.1.3.3.1" xref="S2.E2.m1.13.13.1.1.3.3.1.cmml">​</mo><mstyle displaystyle="true" id="S2.E2.m1.12.12" xref="S2.E2.m1.12.12.cmml"><mfrac id="S2.E2.m1.12.12a" xref="S2.E2.m1.12.12.cmml"><mrow id="S2.E2.m1.6.6.4" xref="S2.E2.m1.6.6.4.cmml"><msubsup id="S2.E2.m1.6.6.4.6" xref="S2.E2.m1.6.6.4.6.cmml"><mi id="S2.E2.m1.6.6.4.6.2.2" xref="S2.E2.m1.6.6.4.6.2.2.cmml">X</mi><mrow id="S2.E2.m1.4.4.2.2.2.4" xref="S2.E2.m1.4.4.2.2.2.3.cmml"><mi id="S2.E2.m1.3.3.1.1.1.1" xref="S2.E2.m1.3.3.1.1.1.1.cmml">i</mi><mo id="S2.E2.m1.4.4.2.2.2.4.1" xref="S2.E2.m1.4.4.2.2.2.3.cmml">,</mo><mi id="S2.E2.m1.4.4.2.2.2.2" xref="S2.E2.m1.4.4.2.2.2.2.cmml">t</mi></mrow><mi id="S2.E2.m1.6.6.4.6.3" xref="S2.E2.m1.6.6.4.6.3.cmml">T</mi></msubsup><mo lspace="0em" rspace="0em" id="S2.E2.m1.6.6.4.5" xref="S2.E2.m1.6.6.4.5.cmml">​</mo><msub id="S2.E2.m1.6.6.4.7" xref="S2.E2.m1.6.6.4.7.cmml"><mi id="S2.E2.m1.6.6.4.7.2" xref="S2.E2.m1.6.6.4.7.2.cmml">X</mi><mrow id="S2.E2.m1.6.6.4.4.2.2" xref="S2.E2.m1.6.6.4.4.2.3.cmml"><mrow id="S2.E2.m1.6.6.4.4.2.2.1" xref="S2.E2.m1.6.6.4.4.2.2.1.cmml"><mi id="S2.E2.m1.6.6.4.4.2.2.1.2" xref="S2.E2.m1.6.6.4.4.2.2.1.2.cmml">i</mi><mo id="S2.E2.m1.6.6.4.4.2.2.1.1" xref="S2.E2.m1.6.6.4.4.2.2.1.1.cmml">+</mo><mn id="S2.E2.m1.6.6.4.4.2.2.1.3" xref="S2.E2.m1.6.6.4.4.2.2.1.3.cmml">1</mn></mrow><mo id="S2.E2.m1.6.6.4.4.2.2.2" xref="S2.E2.m1.6.6.4.4.2.3.cmml">,</mo><mi id="S2.E2.m1.5.5.3.3.1.1" xref="S2.E2.m1.5.5.3.3.1.1.cmml">t</mi></mrow></msub></mrow><mrow id="S2.E2.m1.12.12.10" xref="S2.E2.m1.12.12.10.cmml"><msub id="S2.E2.m1.11.11.9.5" xref="S2.E2.m1.11.11.9.5.cmml"><mrow id="S2.E2.m1.11.11.9.5.1.1" xref="S2.E2.m1.11.11.9.5.1.2.cmml"><mo stretchy="false" id="S2.E2.m1.11.11.9.5.1.1.2" xref="S2.E2.m1.11.11.9.5.1.2.1.cmml">‖</mo><msub id="S2.E2.m1.11.11.9.5.1.1.1" xref="S2.E2.m1.11.11.9.5.1.1.1.cmml"><mi id="S2.E2.m1.11.11.9.5.1.1.1.2" xref="S2.E2.m1.11.11.9.5.1.1.1.2.cmml">X</mi><mrow id="S2.E2.m1.8.8.6.2.2.4" xref="S2.E2.m1.8.8.6.2.2.3.cmml"><mi id="S2.E2.m1.7.7.5.1.1.1" xref="S2.E2.m1.7.7.5.1.1.1.cmml">i</mi><mo id="S2.E2.m1.8.8.6.2.2.4.1" xref="S2.E2.m1.8.8.6.2.2.3.cmml">,</mo><mi id="S2.E2.m1.8.8.6.2.2.2" xref="S2.E2.m1.8.8.6.2.2.2.cmml">t</mi></mrow></msub><mo stretchy="false" id="S2.E2.m1.11.11.9.5.1.1.3" xref="S2.E2.m1.11.11.9.5.1.2.1.cmml">‖</mo></mrow><mn id="S2.E2.m1.11.11.9.5.3" xref="S2.E2.m1.11.11.9.5.3.cmml">2</mn></msub><mo lspace="0em" rspace="0em" id="S2.E2.m1.12.12.10.7" xref="S2.E2.m1.12.12.10.7.cmml">​</mo><msub id="S2.E2.m1.12.12.10.6" xref="S2.E2.m1.12.12.10.6.cmml"><mrow id="S2.E2.m1.12.12.10.6.1.1" xref="S2.E2.m1.12.12.10.6.1.2.cmml"><mo stretchy="false" id="S2.E2.m1.12.12.10.6.1.1.2" xref="S2.E2.m1.12.12.10.6.1.2.1.cmml">‖</mo><msub id="S2.E2.m1.12.12.10.6.1.1.1" xref="S2.E2.m1.12.12.10.6.1.1.1.cmml"><mi id="S2.E2.m1.12.12.10.6.1.1.1.2" xref="S2.E2.m1.12.12.10.6.1.1.1.2.cmml">X</mi><mrow id="S2.E2.m1.10.10.8.4.2.2" xref="S2.E2.m1.10.10.8.4.2.3.cmml"><mrow id="S2.E2.m1.10.10.8.4.2.2.1" xref="S2.E2.m1.10.10.8.4.2.2.1.cmml"><mi id="S2.E2.m1.10.10.8.4.2.2.1.2" xref="S2.E2.m1.10.10.8.4.2.2.1.2.cmml">i</mi><mo id="S2.E2.m1.10.10.8.4.2.2.1.1" xref="S2.E2.m1.10.10.8.4.2.2.1.1.cmml">+</mo><mn id="S2.E2.m1.10.10.8.4.2.2.1.3" xref="S2.E2.m1.10.10.8.4.2.2.1.3.cmml">1</mn></mrow><mo id="S2.E2.m1.10.10.8.4.2.2.2" xref="S2.E2.m1.10.10.8.4.2.3.cmml">,</mo><mi id="S2.E2.m1.9.9.7.3.1.1" xref="S2.E2.m1.9.9.7.3.1.1.cmml">t</mi></mrow></msub><mo stretchy="false" id="S2.E2.m1.12.12.10.6.1.1.3" xref="S2.E2.m1.12.12.10.6.1.2.1.cmml">‖</mo></mrow><mn id="S2.E2.m1.12.12.10.6.3" xref="S2.E2.m1.12.12.10.6.3.cmml">2</mn></msub></mrow></mfrac></mstyle></mrow></mrow></mrow><mo id="S2.E2.m1.13.13.1.2" xref="S2.E2.m1.13.13.1.1.cmml">,</mo></mrow><annotation-xml encoding="MathML-Content" id="S2.E2.m1.13b"><apply id="S2.E2.m1.13.13.1.1.cmml" xref="S2.E2.m1.13.13.1"><eq id="S2.E2.m1.13.13.1.1.1.cmml" xref="S2.E2.m1.13.13.1.1.1"></eq><apply id="S2.E2.m1.13.13.1.1.2.cmml" xref="S2.E2.m1.13.13.1.1.2"><csymbol cd="ambiguous" id="S2.E2.m1.13.13.1.1.2.1.cmml" xref="S2.E2.m1.13.13.1.1.2">subscript</csymbol><ci id="S2.E2.m1.13.13.1.1.2.2a.cmml" xref="S2.E2.m1.13.13.1.1.2.2"><mtext id="S2.E2.m1.13.13.1.1.2.2.cmml" xref="S2.E2.m1.13.13.1.1.2.2">BI</mtext></ci><ci id="S2.E2.m1.13.13.1.1.2.3.cmml" xref="S2.E2.m1.13.13.1.1.2.3">𝑖</ci></apply><apply id="S2.E2.m1.13.13.1.1.3.cmml" xref="S2.E2.m1.13.13.1.1.3"><minus id="S2.E2.m1.13.13.1.1.3.1.cmml" xref="S2.E2.m1.13.13.1.1.3.1"></minus><cn type="integer" id="S2.E2.m1.13.13.1.1.3.2.cmml" xref="S2.E2.m1.13.13.1.1.3.2">1</cn><apply id="S2.E2.m1.13.13.1.1.3.3.cmml" xref="S2.E2.m1.13.13.1.1.3.3"><times id="S2.E2.m1.13.13.1.1.3.3.1.cmml" xref="S2.E2.m1.13.13.1.1.3.3.1"></times><apply id="S2.E2.m1.13.13.1.1.3.3.2.cmml" xref="S2.E2.m1.13.13.1.1.3.3.2"><csymbol cd="ambiguous" id="S2.E2.m1.13.13.1.1.3.3.2.1.cmml" xref="S2.E2.m1.13.13.1.1.3.3.2">subscript</csymbol><ci id="S2.E2.m1.13.13.1.1.3.3.2.2.cmml" xref="S2.E2.m1.13.13.1.1.3.3.2.2">𝔼</ci><list id="S2.E2.m1.2.2.2.3.cmml" xref="S2.E2.m1.2.2.2.4"><ci id="S2.E2.m1.1.1.1.1.cmml" xref="S2.E2.m1.1.1.1.1">𝑋</ci><ci id="S2.E2.m1.2.2.2.2.cmml" xref="S2.E2.m1.2.2.2.2">𝑡</ci></list></apply><apply id="S2.E2.m1.12.12.cmml" xref="S2.E2.m1.12.12"><divide id="S2.E2.m1.12.12.11.cmml" xref="S2.E2.m1.12.12"></divide><apply id="S2.E2.m1.6.6.4.cmml" xref="S2.E2.m1.6.6.4"><times id="S2.E2.m1.6.6.4.5.cmml" xref="S2.E2.m1.6.6.4.5"></times><apply id="S2.E2.m1.6.6.4.6.cmml" xref="S2.E2.m1.6.6.4.6"><csymbol cd="ambiguous" id="S2.E2.m1.6.6.4.6.1.cmml" xref="S2.E2.m1.6.6.4.6">superscript</csymbol><apply id="S2.E2.m1.6.6.4.6.2.cmml" xref="S2.E2.m1.6.6.4.6"><csymbol cd="ambiguous" id="S2.E2.m1.6.6.4.6.2.1.cmml" xref="S2.E2.m1.6.6.4.6">subscript</csymbol><ci id="S2.E2.m1.6.6.4.6.2.2.cmml" xref="S2.E2.m1.6.6.4.6.2.2">𝑋</ci><list id="S2.E2.m1.4.4.2.2.2.3.cmml" xref="S2.E2.m1.4.4.2.2.2.4"><ci id="S2.E2.m1.3.3.1.1.1.1.cmml" xref="S2.E2.m1.3.3.1.1.1.1">𝑖</ci><ci id="S2.E2.m1.4.4.2.2.2.2.cmml" xref="S2.E2.m1.4.4.2.2.2.2">𝑡</ci></list></apply><ci id="S2.E2.m1.6.6.4.6.3.cmml" xref="S2.E2.m1.6.6.4.6.3">𝑇</ci></apply><apply id="S2.E2.m1.6.6.4.7.cmml" xref="S2.E2.m1.6.6.4.7"><csymbol cd="ambiguous" id="S2.E2.m1.6.6.4.7.1.cmml" xref="S2.E2.m1.6.6.4.7">subscript</csymbol><ci id="S2.E2.m1.6.6.4.7.2.cmml" xref="S2.E2.m1.6.6.4.7.2">𝑋</ci><list id="S2.E2.m1.6.6.4.4.2.3.cmml" xref="S2.E2.m1.6.6.4.4.2.2"><apply id="S2.E2.m1.6.6.4.4.2.2.1.cmml" xref="S2.E2.m1.6.6.4.4.2.2.1"><plus id="S2.E2.m1.6.6.4.4.2.2.1.1.cmml" xref="S2.E2.m1.6.6.4.4.2.2.1.1"></plus><ci id="S2.E2.m1.6.6.4.4.2.2.1.2.cmml" xref="S2.E2.m1.6.6.4.4.2.2.1.2">𝑖</ci><cn type="integer" id="S2.E2.m1.6.6.4.4.2.2.1.3.cmml" xref="S2.E2.m1.6.6.4.4.2.2.1.3">1</cn></apply><ci id="S2.E2.m1.5.5.3.3.1.1.cmml" xref="S2.E2.m1.5.5.3.3.1.1">𝑡</ci></list></apply></apply><apply id="S2.E2.m1.12.12.10.cmml" xref="S2.E2.m1.12.12.10"><times id="S2.E2.m1.12.12.10.7.cmml" xref="S2.E2.m1.12.12.10.7"></times><apply id="S2.E2.m1.11.11.9.5.cmml" xref="S2.E2.m1.11.11.9.5"><csymbol cd="ambiguous" id="S2.E2.m1.11.11.9.5.2.cmml" xref="S2.E2.m1.11.11.9.5">subscript</csymbol><apply id="S2.E2.m1.11.11.9.5.1.2.cmml" xref="S2.E2.m1.11.11.9.5.1.1"><csymbol cd="latexml" id="S2.E2.m1.11.11.9.5.1.2.1.cmml" xref="S2.E2.m1.11.11.9.5.1.1.2">norm</csymbol><apply id="S2.E2.m1.11.11.9.5.1.1.1.cmml" xref="S2.E2.m1.11.11.9.5.1.1.1"><csymbol cd="ambiguous" id="S2.E2.m1.11.11.9.5.1.1.1.1.cmml" xref="S2.E2.m1.11.11.9.5.1.1.1">subscript</csymbol><ci id="S2.E2.m1.11.11.9.5.1.1.1.2.cmml" xref="S2.E2.m1.11.11.9.5.1.1.1.2">𝑋</ci><list id="S2.E2.m1.8.8.6.2.2.3.cmml" xref="S2.E2.m1.8.8.6.2.2.4"><ci id="S2.E2.m1.7.7.5.1.1.1.cmml" xref="S2.E2.m1.7.7.5.1.1.1">𝑖</ci><ci id="S2.E2.m1.8.8.6.2.2.2.cmml" xref="S2.E2.m1.8.8.6.2.2.2">𝑡</ci></list></apply></apply><cn type="integer" id="S2.E2.m1.11.11.9.5.3.cmml" xref="S2.E2.m1.11.11.9.5.3">2</cn></apply><apply id="S2.E2.m1.12.12.10.6.cmml" xref="S2.E2.m1.12.12.10.6"><csymbol cd="ambiguous" id="S2.E2.m1.12.12.10.6.2.cmml" xref="S2.E2.m1.12.12.10.6">subscript</csymbol><apply id="S2.E2.m1.12.12.10.6.1.2.cmml" xref="S2.E2.m1.12.12.10.6.1.1"><csymbol cd="latexml" id="S2.E2.m1.12.12.10.6.1.2.1.cmml" xref="S2.E2.m1.12.12.10.6.1.1.2">norm</csymbol><apply id="S2.E2.m1.12.12.10.6.1.1.1.cmml" xref="S2.E2.m1.12.12.10.6.1.1.1"><csymbol cd="ambiguous" id="S2.E2.m1.12.12.10.6.1.1.1.1.cmml" xref="S2.E2.m1.12.12.10.6.1.1.1">subscript</csymbol><ci id="S2.E2.m1.12.12.10.6.1.1.1.2.cmml" xref="S2.E2.m1.12.12.10.6.1.1.1.2">𝑋</ci><list id="S2.E2.m1.10.10.8.4.2.3.cmml" xref="S2.E2.m1.10.10.8.4.2.2"><apply id="S2.E2.m1.10.10.8.4.2.2.1.cmml" xref="S2.E2.m1.10.10.8.4.2.2.1"><plus id="S2.E2.m1.10.10.8.4.2.2.1.1.cmml" xref="S2.E2.m1.10.10.8.4.2.2.1.1"></plus><ci id="S2.E2.m1.10.10.8.4.2.2.1.2.cmml" xref="S2.E2.m1.10.10.8.4.2.2.1.2">𝑖</ci><cn type="integer" id="S2.E2.m1.10.10.8.4.2.2.1.3.cmml" xref="S2.E2.m1.10.10.8.4.2.2.1.3">1</cn></apply><ci id="S2.E2.m1.9.9.7.3.1.1.cmml" xref="S2.E2.m1.9.9.7.3.1.1">𝑡</ci></list></apply></apply><cn type="integer" id="S2.E2.m1.12.12.10.6.3.cmml" xref="S2.E2.m1.12.12.10.6.3">2</cn></apply></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E2.m1.13c">\displaystyle\text{BI}_{i}=1-\mathbb{E}_{X,t}\frac{X_{i,t}^{T}X_{i+1,t}}{||X_{i,t}||_{2}||X_{i+1,t}||_{2}},</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(2)</span></td>
</tr></tbody>
</table>
<p id="S2.SS2.p1.5" class="ltx_p">where <math id="S2.SS2.p1.3.m1.2" class="ltx_Math" alttext="X_{i,t}" display="inline"><semantics id="S2.SS2.p1.3.m1.2a"><msub id="S2.SS2.p1.3.m1.2.3" xref="S2.SS2.p1.3.m1.2.3.cmml"><mi id="S2.SS2.p1.3.m1.2.3.2" xref="S2.SS2.p1.3.m1.2.3.2.cmml">X</mi><mrow id="S2.SS2.p1.3.m1.2.2.2.4" xref="S2.SS2.p1.3.m1.2.2.2.3.cmml"><mi id="S2.SS2.p1.3.m1.1.1.1.1" xref="S2.SS2.p1.3.m1.1.1.1.1.cmml">i</mi><mo id="S2.SS2.p1.3.m1.2.2.2.4.1" xref="S2.SS2.p1.3.m1.2.2.2.3.cmml">,</mo><mi id="S2.SS2.p1.3.m1.2.2.2.2" xref="S2.SS2.p1.3.m1.2.2.2.2.cmml">t</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S2.SS2.p1.3.m1.2b"><apply id="S2.SS2.p1.3.m1.2.3.cmml" xref="S2.SS2.p1.3.m1.2.3"><csymbol cd="ambiguous" id="S2.SS2.p1.3.m1.2.3.1.cmml" xref="S2.SS2.p1.3.m1.2.3">subscript</csymbol><ci id="S2.SS2.p1.3.m1.2.3.2.cmml" xref="S2.SS2.p1.3.m1.2.3.2">𝑋</ci><list id="S2.SS2.p1.3.m1.2.2.2.3.cmml" xref="S2.SS2.p1.3.m1.2.2.2.4"><ci id="S2.SS2.p1.3.m1.1.1.1.1.cmml" xref="S2.SS2.p1.3.m1.1.1.1.1">𝑖</ci><ci id="S2.SS2.p1.3.m1.2.2.2.2.cmml" xref="S2.SS2.p1.3.m1.2.2.2.2">𝑡</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p1.3.m1.2c">X_{i,t}</annotation></semantics></math> means the <math id="S2.SS2.p1.4.m2.1" class="ltx_Math" alttext="t^{th}" display="inline"><semantics id="S2.SS2.p1.4.m2.1a"><msup id="S2.SS2.p1.4.m2.1.1" xref="S2.SS2.p1.4.m2.1.1.cmml"><mi id="S2.SS2.p1.4.m2.1.1.2" xref="S2.SS2.p1.4.m2.1.1.2.cmml">t</mi><mrow id="S2.SS2.p1.4.m2.1.1.3" xref="S2.SS2.p1.4.m2.1.1.3.cmml"><mi id="S2.SS2.p1.4.m2.1.1.3.2" xref="S2.SS2.p1.4.m2.1.1.3.2.cmml">t</mi><mo lspace="0em" rspace="0em" id="S2.SS2.p1.4.m2.1.1.3.1" xref="S2.SS2.p1.4.m2.1.1.3.1.cmml">​</mo><mi id="S2.SS2.p1.4.m2.1.1.3.3" xref="S2.SS2.p1.4.m2.1.1.3.3.cmml">h</mi></mrow></msup><annotation-xml encoding="MathML-Content" id="S2.SS2.p1.4.m2.1b"><apply id="S2.SS2.p1.4.m2.1.1.cmml" xref="S2.SS2.p1.4.m2.1.1"><csymbol cd="ambiguous" id="S2.SS2.p1.4.m2.1.1.1.cmml" xref="S2.SS2.p1.4.m2.1.1">superscript</csymbol><ci id="S2.SS2.p1.4.m2.1.1.2.cmml" xref="S2.SS2.p1.4.m2.1.1.2">𝑡</ci><apply id="S2.SS2.p1.4.m2.1.1.3.cmml" xref="S2.SS2.p1.4.m2.1.1.3"><times id="S2.SS2.p1.4.m2.1.1.3.1.cmml" xref="S2.SS2.p1.4.m2.1.1.3.1"></times><ci id="S2.SS2.p1.4.m2.1.1.3.2.cmml" xref="S2.SS2.p1.4.m2.1.1.3.2">𝑡</ci><ci id="S2.SS2.p1.4.m2.1.1.3.3.cmml" xref="S2.SS2.p1.4.m2.1.1.3.3">ℎ</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p1.4.m2.1c">t^{th}</annotation></semantics></math> row of <math id="S2.SS2.p1.5.m3.1" class="ltx_Math" alttext="X_{i}" display="inline"><semantics id="S2.SS2.p1.5.m3.1a"><msub id="S2.SS2.p1.5.m3.1.1" xref="S2.SS2.p1.5.m3.1.1.cmml"><mi id="S2.SS2.p1.5.m3.1.1.2" xref="S2.SS2.p1.5.m3.1.1.2.cmml">X</mi><mi id="S2.SS2.p1.5.m3.1.1.3" xref="S2.SS2.p1.5.m3.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS2.p1.5.m3.1b"><apply id="S2.SS2.p1.5.m3.1.1.cmml" xref="S2.SS2.p1.5.m3.1.1"><csymbol cd="ambiguous" id="S2.SS2.p1.5.m3.1.1.1.cmml" xref="S2.SS2.p1.5.m3.1.1">subscript</csymbol><ci id="S2.SS2.p1.5.m3.1.1.2.cmml" xref="S2.SS2.p1.5.m3.1.1.2">𝑋</ci><ci id="S2.SS2.p1.5.m3.1.1.3.cmml" xref="S2.SS2.p1.5.m3.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p1.5.m3.1c">X_{i}</annotation></semantics></math>.
Our empirical evidence supports that BI effectively reflects the importance of a layer. Figure <a href="#S4.F8" title="Figure 8 ‣ 4.4 Importance metric ‣ 4 Analysis ‣ ShortGPT: Layers in Large Language Models are More Redundant Than You Expect" class="ltx_ref"><span class="ltx_text ltx_ref_tag">8</span></a> shows these different metrics in detail.</p>
</div>
</section>
<section id="S2.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.3 </span>Layer Removal</h3>

<div id="S2.SS3.p1" class="ltx_para">
<p id="S2.SS3.p1.1" class="ltx_p">The method we propose is straightforward: layer removal, which we simply referred as ShortGPT. We delete certain layers in LLMs based on BI score. Figure <a href="#S1.F2" title="Figure 2 ‣ 1 Introduction ‣ ShortGPT: Layers in Large Language Models are More Redundant Than You Expect" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a> illustrates our pruning method. First of all, we construct a calibration set, which is a set of unlabelled text samples such as PG19 <cite class="ltx_cite ltx_citemacro_citep">(Rae et al., <a href="#bib.bib37" title="" class="ltx_ref">2019</a>)</cite>.
Then we collect the hidden states of each layer during inference on these samples. Next, we calculate the BI score based on the collected hidden states. Finally, we sort layers in ascending order according to the BI, and delete the layers with the smaller importance. The number of layers to be deleted can vary to trade off the speed and performance. The details of our layer remove setting can be found in Appendix <a href="#A2" title="Appendix B Benchmark Setup for Removed Layers ‣ ShortGPT: Layers in Large Language Models are More Redundant Than You Expect" class="ltx_ref"><span class="ltx_text ltx_ref_tag">B</span></a></p>
</div>
</section>
</section>
<section id="S3" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">3 </span>Experiments</h2>

<section id="S3.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.1 </span>Experimental Setup</h3>

<section id="S3.SS1.SSS1" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">3.1.1 </span>Models</h4>

<div id="S3.SS1.SSS1.p1" class="ltx_para">
<p id="S3.SS1.SSS1.p1.1" class="ltx_p">To validate the effectiveness of our method, we conducted experiments on existing popular open-source language models, including Llama2-7B <cite class="ltx_cite ltx_citemacro_citep">(Touvron et al., <a href="#bib.bib43" title="" class="ltx_ref">2023</a>)</cite>, Llama2-13B, Baichuan2-7B, and Baichuan2-13B. They are all large language models based on the decoder-only Transformer architecture. LLaMA 2 was trained on more than 2 trillion tokens. Baichuan-series was mainly trained in Chinese and its 13-Billion model replaced the RoPE <cite class="ltx_cite ltx_citemacro_citep">(Su et al., <a href="#bib.bib40" title="" class="ltx_ref">2024</a>)</cite> positional embedding with ALiBi <cite class="ltx_cite ltx_citemacro_citep">(Press et al., <a href="#bib.bib36" title="" class="ltx_ref">2021</a>)</cite>.</p>
</div>
</section>
<section id="S3.SS1.SSS2" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">3.1.2 </span>Benchmarks</h4>

<div id="S3.SS1.SSS2.p1" class="ltx_para">
<p id="S3.SS1.SSS2.p1.1" class="ltx_p">In order to comprehensively evaluate the changes in the ability of large language models before and after pruning, we conducted evaluations on the most commonly used Benchmark MMLU <cite class="ltx_cite ltx_citemacro_citep">(Hendrycks et al., <a href="#bib.bib21" title="" class="ltx_ref">2020</a>)</cite>, CMMLU <cite class="ltx_cite ltx_citemacro_citep">(Li et al., <a href="#bib.bib29" title="" class="ltx_ref">2024</a>)</cite> for evaluating large models. In addition, we also followed LaCo <cite class="ltx_cite ltx_citemacro_citep">(Yang et al., <a href="#bib.bib49" title="" class="ltx_ref">2024</a>)</cite> to evaluate a wider dataset.</p>
</div>
<div id="S3.SS1.SSS2.p2" class="ltx_para">
<p id="S3.SS1.SSS2.p2.1" class="ltx_p"><span id="S3.SS1.SSS2.p2.1.1" class="ltx_text ltx_font_bold">MMLU</span> <cite class="ltx_cite ltx_citemacro_citep">(Hendrycks et al., <a href="#bib.bib21" title="" class="ltx_ref">2020</a>)</cite> is a benchmark aimed at measuring the knowledge acquired during pre-training by specifically evaluating models in zero-shot and few-shot settings. This makes benchmarks more challenging and similar to the way we evaluate humans. This benchmark covers 57 subjects including STEM, humanities, social sciences, etc. Its difficulty ranges from beginner to advanced professional level, and it tests world knowledge and problem-solving ability.</p>
</div>
<div id="S3.SS1.SSS2.p3" class="ltx_para">
<p id="S3.SS1.SSS2.p3.1" class="ltx_p"><span id="S3.SS1.SSS2.p3.1.1" class="ltx_text ltx_font_bold">CMMLU</span> <cite class="ltx_cite ltx_citemacro_citep">(Li et al., <a href="#bib.bib29" title="" class="ltx_ref">2024</a>)</cite> is a comprehensive Chinese language assessment dataset designed specifically to evaluate LLM’s advanced knowledge and reasoning abilities in the context of Chinese language and culture. CMMLU covers 67 topics, from elementary school to university or professional level. Including natural sciences, as well as humanities and social sciences, it also includes many contents with Chinese characteristics.</p>
</div>
<div id="S3.SS1.SSS2.p4" class="ltx_para">
<p id="S3.SS1.SSS2.p4.1" class="ltx_p"><span id="S3.SS1.SSS2.p4.1.1" class="ltx_text ltx_font_bold">CMNLI</span> <cite class="ltx_cite ltx_citemacro_citep">(Xu et al., <a href="#bib.bib46" title="" class="ltx_ref">2020</a>)</cite> is part of the Chinese language understanding assessment benchmark. It consists of two parts: XNLI and MNLI. <span id="S3.SS1.SSS2.p4.1.2" class="ltx_text ltx_font_bold">HellaSwag (HeSw)</span> <cite class="ltx_cite ltx_citemacro_citep">(Zellers et al., <a href="#bib.bib51" title="" class="ltx_ref">2019</a>)</cite> is a challenging dataset for evaluating commonsense NLI that is especially hard for state-of-the-art models, though its questions are trivial for humans. <span id="S3.SS1.SSS2.p4.1.3" class="ltx_text ltx_font_bold">PIQA</span> <cite class="ltx_cite ltx_citemacro_citep">(Bisk et al., <a href="#bib.bib3" title="" class="ltx_ref">2020</a>)</cite> is a multi-choice question and answer dataset that focuses on daily scenarios. This dataset explores the model’s grasp of the laws of the real physical world through daily scenarios. <span id="S3.SS1.SSS2.p4.1.4" class="ltx_text ltx_font_bold">CHID</span> <cite class="ltx_cite ltx_citemacro_citep">(Zheng et al., <a href="#bib.bib53" title="" class="ltx_ref">2019</a>)</cite> is an idiom cloze test dataset that mainly focuses on the selection of candidate words and the representation of idioms. <span id="S3.SS1.SSS2.p4.1.5" class="ltx_text ltx_font_bold">CoQA</span> <cite class="ltx_cite ltx_citemacro_citep">(Reddy et al., <a href="#bib.bib38" title="" class="ltx_ref">2019</a>)</cite> is a large-scale dataset used for conversational question-answering tasks, containing over 127000 questions and their corresponding answers. <span id="S3.SS1.SSS2.p4.1.6" class="ltx_text ltx_font_bold">BoolQ</span> <cite class="ltx_cite ltx_citemacro_citep">(Clark et al., <a href="#bib.bib8" title="" class="ltx_ref">2019</a>)</cite> is a question-answer dataset containing 15942 examples of yes/no questions. These problems occur naturally - they are generated in an environment that is silent and unconstrained. <span id="S3.SS1.SSS2.p4.1.7" class="ltx_text ltx_font_bold">Race</span> <cite class="ltx_cite ltx_citemacro_citep">(Lai et al., <a href="#bib.bib27" title="" class="ltx_ref">2017</a>)</cite> is a large-scale reading comprehension dataset collected from English examinations in China, which are designed for middle school and high school students. <span id="S3.SS1.SSS2.p4.1.8" class="ltx_text ltx_font_bold">XSum<cite class="ltx_cite ltx_citemacro_citep"><span id="S3.SS1.SSS2.p4.1.8.1.1" class="ltx_text ltx_font_medium">(</span>Hasan et al.<span id="S3.SS1.SSS2.p4.1.8.2.2.1.1" class="ltx_text ltx_font_medium">, </span><a href="#bib.bib20" title="" class="ltx_ref">2021</a><span id="S3.SS1.SSS2.p4.1.8.3.3" class="ltx_text ltx_font_medium">)</span></cite></span> is used to evaluate abstract single document summarization systems. The goal is to create a short, one-sentence new summary of what the article is about. <span id="S3.SS1.SSS2.p4.1.9" class="ltx_text ltx_font_bold">C3</span> <cite class="ltx_cite ltx_citemacro_citep">(Sun et al., <a href="#bib.bib41" title="" class="ltx_ref">2020</a>)</cite> is a machine reading comprehension dataset with multiple choices, consisting of multiple-choice questions, reading materials from Chinese proficiency exams, and ethnic Chinese exams. <span id="S3.SS1.SSS2.p4.1.10" class="ltx_text ltx_font_bold">PG19</span> <cite class="ltx_cite ltx_citemacro_citep">(Rae et al., <a href="#bib.bib37" title="" class="ltx_ref">2019</a>)</cite> is a long document dataset from books used to test the effectiveness of language modeling.</p>
</div>
</section>
<section id="S3.SS1.SSS3" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">3.1.3 </span>Baselines</h4>

<div id="S3.SS1.SSS3.p1" class="ltx_para">
<p id="S3.SS1.SSS3.p1.1" class="ltx_p">To evaluate the effectiveness of our method, we compared several structured pruning methods for large language models followed by LaCo <cite class="ltx_cite ltx_citemacro_citep">(Yang et al., <a href="#bib.bib49" title="" class="ltx_ref">2024</a>)</cite>. For our method, we use PG19 for layer importance and perplexity calculation.</p>
</div>
<div id="S3.SS1.SSS3.p2" class="ltx_para">
<p id="S3.SS1.SSS3.p2.1" class="ltx_p"><span id="S3.SS1.SSS3.p2.1.1" class="ltx_text ltx_font_bold">LLMPru.</span> <cite class="ltx_cite ltx_citemacro_citep">(Ma et al., <a href="#bib.bib32" title="" class="ltx_ref">2024</a>)</cite> adopts structural pruning that selectively removes non-critical coupled structures based on gradient information, maximally preserving the majority of the LLM’s functionality. LLMPru. applies post training to the pruned model, but for fair comparison, we do not apply post training to it.</p>
</div>
<div id="S3.SS1.SSS3.p3" class="ltx_para">
<p id="S3.SS1.SSS3.p3.1" class="ltx_p"><span id="S3.SS1.SSS3.p3.1.1" class="ltx_text ltx_font_bold">SliceGPT</span> <cite class="ltx_cite ltx_citemacro_citep">(Ashkboos et al., <a href="#bib.bib1" title="" class="ltx_ref">2024</a>)</cite> is a post-training sparsification scheme that replaces each weight matrix with a smaller matrix, reducing the embedding dimension of the network. Specifically, they applied PCA to the hidden representation from shallow to deep layers, and incorporated the dimension reduction matrix into existing network parameters.</p>
</div>
<div id="S3.SS1.SSS3.p4" class="ltx_para">
<p id="S3.SS1.SSS3.p4.1" class="ltx_p"><span id="S3.SS1.SSS3.p4.1.1" class="ltx_text ltx_font_bold">LaCo</span> <cite class="ltx_cite ltx_citemacro_citep">(Yang et al., <a href="#bib.bib49" title="" class="ltx_ref">2024</a>)</cite> is a pruning method for large language models based on reducing layers. LaCo gradually merges similar layers from deep to shallow and sets a threshold to avoid continuously merging too many layers.</p>
</div>
<figure id="S3.T1" class="ltx_table">
<table id="S3.T1.1" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S3.T1.1.1.1" class="ltx_tr">
<th id="S3.T1.1.1.1.1" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_tt" style="padding:0.75pt 2.2pt;" rowspan="2"><span id="S3.T1.1.1.1.1.1" class="ltx_text" style="font-size:50%;">LLM</span></th>
<th id="S3.T1.1.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_tt" style="padding:0.75pt 2.2pt;" rowspan="2"><span id="S3.T1.1.1.1.2.1" class="ltx_text" style="font-size:50%;">Method</span></th>
<th id="S3.T1.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_tt" style="padding:0.75pt 2.2pt;" rowspan="2"><span id="S3.T1.1.1.1.3.1" class="ltx_text" style="font-size:50%;">Ratio</span></th>
<th id="S3.T1.1.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_tt" style="padding:0.75pt 2.2pt;" colspan="13"><span id="S3.T1.1.1.1.4.1" class="ltx_text" style="font-size:50%;">Benchmarks</span></th>
<th id="S3.T1.1.1.1.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:0.75pt 2.2pt;" rowspan="2"><span id="S3.T1.1.1.1.5.1" class="ltx_text" style="font-size:50%;">Ave.</span></th>
<th id="S3.T1.1.1.1.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding:0.75pt 2.2pt;" rowspan="2"><span id="S3.T1.1.1.1.6.1" class="ltx_text" style="font-size:50%;">Per.</span></th>
</tr>
<tr id="S3.T1.1.2.2" class="ltx_tr">
<th id="S3.T1.1.2.2.1" class="ltx_td ltx_align_center ltx_th ltx_th_column" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.2.2.1.1" class="ltx_text" style="font-size:50%;">CMNLI</span></th>
<th id="S3.T1.1.2.2.2" class="ltx_td ltx_align_center ltx_th ltx_th_column" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.2.2.2.1" class="ltx_text" style="font-size:50%;">HeSw</span></th>
<th id="S3.T1.1.2.2.3" class="ltx_td ltx_align_center ltx_th ltx_th_column" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.2.2.3.1" class="ltx_text" style="font-size:50%;">PIQA</span></th>
<th id="S3.T1.1.2.2.4" class="ltx_td ltx_align_center ltx_th ltx_th_column" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.2.2.4.1" class="ltx_text" style="font-size:50%;">CHID</span></th>
<th id="S3.T1.1.2.2.5" class="ltx_td ltx_align_center ltx_th ltx_th_column" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.2.2.5.1" class="ltx_text" style="font-size:50%;">WSC</span></th>
<th id="S3.T1.1.2.2.6" class="ltx_td ltx_align_center ltx_th ltx_th_column" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.2.2.6.1" class="ltx_text" style="font-size:50%;">CoQA</span></th>
<th id="S3.T1.1.2.2.7" class="ltx_td ltx_align_center ltx_th ltx_th_column" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.2.2.7.1" class="ltx_text" style="font-size:50%;">BoolQ</span></th>
<th id="S3.T1.1.2.2.8" class="ltx_td ltx_align_center ltx_th ltx_th_column" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.2.2.8.1" class="ltx_text" style="font-size:50%;">Race-H</span></th>
<th id="S3.T1.1.2.2.9" class="ltx_td ltx_align_center ltx_th ltx_th_column" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.2.2.9.1" class="ltx_text" style="font-size:50%;">Race-M</span></th>
<th id="S3.T1.1.2.2.10" class="ltx_td ltx_align_center ltx_th ltx_th_column" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.2.2.10.1" class="ltx_text" style="font-size:50%;">XSum</span></th>
<th id="S3.T1.1.2.2.11" class="ltx_td ltx_align_center ltx_th ltx_th_column" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.2.2.11.1" class="ltx_text" style="font-size:50%;">C3</span></th>
<th id="S3.T1.1.2.2.12" class="ltx_td ltx_align_center ltx_th ltx_th_column" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.2.2.12.1" class="ltx_text" style="font-size:50%;">MMLU</span></th>
<th id="S3.T1.1.2.2.13" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.2.2.13.1" class="ltx_text" style="font-size:50%;">CMMLU</span></th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S3.T1.1.3.1" class="ltx_tr">
<td id="S3.T1.1.3.1.1" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" style="padding:0.75pt 2.2pt;" rowspan="5"><span id="S3.T1.1.3.1.1.1" class="ltx_text" style="font-size:50%;">Llama2-7B</span></td>
<td id="S3.T1.1.3.1.2" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.3.1.2.1" class="ltx_text" style="font-size:50%;">Dense</span></td>
<td id="S3.T1.1.3.1.3" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.3.1.3.1" class="ltx_text" style="font-size:50%;">0.00%</span></td>
<td id="S3.T1.1.3.1.4" class="ltx_td ltx_align_center ltx_border_t" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.3.1.4.1" class="ltx_text" style="font-size:50%;">32.99</span></td>
<td id="S3.T1.1.3.1.5" class="ltx_td ltx_align_center ltx_border_t" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.3.1.5.1" class="ltx_text" style="font-size:50%;">71.26</span></td>
<td id="S3.T1.1.3.1.6" class="ltx_td ltx_align_center ltx_border_t" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.3.1.6.1" class="ltx_text" style="font-size:50%;">77.91</span></td>
<td id="S3.T1.1.3.1.7" class="ltx_td ltx_align_center ltx_border_t" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.3.1.7.1" class="ltx_text" style="font-size:50%;">41.66</span></td>
<td id="S3.T1.1.3.1.8" class="ltx_td ltx_align_center ltx_border_t" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.3.1.8.1" class="ltx_text" style="font-size:50%;">50.00</span></td>
<td id="S3.T1.1.3.1.9" class="ltx_td ltx_align_center ltx_border_t" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.3.1.9.1" class="ltx_text" style="font-size:50%;">64.62</span></td>
<td id="S3.T1.1.3.1.10" class="ltx_td ltx_align_center ltx_border_t" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.3.1.10.1" class="ltx_text" style="font-size:50%;">71.62</span></td>
<td id="S3.T1.1.3.1.11" class="ltx_td ltx_align_center ltx_border_t" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.3.1.11.1" class="ltx_text" style="font-size:50%;">35.71</span></td>
<td id="S3.T1.1.3.1.12" class="ltx_td ltx_align_center ltx_border_t" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.3.1.12.1" class="ltx_text" style="font-size:50%;">34.19</span></td>
<td id="S3.T1.1.3.1.13" class="ltx_td ltx_align_center ltx_border_t" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.3.1.13.1" class="ltx_text" style="font-size:50%;">19.40</span></td>
<td id="S3.T1.1.3.1.14" class="ltx_td ltx_align_center ltx_border_t" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.3.1.14.1" class="ltx_text" style="font-size:50%;">43.56</span></td>
<td id="S3.T1.1.3.1.15" class="ltx_td ltx_align_center ltx_border_t" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.3.1.15.1" class="ltx_text" style="font-size:50%;">45.39</span></td>
<td id="S3.T1.1.3.1.16" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.3.1.16.1" class="ltx_text" style="font-size:50%;">32.92</span></td>
<td id="S3.T1.1.3.1.17" class="ltx_td ltx_align_center ltx_border_t" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.3.1.17.1" class="ltx_text" style="font-size:50%;">44.52</span></td>
<td id="S3.T1.1.3.1.18" class="ltx_td ltx_align_center ltx_border_t" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.3.1.18.1" class="ltx_text" style="font-size:50%;">100.00</span></td>
</tr>
<tr id="S3.T1.1.4.2" class="ltx_tr">
<td id="S3.T1.1.4.2.1" class="ltx_td ltx_align_center ltx_border_r" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.4.2.1.1" class="ltx_text" style="font-size:50%;">LLMPrun.</span></td>
<td id="S3.T1.1.4.2.2" class="ltx_td ltx_align_center ltx_border_r" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.4.2.2.1" class="ltx_text" style="font-size:50%;">27.0%</span></td>
<td id="S3.T1.1.4.2.3" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.4.2.3.1" class="ltx_text ltx_font_bold" style="font-size:50%;">34.33</span></td>
<td id="S3.T1.1.4.2.4" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.4.2.4.1" class="ltx_text ltx_font_bold" style="font-size:50%;">56.46</span></td>
<td id="S3.T1.1.4.2.5" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.4.2.5.1" class="ltx_text ltx_font_bold" style="font-size:50%;">71.22</span></td>
<td id="S3.T1.1.4.2.6" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.4.2.6.1" class="ltx_text" style="font-size:50%;">25.25</span></td>
<td id="S3.T1.1.4.2.7" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.4.2.7.1" class="ltx_text" style="font-size:50%;">36.54</span></td>
<td id="S3.T1.1.4.2.8" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.4.2.8.1" class="ltx_text" style="font-size:50%;">42.51</span></td>
<td id="S3.T1.1.4.2.9" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.4.2.9.1" class="ltx_text" style="font-size:50%;">55.20</span></td>
<td id="S3.T1.1.4.2.10" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.4.2.10.1" class="ltx_text" style="font-size:50%;">22.56</span></td>
<td id="S3.T1.1.4.2.11" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.4.2.11.1" class="ltx_text" style="font-size:50%;">22.35</span></td>
<td id="S3.T1.1.4.2.12" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.4.2.12.1" class="ltx_text" style="font-size:50%;">11.51</span></td>
<td id="S3.T1.1.4.2.13" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.4.2.13.1" class="ltx_text" style="font-size:50%;">25.64</span></td>
<td id="S3.T1.1.4.2.14" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.4.2.14.1" class="ltx_text" style="font-size:50%;">23.33</span></td>
<td id="S3.T1.1.4.2.15" class="ltx_td ltx_align_center ltx_border_r" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.4.2.15.1" class="ltx_text" style="font-size:50%;">25.25</span></td>
<td id="S3.T1.1.4.2.16" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.4.2.16.1" class="ltx_text" style="font-size:50%;">32.84</span></td>
<td id="S3.T1.1.4.2.17" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.4.2.17.1" class="ltx_text" style="font-size:50%;">73.76</span></td>
</tr>
<tr id="S3.T1.1.5.3" class="ltx_tr">
<td id="S3.T1.1.5.3.1" class="ltx_td ltx_align_center ltx_border_r" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.5.3.1.1" class="ltx_text" style="font-size:50%;">SliceGPT</span></td>
<td id="S3.T1.1.5.3.2" class="ltx_td ltx_align_center ltx_border_r" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.5.3.2.1" class="ltx_text" style="font-size:50%;">26.4%</span></td>
<td id="S3.T1.1.5.3.3" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.5.3.3.1" class="ltx_text" style="font-size:50%;">31.70</span></td>
<td id="S3.T1.1.5.3.4" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.5.3.4.1" class="ltx_text" style="font-size:50%;">50.27</span></td>
<td id="S3.T1.1.5.3.5" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.5.3.5.1" class="ltx_text" style="font-size:50%;">66.21</span></td>
<td id="S3.T1.1.5.3.6" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.5.3.6.1" class="ltx_text" style="font-size:50%;">20.79</span></td>
<td id="S3.T1.1.5.3.7" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.5.3.7.1" class="ltx_text" style="font-size:50%;">36.54</span></td>
<td id="S3.T1.1.5.3.8" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.5.3.8.1" class="ltx_text" style="font-size:50%;">41.36</span></td>
<td id="S3.T1.1.5.3.9" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.5.3.9.1" class="ltx_text" style="font-size:50%;">38.32</span></td>
<td id="S3.T1.1.5.3.10" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.5.3.10.1" class="ltx_text" style="font-size:50%;">21.07</span></td>
<td id="S3.T1.1.5.3.11" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.5.3.11.1" class="ltx_text" style="font-size:50%;">21.66</span></td>
<td id="S3.T1.1.5.3.12" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.5.3.12.1" class="ltx_text" style="font-size:50%;">4.89</span></td>
<td id="S3.T1.1.5.3.13" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.5.3.13.1" class="ltx_text ltx_font_bold" style="font-size:50%;">39.78</span></td>
<td id="S3.T1.1.5.3.14" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.5.3.14.1" class="ltx_text" style="font-size:50%;">28.92</span></td>
<td id="S3.T1.1.5.3.15" class="ltx_td ltx_align_center ltx_border_r" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.5.3.15.1" class="ltx_text" style="font-size:50%;">25.37</span></td>
<td id="S3.T1.1.5.3.16" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.5.3.16.1" class="ltx_text" style="font-size:50%;">32.84</span></td>
<td id="S3.T1.1.5.3.17" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.5.3.17.1" class="ltx_text" style="font-size:50%;">73.76</span></td>
</tr>
<tr id="S3.T1.1.6.4" class="ltx_tr">
<td id="S3.T1.1.6.4.1" class="ltx_td ltx_align_center ltx_border_r" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.6.4.1.1" class="ltx_text" style="font-size:50%;">LaCo</span></td>
<td id="S3.T1.1.6.4.2" class="ltx_td ltx_align_center ltx_border_r" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.6.4.2.1" class="ltx_text" style="font-size:50%;">27.1%</span></td>
<td id="S3.T1.1.6.4.3" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.6.4.3.1" class="ltx_text" style="font-size:50%;">34.43</span></td>
<td id="S3.T1.1.6.4.4" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.6.4.4.1" class="ltx_text" style="font-size:50%;">55.69</span></td>
<td id="S3.T1.1.6.4.5" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.6.4.5.1" class="ltx_text" style="font-size:50%;">69.80</span></td>
<td id="S3.T1.1.6.4.6" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.6.4.6.1" class="ltx_text ltx_font_bold" style="font-size:50%;">36.14</span></td>
<td id="S3.T1.1.6.4.7" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.6.4.7.1" class="ltx_text" style="font-size:50%;">40.38</span></td>
<td id="S3.T1.1.6.4.8" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.6.4.8.1" class="ltx_text" style="font-size:50%;">45.70</span></td>
<td id="S3.T1.1.6.4.9" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.6.4.9.1" class="ltx_text" style="font-size:50%;">64.07</span></td>
<td id="S3.T1.1.6.4.10" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.6.4.10.1" class="ltx_text" style="font-size:50%;">22.61</span></td>
<td id="S3.T1.1.6.4.11" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.6.4.11.1" class="ltx_text" style="font-size:50%;">23.61</span></td>
<td id="S3.T1.1.6.4.12" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.6.4.12.1" class="ltx_text ltx_font_bold" style="font-size:50%;">15.64</span></td>
<td id="S3.T1.1.6.4.13" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.6.4.13.1" class="ltx_text" style="font-size:50%;">39.67</span></td>
<td id="S3.T1.1.6.4.14" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.6.4.14.1" class="ltx_text" style="font-size:50%;">26.45</span></td>
<td id="S3.T1.1.6.4.15" class="ltx_td ltx_align_center ltx_border_r" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.6.4.15.1" class="ltx_text" style="font-size:50%;">25.24</span></td>
<td id="S3.T1.1.6.4.16" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.6.4.16.1" class="ltx_text" style="font-size:50%;">38.41</span></td>
<td id="S3.T1.1.6.4.17" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.6.4.17.1" class="ltx_text" style="font-size:50%;">86.28</span></td>
</tr>
<tr id="S3.T1.1.7.5" class="ltx_tr">
<td id="S3.T1.1.7.5.1" class="ltx_td ltx_align_center ltx_border_r" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.7.5.1.1" class="ltx_text" style="font-size:50%;">ShortGPT</span></td>
<td id="S3.T1.1.7.5.2" class="ltx_td ltx_align_center ltx_border_r" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.7.5.2.1" class="ltx_text" style="font-size:50%;">27.1%</span></td>
<td id="S3.T1.1.7.5.3" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.7.5.3.1" class="ltx_text" style="font-size:50%;">32.95</span></td>
<td id="S3.T1.1.7.5.4" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.7.5.4.1" class="ltx_text" style="font-size:50%;">53.02</span></td>
<td id="S3.T1.1.7.5.5" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.7.5.5.1" class="ltx_text" style="font-size:50%;">66.43</span></td>
<td id="S3.T1.1.7.5.6" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.7.5.6.1" class="ltx_text" style="font-size:50%;">24.68</span></td>
<td id="S3.T1.1.7.5.7" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.7.5.7.1" class="ltx_text ltx_font_bold" style="font-size:50%;">52.46</span></td>
<td id="S3.T1.1.7.5.8" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.7.5.8.1" class="ltx_text ltx_font_bold" style="font-size:50%;">47.99</span></td>
<td id="S3.T1.1.7.5.9" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.7.5.9.1" class="ltx_text ltx_font_bold" style="font-size:50%;">74.71</span></td>
<td id="S3.T1.1.7.5.10" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.7.5.10.1" class="ltx_text ltx_font_bold" style="font-size:50%;">32.25</span></td>
<td id="S3.T1.1.7.5.11" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.7.5.11.1" class="ltx_text ltx_font_bold" style="font-size:50%;">35.17</span></td>
<td id="S3.T1.1.7.5.12" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.7.5.12.1" class="ltx_text" style="font-size:50%;">0.67</span></td>
<td id="S3.T1.1.7.5.13" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.7.5.13.1" class="ltx_text" style="font-size:50%;">39.62</span></td>
<td id="S3.T1.1.7.5.14" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.7.5.14.1" class="ltx_text ltx_font_bold" style="font-size:50%;">43.96</span></td>
<td id="S3.T1.1.7.5.15" class="ltx_td ltx_align_center ltx_border_r" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.7.5.15.1" class="ltx_text ltx_font_bold" style="font-size:50%;">32.25</span></td>
<td id="S3.T1.1.7.5.16" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.7.5.16.1" class="ltx_text ltx_font_bold" style="font-size:50%;">42.60</span></td>
<td id="S3.T1.1.7.5.17" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.7.5.17.1" class="ltx_text ltx_font_bold" style="font-size:50%;">95.69</span></td>
</tr>
<tr id="S3.T1.1.8.6" class="ltx_tr">
<td id="S3.T1.1.8.6.1" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" style="padding:0.75pt 2.2pt;" rowspan="5"><span id="S3.T1.1.8.6.1.1" class="ltx_text" style="font-size:50%;">Llama2-13B</span></td>
<td id="S3.T1.1.8.6.2" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.8.6.2.1" class="ltx_text" style="font-size:50%;">Dense</span></td>
<td id="S3.T1.1.8.6.3" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.8.6.3.1" class="ltx_text" style="font-size:50%;">0.00%</span></td>
<td id="S3.T1.1.8.6.4" class="ltx_td ltx_align_center ltx_border_t" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.8.6.4.1" class="ltx_text" style="font-size:50%;">32.99</span></td>
<td id="S3.T1.1.8.6.5" class="ltx_td ltx_align_center ltx_border_t" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.8.6.5.1" class="ltx_text" style="font-size:50%;">74.78</span></td>
<td id="S3.T1.1.8.6.6" class="ltx_td ltx_align_center ltx_border_t" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.8.6.6.1" class="ltx_text" style="font-size:50%;">79.71</span></td>
<td id="S3.T1.1.8.6.7" class="ltx_td ltx_align_center ltx_border_t" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.8.6.7.1" class="ltx_text" style="font-size:50%;">47.35</span></td>
<td id="S3.T1.1.8.6.8" class="ltx_td ltx_align_center ltx_border_t" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.8.6.8.1" class="ltx_text" style="font-size:50%;">50.00</span></td>
<td id="S3.T1.1.8.6.9" class="ltx_td ltx_align_center ltx_border_t" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.8.6.9.1" class="ltx_text" style="font-size:50%;">66.91</span></td>
<td id="S3.T1.1.8.6.10" class="ltx_td ltx_align_center ltx_border_t" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.8.6.10.1" class="ltx_text" style="font-size:50%;">82.39</span></td>
<td id="S3.T1.1.8.6.11" class="ltx_td ltx_align_center ltx_border_t" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.8.6.11.1" class="ltx_text" style="font-size:50%;">57.95</span></td>
<td id="S3.T1.1.8.6.12" class="ltx_td ltx_align_center ltx_border_t" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.8.6.12.1" class="ltx_text" style="font-size:50%;">60.38</span></td>
<td id="S3.T1.1.8.6.13" class="ltx_td ltx_align_center ltx_border_t" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.8.6.13.1" class="ltx_text" style="font-size:50%;">23.45</span></td>
<td id="S3.T1.1.8.6.14" class="ltx_td ltx_align_center ltx_border_t" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.8.6.14.1" class="ltx_text" style="font-size:50%;">47.51</span></td>
<td id="S3.T1.1.8.6.15" class="ltx_td ltx_align_center ltx_border_t" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.8.6.15.1" class="ltx_text" style="font-size:50%;">55.00</span></td>
<td id="S3.T1.1.8.6.16" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.8.6.16.1" class="ltx_text" style="font-size:50%;">38.40</span></td>
<td id="S3.T1.1.8.6.17" class="ltx_td ltx_align_center ltx_border_t" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.8.6.17.1" class="ltx_text" style="font-size:50%;">51.91</span></td>
<td id="S3.T1.1.8.6.18" class="ltx_td ltx_align_center ltx_border_t" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.8.6.18.1" class="ltx_text" style="font-size:50%;">100.00</span></td>
</tr>
<tr id="S3.T1.1.9.7" class="ltx_tr">
<td id="S3.T1.1.9.7.1" class="ltx_td ltx_align_center ltx_border_r" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.9.7.1.1" class="ltx_text" style="font-size:50%;">LLMPrun.</span></td>
<td id="S3.T1.1.9.7.2" class="ltx_td ltx_align_center ltx_border_r" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.9.7.2.1" class="ltx_text" style="font-size:50%;">24.4%</span></td>
<td id="S3.T1.1.9.7.3" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.9.7.3.1" class="ltx_text ltx_font_bold" style="font-size:50%;">33.03</span></td>
<td id="S3.T1.1.9.7.4" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.9.7.4.1" class="ltx_text ltx_font_bold" style="font-size:50%;">67.76</span></td>
<td id="S3.T1.1.9.7.5" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.9.7.5.1" class="ltx_text ltx_font_bold" style="font-size:50%;">76.66</span></td>
<td id="S3.T1.1.9.7.6" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.9.7.6.1" class="ltx_text" style="font-size:50%;">35.64</span></td>
<td id="S3.T1.1.9.7.7" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.9.7.7.1" class="ltx_text" style="font-size:50%;">40.38</span></td>
<td id="S3.T1.1.9.7.8" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.9.7.8.1" class="ltx_text" style="font-size:50%;">50.86</span></td>
<td id="S3.T1.1.9.7.9" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.9.7.9.1" class="ltx_text" style="font-size:50%;">56.42</span></td>
<td id="S3.T1.1.9.7.10" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.9.7.10.1" class="ltx_text" style="font-size:50%;">22.47</span></td>
<td id="S3.T1.1.9.7.11" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.9.7.11.1" class="ltx_text" style="font-size:50%;">22.08</span></td>
<td id="S3.T1.1.9.7.12" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.9.7.12.1" class="ltx_text ltx_font_bold" style="font-size:50%;">19.17</span></td>
<td id="S3.T1.1.9.7.13" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.9.7.13.1" class="ltx_text" style="font-size:50%;">32.33</span></td>
<td id="S3.T1.1.9.7.14" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.9.7.14.1" class="ltx_text" style="font-size:50%;">25.21</span></td>
<td id="S3.T1.1.9.7.15" class="ltx_td ltx_align_center ltx_border_r" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.9.7.15.1" class="ltx_text" style="font-size:50%;">24.71</span></td>
<td id="S3.T1.1.9.7.16" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.9.7.16.1" class="ltx_text" style="font-size:50%;">38.97</span></td>
<td id="S3.T1.1.9.7.17" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.9.7.17.1" class="ltx_text" style="font-size:50%;">75.07</span></td>
</tr>
<tr id="S3.T1.1.10.8" class="ltx_tr">
<td id="S3.T1.1.10.8.1" class="ltx_td ltx_align_center ltx_border_r" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.10.8.1.1" class="ltx_text" style="font-size:50%;">SliceGPT</span></td>
<td id="S3.T1.1.10.8.2" class="ltx_td ltx_align_center ltx_border_r" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.10.8.2.1" class="ltx_text" style="font-size:50%;">23.6%</span></td>
<td id="S3.T1.1.10.8.3" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.10.8.3.1" class="ltx_text" style="font-size:50%;">29.82</span></td>
<td id="S3.T1.1.10.8.4" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.10.8.4.1" class="ltx_text" style="font-size:50%;">55.71</span></td>
<td id="S3.T1.1.10.8.5" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.10.8.5.1" class="ltx_text" style="font-size:50%;">69.04</span></td>
<td id="S3.T1.1.10.8.6" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.10.8.6.1" class="ltx_text" style="font-size:50%;">19.31</span></td>
<td id="S3.T1.1.10.8.7" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.10.8.7.1" class="ltx_text" style="font-size:50%;">36.54</span></td>
<td id="S3.T1.1.10.8.8" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.10.8.8.1" class="ltx_text" style="font-size:50%;">47.26</span></td>
<td id="S3.T1.1.10.8.9" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.10.8.9.1" class="ltx_text" style="font-size:50%;">37.86</span></td>
<td id="S3.T1.1.10.8.10" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.10.8.10.1" class="ltx_text" style="font-size:50%;">23.41</span></td>
<td id="S3.T1.1.10.8.11" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.10.8.11.1" class="ltx_text" style="font-size:50%;">24.03</span></td>
<td id="S3.T1.1.10.8.12" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.10.8.12.1" class="ltx_text" style="font-size:50%;">5.27</span></td>
<td id="S3.T1.1.10.8.13" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.10.8.13.1" class="ltx_text" style="font-size:50%;">41.92</span></td>
<td id="S3.T1.1.10.8.14" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.10.8.14.1" class="ltx_text" style="font-size:50%;">37.14</span></td>
<td id="S3.T1.1.10.8.15" class="ltx_td ltx_align_center ltx_border_r" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.10.8.15.1" class="ltx_text" style="font-size:50%;">25.79</span></td>
<td id="S3.T1.1.10.8.16" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.10.8.16.1" class="ltx_text" style="font-size:50%;">34.84</span></td>
<td id="S3.T1.1.10.8.17" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.10.8.17.1" class="ltx_text" style="font-size:50%;">67.11</span></td>
</tr>
<tr id="S3.T1.1.11.9" class="ltx_tr">
<td id="S3.T1.1.11.9.1" class="ltx_td ltx_align_center ltx_border_r" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.11.9.1.1" class="ltx_text" style="font-size:50%;">LaCo</span></td>
<td id="S3.T1.1.11.9.2" class="ltx_td ltx_align_center ltx_border_r" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.11.9.2.1" class="ltx_text" style="font-size:50%;">24.6%</span></td>
<td id="S3.T1.1.11.9.3" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.11.9.3.1" class="ltx_text" style="font-size:50%;">32.86</span></td>
<td id="S3.T1.1.11.9.4" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.11.9.4.1" class="ltx_text" style="font-size:50%;">64.39</span></td>
<td id="S3.T1.1.11.9.5" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.11.9.5.1" class="ltx_text" style="font-size:50%;">74.27</span></td>
<td id="S3.T1.1.11.9.6" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.11.9.6.1" class="ltx_text ltx_font_bold" style="font-size:50%;">40.10</span></td>
<td id="S3.T1.1.11.9.7" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.11.9.7.1" class="ltx_text ltx_font_bold" style="font-size:50%;">52.88</span></td>
<td id="S3.T1.1.11.9.8" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.11.9.8.1" class="ltx_text" style="font-size:50%;">52.66</span></td>
<td id="S3.T1.1.11.9.9" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.11.9.9.1" class="ltx_text ltx_font_bold" style="font-size:50%;">63.98</span></td>
<td id="S3.T1.1.11.9.10" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.11.9.10.1" class="ltx_text" style="font-size:50%;">54.49</span></td>
<td id="S3.T1.1.11.9.11" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.11.9.11.1" class="ltx_text" style="font-size:50%;">56.55</span></td>
<td id="S3.T1.1.11.9.12" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.11.9.12.1" class="ltx_text" style="font-size:50%;">14.45</span></td>
<td id="S3.T1.1.11.9.13" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.11.9.13.1" class="ltx_text" style="font-size:50%;">44.93</span></td>
<td id="S3.T1.1.11.9.14" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.11.9.14.1" class="ltx_text" style="font-size:50%;">45.93</span></td>
<td id="S3.T1.1.11.9.15" class="ltx_td ltx_align_center ltx_border_r" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.11.9.15.1" class="ltx_text" style="font-size:50%;">32.62</span></td>
<td id="S3.T1.1.11.9.16" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.11.9.16.1" class="ltx_text" style="font-size:50%;">48.30</span></td>
<td id="S3.T1.1.11.9.17" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.11.9.17.1" class="ltx_text" style="font-size:50%;">93.05</span></td>
</tr>
<tr id="S3.T1.1.12.10" class="ltx_tr">
<td id="S3.T1.1.12.10.1" class="ltx_td ltx_align_center ltx_border_r" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.12.10.1.1" class="ltx_text" style="font-size:50%;">ShortGPT</span></td>
<td id="S3.T1.1.12.10.2" class="ltx_td ltx_align_center ltx_border_r" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.12.10.2.1" class="ltx_text" style="font-size:50%;">24.6%</span></td>
<td id="S3.T1.1.12.10.3" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.12.10.3.1" class="ltx_text" style="font-size:50%;">33.00</span></td>
<td id="S3.T1.1.12.10.4" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.12.10.4.1" class="ltx_text" style="font-size:50%;">66.64</span></td>
<td id="S3.T1.1.12.10.5" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.12.10.5.1" class="ltx_text" style="font-size:50%;">73.45</span></td>
<td id="S3.T1.1.12.10.6" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.12.10.6.1" class="ltx_text" style="font-size:50%;">36.61</span></td>
<td id="S3.T1.1.12.10.7" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.12.10.7.1" class="ltx_text" style="font-size:50%;">50.00</span></td>
<td id="S3.T1.1.12.10.8" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.12.10.8.1" class="ltx_text ltx_font_bold" style="font-size:50%;">58.64</span></td>
<td id="S3.T1.1.12.10.9" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.12.10.9.1" class="ltx_text" style="font-size:50%;">62.48</span></td>
<td id="S3.T1.1.12.10.10" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.12.10.10.1" class="ltx_text ltx_font_bold" style="font-size:50%;">58.35</span></td>
<td id="S3.T1.1.12.10.11" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.12.10.11.1" class="ltx_text ltx_font_bold" style="font-size:50%;">60.17</span></td>
<td id="S3.T1.1.12.10.12" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.12.10.12.1" class="ltx_text" style="font-size:50%;">17.59</span></td>
<td id="S3.T1.1.12.10.13" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.12.10.13.1" class="ltx_text ltx_font_bold" style="font-size:50%;">46.90</span></td>
<td id="S3.T1.1.12.10.14" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.12.10.14.1" class="ltx_text ltx_font_bold" style="font-size:50%;">54.69</span></td>
<td id="S3.T1.1.12.10.15" class="ltx_td ltx_align_center ltx_border_r" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.12.10.15.1" class="ltx_text ltx_font_bold" style="font-size:50%;">38.38</span></td>
<td id="S3.T1.1.12.10.16" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.12.10.16.1" class="ltx_text ltx_font_bold" style="font-size:50%;">50.53</span></td>
<td id="S3.T1.1.12.10.17" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.12.10.17.1" class="ltx_text ltx_font_bold" style="font-size:50%;">97.34</span></td>
</tr>
<tr id="S3.T1.1.13.11" class="ltx_tr">
<td id="S3.T1.1.13.11.1" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" style="padding:0.75pt 2.2pt;" rowspan="5"><span id="S3.T1.1.13.11.1.1" class="ltx_text" style="font-size:50%;">Baichuan2-7B</span></td>
<td id="S3.T1.1.13.11.2" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.13.11.2.1" class="ltx_text" style="font-size:50%;">Dense</span></td>
<td id="S3.T1.1.13.11.3" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.13.11.3.1" class="ltx_text" style="font-size:50%;">0.00%</span></td>
<td id="S3.T1.1.13.11.4" class="ltx_td ltx_align_center ltx_border_t" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.13.11.4.1" class="ltx_text" style="font-size:50%;">33.37</span></td>
<td id="S3.T1.1.13.11.5" class="ltx_td ltx_align_center ltx_border_t" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.13.11.5.1" class="ltx_text" style="font-size:50%;">67.56</span></td>
<td id="S3.T1.1.13.11.6" class="ltx_td ltx_align_center ltx_border_t" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.13.11.6.1" class="ltx_text" style="font-size:50%;">76.17</span></td>
<td id="S3.T1.1.13.11.7" class="ltx_td ltx_align_center ltx_border_t" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.13.11.7.1" class="ltx_text" style="font-size:50%;">85.56</span></td>
<td id="S3.T1.1.13.11.8" class="ltx_td ltx_align_center ltx_border_t" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.13.11.8.1" class="ltx_text" style="font-size:50%;">50.00</span></td>
<td id="S3.T1.1.13.11.9" class="ltx_td ltx_align_center ltx_border_t" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.13.11.9.1" class="ltx_text" style="font-size:50%;">63.14</span></td>
<td id="S3.T1.1.13.11.10" class="ltx_td ltx_align_center ltx_border_t" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.13.11.10.1" class="ltx_text" style="font-size:50%;">74.10</span></td>
<td id="S3.T1.1.13.11.11" class="ltx_td ltx_align_center ltx_border_t" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.13.11.11.1" class="ltx_text" style="font-size:50%;">26.96</span></td>
<td id="S3.T1.1.13.11.12" class="ltx_td ltx_align_center ltx_border_t" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.13.11.12.1" class="ltx_text" style="font-size:50%;">24.09</span></td>
<td id="S3.T1.1.13.11.13" class="ltx_td ltx_align_center ltx_border_t" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.13.11.13.1" class="ltx_text" style="font-size:50%;">20.82</span></td>
<td id="S3.T1.1.13.11.14" class="ltx_td ltx_align_center ltx_border_t" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.13.11.14.1" class="ltx_text" style="font-size:50%;">64.55</span></td>
<td id="S3.T1.1.13.11.15" class="ltx_td ltx_align_center ltx_border_t" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.13.11.15.1" class="ltx_text" style="font-size:50%;">53.87</span></td>
<td id="S3.T1.1.13.11.16" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.13.11.16.1" class="ltx_text" style="font-size:50%;">56.95</span></td>
<td id="S3.T1.1.13.11.17" class="ltx_td ltx_align_center ltx_border_t" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.13.11.17.1" class="ltx_text" style="font-size:50%;">53.63</span></td>
<td id="S3.T1.1.13.11.18" class="ltx_td ltx_align_center ltx_border_t" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.13.11.18.1" class="ltx_text" style="font-size:50%;">100.00</span></td>
</tr>
<tr id="S3.T1.1.14.12" class="ltx_tr">
<td id="S3.T1.1.14.12.1" class="ltx_td ltx_align_center ltx_border_r" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.14.12.1.1" class="ltx_text" style="font-size:50%;">LLMPrun.</span></td>
<td id="S3.T1.1.14.12.2" class="ltx_td ltx_align_center ltx_border_r" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.14.12.2.1" class="ltx_text" style="font-size:50%;">24.2%</span></td>
<td id="S3.T1.1.14.12.3" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.14.12.3.1" class="ltx_text" style="font-size:50%;">32.28</span></td>
<td id="S3.T1.1.14.12.4" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.14.12.4.1" class="ltx_text" style="font-size:50%;">53.66</span></td>
<td id="S3.T1.1.14.12.5" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.14.12.5.1" class="ltx_text ltx_font_bold" style="font-size:50%;">71.82</span></td>
<td id="S3.T1.1.14.12.6" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.14.12.6.1" class="ltx_text" style="font-size:50%;">69.80</span></td>
<td id="S3.T1.1.14.12.7" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.14.12.7.1" class="ltx_text ltx_font_bold" style="font-size:50%;">53.85</span></td>
<td id="S3.T1.1.14.12.8" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.14.12.8.1" class="ltx_text ltx_font_bold" style="font-size:50%;">47.83</span></td>
<td id="S3.T1.1.14.12.9" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.14.12.9.1" class="ltx_text" style="font-size:50%;">61.19</span></td>
<td id="S3.T1.1.14.12.10" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.14.12.10.1" class="ltx_text" style="font-size:50%;">21.96</span></td>
<td id="S3.T1.1.14.12.11" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.14.12.11.1" class="ltx_text" style="font-size:50%;">22.28</span></td>
<td id="S3.T1.1.14.12.12" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.14.12.12.1" class="ltx_text ltx_font_bold" style="font-size:50%;">15.98</span></td>
<td id="S3.T1.1.14.12.13" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.14.12.13.1" class="ltx_text" style="font-size:50%;">41.64</span></td>
<td id="S3.T1.1.14.12.14" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.14.12.14.1" class="ltx_text" style="font-size:50%;">24.93</span></td>
<td id="S3.T1.1.14.12.15" class="ltx_td ltx_align_center ltx_border_r" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.14.12.15.1" class="ltx_text" style="font-size:50%;">25.69</span></td>
<td id="S3.T1.1.14.12.16" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.14.12.16.1" class="ltx_text" style="font-size:50%;">41.76</span></td>
<td id="S3.T1.1.14.12.17" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.14.12.17.1" class="ltx_text" style="font-size:50%;">77.87</span></td>
</tr>
<tr id="S3.T1.1.15.13" class="ltx_tr">
<td id="S3.T1.1.15.13.1" class="ltx_td ltx_align_center ltx_border_r" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.15.13.1.1" class="ltx_text" style="font-size:50%;">SliceGPT</span></td>
<td id="S3.T1.1.15.13.2" class="ltx_td ltx_align_center ltx_border_r" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.15.13.2.1" class="ltx_text" style="font-size:50%;">22.2%</span></td>
<td id="S3.T1.1.15.13.3" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.15.13.3.1" class="ltx_text" style="font-size:50%;">32.07</span></td>
<td id="S3.T1.1.15.13.4" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.15.13.4.1" class="ltx_text" style="font-size:50%;">25.29</span></td>
<td id="S3.T1.1.15.13.5" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.15.13.5.1" class="ltx_text" style="font-size:50%;">50.33</span></td>
<td id="S3.T1.1.15.13.6" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.15.13.6.1" class="ltx_text" style="font-size:50%;">14.85</span></td>
<td id="S3.T1.1.15.13.7" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.15.13.7.1" class="ltx_text" style="font-size:50%;">36.54</span></td>
<td id="S3.T1.1.15.13.8" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.15.13.8.1" class="ltx_text" style="font-size:50%;">19.57</span></td>
<td id="S3.T1.1.15.13.9" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.15.13.9.1" class="ltx_text" style="font-size:50%;">39.30</span></td>
<td id="S3.T1.1.15.13.10" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.15.13.10.1" class="ltx_text" style="font-size:50%;">23.53</span></td>
<td id="S3.T1.1.15.13.11" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.15.13.11.1" class="ltx_text" style="font-size:50%;">22.49</span></td>
<td id="S3.T1.1.15.13.12" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.15.13.12.1" class="ltx_text" style="font-size:50%;">0.00</span></td>
<td id="S3.T1.1.15.13.13" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.15.13.13.1" class="ltx_text" style="font-size:50%;">26.58</span></td>
<td id="S3.T1.1.15.13.14" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.15.13.14.1" class="ltx_text" style="font-size:50%;">25.18</span></td>
<td id="S3.T1.1.15.13.15" class="ltx_td ltx_align_center ltx_border_r" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.15.13.15.1" class="ltx_text" style="font-size:50%;">25.25</span></td>
<td id="S3.T1.1.15.13.16" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.15.13.16.1" class="ltx_text" style="font-size:50%;">26.23</span></td>
<td id="S3.T1.1.15.13.17" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.15.13.17.1" class="ltx_text" style="font-size:50%;">56.38</span></td>
</tr>
<tr id="S3.T1.1.16.14" class="ltx_tr">
<td id="S3.T1.1.16.14.1" class="ltx_td ltx_align_center ltx_border_r" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.16.14.1.1" class="ltx_text" style="font-size:50%;">LaCo</span></td>
<td id="S3.T1.1.16.14.2" class="ltx_td ltx_align_center ltx_border_r" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.16.14.2.1" class="ltx_text" style="font-size:50%;">24.2%</span></td>
<td id="S3.T1.1.16.14.3" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.16.14.3.1" class="ltx_text" style="font-size:50%;">33.00</span></td>
<td id="S3.T1.1.16.14.4" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.16.14.4.1" class="ltx_text" style="font-size:50%;">52.28</span></td>
<td id="S3.T1.1.16.14.5" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.16.14.5.1" class="ltx_text" style="font-size:50%;">68.50</span></td>
<td id="S3.T1.1.16.14.6" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.16.14.6.1" class="ltx_text ltx_font_bold" style="font-size:50%;">76.24</span></td>
<td id="S3.T1.1.16.14.7" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.16.14.7.1" class="ltx_text" style="font-size:50%;">42.31</span></td>
<td id="S3.T1.1.16.14.8" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.16.14.8.1" class="ltx_text" style="font-size:50%;">47.26</span></td>
<td id="S3.T1.1.16.14.9" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.16.14.9.1" class="ltx_text" style="font-size:50%;">56.15</span></td>
<td id="S3.T1.1.16.14.10" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.16.14.10.1" class="ltx_text" style="font-size:50%;">28.99</span></td>
<td id="S3.T1.1.16.14.11" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.16.14.11.1" class="ltx_text" style="font-size:50%;">27.72</span></td>
<td id="S3.T1.1.16.14.12" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.16.14.12.1" class="ltx_text" style="font-size:50%;">12.03</span></td>
<td id="S3.T1.1.16.14.13" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.16.14.13.1" class="ltx_text" style="font-size:50%;">50.85</span></td>
<td id="S3.T1.1.16.14.14" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.16.14.14.1" class="ltx_text" style="font-size:50%;">31.53</span></td>
<td id="S3.T1.1.16.14.15" class="ltx_td ltx_align_center ltx_border_r" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.16.14.15.1" class="ltx_text" style="font-size:50%;">31.24</span></td>
<td id="S3.T1.1.16.14.16" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.16.14.16.1" class="ltx_text" style="font-size:50%;">42.93</span></td>
<td id="S3.T1.1.16.14.17" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.16.14.17.1" class="ltx_text" style="font-size:50%;">80.05</span></td>
</tr>
<tr id="S3.T1.1.17.15" class="ltx_tr">
<td id="S3.T1.1.17.15.1" class="ltx_td ltx_align_center ltx_border_r" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.17.15.1.1" class="ltx_text" style="font-size:50%;">ShortGPT</span></td>
<td id="S3.T1.1.17.15.2" class="ltx_td ltx_align_center ltx_border_r" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.17.15.2.1" class="ltx_text" style="font-size:50%;">24.2%</span></td>
<td id="S3.T1.1.17.15.3" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.17.15.3.1" class="ltx_text ltx_font_bold" style="font-size:50%;">33.30</span></td>
<td id="S3.T1.1.17.15.4" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.17.15.4.1" class="ltx_text ltx_font_bold" style="font-size:50%;">56.96</span></td>
<td id="S3.T1.1.17.15.5" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.17.15.5.1" class="ltx_text" style="font-size:50%;">67.68</span></td>
<td id="S3.T1.1.17.15.6" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.17.15.6.1" class="ltx_text" style="font-size:50%;">65.63</span></td>
<td id="S3.T1.1.17.15.7" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.17.15.7.1" class="ltx_text" style="font-size:50%;">50.00</span></td>
<td id="S3.T1.1.17.15.8" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.17.15.8.1" class="ltx_text" style="font-size:50%;">46.70</span></td>
<td id="S3.T1.1.17.15.9" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.17.15.9.1" class="ltx_text ltx_font_bold" style="font-size:50%;">67.83</span></td>
<td id="S3.T1.1.17.15.10" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.17.15.10.1" class="ltx_text ltx_font_bold" style="font-size:50%;">53.26</span></td>
<td id="S3.T1.1.17.15.11" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.17.15.11.1" class="ltx_text ltx_font_bold" style="font-size:50%;">46.76</span></td>
<td id="S3.T1.1.17.15.12" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.17.15.12.1" class="ltx_text" style="font-size:50%;">0.04</span></td>
<td id="S3.T1.1.17.15.13" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.17.15.13.1" class="ltx_text ltx_font_bold" style="font-size:50%;">56.33</span></td>
<td id="S3.T1.1.17.15.14" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.17.15.14.1" class="ltx_text ltx_font_bold" style="font-size:50%;">45.77</span></td>
<td id="S3.T1.1.17.15.15" class="ltx_td ltx_align_center ltx_border_r" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.17.15.15.1" class="ltx_text ltx_font_bold" style="font-size:50%;">47.87</span></td>
<td id="S3.T1.1.17.15.16" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.17.15.16.1" class="ltx_text ltx_font_bold" style="font-size:50%;">49.08</span></td>
<td id="S3.T1.1.17.15.17" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.17.15.17.1" class="ltx_text ltx_font_bold" style="font-size:50%;">91.52</span></td>
</tr>
<tr id="S3.T1.1.18.16" class="ltx_tr">
<td id="S3.T1.1.18.16.1" class="ltx_td ltx_align_center ltx_border_bb ltx_border_r ltx_border_t" style="padding:0.75pt 2.2pt;" rowspan="5"><span id="S3.T1.1.18.16.1.1" class="ltx_text" style="font-size:50%;">Baichuan2-13B</span></td>
<td id="S3.T1.1.18.16.2" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.18.16.2.1" class="ltx_text" style="font-size:50%;">Dense</span></td>
<td id="S3.T1.1.18.16.3" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.18.16.3.1" class="ltx_text" style="font-size:50%;">0.00%</span></td>
<td id="S3.T1.1.18.16.4" class="ltx_td ltx_align_center ltx_border_t" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.18.16.4.1" class="ltx_text" style="font-size:50%;">33.21</span></td>
<td id="S3.T1.1.18.16.5" class="ltx_td ltx_align_center ltx_border_t" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.18.16.5.1" class="ltx_text" style="font-size:50%;">71.10</span></td>
<td id="S3.T1.1.18.16.6" class="ltx_td ltx_align_center ltx_border_t" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.18.16.6.1" class="ltx_text" style="font-size:50%;">78.07</span></td>
<td id="S3.T1.1.18.16.7" class="ltx_td ltx_align_center ltx_border_t" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.18.16.7.1" class="ltx_text" style="font-size:50%;">86.51</span></td>
<td id="S3.T1.1.18.16.8" class="ltx_td ltx_align_center ltx_border_t" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.18.16.8.1" class="ltx_text" style="font-size:50%;">50.00</span></td>
<td id="S3.T1.1.18.16.9" class="ltx_td ltx_align_center ltx_border_t" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.18.16.9.1" class="ltx_text" style="font-size:50%;">65.6</span></td>
<td id="S3.T1.1.18.16.10" class="ltx_td ltx_align_center ltx_border_t" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.18.16.10.1" class="ltx_text" style="font-size:50%;">77.89</span></td>
<td id="S3.T1.1.18.16.11" class="ltx_td ltx_align_center ltx_border_t" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.18.16.11.1" class="ltx_text" style="font-size:50%;">67.27</span></td>
<td id="S3.T1.1.18.16.12" class="ltx_td ltx_align_center ltx_border_t" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.18.16.12.1" class="ltx_text" style="font-size:50%;">68.94</span></td>
<td id="S3.T1.1.18.16.13" class="ltx_td ltx_align_center ltx_border_t" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.18.16.13.1" class="ltx_text" style="font-size:50%;">25.02</span></td>
<td id="S3.T1.1.18.16.14" class="ltx_td ltx_align_center ltx_border_t" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.18.16.14.1" class="ltx_text" style="font-size:50%;">65.64</span></td>
<td id="S3.T1.1.18.16.15" class="ltx_td ltx_align_center ltx_border_t" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.18.16.15.1" class="ltx_text" style="font-size:50%;">59.50</span></td>
<td id="S3.T1.1.18.16.16" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.18.16.16.1" class="ltx_text" style="font-size:50%;">61.30</span></td>
<td id="S3.T1.1.18.16.17" class="ltx_td ltx_align_center ltx_border_t" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.18.16.17.1" class="ltx_text" style="font-size:50%;">62.31</span></td>
<td id="S3.T1.1.18.16.18" class="ltx_td ltx_align_center ltx_border_t" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.18.16.18.1" class="ltx_text" style="font-size:50%;">100.00</span></td>
</tr>
<tr id="S3.T1.1.19.17" class="ltx_tr">
<td id="S3.T1.1.19.17.1" class="ltx_td ltx_align_center ltx_border_r" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.19.17.1.1" class="ltx_text" style="font-size:50%;">LLMPrun.</span></td>
<td id="S3.T1.1.19.17.2" class="ltx_td ltx_align_center ltx_border_r" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.19.17.2.1" class="ltx_text" style="font-size:50%;">24.3%</span></td>
<td id="S3.T1.1.19.17.3" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.19.17.3.1" class="ltx_text ltx_font_bold" style="font-size:50%;">33.80</span></td>
<td id="S3.T1.1.19.17.4" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.19.17.4.1" class="ltx_text" style="font-size:50%;">53.57</span></td>
<td id="S3.T1.1.19.17.5" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.19.17.5.1" class="ltx_text ltx_font_bold" style="font-size:50%;">71.82</span></td>
<td id="S3.T1.1.19.17.6" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.19.17.6.1" class="ltx_text" style="font-size:50%;">72.77</span></td>
<td id="S3.T1.1.19.17.7" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.19.17.7.1" class="ltx_text" style="font-size:50%;">37.50</span></td>
<td id="S3.T1.1.19.17.8" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.19.17.8.1" class="ltx_text" style="font-size:50%;">38.82</span></td>
<td id="S3.T1.1.19.17.9" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.19.17.9.1" class="ltx_text" style="font-size:50%;">56.54</span></td>
<td id="S3.T1.1.19.17.10" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.19.17.10.1" class="ltx_text" style="font-size:50%;">21.17</span></td>
<td id="S3.T1.1.19.17.11" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.19.17.11.1" class="ltx_text" style="font-size:50%;">21.61</span></td>
<td id="S3.T1.1.19.17.12" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.19.17.12.1" class="ltx_text" style="font-size:50%;">13.67</span></td>
<td id="S3.T1.1.19.17.13" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.19.17.13.1" class="ltx_text" style="font-size:50%;">39.89</span></td>
<td id="S3.T1.1.19.17.14" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.19.17.14.1" class="ltx_text" style="font-size:50%;">23.19</span></td>
<td id="S3.T1.1.19.17.15" class="ltx_td ltx_align_center ltx_border_r" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.19.17.15.1" class="ltx_text" style="font-size:50%;">25.18</span></td>
<td id="S3.T1.1.19.17.16" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.19.17.16.1" class="ltx_text" style="font-size:50%;">39.20</span></td>
<td id="S3.T1.1.19.17.17" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.19.17.17.1" class="ltx_text" style="font-size:50%;">62.91</span></td>
</tr>
<tr id="S3.T1.1.20.18" class="ltx_tr">
<td id="S3.T1.1.20.18.1" class="ltx_td ltx_align_center ltx_border_r" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.20.18.1.1" class="ltx_text" style="font-size:50%;">SliceGPT</span></td>
<td id="S3.T1.1.20.18.2" class="ltx_td ltx_align_center ltx_border_r" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.20.18.2.1" class="ltx_text" style="font-size:50%;">22.8%</span></td>
<td id="S3.T1.1.20.18.3" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.20.18.3.1" class="ltx_text" style="font-size:50%;">32.07</span></td>
<td id="S3.T1.1.20.18.4" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.20.18.4.1" class="ltx_text" style="font-size:50%;">25.85</span></td>
<td id="S3.T1.1.20.18.5" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.20.18.5.1" class="ltx_text" style="font-size:50%;">51.03</span></td>
<td id="S3.T1.1.20.18.6" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.20.18.6.1" class="ltx_text" style="font-size:50%;">10.40</span></td>
<td id="S3.T1.1.20.18.7" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.20.18.7.1" class="ltx_text" style="font-size:50%;">36.54</span></td>
<td id="S3.T1.1.20.18.8" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.20.18.8.1" class="ltx_text" style="font-size:50%;">18.02</span></td>
<td id="S3.T1.1.20.18.9" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.20.18.9.1" class="ltx_text" style="font-size:50%;">37.83</span></td>
<td id="S3.T1.1.20.18.10" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.20.18.10.1" class="ltx_text" style="font-size:50%;">21.56</span></td>
<td id="S3.T1.1.20.18.11" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.20.18.11.1" class="ltx_text" style="font-size:50%;">21.52</span></td>
<td id="S3.T1.1.20.18.12" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.20.18.12.1" class="ltx_text" style="font-size:50%;">0.00</span></td>
<td id="S3.T1.1.20.18.13" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.20.18.13.1" class="ltx_text" style="font-size:50%;">24.99</span></td>
<td id="S3.T1.1.20.18.14" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.20.18.14.1" class="ltx_text" style="font-size:50%;">22.95</span></td>
<td id="S3.T1.1.20.18.15" class="ltx_td ltx_align_center ltx_border_r" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.20.18.15.1" class="ltx_text" style="font-size:50%;">25.26</span></td>
<td id="S3.T1.1.20.18.16" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.20.18.16.1" class="ltx_text" style="font-size:50%;">25.03</span></td>
<td id="S3.T1.1.20.18.17" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.20.18.17.1" class="ltx_text" style="font-size:50%;">40.17</span></td>
</tr>
<tr id="S3.T1.1.21.19" class="ltx_tr">
<td id="S3.T1.1.21.19.1" class="ltx_td ltx_align_center ltx_border_r" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.21.19.1.1" class="ltx_text" style="font-size:50%;">LaCo</span></td>
<td id="S3.T1.1.21.19.2" class="ltx_td ltx_align_center ltx_border_r" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.21.19.2.1" class="ltx_text" style="font-size:50%;">24.7%</span></td>
<td id="S3.T1.1.21.19.3" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.21.19.3.1" class="ltx_text" style="font-size:50%;">33.03</span></td>
<td id="S3.T1.1.21.19.4" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.21.19.4.1" class="ltx_text ltx_font_bold" style="font-size:50%;">60.71</span></td>
<td id="S3.T1.1.21.19.5" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.21.19.5.1" class="ltx_text" style="font-size:50%;">68.88</span></td>
<td id="S3.T1.1.21.19.6" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.21.19.6.1" class="ltx_text" style="font-size:50%;">76.73</span></td>
<td id="S3.T1.1.21.19.7" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.21.19.7.1" class="ltx_text" style="font-size:50%;">44.23</span></td>
<td id="S3.T1.1.21.19.8" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.21.19.8.1" class="ltx_text ltx_font_bold" style="font-size:50%;">55.45</span></td>
<td id="S3.T1.1.21.19.9" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.21.19.9.1" class="ltx_text" style="font-size:50%;">62.35</span></td>
<td id="S3.T1.1.21.19.10" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.21.19.10.1" class="ltx_text ltx_font_bold" style="font-size:50%;">56.92</span></td>
<td id="S3.T1.1.21.19.11" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.21.19.11.1" class="ltx_text ltx_font_bold" style="font-size:50%;">57.80</span></td>
<td id="S3.T1.1.21.19.12" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.21.19.12.1" class="ltx_text" style="font-size:50%;">12.32</span></td>
<td id="S3.T1.1.21.19.13" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.21.19.13.1" class="ltx_text ltx_font_bold" style="font-size:50%;">61.10</span></td>
<td id="S3.T1.1.21.19.14" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.21.19.14.1" class="ltx_text" style="font-size:50%;">51.35</span></td>
<td id="S3.T1.1.21.19.15" class="ltx_td ltx_align_center ltx_border_r" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.21.19.15.1" class="ltx_text" style="font-size:50%;">53.65</span></td>
<td id="S3.T1.1.21.19.16" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.21.19.16.1" class="ltx_text" style="font-size:50%;">53.43</span></td>
<td id="S3.T1.1.21.19.17" class="ltx_td ltx_align_center" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.21.19.17.1" class="ltx_text" style="font-size:50%;">85.75</span></td>
</tr>
<tr id="S3.T1.1.22.20" class="ltx_tr">
<td id="S3.T1.1.22.20.1" class="ltx_td ltx_align_center ltx_border_bb ltx_border_r" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.22.20.1.1" class="ltx_text" style="font-size:50%;">ShortGPT</span></td>
<td id="S3.T1.1.22.20.2" class="ltx_td ltx_align_center ltx_border_bb ltx_border_r" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.22.20.2.1" class="ltx_text" style="font-size:50%;">24.7%</span></td>
<td id="S3.T1.1.22.20.3" class="ltx_td ltx_align_center ltx_border_bb" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.22.20.3.1" class="ltx_text" style="font-size:50%;">32.81</span></td>
<td id="S3.T1.1.22.20.4" class="ltx_td ltx_align_center ltx_border_bb" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.22.20.4.1" class="ltx_text" style="font-size:50%;">60.55</span></td>
<td id="S3.T1.1.22.20.5" class="ltx_td ltx_align_center ltx_border_bb" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.22.20.5.1" class="ltx_text ltx_font_bold" style="font-size:50%;">71.60</span></td>
<td id="S3.T1.1.22.20.6" class="ltx_td ltx_align_center ltx_border_bb" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.22.20.6.1" class="ltx_text ltx_font_bold" style="font-size:50%;">80.17</span></td>
<td id="S3.T1.1.22.20.7" class="ltx_td ltx_align_center ltx_border_bb" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.22.20.7.1" class="ltx_text ltx_font_bold" style="font-size:50%;">47.13</span></td>
<td id="S3.T1.1.22.20.8" class="ltx_td ltx_align_center ltx_border_bb" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.22.20.8.1" class="ltx_text" style="font-size:50%;">54.30</span></td>
<td id="S3.T1.1.22.20.9" class="ltx_td ltx_align_center ltx_border_bb" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.22.20.9.1" class="ltx_text ltx_font_bold" style="font-size:50%;">62.54</span></td>
<td id="S3.T1.1.22.20.10" class="ltx_td ltx_align_center ltx_border_bb" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.22.20.10.1" class="ltx_text" style="font-size:50%;">55.77</span></td>
<td id="S3.T1.1.22.20.11" class="ltx_td ltx_align_center ltx_border_bb" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.22.20.11.1" class="ltx_text" style="font-size:50%;">56.41</span></td>
<td id="S3.T1.1.22.20.12" class="ltx_td ltx_align_center ltx_border_bb" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.22.20.12.1" class="ltx_text ltx_font_bold" style="font-size:50%;">15.14</span></td>
<td id="S3.T1.1.22.20.13" class="ltx_td ltx_align_center ltx_border_bb" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.22.20.13.1" class="ltx_text" style="font-size:50%;">60.16</span></td>
<td id="S3.T1.1.22.20.14" class="ltx_td ltx_align_center ltx_border_bb" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.22.20.14.1" class="ltx_text ltx_font_bold" style="font-size:50%;">52.11</span></td>
<td id="S3.T1.1.22.20.15" class="ltx_td ltx_align_center ltx_border_bb ltx_border_r" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.22.20.15.1" class="ltx_text ltx_font_bold" style="font-size:50%;">58.86</span></td>
<td id="S3.T1.1.22.20.16" class="ltx_td ltx_align_center ltx_border_bb" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.22.20.16.1" class="ltx_text ltx_font_bold" style="font-size:50%;">54.43</span></td>
<td id="S3.T1.1.22.20.17" class="ltx_td ltx_align_center ltx_border_bb" style="padding:0.75pt 2.2pt;"><span id="S3.T1.1.22.20.17.1" class="ltx_text ltx_font_bold" style="font-size:50%;">87.33</span></td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering" style="font-size:50%;"><span class="ltx_tag ltx_tag_table">Table 1: </span>Comparison of pruning methods on multiple natural language benchmarks. The results of LLMPrun., SliceGPT and LaCo are reported from LaCo.</figcaption>
</figure>
</section>
</section>
<section id="S3.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.2 </span>Main Results</h3>

<div id="S3.SS2.p1" class="ltx_para">
<p id="S3.SS2.p1.1" class="ltx_p">To validate the efficacy of our proposed method, we conducted comparative experiments against benchmark and baseline techniques commonly employed in large language model evaluation. Considering the current structured pruning methods generally reduce parameters by no more than 30%, we performed experiments with approximately 1/4 of the parameters pruned. The experimental results are presented in Table <a href="#S3.T1" title="Table 1 ‣ 3.1.3 Baselines ‣ 3.1 Experimental Setup ‣ 3 Experiments ‣ ShortGPT: Layers in Large Language Models are More Redundant Than You Expect" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>. Additional experiments exploring different parameter reduction proportions will be discussed in the subsequent section.</p>
</div>
<div id="S3.SS2.p2" class="ltx_para">
<p id="S3.SS2.p2.1" class="ltx_p">The results demonstrate that the performance of the model pruned by our method significantly surpasses that of the baseline method, maintaining most of the large language model’s capabilities, such as reasoning, language understanding, knowledge retention, and examination performance, to the greatest extent possible. Furthermore, we note that the approach of reducing the number of layers (ShortGPT/LaCo) outperforms the method of reducing the embedding dimensions (LLMPru./SliceGPT), implying that the model exhibits more redundancy in-depth than in width. Further experimental analysis will be presented in the ensuing section.</p>
</div>
</section>
</section>
<section id="S4" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">4 </span>Analysis</h2>

<section id="S4.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.1 </span>Varying prune ratio</h3>

<figure id="S4.F5" class="ltx_figure"><img src="/html/2403.03853/assets/x7.png" id="S4.F5.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="415" height="281" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 5: </span>Pruning by different metrics, with increasing pruning ratio.</figcaption>
</figure>
<div id="S4.SS1.p1" class="ltx_para">
<p id="S4.SS1.p1.1" class="ltx_p">As a pruning method, we further validated the effects of different pruning ratios on model performance. Experiments were conducted on the Llama2-7B-Base and Baichuan2-7B-Base models, observing the Perplexity and MMLU. The pruning ratios ranged from 0% to 97%, employing the strategy of deleting layers based on the importance proposed in this paper, resulting in models with 1 to 32 layers. The results in Figure <a href="#S4.F5" title="Figure 5 ‣ 4.1 Varying prune ratio ‣ 4 Analysis ‣ ShortGPT: Layers in Large Language Models are More Redundant Than You Expect" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a> indicate that as the pruning ratio increases, the performance of the model declines. However, the MMLU score drops significantly at a certain layer, which may imply the existence of certain special layers within the network that play a critically important role.</p>
</div>
</section>
<section id="S4.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.2 </span>Depth Redundancy v.s. Width Redundancy</h3>

<figure id="S4.F6" class="ltx_figure"><img src="/html/2403.03853/assets/x8.png" id="S4.F6.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="415" height="257" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 6: </span>Redundancy on width, removing a single head from model. The x-axis is the head id.</figcaption>
</figure>
<div id="S4.SS2.p1" class="ltx_para">
<p id="S4.SS2.p1.1" class="ltx_p">In the preceding sections, we analyzed and explored the redundancy of large language models (LLMs) in terms of depth (layers). However, we also investigated redundancy in terms of width, specifically focusing on the attention heads. Our method involves sequentially removing each head from the multi-head attention mechanism and adjusting the parameters of the attention output projection to ensure the output shape of the Attention block remains unchanged. The Feed-Forward Network (FFN) remains unaltered. Similar to the approach in Section <a href="#S2.SS1" title="2.1 Layer redundancy ‣ 2 Methodology ‣ ShortGPT: Layers in Large Language Models are More Redundant Than You Expect" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2.1</span></a>, we observe the impact of each head’s removal on the final performance of the network. Figure <a href="#S4.F6" title="Figure 6 ‣ 4.2 Depth Redundancy v.s. Width Redundancy ‣ 4 Analysis ‣ ShortGPT: Layers in Large Language Models are More Redundant Than You Expect" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a> illustrates the changes in Perplexity and MMLU scores after each head is removed. The results indicate that LLMs exhibit a high degree of redundancy in width, comparable to their redundancy in depth. However, this width redundancy does not follow any discernible pattern and varies across different models. We hypothesize that this variation may be attributed to the symmetry among heads.</p>
</div>
</section>
<section id="S4.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.3 </span>Redundancy on non-transformer LLM</h3>

<figure id="S4.F7" class="ltx_figure"><img src="/html/2403.03853/assets/x9.png" id="S4.F7.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="323" height="180" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 7: </span>Redundancy on RWKV, removing single layers on RWKV-7B.</figcaption>
</figure>
<div id="S4.SS3.p1" class="ltx_para">
<p id="S4.SS3.p1.1" class="ltx_p">We further investigate whether the redundancy observed in depth is a result of the Transformer’s architecture. Given that the vast majority of popular LLMs are based on the Transformer structure. We selected the RWKV-7B model proposed in <cite class="ltx_cite ltx_citemacro_citep">(Peng et al., <a href="#bib.bib34" title="" class="ltx_ref">2023</a>)</cite>, a formidable competitor to the Transformer architecture which indeed consists of identical layers to a certain extent. We analyzed this model using the methodology from <a href="#S2.SS1" title="2.1 Layer redundancy ‣ 2 Methodology ‣ ShortGPT: Layers in Large Language Models are More Redundant Than You Expect" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2.1</span></a>. Figure <a href="#S4.F7" title="Figure 7 ‣ 4.3 Redundancy on non-transformer LLM ‣ 4 Analysis ‣ ShortGPT: Layers in Large Language Models are More Redundant Than You Expect" class="ltx_ref"><span class="ltx_text ltx_ref_tag">7</span></a> shows the redundancy of the RWKV-7B <span id="footnote1" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">1</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">1</sup><span class="ltx_tag ltx_tag_note">1</span> We use rwkv-v5-world-7B from https://huggingface.co/BlinkDL/rwkv-5-world</span></span></span> model. Through this figure, we can observe that the RWKV-7B model also exhibits a high level of redundancy. This may suggest that the redundancy is universal across LLMs.</p>
</div>
</section>
<section id="S4.SS4" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.4 </span>Importance metric</h3>

<figure id="S4.F8" class="ltx_figure"><img src="/html/2403.03853/assets/x10.png" id="S4.F8.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="276" height="206" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 8: </span>Different importance metrics. Perplexity is calculated by removing each single layer, other metrics is calculated by hidden states of each layer</figcaption>
</figure>
<div id="S4.SS4.p1" class="ltx_para">
<p id="S4.SS4.p1.1" class="ltx_p">The overarching concept of our method is to rank layers by importance and remove the less significant ones. The definition of importance metrics has a profound impact on the outcome. In this section, we define and compare various important metrics:</p>
</div>
<div id="S4.SS4.p2" class="ltx_para">
<p id="S4.SS4.p2.1" class="ltx_p"><span id="S4.SS4.p2.1.1" class="ltx_text ltx_font_bold">Sequential</span>: The importance is directly proportional to the sequence order, with shallower layers being more important. This can be implemented by taking the negative of each layer’s index as the importance metric.</p>
</div>
<div id="S4.SS4.p3" class="ltx_para">
<p id="S4.SS4.p3.1" class="ltx_p"><span id="S4.SS4.p3.1.1" class="ltx_text ltx_font_bold">Reverse-order/Norm</span>: This metric posits that importance is inversely proportional to the sequence order. It assigns higher importance scores to the shallower layers Inspired by Figure <a href="#S2.F3" title="Figure 3 ‣ 2.1 Layer redundancy ‣ 2 Methodology ‣ ShortGPT: Layers in Large Language Models are More Redundant Than You Expect" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>. This method gives the same order as measuring importance by hidden states norm mentioned in <span class="ltx_ref ltx_missing_label ltx_ref_self">LABEL:layerimportacne</span>. Here, the importance metric is the layer’s index itself.</p>
</div>
<div id="S4.SS4.p4" class="ltx_para">
<p id="S4.SS4.p4.2" class="ltx_p"><span id="S4.SS4.p4.2.1" class="ltx_text ltx_font_bold">Relative Magnitude</span>: Proposed in <cite class="ltx_cite ltx_citemacro_citep">(Samragh et al., <a href="#bib.bib39" title="" class="ltx_ref">2023</a>)</cite>, this metric assumes layers with larger <math id="S4.SS4.p4.1.m1.2" class="ltx_Math" alttext="||\frac{f(x)}{x+f(x)}||" display="inline"><semantics id="S4.SS4.p4.1.m1.2a"><mrow id="S4.SS4.p4.1.m1.2.3.2" xref="S4.SS4.p4.1.m1.2.3.1.cmml"><mo stretchy="false" id="S4.SS4.p4.1.m1.2.3.2.1" xref="S4.SS4.p4.1.m1.2.3.1.1.cmml">‖</mo><mfrac id="S4.SS4.p4.1.m1.2.2" xref="S4.SS4.p4.1.m1.2.2.cmml"><mrow id="S4.SS4.p4.1.m1.1.1.1" xref="S4.SS4.p4.1.m1.1.1.1.cmml"><mi id="S4.SS4.p4.1.m1.1.1.1.3" xref="S4.SS4.p4.1.m1.1.1.1.3.cmml">f</mi><mo lspace="0em" rspace="0em" id="S4.SS4.p4.1.m1.1.1.1.2" xref="S4.SS4.p4.1.m1.1.1.1.2.cmml">​</mo><mrow id="S4.SS4.p4.1.m1.1.1.1.4.2" xref="S4.SS4.p4.1.m1.1.1.1.cmml"><mo stretchy="false" id="S4.SS4.p4.1.m1.1.1.1.4.2.1" xref="S4.SS4.p4.1.m1.1.1.1.cmml">(</mo><mi id="S4.SS4.p4.1.m1.1.1.1.1" xref="S4.SS4.p4.1.m1.1.1.1.1.cmml">x</mi><mo stretchy="false" id="S4.SS4.p4.1.m1.1.1.1.4.2.2" xref="S4.SS4.p4.1.m1.1.1.1.cmml">)</mo></mrow></mrow><mrow id="S4.SS4.p4.1.m1.2.2.2" xref="S4.SS4.p4.1.m1.2.2.2.cmml"><mi id="S4.SS4.p4.1.m1.2.2.2.3" xref="S4.SS4.p4.1.m1.2.2.2.3.cmml">x</mi><mo id="S4.SS4.p4.1.m1.2.2.2.2" xref="S4.SS4.p4.1.m1.2.2.2.2.cmml">+</mo><mrow id="S4.SS4.p4.1.m1.2.2.2.4" xref="S4.SS4.p4.1.m1.2.2.2.4.cmml"><mi id="S4.SS4.p4.1.m1.2.2.2.4.2" xref="S4.SS4.p4.1.m1.2.2.2.4.2.cmml">f</mi><mo lspace="0em" rspace="0em" id="S4.SS4.p4.1.m1.2.2.2.4.1" xref="S4.SS4.p4.1.m1.2.2.2.4.1.cmml">​</mo><mrow id="S4.SS4.p4.1.m1.2.2.2.4.3.2" xref="S4.SS4.p4.1.m1.2.2.2.4.cmml"><mo stretchy="false" id="S4.SS4.p4.1.m1.2.2.2.4.3.2.1" xref="S4.SS4.p4.1.m1.2.2.2.4.cmml">(</mo><mi id="S4.SS4.p4.1.m1.2.2.2.1" xref="S4.SS4.p4.1.m1.2.2.2.1.cmml">x</mi><mo stretchy="false" id="S4.SS4.p4.1.m1.2.2.2.4.3.2.2" xref="S4.SS4.p4.1.m1.2.2.2.4.cmml">)</mo></mrow></mrow></mrow></mfrac><mo stretchy="false" id="S4.SS4.p4.1.m1.2.3.2.2" xref="S4.SS4.p4.1.m1.2.3.1.1.cmml">‖</mo></mrow><annotation-xml encoding="MathML-Content" id="S4.SS4.p4.1.m1.2b"><apply id="S4.SS4.p4.1.m1.2.3.1.cmml" xref="S4.SS4.p4.1.m1.2.3.2"><csymbol cd="latexml" id="S4.SS4.p4.1.m1.2.3.1.1.cmml" xref="S4.SS4.p4.1.m1.2.3.2.1">norm</csymbol><apply id="S4.SS4.p4.1.m1.2.2.cmml" xref="S4.SS4.p4.1.m1.2.2"><divide id="S4.SS4.p4.1.m1.2.2.3.cmml" xref="S4.SS4.p4.1.m1.2.2"></divide><apply id="S4.SS4.p4.1.m1.1.1.1.cmml" xref="S4.SS4.p4.1.m1.1.1.1"><times id="S4.SS4.p4.1.m1.1.1.1.2.cmml" xref="S4.SS4.p4.1.m1.1.1.1.2"></times><ci id="S4.SS4.p4.1.m1.1.1.1.3.cmml" xref="S4.SS4.p4.1.m1.1.1.1.3">𝑓</ci><ci id="S4.SS4.p4.1.m1.1.1.1.1.cmml" xref="S4.SS4.p4.1.m1.1.1.1.1">𝑥</ci></apply><apply id="S4.SS4.p4.1.m1.2.2.2.cmml" xref="S4.SS4.p4.1.m1.2.2.2"><plus id="S4.SS4.p4.1.m1.2.2.2.2.cmml" xref="S4.SS4.p4.1.m1.2.2.2.2"></plus><ci id="S4.SS4.p4.1.m1.2.2.2.3.cmml" xref="S4.SS4.p4.1.m1.2.2.2.3">𝑥</ci><apply id="S4.SS4.p4.1.m1.2.2.2.4.cmml" xref="S4.SS4.p4.1.m1.2.2.2.4"><times id="S4.SS4.p4.1.m1.2.2.2.4.1.cmml" xref="S4.SS4.p4.1.m1.2.2.2.4.1"></times><ci id="S4.SS4.p4.1.m1.2.2.2.4.2.cmml" xref="S4.SS4.p4.1.m1.2.2.2.4.2">𝑓</ci><ci id="S4.SS4.p4.1.m1.2.2.2.1.cmml" xref="S4.SS4.p4.1.m1.2.2.2.1">𝑥</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS4.p4.1.m1.2c">||\frac{f(x)}{x+f(x)}||</annotation></semantics></math> are of higher importance, where <math id="S4.SS4.p4.2.m2.1" class="ltx_Math" alttext="f" display="inline"><semantics id="S4.SS4.p4.2.m2.1a"><mi id="S4.SS4.p4.2.m2.1.1" xref="S4.SS4.p4.2.m2.1.1.cmml">f</mi><annotation-xml encoding="MathML-Content" id="S4.SS4.p4.2.m2.1b"><ci id="S4.SS4.p4.2.m2.1.1.cmml" xref="S4.SS4.p4.2.m2.1.1">𝑓</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS4.p4.2.m2.1c">f</annotation></semantics></math> is the layer transformation function.</p>
</div>
<div id="S4.SS4.p5" class="ltx_para">
<p id="S4.SS4.p5.1" class="ltx_p"><span id="S4.SS4.p5.1.1" class="ltx_text ltx_font_bold">BI</span>: BI mentioned in previous section <a href="#S2.SS2" title="2.2 Layer importance ‣ 2 Methodology ‣ ShortGPT: Layers in Large Language Models are More Redundant Than You Expect" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2.2</span></a>.</p>
</div>
<div id="S4.SS4.p6" class="ltx_para">
<p id="S4.SS4.p6.1" class="ltx_p">Figure <a href="#S4.F8" title="Figure 8 ‣ 4.4 Importance metric ‣ 4 Analysis ‣ ShortGPT: Layers in Large Language Models are More Redundant Than You Expect" class="ltx_ref"><span class="ltx_text ltx_ref_tag">8</span></a> demonstrates the different metrics. We observe that shallower layers in the LLM network are more crucial than deeper ones. Figure <a href="#S4.F5" title="Figure 5 ‣ 4.1 Varying prune ratio ‣ 4 Analysis ‣ ShortGPT: Layers in Large Language Models are More Redundant Than You Expect" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a> shows the results of removing layers by different metrics, demonstrating that Our proposed BI outperforms other metrics. The method of <cite class="ltx_cite ltx_citemacro_citep">(Samragh et al., <a href="#bib.bib39" title="" class="ltx_ref">2023</a>)</cite> is highly competitive, indicating that relative values can also reflect the importance to a certain extent. It is worth noting that the hidden states norm seems to be a good metric when only considering the MMLU benchmark, but the perplexity is relatively poor, which significantly impacts tasks related to text generation. This also indicates the limitations of the current LLMs evaluation methods.</p>
</div>
</section>
<section id="S4.SS5" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.5 </span>Orthogonal to Quantization</h3>

<div id="S4.SS5.p1" class="ltx_para">
<p id="S4.SS5.p1.1" class="ltx_p">In this section, we show that our method is orthogonal to quantization methods. We apply our method to a model <span id="footnote2" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">2</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">2</sup><span class="ltx_tag ltx_tag_note">2</span>We take the model from https://huggingface.co/TheBloke/Llama-2-7B-GPTQ</span></span></span> quantized by GPTQ algorithm. Table <a href="#S4.T2" title="Table 2 ‣ 4.5 Orthogonal to Quantization ‣ 4 Analysis ‣ ShortGPT: Layers in Large Language Models are More Redundant Than You Expect" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a> shows that our method is compatible with the quantization-like method, which could reduce the footprint further.</p>
</div>
<figure id="S4.T2" class="ltx_table">
<table id="S4.T2.1" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S4.T2.1.1.1" class="ltx_tr">
<th id="S4.T2.1.1.1.1" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_l ltx_border_r ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;">Model</th>
<th id="S4.T2.1.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;">Ratio/Layer</th>
<th id="S4.T2.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;">Perplexity</th>
<th id="S4.T2.1.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;">MMLU</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S4.T2.1.2.1" class="ltx_tr">
<td id="S4.T2.1.2.1.1" class="ltx_td ltx_align_center ltx_border_l ltx_border_r ltx_border_tt" style="padding-top:1pt;padding-bottom:1pt;">Baseline</td>
<td id="S4.T2.1.2.1.2" class="ltx_td ltx_align_center ltx_border_r ltx_border_tt" style="padding-top:1pt;padding-bottom:1pt;">0%/32</td>
<td id="S4.T2.1.2.1.3" class="ltx_td ltx_align_center ltx_border_r ltx_border_tt" style="padding-top:1pt;padding-bottom:1pt;">8.4999</td>
<td id="S4.T2.1.2.1.4" class="ltx_td ltx_align_center ltx_border_r ltx_border_tt" style="padding-top:1pt;padding-bottom:1pt;">37.99</td>
</tr>
<tr id="S4.T2.1.3.2" class="ltx_tr">
<td id="S4.T2.1.3.2.1" class="ltx_td ltx_align_center ltx_border_b ltx_border_l ltx_border_r ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;">ShortGPT</td>
<td id="S4.T2.1.3.2.2" class="ltx_td ltx_align_center ltx_border_b ltx_border_r ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;">27.1%/23</td>
<td id="S4.T2.1.3.2.3" class="ltx_td ltx_align_center ltx_border_b ltx_border_r ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;">42.6951</td>
<td id="S4.T2.1.3.2.4" class="ltx_td ltx_align_center ltx_border_b ltx_border_r ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;">36.69</td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 2: </span>Layer removal results on Llama2-7B-Base-GPTQ.</figcaption>
</figure>
</section>
</section>
<section id="S5" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">5 </span>Limitation</h2>

<div id="S5.p1" class="ltx_para">
<p id="S5.p1.1" class="ltx_p">Although our method demonstrates strong competitiveness compared to current pruning methods, with some benchmarks even suggesting that deleting certain layers has no impact on the final result, this does not imply that layer removal comes without drawbacks. Due to the limitations of benchmarking, current evaluations may not fully capture the impact of layer removal on model performance. Our experiments reveal that the effect of layer removal is significantly more pronounced on generative tasks compared to multiple-choice tasks. On benchmarks such as GSM8K <cite class="ltx_cite ltx_citemacro_citep">(Cobbe et al., <a href="#bib.bib9" title="" class="ltx_ref">2021</a>)</cite> and HumanEval <cite class="ltx_cite ltx_citemacro_citep">(Chen et al., <a href="#bib.bib5" title="" class="ltx_ref">2021</a>)</cite>, removing 25% of the layers often leads to a severe performance drop, with scores approaching zero. Nevertheless, this study proves that models maintain strong semantic understanding and processing capabilities after layer removal. Post-training techniques could potentially further mitigate performance losses, which is an area of ongoing research.</p>
</div>
</section>
<section id="S6" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">6 </span>Related works</h2>

<div id="S6.p1" class="ltx_para">
<p id="S6.p1.1" class="ltx_p">To reduce the inference cost of large language models and increase their practical applications, there have been many recent works on compressing models, which can be classified into four categories: model pruning, knowledge distillation, quantization, and low-rank factorization.</p>
</div>
<div id="S6.p2" class="ltx_para">
<p id="S6.p2.1" class="ltx_p"><span id="S6.p2.1.1" class="ltx_text ltx_font_bold">Model pruning:</span> model pruning <cite class="ltx_cite ltx_citemacro_citep">(LeCun et al., <a href="#bib.bib28" title="" class="ltx_ref">1989</a>; Han et al., <a href="#bib.bib19" title="" class="ltx_ref">2015</a>)</cite> is a classic and effective method of reducing model redundancy modules to compress models. The model pruning methods mainly include unstructured pruning and structured pruning. The unstructured pruning simplifies an LLM by removing specific parameters without considering its internal structure, such as SparseGPT <cite class="ltx_cite ltx_citemacro_citep">(Frantar and Alistarh, <a href="#bib.bib15" title="" class="ltx_ref">2023</a>)</cite> and LoRAPrune <cite class="ltx_cite ltx_citemacro_citep">(Zhang et al., <a href="#bib.bib52" title="" class="ltx_ref">2023</a>)</cite>. However, this method disregards the overall LLM structure, resulting in an irregular sparse model composition. Another more practical approach is structured pruning, GUM<cite class="ltx_cite ltx_citemacro_citep">(Syed et al., <a href="#bib.bib42" title="" class="ltx_ref">2023</a>)</cite> makes an analysis of several structured pruning methods for decoder-only LLMs. LLM-Pruner <cite class="ltx_cite ltx_citemacro_citep">(Ma et al., <a href="#bib.bib32" title="" class="ltx_ref">2024</a>)</cite> selectively removes non-critical structures according to gradient information. ShearedLLaMA <cite class="ltx_cite ltx_citemacro_citep">(Xia et al., <a href="#bib.bib45" title="" class="ltx_ref">2023</a>)</cite> employs targeted structured pruning and dynamic batch loading. LaCo <cite class="ltx_cite ltx_citemacro_citep">(Yang et al., <a href="#bib.bib49" title="" class="ltx_ref">2024</a>)</cite> used layer merging to compress the model. Compared to the previous method, our method is a simple and efficient structured pruning method.</p>
</div>
<div id="S6.p3" class="ltx_para">
<p id="S6.p3.1" class="ltx_p"><span id="S6.p3.1.1" class="ltx_text ltx_font_bold">Knowledge distillation:</span> Knowledge distillation (KD) <cite class="ltx_cite ltx_citemacro_citep">(Hinton et al., <a href="#bib.bib22" title="" class="ltx_ref">2015</a>; Gou et al., <a href="#bib.bib17" title="" class="ltx_ref">2021</a>)</cite> is another way of compressing models, where a larger teacher network provides knowledge to a smaller student network. Using the prompt response pairs by LLM APIs to fine-tune small models can get promised results <cite class="ltx_cite ltx_citemacro_citep">(Li et al., <a href="#bib.bib30" title="" class="ltx_ref">2022</a>)</cite>. In addition, when we can access transparent teacher models, the result can be improved more. MiniLLM <cite class="ltx_cite ltx_citemacro_citep">(Gu et al., <a href="#bib.bib18" title="" class="ltx_ref">2023</a>)</cite> uses reverse Kullback-Leibler divergence to prevent the student model from overestimating the low-probability regions of the teacher distribution. DistlLLM <cite class="ltx_cite ltx_citemacro_citep">(Ko et al., <a href="#bib.bib26" title="" class="ltx_ref">2024</a>)</cite> uses an adaptive off-policy approach designed to enhance the efficiency in utilizing student-generated outputs. However, compared to model pruning, this method often requires higher computational resources.</p>
</div>
<div id="S6.p4" class="ltx_para">
<p id="S6.p4.1" class="ltx_p"><span id="S6.p4.1.1" class="ltx_text ltx_font_bold">Quantization:</span> quantization <cite class="ltx_cite ltx_citemacro_citep">(Liu et al., <a href="#bib.bib31" title="" class="ltx_ref">2021</a>; Gholami et al., <a href="#bib.bib16" title="" class="ltx_ref">2022</a>; Dettmers et al., <a href="#bib.bib11" title="" class="ltx_ref">2022</a>, <a href="#bib.bib12" title="" class="ltx_ref">2024</a>)</cite> is a widely accepted technique in the field of model compression, which can significantly save the storage and computational costs of deep learning models. Traditional models are generally stored as floating-point numbers, but quantization converts them into integers or other discrete forms. LUT-GEMM <cite class="ltx_cite ltx_citemacro_citep">(Park et al., <a href="#bib.bib33" title="" class="ltx_ref">2022</a>)</cite> quantifies only weights and optimizes matrix multiplication in LLM using BCQ format. SPQR <cite class="ltx_cite ltx_citemacro_citep">(Dettmers et al., <a href="#bib.bib13" title="" class="ltx_ref">2023</a>)</cite> identifies and isolates abnormal weights, stores them with higher accuracy, and compresses all other weights into 3-4 bits. Our model pruning method and quantization method are orthogonal, which means quantification based on our pruned model can further compress the model.</p>
</div>
<div id="S6.p5" class="ltx_para">
<p id="S6.p5.1" class="ltx_p"><span id="S6.p5.1.1" class="ltx_text ltx_font_bold">Low-rank factorization:</span> low-rank decomposition <cite class="ltx_cite ltx_citemacro_citep">(Cheng et al., <a href="#bib.bib6" title="" class="ltx_ref">2017</a>; Povey et al., <a href="#bib.bib35" title="" class="ltx_ref">2018</a>)</cite> is a model compression technique aimed at approximating a given weight matrix by decomposing it into two or more smaller matrices with significantly lower dimensions. TensorGPT <cite class="ltx_cite ltx_citemacro_citep">(Xu et al., <a href="#bib.bib47" title="" class="ltx_ref">2023</a>)</cite> stores large embeddings in a low-rank tensor format, reducing the space complexity of LLMs and making them available on edge devices. Recently, SliceGPT <cite class="ltx_cite ltx_citemacro_citep">(Ashkboos et al., <a href="#bib.bib1" title="" class="ltx_ref">2024</a>)</cite> has also implemented structured compression of models based on matrix factorization of hidden states and absorbed the new matrix into the parameters of the origin networks.</p>
</div>
<div id="S6.p6" class="ltx_para">
<p id="S6.p6.1" class="ltx_p"><span id="S6.p6.1.1" class="ltx_text ltx_font_bold">Model redundancy:</span> researchers have long noticed the significant redundancy in nonlinear models <cite class="ltx_cite ltx_citemacro_citep">(Catchpole and Morgan, <a href="#bib.bib4" title="" class="ltx_ref">1997</a>)</cite>. In recent years, the transformer model architecture has been widely applied, and researchers have also studied its redundancy. In <cite class="ltx_cite ltx_citemacro_citep">(Bian et al., <a href="#bib.bib2" title="" class="ltx_ref">2021</a>)</cite>, researchers analyzed redundancy in attention mechanisms, in which clear and similar redundancy patterns (cluster structure) are observed among attention heads. In <cite class="ltx_cite ltx_citemacro_citep">(Dalvi et al., <a href="#bib.bib10" title="" class="ltx_ref">2020</a>)</cite>, researchers dissect two pre-trained models, BERT <cite class="ltx_cite ltx_citemacro_citep">(Devlin et al., <a href="#bib.bib14" title="" class="ltx_ref">2018</a>)</cite> and XLNet <cite class="ltx_cite ltx_citemacro_citep">(Yang et al., <a href="#bib.bib50" title="" class="ltx_ref">2019</a>)</cite>, studying how much redundancy they exhibit at a representation level and a more fine-grained neuron-level. However, the redundancy in current large language models based on decoder-only structures still needs to be explored.</p>
</div>
</section>
<section id="S7" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">7 </span>Conclusion</h2>

<div id="S7.p1" class="ltx_para">
<p id="S7.p1.1" class="ltx_p">This work has introduced a novel approach to pruning Large Language Models (LLMs) based on layer redundancy and an "importance" metric defined as attention entropy. Our study reveals a significant degree of layer-wise redundancy in LLMs, indicating that certain layers contribute minimally to the overall network function and can thus be removed without substantially compromising model performance. By employing a straightforward layer removal strategy guided by the calculated importance of each layer, we have demonstrated that it is possible to maintain up to 95% of an LLM’s performance while reducing the model’s parameter count and computational requirements by approximately 25%. This achievement not only surpasses previous state-of-the-art pruning methods but also underscores the potential for further optimizations in LLM deployment strategies.</p>
</div>
<div id="S7.p2" class="ltx_para">
<p id="S7.p2.1" class="ltx_p">Our findings suggest that the redundancy inherent in LLMs is largely depth-based rather than width-based, highlighting an avenue for future research into the structural efficiency of neural networks. Moreover, our pruning approach, characterized by its simplicity and effectiveness, is compatible with other compression techniques such as quantization, offering a composite pathway to model size reduction that is both significant and versatile.</p>
</div>
<div id="S7.p3" class="ltx_para">
<p id="S7.p3.1" class="ltx_p">The implications of our research extend beyond academic interest, offering practical solutions for deploying advanced LLMs in resource-constrained environments. By enabling more efficient model architectures without the need for extensive retraining, our pruning method facilitates broader access to cutting-edge AI capabilities across a variety of platforms and devices.</p>
</div>
<div id="S7.p4" class="ltx_para">
<p id="S7.p4.1" class="ltx_p">In conclusion, our investigation into layer redundancy and the development of an importance-based pruning strategy represent a meaningful advancement in the optimization of Large Language Models. As the demand for sophisticated AI tools continues to grow, approaches like ours will play a crucial role in making these technologies more accessible and sustainable. Future work will focus on refining our understanding of model redundancy and exploring additional methods for enhancing the efficiency of neural network models.</p>
</div>
</section>
<section id="bib" class="ltx_bibliography">
<h2 class="ltx_title ltx_title_bibliography">References</h2>

<ul class="ltx_biblist">
<li id="bib.bib1" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ashkboos et al. (2024)</span>
<span class="ltx_bibblock">
Saleh Ashkboos, Maximilian L Croci, Marcelo Gennari do Nascimento, Torsten
Hoefler, and James Hensman. 2024.

</span>
<span class="ltx_bibblock">Slicegpt: Compress large language models by deleting rows and
columns.

</span>
<span class="ltx_bibblock"><em id="bib.bib1.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2401.15024</em>.

</span>
</li>
<li id="bib.bib2" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Bian et al. (2021)</span>
<span class="ltx_bibblock">
Yuchen Bian, Jiaji Huang, Xingyu Cai, Jiahong Yuan, and Kenneth Church. 2021.

</span>
<span class="ltx_bibblock">On attention redundancy: A comprehensive study.

</span>
<span class="ltx_bibblock">In <em id="bib.bib2.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2021 conference of the north american
chapter of the association for computational linguistics: human language
technologies</em>, pages 930–945.

</span>
</li>
<li id="bib.bib3" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Bisk et al. (2020)</span>
<span class="ltx_bibblock">
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. 2020.

</span>
<span class="ltx_bibblock">Piqa: Reasoning about physical commonsense in natural language.

</span>
<span class="ltx_bibblock">In <em id="bib.bib3.1.1" class="ltx_emph ltx_font_italic">Proceedings of the AAAI conference on artificial
intelligence</em>, pages 7432–7439.

</span>
</li>
<li id="bib.bib4" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Catchpole and Morgan (1997)</span>
<span class="ltx_bibblock">
Edward A Catchpole and Byron JT Morgan. 1997.

</span>
<span class="ltx_bibblock">Detecting parameter redundancy.

</span>
<span class="ltx_bibblock"><em id="bib.bib4.1.1" class="ltx_emph ltx_font_italic">Biometrika</em>, 84(1):187–196.

</span>
</li>
<li id="bib.bib5" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Chen et al. (2021)</span>
<span class="ltx_bibblock">
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira
Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg
Brockman, et al. 2021.

</span>
<span class="ltx_bibblock">Evaluating large language models trained on code.

</span>
<span class="ltx_bibblock"><em id="bib.bib5.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2107.03374</em>.

</span>
</li>
<li id="bib.bib6" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Cheng et al. (2017)</span>
<span class="ltx_bibblock">
Yu Cheng, Duo Wang, Pan Zhou, and Tao Zhang. 2017.

</span>
<span class="ltx_bibblock">A survey of model compression and acceleration for deep neural
networks.

</span>
<span class="ltx_bibblock"><em id="bib.bib6.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:1710.09282</em>.

</span>
</li>
<li id="bib.bib7" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Chowdhery et al. (2023)</span>
<span class="ltx_bibblock">
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra,
Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian
Gehrmann, et al. 2023.

</span>
<span class="ltx_bibblock">Palm: Scaling language modeling with pathways.

</span>
<span class="ltx_bibblock"><em id="bib.bib7.1.1" class="ltx_emph ltx_font_italic">Journal of Machine Learning Research</em>, 24(240):1–113.

</span>
</li>
<li id="bib.bib8" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Clark et al. (2019)</span>
<span class="ltx_bibblock">
Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael
Collins, and Kristina Toutanova. 2019.

</span>
<span class="ltx_bibblock">Boolq: Exploring the surprising difficulty of natural yes/no
questions.

</span>
<span class="ltx_bibblock">In <em id="bib.bib8.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2019 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language
Technologies, Volume 1 (Long and Short Papers)</em>, pages 2924–2936.

</span>
</li>
<li id="bib.bib9" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Cobbe et al. (2021)</span>
<span class="ltx_bibblock">
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz
Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano,
et al. 2021.

</span>
<span class="ltx_bibblock">Training verifiers to solve math word problems.

</span>
<span class="ltx_bibblock"><em id="bib.bib9.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2110.14168</em>.

</span>
</li>
<li id="bib.bib10" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Dalvi et al. (2020)</span>
<span class="ltx_bibblock">
Fahim Dalvi, Hassan Sajjad, Nadir Durrani, and Yonatan Belinkov. 2020.

</span>
<span class="ltx_bibblock"><a target="_blank" href="http://arxiv.org/abs/2004.04010" title="" class="ltx_ref ltx_href">Analyzing redundancy in
pretrained transformer models</a>.

</span>
</li>
<li id="bib.bib11" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Dettmers et al. (2022)</span>
<span class="ltx_bibblock">
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. 2022.

</span>
<span class="ltx_bibblock">Llm. int8 (): 8-bit matrix multiplication for transformers at scale.

</span>
<span class="ltx_bibblock"><em id="bib.bib11.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2208.07339</em>.

</span>
</li>
<li id="bib.bib12" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Dettmers et al. (2024)</span>
<span class="ltx_bibblock">
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2024.

</span>
<span class="ltx_bibblock">Qlora: Efficient finetuning of quantized llms.

</span>
<span class="ltx_bibblock"><em id="bib.bib12.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>, 36.

</span>
</li>
<li id="bib.bib13" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Dettmers et al. (2023)</span>
<span class="ltx_bibblock">
Tim Dettmers, Ruslan Svirschevski, Vage Egiazarian, Denis Kuznedelev, Elias
Frantar, Saleh Ashkboos, Alexander Borzunov, Torsten Hoefler, and Dan
Alistarh. 2023.

</span>
<span class="ltx_bibblock">Spqr: A sparse-quantized representation for near-lossless llm weight
compression.

</span>
<span class="ltx_bibblock"><em id="bib.bib13.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2306.03078</em>.

</span>
</li>
<li id="bib.bib14" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Devlin et al. (2018)</span>
<span class="ltx_bibblock">
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.

</span>
<span class="ltx_bibblock">Bert: Pre-training of deep bidirectional transformers for language
understanding.

</span>
<span class="ltx_bibblock"><em id="bib.bib14.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:1810.04805</em>.

</span>
</li>
<li id="bib.bib15" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Frantar and Alistarh (2023)</span>
<span class="ltx_bibblock">
Elias Frantar and Dan Alistarh. 2023.

</span>
<span class="ltx_bibblock">Massive language models can be accurately pruned in one-shot.

</span>
<span class="ltx_bibblock"><em id="bib.bib15.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2301.00774</em>.

</span>
</li>
<li id="bib.bib16" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Gholami et al. (2022)</span>
<span class="ltx_bibblock">
Amir Gholami, Sehoon Kim, Zhen Dong, Zhewei Yao, Michael W Mahoney, and Kurt
Keutzer. 2022.

</span>
<span class="ltx_bibblock">A survey of quantization methods for efficient neural network
inference.

</span>
<span class="ltx_bibblock">In <em id="bib.bib16.1.1" class="ltx_emph ltx_font_italic">Low-Power Computer Vision</em>, pages 291–326. Chapman and
Hall/CRC.

</span>
</li>
<li id="bib.bib17" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Gou et al. (2021)</span>
<span class="ltx_bibblock">
Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. 2021.

</span>
<span class="ltx_bibblock">Knowledge distillation: A survey.

</span>
<span class="ltx_bibblock"><em id="bib.bib17.1.1" class="ltx_emph ltx_font_italic">International Journal of Computer Vision</em>, 129:1789–1819.

</span>
</li>
<li id="bib.bib18" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Gu et al. (2023)</span>
<span class="ltx_bibblock">
Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. 2023.

</span>
<span class="ltx_bibblock">Knowledge distillation of large language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib18.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2306.08543</em>.

</span>
</li>
<li id="bib.bib19" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Han et al. (2015)</span>
<span class="ltx_bibblock">
Song Han, Jeff Pool, John Tran, and William Dally. 2015.

</span>
<span class="ltx_bibblock">Learning both weights and connections for efficient neural network.

</span>
<span class="ltx_bibblock"><em id="bib.bib19.1.1" class="ltx_emph ltx_font_italic">Advances in neural information processing systems</em>, 28.

</span>
</li>
<li id="bib.bib20" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hasan et al. (2021)</span>
<span class="ltx_bibblock">
Tahmid Hasan, Abhik Bhattacharjee, Md Saiful Islam, Kazi Mubasshir, Yuan-Fang
Li, Yong-Bin Kang, M Sohel Rahman, and Rifat Shahriyar. 2021.

</span>
<span class="ltx_bibblock">Xl-sum: Large-scale multilingual abstractive summarization for 44
languages.

</span>
<span class="ltx_bibblock">In <em id="bib.bib20.1.1" class="ltx_emph ltx_font_italic">Findings of the Association for Computational Linguistics:
ACL-IJCNLP 2021</em>, pages 4693–4703.

</span>
</li>
<li id="bib.bib21" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hendrycks et al. (2020)</span>
<span class="ltx_bibblock">
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn
Song, and Jacob Steinhardt. 2020.

</span>
<span class="ltx_bibblock">Measuring massive multitask language understanding.

</span>
<span class="ltx_bibblock"><em id="bib.bib21.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2009.03300</em>.

</span>
</li>
<li id="bib.bib22" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hinton et al. (2015)</span>
<span class="ltx_bibblock">
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015.

</span>
<span class="ltx_bibblock">Distilling the knowledge in a neural network.

</span>
<span class="ltx_bibblock"><em id="bib.bib22.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:1503.02531</em>.

</span>
</li>
<li id="bib.bib23" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hoffmann et al. (2022)</span>
<span class="ltx_bibblock">
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor
Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes
Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den
Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich
Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. 2022.

</span>
<span class="ltx_bibblock"><a target="_blank" href="http://arxiv.org/abs/2203.15556" title="" class="ltx_ref ltx_href">Training compute-optimal
large language models</a>.

</span>
</li>
<li id="bib.bib24" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Huang et al. (2021)</span>
<span class="ltx_bibblock">
Feiqing Huang, Yuefeng Si, Yao Zheng, and Guodong Li. 2021.

</span>
<span class="ltx_bibblock"><a target="_blank" href="http://arxiv.org/abs/2112.04857" title="" class="ltx_ref ltx_href">A new measure of model
redundancy for compressed convolutional neural networks</a>.

</span>
</li>
<li id="bib.bib25" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Kaplan et al. (2020)</span>
<span class="ltx_bibblock">
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon
Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020.

</span>
<span class="ltx_bibblock"><a target="_blank" href="http://arxiv.org/abs/2001.08361" title="" class="ltx_ref ltx_href">Scaling laws for neural
language models</a>.

</span>
</li>
<li id="bib.bib26" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ko et al. (2024)</span>
<span class="ltx_bibblock">
Jongwoo Ko, Sungnyun Kim, Tianyi Chen, and Se-Young Yun. 2024.

</span>
<span class="ltx_bibblock">Distillm: Towards streamlined distillation for large language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib26.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2402.03898</em>.

</span>
</li>
<li id="bib.bib27" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lai et al. (2017)</span>
<span class="ltx_bibblock">
Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017.

</span>
<span class="ltx_bibblock">Race: Large-scale reading comprehension dataset from examinations.

</span>
<span class="ltx_bibblock">In <em id="bib.bib27.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2017 Conference on Empirical Methods in
Natural Language Processing</em>, pages 785–794.

</span>
</li>
<li id="bib.bib28" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">LeCun et al. (1989)</span>
<span class="ltx_bibblock">
Yann LeCun, John Denker, and Sara Solla. 1989.

</span>
<span class="ltx_bibblock">Optimal brain damage.

</span>
<span class="ltx_bibblock"><em id="bib.bib28.1.1" class="ltx_emph ltx_font_italic">Advances in neural information processing systems</em>, 2.

</span>
</li>
<li id="bib.bib29" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Li et al. (2024)</span>
<span class="ltx_bibblock">
Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan
Duan, and Timothy Baldwin. 2024.

</span>
<span class="ltx_bibblock"><a target="_blank" href="http://arxiv.org/abs/2306.09212" title="" class="ltx_ref ltx_href">Cmmlu: Measuring massive
multitask language understanding in chinese</a>.

</span>
</li>
<li id="bib.bib30" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Li et al. (2022)</span>
<span class="ltx_bibblock">
Shiyang Li, Jianshu Chen, Yelong Shen, Zhiyu Chen, Xinlu Zhang, Zekun Li, Hong
Wang, Jing Qian, Baolin Peng, Yi Mao, et al. 2022.

</span>
<span class="ltx_bibblock">Explanations from large language models make small reasoners better.

</span>
<span class="ltx_bibblock"><em id="bib.bib30.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2210.06726</em>.

</span>
</li>
<li id="bib.bib31" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Liu et al. (2021)</span>
<span class="ltx_bibblock">
Zhenhua Liu, Yunhe Wang, Kai Han, Wei Zhang, Siwei Ma, and Wen Gao. 2021.

</span>
<span class="ltx_bibblock">Post-training quantization for vision transformer.

</span>
<span class="ltx_bibblock"><em id="bib.bib31.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>,
34:28092–28103.

</span>
</li>
<li id="bib.bib32" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ma et al. (2024)</span>
<span class="ltx_bibblock">
Xinyin Ma, Gongfan Fang, and Xinchao Wang. 2024.

</span>
<span class="ltx_bibblock">Llm-pruner: On the structural pruning of large language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib32.1.1" class="ltx_emph ltx_font_italic">Advances in neural information processing systems</em>, 36.

</span>
</li>
<li id="bib.bib33" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Park et al. (2022)</span>
<span class="ltx_bibblock">
Gunho Park, Baeseong Park, Se Jung Kwon, Byeongwook Kim, Youngjoo Lee, and
Dongsoo Lee. 2022.

</span>
<span class="ltx_bibblock">nuqmm: Quantized matmul for efficient inference of large-scale
generative language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib33.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2206.09557</em>.

</span>
</li>
<li id="bib.bib34" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Peng et al. (2023)</span>
<span class="ltx_bibblock">
Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Huanqi
Cao, Xin Cheng, Michael Chung, Matteo Grella, Kranthi Kiran GV, et al. 2023.

</span>
<span class="ltx_bibblock">Rwkv: Reinventing rnns for the transformer era.

</span>
<span class="ltx_bibblock"><em id="bib.bib34.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2305.13048</em>.

</span>
</li>
<li id="bib.bib35" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Povey et al. (2018)</span>
<span class="ltx_bibblock">
Daniel Povey, Gaofeng Cheng, Yiming Wang, Ke Li, Hainan Xu, Mahsa Yarmohammadi,
and Sanjeev Khudanpur. 2018.

</span>
<span class="ltx_bibblock">Semi-orthogonal low-rank matrix factorization for deep neural
networks.

</span>
<span class="ltx_bibblock">In <em id="bib.bib35.1.1" class="ltx_emph ltx_font_italic">Interspeech</em>, pages 3743–3747.

</span>
</li>
<li id="bib.bib36" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Press et al. (2021)</span>
<span class="ltx_bibblock">
Ofir Press, Noah A Smith, and Mike Lewis. 2021.

</span>
<span class="ltx_bibblock">Train short, test long: Attention with linear biases enables input
length extrapolation.

</span>
<span class="ltx_bibblock"><em id="bib.bib36.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2108.12409</em>.

</span>
</li>
<li id="bib.bib37" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Rae et al. (2019)</span>
<span class="ltx_bibblock">
Jack W Rae, Anna Potapenko, Siddhant M Jayakumar, Chloe Hillier, and Timothy P
Lillicrap. 2019.

</span>
<span class="ltx_bibblock">Compressive transformers for long-range sequence modelling.

</span>
<span class="ltx_bibblock">In <em id="bib.bib37.1.1" class="ltx_emph ltx_font_italic">International Conference on Learning Representations</em>.

</span>
</li>
<li id="bib.bib38" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Reddy et al. (2019)</span>
<span class="ltx_bibblock">
Siva Reddy, Danqi Chen, and Christopher D Manning. 2019.

</span>
<span class="ltx_bibblock">Coqa: A conversational question answering challenge.

</span>
<span class="ltx_bibblock"><em id="bib.bib38.1.1" class="ltx_emph ltx_font_italic">Transactions of the Association for Computational Linguistics</em>,
7:249–266.

</span>
</li>
<li id="bib.bib39" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Samragh et al. (2023)</span>
<span class="ltx_bibblock">
Mohammad Samragh, Mehrdad Farajtabar, Sachin Mehta, Raviteja Vemulapalli,
Fartash Faghri, Devang Naik, Oncel Tuzel, and Mohammad Rastegari. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="http://arxiv.org/abs/2312.09299" title="" class="ltx_ref ltx_href">Weight subcloning: direct
initialization of transformers using larger pretrained ones</a>.

</span>
</li>
<li id="bib.bib40" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Su et al. (2024)</span>
<span class="ltx_bibblock">
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu.
2024.

</span>
<span class="ltx_bibblock">Roformer: Enhanced transformer with rotary position embedding.

</span>
<span class="ltx_bibblock"><em id="bib.bib40.1.1" class="ltx_emph ltx_font_italic">Neurocomputing</em>, 568:127063.

</span>
</li>
<li id="bib.bib41" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Sun et al. (2020)</span>
<span class="ltx_bibblock">
Kai Sun, Dian Yu, Dong Yu, and Claire Cardie. 2020.

</span>
<span class="ltx_bibblock">Investigating prior knowledge for challenging chinese machine reading
comprehension.

</span>
<span class="ltx_bibblock"><em id="bib.bib41.1.1" class="ltx_emph ltx_font_italic">Transactions of the Association for Computational Linguistics</em>,
8:141–155.

</span>
</li>
<li id="bib.bib42" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Syed et al. (2023)</span>
<span class="ltx_bibblock">
Aaquib Syed, Phillip Huang Guo, and Vijaykaarti Sundarapandiyan. 2023.

</span>
<span class="ltx_bibblock">Prune and tune: Improving efficient pruning techniques for massive
language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib42.1.1" class="ltx_emph ltx_font_italic">Arxiv</em>.

</span>
</li>
<li id="bib.bib43" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Touvron et al. (2023)</span>
<span class="ltx_bibblock">
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine
Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale,
et al. 2023.

</span>
<span class="ltx_bibblock">Llama 2: Open foundation and fine-tuned chat models.

</span>
<span class="ltx_bibblock"><em id="bib.bib43.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2307.09288</em>.

</span>
</li>
<li id="bib.bib44" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Vaswani et al. (2017)</span>
<span class="ltx_bibblock">
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones,
Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.

</span>
<span class="ltx_bibblock">Attention is all you need.

</span>
<span class="ltx_bibblock"><em id="bib.bib44.1.1" class="ltx_emph ltx_font_italic">Advances in neural information processing systems</em>, 30.

</span>
</li>
<li id="bib.bib45" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Xia et al. (2023)</span>
<span class="ltx_bibblock">
Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen. 2023.

</span>
<span class="ltx_bibblock">Sheared llama: Accelerating language model pre-training via
structured pruning.

</span>
<span class="ltx_bibblock"><em id="bib.bib45.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2310.06694</em>.

</span>
</li>
<li id="bib.bib46" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Xu et al. (2020)</span>
<span class="ltx_bibblock">
Liang Xu, Hai Hu, Xuanwei Zhang, Lu Li, Chenjie Cao, Yudong Li, Yechen Xu, Kai
Sun, Dian Yu, Cong Yu, et al. 2020.

</span>
<span class="ltx_bibblock">Clue: A chinese language understanding evaluation benchmark.

</span>
<span class="ltx_bibblock">In <em id="bib.bib46.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 28th International Conference on
Computational Linguistics</em>, pages 4762–4772.

</span>
</li>
<li id="bib.bib47" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Xu et al. (2023)</span>
<span class="ltx_bibblock">
Mingxue Xu, Yao Lei Xu, and Danilo P Mandic. 2023.

</span>
<span class="ltx_bibblock">Tensorgpt: Efficient compression of the embedding layer in llms based
on the tensor-train decomposition.

</span>
<span class="ltx_bibblock"><em id="bib.bib47.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2307.00526</em>.

</span>
</li>
<li id="bib.bib48" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Yang et al. (2023)</span>
<span class="ltx_bibblock">
Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu
Lv, Da Pan, Dian Wang, Dong Yan, et al. 2023.

</span>
<span class="ltx_bibblock">Baichuan 2: Open large-scale language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib48.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2309.10305</em>.

</span>
</li>
<li id="bib.bib49" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Yang et al. (2024)</span>
<span class="ltx_bibblock">
Yifei Yang, Zouying Cao, and Hai Zhao. 2024.

</span>
<span class="ltx_bibblock"><a target="_blank" href="http://arxiv.org/abs/2402.11187" title="" class="ltx_ref ltx_href">Laco: Large language model
pruning via layer collapse</a>.

</span>
</li>
<li id="bib.bib50" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Yang et al. (2019)</span>
<span class="ltx_bibblock">
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov,
and Quoc V Le. 2019.

</span>
<span class="ltx_bibblock">Xlnet: Generalized autoregressive pretraining for language
understanding.

</span>
<span class="ltx_bibblock"><em id="bib.bib50.1.1" class="ltx_emph ltx_font_italic">Advances in neural information processing systems</em>, 32.

</span>
</li>
<li id="bib.bib51" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zellers et al. (2019)</span>
<span class="ltx_bibblock">
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019.

</span>
<span class="ltx_bibblock">Hellaswag: Can a machine really finish your sentence?

</span>
<span class="ltx_bibblock">In <em id="bib.bib51.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 57th Annual Meeting of the Association
for Computational Linguistics</em>, pages 4791–4800.

</span>
</li>
<li id="bib.bib52" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhang et al. (2023)</span>
<span class="ltx_bibblock">
Mingyang Zhang, Chunhua Shen, Zhen Yang, Linlin Ou, Xinyi Yu, Bohan Zhuang,
et al. 2023.

</span>
<span class="ltx_bibblock">Pruning meets low-rank parameter-efficient fine-tuning.

</span>
<span class="ltx_bibblock"><em id="bib.bib52.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2305.18403</em>.

</span>
</li>
<li id="bib.bib53" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zheng et al. (2019)</span>
<span class="ltx_bibblock">
Chujie Zheng, Minlie Huang, and Aixin Sun. 2019.

</span>
<span class="ltx_bibblock">Chid: A large-scale chinese idiom dataset for cloze test.

</span>
<span class="ltx_bibblock">In <em id="bib.bib53.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 57th Annual Meeting of the Association
for Computational Linguistics</em>, pages 778–787.

</span>
</li>
<li id="bib.bib54" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhu et al. (2023)</span>
<span class="ltx_bibblock">
Xunyu Zhu, Jian Li, Yong Liu, Can Ma, and Weiping Wang. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="http://arxiv.org/abs/2308.07633" title="" class="ltx_ref ltx_href">A survey on model
compression for large language models</a>.

</span>
</li>
</ul>
</section>
<div class="ltx_pagination ltx_role_newpage"></div>
<section id="A1" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix A </span>Detailed Strategy for Layer Removal</h2>

<figure id="A1.T3" class="ltx_table">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 3: </span>Strategies for Layer Removal in Models.</figcaption>
<table id="A1.T3.1" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A1.T3.1.1.1" class="ltx_tr">
<th id="A1.T3.1.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_tt" style="padding-top:1pt;padding-bottom:1pt;">Strategy</th>
<th id="A1.T3.1.1.1.2" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top ltx_th ltx_th_column ltx_border_tt" style="padding-top:1pt;padding-bottom:1pt;">
<span id="A1.T3.1.1.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T3.1.1.1.2.1.1" class="ltx_p" style="width:227.6pt;">Description</span>
</span>
</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A1.T3.1.2.1" class="ltx_tr">
<th id="A1.T3.1.2.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;">Sequential</th>
<td id="A1.T3.1.2.1.2" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;">
<span id="A1.T3.1.2.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T3.1.2.1.2.1.1" class="ltx_p" style="width:227.6pt;">Layers are removed sequentially from the beginning of the model. The process starts with layer 0 and progressively includes more layers for removal (e.g., {0}, {0, 1}, …).</span>
</span>
</td>
</tr>
<tr id="A1.T3.1.3.2" class="ltx_tr">
<th id="A1.T3.1.3.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row" style="padding-top:1pt;padding-bottom:1pt;">Reverse-order</th>
<td id="A1.T3.1.3.2.2" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top" style="padding-top:1pt;padding-bottom:1pt;">
<span id="A1.T3.1.3.2.2.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T3.1.3.2.2.1.1" class="ltx_p" style="width:227.6pt;">This strategy involves starting from the model’s final layer and progressively removing layers in reverse order (e.g., {-1}, {-1, -2}, …).</span>
</span>
</td>
</tr>
<tr id="A1.T3.1.4.3" class="ltx_tr">
<th id="A1.T3.1.4.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row" style="padding-top:1pt;padding-bottom:1pt;">Relative Magnitude</th>
<td id="A1.T3.1.4.3.2" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top" style="padding-top:1pt;padding-bottom:1pt;">
<span id="A1.T3.1.4.3.2.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T3.1.4.3.2.1.1" class="ltx_p" style="width:227.6pt;">Layers are removed in ascending order based on their Relative Magnitude values. The removal process accumulates layers from those with the smallest to the largest values, mirroring the sequential strategy’s accumulation method.</span>
</span>
</td>
</tr>
<tr id="A1.T3.1.5.4" class="ltx_tr">
<th id="A1.T3.1.5.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb" style="padding-top:1pt;padding-bottom:1pt;">BI (Block Influence)</th>
<td id="A1.T3.1.5.4.2" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top ltx_border_bb" style="padding-top:1pt;padding-bottom:1pt;">
<span id="A1.T3.1.5.4.2.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T3.1.5.4.2.1.1" class="ltx_p" style="width:227.6pt;">Follows a similar accumulation approach as the Sequential strategy, but layers are ordered and removed according to their BI values, starting from the lowest and moving to the highest.</span>
</span>
</td>
</tr>
</tbody>
</table>
</figure>
</section>
<section id="A2" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix B </span>Benchmark Setup for Removed Layers</h2>

<div id="A2.p1" class="ltx_para">
<p id="A2.p1.1" class="ltx_p">Before delineating the specific layer removal configurations in the benchmarked models, a concise overview of the structural modifications is warranted. The Llama-2-7B and Baichuan-2-7B models, initially designed with 32 layers, have been streamlined by excising 9 layers, corresponding to a 28% reduction of the total layer framework. The 13B variants, originally constituted of 40 layers, experienced a decrement of 10 layers, marking a 25% diminution. The selection criterion for layer removal was based on ascending Block Influence (BI) values, ensuring a data-driven approach to minimization. This premise underpins the benchmark configurations.</p>
</div>
<figure id="A2.T4" class="ltx_table">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 4: </span>Setup of Removed Layers for Benchmark Models</figcaption>
<table id="A2.T4.1" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A2.T4.1.1.1" class="ltx_tr">
<th id="A2.T4.1.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_tt" style="padding-top:1pt;padding-bottom:1pt;">Model</th>
<th id="A2.T4.1.1.1.2" class="ltx_td ltx_nopad_r ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding-top:1pt;padding-bottom:1pt;">Removed Layers</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A2.T4.1.2.1" class="ltx_tr">
<th id="A2.T4.1.2.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;">Llama-2-7B</th>
<td id="A2.T4.1.2.1.2" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;">27, 26, 25, 28, 24, 29, 23, 21, 22</td>
</tr>
<tr id="A2.T4.1.3.2" class="ltx_tr">
<th id="A2.T4.1.3.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row" style="padding-top:1pt;padding-bottom:1pt;">Llama-2-13B</th>
<td id="A2.T4.1.3.2.2" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">33, 31, 32, 30, 29, 34, 28, 35, 27, 26</td>
</tr>
<tr id="A2.T4.1.4.3" class="ltx_tr">
<th id="A2.T4.1.4.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row" style="padding-top:1pt;padding-bottom:1pt;">Baichuan-2-7B</th>
<td id="A2.T4.1.4.3.2" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-top:1pt;padding-bottom:1pt;">26, 27, 25, 28, 24, 29, 23, 22, 30</td>
</tr>
<tr id="A2.T4.1.5.4" class="ltx_tr">
<th id="A2.T4.1.5.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb" style="padding-top:1pt;padding-bottom:1pt;">Baichuan-2-13B</th>
<td id="A2.T4.1.5.4.2" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_bb" style="padding-top:1pt;padding-bottom:1pt;">32, 31, 33, 30, 34, 29, 28, 35, 27, 26</td>
</tr>
</tbody>
</table>
</figure>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
<section id="A3" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix C </span>Layer Removal on Baichuan2-series Model</h2>

<figure id="A3.F9" class="ltx_figure"><img src="/html/2403.03853/assets/x11.png" id="A3.F9.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="368" height="249" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 9: </span>Pruning by different metrics on Baichuan2-series model.</figcaption>
</figure>
</section>
<section id="A4" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix D </span>Layer Removal on HumanEval.</h2>

<div id="A4.p1" class="ltx_para">
<p id="A4.p1.1" class="ltx_p">We also found that layer removal has a significant impact on the performance of the model in generative tasks. Specifically, we conducted experiments on the Baichuan2-13B model on HumanEval <cite class="ltx_cite ltx_citemacro_citep">(Chen et al., <a href="#bib.bib5" title="" class="ltx_ref">2021</a>)</cite>, which is a dataset used to evaluate the performance of code generation models. The results are shown in Figure 10. After removing 10 layers, the performance on HumanEval almost completely degraded. We speculate that it may be due to significant cumulative errors in generative tasks.</p>
</div>
<figure id="A4.F10" class="ltx_figure"><img src="/html/2403.03853/assets/x12.png" id="A4.F10.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="368" height="176" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 10: </span> Performance degragation on generation task.</figcaption>
</figure>
</section>
</article>
</div>
<div class="ar5iv-footer"><a href="/html/2403.03852" class="ar5iv-nav-button ar5iv-nav-button-prev">◄</a>
    <a class="ar5iv-home-button" href="/"><img height="40" alt="ar5iv homepage" src="/assets/ar5iv.png"></a>
    <a href="/feeling_lucky" class="ar5iv-text-button">Feeling<br>lucky?</a>
    <a href="/log/2403.03853" class="ar5iv-text-button ar5iv-severity-warning">Conversion<br>report</a>
    <a class="ar5iv-text-button" target="_blank" href="https://github.com/dginev/ar5iv/issues/new?template=improve-article--arxiv-id-.md&title=Improve+article+2403.03853">Report<br>an issue</a>
    <a href="https://arxiv.org/abs/2403.03853" class="ar5iv-text-button arxiv-ui-theme">View&nbsp;original<br>on&nbsp;arXiv</a><a href="/html/2403.03854" class="ar5iv-nav-button ar5iv-nav-button-next">►</a>
</div><footer class="ltx_page_footer">
<a class="ar5iv-toggle-color-scheme" href="javascript:toggleColorScheme()" title="Toggle ar5iv color scheme"><span class="color-scheme-icon"></span></a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/license" target="_blank">Copyright</a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/policies/privacy_policy" target="_blank">Privacy Policy</a>

<div class="ltx_page_logo">Generated  on Fri Apr  5 17:35:43 2024 by <a target="_blank" href="http://dlmf.nist.gov/LaTeXML/" class="ltx_LaTeXML_logo"><span style="letter-spacing:-0.2em; margin-right:0.1em;">L<span class="ltx_font_smallcaps" style="position:relative; bottom:2.2pt;">a</span>T<span class="ltx_font_smallcaps" style="font-size:120%;position:relative; bottom:-0.2ex;">e</span></span><span style="font-size:90%; position:relative; bottom:-0.2ex;">XML</span><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAsAAAAOCAYAAAD5YeaVAAAAAXNSR0IArs4c6QAAAAZiS0dEAP8A/wD/oL2nkwAAAAlwSFlzAAALEwAACxMBAJqcGAAAAAd0SU1FB9wKExQZLWTEaOUAAAAddEVYdENvbW1lbnQAQ3JlYXRlZCB3aXRoIFRoZSBHSU1Q72QlbgAAAdpJREFUKM9tkL+L2nAARz9fPZNCKFapUn8kyI0e4iRHSR1Kb8ng0lJw6FYHFwv2LwhOpcWxTjeUunYqOmqd6hEoRDhtDWdA8ApRYsSUCDHNt5ul13vz4w0vWCgUnnEc975arX6ORqN3VqtVZbfbTQC4uEHANM3jSqXymFI6yWazP2KxWAXAL9zCUa1Wy2tXVxheKA9YNoR8Pt+aTqe4FVVVvz05O6MBhqUIBGk8Hn8HAOVy+T+XLJfLS4ZhTiRJgqIoVBRFIoric47jPnmeB1mW/9rr9ZpSSn3Lsmir1fJZlqWlUonKsvwWwD8ymc/nXwVBeLjf7xEKhdBut9Hr9WgmkyGEkJwsy5eHG5vN5g0AKIoCAEgkEkin0wQAfN9/cXPdheu6P33fBwB4ngcAcByHJpPJl+fn54mD3Gg0NrquXxeLRQAAwzAYj8cwTZPwPH9/sVg8PXweDAauqqr2cDjEer1GJBLBZDJBs9mE4zjwfZ85lAGg2+06hmGgXq+j3+/DsixYlgVN03a9Xu8jgCNCyIegIAgx13Vfd7vdu+FweG8YRkjXdWy329+dTgeSJD3ieZ7RNO0VAXAPwDEAO5VKndi2fWrb9jWl9Esul6PZbDY9Go1OZ7PZ9z/lyuD3OozU2wAAAABJRU5ErkJggg==" alt="Mascot Sammy"></a>
</div></footer>
</div>

    <script>
      var canMathML = typeof(MathMLElement) == "function";
      if (!canMathML) {
        var body = document.querySelector("body");
        body.firstElementChild.setAttribute('style', 'opacity: 0;');
        var loading = document.createElement("div");
        loading.setAttribute("id", "mathjax-loading-spinner");
        var message = document.createElement("div");
        message.setAttribute("id", "mathjax-loading-message");
        message.innerText = "Typesetting Equations...";
        body.prepend(loading);
        body.prepend(message);

        var el = document.createElement("script");
        el.src = "https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js";
        document.querySelector("head").appendChild(el);

        window.MathJax = {
          startup: {
            pageReady: () => {
              return MathJax.startup.defaultPageReady().then(() => {
                body.removeChild(loading);
                body.removeChild(message);
                body.firstElementChild.removeAttribute('style');
              }); } } };
      }
    </script>
    <script>
    // Auxiliary function, building the preview feature when
    // an inline citation is clicked
    function clicked_cite(e) {
      e.preventDefault();
      let cite = this.closest('.ltx_cite');
      let next = cite.nextSibling;
      if (next && next.nodeType == Node.ELEMENT_NODE && next.getAttribute('class') == "ar5iv-bibitem-preview") {
        next.remove();
        return; }
      // Before adding a preview modal,
      // cleanup older previews, in case they're still open
      document.querySelectorAll('span.ar5iv-bibitem-preview').forEach(function(node) {
        node.remove();
      })

      // Create the preview
      preview = document.createElement('span');
      preview.setAttribute('class','ar5iv-bibitem-preview');
      let target = document.getElementById(this.getAttribute('href').slice(1));
      target.childNodes.forEach(function (child) {
        preview.append(child.cloneNode(true));
      });
      let close_x = document.createElement('button');
      close_x.setAttribute("aria-label","Close modal for bibliography item preview");
      close_x.textContent = "×";
      close_x.setAttribute('class', 'ar5iv-button-close-preview');
      close_x.setAttribute('onclick','this.parentNode.remove()');
      preview.append(close_x);
      preview.querySelectorAll('.ltx_tag_bibitem').forEach(function(node) {
        node.remove();
      });
      cite.parentNode.insertBefore(preview, cite.nextSibling);
      return;
    }
    // Global Document initialization:
    // - assign the preview feature to all inline citation links
    document.querySelectorAll(".ltx_cite .ltx_ref").forEach(function (link) {
      link.addEventListener("click", clicked_cite);
    });
    </script>
    </body>
</html>

2403.03853
As Large Language Models (LLMs)  continue to advance in performance, their size has escalated significantly, with current LLMs containing billions or even trillions of parameters. However, in this study, we discovered that many layers of LLMs exhibit high similarity, and some layers play a negligible role in network functionality. Based on this observation, we define a metric called Block Influence (BI)  to gauge the significance of each layer in LLMs. We then propose a straightforward pruning approach: layer removal, in which we directly delete the redundant layers in LLMs based on their BI scores. Experiments demonstrate that our method, which we call ShortGPT, significantly outperforms previous state-of-the-art (SOTA)  methods in model pruning. Moreover, ShortGPT is orthogonal to quantization-like methods, enabling further reduction in parameters and computation. The ability to achieve better results through simple layer removal, as opposed to more complex pruning techniques, suggests a high degree of redundancy in the model architecture.
abstract
ShortGPT: Layers in Large Language Models are More Redundant Than You Expect
Abstract
2403.03853
1IntroductionThe field of large language models (LLMs)  has witnessed rapid development recently, with LLMs achieving impressive performance across various domains. Guided by the scaling laws identified in prior work, current LLM research has tended to increase model parameters consistently to boost performance. However, the resulting massive models containing billions or even trillions of parameters pose stringent hardware requirements, hindering their practical deployment and use.To alleviate the demanding hardware requirements for deploying massive trained models, many researchers have focused on model compression techniquesthat reduce the inference cost. Model compression methods can be broadly categorized into two categories: quantizationand pruning. Quantization methods quantize the weights and activations to lower precision. However, the acceleration benefits of quantization are dependent on hardware support and sometimes require additional fine-tuning to maintain good performance. In contrast, pruning methods remove redundant model parameters to reduce the overall parameter count. Pruning can be directly applied to a trained model without retraining and is generally more hardware-friendly than quantization approaches.While recently there has been significant progress in large language model compression methods based on model pruning, existing methods are often designed to be relatively complex. Some require the use of gradient informationor only focus on compression in width, rendering them overly sophisticated for practical applications. Therefore, there is a need to explore simple and efficient model pruning approaches tailored specifically for large language models.Identifying effective model pruning techniques necessitates studying model redundancy. Previous research on model redundancy has typically focused on relatively smaller models, such as convolutional neural networks (CNNs)  or small-sized Transformers. For model pruning in LLMs, most previous works have concentrated on tensor-wise redundancy analysis, investigating the redundancy within each parameter tensor. However, a key finding in this paper is that LLMs exhibit significant redundancy at the layer level, enabling the simple removal of entire layers without substantially impacting downstream task performance. For instance, when deleting the last 10 layers (25% of the total 40 layers)  from the LLaMA 2-13B model, the results on the MMLU benchmarkonly drop from 55.0 to 52.2 (a 95% retention) . Furthermore, by removing the last 22 layers (55% of the total 40 layers) , resulting in a 5.6 billion parameter model, we can still obtain a score of 47.2 on MMLU without any fine-tuning, even outperforming the LLaMA 2-7B model.In this paper, we propose to analyze the layer-wise redundancy through the lens of Block Influence (BI) , which measures the hidden states transformations during the modeling process of LLMs. We find BI is a more relevant indicator of the layer’s importance in LLMs, and we can perform model pruning by simply deleting redundant layers with BI. This straightforward act of removing specific layers significantly outperforms previous more complex pruning methods. Our findings highlight substantial redundancy in current LLM architectures and shed light on opportunities for more efficient LLM training in the future.The main contributions of our paper are summarized as follows:•We analyze the redundancy in large language models (LLMs)  and find that they exhibit significant redundancy at the layer level. This insight inspires us to prune LLMs by simply removing redundant layers.•We propose a metric called Block Influence (BI)  as an effective indicator of layer importance. Through a quantitative analysis, we demonstrate that LLMs possess redundancy both in depth (layers)  and width (parameters within layers) .•Based on the BI metric, we propose a simple yet effective pruning strategy by removing layers with low BI scores. Experimental results show that our method maintains 92% performance while reducing approximately 25% of the parameters and computation, outperforming previous state-of-the-art methods.•Furthermore, we demonstrate that our layer pruning approach is orthogonal to quantization methods, meaning it can be combined with quantization techniques to further reduce the deployment overhead of LLMs.(a) Layer pruning order. We first calculate the BI score of each layer in LLM, and then prune the redundant layer in ascending BI score.(b) Performance of ShortGPT with increasing pruning ratio.Figure 2:Our Method: firstly, we sort layers by their BI score (introduced in Sec.LABEL:layerimportacne) , and then delete layers in ascending order. All results are based on Llama2-7B-Base.
introduction
ShortGPT: Layers in Large Language Models are More Redundant Than You Expect
1Introduction
2403.03853
2MethodologyIn this section, we present the methodological framework of our layer deletion approach for LLMs, elucidating the underlying principles and techniques employed. We first quantify the layer redundancy problem present in current prominent LLMs, such as LLaMA 2 and Baichuan 2. Then, we introduce a metric called Block Influence (BI) , designed to assess the transformation of hidden states by each layer during LLM inference. Building upon BI, we apply layer deletion to LLMs, thereby reducing their inference cost without compromising their predictive accuracy or linguistic capabilities.2.1Layer redundancyThe predominant LLMs in current are primarily based on the Transformer. The transformer architecture is based on an attention mechanism, typically consisting of several residual layers stacked upon each other. Transformer is a sequence to sequence mapping, which can be defined as, where,,is the length of the sequence,is the vocabulary size,is the learnable parameters. The formal expression of an L-layer transformer is as follows:Y(1) whereis the word embedding matrix,is the output projection matrix of the transformer, which are sometimes tied with the,is the hidden dim of the transformer.ATTNrefers to the attention layer andFFNmeans the feed-forward layers,is the hidden states of thelayers.
Given that the transformer structure is composed of several identical layers, a natural question is what the differences and connections between the functionalities and roles of these identical layers are. Previous works found that the Transformer possesses certain semantic capabilities in earlier layers. In this work, we uncover a significant level of redundancy between the layers of the transformer.Layer Redundancy: A network with high redundancy should contain some redundant layers, which have a minimal impact on the network’s final performance. This may be due to these layers having homogenized functionalities compared to other layers in the network. We have found a high degree of redundancy in the current LLMs by omitting specific layers.(a) Llama2-7B-Base(b) Baichuan2-7B-BaseFigure 3:Performance of removing single layer.We observed that in many cases, the network’s prediction ofremains largely unchanged. This phenomenon demonstrates a significant level of redundancy across the layers of the transformer. Figure3shows the perplexity and MMLUscore of omitting a certain layer in the Llama2-7B-Base, an English based LLMs, and Baichuan2-7B-Basewhich is mainly focused on Chinese. For more benchmark results, please refer to the section3. Figure3reveals that some layers do not play a crucial role in LLMs. Dropping a certain layer from the LLMs may have a minimal effect on the final results, and the same holds for the perplexity. Moreover, this redundancy is primarily manifested in the middle to later layers of the network, with the initial layers and the final layer often being more critical.Figure 4:Illustration of our Block Influence(BI)  metric. We omit LayerNorm for visual conciseness.2.2Layer importanceAs outlined in the preceding section, the layers of LLMs exhibit redundancy, with varying degrees of redundancy across different layers. Notably, the deeper layers demonstrate a higher level of redundancy. To remove these redundant layers, we need a model-intrinsic metric to measure the importance of the layer. An intuitive method is to use the magnitude of each layer’s output as a measure of its importance, as a larger magnitude implies more activated neurons. In, they proposed to use relative magnitudeto measure the importance of layers. To characterize the influence of a certain layer. We defined a new metric, Block Influence (BI) , under the assumption that the more a transformer block changes the hidden states, the more influential this layer is. As Figure4shows, the BI score ofblocks can be calculated as follows:(2) wheremeans therow of.
Our empirical evidence supports that BI effectively reflects the importance of a layer. Figure8shows these different metrics in detail.2.3Layer RemovalThe method we propose is straightforward: layer removal, which we simply referred as ShortGPT. We delete certain layers in LLMs based on BI score. Figure2illustrates our pruning method. First of all, we construct a calibration set, which is a set of unlabelled text samples such as PG19.
Then we collect the hidden states of each layer during inference on these samples. Next, we calculate the BI score based on the collected hidden states. Finally, we sort layers in ascending order according to the BI, and delete the layers with the smaller importance. The number of layers to be deleted can vary to trade off the speed and performance. The details of our layer remove setting can be found in AppendixB
body
ShortGPT: Layers in Large Language Models are More Redundant Than You Expect
2Methodology
2403.03853
3Experiments3.1Experimental Setup3.1.1ModelsTo validate the effectiveness of our method, we conducted experiments on existing popular open-source language models, including Llama2-7B, Llama2-13B, Baichuan2-7B, and Baichuan2-13B. They are all large language models based on the decoder-only Transformer architecture. LLaMA 2 was trained on more than 2 trillion tokens. Baichuan-series was mainly trained in Chinese and its 13-Billion model replaced the RoPEpositional embedding with ALiBi.3.1.2BenchmarksIn order to comprehensively evaluate the changes in the ability of large language models before and after pruning, we conducted evaluations on the most commonly used Benchmark MMLU, CMMLUfor evaluating large models. In addition, we also followed LaCoto evaluate a wider dataset.MMLUis a benchmark aimed at measuring the knowledge acquired during pre-training by specifically evaluating models in zero-shot and few-shot settings. This makes benchmarks more challenging and similar to the way we evaluate humans. This benchmark covers 57 subjects including STEM, humanities, social sciences, etc. Its difficulty ranges from beginner to advanced professional level, and it tests world knowledge and problem-solving ability.CMMLUis a comprehensive Chinese language assessment dataset designed specifically to evaluate LLM’s advanced knowledge and reasoning abilities in the context of Chinese language and culture. CMMLU covers 67 topics, from elementary school to university or professional level. Including natural sciences, as well as humanities and social sciences, it also includes many contents with Chinese characteristics.CMNLIis part of the Chinese language understanding assessment benchmark. It consists of two parts: XNLI and MNLI.HellaSwag (HeSw) is a challenging dataset for evaluating commonsense NLI that is especially hard for state-of-the-art models, though its questions are trivial for humans.PIQAis a multi-choice question and answer dataset that focuses on daily scenarios. This dataset explores the model’s grasp of the laws of the real physical world through daily scenarios.CHIDis an idiom cloze test dataset that mainly focuses on the selection of candidate words and the representation of idioms.CoQAis a large-scale dataset used for conversational question-answering tasks, containing over 127000 questions and their corresponding answers.BoolQis a question-answer dataset containing 15942 examples of yes/no questions. These problems occur naturally - they are generated in an environment that is silent and unconstrained.Raceis a large-scale reading comprehension dataset collected from English examinations in China, which are designed for middle school and high school students.XSumis used to evaluate abstract single document summarization systems. The goal is to create a short, one-sentence new summary of what the article is about.C3is a machine reading comprehension dataset with multiple choices, consisting of multiple-choice questions, reading materials from Chinese proficiency exams, and ethnic Chinese exams.PG19is a long document dataset from books used to test the effectiveness of language modeling.3.1.3BaselinesTo evaluate the effectiveness of our method, we compared several structured pruning methods for large language models followed by LaCo. For our method, we use PG19 for layer importance and perplexity calculation.LLMPru.adopts structural pruning that selectively removes non-critical coupled structures based on gradient information, maximally preserving the majority of the LLM’s functionality. LLMPru. applies post training to the pruned model, but for fair comparison, we do not apply post training to it.SliceGPTis a post-training sparsification scheme that replaces each weight matrix with a smaller matrix, reducing the embedding dimension of the network. Specifically, they applied PCA to the hidden representation from shallow to deep layers, and incorporated the dimension reduction matrix into existing network parameters.LaCois a pruning method for large language models based on reducing layers. LaCo gradually merges similar layers from deep to shallow and sets a threshold to avoid continuously merging too many layers.LLMMethodRatioBenchmarksAve.Per.CMNLIHeSwPIQACHIDWSCCoQABoolQRace-HRace-MXSumC3MMLUCMMLULlama2-7BDense0.00%32.9971.2677.9141.6650.0064.6271.6235.7134.1919.4043.5645.3932.9244.52100.00LLMPrun.27.0%34.3356.4671.2225.2536.5442.5155.2022.5622.3511.5125.6423.3325.2532.8473.76SliceGPT26.4%31.7050.2766.2120.7936.5441.3638.3221.0721.664.8939.7828.9225.3732.8473.76LaCo27.1%34.4355.6969.8036.1440.3845.7064.0722.6123.6115.6439.6726.4525.2438.4186.28ShortGPT27.1%32.9553.0266.4324.6852.4647.9974.7132.2535.170.6739.6243.9632.2542.6095.69Llama2-13BDense0.00%32.9974.7879.7147.3550.0066.9182.3957.9560.3823.4547.5155.0038.4051.91100.00LLMPrun.24.4%33.0367.7676.6635.6440.3850.8656.4222.4722.0819.1732.3325.2124.7138.9775.07SliceGPT23.6%29.8255.7169.0419.3136.5447.2637.8623.4124.035.2741.9237.1425.7934.8467.11LaCo24.6%32.8664.3974.2740.1052.8852.6663.9854.4956.5514.4544.9345.9332.6248.3093.05ShortGPT24.6%33.0066.6473.4536.6150.0058.6462.4858.3560.1717.5946.9054.6938.3850.5397.34Baichuan2-7BDense0.00%33.3767.5676.1785.5650.0063.1474.1026.9624.0920.8264.5553.8756.9553.63100.00LLMPrun.24.2%32.2853.6671.8269.8053.8547.8361.1921.9622.2815.9841.6424.9325.6941.7677.87SliceGPT22.2%32.0725.2950.3314.8536.5419.5739.3023.5322.490.0026.5825.1825.2526.2356.38LaCo24.2%33.0052.2868.5076.2442.3147.2656.1528.9927.7212.0350.8531.5331.2442.9380.05ShortGPT24.2%33.3056.9667.6865.6350.0046.7067.8353.2646.760.0456.3345.7747.8749.0891.52Baichuan2-13BDense0.00%33.2171.1078.0786.5150.0065.677.8967.2768.9425.0265.6459.5061.3062.31100.00LLMPrun.24.3%33.8053.5771.8272.7737.5038.8256.5421.1721.6113.6739.8923.1925.1839.2062.91SliceGPT22.8%32.0725.8551.0310.4036.5418.0237.8321.5621.520.0024.9922.9525.2625.0340.17LaCo24.7%33.0360.7168.8876.7344.2355.4562.3556.9257.8012.3261.1051.3553.6553.4385.75ShortGPT24.7%32.8160.5571.6080.1747.1354.3062.5455.7756.4115.1460.1652.1158.8654.4387.33Table 1:Comparison of pruning methods on multiple natural language benchmarks. The results of LLMPrun., SliceGPT and LaCo are reported from LaCo.3.2Main ResultsTo validate the efficacy of our proposed method, we conducted comparative experiments against benchmark and baseline techniques commonly employed in large language model evaluation. Considering the current structured pruning methods generally reduce parameters by no more than 30%, we performed experiments with approximately 1/4 of the parameters pruned. The experimental results are presented in Table1. Additional experiments exploring different parameter reduction proportions will be discussed in the subsequent section.The results demonstrate that the performance of the model pruned by our method significantly surpasses that of the baseline method, maintaining most of the large language model’s capabilities, such as reasoning, language understanding, knowledge retention, and examination performance, to the greatest extent possible. Furthermore, we note that the approach of reducing the number of layers (ShortGPT/LaCo)  outperforms the method of reducing the embedding dimensions (LLMPru./SliceGPT) , implying that the model exhibits more redundancy in-depth than in width. Further experimental analysis will be presented in the ensuing section.
body
ShortGPT: Layers in Large Language Models are More Redundant Than You Expect
3Experiments
2403.03853
4Analysis4.1Varying prune ratioFigure 5:Pruning by different metrics, with increasing pruning ratio.As a pruning method, we further validated the effects of different pruning ratios on model performance. Experiments were conducted on the Llama2-7B-Base and Baichuan2-7B-Base models, observing the Perplexity and MMLU. The pruning ratios ranged from 0% to 97%, employing the strategy of deleting layers based on the importance proposed in this paper, resulting in models with 1 to 32 layers. The results in Figure5indicate that as the pruning ratio increases, the performance of the model declines. However, the MMLU score drops significantly at a certain layer, which may imply the existence of certain special layers within the network that play a critically important role.4.2Depth Redundancy v.s. Width RedundancyFigure 6:Redundancy on width, removing a single head from model. The x-axis is the head id.In the preceding sections, we analyzed and explored the redundancy of large language models (LLMs)  in terms of depth (layers) . However, we also investigated redundancy in terms of width, specifically focusing on the attention heads. Our method involves sequentially removing each head from the multi-head attention mechanism and adjusting the parameters of the attention output projection to ensure the output shape of the Attention block remains unchanged. The Feed-Forward Network (FFN)  remains unaltered. Similar to the approach in Section2.1, we observe the impact of each head’s removal on the final performance of the network. Figure6illustrates the changes in Perplexity and MMLU scores after each head is removed. The results indicate that LLMs exhibit a high degree of redundancy in width, comparable to their redundancy in depth. However, this width redundancy does not follow any discernible pattern and varies across different models. We hypothesize that this variation may be attributed to the symmetry among heads.4.3Redundancy on non-transformer LLMFigure 7:Redundancy on RWKV, removing single layers on RWKV-7B.We further investigate whether the redundancy observed in depth is a result of the Transformer’s architecture. Given that the vast majority of popular LLMs are based on the Transformer structure. We selected the RWKV-7B model proposed in, a formidable competitor to the Transformer architecture which indeed consists of identical layers to a certain extent. We analyzed this model using the methodology from2.1. Figure7shows the redundancy of the RWKV-7B111We use rwkv-v5-world-7B from https://huggingface.co/BlinkDL/rwkv-5-worldmodel. Through this figure, we can observe that the RWKV-7B model also exhibits a high level of redundancy. This may suggest that the redundancy is universal across LLMs.4.4Importance metricFigure 8:Different importance metrics. Perplexity is calculated by removing each single layer, other metrics is calculated by hidden states of each layerThe overarching concept of our method is to rank layers by importance and remove the less significant ones. The definition of importance metrics has a profound impact on the outcome. In this section, we define and compare various important metrics:Sequential: The importance is directly proportional to the sequence order, with shallower layers being more important. This can be implemented by taking the negative of each layer’s index as the importance metric.Reverse-order/Norm: This metric posits that importance is inversely proportional to the sequence order. It assigns higher importance scores to the shallower layers Inspired by Figure3. This method gives the same order as measuring importance by hidden states norm mentioned inLABEL:layerimportacne. Here, the importance metric is the layer’s index itself.Relative Magnitude: Proposed in, this metric assumes layers with largerare of higher importance, whereis the layer transformation function.BI: BI mentioned in previous section2.2.Figure8demonstrates the different metrics. We observe that shallower layers in the LLM network are more crucial than deeper ones. Figure5shows the results of removing layers by different metrics, demonstrating that Our proposed BI outperforms other metrics. The method ofis highly competitive, indicating that relative values can also reflect the importance to a certain extent. It is worth noting that the hidden states norm seems to be a good metric when only considering the MMLU benchmark, but the perplexity is relatively poor, which significantly impacts tasks related to text generation. This also indicates the limitations of the current LLMs evaluation methods.4.5Orthogonal to QuantizationIn this section, we show that our method is orthogonal to quantization methods. We apply our method to a model222We take the model from https://huggingface.co/TheBloke/Llama-2-7B-GPTQquantized by GPTQ algorithm. Table2shows that our method is compatible with the quantization-like method, which could reduce the footprint further.ModelRatio/LayerPerplexityMMLUBaseline0%/328.499937.99ShortGPT27.1%/2342.695136.69Table 2:Layer removal results on Llama2-7B-Base-GPTQ.
body
ShortGPT: Layers in Large Language Models are More Redundant Than You Expect
4Analysis
2403.03853
5LimitationAlthough our method demonstrates strong competitiveness compared to current pruning methods, with some benchmarks even suggesting that deleting certain layers has no impact on the final result, this does not imply that layer removal comes without drawbacks. Due to the limitations of benchmarking, current evaluations may not fully capture the impact of layer removal on model performance. Our experiments reveal that the effect of layer removal is significantly more pronounced on generative tasks compared to multiple-choice tasks. On benchmarks such as GSM8Kand HumanEval, removing 25% of the layers often leads to a severe performance drop, with scores approaching zero. Nevertheless, this study proves that models maintain strong semantic understanding and processing capabilities after layer removal. Post-training techniques could potentially further mitigate performance losses, which is an area of ongoing research.
body
ShortGPT: Layers in Large Language Models are More Redundant Than You Expect
5Limitation
2403.03853
6Related worksTo reduce the inference cost of large language models and increase their practical applications, there have been many recent works on compressing models, which can be classified into four categories: model pruning, knowledge distillation, quantization, and low-rank factorization.Model pruning:model pruningis a classic and effective method of reducing model redundancy modules to compress models. The model pruning methods mainly include unstructured pruning and structured pruning. The unstructured pruning simplifies an LLM by removing specific parameters without considering its internal structure, such as SparseGPTand LoRAPrune. However, this method disregards the overall LLM structure, resulting in an irregular sparse model composition. Another more practical approach is structured pruning, GUMmakes an analysis of several structured pruning methods for decoder-only LLMs. LLM-Prunerselectively removes non-critical structures according to gradient information. ShearedLLaMAemploys targeted structured pruning and dynamic batch loading. LaCoused layer merging to compress the model. Compared to the previous method, our method is a simple and efficient structured pruning method.Knowledge distillation:Knowledge distillation (KD) is another way of compressing models, where a larger teacher network provides knowledge to a smaller student network. Using the prompt response pairs by LLM APIs to fine-tune small models can get promised results. In addition, when we can access transparent teacher models, the result can be improved more. MiniLLMuses reverse Kullback-Leibler divergence to prevent the student model from overestimating the low-probability regions of the teacher distribution. DistlLLMuses an adaptive off-policy approach designed to enhance the efficiency in utilizing student-generated outputs. However, compared to model pruning, this method often requires higher computational resources.Quantization:quantizationis a widely accepted technique in the field of model compression, which can significantly save the storage and computational costs of deep learning models. Traditional models are generally stored as floating-point numbers, but quantization converts them into integers or other discrete forms. LUT-GEMMquantifies only weights and optimizes matrix multiplication in LLM using BCQ format. SPQRidentifies and isolates abnormal weights, stores them with higher accuracy, and compresses all other weights into 3-4 bits. Our model pruning method and quantization method are orthogonal, which means quantification based on our pruned model can further compress the model.Low-rank factorization:low-rank decompositionis a model compression technique aimed at approximating a given weight matrix by decomposing it into two or more smaller matrices with significantly lower dimensions. TensorGPTstores large embeddings in a low-rank tensor format, reducing the space complexity of LLMs and making them available on edge devices. Recently, SliceGPThas also implemented structured compression of models based on matrix factorization of hidden states and absorbed the new matrix into the parameters of the origin networks.Model redundancy:researchers have long noticed the significant redundancy in nonlinear models. In recent years, the transformer model architecture has been widely applied, and researchers have also studied its redundancy. In, researchers analyzed redundancy in attention mechanisms, in which clear and similar redundancy patterns (cluster structure)  are observed among attention heads. In, researchers dissect two pre-trained models, BERTand XLNet, studying how much redundancy they exhibit at a representation level and a more fine-grained neuron-level. However, the redundancy in current large language models based on decoder-only structures still needs to be explored.
body
ShortGPT: Layers in Large Language Models are More Redundant Than You Expect
6Related works
2403.03853
7ConclusionThis work has introduced a novel approach to pruning Large Language Models (LLMs)  based on layer redundancy and an "importance" metric defined as attention entropy. Our study reveals a significant degree of layer-wise redundancy in LLMs, indicating that certain layers contribute minimally to the overall network function and can thus be removed without substantially compromising model performance. By employing a straightforward layer removal strategy guided by the calculated importance of each layer, we have demonstrated that it is possible to maintain up to 95% of an LLM’s performance while reducing the model’s parameter count and computational requirements by approximately 25%. This achievement not only surpasses previous state-of-the-art pruning methods but also underscores the potential for further optimizations in LLM deployment strategies.Our findings suggest that the redundancy inherent in LLMs is largely depth-based rather than width-based, highlighting an avenue for future research into the structural efficiency of neural networks. Moreover, our pruning approach, characterized by its simplicity and effectiveness, is compatible with other compression techniques such as quantization, offering a composite pathway to model size reduction that is both significant and versatile.The implications of our research extend beyond academic interest, offering practical solutions for deploying advanced LLMs in resource-constrained environments. By enabling more efficient model architectures without the need for extensive retraining, our pruning method facilitates broader access to cutting-edge AI capabilities across a variety of platforms and devices.In conclusion, our investigation into layer redundancy and the development of an importance-based pruning strategy represent a meaningful advancement in the optimization of Large Language Models. As the demand for sophisticated AI tools continues to grow, approaches like ours will play a crucial role in making these technologies more accessible and sustainable. Future work will focus on refining our understanding of model redundancy and exploring additional methods for enhancing the efficiency of neural network models.
conclusion
ShortGPT: Layers in Large Language Models are More Redundant Than You Expect
7Conclusion