<!DOCTYPE html><html lang="en">
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<title>[2309.05516] Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs</title><meta property="og:description" content="Large Language Models (LLMs) have proven their exceptional capabilities in performing language-related tasks. However, their deployment poses significant challenges due to their considerable memory and storage requirem…">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="twitter:card" content="summary">
<meta name="twitter:title" content="Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs">
<meta name="twitter:image:src" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta name="twitter:image:alt" content="ar5iv logo">
<meta property="og:title" content="Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs">
<meta property="og:site_name" content="ar5iv">
<meta property="og:image" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta property="og:type" content="article">
<meta property="og:url" content="https://ar5iv.labs.arxiv.org/html/2309.05516">

<!--Generated on Wed Feb 28 07:06:44 2024 by LaTeXML (version 0.8.8) http://dlmf.nist.gov/LaTeXML/.-->
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">

<script>
  function detectColorScheme(){
    var theme="light";
    var current_theme = localStorage.getItem("ar5iv_theme");
    if(current_theme){
      if(current_theme == "dark"){
        theme = "dark";
      } }
    else if(!window.matchMedia) { return false; }
    else if(window.matchMedia("(prefers-color-scheme: dark)").matches) {
      theme = "dark"; }
    if (theme=="dark") {
      document.documentElement.setAttribute("data-theme", "dark");
    } else {
      document.documentElement.setAttribute("data-theme", "light"); } }

  detectColorScheme();

  function toggleColorScheme(){
    var current_theme = localStorage.getItem("ar5iv_theme");
    if (current_theme) {
      if (current_theme == "light") {
        localStorage.setItem("ar5iv_theme", "dark"); }
      else {
        localStorage.setItem("ar5iv_theme", "light"); } }
    else {
        localStorage.setItem("ar5iv_theme", "dark"); }
    detectColorScheme(); }
</script>
<link media="all" rel="stylesheet" href="/assets/ar5iv-fonts.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv-site.0.2.2.css">
</head>
<body>
<div class="ltx_page_main">
<div class="ltx_page_content">
<article class="ltx_document ltx_authors_1line">
<h1 class="ltx_title ltx_title_document">Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs</h1>
<div class="ltx_authors">
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Wenhua Cheng  , Weiwei Zhang, Haihao Shen, Yiyang Cai, Xin He &amp; Kaokao Lv 
<br class="ltx_break">Intel
</span><span class="ltx_author_notes"> Corresponding author wenhua.cheng@intel.com</span></span>
</div>

<div class="ltx_abstract">
<h6 class="ltx_title ltx_title_abstract">Abstract</h6>
<p id="id1.id1" class="ltx_p">Large Language Models (LLMs) have proven their exceptional capabilities in performing language-related tasks. However, their deployment poses significant challenges due to their considerable memory and storage requirements. In response to this issue, weight-only quantization, particularly 3 and 4-bit weight-only quantization, has emerged as one of the most viable solutions. As the number of bits decreases, the quantization grid broadens, thus emphasizing the importance of up and down rounding. While previous studies have demonstrated that fine-tuning up and down rounding with the addition of perturbations can enhance accuracy in some scenarios, our study is driven by the precise and limited boundary of these perturbations, where only the threshold for altering the rounding value is of significance. Consequently, we propose a concise and highly effective approach for optimizing the weight rounding task. Our method, named SignRound, involves lightweight block-wise tuning using signed gradient descent, enabling us to achieve outstanding results within 400 steps. SignRound competes impressively against recent methods without introducing additional inference overhead.
The source code will be publicly available at <a target="_blank" href="https://github.com/intel/neural-compressor" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/intel/neural-compressor</a> soon.</p>
</div>
<section id="S1" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">1 </span>Introduction</h2>

<div id="S1.p1" class="ltx_para ltx_noindent">
<p id="S1.p1.1" class="ltx_p">Large language models (LLMs) have demonstrated exceptional proficiency on language-related tasks<cite class="ltx_cite ltx_citemacro_citep">(<a href="#bib.bib24" title="" class="ltx_ref">OpenAI, </a>; Touvron et al., <a href="#bib.bib32" title="" class="ltx_ref">2023a</a>)</cite>. Nevertheless, the deployment of LLMs presents notable hurdles due to their extensive memory and storage needs. Moreover, the computational demands of these models leads to the challenges for real-time applications. Consequently, it becomes imperative to explore techniques like quantization to facilitate the efficient deployment of LLMs.</p>
</div>
<div id="S1.p2" class="ltx_para ltx_noindent">
<p id="S1.p2.1" class="ltx_p">Quantization techniques can be broadly classified into two categories: quantization-aware training (QAT) <cite class="ltx_cite ltx_citemacro_citep">(Esser et al., <a href="#bib.bib6" title="" class="ltx_ref">2019</a>; Zhuang et al., <a href="#bib.bib46" title="" class="ltx_ref">2021</a>; Lee et al., <a href="#bib.bib13" title="" class="ltx_ref">2021</a>; Liu et al., <a href="#bib.bib19" title="" class="ltx_ref">2023</a>)</cite> and post-training quantization (PTQ) <cite class="ltx_cite ltx_citemacro_citep">(Nagel et al., <a href="#bib.bib22" title="" class="ltx_ref">2019</a>; Xiao et al., <a href="#bib.bib38" title="" class="ltx_ref">2022</a>; Frantar et al., <a href="#bib.bib8" title="" class="ltx_ref">2022</a>; Nagel et al., <a href="#bib.bib23" title="" class="ltx_ref">2020</a>)</cite>. QAT involves training the model with quantization in mind. During QAT, the model is trained using simulated lower-precision representations, allowing it to learn and adapt to the effects of quantization. This approach often yields better accuracy compared to PTQ. However, QAT comes with certain drawbacks, including increased training complexity, longer training times, and the need to tune hyperparameters. Applying QAT to LLMs can be particularly costly, despite recent efforts <cite class="ltx_cite ltx_citemacro_citep">(Hu et al., <a href="#bib.bib11" title="" class="ltx_ref">2021</a>; Dettmers et al., <a href="#bib.bib5" title="" class="ltx_ref">2023</a>)</cite> to improve the efficiency of fine-tuning LLMs. In contrast, PTQ directly quantizes the model without any simulated training or fine-tuning. While PTQ is a concise approach, it is susceptible to significant accuracy drops. This highlights the need for further advancements in PTQ methods to enhance their accuracy preservation capabilities.</p>
</div>
<div id="S1.p3" class="ltx_para ltx_noindent">
<p id="S1.p3.1" class="ltx_p">Two types of tensors could be quantized: activations and weights. Weight-only quantization has gained prominence in recent times as it offers a favorable tradeoff for LLMs. Quantizing activations for LLMs can be challenging <cite class="ltx_cite ltx_citemacro_citep">(Wei et al., <a href="#bib.bib36" title="" class="ltx_ref">2022b</a>; Xiao et al., <a href="#bib.bib39" title="" class="ltx_ref">2023</a>; Bondarenko et al., <a href="#bib.bib3" title="" class="ltx_ref">2023</a>)</cite>, making weight-only quantization a more practical choice. Additionally, the primary bottleneck in generating new tokens for LLMs often lies in memory bandwidth <cite class="ltx_cite ltx_citemacro_citep">(Kim et al., <a href="#bib.bib12" title="" class="ltx_ref">2023</a>)</cite>, further emphasizing the significance of weight-only quantization. In this work, we only focus on weight only quantization.</p>
</div>
<div id="S1.p4" class="ltx_para ltx_noindent">
<p id="S1.p4.1" class="ltx_p">In order to quantize the weights, a rounding operation is necessary, with rounding-to-nearest (RTN) being the predominant method. RTN quantizes each element independently by simply rounding it to the nearest integer. However, RTN fails to consider the relationships between weights and weights, as well as weights and activations. The potential of an advanced rounding strategy to improve accuracy has been initially demonstrated by Nagel et al. <cite class="ltx_cite ltx_citemacro_citep">(Nagel et al., <a href="#bib.bib23" title="" class="ltx_ref">2020</a>)</cite>. They addressed the rounding task by formulating it as a quadratic unconstrained binary optimization problem and approximated the task loss by employing a Taylor series expansion. However, relying exclusively on the second-order term may not produce accurate results. This is because rounding can introduce considerable weight modifications that may make other order terms significant and non-negligible.</p>
</div>
<div id="S1.p5" class="ltx_para ltx_noindent">
<p id="S1.p5.1" class="ltx_p">We prefer the signed gradient descent method to effectively tackle the issue of sub-optimal rounding solutions. This approach is inspired by the well-defined boundaries of the solution space, which are confined to the range of [-0.5, 0.5], where only the threshold for altering the rounding value is of significance. Figure <a href="#S3.F1" title="Figure 1 ‣ 3.1 SignRound ‣ 3 Methodology ‣ Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a> provides an overview of our method, SignRound. It utilizes signed gradient descent to fine-tune the up and down rounding through block-wise output reconstruction, resulting in enhanced flexibility and faster convergence. Our contributions are primarily threefold:</p>
</div>
<div id="S1.p6" class="ltx_para ltx_noindent">
<ul id="S1.I1" class="ltx_itemize">
<li id="S1.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S1.I1.i1.p1" class="ltx_para">
<p id="S1.I1.i1.p1.1" class="ltx_p">We introduce a succinct and potent method for optimizing the weight-rounding task. Our approach utilizes a minimal amount of unlabeled data and executes quantization in a block-wise fashion. Moreover, it is worth noting that our method does not introduce any additional overhead during inference, further enhancing its general practicality.</p>
</div>
</li>
<li id="S1.I1.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S1.I1.i2.p1" class="ltx_para">
<p id="S1.I1.i2.p1.1" class="ltx_p">Our findings demonstrate that a mere alteration of approximately 5% of the rounding values can significantly enhance the performance of some quantization models.</p>
</div>
</li>
<li id="S1.I1.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S1.I1.i3.p1" class="ltx_para ltx_noindent">
<p id="S1.I1.i3.p1.1" class="ltx_p">Our empirical results exhibit substantial performance enhancements over the established baseline of RTN, and our method contends favorably against recent techniques.</p>
</div>
</li>
</ul>
</div>
</section>
<section id="S2" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">2 </span>Related Work</h2>

<section id="S2.SS0.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Quantization Aware Training.</h4>

<div id="S2.SS0.SSS0.Px1.p1" class="ltx_para ltx_noindent">
<p id="S2.SS0.SSS0.Px1.p1.1" class="ltx_p">QAT methods have gained widespread popularity in model compression, as they enable the fine-tuning process, often leading to superior accuracy compared to the PTQ method. In their work, <cite class="ltx_cite ltx_citemacro_citep">(Esser et al., <a href="#bib.bib6" title="" class="ltx_ref">2019</a>)</cite> proposed a novel approach that estimates and scales the task loss gradient at each weight and activation layer’s quantizer step size, allowing for joint learning with other network parameters. <cite class="ltx_cite ltx_citemacro_citep">(Zhuang et al., <a href="#bib.bib46" title="" class="ltx_ref">2021</a>)</cite> put forward a progressive quantization scheme that involves quantizing activations after weights. Additionally, CPQ <cite class="ltx_cite ltx_citemacro_citep">(Lee et al., <a href="#bib.bib13" title="" class="ltx_ref">2021</a>)</cite> effectively identified the optimal quantization grids while naturally encouraging the underlying full-precision weights to gather around those quantization grids cohesively during training. While QAT methods are popular in relatively small-scale models, their application in LLMs is limited due to the high computational cost associated with training or fine-tuning.</p>
</div>
</section>
<section id="S2.SS0.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Post-training Quantization (PTQ).</h4>

<div id="S2.SS0.SSS0.Px2.p1" class="ltx_para ltx_noindent">
<p id="S2.SS0.SSS0.Px2.p1.1" class="ltx_p">PTQ methods simplify the quantization process without the needs of additional training. <cite class="ltx_cite ltx_citemacro_citep">(Nagel et al., <a href="#bib.bib22" title="" class="ltx_ref">2019</a>)</cite> focused on minimizing quantization error through weight equalization and bias correction techniques. <cite class="ltx_cite ltx_citemacro_citep">(Liu et al., <a href="#bib.bib20" title="" class="ltx_ref">2021</a>)</cite> specifically addressed the quantization of vision transformers, introducing a ranking loss to preserve the relative order of self-attention results after quantization and exploring a mixed-precision quantization scheme. <cite class="ltx_cite ltx_citemacro_citep">(Frantar &amp; Alistarh, <a href="#bib.bib7" title="" class="ltx_ref">2022</a>)</cite> leveraged Optimal Brain Surgeon <cite class="ltx_cite ltx_citemacro_citep">(Hassibi et al., <a href="#bib.bib10" title="" class="ltx_ref">1993</a>)</cite> to tune weights during model compression. Both Hawq <cite class="ltx_cite ltx_citemacro_citep">(Yao et al., <a href="#bib.bib40" title="" class="ltx_ref">2021</a>)</cite> and HAQ <cite class="ltx_cite ltx_citemacro_citep">(Wang et al., <a href="#bib.bib34" title="" class="ltx_ref">2019</a>)</cite> aimed to identify important layers and maintain higher precision for them. Given its low resource requirement, PTQ is particularly suitable for the quantization of Large Language Models (LLMs). We will next focus on the quantization methods designed for LLMs, most of which fall under the category of PTQ.</p>
</div>
</section>
<section id="S2.SS0.SSS0.Px3" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Large Language Models Quantization.</h4>

<div id="S2.SS0.SSS0.Px3.p1" class="ltx_para ltx_noindent">
<p id="S2.SS0.SSS0.Px3.p1.1" class="ltx_p">Significant advancements have been made in addressing the pressing demand for quantizing large language models (LLMs). LLM.int8() <cite class="ltx_cite ltx_citemacro_citep">(Dettmers et al., <a href="#bib.bib4" title="" class="ltx_ref">2022</a>)</cite> introduced a mixed precision approach to preserve essential channels in high precision. ZeroQuantV2 <cite class="ltx_cite ltx_citemacro_citep">(Yao et al., <a href="#bib.bib41" title="" class="ltx_ref">2023</a>)</cite> employed low-rank matrices to enhance model quality recovery. RPTQ <cite class="ltx_cite ltx_citemacro_citep">(Yuan et al., <a href="#bib.bib42" title="" class="ltx_ref">2023</a>)</cite> mitigated the impact of range differences between channel by rearranging the channels and quantizing them in clusters. Other methods, such as SPIQ <cite class="ltx_cite ltx_citemacro_citep">(Yvinec et al., <a href="#bib.bib43" title="" class="ltx_ref">2023</a>)</cite>, SmoothQuant <cite class="ltx_cite ltx_citemacro_citep">(Xiao et al., <a href="#bib.bib38" title="" class="ltx_ref">2022</a>)</cite>, Outlier Suppression+ <cite class="ltx_cite ltx_citemacro_citep">(Wei et al., <a href="#bib.bib37" title="" class="ltx_ref">2023</a>)</cite>, utilized handcrafted equivalent transformations to mitigate quantization errors. While these approaches are effective, their applicability is limited due to the performance overhead involved during inference, because there is no chance to fuse the transformation scale to the model itself on certain model architectures. LLM-QAT <cite class="ltx_cite ltx_citemacro_citep">(Liu et al., <a href="#bib.bib19" title="" class="ltx_ref">2023</a>)</cite> employs QAT to enhance the performance of W4A8.
In the context of weight-only quantization, GPTQ <cite class="ltx_cite ltx_citemacro_citep">(Frantar et al., <a href="#bib.bib8" title="" class="ltx_ref">2022</a>)</cite> optimized weights using the Optimal Brain Surgeon <cite class="ltx_cite ltx_citemacro_citep">(Hassibi et al., <a href="#bib.bib10" title="" class="ltx_ref">1993</a>)</cite> technique, achieving low-bit quantization on LLMs with minimal computational overhead. AWQ <cite class="ltx_cite ltx_citemacro_citep">(Lin et al., <a href="#bib.bib18" title="" class="ltx_ref">2023</a>)</cite> followed the equivalent transformation approach with additional tuning in a constrained space, and has the similar limitations as SmoothQuant <cite class="ltx_cite ltx_citemacro_citep">(Xiao et al., <a href="#bib.bib38" title="" class="ltx_ref">2022</a>)</cite>. SqueezeLLM <cite class="ltx_cite ltx_citemacro_citep">(Kim et al., <a href="#bib.bib12" title="" class="ltx_ref">2023</a>)</cite> employed sensitivity-based non-uniform quantization and dense-and-sparse decomposition to achieve lossless compression to ultra-low precision. While recent advancements in LLM quantization have made significant progress, there is still room for improvement in achieving minimal quantization loss without introducing inference overhead.</p>
</div>
</section>
<section id="S2.SS0.SSS0.Px4" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Rounding Methods.</h4>

<div id="S2.SS0.SSS0.Px4.p1" class="ltx_para ltx_noindent">
<p id="S2.SS0.SSS0.Px4.p1.1" class="ltx_p">Adaptive Rounding <cite class="ltx_cite ltx_citemacro_citep">(Nagel et al., <a href="#bib.bib23" title="" class="ltx_ref">2020</a>)</cite> has already showcased the potential of an advanced rounding strategy to enhance accuracy <cite class="ltx_cite ltx_citemacro_citep">(Li et al., <a href="#bib.bib16" title="" class="ltx_ref">2021</a>; Wei et al., <a href="#bib.bib35" title="" class="ltx_ref">2022a</a>)</cite>. They used the rounding task as a quadratic unconstrained binary optimization problem by approximating the task loss through a Taylor series expansion. However, considering only the second-order term may not yield accurate results. This is because the rounding value gets multiplied by a scaling coefficient during de-quantization, potentially introducing significant weight changes that make other order terms non-negligible. FlexRound <cite class="ltx_cite ltx_citemacro_citep">(Lee et al., <a href="#bib.bib14" title="" class="ltx_ref">2023</a>)</cite> introduces a more flexible approach to rounding by incorporating element-wise division. This allows for simultaneous learning of a shared quantization grid size and individual scales for each pre-trained weight. However, it’s not easily scalable to apply to LLMs due to the needs of specialized hyperparameters for each specific model and task. AQuant <cite class="ltx_cite ltx_citemacro_citep">(Li et al., <a href="#bib.bib17" title="" class="ltx_ref">2022</a>)</cite> introduced a dynamic approach where the border becomes a function dependent on the activation value to reduce the quantization error of activation. We specifically concentrate on the up and down rounding task for weight quantization in this work.</p>
</div>
</section>
<section id="S2.SS0.SSS0.Px5" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Signed Gradient Descent.</h4>

<div id="S2.SS0.SSS0.Px5.p1" class="ltx_para ltx_noindent">
<p id="S2.SS0.SSS0.Px5.p1.1" class="ltx_p">Signed gradient descent is not commonly utilized and is typically applied in specific scenarios, such as reducing communication costs. This is because signed gradient carries significantly less information compared to original gradient. Recent studies have shed light on the advantages of sign-based methods over gradient descent in certain conditions. Safaryan et al. <cite class="ltx_cite ltx_citemacro_citep">(Safaryan &amp; Richtárik, <a href="#bib.bib29" title="" class="ltx_ref">2021</a>)</cite> found that sign-based methods are preferable when the Hessian matrix is concentrated on its diagonal and the maximal eigenvalue is much larger than the average eigenvalue. Li et al. <cite class="ltx_cite ltx_citemacro_citep">(Li et al., <a href="#bib.bib15" title="" class="ltx_ref">2023</a>)</cite> investigated a variant of sign-based gradient descent that exhibits faster convergence. Additionally, Safaryan et al. <cite class="ltx_cite ltx_citemacro_citep">(Safaryan &amp; Richtárik, <a href="#bib.bib29" title="" class="ltx_ref">2021</a>)</cite> proposed a stochastic sign descent with momentum, which converges under the standard bounded variance assumption with the optimal asymptotic rate. These findings contribute to a better understanding of the potential benefits and applications of signed gradient descent methods.</p>
</div>
</section>
</section>
<section id="S3" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">3 </span>Methodology</h2>

<div id="S3.p1" class="ltx_para ltx_noindent">
<p id="S3.p1.4" class="ltx_p">We provide an overview of quantization before diving into the details of our approach. To quantize and de-quantize the weights, the following operation as shown in Eq.<a href="#S3.E1" title="In 3 Methodology ‣ Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a> is used (disregarding zero point for simplicity).</p>
<table id="S3.E1" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S3.E1.m1.1" class="ltx_math_unparsed" alttext="\widetilde{W}=s*clip(\left\lfloor\frac{W}{s}\right\rceil,n,m),n,m\in\mathbb{N}" display="block"><semantics id="S3.E1.m1.1a"><mrow id="S3.E1.m1.1b"><mover accent="true" id="S3.E1.m1.1.1"><mi id="S3.E1.m1.1.1.2">W</mi><mo id="S3.E1.m1.1.1.1">~</mo></mover><mo id="S3.E1.m1.1.2">=</mo><mi id="S3.E1.m1.1.3">s</mi><mo lspace="0.222em" rspace="0.222em" id="S3.E1.m1.1.4">∗</mo><mi id="S3.E1.m1.1.5">c</mi><mi id="S3.E1.m1.1.6">l</mi><mi id="S3.E1.m1.1.7">i</mi><mi id="S3.E1.m1.1.8">p</mi><mrow id="S3.E1.m1.1.9"><mo stretchy="false" id="S3.E1.m1.1.9.1">(</mo><mrow id="S3.E1.m1.1.9.2"><mo id="S3.E1.m1.1.9.2.1">⌊</mo><mfrac id="S3.E1.m1.1.9.2.2"><mi id="S3.E1.m1.1.9.2.2.2">W</mi><mi id="S3.E1.m1.1.9.2.2.3">s</mi></mfrac><mo id="S3.E1.m1.1.9.2.3">⌉</mo></mrow><mo id="S3.E1.m1.1.9.3">,</mo><mi id="S3.E1.m1.1.9.4">n</mi><mo id="S3.E1.m1.1.9.5">,</mo><mi id="S3.E1.m1.1.9.6">m</mi><mo stretchy="false" id="S3.E1.m1.1.9.7">)</mo></mrow><mo id="S3.E1.m1.1.10">,</mo><mi id="S3.E1.m1.1.11">n</mi><mo id="S3.E1.m1.1.12">,</mo><mi id="S3.E1.m1.1.13">m</mi><mo id="S3.E1.m1.1.14">∈</mo><mi id="S3.E1.m1.1.15">ℕ</mi></mrow><annotation encoding="application/x-tex" id="S3.E1.m1.1c">\widetilde{W}=s*clip(\left\lfloor\frac{W}{s}\right\rceil,n,m),n,m\in\mathbb{N}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(1)</span></td>
</tr></tbody>
</table>
<p id="S3.p1.3" class="ltx_p">where <math id="S3.p1.1.m1.1" class="ltx_Math" alttext="s" display="inline"><semantics id="S3.p1.1.m1.1a"><mi id="S3.p1.1.m1.1.1" xref="S3.p1.1.m1.1.1.cmml">s</mi><annotation-xml encoding="MathML-Content" id="S3.p1.1.m1.1b"><ci id="S3.p1.1.m1.1.1.cmml" xref="S3.p1.1.m1.1.1">𝑠</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.p1.1.m1.1c">s</annotation></semantics></math> is the quantization scale, which is a positive scalar value. However, it is important to mention that our method can be easily extended to cases where <math id="S3.p1.2.m2.1" class="ltx_Math" alttext="s" display="inline"><semantics id="S3.p1.2.m2.1a"><mi id="S3.p1.2.m2.1.1" xref="S3.p1.2.m2.1.1.cmml">s</mi><annotation-xml encoding="MathML-Content" id="S3.p1.2.m2.1b"><ci id="S3.p1.2.m2.1.1.cmml" xref="S3.p1.2.m2.1.1">𝑠</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.p1.2.m2.1c">s</annotation></semantics></math> is a vector or tensor. And the rounding operation <math id="S3.p1.3.m3.1" class="ltx_Math" alttext="\left\lfloor\cdot\right\rceil" display="inline"><semantics id="S3.p1.3.m3.1a"><mrow id="S3.p1.3.m3.1.2.2" xref="S3.p1.3.m3.1.2.1.cmml"><mo id="S3.p1.3.m3.1.2.2.1" xref="S3.p1.3.m3.1.2.1.1.cmml">⌊</mo><mo lspace="0em" rspace="0em" id="S3.p1.3.m3.1.1" xref="S3.p1.3.m3.1.1.cmml">⋅</mo><mo id="S3.p1.3.m3.1.2.2.2" xref="S3.p1.3.m3.1.2.1.1.cmml">⌉</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.p1.3.m3.1b"><apply id="S3.p1.3.m3.1.2.1.cmml" xref="S3.p1.3.m3.1.2.2"><csymbol cd="latexml" id="S3.p1.3.m3.1.2.1.1.cmml" xref="S3.p1.3.m3.1.2.2.1">delimited-⌊⌉</csymbol><ci id="S3.p1.3.m3.1.1.cmml" xref="S3.p1.3.m3.1.1">⋅</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.p1.3.m3.1c">\left\lfloor\cdot\right\rceil</annotation></semantics></math> is typically performed using the RTN method. While RTN is a concise approach, it quantizes each element independently, thereby losing the ability to model the correlation among different weights or activations.</p>
</div>
<div id="S3.p2" class="ltx_para ltx_noindent">
<p id="S3.p2.5" class="ltx_p">To introduce more flexibility into the rounding operation, a tensor <math id="S3.p2.1.m1.1" class="ltx_Math" alttext="V" display="inline"><semantics id="S3.p2.1.m1.1a"><mi id="S3.p2.1.m1.1.1" xref="S3.p2.1.m1.1.1.cmml">V</mi><annotation-xml encoding="MathML-Content" id="S3.p2.1.m1.1b"><ci id="S3.p2.1.m1.1.1.cmml" xref="S3.p2.1.m1.1.1">𝑉</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.p2.1.m1.1c">V</annotation></semantics></math> with the same shape of <math id="S3.p2.2.m2.1" class="ltx_Math" alttext="W" display="inline"><semantics id="S3.p2.2.m2.1a"><mi id="S3.p2.2.m2.1.1" xref="S3.p2.2.m2.1.1.cmml">W</mi><annotation-xml encoding="MathML-Content" id="S3.p2.2.m2.1b"><ci id="S3.p2.2.m2.1.1.cmml" xref="S3.p2.2.m2.1.1">𝑊</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.p2.2.m2.1c">W</annotation></semantics></math> is introduced. Each element of <math id="S3.p2.3.m3.1" class="ltx_Math" alttext="V" display="inline"><semantics id="S3.p2.3.m3.1a"><mi id="S3.p2.3.m3.1.1" xref="S3.p2.3.m3.1.1.cmml">V</mi><annotation-xml encoding="MathML-Content" id="S3.p2.3.m3.1b"><ci id="S3.p2.3.m3.1.1.cmml" xref="S3.p2.3.m3.1.1">𝑉</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.p2.3.m3.1c">V</annotation></semantics></math> falls within the range of <math id="S3.p2.4.m4.2" class="ltx_Math" alttext="[-B,B]" display="inline"><semantics id="S3.p2.4.m4.2a"><mrow id="S3.p2.4.m4.2.2.1" xref="S3.p2.4.m4.2.2.2.cmml"><mo stretchy="false" id="S3.p2.4.m4.2.2.1.2" xref="S3.p2.4.m4.2.2.2.cmml">[</mo><mrow id="S3.p2.4.m4.2.2.1.1" xref="S3.p2.4.m4.2.2.1.1.cmml"><mo id="S3.p2.4.m4.2.2.1.1a" xref="S3.p2.4.m4.2.2.1.1.cmml">−</mo><mi id="S3.p2.4.m4.2.2.1.1.2" xref="S3.p2.4.m4.2.2.1.1.2.cmml">B</mi></mrow><mo id="S3.p2.4.m4.2.2.1.3" xref="S3.p2.4.m4.2.2.2.cmml">,</mo><mi id="S3.p2.4.m4.1.1" xref="S3.p2.4.m4.1.1.cmml">B</mi><mo stretchy="false" id="S3.p2.4.m4.2.2.1.4" xref="S3.p2.4.m4.2.2.2.cmml">]</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.p2.4.m4.2b"><interval closure="closed" id="S3.p2.4.m4.2.2.2.cmml" xref="S3.p2.4.m4.2.2.1"><apply id="S3.p2.4.m4.2.2.1.1.cmml" xref="S3.p2.4.m4.2.2.1.1"><minus id="S3.p2.4.m4.2.2.1.1.1.cmml" xref="S3.p2.4.m4.2.2.1.1"></minus><ci id="S3.p2.4.m4.2.2.1.1.2.cmml" xref="S3.p2.4.m4.2.2.1.1.2">𝐵</ci></apply><ci id="S3.p2.4.m4.1.1.cmml" xref="S3.p2.4.m4.1.1">𝐵</ci></interval></annotation-xml><annotation encoding="application/x-tex" id="S3.p2.4.m4.2c">[-B,B]</annotation></semantics></math>, in which <math id="S3.p2.5.m5.1" class="ltx_Math" alttext="B" display="inline"><semantics id="S3.p2.5.m5.1a"><mi id="S3.p2.5.m5.1.1" xref="S3.p2.5.m5.1.1.cmml">B</mi><annotation-xml encoding="MathML-Content" id="S3.p2.5.m5.1b"><ci id="S3.p2.5.m5.1.1.cmml" xref="S3.p2.5.m5.1.1">𝐵</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.p2.5.m5.1c">B</annotation></semantics></math> is set to 0.5 in all of experiments to ensure that the changes made only impact the rounding value.</p>
</div>
<div id="S3.p3" class="ltx_para ltx_noindent">
<table id="S3.E2" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S3.E2.m1.1" class="ltx_math_unparsed" alttext="\widetilde{W}=s*clip(\left\lfloor\frac{W}{s}+V\right\rceil,n,m),n,m\in\mathbb{N}" display="block"><semantics id="S3.E2.m1.1a"><mrow id="S3.E2.m1.1b"><mover accent="true" id="S3.E2.m1.1.1"><mi id="S3.E2.m1.1.1.2">W</mi><mo id="S3.E2.m1.1.1.1">~</mo></mover><mo id="S3.E2.m1.1.2">=</mo><mi id="S3.E2.m1.1.3">s</mi><mo lspace="0.222em" rspace="0.222em" id="S3.E2.m1.1.4">∗</mo><mi id="S3.E2.m1.1.5">c</mi><mi id="S3.E2.m1.1.6">l</mi><mi id="S3.E2.m1.1.7">i</mi><mi id="S3.E2.m1.1.8">p</mi><mrow id="S3.E2.m1.1.9"><mo stretchy="false" id="S3.E2.m1.1.9.1">(</mo><mrow id="S3.E2.m1.1.9.2"><mo id="S3.E2.m1.1.9.2.1">⌊</mo><mfrac id="S3.E2.m1.1.9.2.2"><mi id="S3.E2.m1.1.9.2.2.2">W</mi><mi id="S3.E2.m1.1.9.2.2.3">s</mi></mfrac><mo id="S3.E2.m1.1.9.2.3">+</mo><mi id="S3.E2.m1.1.9.2.4">V</mi><mo id="S3.E2.m1.1.9.2.5">⌉</mo></mrow><mo id="S3.E2.m1.1.9.3">,</mo><mi id="S3.E2.m1.1.9.4">n</mi><mo id="S3.E2.m1.1.9.5">,</mo><mi id="S3.E2.m1.1.9.6">m</mi><mo stretchy="false" id="S3.E2.m1.1.9.7">)</mo></mrow><mo id="S3.E2.m1.1.10">,</mo><mi id="S3.E2.m1.1.11">n</mi><mo id="S3.E2.m1.1.12">,</mo><mi id="S3.E2.m1.1.13">m</mi><mo id="S3.E2.m1.1.14">∈</mo><mi id="S3.E2.m1.1.15">ℕ</mi></mrow><annotation encoding="application/x-tex" id="S3.E2.m1.1c">\widetilde{W}=s*clip(\left\lfloor\frac{W}{s}+V\right\rceil,n,m),n,m\in\mathbb{N}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(2)</span></td>
</tr></tbody>
</table>
</div>
<div id="S3.p4" class="ltx_para ltx_noindent">
<p id="S3.p4.3" class="ltx_p">This adjustment allows for a more adaptable and context-aware quantization process.
If we try to reconstruct the output of layers, the loss could be formulated as</p>
<table id="S3.E3" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S3.E3.m1.1" class="ltx_Math" alttext="L=||WX-\widetilde{W}X||_{F}^{2}" display="block"><semantics id="S3.E3.m1.1a"><mrow id="S3.E3.m1.1.1" xref="S3.E3.m1.1.1.cmml"><mi id="S3.E3.m1.1.1.3" xref="S3.E3.m1.1.1.3.cmml">L</mi><mo id="S3.E3.m1.1.1.2" xref="S3.E3.m1.1.1.2.cmml">=</mo><msubsup id="S3.E3.m1.1.1.1" xref="S3.E3.m1.1.1.1.cmml"><mrow id="S3.E3.m1.1.1.1.1.1.1" xref="S3.E3.m1.1.1.1.1.1.2.cmml"><mo stretchy="false" id="S3.E3.m1.1.1.1.1.1.1.2" xref="S3.E3.m1.1.1.1.1.1.2.1.cmml">‖</mo><mrow id="S3.E3.m1.1.1.1.1.1.1.1" xref="S3.E3.m1.1.1.1.1.1.1.1.cmml"><mrow id="S3.E3.m1.1.1.1.1.1.1.1.2" xref="S3.E3.m1.1.1.1.1.1.1.1.2.cmml"><mi id="S3.E3.m1.1.1.1.1.1.1.1.2.2" xref="S3.E3.m1.1.1.1.1.1.1.1.2.2.cmml">W</mi><mo lspace="0em" rspace="0em" id="S3.E3.m1.1.1.1.1.1.1.1.2.1" xref="S3.E3.m1.1.1.1.1.1.1.1.2.1.cmml">​</mo><mi id="S3.E3.m1.1.1.1.1.1.1.1.2.3" xref="S3.E3.m1.1.1.1.1.1.1.1.2.3.cmml">X</mi></mrow><mo id="S3.E3.m1.1.1.1.1.1.1.1.1" xref="S3.E3.m1.1.1.1.1.1.1.1.1.cmml">−</mo><mrow id="S3.E3.m1.1.1.1.1.1.1.1.3" xref="S3.E3.m1.1.1.1.1.1.1.1.3.cmml"><mover accent="true" id="S3.E3.m1.1.1.1.1.1.1.1.3.2" xref="S3.E3.m1.1.1.1.1.1.1.1.3.2.cmml"><mi id="S3.E3.m1.1.1.1.1.1.1.1.3.2.2" xref="S3.E3.m1.1.1.1.1.1.1.1.3.2.2.cmml">W</mi><mo id="S3.E3.m1.1.1.1.1.1.1.1.3.2.1" xref="S3.E3.m1.1.1.1.1.1.1.1.3.2.1.cmml">~</mo></mover><mo lspace="0em" rspace="0em" id="S3.E3.m1.1.1.1.1.1.1.1.3.1" xref="S3.E3.m1.1.1.1.1.1.1.1.3.1.cmml">​</mo><mi id="S3.E3.m1.1.1.1.1.1.1.1.3.3" xref="S3.E3.m1.1.1.1.1.1.1.1.3.3.cmml">X</mi></mrow></mrow><mo stretchy="false" id="S3.E3.m1.1.1.1.1.1.1.3" xref="S3.E3.m1.1.1.1.1.1.2.1.cmml">‖</mo></mrow><mi id="S3.E3.m1.1.1.1.1.3" xref="S3.E3.m1.1.1.1.1.3.cmml">F</mi><mn id="S3.E3.m1.1.1.1.3" xref="S3.E3.m1.1.1.1.3.cmml">2</mn></msubsup></mrow><annotation-xml encoding="MathML-Content" id="S3.E3.m1.1b"><apply id="S3.E3.m1.1.1.cmml" xref="S3.E3.m1.1.1"><eq id="S3.E3.m1.1.1.2.cmml" xref="S3.E3.m1.1.1.2"></eq><ci id="S3.E3.m1.1.1.3.cmml" xref="S3.E3.m1.1.1.3">𝐿</ci><apply id="S3.E3.m1.1.1.1.cmml" xref="S3.E3.m1.1.1.1"><csymbol cd="ambiguous" id="S3.E3.m1.1.1.1.2.cmml" xref="S3.E3.m1.1.1.1">superscript</csymbol><apply id="S3.E3.m1.1.1.1.1.cmml" xref="S3.E3.m1.1.1.1"><csymbol cd="ambiguous" id="S3.E3.m1.1.1.1.1.2.cmml" xref="S3.E3.m1.1.1.1">subscript</csymbol><apply id="S3.E3.m1.1.1.1.1.1.2.cmml" xref="S3.E3.m1.1.1.1.1.1.1"><csymbol cd="latexml" id="S3.E3.m1.1.1.1.1.1.2.1.cmml" xref="S3.E3.m1.1.1.1.1.1.1.2">norm</csymbol><apply id="S3.E3.m1.1.1.1.1.1.1.1.cmml" xref="S3.E3.m1.1.1.1.1.1.1.1"><minus id="S3.E3.m1.1.1.1.1.1.1.1.1.cmml" xref="S3.E3.m1.1.1.1.1.1.1.1.1"></minus><apply id="S3.E3.m1.1.1.1.1.1.1.1.2.cmml" xref="S3.E3.m1.1.1.1.1.1.1.1.2"><times id="S3.E3.m1.1.1.1.1.1.1.1.2.1.cmml" xref="S3.E3.m1.1.1.1.1.1.1.1.2.1"></times><ci id="S3.E3.m1.1.1.1.1.1.1.1.2.2.cmml" xref="S3.E3.m1.1.1.1.1.1.1.1.2.2">𝑊</ci><ci id="S3.E3.m1.1.1.1.1.1.1.1.2.3.cmml" xref="S3.E3.m1.1.1.1.1.1.1.1.2.3">𝑋</ci></apply><apply id="S3.E3.m1.1.1.1.1.1.1.1.3.cmml" xref="S3.E3.m1.1.1.1.1.1.1.1.3"><times id="S3.E3.m1.1.1.1.1.1.1.1.3.1.cmml" xref="S3.E3.m1.1.1.1.1.1.1.1.3.1"></times><apply id="S3.E3.m1.1.1.1.1.1.1.1.3.2.cmml" xref="S3.E3.m1.1.1.1.1.1.1.1.3.2"><ci id="S3.E3.m1.1.1.1.1.1.1.1.3.2.1.cmml" xref="S3.E3.m1.1.1.1.1.1.1.1.3.2.1">~</ci><ci id="S3.E3.m1.1.1.1.1.1.1.1.3.2.2.cmml" xref="S3.E3.m1.1.1.1.1.1.1.1.3.2.2">𝑊</ci></apply><ci id="S3.E3.m1.1.1.1.1.1.1.1.3.3.cmml" xref="S3.E3.m1.1.1.1.1.1.1.1.3.3">𝑋</ci></apply></apply></apply><ci id="S3.E3.m1.1.1.1.1.3.cmml" xref="S3.E3.m1.1.1.1.1.3">𝐹</ci></apply><cn type="integer" id="S3.E3.m1.1.1.1.3.cmml" xref="S3.E3.m1.1.1.1.3">2</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E3.m1.1c">L=||WX-\widetilde{W}X||_{F}^{2}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(3)</span></td>
</tr></tbody>
</table>
<p id="S3.p4.2" class="ltx_p">where <math id="S3.p4.1.m1.1" class="ltx_Math" alttext="X" display="inline"><semantics id="S3.p4.1.m1.1a"><mi id="S3.p4.1.m1.1.1" xref="S3.p4.1.m1.1.1.cmml">X</mi><annotation-xml encoding="MathML-Content" id="S3.p4.1.m1.1b"><ci id="S3.p4.1.m1.1.1.cmml" xref="S3.p4.1.m1.1.1">𝑋</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.p4.1.m1.1c">X</annotation></semantics></math> is the input of the layer and <math id="S3.p4.2.m2.1" class="ltx_math_unparsed" alttext="||\cdot||_{F}" display="inline"><semantics id="S3.p4.2.m2.1a"><mrow id="S3.p4.2.m2.1b"><mo fence="false" rspace="0.167em" stretchy="false" id="S3.p4.2.m2.1.1">|</mo><mo fence="false" stretchy="false" id="S3.p4.2.m2.1.2">|</mo><mo lspace="0em" rspace="0em" id="S3.p4.2.m2.1.3">⋅</mo><mo fence="false" rspace="0.167em" stretchy="false" id="S3.p4.2.m2.1.4">|</mo><msub id="S3.p4.2.m2.1.5"><mo fence="false" stretchy="false" id="S3.p4.2.m2.1.5.2">|</mo><mi id="S3.p4.2.m2.1.5.3">F</mi></msub></mrow><annotation encoding="application/x-tex" id="S3.p4.2.m2.1c">||\cdot||_{F}</annotation></semantics></math> denotes the Frobenius norm. Then the final optimization task is described as the following</p>
<table id="S3.E4" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S3.E4.m1.1" class="ltx_Math" alttext="\mathop{\arg min}\limits_{V}||WX-\widetilde{W}X||_{F}^{2}" display="block"><semantics id="S3.E4.m1.1a"><mrow id="S3.E4.m1.1.1" xref="S3.E4.m1.1.1.cmml"><munder id="S3.E4.m1.1.1.2" xref="S3.E4.m1.1.1.2.cmml"><mrow id="S3.E4.m1.1.1.2.2" xref="S3.E4.m1.1.1.2.2.cmml"><mi id="S3.E4.m1.1.1.2.2.1" xref="S3.E4.m1.1.1.2.2.1.cmml">arg</mi><mo lspace="0.167em" id="S3.E4.m1.1.1.2.2a" xref="S3.E4.m1.1.1.2.2.cmml">⁡</mo><mrow id="S3.E4.m1.1.1.2.2.2" xref="S3.E4.m1.1.1.2.2.2.cmml"><mi id="S3.E4.m1.1.1.2.2.2.2" xref="S3.E4.m1.1.1.2.2.2.2.cmml">m</mi><mo lspace="0em" rspace="0em" id="S3.E4.m1.1.1.2.2.2.1" xref="S3.E4.m1.1.1.2.2.2.1.cmml">​</mo><mi id="S3.E4.m1.1.1.2.2.2.3" xref="S3.E4.m1.1.1.2.2.2.3.cmml">i</mi><mo lspace="0em" rspace="0em" id="S3.E4.m1.1.1.2.2.2.1a" xref="S3.E4.m1.1.1.2.2.2.1.cmml">​</mo><mi id="S3.E4.m1.1.1.2.2.2.4" xref="S3.E4.m1.1.1.2.2.2.4.cmml">n</mi></mrow></mrow><mi id="S3.E4.m1.1.1.2.3" xref="S3.E4.m1.1.1.2.3.cmml">V</mi></munder><msubsup id="S3.E4.m1.1.1.1" xref="S3.E4.m1.1.1.1.cmml"><mrow id="S3.E4.m1.1.1.1.1.1.1" xref="S3.E4.m1.1.1.1.1.1.2.cmml"><mo stretchy="false" id="S3.E4.m1.1.1.1.1.1.1.2" xref="S3.E4.m1.1.1.1.1.1.2.1.cmml">‖</mo><mrow id="S3.E4.m1.1.1.1.1.1.1.1" xref="S3.E4.m1.1.1.1.1.1.1.1.cmml"><mrow id="S3.E4.m1.1.1.1.1.1.1.1.2" xref="S3.E4.m1.1.1.1.1.1.1.1.2.cmml"><mi id="S3.E4.m1.1.1.1.1.1.1.1.2.2" xref="S3.E4.m1.1.1.1.1.1.1.1.2.2.cmml">W</mi><mo lspace="0em" rspace="0em" id="S3.E4.m1.1.1.1.1.1.1.1.2.1" xref="S3.E4.m1.1.1.1.1.1.1.1.2.1.cmml">​</mo><mi id="S3.E4.m1.1.1.1.1.1.1.1.2.3" xref="S3.E4.m1.1.1.1.1.1.1.1.2.3.cmml">X</mi></mrow><mo id="S3.E4.m1.1.1.1.1.1.1.1.1" xref="S3.E4.m1.1.1.1.1.1.1.1.1.cmml">−</mo><mrow id="S3.E4.m1.1.1.1.1.1.1.1.3" xref="S3.E4.m1.1.1.1.1.1.1.1.3.cmml"><mover accent="true" id="S3.E4.m1.1.1.1.1.1.1.1.3.2" xref="S3.E4.m1.1.1.1.1.1.1.1.3.2.cmml"><mi id="S3.E4.m1.1.1.1.1.1.1.1.3.2.2" xref="S3.E4.m1.1.1.1.1.1.1.1.3.2.2.cmml">W</mi><mo id="S3.E4.m1.1.1.1.1.1.1.1.3.2.1" xref="S3.E4.m1.1.1.1.1.1.1.1.3.2.1.cmml">~</mo></mover><mo lspace="0em" rspace="0em" id="S3.E4.m1.1.1.1.1.1.1.1.3.1" xref="S3.E4.m1.1.1.1.1.1.1.1.3.1.cmml">​</mo><mi id="S3.E4.m1.1.1.1.1.1.1.1.3.3" xref="S3.E4.m1.1.1.1.1.1.1.1.3.3.cmml">X</mi></mrow></mrow><mo stretchy="false" id="S3.E4.m1.1.1.1.1.1.1.3" xref="S3.E4.m1.1.1.1.1.1.2.1.cmml">‖</mo></mrow><mi id="S3.E4.m1.1.1.1.1.3" xref="S3.E4.m1.1.1.1.1.3.cmml">F</mi><mn id="S3.E4.m1.1.1.1.3" xref="S3.E4.m1.1.1.1.3.cmml">2</mn></msubsup></mrow><annotation-xml encoding="MathML-Content" id="S3.E4.m1.1b"><apply id="S3.E4.m1.1.1.cmml" xref="S3.E4.m1.1.1"><apply id="S3.E4.m1.1.1.2.cmml" xref="S3.E4.m1.1.1.2"><csymbol cd="ambiguous" id="S3.E4.m1.1.1.2.1.cmml" xref="S3.E4.m1.1.1.2">subscript</csymbol><apply id="S3.E4.m1.1.1.2.2.cmml" xref="S3.E4.m1.1.1.2.2"><arg id="S3.E4.m1.1.1.2.2.1.cmml" xref="S3.E4.m1.1.1.2.2.1"></arg><apply id="S3.E4.m1.1.1.2.2.2.cmml" xref="S3.E4.m1.1.1.2.2.2"><times id="S3.E4.m1.1.1.2.2.2.1.cmml" xref="S3.E4.m1.1.1.2.2.2.1"></times><ci id="S3.E4.m1.1.1.2.2.2.2.cmml" xref="S3.E4.m1.1.1.2.2.2.2">𝑚</ci><ci id="S3.E4.m1.1.1.2.2.2.3.cmml" xref="S3.E4.m1.1.1.2.2.2.3">𝑖</ci><ci id="S3.E4.m1.1.1.2.2.2.4.cmml" xref="S3.E4.m1.1.1.2.2.2.4">𝑛</ci></apply></apply><ci id="S3.E4.m1.1.1.2.3.cmml" xref="S3.E4.m1.1.1.2.3">𝑉</ci></apply><apply id="S3.E4.m1.1.1.1.cmml" xref="S3.E4.m1.1.1.1"><csymbol cd="ambiguous" id="S3.E4.m1.1.1.1.2.cmml" xref="S3.E4.m1.1.1.1">superscript</csymbol><apply id="S3.E4.m1.1.1.1.1.cmml" xref="S3.E4.m1.1.1.1"><csymbol cd="ambiguous" id="S3.E4.m1.1.1.1.1.2.cmml" xref="S3.E4.m1.1.1.1">subscript</csymbol><apply id="S3.E4.m1.1.1.1.1.1.2.cmml" xref="S3.E4.m1.1.1.1.1.1.1"><csymbol cd="latexml" id="S3.E4.m1.1.1.1.1.1.2.1.cmml" xref="S3.E4.m1.1.1.1.1.1.1.2">norm</csymbol><apply id="S3.E4.m1.1.1.1.1.1.1.1.cmml" xref="S3.E4.m1.1.1.1.1.1.1.1"><minus id="S3.E4.m1.1.1.1.1.1.1.1.1.cmml" xref="S3.E4.m1.1.1.1.1.1.1.1.1"></minus><apply id="S3.E4.m1.1.1.1.1.1.1.1.2.cmml" xref="S3.E4.m1.1.1.1.1.1.1.1.2"><times id="S3.E4.m1.1.1.1.1.1.1.1.2.1.cmml" xref="S3.E4.m1.1.1.1.1.1.1.1.2.1"></times><ci id="S3.E4.m1.1.1.1.1.1.1.1.2.2.cmml" xref="S3.E4.m1.1.1.1.1.1.1.1.2.2">𝑊</ci><ci id="S3.E4.m1.1.1.1.1.1.1.1.2.3.cmml" xref="S3.E4.m1.1.1.1.1.1.1.1.2.3">𝑋</ci></apply><apply id="S3.E4.m1.1.1.1.1.1.1.1.3.cmml" xref="S3.E4.m1.1.1.1.1.1.1.1.3"><times id="S3.E4.m1.1.1.1.1.1.1.1.3.1.cmml" xref="S3.E4.m1.1.1.1.1.1.1.1.3.1"></times><apply id="S3.E4.m1.1.1.1.1.1.1.1.3.2.cmml" xref="S3.E4.m1.1.1.1.1.1.1.1.3.2"><ci id="S3.E4.m1.1.1.1.1.1.1.1.3.2.1.cmml" xref="S3.E4.m1.1.1.1.1.1.1.1.3.2.1">~</ci><ci id="S3.E4.m1.1.1.1.1.1.1.1.3.2.2.cmml" xref="S3.E4.m1.1.1.1.1.1.1.1.3.2.2">𝑊</ci></apply><ci id="S3.E4.m1.1.1.1.1.1.1.1.3.3.cmml" xref="S3.E4.m1.1.1.1.1.1.1.1.3.3">𝑋</ci></apply></apply></apply><ci id="S3.E4.m1.1.1.1.1.3.cmml" xref="S3.E4.m1.1.1.1.1.3">𝐹</ci></apply><cn type="integer" id="S3.E4.m1.1.1.1.3.cmml" xref="S3.E4.m1.1.1.1.3">2</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E4.m1.1c">\mathop{\arg min}\limits_{V}||WX-\widetilde{W}X||_{F}^{2}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(4)</span></td>
</tr></tbody>
</table>
</div>
<section id="S3.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.1 </span>SignRound</h3>

<figure id="S3.F1" class="ltx_figure"><img src="/html/2309.05516/assets/imgs/overview.png" id="S3.F1.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="598" height="163" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 1: </span>An illustration of SignRound. Unlike the direct rounding in RTN, SignRound performs signed gradient descent to fine-tune the up and down rounding through block-wise output reconstruction. After lightweight forward and backward steps, <span id="S3.F1.2.1" class="ltx_text ltx_font_bold">W</span>INT4 has been well optimized towards the minimal loss, therefore ready for the final inference deployment. Note that Quant and Dequant are two standard operations for quantization and dequantization respectively.</figcaption>
</figure>
<div id="S3.SS1.p1" class="ltx_para ltx_noindent">
<p id="S3.SS1.p1.2" class="ltx_p">Since <math id="S3.SS1.p1.1.m1.1" class="ltx_Math" alttext="V" display="inline"><semantics id="S3.SS1.p1.1.m1.1a"><mi id="S3.SS1.p1.1.m1.1.1" xref="S3.SS1.p1.1.m1.1.1.cmml">V</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.1.m1.1b"><ci id="S3.SS1.p1.1.m1.1.1.cmml" xref="S3.SS1.p1.1.m1.1.1">𝑉</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.1.m1.1c">V</annotation></semantics></math> has a clear boundary, i.e. <math id="S3.SS1.p1.2.m2.2" class="ltx_Math" alttext="[-0.5,0.5]" display="inline"><semantics id="S3.SS1.p1.2.m2.2a"><mrow id="S3.SS1.p1.2.m2.2.2.1" xref="S3.SS1.p1.2.m2.2.2.2.cmml"><mo stretchy="false" id="S3.SS1.p1.2.m2.2.2.1.2" xref="S3.SS1.p1.2.m2.2.2.2.cmml">[</mo><mrow id="S3.SS1.p1.2.m2.2.2.1.1" xref="S3.SS1.p1.2.m2.2.2.1.1.cmml"><mo id="S3.SS1.p1.2.m2.2.2.1.1a" xref="S3.SS1.p1.2.m2.2.2.1.1.cmml">−</mo><mn id="S3.SS1.p1.2.m2.2.2.1.1.2" xref="S3.SS1.p1.2.m2.2.2.1.1.2.cmml">0.5</mn></mrow><mo id="S3.SS1.p1.2.m2.2.2.1.3" xref="S3.SS1.p1.2.m2.2.2.2.cmml">,</mo><mn id="S3.SS1.p1.2.m2.1.1" xref="S3.SS1.p1.2.m2.1.1.cmml">0.5</mn><mo stretchy="false" id="S3.SS1.p1.2.m2.2.2.1.4" xref="S3.SS1.p1.2.m2.2.2.2.cmml">]</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.2.m2.2b"><interval closure="closed" id="S3.SS1.p1.2.m2.2.2.2.cmml" xref="S3.SS1.p1.2.m2.2.2.1"><apply id="S3.SS1.p1.2.m2.2.2.1.1.cmml" xref="S3.SS1.p1.2.m2.2.2.1.1"><minus id="S3.SS1.p1.2.m2.2.2.1.1.1.cmml" xref="S3.SS1.p1.2.m2.2.2.1.1"></minus><cn type="float" id="S3.SS1.p1.2.m2.2.2.1.1.2.cmml" xref="S3.SS1.p1.2.m2.2.2.1.1.2">0.5</cn></apply><cn type="float" id="S3.SS1.p1.2.m2.1.1.cmml" xref="S3.SS1.p1.2.m2.1.1">0.5</cn></interval></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.2.m2.2c">[-0.5,0.5]</annotation></semantics></math>, and only the threshold for altering the rounding value is of significance, we prefer scaled signed gradient descent instead of normal gradient descent to optimize this task. Figure <a href="#S3.F1" title="Figure 1 ‣ 3.1 SignRound ‣ 3 Methodology ‣ Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a> shows an illustration of our method. More precisely, we follow the below optimization to approach the sub-optimal solution of Eq. <a href="#S3.E4" title="In 3 Methodology ‣ Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>.</p>
</div>
<div id="S3.SS1.p2" class="ltx_para ltx_noindent">
<table id="S3.E5" class="ltx_equationgroup ltx_eqn_table">
<tbody>
<tr id="S3.E5X" class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_eqn_cell"></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="S3.E5X.2.1.1.m1.1" class="ltx_Math" alttext="\displaystyle V_{t+1}=V_{t}-lr*sign(\frac{\partial L}{\partial V})" display="inline"><semantics id="S3.E5X.2.1.1.m1.1a"><mrow id="S3.E5X.2.1.1.m1.1.2" xref="S3.E5X.2.1.1.m1.1.2.cmml"><msub id="S3.E5X.2.1.1.m1.1.2.2" xref="S3.E5X.2.1.1.m1.1.2.2.cmml"><mi id="S3.E5X.2.1.1.m1.1.2.2.2" xref="S3.E5X.2.1.1.m1.1.2.2.2.cmml">V</mi><mrow id="S3.E5X.2.1.1.m1.1.2.2.3" xref="S3.E5X.2.1.1.m1.1.2.2.3.cmml"><mi id="S3.E5X.2.1.1.m1.1.2.2.3.2" xref="S3.E5X.2.1.1.m1.1.2.2.3.2.cmml">t</mi><mo id="S3.E5X.2.1.1.m1.1.2.2.3.1" xref="S3.E5X.2.1.1.m1.1.2.2.3.1.cmml">+</mo><mn id="S3.E5X.2.1.1.m1.1.2.2.3.3" xref="S3.E5X.2.1.1.m1.1.2.2.3.3.cmml">1</mn></mrow></msub><mo id="S3.E5X.2.1.1.m1.1.2.1" xref="S3.E5X.2.1.1.m1.1.2.1.cmml">=</mo><mrow id="S3.E5X.2.1.1.m1.1.2.3" xref="S3.E5X.2.1.1.m1.1.2.3.cmml"><msub id="S3.E5X.2.1.1.m1.1.2.3.2" xref="S3.E5X.2.1.1.m1.1.2.3.2.cmml"><mi id="S3.E5X.2.1.1.m1.1.2.3.2.2" xref="S3.E5X.2.1.1.m1.1.2.3.2.2.cmml">V</mi><mi id="S3.E5X.2.1.1.m1.1.2.3.2.3" xref="S3.E5X.2.1.1.m1.1.2.3.2.3.cmml">t</mi></msub><mo id="S3.E5X.2.1.1.m1.1.2.3.1" xref="S3.E5X.2.1.1.m1.1.2.3.1.cmml">−</mo><mrow id="S3.E5X.2.1.1.m1.1.2.3.3" xref="S3.E5X.2.1.1.m1.1.2.3.3.cmml"><mrow id="S3.E5X.2.1.1.m1.1.2.3.3.2" xref="S3.E5X.2.1.1.m1.1.2.3.3.2.cmml"><mrow id="S3.E5X.2.1.1.m1.1.2.3.3.2.2" xref="S3.E5X.2.1.1.m1.1.2.3.3.2.2.cmml"><mi id="S3.E5X.2.1.1.m1.1.2.3.3.2.2.2" xref="S3.E5X.2.1.1.m1.1.2.3.3.2.2.2.cmml">l</mi><mo lspace="0em" rspace="0em" id="S3.E5X.2.1.1.m1.1.2.3.3.2.2.1" xref="S3.E5X.2.1.1.m1.1.2.3.3.2.2.1.cmml">​</mo><mi id="S3.E5X.2.1.1.m1.1.2.3.3.2.2.3" xref="S3.E5X.2.1.1.m1.1.2.3.3.2.2.3.cmml">r</mi></mrow><mo lspace="0.222em" rspace="0.222em" id="S3.E5X.2.1.1.m1.1.2.3.3.2.1" xref="S3.E5X.2.1.1.m1.1.2.3.3.2.1.cmml">∗</mo><mi id="S3.E5X.2.1.1.m1.1.2.3.3.2.3" xref="S3.E5X.2.1.1.m1.1.2.3.3.2.3.cmml">s</mi></mrow><mo lspace="0em" rspace="0em" id="S3.E5X.2.1.1.m1.1.2.3.3.1" xref="S3.E5X.2.1.1.m1.1.2.3.3.1.cmml">​</mo><mi id="S3.E5X.2.1.1.m1.1.2.3.3.3" xref="S3.E5X.2.1.1.m1.1.2.3.3.3.cmml">i</mi><mo lspace="0em" rspace="0em" id="S3.E5X.2.1.1.m1.1.2.3.3.1a" xref="S3.E5X.2.1.1.m1.1.2.3.3.1.cmml">​</mo><mi id="S3.E5X.2.1.1.m1.1.2.3.3.4" xref="S3.E5X.2.1.1.m1.1.2.3.3.4.cmml">g</mi><mo lspace="0em" rspace="0em" id="S3.E5X.2.1.1.m1.1.2.3.3.1b" xref="S3.E5X.2.1.1.m1.1.2.3.3.1.cmml">​</mo><mi id="S3.E5X.2.1.1.m1.1.2.3.3.5" xref="S3.E5X.2.1.1.m1.1.2.3.3.5.cmml">n</mi><mo lspace="0em" rspace="0em" id="S3.E5X.2.1.1.m1.1.2.3.3.1c" xref="S3.E5X.2.1.1.m1.1.2.3.3.1.cmml">​</mo><mrow id="S3.E5X.2.1.1.m1.1.2.3.3.6.2" xref="S3.E5X.2.1.1.m1.1.1.cmml"><mo stretchy="false" id="S3.E5X.2.1.1.m1.1.2.3.3.6.2.1" xref="S3.E5X.2.1.1.m1.1.1.cmml">(</mo><mstyle displaystyle="true" id="S3.E5X.2.1.1.m1.1.1" xref="S3.E5X.2.1.1.m1.1.1.cmml"><mfrac id="S3.E5X.2.1.1.m1.1.1a" xref="S3.E5X.2.1.1.m1.1.1.cmml"><mrow id="S3.E5X.2.1.1.m1.1.1.2" xref="S3.E5X.2.1.1.m1.1.1.2.cmml"><mo rspace="0em" id="S3.E5X.2.1.1.m1.1.1.2.1" xref="S3.E5X.2.1.1.m1.1.1.2.1.cmml">∂</mo><mi id="S3.E5X.2.1.1.m1.1.1.2.2" xref="S3.E5X.2.1.1.m1.1.1.2.2.cmml">L</mi></mrow><mrow id="S3.E5X.2.1.1.m1.1.1.3" xref="S3.E5X.2.1.1.m1.1.1.3.cmml"><mo rspace="0em" id="S3.E5X.2.1.1.m1.1.1.3.1" xref="S3.E5X.2.1.1.m1.1.1.3.1.cmml">∂</mo><mi id="S3.E5X.2.1.1.m1.1.1.3.2" xref="S3.E5X.2.1.1.m1.1.1.3.2.cmml">V</mi></mrow></mfrac></mstyle><mo stretchy="false" id="S3.E5X.2.1.1.m1.1.2.3.3.6.2.2" xref="S3.E5X.2.1.1.m1.1.1.cmml">)</mo></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.E5X.2.1.1.m1.1b"><apply id="S3.E5X.2.1.1.m1.1.2.cmml" xref="S3.E5X.2.1.1.m1.1.2"><eq id="S3.E5X.2.1.1.m1.1.2.1.cmml" xref="S3.E5X.2.1.1.m1.1.2.1"></eq><apply id="S3.E5X.2.1.1.m1.1.2.2.cmml" xref="S3.E5X.2.1.1.m1.1.2.2"><csymbol cd="ambiguous" id="S3.E5X.2.1.1.m1.1.2.2.1.cmml" xref="S3.E5X.2.1.1.m1.1.2.2">subscript</csymbol><ci id="S3.E5X.2.1.1.m1.1.2.2.2.cmml" xref="S3.E5X.2.1.1.m1.1.2.2.2">𝑉</ci><apply id="S3.E5X.2.1.1.m1.1.2.2.3.cmml" xref="S3.E5X.2.1.1.m1.1.2.2.3"><plus id="S3.E5X.2.1.1.m1.1.2.2.3.1.cmml" xref="S3.E5X.2.1.1.m1.1.2.2.3.1"></plus><ci id="S3.E5X.2.1.1.m1.1.2.2.3.2.cmml" xref="S3.E5X.2.1.1.m1.1.2.2.3.2">𝑡</ci><cn type="integer" id="S3.E5X.2.1.1.m1.1.2.2.3.3.cmml" xref="S3.E5X.2.1.1.m1.1.2.2.3.3">1</cn></apply></apply><apply id="S3.E5X.2.1.1.m1.1.2.3.cmml" xref="S3.E5X.2.1.1.m1.1.2.3"><minus id="S3.E5X.2.1.1.m1.1.2.3.1.cmml" xref="S3.E5X.2.1.1.m1.1.2.3.1"></minus><apply id="S3.E5X.2.1.1.m1.1.2.3.2.cmml" xref="S3.E5X.2.1.1.m1.1.2.3.2"><csymbol cd="ambiguous" id="S3.E5X.2.1.1.m1.1.2.3.2.1.cmml" xref="S3.E5X.2.1.1.m1.1.2.3.2">subscript</csymbol><ci id="S3.E5X.2.1.1.m1.1.2.3.2.2.cmml" xref="S3.E5X.2.1.1.m1.1.2.3.2.2">𝑉</ci><ci id="S3.E5X.2.1.1.m1.1.2.3.2.3.cmml" xref="S3.E5X.2.1.1.m1.1.2.3.2.3">𝑡</ci></apply><apply id="S3.E5X.2.1.1.m1.1.2.3.3.cmml" xref="S3.E5X.2.1.1.m1.1.2.3.3"><times id="S3.E5X.2.1.1.m1.1.2.3.3.1.cmml" xref="S3.E5X.2.1.1.m1.1.2.3.3.1"></times><apply id="S3.E5X.2.1.1.m1.1.2.3.3.2.cmml" xref="S3.E5X.2.1.1.m1.1.2.3.3.2"><times id="S3.E5X.2.1.1.m1.1.2.3.3.2.1.cmml" xref="S3.E5X.2.1.1.m1.1.2.3.3.2.1"></times><apply id="S3.E5X.2.1.1.m1.1.2.3.3.2.2.cmml" xref="S3.E5X.2.1.1.m1.1.2.3.3.2.2"><times id="S3.E5X.2.1.1.m1.1.2.3.3.2.2.1.cmml" xref="S3.E5X.2.1.1.m1.1.2.3.3.2.2.1"></times><ci id="S3.E5X.2.1.1.m1.1.2.3.3.2.2.2.cmml" xref="S3.E5X.2.1.1.m1.1.2.3.3.2.2.2">𝑙</ci><ci id="S3.E5X.2.1.1.m1.1.2.3.3.2.2.3.cmml" xref="S3.E5X.2.1.1.m1.1.2.3.3.2.2.3">𝑟</ci></apply><ci id="S3.E5X.2.1.1.m1.1.2.3.3.2.3.cmml" xref="S3.E5X.2.1.1.m1.1.2.3.3.2.3">𝑠</ci></apply><ci id="S3.E5X.2.1.1.m1.1.2.3.3.3.cmml" xref="S3.E5X.2.1.1.m1.1.2.3.3.3">𝑖</ci><ci id="S3.E5X.2.1.1.m1.1.2.3.3.4.cmml" xref="S3.E5X.2.1.1.m1.1.2.3.3.4">𝑔</ci><ci id="S3.E5X.2.1.1.m1.1.2.3.3.5.cmml" xref="S3.E5X.2.1.1.m1.1.2.3.3.5">𝑛</ci><apply id="S3.E5X.2.1.1.m1.1.1.cmml" xref="S3.E5X.2.1.1.m1.1.2.3.3.6.2"><divide id="S3.E5X.2.1.1.m1.1.1.1.cmml" xref="S3.E5X.2.1.1.m1.1.2.3.3.6.2"></divide><apply id="S3.E5X.2.1.1.m1.1.1.2.cmml" xref="S3.E5X.2.1.1.m1.1.1.2"><partialdiff id="S3.E5X.2.1.1.m1.1.1.2.1.cmml" xref="S3.E5X.2.1.1.m1.1.1.2.1"></partialdiff><ci id="S3.E5X.2.1.1.m1.1.1.2.2.cmml" xref="S3.E5X.2.1.1.m1.1.1.2.2">𝐿</ci></apply><apply id="S3.E5X.2.1.1.m1.1.1.3.cmml" xref="S3.E5X.2.1.1.m1.1.1.3"><partialdiff id="S3.E5X.2.1.1.m1.1.1.3.1.cmml" xref="S3.E5X.2.1.1.m1.1.1.3.1"></partialdiff><ci id="S3.E5X.2.1.1.m1.1.1.3.2.cmml" xref="S3.E5X.2.1.1.m1.1.1.3.2">𝑉</ci></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E5X.2.1.1.m1.1c">\displaystyle V_{t+1}=V_{t}-lr*sign(\frac{\partial L}{\partial V})</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="2" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equationgroup ltx_align_right">(5)</span></td>
</tr>
<tr id="S3.E5Xa" class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_eqn_cell"></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="S3.E5Xa.2.1.1.m1.4" class="ltx_Math" alttext="\displaystyle\mathrm{s.t.}|\sum_{t}lr*sign(\frac{\partial L}{\partial V})|\leq B" display="inline"><semantics id="S3.E5Xa.2.1.1.m1.4a"><mrow id="S3.E5Xa.2.1.1.m1.4.4.1" xref="S3.E5Xa.2.1.1.m1.4.4.2.cmml"><mi mathvariant="normal" id="S3.E5Xa.2.1.1.m1.2.2" xref="S3.E5Xa.2.1.1.m1.2.2.cmml">s</mi><mo lspace="0em" rspace="0.167em" id="S3.E5Xa.2.1.1.m1.4.4.1.2" xref="S3.E5Xa.2.1.1.m1.4.4.2a.cmml">.</mo><mi mathvariant="normal" id="S3.E5Xa.2.1.1.m1.3.3" xref="S3.E5Xa.2.1.1.m1.3.3.cmml">t</mi><mo lspace="0em" rspace="0.167em" id="S3.E5Xa.2.1.1.m1.4.4.1.3" xref="S3.E5Xa.2.1.1.m1.4.4.2a.cmml">.</mo><mrow id="S3.E5Xa.2.1.1.m1.4.4.1.1" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.cmml"><mrow id="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.1.2.cmml"><mo stretchy="false" id="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.2" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.1.2.1.cmml">|</mo><mrow id="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.cmml"><mstyle displaystyle="true" id="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.1" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.1.cmml"><munder id="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.1a" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.1.cmml"><mo movablelimits="false" id="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.1.2" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.1.2.cmml">∑</mo><mi id="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.1.3" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.1.3.cmml">t</mi></munder></mstyle><mrow id="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.2" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.2.cmml"><mrow id="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.2.2" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.2.2.cmml"><mrow id="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.2.2.2" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.2.2.2.cmml"><mi id="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.2.2.2.2" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.2.2.2.2.cmml">l</mi><mo lspace="0em" rspace="0em" id="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.2.2.2.1" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.2.2.2.1.cmml">​</mo><mi id="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.2.2.2.3" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.2.2.2.3.cmml">r</mi></mrow><mo lspace="0.222em" rspace="0.222em" id="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.2.2.1" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.2.2.1.cmml">∗</mo><mi id="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.2.2.3" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.2.2.3.cmml">s</mi></mrow><mo lspace="0em" rspace="0em" id="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.2.1" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.2.1.cmml">​</mo><mi id="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.2.3" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.2.3.cmml">i</mi><mo lspace="0em" rspace="0em" id="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.2.1a" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.2.1.cmml">​</mo><mi id="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.2.4" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.2.4.cmml">g</mi><mo lspace="0em" rspace="0em" id="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.2.1b" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.2.1.cmml">​</mo><mi id="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.2.5" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.2.5.cmml">n</mi><mo lspace="0em" rspace="0em" id="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.2.1c" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.2.1.cmml">​</mo><mrow id="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.2.6.2" xref="S3.E5Xa.2.1.1.m1.1.1.cmml"><mo stretchy="false" id="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.2.6.2.1" xref="S3.E5Xa.2.1.1.m1.1.1.cmml">(</mo><mstyle displaystyle="true" id="S3.E5Xa.2.1.1.m1.1.1" xref="S3.E5Xa.2.1.1.m1.1.1.cmml"><mfrac id="S3.E5Xa.2.1.1.m1.1.1a" xref="S3.E5Xa.2.1.1.m1.1.1.cmml"><mrow id="S3.E5Xa.2.1.1.m1.1.1.2" xref="S3.E5Xa.2.1.1.m1.1.1.2.cmml"><mo rspace="0em" id="S3.E5Xa.2.1.1.m1.1.1.2.1" xref="S3.E5Xa.2.1.1.m1.1.1.2.1.cmml">∂</mo><mi id="S3.E5Xa.2.1.1.m1.1.1.2.2" xref="S3.E5Xa.2.1.1.m1.1.1.2.2.cmml">L</mi></mrow><mrow id="S3.E5Xa.2.1.1.m1.1.1.3" xref="S3.E5Xa.2.1.1.m1.1.1.3.cmml"><mo rspace="0em" id="S3.E5Xa.2.1.1.m1.1.1.3.1" xref="S3.E5Xa.2.1.1.m1.1.1.3.1.cmml">∂</mo><mi id="S3.E5Xa.2.1.1.m1.1.1.3.2" xref="S3.E5Xa.2.1.1.m1.1.1.3.2.cmml">V</mi></mrow></mfrac></mstyle><mo stretchy="false" id="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.2.6.2.2" xref="S3.E5Xa.2.1.1.m1.1.1.cmml">)</mo></mrow></mrow></mrow><mo stretchy="false" id="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.3" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.1.2.1.cmml">|</mo></mrow><mo id="S3.E5Xa.2.1.1.m1.4.4.1.1.2" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.2.cmml">≤</mo><mi id="S3.E5Xa.2.1.1.m1.4.4.1.1.3" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.3.cmml">B</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.E5Xa.2.1.1.m1.4b"><apply id="S3.E5Xa.2.1.1.m1.4.4.2.cmml" xref="S3.E5Xa.2.1.1.m1.4.4.1"><csymbol cd="ambiguous" id="S3.E5Xa.2.1.1.m1.4.4.2a.cmml" xref="S3.E5Xa.2.1.1.m1.4.4.1.2">formulae-sequence</csymbol><ci id="S3.E5Xa.2.1.1.m1.2.2.cmml" xref="S3.E5Xa.2.1.1.m1.2.2">s</ci><ci id="S3.E5Xa.2.1.1.m1.3.3.cmml" xref="S3.E5Xa.2.1.1.m1.3.3">t</ci><apply id="S3.E5Xa.2.1.1.m1.4.4.1.1.cmml" xref="S3.E5Xa.2.1.1.m1.4.4.1.1"><leq id="S3.E5Xa.2.1.1.m1.4.4.1.1.2.cmml" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.2"></leq><apply id="S3.E5Xa.2.1.1.m1.4.4.1.1.1.2.cmml" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1"><abs id="S3.E5Xa.2.1.1.m1.4.4.1.1.1.2.1.cmml" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.2"></abs><apply id="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.cmml" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1"><apply id="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.1.cmml" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.1.1.cmml" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.1">subscript</csymbol><sum id="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.1.2.cmml" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.1.2"></sum><ci id="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.1.3.cmml" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.1.3">𝑡</ci></apply><apply id="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.2.cmml" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.2"><times id="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.2.1.cmml" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.2.1"></times><apply id="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.2.2.cmml" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.2.2"><times id="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.2.2.1.cmml" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.2.2.1"></times><apply id="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.2.2.2.cmml" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.2.2.2"><times id="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.2.2.2.1.cmml" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.2.2.2.1"></times><ci id="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.2.2.2.2.cmml" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.2.2.2.2">𝑙</ci><ci id="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.2.2.2.3.cmml" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.2.2.2.3">𝑟</ci></apply><ci id="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.2.2.3.cmml" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.2.2.3">𝑠</ci></apply><ci id="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.2.3.cmml" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.2.3">𝑖</ci><ci id="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.2.4.cmml" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.2.4">𝑔</ci><ci id="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.2.5.cmml" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.2.5">𝑛</ci><apply id="S3.E5Xa.2.1.1.m1.1.1.cmml" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.2.6.2"><divide id="S3.E5Xa.2.1.1.m1.1.1.1.cmml" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.2.6.2"></divide><apply id="S3.E5Xa.2.1.1.m1.1.1.2.cmml" xref="S3.E5Xa.2.1.1.m1.1.1.2"><partialdiff id="S3.E5Xa.2.1.1.m1.1.1.2.1.cmml" xref="S3.E5Xa.2.1.1.m1.1.1.2.1"></partialdiff><ci id="S3.E5Xa.2.1.1.m1.1.1.2.2.cmml" xref="S3.E5Xa.2.1.1.m1.1.1.2.2">𝐿</ci></apply><apply id="S3.E5Xa.2.1.1.m1.1.1.3.cmml" xref="S3.E5Xa.2.1.1.m1.1.1.3"><partialdiff id="S3.E5Xa.2.1.1.m1.1.1.3.1.cmml" xref="S3.E5Xa.2.1.1.m1.1.1.3.1"></partialdiff><ci id="S3.E5Xa.2.1.1.m1.1.1.3.2.cmml" xref="S3.E5Xa.2.1.1.m1.1.1.3.2">𝑉</ci></apply></apply></apply></apply></apply><ci id="S3.E5Xa.2.1.1.m1.4.4.1.1.3.cmml" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.3">𝐵</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E5Xa.2.1.1.m1.4c">\displaystyle\mathrm{s.t.}|\sum_{t}lr*sign(\frac{\partial L}{\partial V})|\leq B</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr>
</tbody>
</table>
</div>
<div id="S3.SS1.p3" class="ltx_para ltx_noindent">
<p id="S3.SS1.p3.2" class="ltx_p">where <math id="S3.SS1.p3.1.m1.1" class="ltx_Math" alttext="t" display="inline"><semantics id="S3.SS1.p3.1.m1.1a"><mi id="S3.SS1.p3.1.m1.1.1" xref="S3.SS1.p3.1.m1.1.1.cmml">t</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p3.1.m1.1b"><ci id="S3.SS1.p3.1.m1.1.1.cmml" xref="S3.SS1.p3.1.m1.1.1">𝑡</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p3.1.m1.1c">t</annotation></semantics></math> is the optimizing step, lr is the learning rate, <math id="S3.SS1.p3.2.m2.1" class="ltx_math_unparsed" alttext="|\cdot|" display="inline"><semantics id="S3.SS1.p3.2.m2.1a"><mrow id="S3.SS1.p3.2.m2.1b"><mo fence="false" stretchy="false" id="S3.SS1.p3.2.m2.1.1">|</mo><mo lspace="0em" rspace="0em" id="S3.SS1.p3.2.m2.1.2">⋅</mo><mo fence="false" stretchy="false" id="S3.SS1.p3.2.m2.1.3">|</mo></mrow><annotation encoding="application/x-tex" id="S3.SS1.p3.2.m2.1c">|\cdot|</annotation></semantics></math> is the absolute operation and B is the boundary we use, which is set to 0.5 in all our experiments.</p>
</div>
<div id="S3.SS1.p4" class="ltx_para ltx_noindent">
<p id="S3.SS1.p4.2" class="ltx_p">Further, by employing straight-through estimator (STE) <cite class="ltx_cite ltx_citemacro_citep">(Bengio et al., <a href="#bib.bib1" title="" class="ltx_ref">2013</a>)</cite>, it can be easily demonstrated that <math id="S3.SS1.p4.1.m1.2" class="ltx_Math" alttext="sign(\frac{\partial L}{\partial V})=sign(\frac{\partial L}{\partial W})" display="inline"><semantics id="S3.SS1.p4.1.m1.2a"><mrow id="S3.SS1.p4.1.m1.2.3" xref="S3.SS1.p4.1.m1.2.3.cmml"><mrow id="S3.SS1.p4.1.m1.2.3.2" xref="S3.SS1.p4.1.m1.2.3.2.cmml"><mi id="S3.SS1.p4.1.m1.2.3.2.2" xref="S3.SS1.p4.1.m1.2.3.2.2.cmml">s</mi><mo lspace="0em" rspace="0em" id="S3.SS1.p4.1.m1.2.3.2.1" xref="S3.SS1.p4.1.m1.2.3.2.1.cmml">​</mo><mi id="S3.SS1.p4.1.m1.2.3.2.3" xref="S3.SS1.p4.1.m1.2.3.2.3.cmml">i</mi><mo lspace="0em" rspace="0em" id="S3.SS1.p4.1.m1.2.3.2.1a" xref="S3.SS1.p4.1.m1.2.3.2.1.cmml">​</mo><mi id="S3.SS1.p4.1.m1.2.3.2.4" xref="S3.SS1.p4.1.m1.2.3.2.4.cmml">g</mi><mo lspace="0em" rspace="0em" id="S3.SS1.p4.1.m1.2.3.2.1b" xref="S3.SS1.p4.1.m1.2.3.2.1.cmml">​</mo><mi id="S3.SS1.p4.1.m1.2.3.2.5" xref="S3.SS1.p4.1.m1.2.3.2.5.cmml">n</mi><mo lspace="0em" rspace="0em" id="S3.SS1.p4.1.m1.2.3.2.1c" xref="S3.SS1.p4.1.m1.2.3.2.1.cmml">​</mo><mrow id="S3.SS1.p4.1.m1.2.3.2.6.2" xref="S3.SS1.p4.1.m1.1.1.cmml"><mo stretchy="false" id="S3.SS1.p4.1.m1.2.3.2.6.2.1" xref="S3.SS1.p4.1.m1.1.1.cmml">(</mo><mfrac id="S3.SS1.p4.1.m1.1.1" xref="S3.SS1.p4.1.m1.1.1.cmml"><mrow id="S3.SS1.p4.1.m1.1.1.2" xref="S3.SS1.p4.1.m1.1.1.2.cmml"><mo rspace="0em" id="S3.SS1.p4.1.m1.1.1.2.1" xref="S3.SS1.p4.1.m1.1.1.2.1.cmml">∂</mo><mi id="S3.SS1.p4.1.m1.1.1.2.2" xref="S3.SS1.p4.1.m1.1.1.2.2.cmml">L</mi></mrow><mrow id="S3.SS1.p4.1.m1.1.1.3" xref="S3.SS1.p4.1.m1.1.1.3.cmml"><mo rspace="0em" id="S3.SS1.p4.1.m1.1.1.3.1" xref="S3.SS1.p4.1.m1.1.1.3.1.cmml">∂</mo><mi id="S3.SS1.p4.1.m1.1.1.3.2" xref="S3.SS1.p4.1.m1.1.1.3.2.cmml">V</mi></mrow></mfrac><mo stretchy="false" id="S3.SS1.p4.1.m1.2.3.2.6.2.2" xref="S3.SS1.p4.1.m1.1.1.cmml">)</mo></mrow></mrow><mo id="S3.SS1.p4.1.m1.2.3.1" xref="S3.SS1.p4.1.m1.2.3.1.cmml">=</mo><mrow id="S3.SS1.p4.1.m1.2.3.3" xref="S3.SS1.p4.1.m1.2.3.3.cmml"><mi id="S3.SS1.p4.1.m1.2.3.3.2" xref="S3.SS1.p4.1.m1.2.3.3.2.cmml">s</mi><mo lspace="0em" rspace="0em" id="S3.SS1.p4.1.m1.2.3.3.1" xref="S3.SS1.p4.1.m1.2.3.3.1.cmml">​</mo><mi id="S3.SS1.p4.1.m1.2.3.3.3" xref="S3.SS1.p4.1.m1.2.3.3.3.cmml">i</mi><mo lspace="0em" rspace="0em" id="S3.SS1.p4.1.m1.2.3.3.1a" xref="S3.SS1.p4.1.m1.2.3.3.1.cmml">​</mo><mi id="S3.SS1.p4.1.m1.2.3.3.4" xref="S3.SS1.p4.1.m1.2.3.3.4.cmml">g</mi><mo lspace="0em" rspace="0em" id="S3.SS1.p4.1.m1.2.3.3.1b" xref="S3.SS1.p4.1.m1.2.3.3.1.cmml">​</mo><mi id="S3.SS1.p4.1.m1.2.3.3.5" xref="S3.SS1.p4.1.m1.2.3.3.5.cmml">n</mi><mo lspace="0em" rspace="0em" id="S3.SS1.p4.1.m1.2.3.3.1c" xref="S3.SS1.p4.1.m1.2.3.3.1.cmml">​</mo><mrow id="S3.SS1.p4.1.m1.2.3.3.6.2" xref="S3.SS1.p4.1.m1.2.2.cmml"><mo stretchy="false" id="S3.SS1.p4.1.m1.2.3.3.6.2.1" xref="S3.SS1.p4.1.m1.2.2.cmml">(</mo><mfrac id="S3.SS1.p4.1.m1.2.2" xref="S3.SS1.p4.1.m1.2.2.cmml"><mrow id="S3.SS1.p4.1.m1.2.2.2" xref="S3.SS1.p4.1.m1.2.2.2.cmml"><mo rspace="0em" id="S3.SS1.p4.1.m1.2.2.2.1" xref="S3.SS1.p4.1.m1.2.2.2.1.cmml">∂</mo><mi id="S3.SS1.p4.1.m1.2.2.2.2" xref="S3.SS1.p4.1.m1.2.2.2.2.cmml">L</mi></mrow><mrow id="S3.SS1.p4.1.m1.2.2.3" xref="S3.SS1.p4.1.m1.2.2.3.cmml"><mo rspace="0em" id="S3.SS1.p4.1.m1.2.2.3.1" xref="S3.SS1.p4.1.m1.2.2.3.1.cmml">∂</mo><mi id="S3.SS1.p4.1.m1.2.2.3.2" xref="S3.SS1.p4.1.m1.2.2.3.2.cmml">W</mi></mrow></mfrac><mo stretchy="false" id="S3.SS1.p4.1.m1.2.3.3.6.2.2" xref="S3.SS1.p4.1.m1.2.2.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p4.1.m1.2b"><apply id="S3.SS1.p4.1.m1.2.3.cmml" xref="S3.SS1.p4.1.m1.2.3"><eq id="S3.SS1.p4.1.m1.2.3.1.cmml" xref="S3.SS1.p4.1.m1.2.3.1"></eq><apply id="S3.SS1.p4.1.m1.2.3.2.cmml" xref="S3.SS1.p4.1.m1.2.3.2"><times id="S3.SS1.p4.1.m1.2.3.2.1.cmml" xref="S3.SS1.p4.1.m1.2.3.2.1"></times><ci id="S3.SS1.p4.1.m1.2.3.2.2.cmml" xref="S3.SS1.p4.1.m1.2.3.2.2">𝑠</ci><ci id="S3.SS1.p4.1.m1.2.3.2.3.cmml" xref="S3.SS1.p4.1.m1.2.3.2.3">𝑖</ci><ci id="S3.SS1.p4.1.m1.2.3.2.4.cmml" xref="S3.SS1.p4.1.m1.2.3.2.4">𝑔</ci><ci id="S3.SS1.p4.1.m1.2.3.2.5.cmml" xref="S3.SS1.p4.1.m1.2.3.2.5">𝑛</ci><apply id="S3.SS1.p4.1.m1.1.1.cmml" xref="S3.SS1.p4.1.m1.2.3.2.6.2"><divide id="S3.SS1.p4.1.m1.1.1.1.cmml" xref="S3.SS1.p4.1.m1.2.3.2.6.2"></divide><apply id="S3.SS1.p4.1.m1.1.1.2.cmml" xref="S3.SS1.p4.1.m1.1.1.2"><partialdiff id="S3.SS1.p4.1.m1.1.1.2.1.cmml" xref="S3.SS1.p4.1.m1.1.1.2.1"></partialdiff><ci id="S3.SS1.p4.1.m1.1.1.2.2.cmml" xref="S3.SS1.p4.1.m1.1.1.2.2">𝐿</ci></apply><apply id="S3.SS1.p4.1.m1.1.1.3.cmml" xref="S3.SS1.p4.1.m1.1.1.3"><partialdiff id="S3.SS1.p4.1.m1.1.1.3.1.cmml" xref="S3.SS1.p4.1.m1.1.1.3.1"></partialdiff><ci id="S3.SS1.p4.1.m1.1.1.3.2.cmml" xref="S3.SS1.p4.1.m1.1.1.3.2">𝑉</ci></apply></apply></apply><apply id="S3.SS1.p4.1.m1.2.3.3.cmml" xref="S3.SS1.p4.1.m1.2.3.3"><times id="S3.SS1.p4.1.m1.2.3.3.1.cmml" xref="S3.SS1.p4.1.m1.2.3.3.1"></times><ci id="S3.SS1.p4.1.m1.2.3.3.2.cmml" xref="S3.SS1.p4.1.m1.2.3.3.2">𝑠</ci><ci id="S3.SS1.p4.1.m1.2.3.3.3.cmml" xref="S3.SS1.p4.1.m1.2.3.3.3">𝑖</ci><ci id="S3.SS1.p4.1.m1.2.3.3.4.cmml" xref="S3.SS1.p4.1.m1.2.3.3.4">𝑔</ci><ci id="S3.SS1.p4.1.m1.2.3.3.5.cmml" xref="S3.SS1.p4.1.m1.2.3.3.5">𝑛</ci><apply id="S3.SS1.p4.1.m1.2.2.cmml" xref="S3.SS1.p4.1.m1.2.3.3.6.2"><divide id="S3.SS1.p4.1.m1.2.2.1.cmml" xref="S3.SS1.p4.1.m1.2.3.3.6.2"></divide><apply id="S3.SS1.p4.1.m1.2.2.2.cmml" xref="S3.SS1.p4.1.m1.2.2.2"><partialdiff id="S3.SS1.p4.1.m1.2.2.2.1.cmml" xref="S3.SS1.p4.1.m1.2.2.2.1"></partialdiff><ci id="S3.SS1.p4.1.m1.2.2.2.2.cmml" xref="S3.SS1.p4.1.m1.2.2.2.2">𝐿</ci></apply><apply id="S3.SS1.p4.1.m1.2.2.3.cmml" xref="S3.SS1.p4.1.m1.2.2.3"><partialdiff id="S3.SS1.p4.1.m1.2.2.3.1.cmml" xref="S3.SS1.p4.1.m1.2.2.3.1"></partialdiff><ci id="S3.SS1.p4.1.m1.2.2.3.2.cmml" xref="S3.SS1.p4.1.m1.2.2.3.2">𝑊</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p4.1.m1.2c">sign(\frac{\partial L}{\partial V})=sign(\frac{\partial L}{\partial W})</annotation></semantics></math> in Eq. <a href="#S3.E5" title="In 3.1 SignRound ‣ 3 Methodology ‣ Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a> as following since elements of <math id="S3.SS1.p4.2.m2.1" class="ltx_Math" alttext="s" display="inline"><semantics id="S3.SS1.p4.2.m2.1a"><mi id="S3.SS1.p4.2.m2.1.1" xref="S3.SS1.p4.2.m2.1.1.cmml">s</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p4.2.m2.1b"><ci id="S3.SS1.p4.2.m2.1.1.cmml" xref="S3.SS1.p4.2.m2.1.1">𝑠</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p4.2.m2.1c">s</annotation></semantics></math> are all positive.</p>
</div>
<div id="S3.SS1.p5" class="ltx_para ltx_noindent">
<table id="S3.E6" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S3.E6.m1.1" class="ltx_Math" alttext="\frac{\partial L}{\partial W}=-2(WX-\widetilde{W}X)X^{T}" display="block"><semantics id="S3.E6.m1.1a"><mrow id="S3.E6.m1.1.1" xref="S3.E6.m1.1.1.cmml"><mfrac id="S3.E6.m1.1.1.3" xref="S3.E6.m1.1.1.3.cmml"><mrow id="S3.E6.m1.1.1.3.2" xref="S3.E6.m1.1.1.3.2.cmml"><mo rspace="0em" id="S3.E6.m1.1.1.3.2.1" xref="S3.E6.m1.1.1.3.2.1.cmml">∂</mo><mi id="S3.E6.m1.1.1.3.2.2" xref="S3.E6.m1.1.1.3.2.2.cmml">L</mi></mrow><mrow id="S3.E6.m1.1.1.3.3" xref="S3.E6.m1.1.1.3.3.cmml"><mo rspace="0em" id="S3.E6.m1.1.1.3.3.1" xref="S3.E6.m1.1.1.3.3.1.cmml">∂</mo><mi id="S3.E6.m1.1.1.3.3.2" xref="S3.E6.m1.1.1.3.3.2.cmml">W</mi></mrow></mfrac><mo id="S3.E6.m1.1.1.2" xref="S3.E6.m1.1.1.2.cmml">=</mo><mrow id="S3.E6.m1.1.1.1" xref="S3.E6.m1.1.1.1.cmml"><mo id="S3.E6.m1.1.1.1a" xref="S3.E6.m1.1.1.1.cmml">−</mo><mrow id="S3.E6.m1.1.1.1.1" xref="S3.E6.m1.1.1.1.1.cmml"><mn id="S3.E6.m1.1.1.1.1.3" xref="S3.E6.m1.1.1.1.1.3.cmml">2</mn><mo lspace="0em" rspace="0em" id="S3.E6.m1.1.1.1.1.2" xref="S3.E6.m1.1.1.1.1.2.cmml">​</mo><mrow id="S3.E6.m1.1.1.1.1.1.1" xref="S3.E6.m1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.E6.m1.1.1.1.1.1.1.2" xref="S3.E6.m1.1.1.1.1.1.1.1.cmml">(</mo><mrow id="S3.E6.m1.1.1.1.1.1.1.1" xref="S3.E6.m1.1.1.1.1.1.1.1.cmml"><mrow id="S3.E6.m1.1.1.1.1.1.1.1.2" xref="S3.E6.m1.1.1.1.1.1.1.1.2.cmml"><mi id="S3.E6.m1.1.1.1.1.1.1.1.2.2" xref="S3.E6.m1.1.1.1.1.1.1.1.2.2.cmml">W</mi><mo lspace="0em" rspace="0em" id="S3.E6.m1.1.1.1.1.1.1.1.2.1" xref="S3.E6.m1.1.1.1.1.1.1.1.2.1.cmml">​</mo><mi id="S3.E6.m1.1.1.1.1.1.1.1.2.3" xref="S3.E6.m1.1.1.1.1.1.1.1.2.3.cmml">X</mi></mrow><mo id="S3.E6.m1.1.1.1.1.1.1.1.1" xref="S3.E6.m1.1.1.1.1.1.1.1.1.cmml">−</mo><mrow id="S3.E6.m1.1.1.1.1.1.1.1.3" xref="S3.E6.m1.1.1.1.1.1.1.1.3.cmml"><mover accent="true" id="S3.E6.m1.1.1.1.1.1.1.1.3.2" xref="S3.E6.m1.1.1.1.1.1.1.1.3.2.cmml"><mi id="S3.E6.m1.1.1.1.1.1.1.1.3.2.2" xref="S3.E6.m1.1.1.1.1.1.1.1.3.2.2.cmml">W</mi><mo id="S3.E6.m1.1.1.1.1.1.1.1.3.2.1" xref="S3.E6.m1.1.1.1.1.1.1.1.3.2.1.cmml">~</mo></mover><mo lspace="0em" rspace="0em" id="S3.E6.m1.1.1.1.1.1.1.1.3.1" xref="S3.E6.m1.1.1.1.1.1.1.1.3.1.cmml">​</mo><mi id="S3.E6.m1.1.1.1.1.1.1.1.3.3" xref="S3.E6.m1.1.1.1.1.1.1.1.3.3.cmml">X</mi></mrow></mrow><mo stretchy="false" id="S3.E6.m1.1.1.1.1.1.1.3" xref="S3.E6.m1.1.1.1.1.1.1.1.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="S3.E6.m1.1.1.1.1.2a" xref="S3.E6.m1.1.1.1.1.2.cmml">​</mo><msup id="S3.E6.m1.1.1.1.1.4" xref="S3.E6.m1.1.1.1.1.4.cmml"><mi id="S3.E6.m1.1.1.1.1.4.2" xref="S3.E6.m1.1.1.1.1.4.2.cmml">X</mi><mi id="S3.E6.m1.1.1.1.1.4.3" xref="S3.E6.m1.1.1.1.1.4.3.cmml">T</mi></msup></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.E6.m1.1b"><apply id="S3.E6.m1.1.1.cmml" xref="S3.E6.m1.1.1"><eq id="S3.E6.m1.1.1.2.cmml" xref="S3.E6.m1.1.1.2"></eq><apply id="S3.E6.m1.1.1.3.cmml" xref="S3.E6.m1.1.1.3"><divide id="S3.E6.m1.1.1.3.1.cmml" xref="S3.E6.m1.1.1.3"></divide><apply id="S3.E6.m1.1.1.3.2.cmml" xref="S3.E6.m1.1.1.3.2"><partialdiff id="S3.E6.m1.1.1.3.2.1.cmml" xref="S3.E6.m1.1.1.3.2.1"></partialdiff><ci id="S3.E6.m1.1.1.3.2.2.cmml" xref="S3.E6.m1.1.1.3.2.2">𝐿</ci></apply><apply id="S3.E6.m1.1.1.3.3.cmml" xref="S3.E6.m1.1.1.3.3"><partialdiff id="S3.E6.m1.1.1.3.3.1.cmml" xref="S3.E6.m1.1.1.3.3.1"></partialdiff><ci id="S3.E6.m1.1.1.3.3.2.cmml" xref="S3.E6.m1.1.1.3.3.2">𝑊</ci></apply></apply><apply id="S3.E6.m1.1.1.1.cmml" xref="S3.E6.m1.1.1.1"><minus id="S3.E6.m1.1.1.1.2.cmml" xref="S3.E6.m1.1.1.1"></minus><apply id="S3.E6.m1.1.1.1.1.cmml" xref="S3.E6.m1.1.1.1.1"><times id="S3.E6.m1.1.1.1.1.2.cmml" xref="S3.E6.m1.1.1.1.1.2"></times><cn type="integer" id="S3.E6.m1.1.1.1.1.3.cmml" xref="S3.E6.m1.1.1.1.1.3">2</cn><apply id="S3.E6.m1.1.1.1.1.1.1.1.cmml" xref="S3.E6.m1.1.1.1.1.1.1"><minus id="S3.E6.m1.1.1.1.1.1.1.1.1.cmml" xref="S3.E6.m1.1.1.1.1.1.1.1.1"></minus><apply id="S3.E6.m1.1.1.1.1.1.1.1.2.cmml" xref="S3.E6.m1.1.1.1.1.1.1.1.2"><times id="S3.E6.m1.1.1.1.1.1.1.1.2.1.cmml" xref="S3.E6.m1.1.1.1.1.1.1.1.2.1"></times><ci id="S3.E6.m1.1.1.1.1.1.1.1.2.2.cmml" xref="S3.E6.m1.1.1.1.1.1.1.1.2.2">𝑊</ci><ci id="S3.E6.m1.1.1.1.1.1.1.1.2.3.cmml" xref="S3.E6.m1.1.1.1.1.1.1.1.2.3">𝑋</ci></apply><apply id="S3.E6.m1.1.1.1.1.1.1.1.3.cmml" xref="S3.E6.m1.1.1.1.1.1.1.1.3"><times id="S3.E6.m1.1.1.1.1.1.1.1.3.1.cmml" xref="S3.E6.m1.1.1.1.1.1.1.1.3.1"></times><apply id="S3.E6.m1.1.1.1.1.1.1.1.3.2.cmml" xref="S3.E6.m1.1.1.1.1.1.1.1.3.2"><ci id="S3.E6.m1.1.1.1.1.1.1.1.3.2.1.cmml" xref="S3.E6.m1.1.1.1.1.1.1.1.3.2.1">~</ci><ci id="S3.E6.m1.1.1.1.1.1.1.1.3.2.2.cmml" xref="S3.E6.m1.1.1.1.1.1.1.1.3.2.2">𝑊</ci></apply><ci id="S3.E6.m1.1.1.1.1.1.1.1.3.3.cmml" xref="S3.E6.m1.1.1.1.1.1.1.1.3.3">𝑋</ci></apply></apply><apply id="S3.E6.m1.1.1.1.1.4.cmml" xref="S3.E6.m1.1.1.1.1.4"><csymbol cd="ambiguous" id="S3.E6.m1.1.1.1.1.4.1.cmml" xref="S3.E6.m1.1.1.1.1.4">superscript</csymbol><ci id="S3.E6.m1.1.1.1.1.4.2.cmml" xref="S3.E6.m1.1.1.1.1.4.2">𝑋</ci><ci id="S3.E6.m1.1.1.1.1.4.3.cmml" xref="S3.E6.m1.1.1.1.1.4.3">𝑇</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E6.m1.1c">\frac{\partial L}{\partial W}=-2(WX-\widetilde{W}X)X^{T}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(6)</span></td>
</tr></tbody>
</table>
</div>
<div id="S3.SS1.p6" class="ltx_para ltx_noindent">
<table id="S3.E7" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S3.E7.m1.1" class="ltx_Math" alttext="\frac{\partial L}{\partial V}=-2s(WX-\widetilde{W}X)X^{T}" display="block"><semantics id="S3.E7.m1.1a"><mrow id="S3.E7.m1.1.1" xref="S3.E7.m1.1.1.cmml"><mfrac id="S3.E7.m1.1.1.3" xref="S3.E7.m1.1.1.3.cmml"><mrow id="S3.E7.m1.1.1.3.2" xref="S3.E7.m1.1.1.3.2.cmml"><mo rspace="0em" id="S3.E7.m1.1.1.3.2.1" xref="S3.E7.m1.1.1.3.2.1.cmml">∂</mo><mi id="S3.E7.m1.1.1.3.2.2" xref="S3.E7.m1.1.1.3.2.2.cmml">L</mi></mrow><mrow id="S3.E7.m1.1.1.3.3" xref="S3.E7.m1.1.1.3.3.cmml"><mo rspace="0em" id="S3.E7.m1.1.1.3.3.1" xref="S3.E7.m1.1.1.3.3.1.cmml">∂</mo><mi id="S3.E7.m1.1.1.3.3.2" xref="S3.E7.m1.1.1.3.3.2.cmml">V</mi></mrow></mfrac><mo id="S3.E7.m1.1.1.2" xref="S3.E7.m1.1.1.2.cmml">=</mo><mrow id="S3.E7.m1.1.1.1" xref="S3.E7.m1.1.1.1.cmml"><mo id="S3.E7.m1.1.1.1a" xref="S3.E7.m1.1.1.1.cmml">−</mo><mrow id="S3.E7.m1.1.1.1.1" xref="S3.E7.m1.1.1.1.1.cmml"><mn id="S3.E7.m1.1.1.1.1.3" xref="S3.E7.m1.1.1.1.1.3.cmml">2</mn><mo lspace="0em" rspace="0em" id="S3.E7.m1.1.1.1.1.2" xref="S3.E7.m1.1.1.1.1.2.cmml">​</mo><mi id="S3.E7.m1.1.1.1.1.4" xref="S3.E7.m1.1.1.1.1.4.cmml">s</mi><mo lspace="0em" rspace="0em" id="S3.E7.m1.1.1.1.1.2a" xref="S3.E7.m1.1.1.1.1.2.cmml">​</mo><mrow id="S3.E7.m1.1.1.1.1.1.1" xref="S3.E7.m1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.E7.m1.1.1.1.1.1.1.2" xref="S3.E7.m1.1.1.1.1.1.1.1.cmml">(</mo><mrow id="S3.E7.m1.1.1.1.1.1.1.1" xref="S3.E7.m1.1.1.1.1.1.1.1.cmml"><mrow id="S3.E7.m1.1.1.1.1.1.1.1.2" xref="S3.E7.m1.1.1.1.1.1.1.1.2.cmml"><mi id="S3.E7.m1.1.1.1.1.1.1.1.2.2" xref="S3.E7.m1.1.1.1.1.1.1.1.2.2.cmml">W</mi><mo lspace="0em" rspace="0em" id="S3.E7.m1.1.1.1.1.1.1.1.2.1" xref="S3.E7.m1.1.1.1.1.1.1.1.2.1.cmml">​</mo><mi id="S3.E7.m1.1.1.1.1.1.1.1.2.3" xref="S3.E7.m1.1.1.1.1.1.1.1.2.3.cmml">X</mi></mrow><mo id="S3.E7.m1.1.1.1.1.1.1.1.1" xref="S3.E7.m1.1.1.1.1.1.1.1.1.cmml">−</mo><mrow id="S3.E7.m1.1.1.1.1.1.1.1.3" xref="S3.E7.m1.1.1.1.1.1.1.1.3.cmml"><mover accent="true" id="S3.E7.m1.1.1.1.1.1.1.1.3.2" xref="S3.E7.m1.1.1.1.1.1.1.1.3.2.cmml"><mi id="S3.E7.m1.1.1.1.1.1.1.1.3.2.2" xref="S3.E7.m1.1.1.1.1.1.1.1.3.2.2.cmml">W</mi><mo id="S3.E7.m1.1.1.1.1.1.1.1.3.2.1" xref="S3.E7.m1.1.1.1.1.1.1.1.3.2.1.cmml">~</mo></mover><mo lspace="0em" rspace="0em" id="S3.E7.m1.1.1.1.1.1.1.1.3.1" xref="S3.E7.m1.1.1.1.1.1.1.1.3.1.cmml">​</mo><mi id="S3.E7.m1.1.1.1.1.1.1.1.3.3" xref="S3.E7.m1.1.1.1.1.1.1.1.3.3.cmml">X</mi></mrow></mrow><mo stretchy="false" id="S3.E7.m1.1.1.1.1.1.1.3" xref="S3.E7.m1.1.1.1.1.1.1.1.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="S3.E7.m1.1.1.1.1.2b" xref="S3.E7.m1.1.1.1.1.2.cmml">​</mo><msup id="S3.E7.m1.1.1.1.1.5" xref="S3.E7.m1.1.1.1.1.5.cmml"><mi id="S3.E7.m1.1.1.1.1.5.2" xref="S3.E7.m1.1.1.1.1.5.2.cmml">X</mi><mi id="S3.E7.m1.1.1.1.1.5.3" xref="S3.E7.m1.1.1.1.1.5.3.cmml">T</mi></msup></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.E7.m1.1b"><apply id="S3.E7.m1.1.1.cmml" xref="S3.E7.m1.1.1"><eq id="S3.E7.m1.1.1.2.cmml" xref="S3.E7.m1.1.1.2"></eq><apply id="S3.E7.m1.1.1.3.cmml" xref="S3.E7.m1.1.1.3"><divide id="S3.E7.m1.1.1.3.1.cmml" xref="S3.E7.m1.1.1.3"></divide><apply id="S3.E7.m1.1.1.3.2.cmml" xref="S3.E7.m1.1.1.3.2"><partialdiff id="S3.E7.m1.1.1.3.2.1.cmml" xref="S3.E7.m1.1.1.3.2.1"></partialdiff><ci id="S3.E7.m1.1.1.3.2.2.cmml" xref="S3.E7.m1.1.1.3.2.2">𝐿</ci></apply><apply id="S3.E7.m1.1.1.3.3.cmml" xref="S3.E7.m1.1.1.3.3"><partialdiff id="S3.E7.m1.1.1.3.3.1.cmml" xref="S3.E7.m1.1.1.3.3.1"></partialdiff><ci id="S3.E7.m1.1.1.3.3.2.cmml" xref="S3.E7.m1.1.1.3.3.2">𝑉</ci></apply></apply><apply id="S3.E7.m1.1.1.1.cmml" xref="S3.E7.m1.1.1.1"><minus id="S3.E7.m1.1.1.1.2.cmml" xref="S3.E7.m1.1.1.1"></minus><apply id="S3.E7.m1.1.1.1.1.cmml" xref="S3.E7.m1.1.1.1.1"><times id="S3.E7.m1.1.1.1.1.2.cmml" xref="S3.E7.m1.1.1.1.1.2"></times><cn type="integer" id="S3.E7.m1.1.1.1.1.3.cmml" xref="S3.E7.m1.1.1.1.1.3">2</cn><ci id="S3.E7.m1.1.1.1.1.4.cmml" xref="S3.E7.m1.1.1.1.1.4">𝑠</ci><apply id="S3.E7.m1.1.1.1.1.1.1.1.cmml" xref="S3.E7.m1.1.1.1.1.1.1"><minus id="S3.E7.m1.1.1.1.1.1.1.1.1.cmml" xref="S3.E7.m1.1.1.1.1.1.1.1.1"></minus><apply id="S3.E7.m1.1.1.1.1.1.1.1.2.cmml" xref="S3.E7.m1.1.1.1.1.1.1.1.2"><times id="S3.E7.m1.1.1.1.1.1.1.1.2.1.cmml" xref="S3.E7.m1.1.1.1.1.1.1.1.2.1"></times><ci id="S3.E7.m1.1.1.1.1.1.1.1.2.2.cmml" xref="S3.E7.m1.1.1.1.1.1.1.1.2.2">𝑊</ci><ci id="S3.E7.m1.1.1.1.1.1.1.1.2.3.cmml" xref="S3.E7.m1.1.1.1.1.1.1.1.2.3">𝑋</ci></apply><apply id="S3.E7.m1.1.1.1.1.1.1.1.3.cmml" xref="S3.E7.m1.1.1.1.1.1.1.1.3"><times id="S3.E7.m1.1.1.1.1.1.1.1.3.1.cmml" xref="S3.E7.m1.1.1.1.1.1.1.1.3.1"></times><apply id="S3.E7.m1.1.1.1.1.1.1.1.3.2.cmml" xref="S3.E7.m1.1.1.1.1.1.1.1.3.2"><ci id="S3.E7.m1.1.1.1.1.1.1.1.3.2.1.cmml" xref="S3.E7.m1.1.1.1.1.1.1.1.3.2.1">~</ci><ci id="S3.E7.m1.1.1.1.1.1.1.1.3.2.2.cmml" xref="S3.E7.m1.1.1.1.1.1.1.1.3.2.2">𝑊</ci></apply><ci id="S3.E7.m1.1.1.1.1.1.1.1.3.3.cmml" xref="S3.E7.m1.1.1.1.1.1.1.1.3.3">𝑋</ci></apply></apply><apply id="S3.E7.m1.1.1.1.1.5.cmml" xref="S3.E7.m1.1.1.1.1.5"><csymbol cd="ambiguous" id="S3.E7.m1.1.1.1.1.5.1.cmml" xref="S3.E7.m1.1.1.1.1.5">superscript</csymbol><ci id="S3.E7.m1.1.1.1.1.5.2.cmml" xref="S3.E7.m1.1.1.1.1.5.2">𝑋</ci><ci id="S3.E7.m1.1.1.1.1.5.3.cmml" xref="S3.E7.m1.1.1.1.1.5.3">𝑇</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E7.m1.1c">\frac{\partial L}{\partial V}=-2s(WX-\widetilde{W}X)X^{T}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(7)</span></td>
</tr></tbody>
</table>
</div>
<div id="S3.SS1.p7" class="ltx_para ltx_noindent">
<p id="S3.SS1.p7.1" class="ltx_p">So our optimization could be simplified as</p>
</div>
<div id="S3.SS1.p8" class="ltx_para ltx_noindent">
<table id="S3.E8" class="ltx_equationgroup ltx_eqn_table">
<tbody>
<tr id="S3.E8X" class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_eqn_cell"></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="S3.E8X.2.1.1.m1.1" class="ltx_Math" alttext="\displaystyle V_{t+1}=V_{t}-lr*sign(\frac{\partial L}{\partial W})" display="inline"><semantics id="S3.E8X.2.1.1.m1.1a"><mrow id="S3.E8X.2.1.1.m1.1.2" xref="S3.E8X.2.1.1.m1.1.2.cmml"><msub id="S3.E8X.2.1.1.m1.1.2.2" xref="S3.E8X.2.1.1.m1.1.2.2.cmml"><mi id="S3.E8X.2.1.1.m1.1.2.2.2" xref="S3.E8X.2.1.1.m1.1.2.2.2.cmml">V</mi><mrow id="S3.E8X.2.1.1.m1.1.2.2.3" xref="S3.E8X.2.1.1.m1.1.2.2.3.cmml"><mi id="S3.E8X.2.1.1.m1.1.2.2.3.2" xref="S3.E8X.2.1.1.m1.1.2.2.3.2.cmml">t</mi><mo id="S3.E8X.2.1.1.m1.1.2.2.3.1" xref="S3.E8X.2.1.1.m1.1.2.2.3.1.cmml">+</mo><mn id="S3.E8X.2.1.1.m1.1.2.2.3.3" xref="S3.E8X.2.1.1.m1.1.2.2.3.3.cmml">1</mn></mrow></msub><mo id="S3.E8X.2.1.1.m1.1.2.1" xref="S3.E8X.2.1.1.m1.1.2.1.cmml">=</mo><mrow id="S3.E8X.2.1.1.m1.1.2.3" xref="S3.E8X.2.1.1.m1.1.2.3.cmml"><msub id="S3.E8X.2.1.1.m1.1.2.3.2" xref="S3.E8X.2.1.1.m1.1.2.3.2.cmml"><mi id="S3.E8X.2.1.1.m1.1.2.3.2.2" xref="S3.E8X.2.1.1.m1.1.2.3.2.2.cmml">V</mi><mi id="S3.E8X.2.1.1.m1.1.2.3.2.3" xref="S3.E8X.2.1.1.m1.1.2.3.2.3.cmml">t</mi></msub><mo id="S3.E8X.2.1.1.m1.1.2.3.1" xref="S3.E8X.2.1.1.m1.1.2.3.1.cmml">−</mo><mrow id="S3.E8X.2.1.1.m1.1.2.3.3" xref="S3.E8X.2.1.1.m1.1.2.3.3.cmml"><mrow id="S3.E8X.2.1.1.m1.1.2.3.3.2" xref="S3.E8X.2.1.1.m1.1.2.3.3.2.cmml"><mrow id="S3.E8X.2.1.1.m1.1.2.3.3.2.2" xref="S3.E8X.2.1.1.m1.1.2.3.3.2.2.cmml"><mi id="S3.E8X.2.1.1.m1.1.2.3.3.2.2.2" xref="S3.E8X.2.1.1.m1.1.2.3.3.2.2.2.cmml">l</mi><mo lspace="0em" rspace="0em" id="S3.E8X.2.1.1.m1.1.2.3.3.2.2.1" xref="S3.E8X.2.1.1.m1.1.2.3.3.2.2.1.cmml">​</mo><mi id="S3.E8X.2.1.1.m1.1.2.3.3.2.2.3" xref="S3.E8X.2.1.1.m1.1.2.3.3.2.2.3.cmml">r</mi></mrow><mo lspace="0.222em" rspace="0.222em" id="S3.E8X.2.1.1.m1.1.2.3.3.2.1" xref="S3.E8X.2.1.1.m1.1.2.3.3.2.1.cmml">∗</mo><mi id="S3.E8X.2.1.1.m1.1.2.3.3.2.3" xref="S3.E8X.2.1.1.m1.1.2.3.3.2.3.cmml">s</mi></mrow><mo lspace="0em" rspace="0em" id="S3.E8X.2.1.1.m1.1.2.3.3.1" xref="S3.E8X.2.1.1.m1.1.2.3.3.1.cmml">​</mo><mi id="S3.E8X.2.1.1.m1.1.2.3.3.3" xref="S3.E8X.2.1.1.m1.1.2.3.3.3.cmml">i</mi><mo lspace="0em" rspace="0em" id="S3.E8X.2.1.1.m1.1.2.3.3.1a" xref="S3.E8X.2.1.1.m1.1.2.3.3.1.cmml">​</mo><mi id="S3.E8X.2.1.1.m1.1.2.3.3.4" xref="S3.E8X.2.1.1.m1.1.2.3.3.4.cmml">g</mi><mo lspace="0em" rspace="0em" id="S3.E8X.2.1.1.m1.1.2.3.3.1b" xref="S3.E8X.2.1.1.m1.1.2.3.3.1.cmml">​</mo><mi id="S3.E8X.2.1.1.m1.1.2.3.3.5" xref="S3.E8X.2.1.1.m1.1.2.3.3.5.cmml">n</mi><mo lspace="0em" rspace="0em" id="S3.E8X.2.1.1.m1.1.2.3.3.1c" xref="S3.E8X.2.1.1.m1.1.2.3.3.1.cmml">​</mo><mrow id="S3.E8X.2.1.1.m1.1.2.3.3.6.2" xref="S3.E8X.2.1.1.m1.1.1.cmml"><mo stretchy="false" id="S3.E8X.2.1.1.m1.1.2.3.3.6.2.1" xref="S3.E8X.2.1.1.m1.1.1.cmml">(</mo><mstyle displaystyle="true" id="S3.E8X.2.1.1.m1.1.1" xref="S3.E8X.2.1.1.m1.1.1.cmml"><mfrac id="S3.E8X.2.1.1.m1.1.1a" xref="S3.E8X.2.1.1.m1.1.1.cmml"><mrow id="S3.E8X.2.1.1.m1.1.1.2" xref="S3.E8X.2.1.1.m1.1.1.2.cmml"><mo rspace="0em" id="S3.E8X.2.1.1.m1.1.1.2.1" xref="S3.E8X.2.1.1.m1.1.1.2.1.cmml">∂</mo><mi id="S3.E8X.2.1.1.m1.1.1.2.2" xref="S3.E8X.2.1.1.m1.1.1.2.2.cmml">L</mi></mrow><mrow id="S3.E8X.2.1.1.m1.1.1.3" xref="S3.E8X.2.1.1.m1.1.1.3.cmml"><mo rspace="0em" id="S3.E8X.2.1.1.m1.1.1.3.1" xref="S3.E8X.2.1.1.m1.1.1.3.1.cmml">∂</mo><mi id="S3.E8X.2.1.1.m1.1.1.3.2" xref="S3.E8X.2.1.1.m1.1.1.3.2.cmml">W</mi></mrow></mfrac></mstyle><mo stretchy="false" id="S3.E8X.2.1.1.m1.1.2.3.3.6.2.2" xref="S3.E8X.2.1.1.m1.1.1.cmml">)</mo></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.E8X.2.1.1.m1.1b"><apply id="S3.E8X.2.1.1.m1.1.2.cmml" xref="S3.E8X.2.1.1.m1.1.2"><eq id="S3.E8X.2.1.1.m1.1.2.1.cmml" xref="S3.E8X.2.1.1.m1.1.2.1"></eq><apply id="S3.E8X.2.1.1.m1.1.2.2.cmml" xref="S3.E8X.2.1.1.m1.1.2.2"><csymbol cd="ambiguous" id="S3.E8X.2.1.1.m1.1.2.2.1.cmml" xref="S3.E8X.2.1.1.m1.1.2.2">subscript</csymbol><ci id="S3.E8X.2.1.1.m1.1.2.2.2.cmml" xref="S3.E8X.2.1.1.m1.1.2.2.2">𝑉</ci><apply id="S3.E8X.2.1.1.m1.1.2.2.3.cmml" xref="S3.E8X.2.1.1.m1.1.2.2.3"><plus id="S3.E8X.2.1.1.m1.1.2.2.3.1.cmml" xref="S3.E8X.2.1.1.m1.1.2.2.3.1"></plus><ci id="S3.E8X.2.1.1.m1.1.2.2.3.2.cmml" xref="S3.E8X.2.1.1.m1.1.2.2.3.2">𝑡</ci><cn type="integer" id="S3.E8X.2.1.1.m1.1.2.2.3.3.cmml" xref="S3.E8X.2.1.1.m1.1.2.2.3.3">1</cn></apply></apply><apply id="S3.E8X.2.1.1.m1.1.2.3.cmml" xref="S3.E8X.2.1.1.m1.1.2.3"><minus id="S3.E8X.2.1.1.m1.1.2.3.1.cmml" xref="S3.E8X.2.1.1.m1.1.2.3.1"></minus><apply id="S3.E8X.2.1.1.m1.1.2.3.2.cmml" xref="S3.E8X.2.1.1.m1.1.2.3.2"><csymbol cd="ambiguous" id="S3.E8X.2.1.1.m1.1.2.3.2.1.cmml" xref="S3.E8X.2.1.1.m1.1.2.3.2">subscript</csymbol><ci id="S3.E8X.2.1.1.m1.1.2.3.2.2.cmml" xref="S3.E8X.2.1.1.m1.1.2.3.2.2">𝑉</ci><ci id="S3.E8X.2.1.1.m1.1.2.3.2.3.cmml" xref="S3.E8X.2.1.1.m1.1.2.3.2.3">𝑡</ci></apply><apply id="S3.E8X.2.1.1.m1.1.2.3.3.cmml" xref="S3.E8X.2.1.1.m1.1.2.3.3"><times id="S3.E8X.2.1.1.m1.1.2.3.3.1.cmml" xref="S3.E8X.2.1.1.m1.1.2.3.3.1"></times><apply id="S3.E8X.2.1.1.m1.1.2.3.3.2.cmml" xref="S3.E8X.2.1.1.m1.1.2.3.3.2"><times id="S3.E8X.2.1.1.m1.1.2.3.3.2.1.cmml" xref="S3.E8X.2.1.1.m1.1.2.3.3.2.1"></times><apply id="S3.E8X.2.1.1.m1.1.2.3.3.2.2.cmml" xref="S3.E8X.2.1.1.m1.1.2.3.3.2.2"><times id="S3.E8X.2.1.1.m1.1.2.3.3.2.2.1.cmml" xref="S3.E8X.2.1.1.m1.1.2.3.3.2.2.1"></times><ci id="S3.E8X.2.1.1.m1.1.2.3.3.2.2.2.cmml" xref="S3.E8X.2.1.1.m1.1.2.3.3.2.2.2">𝑙</ci><ci id="S3.E8X.2.1.1.m1.1.2.3.3.2.2.3.cmml" xref="S3.E8X.2.1.1.m1.1.2.3.3.2.2.3">𝑟</ci></apply><ci id="S3.E8X.2.1.1.m1.1.2.3.3.2.3.cmml" xref="S3.E8X.2.1.1.m1.1.2.3.3.2.3">𝑠</ci></apply><ci id="S3.E8X.2.1.1.m1.1.2.3.3.3.cmml" xref="S3.E8X.2.1.1.m1.1.2.3.3.3">𝑖</ci><ci id="S3.E8X.2.1.1.m1.1.2.3.3.4.cmml" xref="S3.E8X.2.1.1.m1.1.2.3.3.4">𝑔</ci><ci id="S3.E8X.2.1.1.m1.1.2.3.3.5.cmml" xref="S3.E8X.2.1.1.m1.1.2.3.3.5">𝑛</ci><apply id="S3.E8X.2.1.1.m1.1.1.cmml" xref="S3.E8X.2.1.1.m1.1.2.3.3.6.2"><divide id="S3.E8X.2.1.1.m1.1.1.1.cmml" xref="S3.E8X.2.1.1.m1.1.2.3.3.6.2"></divide><apply id="S3.E8X.2.1.1.m1.1.1.2.cmml" xref="S3.E8X.2.1.1.m1.1.1.2"><partialdiff id="S3.E8X.2.1.1.m1.1.1.2.1.cmml" xref="S3.E8X.2.1.1.m1.1.1.2.1"></partialdiff><ci id="S3.E8X.2.1.1.m1.1.1.2.2.cmml" xref="S3.E8X.2.1.1.m1.1.1.2.2">𝐿</ci></apply><apply id="S3.E8X.2.1.1.m1.1.1.3.cmml" xref="S3.E8X.2.1.1.m1.1.1.3"><partialdiff id="S3.E8X.2.1.1.m1.1.1.3.1.cmml" xref="S3.E8X.2.1.1.m1.1.1.3.1"></partialdiff><ci id="S3.E8X.2.1.1.m1.1.1.3.2.cmml" xref="S3.E8X.2.1.1.m1.1.1.3.2">𝑊</ci></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E8X.2.1.1.m1.1c">\displaystyle V_{t+1}=V_{t}-lr*sign(\frac{\partial L}{\partial W})</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="2" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equationgroup ltx_align_right">(8)</span></td>
</tr>
<tr id="S3.E8Xa" class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_eqn_cell"></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="S3.E8Xa.2.1.1.m1.4" class="ltx_Math" alttext="\displaystyle\mathrm{s.t.}|\sum_{t}lr*sign(\frac{\partial L}{\partial W})|\leq B" display="inline"><semantics id="S3.E8Xa.2.1.1.m1.4a"><mrow id="S3.E8Xa.2.1.1.m1.4.4.1" xref="S3.E8Xa.2.1.1.m1.4.4.2.cmml"><mi mathvariant="normal" id="S3.E8Xa.2.1.1.m1.2.2" xref="S3.E8Xa.2.1.1.m1.2.2.cmml">s</mi><mo lspace="0em" rspace="0.167em" id="S3.E8Xa.2.1.1.m1.4.4.1.2" xref="S3.E8Xa.2.1.1.m1.4.4.2a.cmml">.</mo><mi mathvariant="normal" id="S3.E8Xa.2.1.1.m1.3.3" xref="S3.E8Xa.2.1.1.m1.3.3.cmml">t</mi><mo lspace="0em" rspace="0.167em" id="S3.E8Xa.2.1.1.m1.4.4.1.3" xref="S3.E8Xa.2.1.1.m1.4.4.2a.cmml">.</mo><mrow id="S3.E8Xa.2.1.1.m1.4.4.1.1" xref="S3.E8Xa.2.1.1.m1.4.4.1.1.cmml"><mrow id="S3.E8Xa.2.1.1.m1.4.4.1.1.1.1" xref="S3.E8Xa.2.1.1.m1.4.4.1.1.1.2.cmml"><mo stretchy="false" id="S3.E8Xa.2.1.1.m1.4.4.1.1.1.1.2" xref="S3.E8Xa.2.1.1.m1.4.4.1.1.1.2.1.cmml">|</mo><mrow id="S3.E8Xa.2.1.1.m1.4.4.1.1.1.1.1" xref="S3.E8Xa.2.1.1.m1.4.4.1.1.1.1.1.cmml"><mstyle displaystyle="true" id="S3.E8Xa.2.1.1.m1.4.4.1.1.1.1.1.1" xref="S3.E8Xa.2.1.1.m1.4.4.1.1.1.1.1.1.cmml"><munder id="S3.E8Xa.2.1.1.m1.4.4.1.1.1.1.1.1a" xref="S3.E8Xa.2.1.1.m1.4.4.1.1.1.1.1.1.cmml"><mo movablelimits="false" id="S3.E8Xa.2.1.1.m1.4.4.1.1.1.1.1.1.2" xref="S3.E8Xa.2.1.1.m1.4.4.1.1.1.1.1.1.2.cmml">∑</mo><mi id="S3.E8Xa.2.1.1.m1.4.4.1.1.1.1.1.1.3" xref="S3.E8Xa.2.1.1.m1.4.4.1.1.1.1.1.1.3.cmml">t</mi></munder></mstyle><mrow id="S3.E8Xa.2.1.1.m1.4.4.1.1.1.1.1.2" xref="S3.E8Xa.2.1.1.m1.4.4.1.1.1.1.1.2.cmml"><mrow id="S3.E8Xa.2.1.1.m1.4.4.1.1.1.1.1.2.2" xref="S3.E8Xa.2.1.1.m1.4.4.1.1.1.1.1.2.2.cmml"><mrow id="S3.E8Xa.2.1.1.m1.4.4.1.1.1.1.1.2.2.2" xref="S3.E8Xa.2.1.1.m1.4.4.1.1.1.1.1.2.2.2.cmml"><mi id="S3.E8Xa.2.1.1.m1.4.4.1.1.1.1.1.2.2.2.2" xref="S3.E8Xa.2.1.1.m1.4.4.1.1.1.1.1.2.2.2.2.cmml">l</mi><mo lspace="0em" rspace="0em" id="S3.E8Xa.2.1.1.m1.4.4.1.1.1.1.1.2.2.2.1" xref="S3.E8Xa.2.1.1.m1.4.4.1.1.1.1.1.2.2.2.1.cmml">​</mo><mi id="S3.E8Xa.2.1.1.m1.4.4.1.1.1.1.1.2.2.2.3" xref="S3.E8Xa.2.1.1.m1.4.4.1.1.1.1.1.2.2.2.3.cmml">r</mi></mrow><mo lspace="0.222em" rspace="0.222em" id="S3.E8Xa.2.1.1.m1.4.4.1.1.1.1.1.2.2.1" xref="S3.E8Xa.2.1.1.m1.4.4.1.1.1.1.1.2.2.1.cmml">∗</mo><mi id="S3.E8Xa.2.1.1.m1.4.4.1.1.1.1.1.2.2.3" xref="S3.E8Xa.2.1.1.m1.4.4.1.1.1.1.1.2.2.3.cmml">s</mi></mrow><mo lspace="0em" rspace="0em" id="S3.E8Xa.2.1.1.m1.4.4.1.1.1.1.1.2.1" xref="S3.E8Xa.2.1.1.m1.4.4.1.1.1.1.1.2.1.cmml">​</mo><mi id="S3.E8Xa.2.1.1.m1.4.4.1.1.1.1.1.2.3" xref="S3.E8Xa.2.1.1.m1.4.4.1.1.1.1.1.2.3.cmml">i</mi><mo lspace="0em" rspace="0em" id="S3.E8Xa.2.1.1.m1.4.4.1.1.1.1.1.2.1a" xref="S3.E8Xa.2.1.1.m1.4.4.1.1.1.1.1.2.1.cmml">​</mo><mi id="S3.E8Xa.2.1.1.m1.4.4.1.1.1.1.1.2.4" xref="S3.E8Xa.2.1.1.m1.4.4.1.1.1.1.1.2.4.cmml">g</mi><mo lspace="0em" rspace="0em" id="S3.E8Xa.2.1.1.m1.4.4.1.1.1.1.1.2.1b" xref="S3.E8Xa.2.1.1.m1.4.4.1.1.1.1.1.2.1.cmml">​</mo><mi id="S3.E8Xa.2.1.1.m1.4.4.1.1.1.1.1.2.5" xref="S3.E8Xa.2.1.1.m1.4.4.1.1.1.1.1.2.5.cmml">n</mi><mo lspace="0em" rspace="0em" id="S3.E8Xa.2.1.1.m1.4.4.1.1.1.1.1.2.1c" xref="S3.E8Xa.2.1.1.m1.4.4.1.1.1.1.1.2.1.cmml">​</mo><mrow id="S3.E8Xa.2.1.1.m1.4.4.1.1.1.1.1.2.6.2" xref="S3.E8Xa.2.1.1.m1.1.1.cmml"><mo stretchy="false" id="S3.E8Xa.2.1.1.m1.4.4.1.1.1.1.1.2.6.2.1" xref="S3.E8Xa.2.1.1.m1.1.1.cmml">(</mo><mstyle displaystyle="true" id="S3.E8Xa.2.1.1.m1.1.1" xref="S3.E8Xa.2.1.1.m1.1.1.cmml"><mfrac id="S3.E8Xa.2.1.1.m1.1.1a" xref="S3.E8Xa.2.1.1.m1.1.1.cmml"><mrow id="S3.E8Xa.2.1.1.m1.1.1.2" xref="S3.E8Xa.2.1.1.m1.1.1.2.cmml"><mo rspace="0em" id="S3.E8Xa.2.1.1.m1.1.1.2.1" xref="S3.E8Xa.2.1.1.m1.1.1.2.1.cmml">∂</mo><mi id="S3.E8Xa.2.1.1.m1.1.1.2.2" xref="S3.E8Xa.2.1.1.m1.1.1.2.2.cmml">L</mi></mrow><mrow id="S3.E8Xa.2.1.1.m1.1.1.3" xref="S3.E8Xa.2.1.1.m1.1.1.3.cmml"><mo rspace="0em" id="S3.E8Xa.2.1.1.m1.1.1.3.1" xref="S3.E8Xa.2.1.1.m1.1.1.3.1.cmml">∂</mo><mi id="S3.E8Xa.2.1.1.m1.1.1.3.2" xref="S3.E8Xa.2.1.1.m1.1.1.3.2.cmml">W</mi></mrow></mfrac></mstyle><mo stretchy="false" id="S3.E8Xa.2.1.1.m1.4.4.1.1.1.1.1.2.6.2.2" xref="S3.E8Xa.2.1.1.m1.1.1.cmml">)</mo></mrow></mrow></mrow><mo stretchy="false" id="S3.E8Xa.2.1.1.m1.4.4.1.1.1.1.3" xref="S3.E8Xa.2.1.1.m1.4.4.1.1.1.2.1.cmml">|</mo></mrow><mo id="S3.E8Xa.2.1.1.m1.4.4.1.1.2" xref="S3.E8Xa.2.1.1.m1.4.4.1.1.2.cmml">≤</mo><mi id="S3.E8Xa.2.1.1.m1.4.4.1.1.3" xref="S3.E8Xa.2.1.1.m1.4.4.1.1.3.cmml">B</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.E8Xa.2.1.1.m1.4b"><apply id="S3.E8Xa.2.1.1.m1.4.4.2.cmml" xref="S3.E8Xa.2.1.1.m1.4.4.1"><csymbol cd="ambiguous" id="S3.E8Xa.2.1.1.m1.4.4.2a.cmml" xref="S3.E8Xa.2.1.1.m1.4.4.1.2">formulae-sequence</csymbol><ci id="S3.E8Xa.2.1.1.m1.2.2.cmml" xref="S3.E8Xa.2.1.1.m1.2.2">s</ci><ci id="S3.E8Xa.2.1.1.m1.3.3.cmml" xref="S3.E8Xa.2.1.1.m1.3.3">t</ci><apply id="S3.E8Xa.2.1.1.m1.4.4.1.1.cmml" xref="S3.E8Xa.2.1.1.m1.4.4.1.1"><leq id="S3.E8Xa.2.1.1.m1.4.4.1.1.2.cmml" xref="S3.E8Xa.2.1.1.m1.4.4.1.1.2"></leq><apply id="S3.E8Xa.2.1.1.m1.4.4.1.1.1.2.cmml" xref="S3.E8Xa.2.1.1.m1.4.4.1.1.1.1"><abs id="S3.E8Xa.2.1.1.m1.4.4.1.1.1.2.1.cmml" xref="S3.E8Xa.2.1.1.m1.4.4.1.1.1.1.2"></abs><apply id="S3.E8Xa.2.1.1.m1.4.4.1.1.1.1.1.cmml" xref="S3.E8Xa.2.1.1.m1.4.4.1.1.1.1.1"><apply id="S3.E8Xa.2.1.1.m1.4.4.1.1.1.1.1.1.cmml" xref="S3.E8Xa.2.1.1.m1.4.4.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.E8Xa.2.1.1.m1.4.4.1.1.1.1.1.1.1.cmml" xref="S3.E8Xa.2.1.1.m1.4.4.1.1.1.1.1.1">subscript</csymbol><sum id="S3.E8Xa.2.1.1.m1.4.4.1.1.1.1.1.1.2.cmml" xref="S3.E8Xa.2.1.1.m1.4.4.1.1.1.1.1.1.2"></sum><ci id="S3.E8Xa.2.1.1.m1.4.4.1.1.1.1.1.1.3.cmml" xref="S3.E8Xa.2.1.1.m1.4.4.1.1.1.1.1.1.3">𝑡</ci></apply><apply id="S3.E8Xa.2.1.1.m1.4.4.1.1.1.1.1.2.cmml" xref="S3.E8Xa.2.1.1.m1.4.4.1.1.1.1.1.2"><times id="S3.E8Xa.2.1.1.m1.4.4.1.1.1.1.1.2.1.cmml" xref="S3.E8Xa.2.1.1.m1.4.4.1.1.1.1.1.2.1"></times><apply id="S3.E8Xa.2.1.1.m1.4.4.1.1.1.1.1.2.2.cmml" xref="S3.E8Xa.2.1.1.m1.4.4.1.1.1.1.1.2.2"><times id="S3.E8Xa.2.1.1.m1.4.4.1.1.1.1.1.2.2.1.cmml" xref="S3.E8Xa.2.1.1.m1.4.4.1.1.1.1.1.2.2.1"></times><apply id="S3.E8Xa.2.1.1.m1.4.4.1.1.1.1.1.2.2.2.cmml" xref="S3.E8Xa.2.1.1.m1.4.4.1.1.1.1.1.2.2.2"><times id="S3.E8Xa.2.1.1.m1.4.4.1.1.1.1.1.2.2.2.1.cmml" xref="S3.E8Xa.2.1.1.m1.4.4.1.1.1.1.1.2.2.2.1"></times><ci id="S3.E8Xa.2.1.1.m1.4.4.1.1.1.1.1.2.2.2.2.cmml" xref="S3.E8Xa.2.1.1.m1.4.4.1.1.1.1.1.2.2.2.2">𝑙</ci><ci id="S3.E8Xa.2.1.1.m1.4.4.1.1.1.1.1.2.2.2.3.cmml" xref="S3.E8Xa.2.1.1.m1.4.4.1.1.1.1.1.2.2.2.3">𝑟</ci></apply><ci id="S3.E8Xa.2.1.1.m1.4.4.1.1.1.1.1.2.2.3.cmml" xref="S3.E8Xa.2.1.1.m1.4.4.1.1.1.1.1.2.2.3">𝑠</ci></apply><ci id="S3.E8Xa.2.1.1.m1.4.4.1.1.1.1.1.2.3.cmml" xref="S3.E8Xa.2.1.1.m1.4.4.1.1.1.1.1.2.3">𝑖</ci><ci id="S3.E8Xa.2.1.1.m1.4.4.1.1.1.1.1.2.4.cmml" xref="S3.E8Xa.2.1.1.m1.4.4.1.1.1.1.1.2.4">𝑔</ci><ci id="S3.E8Xa.2.1.1.m1.4.4.1.1.1.1.1.2.5.cmml" xref="S3.E8Xa.2.1.1.m1.4.4.1.1.1.1.1.2.5">𝑛</ci><apply id="S3.E8Xa.2.1.1.m1.1.1.cmml" xref="S3.E8Xa.2.1.1.m1.4.4.1.1.1.1.1.2.6.2"><divide id="S3.E8Xa.2.1.1.m1.1.1.1.cmml" xref="S3.E8Xa.2.1.1.m1.4.4.1.1.1.1.1.2.6.2"></divide><apply id="S3.E8Xa.2.1.1.m1.1.1.2.cmml" xref="S3.E8Xa.2.1.1.m1.1.1.2"><partialdiff id="S3.E8Xa.2.1.1.m1.1.1.2.1.cmml" xref="S3.E8Xa.2.1.1.m1.1.1.2.1"></partialdiff><ci id="S3.E8Xa.2.1.1.m1.1.1.2.2.cmml" xref="S3.E8Xa.2.1.1.m1.1.1.2.2">𝐿</ci></apply><apply id="S3.E8Xa.2.1.1.m1.1.1.3.cmml" xref="S3.E8Xa.2.1.1.m1.1.1.3"><partialdiff id="S3.E8Xa.2.1.1.m1.1.1.3.1.cmml" xref="S3.E8Xa.2.1.1.m1.1.1.3.1"></partialdiff><ci id="S3.E8Xa.2.1.1.m1.1.1.3.2.cmml" xref="S3.E8Xa.2.1.1.m1.1.1.3.2">𝑊</ci></apply></apply></apply></apply></apply><ci id="S3.E8Xa.2.1.1.m1.4.4.1.1.3.cmml" xref="S3.E8Xa.2.1.1.m1.4.4.1.1.3">𝐵</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E8Xa.2.1.1.m1.4c">\displaystyle\mathrm{s.t.}|\sum_{t}lr*sign(\frac{\partial L}{\partial W})|\leq B</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr>
</tbody>
</table>
</div>
<div id="S3.SS1.p9" class="ltx_para ltx_noindent">
<p id="S3.SS1.p9.1" class="ltx_p">Moreover, as Eq <a href="#S3.E3" title="In 3 Methodology ‣ Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a> averages the loss of each element, which presumes that each one contributes equally to the network, that basically is not true. To alleviate this issue, we optimize the rounding task blockwise. To clarify, in our context, we use the term ’layer’ to refer to a linear/convolution layer, while ’block’ denotes a transformer block that typically consists of several linear layers.</p>
</div>
<div id="S3.SS1.p10" class="ltx_para ltx_noindent">
<p id="S3.SS1.p10.1" class="ltx_p">The above pseudocode <a href="#alg1" title="Algorithm 1 ‣ 3.1 SignRound ‣ 3 Methodology ‣ Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a> presents more details of SignRound.</p>
</div>
<figure id="alg1" class="ltx_float ltx_float_algorithm ltx_framed ltx_framed_top">

<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_float"><span id="alg1.11.1.1" class="ltx_text ltx_font_bold">Algorithm 1</span> </span> SignRound</figcaption><div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_1">
<p id="alg1.8" class="ltx_p ltx_figure_panel"><span id="alg1.8.1" class="ltx_text ltx_font_bold">Input:</span> Calibration Data <math id="alg1.1.m1.1" class="ltx_Math" alttext="\mathcal{D}" display="inline"><semantics id="alg1.1.m1.1a"><mi class="ltx_font_mathcaligraphic" id="alg1.1.m1.1.1" xref="alg1.1.m1.1.1.cmml">𝒟</mi><annotation-xml encoding="MathML-Content" id="alg1.1.m1.1b"><ci id="alg1.1.m1.1.1.cmml" xref="alg1.1.m1.1.1">𝒟</ci></annotation-xml><annotation encoding="application/x-tex" id="alg1.1.m1.1c">\mathcal{D}</annotation></semantics></math>, learning rate <math id="alg1.2.m2.1" class="ltx_Math" alttext="lr" display="inline"><semantics id="alg1.2.m2.1a"><mrow id="alg1.2.m2.1.1" xref="alg1.2.m2.1.1.cmml"><mi id="alg1.2.m2.1.1.2" xref="alg1.2.m2.1.1.2.cmml">l</mi><mo lspace="0em" rspace="0em" id="alg1.2.m2.1.1.1" xref="alg1.2.m2.1.1.1.cmml">​</mo><mi id="alg1.2.m2.1.1.3" xref="alg1.2.m2.1.1.3.cmml">r</mi></mrow><annotation-xml encoding="MathML-Content" id="alg1.2.m2.1b"><apply id="alg1.2.m2.1.1.cmml" xref="alg1.2.m2.1.1"><times id="alg1.2.m2.1.1.1.cmml" xref="alg1.2.m2.1.1.1"></times><ci id="alg1.2.m2.1.1.2.cmml" xref="alg1.2.m2.1.1.2">𝑙</ci><ci id="alg1.2.m2.1.1.3.cmml" xref="alg1.2.m2.1.1.3">𝑟</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.2.m2.1c">lr</annotation></semantics></math>, total steps <math id="alg1.3.m3.1" class="ltx_Math" alttext="T" display="inline"><semantics id="alg1.3.m3.1a"><mi id="alg1.3.m3.1.1" xref="alg1.3.m3.1.1.cmml">T</mi><annotation-xml encoding="MathML-Content" id="alg1.3.m3.1b"><ci id="alg1.3.m3.1.1.cmml" xref="alg1.3.m3.1.1">𝑇</ci></annotation-xml><annotation encoding="application/x-tex" id="alg1.3.m3.1c">T</annotation></semantics></math>, Model <math id="alg1.4.m4.1" class="ltx_Math" alttext="M" display="inline"><semantics id="alg1.4.m4.1a"><mi id="alg1.4.m4.1.1" xref="alg1.4.m4.1.1.cmml">M</mi><annotation-xml encoding="MathML-Content" id="alg1.4.m4.1b"><ci id="alg1.4.m4.1.1.cmml" xref="alg1.4.m4.1.1">𝑀</ci></annotation-xml><annotation encoding="application/x-tex" id="alg1.4.m4.1c">M</annotation></semantics></math>, block module <math id="alg1.5.m5.1" class="ltx_Math" alttext="m_{w}" display="inline"><semantics id="alg1.5.m5.1a"><msub id="alg1.5.m5.1.1" xref="alg1.5.m5.1.1.cmml"><mi id="alg1.5.m5.1.1.2" xref="alg1.5.m5.1.1.2.cmml">m</mi><mi id="alg1.5.m5.1.1.3" xref="alg1.5.m5.1.1.3.cmml">w</mi></msub><annotation-xml encoding="MathML-Content" id="alg1.5.m5.1b"><apply id="alg1.5.m5.1.1.cmml" xref="alg1.5.m5.1.1"><csymbol cd="ambiguous" id="alg1.5.m5.1.1.1.cmml" xref="alg1.5.m5.1.1">subscript</csymbol><ci id="alg1.5.m5.1.1.2.cmml" xref="alg1.5.m5.1.1.2">𝑚</ci><ci id="alg1.5.m5.1.1.3.cmml" xref="alg1.5.m5.1.1.3">𝑤</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.5.m5.1c">m_{w}</annotation></semantics></math> with weights <math id="alg1.6.m6.1" class="ltx_Math" alttext="w" display="inline"><semantics id="alg1.6.m6.1a"><mi id="alg1.6.m6.1.1" xref="alg1.6.m6.1.1.cmml">w</mi><annotation-xml encoding="MathML-Content" id="alg1.6.m6.1b"><ci id="alg1.6.m6.1.1.cmml" xref="alg1.6.m6.1.1">𝑤</ci></annotation-xml><annotation encoding="application/x-tex" id="alg1.6.m6.1c">w</annotation></semantics></math>, zero initialized <math id="alg1.7.m7.1" class="ltx_Math" alttext="V" display="inline"><semantics id="alg1.7.m7.1a"><mi id="alg1.7.m7.1.1" xref="alg1.7.m7.1.1.cmml">V</mi><annotation-xml encoding="MathML-Content" id="alg1.7.m7.1b"><ci id="alg1.7.m7.1.1.cmml" xref="alg1.7.m7.1.1">𝑉</ci></annotation-xml><annotation encoding="application/x-tex" id="alg1.7.m7.1c">V</annotation></semantics></math>, batch size <math id="alg1.8.m8.1" class="ltx_Math" alttext="bs" display="inline"><semantics id="alg1.8.m8.1a"><mrow id="alg1.8.m8.1.1" xref="alg1.8.m8.1.1.cmml"><mi id="alg1.8.m8.1.1.2" xref="alg1.8.m8.1.1.2.cmml">b</mi><mo lspace="0em" rspace="0em" id="alg1.8.m8.1.1.1" xref="alg1.8.m8.1.1.1.cmml">​</mo><mi id="alg1.8.m8.1.1.3" xref="alg1.8.m8.1.1.3.cmml">s</mi></mrow><annotation-xml encoding="MathML-Content" id="alg1.8.m8.1b"><apply id="alg1.8.m8.1.1.cmml" xref="alg1.8.m8.1.1"><times id="alg1.8.m8.1.1.1.cmml" xref="alg1.8.m8.1.1.1"></times><ci id="alg1.8.m8.1.1.2.cmml" xref="alg1.8.m8.1.1.2">𝑏</ci><ci id="alg1.8.m8.1.1.3.cmml" xref="alg1.8.m8.1.1.3">𝑠</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.8.m8.1c">bs</annotation></semantics></math></p>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<p id="alg1.9" class="ltx_p ltx_figure_panel"><span id="alg1.9.1" class="ltx_text ltx_font_bold">Output:</span> <math id="alg1.9.m1.1" class="ltx_Math" alttext="best\_V" display="inline"><semantics id="alg1.9.m1.1a"><mrow id="alg1.9.m1.1.1" xref="alg1.9.m1.1.1.cmml"><mi id="alg1.9.m1.1.1.2" xref="alg1.9.m1.1.1.2.cmml">b</mi><mo lspace="0em" rspace="0em" id="alg1.9.m1.1.1.1" xref="alg1.9.m1.1.1.1.cmml">​</mo><mi id="alg1.9.m1.1.1.3" xref="alg1.9.m1.1.1.3.cmml">e</mi><mo lspace="0em" rspace="0em" id="alg1.9.m1.1.1.1a" xref="alg1.9.m1.1.1.1.cmml">​</mo><mi id="alg1.9.m1.1.1.4" xref="alg1.9.m1.1.1.4.cmml">s</mi><mo lspace="0em" rspace="0em" id="alg1.9.m1.1.1.1b" xref="alg1.9.m1.1.1.1.cmml">​</mo><mi id="alg1.9.m1.1.1.5" xref="alg1.9.m1.1.1.5.cmml">t</mi><mo lspace="0em" rspace="0em" id="alg1.9.m1.1.1.1c" xref="alg1.9.m1.1.1.1.cmml">​</mo><mi mathvariant="normal" id="alg1.9.m1.1.1.6" xref="alg1.9.m1.1.1.6.cmml">_</mi><mo lspace="0em" rspace="0em" id="alg1.9.m1.1.1.1d" xref="alg1.9.m1.1.1.1.cmml">​</mo><mi id="alg1.9.m1.1.1.7" xref="alg1.9.m1.1.1.7.cmml">V</mi></mrow><annotation-xml encoding="MathML-Content" id="alg1.9.m1.1b"><apply id="alg1.9.m1.1.1.cmml" xref="alg1.9.m1.1.1"><times id="alg1.9.m1.1.1.1.cmml" xref="alg1.9.m1.1.1.1"></times><ci id="alg1.9.m1.1.1.2.cmml" xref="alg1.9.m1.1.1.2">𝑏</ci><ci id="alg1.9.m1.1.1.3.cmml" xref="alg1.9.m1.1.1.3">𝑒</ci><ci id="alg1.9.m1.1.1.4.cmml" xref="alg1.9.m1.1.1.4">𝑠</ci><ci id="alg1.9.m1.1.1.5.cmml" xref="alg1.9.m1.1.1.5">𝑡</ci><ci id="alg1.9.m1.1.1.6.cmml" xref="alg1.9.m1.1.1.6">_</ci><ci id="alg1.9.m1.1.1.7.cmml" xref="alg1.9.m1.1.1.7">𝑉</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.9.m1.1c">best\_V</annotation></semantics></math></p>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<div id="alg1.12" class="ltx_listing ltx_figure_panel ltx_listing">
<div id="alg1.l1" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">1:</span><math id="alg1.l1.m1.1" class="ltx_Math" alttext="V\leftarrow 0" display="inline"><semantics id="alg1.l1.m1.1a"><mrow id="alg1.l1.m1.1.1" xref="alg1.l1.m1.1.1.cmml"><mi id="alg1.l1.m1.1.1.2" xref="alg1.l1.m1.1.1.2.cmml">V</mi><mo stretchy="false" id="alg1.l1.m1.1.1.1" xref="alg1.l1.m1.1.1.1.cmml">←</mo><mn id="alg1.l1.m1.1.1.3" xref="alg1.l1.m1.1.1.3.cmml">0</mn></mrow><annotation-xml encoding="MathML-Content" id="alg1.l1.m1.1b"><apply id="alg1.l1.m1.1.1.cmml" xref="alg1.l1.m1.1.1"><ci id="alg1.l1.m1.1.1.1.cmml" xref="alg1.l1.m1.1.1.1">←</ci><ci id="alg1.l1.m1.1.1.2.cmml" xref="alg1.l1.m1.1.1.2">𝑉</ci><cn type="integer" id="alg1.l1.m1.1.1.3.cmml" xref="alg1.l1.m1.1.1.3">0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l1.m1.1c">V\leftarrow 0</annotation></semantics></math>, <math id="alg1.l1.m2.1" class="ltx_Math" alttext="best\_V\leftarrow 0" display="inline"><semantics id="alg1.l1.m2.1a"><mrow id="alg1.l1.m2.1.1" xref="alg1.l1.m2.1.1.cmml"><mrow id="alg1.l1.m2.1.1.2" xref="alg1.l1.m2.1.1.2.cmml"><mi id="alg1.l1.m2.1.1.2.2" xref="alg1.l1.m2.1.1.2.2.cmml">b</mi><mo lspace="0em" rspace="0em" id="alg1.l1.m2.1.1.2.1" xref="alg1.l1.m2.1.1.2.1.cmml">​</mo><mi id="alg1.l1.m2.1.1.2.3" xref="alg1.l1.m2.1.1.2.3.cmml">e</mi><mo lspace="0em" rspace="0em" id="alg1.l1.m2.1.1.2.1a" xref="alg1.l1.m2.1.1.2.1.cmml">​</mo><mi id="alg1.l1.m2.1.1.2.4" xref="alg1.l1.m2.1.1.2.4.cmml">s</mi><mo lspace="0em" rspace="0em" id="alg1.l1.m2.1.1.2.1b" xref="alg1.l1.m2.1.1.2.1.cmml">​</mo><mi id="alg1.l1.m2.1.1.2.5" xref="alg1.l1.m2.1.1.2.5.cmml">t</mi><mo lspace="0em" rspace="0em" id="alg1.l1.m2.1.1.2.1c" xref="alg1.l1.m2.1.1.2.1.cmml">​</mo><mi mathvariant="normal" id="alg1.l1.m2.1.1.2.6" xref="alg1.l1.m2.1.1.2.6.cmml">_</mi><mo lspace="0em" rspace="0em" id="alg1.l1.m2.1.1.2.1d" xref="alg1.l1.m2.1.1.2.1.cmml">​</mo><mi id="alg1.l1.m2.1.1.2.7" xref="alg1.l1.m2.1.1.2.7.cmml">V</mi></mrow><mo stretchy="false" id="alg1.l1.m2.1.1.1" xref="alg1.l1.m2.1.1.1.cmml">←</mo><mn id="alg1.l1.m2.1.1.3" xref="alg1.l1.m2.1.1.3.cmml">0</mn></mrow><annotation-xml encoding="MathML-Content" id="alg1.l1.m2.1b"><apply id="alg1.l1.m2.1.1.cmml" xref="alg1.l1.m2.1.1"><ci id="alg1.l1.m2.1.1.1.cmml" xref="alg1.l1.m2.1.1.1">←</ci><apply id="alg1.l1.m2.1.1.2.cmml" xref="alg1.l1.m2.1.1.2"><times id="alg1.l1.m2.1.1.2.1.cmml" xref="alg1.l1.m2.1.1.2.1"></times><ci id="alg1.l1.m2.1.1.2.2.cmml" xref="alg1.l1.m2.1.1.2.2">𝑏</ci><ci id="alg1.l1.m2.1.1.2.3.cmml" xref="alg1.l1.m2.1.1.2.3">𝑒</ci><ci id="alg1.l1.m2.1.1.2.4.cmml" xref="alg1.l1.m2.1.1.2.4">𝑠</ci><ci id="alg1.l1.m2.1.1.2.5.cmml" xref="alg1.l1.m2.1.1.2.5">𝑡</ci><ci id="alg1.l1.m2.1.1.2.6.cmml" xref="alg1.l1.m2.1.1.2.6">_</ci><ci id="alg1.l1.m2.1.1.2.7.cmml" xref="alg1.l1.m2.1.1.2.7">𝑉</ci></apply><cn type="integer" id="alg1.l1.m2.1.1.3.cmml" xref="alg1.l1.m2.1.1.3">0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l1.m2.1c">best\_V\leftarrow 0</annotation></semantics></math>, <math id="alg1.l1.m3.1" class="ltx_Math" alttext="best\_l\leftarrow maximum" display="inline"><semantics id="alg1.l1.m3.1a"><mrow id="alg1.l1.m3.1.1" xref="alg1.l1.m3.1.1.cmml"><mrow id="alg1.l1.m3.1.1.2" xref="alg1.l1.m3.1.1.2.cmml"><mi id="alg1.l1.m3.1.1.2.2" xref="alg1.l1.m3.1.1.2.2.cmml">b</mi><mo lspace="0em" rspace="0em" id="alg1.l1.m3.1.1.2.1" xref="alg1.l1.m3.1.1.2.1.cmml">​</mo><mi id="alg1.l1.m3.1.1.2.3" xref="alg1.l1.m3.1.1.2.3.cmml">e</mi><mo lspace="0em" rspace="0em" id="alg1.l1.m3.1.1.2.1a" xref="alg1.l1.m3.1.1.2.1.cmml">​</mo><mi id="alg1.l1.m3.1.1.2.4" xref="alg1.l1.m3.1.1.2.4.cmml">s</mi><mo lspace="0em" rspace="0em" id="alg1.l1.m3.1.1.2.1b" xref="alg1.l1.m3.1.1.2.1.cmml">​</mo><mi id="alg1.l1.m3.1.1.2.5" xref="alg1.l1.m3.1.1.2.5.cmml">t</mi><mo lspace="0em" rspace="0em" id="alg1.l1.m3.1.1.2.1c" xref="alg1.l1.m3.1.1.2.1.cmml">​</mo><mi mathvariant="normal" id="alg1.l1.m3.1.1.2.6" xref="alg1.l1.m3.1.1.2.6.cmml">_</mi><mo lspace="0em" rspace="0em" id="alg1.l1.m3.1.1.2.1d" xref="alg1.l1.m3.1.1.2.1.cmml">​</mo><mi id="alg1.l1.m3.1.1.2.7" xref="alg1.l1.m3.1.1.2.7.cmml">l</mi></mrow><mo stretchy="false" id="alg1.l1.m3.1.1.1" xref="alg1.l1.m3.1.1.1.cmml">←</mo><mrow id="alg1.l1.m3.1.1.3" xref="alg1.l1.m3.1.1.3.cmml"><mi id="alg1.l1.m3.1.1.3.2" xref="alg1.l1.m3.1.1.3.2.cmml">m</mi><mo lspace="0em" rspace="0em" id="alg1.l1.m3.1.1.3.1" xref="alg1.l1.m3.1.1.3.1.cmml">​</mo><mi id="alg1.l1.m3.1.1.3.3" xref="alg1.l1.m3.1.1.3.3.cmml">a</mi><mo lspace="0em" rspace="0em" id="alg1.l1.m3.1.1.3.1a" xref="alg1.l1.m3.1.1.3.1.cmml">​</mo><mi id="alg1.l1.m3.1.1.3.4" xref="alg1.l1.m3.1.1.3.4.cmml">x</mi><mo lspace="0em" rspace="0em" id="alg1.l1.m3.1.1.3.1b" xref="alg1.l1.m3.1.1.3.1.cmml">​</mo><mi id="alg1.l1.m3.1.1.3.5" xref="alg1.l1.m3.1.1.3.5.cmml">i</mi><mo lspace="0em" rspace="0em" id="alg1.l1.m3.1.1.3.1c" xref="alg1.l1.m3.1.1.3.1.cmml">​</mo><mi id="alg1.l1.m3.1.1.3.6" xref="alg1.l1.m3.1.1.3.6.cmml">m</mi><mo lspace="0em" rspace="0em" id="alg1.l1.m3.1.1.3.1d" xref="alg1.l1.m3.1.1.3.1.cmml">​</mo><mi id="alg1.l1.m3.1.1.3.7" xref="alg1.l1.m3.1.1.3.7.cmml">u</mi><mo lspace="0em" rspace="0em" id="alg1.l1.m3.1.1.3.1e" xref="alg1.l1.m3.1.1.3.1.cmml">​</mo><mi id="alg1.l1.m3.1.1.3.8" xref="alg1.l1.m3.1.1.3.8.cmml">m</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg1.l1.m3.1b"><apply id="alg1.l1.m3.1.1.cmml" xref="alg1.l1.m3.1.1"><ci id="alg1.l1.m3.1.1.1.cmml" xref="alg1.l1.m3.1.1.1">←</ci><apply id="alg1.l1.m3.1.1.2.cmml" xref="alg1.l1.m3.1.1.2"><times id="alg1.l1.m3.1.1.2.1.cmml" xref="alg1.l1.m3.1.1.2.1"></times><ci id="alg1.l1.m3.1.1.2.2.cmml" xref="alg1.l1.m3.1.1.2.2">𝑏</ci><ci id="alg1.l1.m3.1.1.2.3.cmml" xref="alg1.l1.m3.1.1.2.3">𝑒</ci><ci id="alg1.l1.m3.1.1.2.4.cmml" xref="alg1.l1.m3.1.1.2.4">𝑠</ci><ci id="alg1.l1.m3.1.1.2.5.cmml" xref="alg1.l1.m3.1.1.2.5">𝑡</ci><ci id="alg1.l1.m3.1.1.2.6.cmml" xref="alg1.l1.m3.1.1.2.6">_</ci><ci id="alg1.l1.m3.1.1.2.7.cmml" xref="alg1.l1.m3.1.1.2.7">𝑙</ci></apply><apply id="alg1.l1.m3.1.1.3.cmml" xref="alg1.l1.m3.1.1.3"><times id="alg1.l1.m3.1.1.3.1.cmml" xref="alg1.l1.m3.1.1.3.1"></times><ci id="alg1.l1.m3.1.1.3.2.cmml" xref="alg1.l1.m3.1.1.3.2">𝑚</ci><ci id="alg1.l1.m3.1.1.3.3.cmml" xref="alg1.l1.m3.1.1.3.3">𝑎</ci><ci id="alg1.l1.m3.1.1.3.4.cmml" xref="alg1.l1.m3.1.1.3.4">𝑥</ci><ci id="alg1.l1.m3.1.1.3.5.cmml" xref="alg1.l1.m3.1.1.3.5">𝑖</ci><ci id="alg1.l1.m3.1.1.3.6.cmml" xref="alg1.l1.m3.1.1.3.6">𝑚</ci><ci id="alg1.l1.m3.1.1.3.7.cmml" xref="alg1.l1.m3.1.1.3.7">𝑢</ci><ci id="alg1.l1.m3.1.1.3.8.cmml" xref="alg1.l1.m3.1.1.3.8">𝑚</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l1.m3.1c">best\_l\leftarrow maximum</annotation></semantics></math>

</div>
<div id="alg1.l2" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">2:</span><span id="alg1.l2.1" class="ltx_text ltx_font_bold">for</span> <math id="alg1.l2.m1.1" class="ltx_Math" alttext="i\leftarrow 0" display="inline"><semantics id="alg1.l2.m1.1a"><mrow id="alg1.l2.m1.1.1" xref="alg1.l2.m1.1.1.cmml"><mi id="alg1.l2.m1.1.1.2" xref="alg1.l2.m1.1.1.2.cmml">i</mi><mo stretchy="false" id="alg1.l2.m1.1.1.1" xref="alg1.l2.m1.1.1.1.cmml">←</mo><mn id="alg1.l2.m1.1.1.3" xref="alg1.l2.m1.1.1.3.cmml">0</mn></mrow><annotation-xml encoding="MathML-Content" id="alg1.l2.m1.1b"><apply id="alg1.l2.m1.1.1.cmml" xref="alg1.l2.m1.1.1"><ci id="alg1.l2.m1.1.1.1.cmml" xref="alg1.l2.m1.1.1.1">←</ci><ci id="alg1.l2.m1.1.1.2.cmml" xref="alg1.l2.m1.1.1.2">𝑖</ci><cn type="integer" id="alg1.l2.m1.1.1.3.cmml" xref="alg1.l2.m1.1.1.3">0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l2.m1.1c">i\leftarrow 0</annotation></semantics></math> to <math id="alg1.l2.m2.1" class="ltx_Math" alttext="T" display="inline"><semantics id="alg1.l2.m2.1a"><mi id="alg1.l2.m2.1.1" xref="alg1.l2.m2.1.1.cmml">T</mi><annotation-xml encoding="MathML-Content" id="alg1.l2.m2.1b"><ci id="alg1.l2.m2.1.1.cmml" xref="alg1.l2.m2.1.1">𝑇</ci></annotation-xml><annotation encoding="application/x-tex" id="alg1.l2.m2.1c">T</annotation></semantics></math> <span id="alg1.l2.2" class="ltx_text ltx_font_bold">do</span>

</div>
<div id="alg1.l3" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">3:</span>     <math id="alg1.l3.m1.2" class="ltx_Math" alttext="d\leftarrow\textit{draw $bs$ samples from $\mathcal{D}$}" display="inline"><semantics id="alg1.l3.m1.2a"><mrow id="alg1.l3.m1.2.3" xref="alg1.l3.m1.2.3.cmml"><mi id="alg1.l3.m1.2.3.2" xref="alg1.l3.m1.2.3.2.cmml">d</mi><mo stretchy="false" id="alg1.l3.m1.2.3.1" xref="alg1.l3.m1.2.3.1.cmml">←</mo><mrow id="alg1.l3.m1.2.2.2" xref="alg1.l3.m1.2.2.2c.cmml"><mtext class="ltx_mathvariant_italic" id="alg1.l3.m1.2.2.2a" xref="alg1.l3.m1.2.2.2c.cmml">draw </mtext><mrow id="alg1.l3.m1.1.1.1.m1.1.1" xref="alg1.l3.m1.1.1.1.m1.1.1.cmml"><mi id="alg1.l3.m1.1.1.1.m1.1.1.2" xref="alg1.l3.m1.1.1.1.m1.1.1.2.cmml">b</mi><mo lspace="0em" rspace="0em" id="alg1.l3.m1.1.1.1.m1.1.1.1" xref="alg1.l3.m1.1.1.1.m1.1.1.1.cmml">​</mo><mi id="alg1.l3.m1.1.1.1.m1.1.1.3" xref="alg1.l3.m1.1.1.1.m1.1.1.3.cmml">s</mi></mrow><mtext class="ltx_mathvariant_italic" id="alg1.l3.m1.2.2.2b" xref="alg1.l3.m1.2.2.2c.cmml"> samples from </mtext><mi class="ltx_font_mathcaligraphic" id="alg1.l3.m1.2.2.2.m2.1.1" xref="alg1.l3.m1.2.2.2.m2.1.1.cmml">𝒟</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg1.l3.m1.2b"><apply id="alg1.l3.m1.2.3.cmml" xref="alg1.l3.m1.2.3"><ci id="alg1.l3.m1.2.3.1.cmml" xref="alg1.l3.m1.2.3.1">←</ci><ci id="alg1.l3.m1.2.3.2.cmml" xref="alg1.l3.m1.2.3.2">𝑑</ci><ci id="alg1.l3.m1.2.2.2c.cmml" xref="alg1.l3.m1.2.2.2"><mrow id="alg1.l3.m1.2.2.2.cmml" xref="alg1.l3.m1.2.2.2"><mtext class="ltx_mathvariant_italic" id="alg1.l3.m1.2.2.2a.cmml" xref="alg1.l3.m1.2.2.2">draw </mtext><mrow id="alg1.l3.m1.1.1.1.m1.1.1.cmml" xref="alg1.l3.m1.1.1.1.m1.1.1"><mi id="alg1.l3.m1.1.1.1.m1.1.1.2.cmml" xref="alg1.l3.m1.1.1.1.m1.1.1.2">b</mi><mo id="alg1.l3.m1.1.1.1.m1.1.1.1.cmml" xref="alg1.l3.m1.1.1.1.m1.1.1.1" lspace='0px' rspace='0px'></mo><mi id="alg1.l3.m1.1.1.1.m1.1.1.3.cmml" xref="alg1.l3.m1.1.1.1.m1.1.1.3">s</mi></mrow><mtext class="ltx_mathvariant_italic" id="alg1.l3.m1.2.2.2b.cmml" xref="alg1.l3.m1.2.2.2"> samples from </mtext><mi class="ltx_font_mathcaligraphic" id="alg1.l3.m1.2.2.2.m2.1.1.cmml" xref="alg1.l3.m1.2.2.2.m2.1.1">𝒟</mi></mrow></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l3.m1.2c">d\leftarrow\textit{draw $bs$ samples from $\mathcal{D}$}</annotation></semantics></math>

</div>
<div id="alg1.l4" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">4:</span>     <math id="alg1.l4.m1.1" class="ltx_Math" alttext="x\leftarrow M(d)_{m}" display="inline"><semantics id="alg1.l4.m1.1a"><mrow id="alg1.l4.m1.1.2" xref="alg1.l4.m1.1.2.cmml"><mi id="alg1.l4.m1.1.2.2" xref="alg1.l4.m1.1.2.2.cmml">x</mi><mo stretchy="false" id="alg1.l4.m1.1.2.1" xref="alg1.l4.m1.1.2.1.cmml">←</mo><mrow id="alg1.l4.m1.1.2.3" xref="alg1.l4.m1.1.2.3.cmml"><mi id="alg1.l4.m1.1.2.3.2" xref="alg1.l4.m1.1.2.3.2.cmml">M</mi><mo lspace="0em" rspace="0em" id="alg1.l4.m1.1.2.3.1" xref="alg1.l4.m1.1.2.3.1.cmml">​</mo><msub id="alg1.l4.m1.1.2.3.3" xref="alg1.l4.m1.1.2.3.3.cmml"><mrow id="alg1.l4.m1.1.2.3.3.2.2" xref="alg1.l4.m1.1.2.3.3.cmml"><mo stretchy="false" id="alg1.l4.m1.1.2.3.3.2.2.1" xref="alg1.l4.m1.1.2.3.3.cmml">(</mo><mi id="alg1.l4.m1.1.1" xref="alg1.l4.m1.1.1.cmml">d</mi><mo stretchy="false" id="alg1.l4.m1.1.2.3.3.2.2.2" xref="alg1.l4.m1.1.2.3.3.cmml">)</mo></mrow><mi id="alg1.l4.m1.1.2.3.3.3" xref="alg1.l4.m1.1.2.3.3.3.cmml">m</mi></msub></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg1.l4.m1.1b"><apply id="alg1.l4.m1.1.2.cmml" xref="alg1.l4.m1.1.2"><ci id="alg1.l4.m1.1.2.1.cmml" xref="alg1.l4.m1.1.2.1">←</ci><ci id="alg1.l4.m1.1.2.2.cmml" xref="alg1.l4.m1.1.2.2">𝑥</ci><apply id="alg1.l4.m1.1.2.3.cmml" xref="alg1.l4.m1.1.2.3"><times id="alg1.l4.m1.1.2.3.1.cmml" xref="alg1.l4.m1.1.2.3.1"></times><ci id="alg1.l4.m1.1.2.3.2.cmml" xref="alg1.l4.m1.1.2.3.2">𝑀</ci><apply id="alg1.l4.m1.1.2.3.3.cmml" xref="alg1.l4.m1.1.2.3.3"><csymbol cd="ambiguous" id="alg1.l4.m1.1.2.3.3.1.cmml" xref="alg1.l4.m1.1.2.3.3">subscript</csymbol><ci id="alg1.l4.m1.1.1.cmml" xref="alg1.l4.m1.1.1">𝑑</ci><ci id="alg1.l4.m1.1.2.3.3.3.cmml" xref="alg1.l4.m1.1.2.3.3.3">𝑚</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l4.m1.1c">x\leftarrow M(d)_{m}</annotation></semantics></math> <span id="alg1.l4.1" class="ltx_text" style="float:right;"><math id="alg1.l4.1.m1.1" class="ltx_Math" alttext="\triangleright" display="inline"><semantics id="alg1.l4.1.m1.1a"><mo id="alg1.l4.1.m1.1.1" xref="alg1.l4.1.m1.1.1.cmml">▷</mo><annotation-xml encoding="MathML-Content" id="alg1.l4.1.m1.1b"><ci id="alg1.l4.1.m1.1.1.cmml" xref="alg1.l4.1.m1.1.1">▷</ci></annotation-xml><annotation encoding="application/x-tex" id="alg1.l4.1.m1.1c">\triangleright</annotation></semantics></math> get the inputs of m
</span>
</div>
<div id="alg1.l5" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">5:</span>     <math id="alg1.l5.m1.1" class="ltx_Math" alttext="y_{f}\leftarrow m_{w}(x)" display="inline"><semantics id="alg1.l5.m1.1a"><mrow id="alg1.l5.m1.1.2" xref="alg1.l5.m1.1.2.cmml"><msub id="alg1.l5.m1.1.2.2" xref="alg1.l5.m1.1.2.2.cmml"><mi id="alg1.l5.m1.1.2.2.2" xref="alg1.l5.m1.1.2.2.2.cmml">y</mi><mi id="alg1.l5.m1.1.2.2.3" xref="alg1.l5.m1.1.2.2.3.cmml">f</mi></msub><mo stretchy="false" id="alg1.l5.m1.1.2.1" xref="alg1.l5.m1.1.2.1.cmml">←</mo><mrow id="alg1.l5.m1.1.2.3" xref="alg1.l5.m1.1.2.3.cmml"><msub id="alg1.l5.m1.1.2.3.2" xref="alg1.l5.m1.1.2.3.2.cmml"><mi id="alg1.l5.m1.1.2.3.2.2" xref="alg1.l5.m1.1.2.3.2.2.cmml">m</mi><mi id="alg1.l5.m1.1.2.3.2.3" xref="alg1.l5.m1.1.2.3.2.3.cmml">w</mi></msub><mo lspace="0em" rspace="0em" id="alg1.l5.m1.1.2.3.1" xref="alg1.l5.m1.1.2.3.1.cmml">​</mo><mrow id="alg1.l5.m1.1.2.3.3.2" xref="alg1.l5.m1.1.2.3.cmml"><mo stretchy="false" id="alg1.l5.m1.1.2.3.3.2.1" xref="alg1.l5.m1.1.2.3.cmml">(</mo><mi id="alg1.l5.m1.1.1" xref="alg1.l5.m1.1.1.cmml">x</mi><mo stretchy="false" id="alg1.l5.m1.1.2.3.3.2.2" xref="alg1.l5.m1.1.2.3.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg1.l5.m1.1b"><apply id="alg1.l5.m1.1.2.cmml" xref="alg1.l5.m1.1.2"><ci id="alg1.l5.m1.1.2.1.cmml" xref="alg1.l5.m1.1.2.1">←</ci><apply id="alg1.l5.m1.1.2.2.cmml" xref="alg1.l5.m1.1.2.2"><csymbol cd="ambiguous" id="alg1.l5.m1.1.2.2.1.cmml" xref="alg1.l5.m1.1.2.2">subscript</csymbol><ci id="alg1.l5.m1.1.2.2.2.cmml" xref="alg1.l5.m1.1.2.2.2">𝑦</ci><ci id="alg1.l5.m1.1.2.2.3.cmml" xref="alg1.l5.m1.1.2.2.3">𝑓</ci></apply><apply id="alg1.l5.m1.1.2.3.cmml" xref="alg1.l5.m1.1.2.3"><times id="alg1.l5.m1.1.2.3.1.cmml" xref="alg1.l5.m1.1.2.3.1"></times><apply id="alg1.l5.m1.1.2.3.2.cmml" xref="alg1.l5.m1.1.2.3.2"><csymbol cd="ambiguous" id="alg1.l5.m1.1.2.3.2.1.cmml" xref="alg1.l5.m1.1.2.3.2">subscript</csymbol><ci id="alg1.l5.m1.1.2.3.2.2.cmml" xref="alg1.l5.m1.1.2.3.2.2">𝑚</ci><ci id="alg1.l5.m1.1.2.3.2.3.cmml" xref="alg1.l5.m1.1.2.3.2.3">𝑤</ci></apply><ci id="alg1.l5.m1.1.1.cmml" xref="alg1.l5.m1.1.1">𝑥</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l5.m1.1c">y_{f}\leftarrow m_{w}(x)</annotation></semantics></math> <span id="alg1.l5.1" class="ltx_text" style="float:right;"><math id="alg1.l5.1.m1.1" class="ltx_Math" alttext="\triangleright" display="inline"><semantics id="alg1.l5.1.m1.1a"><mo id="alg1.l5.1.m1.1.1" xref="alg1.l5.1.m1.1.1.cmml">▷</mo><annotation-xml encoding="MathML-Content" id="alg1.l5.1.m1.1b"><ci id="alg1.l5.1.m1.1.1.cmml" xref="alg1.l5.1.m1.1.1">▷</ci></annotation-xml><annotation encoding="application/x-tex" id="alg1.l5.1.m1.1c">\triangleright</annotation></semantics></math> get the output of original module
</span>
</div>
<div id="alg1.l6" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">6:</span>     <math id="alg1.l6.m1.2" class="ltx_Math" alttext="\widetilde{w}\leftarrow qdq(w,V)" display="inline"><semantics id="alg1.l6.m1.2a"><mrow id="alg1.l6.m1.2.3" xref="alg1.l6.m1.2.3.cmml"><mover accent="true" id="alg1.l6.m1.2.3.2" xref="alg1.l6.m1.2.3.2.cmml"><mi id="alg1.l6.m1.2.3.2.2" xref="alg1.l6.m1.2.3.2.2.cmml">w</mi><mo id="alg1.l6.m1.2.3.2.1" xref="alg1.l6.m1.2.3.2.1.cmml">~</mo></mover><mo stretchy="false" id="alg1.l6.m1.2.3.1" xref="alg1.l6.m1.2.3.1.cmml">←</mo><mrow id="alg1.l6.m1.2.3.3" xref="alg1.l6.m1.2.3.3.cmml"><mi id="alg1.l6.m1.2.3.3.2" xref="alg1.l6.m1.2.3.3.2.cmml">q</mi><mo lspace="0em" rspace="0em" id="alg1.l6.m1.2.3.3.1" xref="alg1.l6.m1.2.3.3.1.cmml">​</mo><mi id="alg1.l6.m1.2.3.3.3" xref="alg1.l6.m1.2.3.3.3.cmml">d</mi><mo lspace="0em" rspace="0em" id="alg1.l6.m1.2.3.3.1a" xref="alg1.l6.m1.2.3.3.1.cmml">​</mo><mi id="alg1.l6.m1.2.3.3.4" xref="alg1.l6.m1.2.3.3.4.cmml">q</mi><mo lspace="0em" rspace="0em" id="alg1.l6.m1.2.3.3.1b" xref="alg1.l6.m1.2.3.3.1.cmml">​</mo><mrow id="alg1.l6.m1.2.3.3.5.2" xref="alg1.l6.m1.2.3.3.5.1.cmml"><mo stretchy="false" id="alg1.l6.m1.2.3.3.5.2.1" xref="alg1.l6.m1.2.3.3.5.1.cmml">(</mo><mi id="alg1.l6.m1.1.1" xref="alg1.l6.m1.1.1.cmml">w</mi><mo id="alg1.l6.m1.2.3.3.5.2.2" xref="alg1.l6.m1.2.3.3.5.1.cmml">,</mo><mi id="alg1.l6.m1.2.2" xref="alg1.l6.m1.2.2.cmml">V</mi><mo stretchy="false" id="alg1.l6.m1.2.3.3.5.2.3" xref="alg1.l6.m1.2.3.3.5.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg1.l6.m1.2b"><apply id="alg1.l6.m1.2.3.cmml" xref="alg1.l6.m1.2.3"><ci id="alg1.l6.m1.2.3.1.cmml" xref="alg1.l6.m1.2.3.1">←</ci><apply id="alg1.l6.m1.2.3.2.cmml" xref="alg1.l6.m1.2.3.2"><ci id="alg1.l6.m1.2.3.2.1.cmml" xref="alg1.l6.m1.2.3.2.1">~</ci><ci id="alg1.l6.m1.2.3.2.2.cmml" xref="alg1.l6.m1.2.3.2.2">𝑤</ci></apply><apply id="alg1.l6.m1.2.3.3.cmml" xref="alg1.l6.m1.2.3.3"><times id="alg1.l6.m1.2.3.3.1.cmml" xref="alg1.l6.m1.2.3.3.1"></times><ci id="alg1.l6.m1.2.3.3.2.cmml" xref="alg1.l6.m1.2.3.3.2">𝑞</ci><ci id="alg1.l6.m1.2.3.3.3.cmml" xref="alg1.l6.m1.2.3.3.3">𝑑</ci><ci id="alg1.l6.m1.2.3.3.4.cmml" xref="alg1.l6.m1.2.3.3.4">𝑞</ci><interval closure="open" id="alg1.l6.m1.2.3.3.5.1.cmml" xref="alg1.l6.m1.2.3.3.5.2"><ci id="alg1.l6.m1.1.1.cmml" xref="alg1.l6.m1.1.1">𝑤</ci><ci id="alg1.l6.m1.2.2.cmml" xref="alg1.l6.m1.2.2">𝑉</ci></interval></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l6.m1.2c">\widetilde{w}\leftarrow qdq(w,V)</annotation></semantics></math> <span id="alg1.l6.1" class="ltx_text" style="float:right;"><math id="alg1.l6.1.m1.1" class="ltx_Math" alttext="\triangleright" display="inline"><semantics id="alg1.l6.1.m1.1a"><mo id="alg1.l6.1.m1.1.1" xref="alg1.l6.1.m1.1.1.cmml">▷</mo><annotation-xml encoding="MathML-Content" id="alg1.l6.1.m1.1b"><ci id="alg1.l6.1.m1.1.1.cmml" xref="alg1.l6.1.m1.1.1">▷</ci></annotation-xml><annotation encoding="application/x-tex" id="alg1.l6.1.m1.1c">\triangleright</annotation></semantics></math> quantize and dequantize w via Eq.<a href="#S3.E2" title="In 3 Methodology ‣ Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>
</span>
</div>
<div id="alg1.l7" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">7:</span>     <math id="alg1.l7.m1.1" class="ltx_Math" alttext="y_{q}\leftarrow m_{\widetilde{w}}(x)" display="inline"><semantics id="alg1.l7.m1.1a"><mrow id="alg1.l7.m1.1.2" xref="alg1.l7.m1.1.2.cmml"><msub id="alg1.l7.m1.1.2.2" xref="alg1.l7.m1.1.2.2.cmml"><mi id="alg1.l7.m1.1.2.2.2" xref="alg1.l7.m1.1.2.2.2.cmml">y</mi><mi id="alg1.l7.m1.1.2.2.3" xref="alg1.l7.m1.1.2.2.3.cmml">q</mi></msub><mo stretchy="false" id="alg1.l7.m1.1.2.1" xref="alg1.l7.m1.1.2.1.cmml">←</mo><mrow id="alg1.l7.m1.1.2.3" xref="alg1.l7.m1.1.2.3.cmml"><msub id="alg1.l7.m1.1.2.3.2" xref="alg1.l7.m1.1.2.3.2.cmml"><mi id="alg1.l7.m1.1.2.3.2.2" xref="alg1.l7.m1.1.2.3.2.2.cmml">m</mi><mover accent="true" id="alg1.l7.m1.1.2.3.2.3" xref="alg1.l7.m1.1.2.3.2.3.cmml"><mi id="alg1.l7.m1.1.2.3.2.3.2" xref="alg1.l7.m1.1.2.3.2.3.2.cmml">w</mi><mo id="alg1.l7.m1.1.2.3.2.3.1" xref="alg1.l7.m1.1.2.3.2.3.1.cmml">~</mo></mover></msub><mo lspace="0em" rspace="0em" id="alg1.l7.m1.1.2.3.1" xref="alg1.l7.m1.1.2.3.1.cmml">​</mo><mrow id="alg1.l7.m1.1.2.3.3.2" xref="alg1.l7.m1.1.2.3.cmml"><mo stretchy="false" id="alg1.l7.m1.1.2.3.3.2.1" xref="alg1.l7.m1.1.2.3.cmml">(</mo><mi id="alg1.l7.m1.1.1" xref="alg1.l7.m1.1.1.cmml">x</mi><mo stretchy="false" id="alg1.l7.m1.1.2.3.3.2.2" xref="alg1.l7.m1.1.2.3.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg1.l7.m1.1b"><apply id="alg1.l7.m1.1.2.cmml" xref="alg1.l7.m1.1.2"><ci id="alg1.l7.m1.1.2.1.cmml" xref="alg1.l7.m1.1.2.1">←</ci><apply id="alg1.l7.m1.1.2.2.cmml" xref="alg1.l7.m1.1.2.2"><csymbol cd="ambiguous" id="alg1.l7.m1.1.2.2.1.cmml" xref="alg1.l7.m1.1.2.2">subscript</csymbol><ci id="alg1.l7.m1.1.2.2.2.cmml" xref="alg1.l7.m1.1.2.2.2">𝑦</ci><ci id="alg1.l7.m1.1.2.2.3.cmml" xref="alg1.l7.m1.1.2.2.3">𝑞</ci></apply><apply id="alg1.l7.m1.1.2.3.cmml" xref="alg1.l7.m1.1.2.3"><times id="alg1.l7.m1.1.2.3.1.cmml" xref="alg1.l7.m1.1.2.3.1"></times><apply id="alg1.l7.m1.1.2.3.2.cmml" xref="alg1.l7.m1.1.2.3.2"><csymbol cd="ambiguous" id="alg1.l7.m1.1.2.3.2.1.cmml" xref="alg1.l7.m1.1.2.3.2">subscript</csymbol><ci id="alg1.l7.m1.1.2.3.2.2.cmml" xref="alg1.l7.m1.1.2.3.2.2">𝑚</ci><apply id="alg1.l7.m1.1.2.3.2.3.cmml" xref="alg1.l7.m1.1.2.3.2.3"><ci id="alg1.l7.m1.1.2.3.2.3.1.cmml" xref="alg1.l7.m1.1.2.3.2.3.1">~</ci><ci id="alg1.l7.m1.1.2.3.2.3.2.cmml" xref="alg1.l7.m1.1.2.3.2.3.2">𝑤</ci></apply></apply><ci id="alg1.l7.m1.1.1.cmml" xref="alg1.l7.m1.1.1">𝑥</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l7.m1.1c">y_{q}\leftarrow m_{\widetilde{w}}(x)</annotation></semantics></math> <span id="alg1.l7.1" class="ltx_text" style="float:right;"><math id="alg1.l7.1.m1.1" class="ltx_Math" alttext="\triangleright" display="inline"><semantics id="alg1.l7.1.m1.1a"><mo id="alg1.l7.1.m1.1.1" xref="alg1.l7.1.m1.1.1.cmml">▷</mo><annotation-xml encoding="MathML-Content" id="alg1.l7.1.m1.1b"><ci id="alg1.l7.1.m1.1.1.cmml" xref="alg1.l7.1.m1.1.1">▷</ci></annotation-xml><annotation encoding="application/x-tex" id="alg1.l7.1.m1.1c">\triangleright</annotation></semantics></math> get the output of quantized module
</span>
</div>
<div id="alg1.l8" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">8:</span>     <math id="alg1.l8.m1.2" class="ltx_Math" alttext="loss\leftarrow mse(y_{q},y_{f})" display="inline"><semantics id="alg1.l8.m1.2a"><mrow id="alg1.l8.m1.2.2" xref="alg1.l8.m1.2.2.cmml"><mrow id="alg1.l8.m1.2.2.4" xref="alg1.l8.m1.2.2.4.cmml"><mi id="alg1.l8.m1.2.2.4.2" xref="alg1.l8.m1.2.2.4.2.cmml">l</mi><mo lspace="0em" rspace="0em" id="alg1.l8.m1.2.2.4.1" xref="alg1.l8.m1.2.2.4.1.cmml">​</mo><mi id="alg1.l8.m1.2.2.4.3" xref="alg1.l8.m1.2.2.4.3.cmml">o</mi><mo lspace="0em" rspace="0em" id="alg1.l8.m1.2.2.4.1a" xref="alg1.l8.m1.2.2.4.1.cmml">​</mo><mi id="alg1.l8.m1.2.2.4.4" xref="alg1.l8.m1.2.2.4.4.cmml">s</mi><mo lspace="0em" rspace="0em" id="alg1.l8.m1.2.2.4.1b" xref="alg1.l8.m1.2.2.4.1.cmml">​</mo><mi id="alg1.l8.m1.2.2.4.5" xref="alg1.l8.m1.2.2.4.5.cmml">s</mi></mrow><mo stretchy="false" id="alg1.l8.m1.2.2.3" xref="alg1.l8.m1.2.2.3.cmml">←</mo><mrow id="alg1.l8.m1.2.2.2" xref="alg1.l8.m1.2.2.2.cmml"><mi id="alg1.l8.m1.2.2.2.4" xref="alg1.l8.m1.2.2.2.4.cmml">m</mi><mo lspace="0em" rspace="0em" id="alg1.l8.m1.2.2.2.3" xref="alg1.l8.m1.2.2.2.3.cmml">​</mo><mi id="alg1.l8.m1.2.2.2.5" xref="alg1.l8.m1.2.2.2.5.cmml">s</mi><mo lspace="0em" rspace="0em" id="alg1.l8.m1.2.2.2.3a" xref="alg1.l8.m1.2.2.2.3.cmml">​</mo><mi id="alg1.l8.m1.2.2.2.6" xref="alg1.l8.m1.2.2.2.6.cmml">e</mi><mo lspace="0em" rspace="0em" id="alg1.l8.m1.2.2.2.3b" xref="alg1.l8.m1.2.2.2.3.cmml">​</mo><mrow id="alg1.l8.m1.2.2.2.2.2" xref="alg1.l8.m1.2.2.2.2.3.cmml"><mo stretchy="false" id="alg1.l8.m1.2.2.2.2.2.3" xref="alg1.l8.m1.2.2.2.2.3.cmml">(</mo><msub id="alg1.l8.m1.1.1.1.1.1.1" xref="alg1.l8.m1.1.1.1.1.1.1.cmml"><mi id="alg1.l8.m1.1.1.1.1.1.1.2" xref="alg1.l8.m1.1.1.1.1.1.1.2.cmml">y</mi><mi id="alg1.l8.m1.1.1.1.1.1.1.3" xref="alg1.l8.m1.1.1.1.1.1.1.3.cmml">q</mi></msub><mo id="alg1.l8.m1.2.2.2.2.2.4" xref="alg1.l8.m1.2.2.2.2.3.cmml">,</mo><msub id="alg1.l8.m1.2.2.2.2.2.2" xref="alg1.l8.m1.2.2.2.2.2.2.cmml"><mi id="alg1.l8.m1.2.2.2.2.2.2.2" xref="alg1.l8.m1.2.2.2.2.2.2.2.cmml">y</mi><mi id="alg1.l8.m1.2.2.2.2.2.2.3" xref="alg1.l8.m1.2.2.2.2.2.2.3.cmml">f</mi></msub><mo stretchy="false" id="alg1.l8.m1.2.2.2.2.2.5" xref="alg1.l8.m1.2.2.2.2.3.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg1.l8.m1.2b"><apply id="alg1.l8.m1.2.2.cmml" xref="alg1.l8.m1.2.2"><ci id="alg1.l8.m1.2.2.3.cmml" xref="alg1.l8.m1.2.2.3">←</ci><apply id="alg1.l8.m1.2.2.4.cmml" xref="alg1.l8.m1.2.2.4"><times id="alg1.l8.m1.2.2.4.1.cmml" xref="alg1.l8.m1.2.2.4.1"></times><ci id="alg1.l8.m1.2.2.4.2.cmml" xref="alg1.l8.m1.2.2.4.2">𝑙</ci><ci id="alg1.l8.m1.2.2.4.3.cmml" xref="alg1.l8.m1.2.2.4.3">𝑜</ci><ci id="alg1.l8.m1.2.2.4.4.cmml" xref="alg1.l8.m1.2.2.4.4">𝑠</ci><ci id="alg1.l8.m1.2.2.4.5.cmml" xref="alg1.l8.m1.2.2.4.5">𝑠</ci></apply><apply id="alg1.l8.m1.2.2.2.cmml" xref="alg1.l8.m1.2.2.2"><times id="alg1.l8.m1.2.2.2.3.cmml" xref="alg1.l8.m1.2.2.2.3"></times><ci id="alg1.l8.m1.2.2.2.4.cmml" xref="alg1.l8.m1.2.2.2.4">𝑚</ci><ci id="alg1.l8.m1.2.2.2.5.cmml" xref="alg1.l8.m1.2.2.2.5">𝑠</ci><ci id="alg1.l8.m1.2.2.2.6.cmml" xref="alg1.l8.m1.2.2.2.6">𝑒</ci><interval closure="open" id="alg1.l8.m1.2.2.2.2.3.cmml" xref="alg1.l8.m1.2.2.2.2.2"><apply id="alg1.l8.m1.1.1.1.1.1.1.cmml" xref="alg1.l8.m1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="alg1.l8.m1.1.1.1.1.1.1.1.cmml" xref="alg1.l8.m1.1.1.1.1.1.1">subscript</csymbol><ci id="alg1.l8.m1.1.1.1.1.1.1.2.cmml" xref="alg1.l8.m1.1.1.1.1.1.1.2">𝑦</ci><ci id="alg1.l8.m1.1.1.1.1.1.1.3.cmml" xref="alg1.l8.m1.1.1.1.1.1.1.3">𝑞</ci></apply><apply id="alg1.l8.m1.2.2.2.2.2.2.cmml" xref="alg1.l8.m1.2.2.2.2.2.2"><csymbol cd="ambiguous" id="alg1.l8.m1.2.2.2.2.2.2.1.cmml" xref="alg1.l8.m1.2.2.2.2.2.2">subscript</csymbol><ci id="alg1.l8.m1.2.2.2.2.2.2.2.cmml" xref="alg1.l8.m1.2.2.2.2.2.2.2">𝑦</ci><ci id="alg1.l8.m1.2.2.2.2.2.2.3.cmml" xref="alg1.l8.m1.2.2.2.2.2.2.3">𝑓</ci></apply></interval></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l8.m1.2c">loss\leftarrow mse(y_{q},y_{f})</annotation></semantics></math> <span id="alg1.l8.1" class="ltx_text" style="float:right;"><math id="alg1.l8.1.m1.1" class="ltx_Math" alttext="\triangleright" display="inline"><semantics id="alg1.l8.1.m1.1a"><mo id="alg1.l8.1.m1.1.1" xref="alg1.l8.1.m1.1.1.cmml">▷</mo><annotation-xml encoding="MathML-Content" id="alg1.l8.1.m1.1b"><ci id="alg1.l8.1.m1.1.1.cmml" xref="alg1.l8.1.m1.1.1">▷</ci></annotation-xml><annotation encoding="application/x-tex" id="alg1.l8.1.m1.1c">\triangleright</annotation></semantics></math> get the loss via Eq.<a href="#S3.E3" title="In 3 Methodology ‣ Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>
</span>
</div>
<div id="alg1.l9" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">9:</span>     <span id="alg1.l9.1" class="ltx_text ltx_font_bold">if</span> <math id="alg1.l9.m1.1" class="ltx_Math" alttext="loss&lt;best\_l" display="inline"><semantics id="alg1.l9.m1.1a"><mrow id="alg1.l9.m1.1.1" xref="alg1.l9.m1.1.1.cmml"><mrow id="alg1.l9.m1.1.1.2" xref="alg1.l9.m1.1.1.2.cmml"><mi id="alg1.l9.m1.1.1.2.2" xref="alg1.l9.m1.1.1.2.2.cmml">l</mi><mo lspace="0em" rspace="0em" id="alg1.l9.m1.1.1.2.1" xref="alg1.l9.m1.1.1.2.1.cmml">​</mo><mi id="alg1.l9.m1.1.1.2.3" xref="alg1.l9.m1.1.1.2.3.cmml">o</mi><mo lspace="0em" rspace="0em" id="alg1.l9.m1.1.1.2.1a" xref="alg1.l9.m1.1.1.2.1.cmml">​</mo><mi id="alg1.l9.m1.1.1.2.4" xref="alg1.l9.m1.1.1.2.4.cmml">s</mi><mo lspace="0em" rspace="0em" id="alg1.l9.m1.1.1.2.1b" xref="alg1.l9.m1.1.1.2.1.cmml">​</mo><mi id="alg1.l9.m1.1.1.2.5" xref="alg1.l9.m1.1.1.2.5.cmml">s</mi></mrow><mo id="alg1.l9.m1.1.1.1" xref="alg1.l9.m1.1.1.1.cmml">&lt;</mo><mrow id="alg1.l9.m1.1.1.3" xref="alg1.l9.m1.1.1.3.cmml"><mi id="alg1.l9.m1.1.1.3.2" xref="alg1.l9.m1.1.1.3.2.cmml">b</mi><mo lspace="0em" rspace="0em" id="alg1.l9.m1.1.1.3.1" xref="alg1.l9.m1.1.1.3.1.cmml">​</mo><mi id="alg1.l9.m1.1.1.3.3" xref="alg1.l9.m1.1.1.3.3.cmml">e</mi><mo lspace="0em" rspace="0em" id="alg1.l9.m1.1.1.3.1a" xref="alg1.l9.m1.1.1.3.1.cmml">​</mo><mi id="alg1.l9.m1.1.1.3.4" xref="alg1.l9.m1.1.1.3.4.cmml">s</mi><mo lspace="0em" rspace="0em" id="alg1.l9.m1.1.1.3.1b" xref="alg1.l9.m1.1.1.3.1.cmml">​</mo><mi id="alg1.l9.m1.1.1.3.5" xref="alg1.l9.m1.1.1.3.5.cmml">t</mi><mo lspace="0em" rspace="0em" id="alg1.l9.m1.1.1.3.1c" xref="alg1.l9.m1.1.1.3.1.cmml">​</mo><mi mathvariant="normal" id="alg1.l9.m1.1.1.3.6" xref="alg1.l9.m1.1.1.3.6.cmml">_</mi><mo lspace="0em" rspace="0em" id="alg1.l9.m1.1.1.3.1d" xref="alg1.l9.m1.1.1.3.1.cmml">​</mo><mi id="alg1.l9.m1.1.1.3.7" xref="alg1.l9.m1.1.1.3.7.cmml">l</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg1.l9.m1.1b"><apply id="alg1.l9.m1.1.1.cmml" xref="alg1.l9.m1.1.1"><lt id="alg1.l9.m1.1.1.1.cmml" xref="alg1.l9.m1.1.1.1"></lt><apply id="alg1.l9.m1.1.1.2.cmml" xref="alg1.l9.m1.1.1.2"><times id="alg1.l9.m1.1.1.2.1.cmml" xref="alg1.l9.m1.1.1.2.1"></times><ci id="alg1.l9.m1.1.1.2.2.cmml" xref="alg1.l9.m1.1.1.2.2">𝑙</ci><ci id="alg1.l9.m1.1.1.2.3.cmml" xref="alg1.l9.m1.1.1.2.3">𝑜</ci><ci id="alg1.l9.m1.1.1.2.4.cmml" xref="alg1.l9.m1.1.1.2.4">𝑠</ci><ci id="alg1.l9.m1.1.1.2.5.cmml" xref="alg1.l9.m1.1.1.2.5">𝑠</ci></apply><apply id="alg1.l9.m1.1.1.3.cmml" xref="alg1.l9.m1.1.1.3"><times id="alg1.l9.m1.1.1.3.1.cmml" xref="alg1.l9.m1.1.1.3.1"></times><ci id="alg1.l9.m1.1.1.3.2.cmml" xref="alg1.l9.m1.1.1.3.2">𝑏</ci><ci id="alg1.l9.m1.1.1.3.3.cmml" xref="alg1.l9.m1.1.1.3.3">𝑒</ci><ci id="alg1.l9.m1.1.1.3.4.cmml" xref="alg1.l9.m1.1.1.3.4">𝑠</ci><ci id="alg1.l9.m1.1.1.3.5.cmml" xref="alg1.l9.m1.1.1.3.5">𝑡</ci><ci id="alg1.l9.m1.1.1.3.6.cmml" xref="alg1.l9.m1.1.1.3.6">_</ci><ci id="alg1.l9.m1.1.1.3.7.cmml" xref="alg1.l9.m1.1.1.3.7">𝑙</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l9.m1.1c">loss&lt;best\_l</annotation></semantics></math> <span id="alg1.l9.2" class="ltx_text ltx_font_bold">then</span>

</div>
<div id="alg1.l10" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">10:</span>         <math id="alg1.l10.m1.1" class="ltx_Math" alttext="best\_V\leftarrow V" display="inline"><semantics id="alg1.l10.m1.1a"><mrow id="alg1.l10.m1.1.1" xref="alg1.l10.m1.1.1.cmml"><mrow id="alg1.l10.m1.1.1.2" xref="alg1.l10.m1.1.1.2.cmml"><mi id="alg1.l10.m1.1.1.2.2" xref="alg1.l10.m1.1.1.2.2.cmml">b</mi><mo lspace="0em" rspace="0em" id="alg1.l10.m1.1.1.2.1" xref="alg1.l10.m1.1.1.2.1.cmml">​</mo><mi id="alg1.l10.m1.1.1.2.3" xref="alg1.l10.m1.1.1.2.3.cmml">e</mi><mo lspace="0em" rspace="0em" id="alg1.l10.m1.1.1.2.1a" xref="alg1.l10.m1.1.1.2.1.cmml">​</mo><mi id="alg1.l10.m1.1.1.2.4" xref="alg1.l10.m1.1.1.2.4.cmml">s</mi><mo lspace="0em" rspace="0em" id="alg1.l10.m1.1.1.2.1b" xref="alg1.l10.m1.1.1.2.1.cmml">​</mo><mi id="alg1.l10.m1.1.1.2.5" xref="alg1.l10.m1.1.1.2.5.cmml">t</mi><mo lspace="0em" rspace="0em" id="alg1.l10.m1.1.1.2.1c" xref="alg1.l10.m1.1.1.2.1.cmml">​</mo><mi mathvariant="normal" id="alg1.l10.m1.1.1.2.6" xref="alg1.l10.m1.1.1.2.6.cmml">_</mi><mo lspace="0em" rspace="0em" id="alg1.l10.m1.1.1.2.1d" xref="alg1.l10.m1.1.1.2.1.cmml">​</mo><mi id="alg1.l10.m1.1.1.2.7" xref="alg1.l10.m1.1.1.2.7.cmml">V</mi></mrow><mo stretchy="false" id="alg1.l10.m1.1.1.1" xref="alg1.l10.m1.1.1.1.cmml">←</mo><mi id="alg1.l10.m1.1.1.3" xref="alg1.l10.m1.1.1.3.cmml">V</mi></mrow><annotation-xml encoding="MathML-Content" id="alg1.l10.m1.1b"><apply id="alg1.l10.m1.1.1.cmml" xref="alg1.l10.m1.1.1"><ci id="alg1.l10.m1.1.1.1.cmml" xref="alg1.l10.m1.1.1.1">←</ci><apply id="alg1.l10.m1.1.1.2.cmml" xref="alg1.l10.m1.1.1.2"><times id="alg1.l10.m1.1.1.2.1.cmml" xref="alg1.l10.m1.1.1.2.1"></times><ci id="alg1.l10.m1.1.1.2.2.cmml" xref="alg1.l10.m1.1.1.2.2">𝑏</ci><ci id="alg1.l10.m1.1.1.2.3.cmml" xref="alg1.l10.m1.1.1.2.3">𝑒</ci><ci id="alg1.l10.m1.1.1.2.4.cmml" xref="alg1.l10.m1.1.1.2.4">𝑠</ci><ci id="alg1.l10.m1.1.1.2.5.cmml" xref="alg1.l10.m1.1.1.2.5">𝑡</ci><ci id="alg1.l10.m1.1.1.2.6.cmml" xref="alg1.l10.m1.1.1.2.6">_</ci><ci id="alg1.l10.m1.1.1.2.7.cmml" xref="alg1.l10.m1.1.1.2.7">𝑉</ci></apply><ci id="alg1.l10.m1.1.1.3.cmml" xref="alg1.l10.m1.1.1.3">𝑉</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l10.m1.1c">best\_V\leftarrow V</annotation></semantics></math>

</div>
<div id="alg1.l11" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">11:</span>         <math id="alg1.l11.m1.1" class="ltx_Math" alttext="best\_l\leftarrow loss" display="inline"><semantics id="alg1.l11.m1.1a"><mrow id="alg1.l11.m1.1.1" xref="alg1.l11.m1.1.1.cmml"><mrow id="alg1.l11.m1.1.1.2" xref="alg1.l11.m1.1.1.2.cmml"><mi id="alg1.l11.m1.1.1.2.2" xref="alg1.l11.m1.1.1.2.2.cmml">b</mi><mo lspace="0em" rspace="0em" id="alg1.l11.m1.1.1.2.1" xref="alg1.l11.m1.1.1.2.1.cmml">​</mo><mi id="alg1.l11.m1.1.1.2.3" xref="alg1.l11.m1.1.1.2.3.cmml">e</mi><mo lspace="0em" rspace="0em" id="alg1.l11.m1.1.1.2.1a" xref="alg1.l11.m1.1.1.2.1.cmml">​</mo><mi id="alg1.l11.m1.1.1.2.4" xref="alg1.l11.m1.1.1.2.4.cmml">s</mi><mo lspace="0em" rspace="0em" id="alg1.l11.m1.1.1.2.1b" xref="alg1.l11.m1.1.1.2.1.cmml">​</mo><mi id="alg1.l11.m1.1.1.2.5" xref="alg1.l11.m1.1.1.2.5.cmml">t</mi><mo lspace="0em" rspace="0em" id="alg1.l11.m1.1.1.2.1c" xref="alg1.l11.m1.1.1.2.1.cmml">​</mo><mi mathvariant="normal" id="alg1.l11.m1.1.1.2.6" xref="alg1.l11.m1.1.1.2.6.cmml">_</mi><mo lspace="0em" rspace="0em" id="alg1.l11.m1.1.1.2.1d" xref="alg1.l11.m1.1.1.2.1.cmml">​</mo><mi id="alg1.l11.m1.1.1.2.7" xref="alg1.l11.m1.1.1.2.7.cmml">l</mi></mrow><mo stretchy="false" id="alg1.l11.m1.1.1.1" xref="alg1.l11.m1.1.1.1.cmml">←</mo><mrow id="alg1.l11.m1.1.1.3" xref="alg1.l11.m1.1.1.3.cmml"><mi id="alg1.l11.m1.1.1.3.2" xref="alg1.l11.m1.1.1.3.2.cmml">l</mi><mo lspace="0em" rspace="0em" id="alg1.l11.m1.1.1.3.1" xref="alg1.l11.m1.1.1.3.1.cmml">​</mo><mi id="alg1.l11.m1.1.1.3.3" xref="alg1.l11.m1.1.1.3.3.cmml">o</mi><mo lspace="0em" rspace="0em" id="alg1.l11.m1.1.1.3.1a" xref="alg1.l11.m1.1.1.3.1.cmml">​</mo><mi id="alg1.l11.m1.1.1.3.4" xref="alg1.l11.m1.1.1.3.4.cmml">s</mi><mo lspace="0em" rspace="0em" id="alg1.l11.m1.1.1.3.1b" xref="alg1.l11.m1.1.1.3.1.cmml">​</mo><mi id="alg1.l11.m1.1.1.3.5" xref="alg1.l11.m1.1.1.3.5.cmml">s</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg1.l11.m1.1b"><apply id="alg1.l11.m1.1.1.cmml" xref="alg1.l11.m1.1.1"><ci id="alg1.l11.m1.1.1.1.cmml" xref="alg1.l11.m1.1.1.1">←</ci><apply id="alg1.l11.m1.1.1.2.cmml" xref="alg1.l11.m1.1.1.2"><times id="alg1.l11.m1.1.1.2.1.cmml" xref="alg1.l11.m1.1.1.2.1"></times><ci id="alg1.l11.m1.1.1.2.2.cmml" xref="alg1.l11.m1.1.1.2.2">𝑏</ci><ci id="alg1.l11.m1.1.1.2.3.cmml" xref="alg1.l11.m1.1.1.2.3">𝑒</ci><ci id="alg1.l11.m1.1.1.2.4.cmml" xref="alg1.l11.m1.1.1.2.4">𝑠</ci><ci id="alg1.l11.m1.1.1.2.5.cmml" xref="alg1.l11.m1.1.1.2.5">𝑡</ci><ci id="alg1.l11.m1.1.1.2.6.cmml" xref="alg1.l11.m1.1.1.2.6">_</ci><ci id="alg1.l11.m1.1.1.2.7.cmml" xref="alg1.l11.m1.1.1.2.7">𝑙</ci></apply><apply id="alg1.l11.m1.1.1.3.cmml" xref="alg1.l11.m1.1.1.3"><times id="alg1.l11.m1.1.1.3.1.cmml" xref="alg1.l11.m1.1.1.3.1"></times><ci id="alg1.l11.m1.1.1.3.2.cmml" xref="alg1.l11.m1.1.1.3.2">𝑙</ci><ci id="alg1.l11.m1.1.1.3.3.cmml" xref="alg1.l11.m1.1.1.3.3">𝑜</ci><ci id="alg1.l11.m1.1.1.3.4.cmml" xref="alg1.l11.m1.1.1.3.4">𝑠</ci><ci id="alg1.l11.m1.1.1.3.5.cmml" xref="alg1.l11.m1.1.1.3.5">𝑠</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l11.m1.1c">best\_l\leftarrow loss</annotation></semantics></math>

</div>
<div id="alg1.l12" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">12:</span>     <span id="alg1.l12.1" class="ltx_text ltx_font_bold">end</span> <span id="alg1.l12.2" class="ltx_text ltx_font_bold">if</span>
</div>
<div id="alg1.l13" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">13:</span>     <math id="alg1.l13.m1.2" class="ltx_Math" alttext="loss.backward()" display="inline"><semantics id="alg1.l13.m1.2a"><mrow id="alg1.l13.m1.2.2.2" xref="alg1.l13.m1.2.2.3.cmml"><mrow id="alg1.l13.m1.1.1.1.1" xref="alg1.l13.m1.1.1.1.1.cmml"><mi id="alg1.l13.m1.1.1.1.1.2" xref="alg1.l13.m1.1.1.1.1.2.cmml">l</mi><mo lspace="0em" rspace="0em" id="alg1.l13.m1.1.1.1.1.1" xref="alg1.l13.m1.1.1.1.1.1.cmml">​</mo><mi id="alg1.l13.m1.1.1.1.1.3" xref="alg1.l13.m1.1.1.1.1.3.cmml">o</mi><mo lspace="0em" rspace="0em" id="alg1.l13.m1.1.1.1.1.1a" xref="alg1.l13.m1.1.1.1.1.1.cmml">​</mo><mi id="alg1.l13.m1.1.1.1.1.4" xref="alg1.l13.m1.1.1.1.1.4.cmml">s</mi><mo lspace="0em" rspace="0em" id="alg1.l13.m1.1.1.1.1.1b" xref="alg1.l13.m1.1.1.1.1.1.cmml">​</mo><mi id="alg1.l13.m1.1.1.1.1.5" xref="alg1.l13.m1.1.1.1.1.5.cmml">s</mi></mrow><mo lspace="0em" rspace="0.167em" id="alg1.l13.m1.2.2.2.3" xref="alg1.l13.m1.2.2.3a.cmml">.</mo><mrow id="alg1.l13.m1.2.2.2.2" xref="alg1.l13.m1.2.2.2.2.cmml"><mi id="alg1.l13.m1.2.2.2.2.2" xref="alg1.l13.m1.2.2.2.2.2.cmml">b</mi><mo lspace="0em" rspace="0em" id="alg1.l13.m1.2.2.2.2.1" xref="alg1.l13.m1.2.2.2.2.1.cmml">​</mo><mi id="alg1.l13.m1.2.2.2.2.3" xref="alg1.l13.m1.2.2.2.2.3.cmml">a</mi><mo lspace="0em" rspace="0em" id="alg1.l13.m1.2.2.2.2.1a" xref="alg1.l13.m1.2.2.2.2.1.cmml">​</mo><mi id="alg1.l13.m1.2.2.2.2.4" xref="alg1.l13.m1.2.2.2.2.4.cmml">c</mi><mo lspace="0em" rspace="0em" id="alg1.l13.m1.2.2.2.2.1b" xref="alg1.l13.m1.2.2.2.2.1.cmml">​</mo><mi id="alg1.l13.m1.2.2.2.2.5" xref="alg1.l13.m1.2.2.2.2.5.cmml">k</mi><mo lspace="0em" rspace="0em" id="alg1.l13.m1.2.2.2.2.1c" xref="alg1.l13.m1.2.2.2.2.1.cmml">​</mo><mi id="alg1.l13.m1.2.2.2.2.6" xref="alg1.l13.m1.2.2.2.2.6.cmml">w</mi><mo lspace="0em" rspace="0em" id="alg1.l13.m1.2.2.2.2.1d" xref="alg1.l13.m1.2.2.2.2.1.cmml">​</mo><mi id="alg1.l13.m1.2.2.2.2.7" xref="alg1.l13.m1.2.2.2.2.7.cmml">a</mi><mo lspace="0em" rspace="0em" id="alg1.l13.m1.2.2.2.2.1e" xref="alg1.l13.m1.2.2.2.2.1.cmml">​</mo><mi id="alg1.l13.m1.2.2.2.2.8" xref="alg1.l13.m1.2.2.2.2.8.cmml">r</mi><mo lspace="0em" rspace="0em" id="alg1.l13.m1.2.2.2.2.1f" xref="alg1.l13.m1.2.2.2.2.1.cmml">​</mo><mi id="alg1.l13.m1.2.2.2.2.9" xref="alg1.l13.m1.2.2.2.2.9.cmml">d</mi><mo lspace="0em" rspace="0em" id="alg1.l13.m1.2.2.2.2.1g" xref="alg1.l13.m1.2.2.2.2.1.cmml">​</mo><mrow id="alg1.l13.m1.2.2.2.2.10.2" xref="alg1.l13.m1.2.2.2.2.cmml"><mo stretchy="false" id="alg1.l13.m1.2.2.2.2.10.2.1" xref="alg1.l13.m1.2.2.2.2.10.1.cmml">(</mo><mo stretchy="false" id="alg1.l13.m1.2.2.2.2.10.2.2" xref="alg1.l13.m1.2.2.2.2.10.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg1.l13.m1.2b"><apply id="alg1.l13.m1.2.2.3.cmml" xref="alg1.l13.m1.2.2.2"><csymbol cd="ambiguous" id="alg1.l13.m1.2.2.3a.cmml" xref="alg1.l13.m1.2.2.2.3">formulae-sequence</csymbol><apply id="alg1.l13.m1.1.1.1.1.cmml" xref="alg1.l13.m1.1.1.1.1"><times id="alg1.l13.m1.1.1.1.1.1.cmml" xref="alg1.l13.m1.1.1.1.1.1"></times><ci id="alg1.l13.m1.1.1.1.1.2.cmml" xref="alg1.l13.m1.1.1.1.1.2">𝑙</ci><ci id="alg1.l13.m1.1.1.1.1.3.cmml" xref="alg1.l13.m1.1.1.1.1.3">𝑜</ci><ci id="alg1.l13.m1.1.1.1.1.4.cmml" xref="alg1.l13.m1.1.1.1.1.4">𝑠</ci><ci id="alg1.l13.m1.1.1.1.1.5.cmml" xref="alg1.l13.m1.1.1.1.1.5">𝑠</ci></apply><apply id="alg1.l13.m1.2.2.2.2.cmml" xref="alg1.l13.m1.2.2.2.2"><times id="alg1.l13.m1.2.2.2.2.1.cmml" xref="alg1.l13.m1.2.2.2.2.1"></times><ci id="alg1.l13.m1.2.2.2.2.2.cmml" xref="alg1.l13.m1.2.2.2.2.2">𝑏</ci><ci id="alg1.l13.m1.2.2.2.2.3.cmml" xref="alg1.l13.m1.2.2.2.2.3">𝑎</ci><ci id="alg1.l13.m1.2.2.2.2.4.cmml" xref="alg1.l13.m1.2.2.2.2.4">𝑐</ci><ci id="alg1.l13.m1.2.2.2.2.5.cmml" xref="alg1.l13.m1.2.2.2.2.5">𝑘</ci><ci id="alg1.l13.m1.2.2.2.2.6.cmml" xref="alg1.l13.m1.2.2.2.2.6">𝑤</ci><ci id="alg1.l13.m1.2.2.2.2.7.cmml" xref="alg1.l13.m1.2.2.2.2.7">𝑎</ci><ci id="alg1.l13.m1.2.2.2.2.8.cmml" xref="alg1.l13.m1.2.2.2.2.8">𝑟</ci><ci id="alg1.l13.m1.2.2.2.2.9.cmml" xref="alg1.l13.m1.2.2.2.2.9">𝑑</ci><list id="alg1.l13.m1.2.2.2.2.10.1.cmml" xref="alg1.l13.m1.2.2.2.2.10.2.1"></list></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l13.m1.2c">loss.backward()</annotation></semantics></math>

</div>
<div id="alg1.l14" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">14:</span>     <span id="alg1.l14.1" class="ltx_text ltx_font_italic">update V via Eq. <a href="#S3.E8" title="In 3.1 SignRound ‣ 3 Methodology ‣ Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs" class="ltx_ref"><span class="ltx_text ltx_ref_tag">8</span></a> </span>

</div>
<div id="alg1.l15" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">15:</span><span id="alg1.l15.1" class="ltx_text ltx_font_bold">end</span> <span id="alg1.l15.2" class="ltx_text ltx_font_bold">for</span>
</div>
</div>
</div>
</div>
</figure>
</section>
</section>
<section id="S4" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">4 </span>Experiments</h2>

<div id="S4.p1" class="ltx_para ltx_noindent">
<p id="S4.p1.1" class="ltx_p">In this section, we conduct a comprehensive evaluation of SignRound from various perspectives. Firstly, we provide a brief overview of the LLM architectures and tasks that are included in our evaluation. Secondly, we present a detailed comparison between our method and some other existing approaches, highlighting the unique features and advantages of SignRound. Thirdly, we conduct additional experiments to further demonstrate the validity of our choices, assess the sensitivity of hyperparameters, and explore other relevant factors. Finally, the runtime is reported in Appendix <a href="#A4" title="Appendix D Runtime ‣ Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs" class="ltx_ref"><span class="ltx_text ltx_ref_tag">D</span></a> for reference.</p>
</div>
<section id="S4.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.1 </span>Experimental Settings</h3>

<section id="S4.SS1.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Evaluation and Datasets.</h4>

<div id="S4.SS1.SSS0.Px1.p1" class="ltx_para ltx_noindent">
<p id="S4.SS1.SSS0.Px1.p1.1" class="ltx_p">We make assessments on several language tasks to satisfy the task-agnostic setting. Specifically, we report average accuracy results on four common sense reasoning tasks including HellaSwag <cite class="ltx_cite ltx_citemacro_citep">(Zellers et al., <a href="#bib.bib44" title="" class="ltx_ref">2019</a>)</cite>, WinoGrande <cite class="ltx_cite ltx_citemacro_citep">(Sakaguchi et al., <a href="#bib.bib30" title="" class="ltx_ref">2021</a>)</cite>, PIQA <cite class="ltx_cite ltx_citemacro_citep">(Bisk et al., <a href="#bib.bib2" title="" class="ltx_ref">2020</a>)</cite> and LAMBADA <cite class="ltx_cite ltx_citemacro_citep">(Paperno et al., <a href="#bib.bib25" title="" class="ltx_ref">2016</a>)</cite>.Additionally, we benchmarked our models on MMLU <cite class="ltx_cite ltx_citemacro_citep">(<span class="ltx_ref ltx_missing_citation ltx_ref_self">hendrycks2020measuring</span>)</cite>, which encompasses 57 tasks spanning STEM, humanities, social science, and more. Evaluation for all these tasks was performed using the lm-eval-harness <cite class="ltx_cite ltx_citemacro_citep">(Gao et al., <a href="#bib.bib9" title="" class="ltx_ref">2021</a>)</cite>. Furthermore, we complement our evaluation with perplexity (ppl) analysis on Wikitext2 <cite class="ltx_cite ltx_citemacro_citep">(Merity et al., <a href="#bib.bib21" title="" class="ltx_ref">2016</a>)</cite> and C4 <cite class="ltx_cite ltx_citemacro_citep">(Raffel et al., <a href="#bib.bib27" title="" class="ltx_ref">2020</a>)</cite>, by following the source code <span id="footnote1" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">1</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">1</sup><span class="ltx_tag ltx_tag_note">1</span>https://github.com/IST-DASLab/gptq</span></span></span> of GPTQ.</p>
</div>
</section>
<section id="S4.SS1.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Quantization Configurations.</h4>

<div id="S4.SS1.SSS0.Px2.p1" class="ltx_para ltx_noindent">
<p id="S4.SS1.SSS0.Px2.p1.1" class="ltx_p">In line with the approach taken in GPTQ <cite class="ltx_cite ltx_citemacro_citep">(Frantar et al., <a href="#bib.bib8" title="" class="ltx_ref">2022</a>)</cite>, we specifically concentrate on weight-only quantization, targeting the linear layers within transformer blocks. Other layers, such as the embedding layer and typically the last layer like lm-head, are excluded from the quantization process. We initially intended to utilize the
pile <cite class="ltx_cite ltx_citemacro_citep">(<span class="ltx_ref ltx_missing_citation ltx_ref_self">gao2020pile</span>)</cite> dataset for calibration, following AWQ <cite class="ltx_cite ltx_citemacro_citep">(Lin et al., <a href="#bib.bib18" title="" class="ltx_ref">2023</a>)</cite> and SmoothQuant <cite class="ltx_cite ltx_citemacro_citep">(Xiao et al., <a href="#bib.bib38" title="" class="ltx_ref">2022</a>)</cite>. However, due to its large size, we have opted to use the readily available pile-10k dataset <span id="footnote2" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">2</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">2</sup><span class="ltx_tag ltx_tag_note">2</span>https://huggingface.co/datasets/NeelNanda/pile-10k</span></span></span>, which consists of the first 10k samples from pile, for both GPTQ and our method. We employ standard uniform per-row asymmetric quantization on the min-max grid. Our evaluation primarily focuses on W4, W4G128, and W3G128, where W4 indicates quantizing weights with 4 bits and G represents finer-granularity grouping as described in <cite class="ltx_cite ltx_citemacro_citep">(Park et al., <a href="#bib.bib26" title="" class="ltx_ref">2022</a>; Frantar et al., <a href="#bib.bib8" title="" class="ltx_ref">2022</a>)</cite>.</p>
</div>
</section>
<section id="S4.SS1.SSS0.Px3" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Large Language Models.</h4>

<div id="S4.SS1.SSS0.Px3.p1" class="ltx_para ltx_noindent">
<p id="S4.SS1.SSS0.Px3.p1.1" class="ltx_p">Our experimental evaluation encompasses a range of widely adopted LLM architectures, such as LLaMAs <cite class="ltx_cite ltx_citemacro_citep">(Touvron et al., <a href="#bib.bib32" title="" class="ltx_ref">2023a</a>)</cite>, LLaMAs v2 <cite class="ltx_cite ltx_citemacro_citep">(Touvron et al., <a href="#bib.bib33" title="" class="ltx_ref">2023b</a>)</cite>, BLOOMs <cite class="ltx_cite ltx_citemacro_citep">(Scao et al., <a href="#bib.bib31" title="" class="ltx_ref">2022</a>)</cite>, and OPTs <cite class="ltx_cite ltx_citemacro_citep">(Zhang et al., <a href="#bib.bib45" title="" class="ltx_ref">2022</a>)</cite>. We cover a wide range of LLM parameters, ranging from millions to billions, to ensure comprehensive coverage and analysis.</p>
</div>
</section>
<section id="S4.SS1.SSS0.Px4" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">SignRound Hyperparameters.</h4>

<div id="S4.SS1.SSS0.Px4.p1" class="ltx_para ltx_noindent">
<p id="S4.SS1.SSS0.Px4.p1.1" class="ltx_p">We selected 512 samples randomly from pile-10k and truncated each sample to a sequence length of 512. The tuning process involves adjusting each block for 400 steps using a learning rate of 2.5e-3, a batch size of 8, and employing a linear learning rate decay. We set the value of B in Eq. <a href="#S3.E8" title="In 3.1 SignRound ‣ 3 Methodology ‣ Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs" class="ltx_ref"><span class="ltx_text ltx_ref_tag">8</span></a> to 0.5. Besides, we adopted automatic mixed precision(AMP) to accelerate the tuning. It’s worth noting that adjusting the sequence length to 2048 yielded improvements in numerous scenarios. However, we did not adopt this as the default setting due to the associated runtime overhead.
For models <math id="S4.SS1.SSS0.Px4.p1.1.m1.1" class="ltx_Math" alttext="\geq" display="inline"><semantics id="S4.SS1.SSS0.Px4.p1.1.m1.1a"><mo id="S4.SS1.SSS0.Px4.p1.1.m1.1.1" xref="S4.SS1.SSS0.Px4.p1.1.m1.1.1.cmml">≥</mo><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS0.Px4.p1.1.m1.1b"><geq id="S4.SS1.SSS0.Px4.p1.1.m1.1.1.cmml" xref="S4.SS1.SSS0.Px4.p1.1.m1.1.1"></geq></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS0.Px4.p1.1.m1.1c">\geq</annotation></semantics></math> 30B, we made configuration adjustments to strike a balance between runtime and performance. Specifically, we reduced the sample count to 256, shorted the sequence length to 256, and disabled AMP.</p>
</div>
<figure id="S4.T1" class="ltx_table">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 1: </span> Average % accuracy(<math id="S4.T1.2.2.m1.1" class="ltx_Math" alttext="\uparrow" display="inline"><semantics id="S4.T1.2.2.m1.1b"><mo stretchy="false" id="S4.T1.2.2.m1.1.1" xref="S4.T1.2.2.m1.1.1.cmml">↑</mo><annotation-xml encoding="MathML-Content" id="S4.T1.2.2.m1.1c"><ci id="S4.T1.2.2.m1.1.1.cmml" xref="S4.T1.2.2.m1.1.1">↑</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.2.2.m1.1d">\uparrow</annotation></semantics></math>) of HellaSwag, WinoGrand, PIQA and LAMBADA for LLaMA &amp; OPT.</figcaption>
<table id="S4.T1.3" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S4.T1.3.1.1" class="ltx_tr">
<th id="S4.T1.3.1.1.1" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;" rowspan="2"><span id="S4.T1.3.1.1.1.1" class="ltx_text">nbits</span></th>
<th id="S4.T1.3.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;" rowspan="2"><span id="S4.T1.3.1.1.2.1" class="ltx_text">methods</span></th>
<th id="S4.T1.3.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;" colspan="4">LLaMA</th>
<th id="S4.T1.3.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;" colspan="5">OPT</th>
</tr>
<tr id="S4.T1.3.2.2" class="ltx_tr">
<th id="S4.T1.3.2.2.1" class="ltx_td ltx_align_center ltx_th ltx_th_column" style="padding-left:5.0pt;padding-right:5.0pt;">7b</th>
<th id="S4.T1.3.2.2.2" class="ltx_td ltx_align_center ltx_th ltx_th_column" style="padding-left:5.0pt;padding-right:5.0pt;">13b</th>
<th id="S4.T1.3.2.2.3" class="ltx_td ltx_align_center ltx_th ltx_th_column" style="padding-left:5.0pt;padding-right:5.0pt;">7bv2</th>
<th id="S4.T1.3.2.2.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;">13bv2</th>
<th id="S4.T1.3.2.2.5" class="ltx_td ltx_align_center ltx_th ltx_th_column" style="padding-left:5.0pt;padding-right:5.0pt;">125m</th>
<th id="S4.T1.3.2.2.6" class="ltx_td ltx_align_center ltx_th ltx_th_column" style="padding-left:5.0pt;padding-right:5.0pt;">1.3b</th>
<th id="S4.T1.3.2.2.7" class="ltx_td ltx_align_center ltx_th ltx_th_column" style="padding-left:5.0pt;padding-right:5.0pt;">2.7b</th>
<th id="S4.T1.3.2.2.8" class="ltx_td ltx_align_center ltx_th ltx_th_column" style="padding-left:5.0pt;padding-right:5.0pt;">6.7b</th>
<th id="S4.T1.3.2.2.9" class="ltx_td ltx_align_center ltx_th ltx_th_column" style="padding-left:5.0pt;padding-right:5.0pt;">13b</th>
</tr>
<tr id="S4.T1.3.3.3" class="ltx_tr">
<th id="S4.T1.3.3.3.1" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">16</th>
<th id="S4.T1.3.3.3.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">FP16</th>
<th id="S4.T1.3.3.3.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">68.80</th>
<th id="S4.T1.3.3.3.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">71.14</th>
<th id="S4.T1.3.3.3.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">69.02</th>
<th id="S4.T1.3.3.3.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">71.20</th>
<th id="S4.T1.3.3.3.7" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">45.09</th>
<th id="S4.T1.3.3.3.8" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">57.66</th>
<th id="S4.T1.3.3.3.9" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">61.04</th>
<th id="S4.T1.3.3.3.10" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">64.92</th>
<th id="S4.T1.3.3.3.11" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">65.49</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S4.T1.3.4.1" class="ltx_tr">
<td id="S4.T1.3.4.1.1" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;" rowspan="3"><span id="S4.T1.3.4.1.1.1" class="ltx_text">W4</span></td>
<td id="S4.T1.3.4.1.2" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">RTN</td>
<td id="S4.T1.3.4.1.3" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">67.38</td>
<td id="S4.T1.3.4.1.4" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">68.82</td>
<td id="S4.T1.3.4.1.5" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">66.98</td>
<td id="S4.T1.3.4.1.6" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T1.3.4.1.6.1" class="ltx_text ltx_font_bold">70.17</span></td>
<td id="S4.T1.3.4.1.7" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">39.41</td>
<td id="S4.T1.3.4.1.8" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">47.22</td>
<td id="S4.T1.3.4.1.9" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">58.61</td>
<td id="S4.T1.3.4.1.10" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">62.99</td>
<td id="S4.T1.3.4.1.11" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">64.08</td>
</tr>
<tr id="S4.T1.3.5.2" class="ltx_tr">
<td id="S4.T1.3.5.2.1" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;">GPTQ</td>
<td id="S4.T1.3.5.2.2" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">64.70</td>
<td id="S4.T1.3.5.2.3" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">70.00</td>
<td id="S4.T1.3.5.2.4" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">66.89</td>
<td id="S4.T1.3.5.2.5" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;">69.24</td>
<td id="S4.T1.3.5.2.6" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">43.58</td>
<td id="S4.T1.3.5.2.7" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">56.15</td>
<td id="S4.T1.3.5.2.8" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">59.92</td>
<td id="S4.T1.3.5.2.9" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">63.09</td>
<td id="S4.T1.3.5.2.10" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">64.83</td>
</tr>
<tr id="S4.T1.3.6.3" class="ltx_tr">
<td id="S4.T1.3.6.3.1" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;">Ours</td>
<td id="S4.T1.3.6.3.2" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T1.3.6.3.2.1" class="ltx_text ltx_font_bold">68.05</span></td>
<td id="S4.T1.3.6.3.3" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T1.3.6.3.3.1" class="ltx_text ltx_font_bold">70.58</span></td>
<td id="S4.T1.3.6.3.4" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T1.3.6.3.4.1" class="ltx_text ltx_font_bold">67.74</span></td>
<td id="S4.T1.3.6.3.5" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;">70.03</td>
<td id="S4.T1.3.6.3.6" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T1.3.6.3.6.1" class="ltx_text ltx_font_bold">44.13</span></td>
<td id="S4.T1.3.6.3.7" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T1.3.6.3.7.1" class="ltx_text ltx_font_bold">56.17</span></td>
<td id="S4.T1.3.6.3.8" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T1.3.6.3.8.1" class="ltx_text ltx_font_bold">60.58</span></td>
<td id="S4.T1.3.6.3.9" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T1.3.6.3.9.1" class="ltx_text ltx_font_bold">64.34</span></td>
<td id="S4.T1.3.6.3.10" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T1.3.6.3.10.1" class="ltx_text ltx_font_bold">65.05</span></td>
</tr>
<tr id="S4.T1.3.7.4" class="ltx_tr">
<td id="S4.T1.3.7.4.1" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;" rowspan="3"><span id="S4.T1.3.7.4.1.1" class="ltx_text">W4G128</span></td>
<td id="S4.T1.3.7.4.2" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">RTN</td>
<td id="S4.T1.3.7.4.3" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">67.85</td>
<td id="S4.T1.3.7.4.4" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">70.84</td>
<td id="S4.T1.3.7.4.5" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">68.32</td>
<td id="S4.T1.3.7.4.6" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">70.72</td>
<td id="S4.T1.3.7.4.7" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T1.3.7.4.7.1" class="ltx_text ltx_font_bold">45.27</span></td>
<td id="S4.T1.3.7.4.8" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">56.47</td>
<td id="S4.T1.3.7.4.9" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">60.70</td>
<td id="S4.T1.3.7.4.10" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">64.03</td>
<td id="S4.T1.3.7.4.11" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">64.84</td>
</tr>
<tr id="S4.T1.3.8.5" class="ltx_tr">
<td id="S4.T1.3.8.5.1" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;">GPTQ</td>
<td id="S4.T1.3.8.5.2" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">66.32</td>
<td id="S4.T1.3.8.5.3" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">70.92</td>
<td id="S4.T1.3.8.5.4" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T1.3.8.5.4.1" class="ltx_text ltx_font_bold">68.90</span></td>
<td id="S4.T1.3.8.5.5" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;">70.68</td>
<td id="S4.T1.3.8.5.6" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">42.88</td>
<td id="S4.T1.3.8.5.7" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">56.99</td>
<td id="S4.T1.3.8.5.8" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T1.3.8.5.8.1" class="ltx_text ltx_font_bold">61.23</span></td>
<td id="S4.T1.3.8.5.9" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">64.75</td>
<td id="S4.T1.3.8.5.10" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">65.37</td>
</tr>
<tr id="S4.T1.3.9.6" class="ltx_tr">
<td id="S4.T1.3.9.6.1" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;">Ours</td>
<td id="S4.T1.3.9.6.2" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T1.3.9.6.2.1" class="ltx_text ltx_font_bold">68.09</span></td>
<td id="S4.T1.3.9.6.3" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T1.3.9.6.3.1" class="ltx_text ltx_font_bold">71.43</span></td>
<td id="S4.T1.3.9.6.4" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">68.65</td>
<td id="S4.T1.3.9.6.5" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T1.3.9.6.5.1" class="ltx_text ltx_font_bold">70.81</span></td>
<td id="S4.T1.3.9.6.6" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">44.23</td>
<td id="S4.T1.3.9.6.7" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T1.3.9.6.7.1" class="ltx_text ltx_font_bold">57.30</span></td>
<td id="S4.T1.3.9.6.8" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">60.86</td>
<td id="S4.T1.3.9.6.9" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T1.3.9.6.9.1" class="ltx_text ltx_font_bold">64.76</span></td>
<td id="S4.T1.3.9.6.10" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T1.3.9.6.10.1" class="ltx_text ltx_font_bold">65.67</span></td>
</tr>
<tr id="S4.T1.3.10.7" class="ltx_tr">
<td id="S4.T1.3.10.7.1" class="ltx_td ltx_align_center ltx_border_b ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;" rowspan="3"><span id="S4.T1.3.10.7.1.1" class="ltx_text">W3G128</span></td>
<td id="S4.T1.3.10.7.2" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">RTN</td>
<td id="S4.T1.3.10.7.3" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">64.94</td>
<td id="S4.T1.3.10.7.4" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">67.70</td>
<td id="S4.T1.3.10.7.5" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">65.92</td>
<td id="S4.T1.3.10.7.6" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">68.70</td>
<td id="S4.T1.3.10.7.7" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">39.11</td>
<td id="S4.T1.3.10.7.8" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">42.61</td>
<td id="S4.T1.3.10.7.9" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">36.99</td>
<td id="S4.T1.3.10.7.10" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">56.09</td>
<td id="S4.T1.3.10.7.11" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">49.56</td>
</tr>
<tr id="S4.T1.3.11.8" class="ltx_tr">
<td id="S4.T1.3.11.8.1" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;">GPTQ</td>
<td id="S4.T1.3.11.8.2" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">58.29</td>
<td id="S4.T1.3.11.8.3" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">68.73</td>
<td id="S4.T1.3.11.8.4" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">65.51</td>
<td id="S4.T1.3.11.8.5" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;">68.73</td>
<td id="S4.T1.3.11.8.6" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">39.78</td>
<td id="S4.T1.3.11.8.7" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">54.43</td>
<td id="S4.T1.3.11.8.8" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">58.47</td>
<td id="S4.T1.3.11.8.9" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T1.3.11.8.9.1" class="ltx_text ltx_font_bold">62.98</span></td>
<td id="S4.T1.3.11.8.10" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T1.3.11.8.10.1" class="ltx_text ltx_font_bold">64.68</span></td>
</tr>
<tr id="S4.T1.3.12.9" class="ltx_tr">
<td id="S4.T1.3.12.9.1" class="ltx_td ltx_align_center ltx_border_b ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;">Ours</td>
<td id="S4.T1.3.12.9.2" class="ltx_td ltx_align_center ltx_border_b" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T1.3.12.9.2.1" class="ltx_text ltx_font_bold">66.62</span></td>
<td id="S4.T1.3.12.9.3" class="ltx_td ltx_align_center ltx_border_b" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T1.3.12.9.3.1" class="ltx_text ltx_font_bold">69.59</span></td>
<td id="S4.T1.3.12.9.4" class="ltx_td ltx_align_center ltx_border_b" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T1.3.12.9.4.1" class="ltx_text ltx_font_bold">66.88</span></td>
<td id="S4.T1.3.12.9.5" class="ltx_td ltx_align_center ltx_border_b ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T1.3.12.9.5.1" class="ltx_text ltx_font_bold">69.70</span></td>
<td id="S4.T1.3.12.9.6" class="ltx_td ltx_align_center ltx_border_b" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T1.3.12.9.6.1" class="ltx_text ltx_font_bold">43.31</span></td>
<td id="S4.T1.3.12.9.7" class="ltx_td ltx_align_center ltx_border_b" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T1.3.12.9.7.1" class="ltx_text ltx_font_bold">55.46</span></td>
<td id="S4.T1.3.12.9.8" class="ltx_td ltx_align_center ltx_border_b" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T1.3.12.9.8.1" class="ltx_text ltx_font_bold">59.12</span></td>
<td id="S4.T1.3.12.9.9" class="ltx_td ltx_align_center ltx_border_b" style="padding-left:5.0pt;padding-right:5.0pt;">53.42</td>
<td id="S4.T1.3.12.9.10" class="ltx_td ltx_align_center ltx_border_b" style="padding-left:5.0pt;padding-right:5.0pt;">63.61</td>
</tr>
</tbody>
</table>
</figure>
<figure id="S4.T2" class="ltx_table">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 2: </span>Average % accuracy(<math id="S4.T2.2.2.m1.1" class="ltx_Math" alttext="\uparrow" display="inline"><semantics id="S4.T2.2.2.m1.1b"><mo stretchy="false" id="S4.T2.2.2.m1.1.1" xref="S4.T2.2.2.m1.1.1.cmml">↑</mo><annotation-xml encoding="MathML-Content" id="S4.T2.2.2.m1.1c"><ci id="S4.T2.2.2.m1.1.1.cmml" xref="S4.T2.2.2.m1.1.1">↑</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.2.2.m1.1d">\uparrow</annotation></semantics></math>) of HellaSwag, WinoGrand, PIQA and LAMBADA for BLOOM. </figcaption>
<table id="S4.T2.3" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S4.T2.3.1.1" class="ltx_tr">
<th id="S4.T2.3.1.1.1" class="ltx_td ltx_th ltx_th_column ltx_th_row ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;"></th>
<th id="S4.T2.3.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;" colspan="4">W4</th>
<th id="S4.T2.3.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;" colspan="4">W4G128</th>
<th id="S4.T2.3.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;" colspan="4">W3G128</th>
</tr>
<tr id="S4.T2.3.2.2" class="ltx_tr">
<th id="S4.T2.3.2.2.1" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_th_row ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">Size</th>
<th id="S4.T2.3.2.2.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">560m</th>
<th id="S4.T2.3.2.2.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">1b7</th>
<th id="S4.T2.3.2.2.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">3b</th>
<th id="S4.T2.3.2.2.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">7b1</th>
<th id="S4.T2.3.2.2.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">560m</th>
<th id="S4.T2.3.2.2.7" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">1b7</th>
<th id="S4.T2.3.2.2.8" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">3b</th>
<th id="S4.T2.3.2.2.9" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">7b1</th>
<th id="S4.T2.3.2.2.10" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">560m</th>
<th id="S4.T2.3.2.2.11" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">1b7</th>
<th id="S4.T2.3.2.2.12" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">3b</th>
<th id="S4.T2.3.2.2.13" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">7b1</th>
</tr>
<tr id="S4.T2.3.3.3" class="ltx_tr">
<th id="S4.T2.3.3.3.1" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_th_row ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">FP16</th>
<th id="S4.T2.3.3.3.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">45.50</th>
<th id="S4.T2.3.3.3.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">52.31</th>
<th id="S4.T2.3.3.3.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">55.48</th>
<th id="S4.T2.3.3.3.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">60.22</th>
<th id="S4.T2.3.3.3.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">45.50</th>
<th id="S4.T2.3.3.3.7" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">52.31</th>
<th id="S4.T2.3.3.3.8" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">55.48</th>
<th id="S4.T2.3.3.3.9" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">60.22</th>
<th id="S4.T2.3.3.3.10" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">45.50</th>
<th id="S4.T2.3.3.3.11" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">52.31</th>
<th id="S4.T2.3.3.3.12" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">55.48</th>
<th id="S4.T2.3.3.3.13" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">60.22</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S4.T2.3.4.1" class="ltx_tr">
<th id="S4.T2.3.4.1.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">RTN</th>
<td id="S4.T2.3.4.1.2" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">43.10</td>
<td id="S4.T2.3.4.1.3" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">49.97</td>
<td id="S4.T2.3.4.1.4" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">53.16</td>
<td id="S4.T2.3.4.1.5" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">57.73</td>
<td id="S4.T2.3.4.1.6" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">44.28</td>
<td id="S4.T2.3.4.1.7" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T2.3.4.1.7.1" class="ltx_text ltx_font_bold">52.08</span></td>
<td id="S4.T2.3.4.1.8" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">54.86</td>
<td id="S4.T2.3.4.1.9" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">59.31</td>
<td id="S4.T2.3.4.1.10" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">40.83</td>
<td id="S4.T2.3.4.1.11" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">47.98</td>
<td id="S4.T2.3.4.1.12" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">52.51</td>
<td id="S4.T2.3.4.1.13" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">57.59</td>
</tr>
<tr id="S4.T2.3.5.2" class="ltx_tr">
<th id="S4.T2.3.5.2.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;">GPTQ</th>
<td id="S4.T2.3.5.2.2" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">43.95</td>
<td id="S4.T2.3.5.2.3" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">50.91</td>
<td id="S4.T2.3.5.2.4" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T2.3.5.2.4.1" class="ltx_text ltx_font_bold">54.65</span></td>
<td id="S4.T2.3.5.2.5" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;">58.27</td>
<td id="S4.T2.3.5.2.6" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">44.79</td>
<td id="S4.T2.3.5.2.7" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T2.3.5.2.7.1" class="ltx_text ltx_font_bold">52.08</span></td>
<td id="S4.T2.3.5.2.8" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T2.3.5.2.8.1" class="ltx_text ltx_font_bold">55.68</span></td>
<td id="S4.T2.3.5.2.9" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;">59.59</td>
<td id="S4.T2.3.5.2.10" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">42.74</td>
<td id="S4.T2.3.5.2.11" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">48.81</td>
<td id="S4.T2.3.5.2.12" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">53.41</td>
<td id="S4.T2.3.5.2.13" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">58.12</td>
</tr>
<tr id="S4.T2.3.6.3" class="ltx_tr">
<th id="S4.T2.3.6.3.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_b ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;">Ours</th>
<td id="S4.T2.3.6.3.2" class="ltx_td ltx_align_center ltx_border_b" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T2.3.6.3.2.1" class="ltx_text ltx_font_bold">45.00</span></td>
<td id="S4.T2.3.6.3.3" class="ltx_td ltx_align_center ltx_border_b" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T2.3.6.3.3.1" class="ltx_text ltx_font_bold">51.47</span></td>
<td id="S4.T2.3.6.3.4" class="ltx_td ltx_align_center ltx_border_b" style="padding-left:5.0pt;padding-right:5.0pt;">54.63</td>
<td id="S4.T2.3.6.3.5" class="ltx_td ltx_align_center ltx_border_b ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T2.3.6.3.5.1" class="ltx_text ltx_font_bold">59.52</span></td>
<td id="S4.T2.3.6.3.6" class="ltx_td ltx_align_center ltx_border_b" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T2.3.6.3.6.1" class="ltx_text ltx_font_bold">45.40</span></td>
<td id="S4.T2.3.6.3.7" class="ltx_td ltx_align_center ltx_border_b" style="padding-left:5.0pt;padding-right:5.0pt;">51.85</td>
<td id="S4.T2.3.6.3.8" class="ltx_td ltx_align_center ltx_border_b" style="padding-left:5.0pt;padding-right:5.0pt;">55.40</td>
<td id="S4.T2.3.6.3.9" class="ltx_td ltx_align_center ltx_border_b ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T2.3.6.3.9.1" class="ltx_text ltx_font_bold">59.83</span></td>
<td id="S4.T2.3.6.3.10" class="ltx_td ltx_align_center ltx_border_b" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T2.3.6.3.10.1" class="ltx_text ltx_font_bold">44.08</span></td>
<td id="S4.T2.3.6.3.11" class="ltx_td ltx_align_center ltx_border_b" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T2.3.6.3.11.1" class="ltx_text ltx_font_bold">50.52</span></td>
<td id="S4.T2.3.6.3.12" class="ltx_td ltx_align_center ltx_border_b" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T2.3.6.3.12.1" class="ltx_text ltx_font_bold">53.64</span></td>
<td id="S4.T2.3.6.3.13" class="ltx_td ltx_align_center ltx_border_b" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T2.3.6.3.13.1" class="ltx_text ltx_font_bold">58.69</span></td>
</tr>
</tbody>
</table>
</figure>
<figure id="S4.T3" class="ltx_table">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table">Table 3: </span>C4 ppl ( <math id="S4.T3.2.m1.1" class="ltx_Math" alttext="\downarrow" display="inline"><semantics id="S4.T3.2.m1.1b"><mo stretchy="false" id="S4.T3.2.m1.1.1" xref="S4.T3.2.m1.1.1.cmml">↓</mo><annotation-xml encoding="MathML-Content" id="S4.T3.2.m1.1c"><ci id="S4.T3.2.m1.1.1.cmml" xref="S4.T3.2.m1.1.1">↓</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.T3.2.m1.1d">\downarrow</annotation></semantics></math>) at W4.</figcaption>
<table id="S4.T3.3" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S4.T3.3.1.1" class="ltx_tr">
<th id="S4.T3.3.1.1.1" class="ltx_td ltx_th ltx_th_column ltx_th_row ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;"></th>
<th id="S4.T3.3.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;" colspan="4">LLaMA</th>
<th id="S4.T3.3.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;" colspan="4">OPT</th>
<th id="S4.T3.3.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;" colspan="4">BLOOM</th>
</tr>
<tr id="S4.T3.3.2.2" class="ltx_tr">
<th id="S4.T3.3.2.2.1" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_th_row ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">Size</th>
<th id="S4.T3.3.2.2.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">7b</th>
<th id="S4.T3.3.2.2.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">13b</th>
<th id="S4.T3.3.2.2.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">7bv2</th>
<th id="S4.T3.3.2.2.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">13bv2</th>
<th id="S4.T3.3.2.2.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">1.3b</th>
<th id="S4.T3.3.2.2.7" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">2.7b</th>
<th id="S4.T3.3.2.2.8" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">6.7b</th>
<th id="S4.T3.3.2.2.9" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">13b</th>
<th id="S4.T3.3.2.2.10" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">560m</th>
<th id="S4.T3.3.2.2.11" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">1b7</th>
<th id="S4.T3.3.2.2.12" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">3b</th>
<th id="S4.T3.3.2.2.13" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">7b1</th>
</tr>
<tr id="S4.T3.3.3.3" class="ltx_tr">
<th id="S4.T3.3.3.3.1" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_th_row ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">FP16</th>
<th id="S4.T3.3.3.3.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">7.34</th>
<th id="S4.T3.3.3.3.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">6.80</th>
<th id="S4.T3.3.3.3.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">7.26</th>
<th id="S4.T3.3.3.3.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">6.73</th>
<th id="S4.T3.3.3.3.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">16.07</th>
<th id="S4.T3.3.3.3.7" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">14.34</th>
<th id="S4.T3.3.3.3.8" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">12.71</th>
<th id="S4.T3.3.3.3.9" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">12.06</th>
<th id="S4.T3.3.3.3.10" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">26.59</th>
<th id="S4.T3.3.3.3.11" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">19.49</th>
<th id="S4.T3.3.3.3.12" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">17.48</th>
<th id="S4.T3.3.3.3.13" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">15.20</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S4.T3.3.4.1" class="ltx_tr">
<th id="S4.T3.3.4.1.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">RTN</th>
<td id="S4.T3.3.4.1.2" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">8.12</td>
<td id="S4.T3.3.4.1.3" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">7.23</td>
<td id="S4.T3.3.4.1.4" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">8.16</td>
<td id="S4.T3.3.4.1.5" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">7.14</td>
<td id="S4.T3.3.4.1.6" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">27.49</td>
<td id="S4.T3.3.4.1.7" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">18.83</td>
<td id="S4.T3.3.4.1.8" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">14.37</td>
<td id="S4.T3.3.4.1.9" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">13.32</td>
<td id="S4.T3.3.4.1.10" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">29.87</td>
<td id="S4.T3.3.4.1.11" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">21.25</td>
<td id="S4.T3.3.4.1.12" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">18.76</td>
<td id="S4.T3.3.4.1.13" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">16.05</td>
</tr>
<tr id="S4.T3.3.5.2" class="ltx_tr">
<th id="S4.T3.3.5.2.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;">GPTQ</th>
<td id="S4.T3.3.5.2.2" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">8.64</td>
<td id="S4.T3.3.5.2.3" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">7.13</td>
<td id="S4.T3.3.5.2.4" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T3.3.5.2.4.1" class="ltx_text ltx_font_bold">7.90</span></td>
<td id="S4.T3.3.5.2.5" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T3.3.5.2.5.1" class="ltx_text ltx_font_bold">6.87</span></td>
<td id="S4.T3.3.5.2.6" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">17.04</td>
<td id="S4.T3.3.5.2.7" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">15.06</td>
<td id="S4.T3.3.5.2.8" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">13.39</td>
<td id="S4.T3.3.5.2.9" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T3.3.5.2.9.1" class="ltx_text ltx_font_bold">12.29</span></td>
<td id="S4.T3.3.5.2.10" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">28.15</td>
<td id="S4.T3.3.5.2.11" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">20.71</td>
<td id="S4.T3.3.5.2.12" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T3.3.5.2.12.1" class="ltx_text ltx_font_bold">18.18</span></td>
<td id="S4.T3.3.5.2.13" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T3.3.5.2.13.1" class="ltx_text ltx_font_bold">15.67</span></td>
</tr>
<tr id="S4.T3.3.6.3" class="ltx_tr">
<th id="S4.T3.3.6.3.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_b ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;">Ours</th>
<td id="S4.T3.3.6.3.2" class="ltx_td ltx_align_center ltx_border_b" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T3.3.6.3.2.1" class="ltx_text ltx_font_bold">7.84</span></td>
<td id="S4.T3.3.6.3.3" class="ltx_td ltx_align_center ltx_border_b" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T3.3.6.3.3.1" class="ltx_text ltx_font_bold">7.05</span></td>
<td id="S4.T3.3.6.3.4" class="ltx_td ltx_align_center ltx_border_b" style="padding-left:5.0pt;padding-right:5.0pt;">11.20</td>
<td id="S4.T3.3.6.3.5" class="ltx_td ltx_align_center ltx_border_b ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;">7.72</td>
<td id="S4.T3.3.6.3.6" class="ltx_td ltx_align_center ltx_border_b" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T3.3.6.3.6.1" class="ltx_text ltx_font_bold">16.92</span></td>
<td id="S4.T3.3.6.3.7" class="ltx_td ltx_align_center ltx_border_b" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T3.3.6.3.7.1" class="ltx_text ltx_font_bold">14.97</span></td>
<td id="S4.T3.3.6.3.8" class="ltx_td ltx_align_center ltx_border_b" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T3.3.6.3.8.1" class="ltx_text ltx_font_bold">13.08</span></td>
<td id="S4.T3.3.6.3.9" class="ltx_td ltx_align_center ltx_border_b ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;">12.48</td>
<td id="S4.T3.3.6.3.10" class="ltx_td ltx_align_center ltx_border_b" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T3.3.6.3.10.1" class="ltx_text ltx_font_bold">28.12</span></td>
<td id="S4.T3.3.6.3.11" class="ltx_td ltx_align_center ltx_border_b" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T3.3.6.3.11.1" class="ltx_text ltx_font_bold">20.41</span></td>
<td id="S4.T3.3.6.3.12" class="ltx_td ltx_align_center ltx_border_b" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T3.3.6.3.12.1" class="ltx_text ltx_font_bold">18.18</span></td>
<td id="S4.T3.3.6.3.13" class="ltx_td ltx_align_center ltx_border_b" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S4.T3.3.6.3.13.1" class="ltx_text ltx_font_bold">15.67</span></td>
</tr>
</tbody>
</table>
</figure>
</section>
</section>
<section id="S4.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.2 </span>Comparing With Other Methods</h3>

<div id="S4.SS2.p1" class="ltx_para ltx_noindent">
<p id="S4.SS2.p1.1" class="ltx_p">We conducted a comprehensive benchmarking of our results against RTN and GPTQ <cite class="ltx_cite ltx_citemacro_citep">(Frantar et al., <a href="#bib.bib8" title="" class="ltx_ref">2022</a>)</cite>. However, it is important to highlight that act-order was not enabled in GPTQ due to the kernel overhead it introduces <cite class="ltx_cite ltx_citemacro_citep">(Lin et al., <a href="#bib.bib18" title="" class="ltx_ref">2023</a>)</cite>, although it has the potential to improve accuracy for certain models. When evaluating perplexity (ppl), we prioritize reporting the ppl on C4 dataset as our primary focus, taking into consideration the potential occurrence of NaN values when assessing perplexity for Wikitext2 and ptb datasets, both for SignRound and GPTQ. Furthermore, we conducted a limited and non-rigorous comparison between our approach and AWQ <cite class="ltx_cite ltx_citemacro_cite">Lin et al. (<a href="#bib.bib18" title="" class="ltx_ref">2023</a>)</cite> in Appendix <a href="#A1.SS1" title="A.1 Non-rigorous comparison with AWQ ‣ Appendix A More results ‣ Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs" class="ltx_ref"><span class="ltx_text ltx_ref_tag">A.1</span></a>.</p>
</div>
<div id="S4.SS2.p2" class="ltx_para ltx_noindent">
<p id="S4.SS2.p2.1" class="ltx_p">We begin by presenting the average accuracy results for the HellaSwag, WinoGrand, PIQA, and LAMBADA tasks across LLaMA, OPT, and BLOOM models with a size below 13B. These results are shown in Table <a href="#S4.T1" title="Table 1 ‣ SignRound Hyperparameters. ‣ 4.1 Experimental Settings ‣ 4 Experiments ‣ Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a> and <a href="#S4.T2" title="Table 2 ‣ SignRound Hyperparameters. ‣ 4.1 Experimental Settings ‣ 4 Experiments ‣ Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>. In conclusion, our method outperforms RTN in 36 out of 39 scenarios, showcasing its effectiveness. Additionally, when comparing our approach to GPTQ, we surpass it in 32 out of 39 scenarios, further highlighting the strengths of our method. While our method showcases overall effectiveness, it is important to acknowledge the presence of outliers, such as OPT6.7B at W3G128. Although the root cause for this has not been identified yet, it could be mitigated by fine-tuning hyperparameters, as discussed in the following sections. For detailed results of LLaMA7B, LLaMA13B, LLAMA7B-V2, and LLAMA13B-V2, please refer to Appendix <a href="#A5" title="Appendix E Detailed Results of some LLaMa models ‣ Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs" class="ltx_ref"><span class="ltx_text ltx_ref_tag">E</span></a>. The results in Appendix <a href="#A5" title="Appendix E Detailed Results of some LLaMa models ‣ Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs" class="ltx_ref"><span class="ltx_text ltx_ref_tag">E</span></a> also highlight that changing the sequence length to 2048 could bring noticeable improvement in many scenarios.</p>
</div>
<div id="S4.SS2.p3" class="ltx_para ltx_noindent">
<p id="S4.SS2.p3.1" class="ltx_p">We then present the perplexity (ppl) results for C4 in Table <a href="#S4.T3" title="Table 3 ‣ SignRound Hyperparameters. ‣ 4.1 Experimental Settings ‣ 4 Experiments ‣ Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>, along with the detailed results for Wikitext2 in Appendix <a href="#A1.SS2" title="A.2 Results of Wikitext2 ppl at W4 ‣ Appendix A More results ‣ Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs" class="ltx_ref"><span class="ltx_text ltx_ref_tag">A.2</span></a>. In conclusion, we achieve better or comparable performance in 9 out of 12 models. In certain cases where the results may not be optimal, we can still fine-tune the hyperparameters to achieve better results, as demonstrated in the subsequent sections.</p>
</div>
<div id="S4.SS2.p4" class="ltx_para ltx_noindent">
<p id="S4.SS2.p4.1" class="ltx_p">Next, we present a comprehensive breakdown of the accuracies achieved by MMLU for LLaMA-7B and LLaMa-7B-V2 in Table <a href="#S4.T4" title="Table 4 ‣ 4.2 Comparing With Other Methods ‣ 4 Experiments ‣ Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>. By analyzing the average accuracies, we observe that SingRound outperforms RTN and GPTQ in 4 out of the 6 scenarios when the best model-wise setting is applied.</p>
</div>
<figure id="S4.T4" class="ltx_table">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 4: </span>Accuracies(<math id="S4.T4.2.2.m1.1" class="ltx_Math" alttext="\uparrow" display="inline"><semantics id="S4.T4.2.2.m1.1b"><mo stretchy="false" id="S4.T4.2.2.m1.1.1" xref="S4.T4.2.2.m1.1.1.cmml">↑</mo><annotation-xml encoding="MathML-Content" id="S4.T4.2.2.m1.1c"><ci id="S4.T4.2.2.m1.1.1.cmml" xref="S4.T4.2.2.m1.1.1">↑</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.T4.2.2.m1.1d">\uparrow</annotation></semantics></math>) of MMLU(5-shot) for LLaMA-7B &amp; LLaMA-7B-V2. ”Ours-2048” indicates that we have modified the sequence length of the calibration dataset from 512 to 2048.</figcaption>
<table id="S4.T4.3" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S4.T4.3.1.1" class="ltx_tr">
<th id="S4.T4.3.1.1.1" class="ltx_td ltx_th ltx_th_column ltx_th_row ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;" rowspan="2"></th>
<th id="S4.T4.3.1.1.2" class="ltx_td ltx_th ltx_th_column ltx_th_row ltx_border_r ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;"></th>
<th id="S4.T4.3.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;" colspan="5">LLaMA-7B</th>
<th id="S4.T4.3.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;" colspan="5">LLaMA-7B-V2</th>
</tr>
<tr id="S4.T4.3.2.2" class="ltx_tr">
<th id="S4.T4.3.2.2.1" class="ltx_td ltx_th ltx_th_column ltx_th_row ltx_border_r" style="padding-left:3.0pt;padding-right:3.0pt;"></th>
<th id="S4.T4.3.2.2.2" class="ltx_td ltx_align_center ltx_th ltx_th_column" style="padding-left:3.0pt;padding-right:3.0pt;">Hums.</th>
<th id="S4.T4.3.2.2.3" class="ltx_td ltx_align_center ltx_th ltx_th_column" style="padding-left:3.0pt;padding-right:3.0pt;">STEM</th>
<th id="S4.T4.3.2.2.4" class="ltx_td ltx_align_center ltx_th ltx_th_column" style="padding-left:3.0pt;padding-right:3.0pt;">Social</th>
<th id="S4.T4.3.2.2.5" class="ltx_td ltx_align_center ltx_th ltx_th_column" style="padding-left:3.0pt;padding-right:3.0pt;">Other</th>
<th id="S4.T4.3.2.2.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r" style="padding-left:3.0pt;padding-right:3.0pt;">Avg.</th>
<th id="S4.T4.3.2.2.7" class="ltx_td ltx_align_center ltx_th ltx_th_column" style="padding-left:3.0pt;padding-right:3.0pt;">Hums.</th>
<th id="S4.T4.3.2.2.8" class="ltx_td ltx_align_center ltx_th ltx_th_column" style="padding-left:3.0pt;padding-right:3.0pt;">STEM</th>
<th id="S4.T4.3.2.2.9" class="ltx_td ltx_align_center ltx_th ltx_th_column" style="padding-left:3.0pt;padding-right:3.0pt;">Social</th>
<th id="S4.T4.3.2.2.10" class="ltx_td ltx_align_center ltx_th ltx_th_column" style="padding-left:3.0pt;padding-right:3.0pt;">Other</th>
<th id="S4.T4.3.2.2.11" class="ltx_td ltx_align_center ltx_th ltx_th_column" style="padding-left:3.0pt;padding-right:3.0pt;">Avg.</th>
</tr>
<tr id="S4.T4.3.3.3" class="ltx_tr">
<th id="S4.T4.3.3.3.1" class="ltx_td ltx_th ltx_th_column ltx_th_row ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;"></th>
<th id="S4.T4.3.3.3.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_th_row ltx_border_r ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">FP16</th>
<th id="S4.T4.3.3.3.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">38.32</th>
<th id="S4.T4.3.3.3.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">31.17</th>
<th id="S4.T4.3.3.3.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">38.05</th>
<th id="S4.T4.3.3.3.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">36.85</th>
<th id="S4.T4.3.3.3.7" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">35.64</th>
<th id="S4.T4.3.3.3.8" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">51.40</th>
<th id="S4.T4.3.3.3.9" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">37.00</th>
<th id="S4.T4.3.3.3.10" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">52.23</th>
<th id="S4.T4.3.3.3.11" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">49.51</th>
<th id="S4.T4.3.3.3.12" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">46.56</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S4.T4.3.4.1" class="ltx_tr">
<th id="S4.T4.3.4.1.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;" rowspan="4"><span id="S4.T4.3.4.1.1.1" class="ltx_text">W4G-1</span></th>
<th id="S4.T4.3.4.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">RTN</th>
<td id="S4.T4.3.4.1.3" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">34.84</td>
<td id="S4.T4.3.4.1.4" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">29.53</td>
<td id="S4.T4.3.4.1.5" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">32.87</td>
<td id="S4.T4.3.4.1.6" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">36.28</td>
<td id="S4.T4.3.4.1.7" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">33.10</td>
<td id="S4.T4.3.4.1.8" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">44.03</td>
<td id="S4.T4.3.4.1.9" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">32.83</td>
<td id="S4.T4.3.4.1.10" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">44.97</td>
<td id="S4.T4.3.4.1.11" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">42.19</td>
<td id="S4.T4.3.4.1.12" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">40.24</td>
</tr>
<tr id="S4.T4.3.5.2" class="ltx_tr">
<th id="S4.T4.3.5.2.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r" style="padding-left:3.0pt;padding-right:3.0pt;">GPTQ</th>
<td id="S4.T4.3.5.2.2" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">33.31</td>
<td id="S4.T4.3.5.2.3" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">26.29</td>
<td id="S4.T4.3.5.2.4" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">29.86</td>
<td id="S4.T4.3.5.2.5" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">33.11</td>
<td id="S4.T4.3.5.2.6" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:3.0pt;padding-right:3.0pt;">30.32</td>
<td id="S4.T4.3.5.2.7" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">46.21</td>
<td id="S4.T4.3.5.2.8" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S4.T4.3.5.2.8.1" class="ltx_text ltx_font_bold">34.29</span></td>
<td id="S4.T4.3.5.2.9" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">46.68</td>
<td id="S4.T4.3.5.2.10" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">44.85</td>
<td id="S4.T4.3.5.2.11" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">42.21</td>
</tr>
<tr id="S4.T4.3.6.3" class="ltx_tr">
<th id="S4.T4.3.6.3.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r" style="padding-left:3.0pt;padding-right:3.0pt;">Ours</th>
<td id="S4.T4.3.6.3.2" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">34.30</td>
<td id="S4.T4.3.6.3.3" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S4.T4.3.6.3.3.1" class="ltx_text ltx_font_bold">31.05</span></td>
<td id="S4.T4.3.6.3.4" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">34.74</td>
<td id="S4.T4.3.6.3.5" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">36.66</td>
<td id="S4.T4.3.6.3.6" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:3.0pt;padding-right:3.0pt;">33.95</td>
<td id="S4.T4.3.6.3.7" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">47.28</td>
<td id="S4.T4.3.6.3.8" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">33.14</td>
<td id="S4.T4.3.6.3.9" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">46.90</td>
<td id="S4.T4.3.6.3.10" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">44.70</td>
<td id="S4.T4.3.6.3.11" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">42.10</td>
</tr>
<tr id="S4.T4.3.7.4" class="ltx_tr">
<th id="S4.T4.3.7.4.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r" style="padding-left:3.0pt;padding-right:3.0pt;">Ours2048</th>
<td id="S4.T4.3.7.4.2" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S4.T4.3.7.4.2.1" class="ltx_text ltx_font_bold">35.10</span></td>
<td id="S4.T4.3.7.4.3" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">30.69</td>
<td id="S4.T4.3.7.4.4" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S4.T4.3.7.4.4.1" class="ltx_text ltx_font_bold">36.43</span></td>
<td id="S4.T4.3.7.4.5" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S4.T4.3.7.4.5.1" class="ltx_text ltx_font_bold">36.85</span></td>
<td id="S4.T4.3.7.4.6" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S4.T4.3.7.4.6.1" class="ltx_text ltx_font_bold">34.42</span></td>
<td id="S4.T4.3.7.4.7" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S4.T4.3.7.4.7.1" class="ltx_text ltx_font_bold">47.40</span></td>
<td id="S4.T4.3.7.4.8" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">33.92</td>
<td id="S4.T4.3.7.4.9" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S4.T4.3.7.4.9.1" class="ltx_text ltx_font_bold">49.61</span></td>
<td id="S4.T4.3.7.4.10" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S4.T4.3.7.4.10.1" class="ltx_text ltx_font_bold">44.91</span></td>
<td id="S4.T4.3.7.4.11" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S4.T4.3.7.4.11.1" class="ltx_text ltx_font_bold">43.00</span></td>
</tr>
<tr id="S4.T4.3.8.5" class="ltx_tr">
<th id="S4.T4.3.8.5.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;" rowspan="4"><span id="S4.T4.3.8.5.1.1" class="ltx_text">W4G128</span></th>
<th id="S4.T4.3.8.5.2" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">RTN</th>
<td id="S4.T4.3.8.5.3" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">36.30</td>
<td id="S4.T4.3.8.5.4" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S4.T4.3.8.5.4.1" class="ltx_text ltx_font_bold">31.67</span></td>
<td id="S4.T4.3.8.5.5" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S4.T4.3.8.5.5.1" class="ltx_text ltx_font_bold">37.40</span></td>
<td id="S4.T4.3.8.5.6" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S4.T4.3.8.5.6.1" class="ltx_text ltx_font_bold">37.99</span></td>
<td id="S4.T4.3.8.5.7" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S4.T4.3.8.5.7.1" class="ltx_text ltx_font_bold">35.48</span></td>
<td id="S4.T4.3.8.5.8" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">49.54</td>
<td id="S4.T4.3.8.5.9" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">36.50</td>
<td id="S4.T4.3.8.5.10" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">50.95</td>
<td id="S4.T4.3.8.5.11" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">47.87</td>
<td id="S4.T4.3.8.5.12" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">45.31</td>
</tr>
<tr id="S4.T4.3.9.6" class="ltx_tr">
<th id="S4.T4.3.9.6.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r" style="padding-left:3.0pt;padding-right:3.0pt;">GPTQ</th>
<td id="S4.T4.3.9.6.2" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S4.T4.3.9.6.2.1" class="ltx_text ltx_font_bold">37.77</span></td>
<td id="S4.T4.3.9.6.3" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">29.64</td>
<td id="S4.T4.3.9.6.4" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">36.38</td>
<td id="S4.T4.3.9.6.5" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">37.45</td>
<td id="S4.T4.3.9.6.6" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:3.0pt;padding-right:3.0pt;">34.83</td>
<td id="S4.T4.3.9.6.7" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">50.30</td>
<td id="S4.T4.3.9.6.8" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">36.51</td>
<td id="S4.T4.3.9.6.9" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">50.91</td>
<td id="S4.T4.3.9.6.10" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">47.69</td>
<td id="S4.T4.3.9.6.11" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">45.43</td>
</tr>
<tr id="S4.T4.3.10.7" class="ltx_tr">
<th id="S4.T4.3.10.7.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r" style="padding-left:3.0pt;padding-right:3.0pt;">Ours</th>
<td id="S4.T4.3.10.7.2" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">36.06</td>
<td id="S4.T4.3.10.7.3" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">30.86</td>
<td id="S4.T4.3.10.7.4" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">35.99</td>
<td id="S4.T4.3.10.7.5" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">36.21</td>
<td id="S4.T4.3.10.7.6" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:3.0pt;padding-right:3.0pt;">34.44</td>
<td id="S4.T4.3.10.7.7" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S4.T4.3.10.7.7.1" class="ltx_text ltx_font_bold">51.39</span></td>
<td id="S4.T4.3.10.7.8" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S4.T4.3.10.7.8.1" class="ltx_text ltx_font_bold">37.87</span></td>
<td id="S4.T4.3.10.7.9" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S4.T4.3.10.7.9.1" class="ltx_text ltx_font_bold">52.56</span></td>
<td id="S4.T4.3.10.7.10" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S4.T4.3.10.7.10.1" class="ltx_text ltx_font_bold">49.69</span></td>
<td id="S4.T4.3.10.7.11" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S4.T4.3.10.7.11.1" class="ltx_text ltx_font_bold">46.95</span></td>
</tr>
<tr id="S4.T4.3.11.8" class="ltx_tr">
<th id="S4.T4.3.11.8.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r" style="padding-left:3.0pt;padding-right:3.0pt;">Ours2048</th>
<td id="S4.T4.3.11.8.2" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">35.66</td>
<td id="S4.T4.3.11.8.3" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">30.05</td>
<td id="S4.T4.3.11.8.4" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">36.16</td>
<td id="S4.T4.3.11.8.5" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">37.57</td>
<td id="S4.T4.3.11.8.6" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:3.0pt;padding-right:3.0pt;">34.46</td>
<td id="S4.T4.3.11.8.7" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">50.12</td>
<td id="S4.T4.3.11.8.8" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">36.70</td>
<td id="S4.T4.3.11.8.9" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">51.44</td>
<td id="S4.T4.3.11.8.10" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">48.20</td>
<td id="S4.T4.3.11.8.11" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">45.69</td>
</tr>
<tr id="S4.T4.3.12.9" class="ltx_tr">
<th id="S4.T4.3.12.9.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_b ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;" rowspan="4"><span id="S4.T4.3.12.9.1.1" class="ltx_text">W3G128</span></th>
<th id="S4.T4.3.12.9.2" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">RTN</th>
<td id="S4.T4.3.12.9.3" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S4.T4.3.12.9.3.1" class="ltx_text ltx_font_bold">32.97</span></td>
<td id="S4.T4.3.12.9.4" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S4.T4.3.12.9.4.1" class="ltx_text ltx_font_bold">30.28</span></td>
<td id="S4.T4.3.12.9.5" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S4.T4.3.12.9.5.1" class="ltx_text ltx_font_bold">33.66</span></td>
<td id="S4.T4.3.12.9.6" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S4.T4.3.12.9.6.1" class="ltx_text ltx_font_bold">32.60</span></td>
<td id="S4.T4.3.12.9.7" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S4.T4.3.12.9.7.1" class="ltx_text ltx_font_bold">32.17</span></td>
<td id="S4.T4.3.12.9.8" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">41.14</td>
<td id="S4.T4.3.12.9.9" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">33.06</td>
<td id="S4.T4.3.12.9.10" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">40.98</td>
<td id="S4.T4.3.12.9.11" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">40.94</td>
<td id="S4.T4.3.12.9.12" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">38.51</td>
</tr>
<tr id="S4.T4.3.13.10" class="ltx_tr">
<th id="S4.T4.3.13.10.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r" style="padding-left:3.0pt;padding-right:3.0pt;">GPTQ</th>
<td id="S4.T4.3.13.10.2" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">30.77</td>
<td id="S4.T4.3.13.10.3" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">28.29</td>
<td id="S4.T4.3.13.10.4" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">30.73</td>
<td id="S4.T4.3.13.10.5" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">31.33</td>
<td id="S4.T4.3.13.10.6" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:3.0pt;padding-right:3.0pt;">30.12</td>
<td id="S4.T4.3.13.10.7" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S4.T4.3.13.10.7.1" class="ltx_text ltx_font_bold">44.66</span></td>
<td id="S4.T4.3.13.10.8" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S4.T4.3.13.10.8.1" class="ltx_text ltx_font_bold">37.55</span></td>
<td id="S4.T4.3.13.10.9" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S4.T4.3.13.10.9.1" class="ltx_text ltx_font_bold">46.36</span></td>
<td id="S4.T4.3.13.10.10" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">43.47</td>
<td id="S4.T4.3.13.10.11" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S4.T4.3.13.10.11.1" class="ltx_text ltx_font_bold">42.48</span></td>
</tr>
<tr id="S4.T4.3.14.11" class="ltx_tr">
<th id="S4.T4.3.14.11.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r" style="padding-left:3.0pt;padding-right:3.0pt;">Ours</th>
<td id="S4.T4.3.14.11.2" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">30.12</td>
<td id="S4.T4.3.14.11.3" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">28.21</td>
<td id="S4.T4.3.14.11.4" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">30.64</td>
<td id="S4.T4.3.14.11.5" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">30.34</td>
<td id="S4.T4.3.14.11.6" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:3.0pt;padding-right:3.0pt;">29.68</td>
<td id="S4.T4.3.14.11.7" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">44.53</td>
<td id="S4.T4.3.14.11.8" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">33.53</td>
<td id="S4.T4.3.14.11.9" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">44.60</td>
<td id="S4.T4.3.14.11.10" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S4.T4.3.14.11.10.1" class="ltx_text ltx_font_bold">43.52</span></td>
<td id="S4.T4.3.14.11.11" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">40.82</td>
</tr>
<tr id="S4.T4.3.15.12" class="ltx_tr">
<th id="S4.T4.3.15.12.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_b ltx_border_r" style="padding-left:3.0pt;padding-right:3.0pt;">Ours2048</th>
<td id="S4.T4.3.15.12.2" class="ltx_td ltx_align_center ltx_border_b" style="padding-left:3.0pt;padding-right:3.0pt;">32.43</td>
<td id="S4.T4.3.15.12.3" class="ltx_td ltx_align_center ltx_border_b" style="padding-left:3.0pt;padding-right:3.0pt;">28.62</td>
<td id="S4.T4.3.15.12.4" class="ltx_td ltx_align_center ltx_border_b" style="padding-left:3.0pt;padding-right:3.0pt;">31.03</td>
<td id="S4.T4.3.15.12.5" class="ltx_td ltx_align_center ltx_border_b" style="padding-left:3.0pt;padding-right:3.0pt;">32.10</td>
<td id="S4.T4.3.15.12.6" class="ltx_td ltx_align_center ltx_border_b ltx_border_r" style="padding-left:3.0pt;padding-right:3.0pt;">30.85</td>
<td id="S4.T4.3.15.12.7" class="ltx_td ltx_align_center ltx_border_b" style="padding-left:3.0pt;padding-right:3.0pt;">42.75</td>
<td id="S4.T4.3.15.12.8" class="ltx_td ltx_align_center ltx_border_b" style="padding-left:3.0pt;padding-right:3.0pt;">32.98</td>
<td id="S4.T4.3.15.12.9" class="ltx_td ltx_align_center ltx_border_b" style="padding-left:3.0pt;padding-right:3.0pt;">42.88</td>
<td id="S4.T4.3.15.12.10" class="ltx_td ltx_align_center ltx_border_b" style="padding-left:3.0pt;padding-right:3.0pt;">41.30</td>
<td id="S4.T4.3.15.12.11" class="ltx_td ltx_align_center ltx_border_b" style="padding-left:3.0pt;padding-right:3.0pt;">39.34</td>
</tr>
</tbody>
</table>
</figure>
<div id="S4.SS2.p5" class="ltx_para ltx_noindent">
<p id="S4.SS2.p5.1" class="ltx_p">We also provide the results for models with a capacity of 30B or greater at W3G128 in Table <a href="#S4.T5" title="Table 5 ‣ 4.2 Comparing With Other Methods ‣ 4 Experiments ‣ Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a> and W4 in Appendix <a href="#A1.SS3" title="A.3 Other results for large models ‣ Appendix A More results ‣ Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs" class="ltx_ref"><span class="ltx_text ltx_ref_tag">A.3</span></a>. Additionally, we discovered that recovering the sequence length to 512 of the calibration dataset yielded improvements in certain scenarios, and thus we include these results. In summary, our approach achieves comparable performance to GPTQ for the given accuracy task. However, we slightly lag behind GPTQ in terms of ppl tasks.</p>
</div>
<figure id="S4.T5" class="ltx_table">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 5: </span>Average % accuracy(<math id="S4.T5.4.4.m1.1" class="ltx_Math" alttext="\uparrow" display="inline"><semantics id="S4.T5.4.4.m1.1b"><mo stretchy="false" id="S4.T5.4.4.m1.1.1" xref="S4.T5.4.4.m1.1.1.cmml">↑</mo><annotation-xml encoding="MathML-Content" id="S4.T5.4.4.m1.1c"><ci id="S4.T5.4.4.m1.1.1.cmml" xref="S4.T5.4.4.m1.1.1">↑</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.T5.4.4.m1.1d">\uparrow</annotation></semantics></math>) of HellaSwag, WinoGrand, PIQA and LAMBADA and C4 ppl(<math id="S4.T5.5.5.m2.1" class="ltx_Math" alttext="\downarrow" display="inline"><semantics id="S4.T5.5.5.m2.1b"><mo stretchy="false" id="S4.T5.5.5.m2.1.1" xref="S4.T5.5.5.m2.1.1.cmml">↓</mo><annotation-xml encoding="MathML-Content" id="S4.T5.5.5.m2.1c"><ci id="S4.T5.5.5.m2.1.1.cmml" xref="S4.T5.5.5.m2.1.1">↓</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.T5.5.5.m2.1d">\downarrow</annotation></semantics></math>) for LLaMA &amp; OPT with <math id="S4.T5.6.6.m3.1" class="ltx_Math" alttext="\geq" display="inline"><semantics id="S4.T5.6.6.m3.1b"><mo id="S4.T5.6.6.m3.1.1" xref="S4.T5.6.6.m3.1.1.cmml">≥</mo><annotation-xml encoding="MathML-Content" id="S4.T5.6.6.m3.1c"><geq id="S4.T5.6.6.m3.1.1.cmml" xref="S4.T5.6.6.m3.1.1"></geq></annotation-xml><annotation encoding="application/x-tex" id="S4.T5.6.6.m3.1d">\geq</annotation></semantics></math> 30B at W3G128. ”Ours-seq512” indicates that we have modified the sequence length of the calibration dataset from 256 to 512.</figcaption>
<table id="S4.T5.7" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="S4.T5.7.1.1" class="ltx_tr">
<th id="S4.T5.7.1.1.1" class="ltx_td ltx_th ltx_th_row ltx_border_r ltx_border_t"></th>
<td id="S4.T5.7.1.1.2" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" colspan="4">Accuracy</td>
<td id="S4.T5.7.1.1.3" class="ltx_td ltx_align_center ltx_border_t" colspan="4">PPL on C4</td>
</tr>
<tr id="S4.T5.7.2.2" class="ltx_tr">
<th id="S4.T5.7.2.2.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r ltx_border_t">
<span id="S4.T5.7.2.2.1.1" class="ltx_text"></span> Type</th>
<td id="S4.T5.7.2.2.2" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" colspan="2">LLaMA</td>
<td id="S4.T5.7.2.2.3" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" colspan="2">OPT</td>
<td id="S4.T5.7.2.2.4" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" colspan="2">LLaMA</td>
<td id="S4.T5.7.2.2.5" class="ltx_td ltx_align_center ltx_border_t" colspan="2">OPT</td>
</tr>
<tr id="S4.T5.7.3.3" class="ltx_tr">
<th id="S4.T5.7.3.3.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r ltx_border_t">Size</th>
<td id="S4.T5.7.3.3.2" class="ltx_td ltx_align_center ltx_border_t">30b</td>
<td id="S4.T5.7.3.3.3" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">65b</td>
<td id="S4.T5.7.3.3.4" class="ltx_td ltx_align_center ltx_border_t">30b</td>
<td id="S4.T5.7.3.3.5" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">66b</td>
<td id="S4.T5.7.3.3.6" class="ltx_td ltx_align_center ltx_border_t">30b</td>
<td id="S4.T5.7.3.3.7" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">65b</td>
<td id="S4.T5.7.3.3.8" class="ltx_td ltx_align_center ltx_border_t">30b</td>
<td id="S4.T5.7.3.3.9" class="ltx_td ltx_align_center ltx_border_t">66b</td>
</tr>
<tr id="S4.T5.7.4.4" class="ltx_tr">
<th id="S4.T5.7.4.4.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r ltx_border_t">FP16</th>
<td id="S4.T5.7.4.4.2" class="ltx_td ltx_align_center ltx_border_t">73.46</td>
<td id="S4.T5.7.4.4.3" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">75.48</td>
<td id="S4.T5.7.4.4.4" class="ltx_td ltx_align_center ltx_border_t">67.87</td>
<td id="S4.T5.7.4.4.5" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">69.54</td>
<td id="S4.T5.7.4.4.6" class="ltx_td ltx_align_center ltx_border_t">6.13</td>
<td id="S4.T5.7.4.4.7" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">5.98</td>
<td id="S4.T5.7.4.4.8" class="ltx_td ltx_align_center ltx_border_t">11.46</td>
<td id="S4.T5.7.4.4.9" class="ltx_td ltx_align_center ltx_border_t">10.99</td>
</tr>
<tr id="S4.T5.7.5.5" class="ltx_tr">
<th id="S4.T5.7.5.5.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r ltx_border_t">RTN</th>
<td id="S4.T5.7.5.5.2" class="ltx_td ltx_align_center ltx_border_t">72.17</td>
<td id="S4.T5.7.5.5.3" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">73.69</td>
<td id="S4.T5.7.5.5.4" class="ltx_td ltx_align_center ltx_border_t">62.83</td>
<td id="S4.T5.7.5.5.5" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">38.00</td>
<td id="S4.T5.7.5.5.6" class="ltx_td ltx_align_center ltx_border_t">6.85</td>
<td id="S4.T5.7.5.5.7" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">6.52</td>
<td id="S4.T5.7.5.5.8" class="ltx_td ltx_align_center ltx_border_t">30.81</td>
<td id="S4.T5.7.5.5.9" class="ltx_td ltx_align_center ltx_border_t">285.41</td>
</tr>
<tr id="S4.T5.7.6.6" class="ltx_tr">
<th id="S4.T5.7.6.6.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r">GPTQ</th>
<td id="S4.T5.7.6.6.2" class="ltx_td ltx_align_center">72.09</td>
<td id="S4.T5.7.6.6.3" class="ltx_td ltx_align_center ltx_border_r"><span id="S4.T5.7.6.6.3.1" class="ltx_text ltx_font_bold">73.97</span></td>
<td id="S4.T5.7.6.6.4" class="ltx_td ltx_align_center"><span id="S4.T5.7.6.6.4.1" class="ltx_text ltx_font_bold">66.76</span></td>
<td id="S4.T5.7.6.6.5" class="ltx_td ltx_align_center ltx_border_r">67.87</td>
<td id="S4.T5.7.6.6.6" class="ltx_td ltx_align_center">6.80</td>
<td id="S4.T5.7.6.6.7" class="ltx_td ltx_align_center ltx_border_r">6.52</td>
<td id="S4.T5.7.6.6.8" class="ltx_td ltx_align_center"><span id="S4.T5.7.6.6.8.1" class="ltx_text ltx_font_bold">11.74</span></td>
<td id="S4.T5.7.6.6.9" class="ltx_td ltx_align_center"><span id="S4.T5.7.6.6.9.1" class="ltx_text ltx_font_bold">11.87</span></td>
</tr>
<tr id="S4.T5.7.7.7" class="ltx_tr">
<th id="S4.T5.7.7.7.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r">Ours-seq256</th>
<td id="S4.T5.7.7.7.2" class="ltx_td ltx_align_center"><span id="S4.T5.7.7.7.2.1" class="ltx_text ltx_font_bold">72.45</span></td>
<td id="S4.T5.7.7.7.3" class="ltx_td ltx_align_center ltx_border_r">73.71</td>
<td id="S4.T5.7.7.7.4" class="ltx_td ltx_align_center">66.51</td>
<td id="S4.T5.7.7.7.5" class="ltx_td ltx_align_center ltx_border_r"><span id="S4.T5.7.7.7.5.1" class="ltx_text ltx_font_bold">68.00</span></td>
<td id="S4.T5.7.7.7.6" class="ltx_td ltx_align_center">6.83</td>
<td id="S4.T5.7.7.7.7" class="ltx_td ltx_align_center ltx_border_r"><span id="S4.T5.7.7.7.7.1" class="ltx_text ltx_font_bold">6.52</span></td>
<td id="S4.T5.7.7.7.8" class="ltx_td ltx_align_center">13.00</td>
<td id="S4.T5.7.7.7.9" class="ltx_td ltx_align_center">13.34</td>
</tr>
<tr id="S4.T5.7.8.8" class="ltx_tr">
<th id="S4.T5.7.8.8.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_b ltx_border_r">Ours-seq512</th>
<td id="S4.T5.7.8.8.2" class="ltx_td ltx_align_center ltx_border_b">71.95</td>
<td id="S4.T5.7.8.8.3" class="ltx_td ltx_align_center ltx_border_b ltx_border_r">73.78</td>
<td id="S4.T5.7.8.8.4" class="ltx_td ltx_align_center ltx_border_b">66.70</td>
<td id="S4.T5.7.8.8.5" class="ltx_td ltx_align_center ltx_border_b ltx_border_r">67.26</td>
<td id="S4.T5.7.8.8.6" class="ltx_td ltx_align_center ltx_border_b"><span id="S4.T5.7.8.8.6.1" class="ltx_text ltx_font_bold">6.79</span></td>
<td id="S4.T5.7.8.8.7" class="ltx_td ltx_align_center ltx_border_b ltx_border_r">6.53</td>
<td id="S4.T5.7.8.8.8" class="ltx_td ltx_align_center ltx_border_b">12.50</td>
<td id="S4.T5.7.8.8.9" class="ltx_td ltx_align_center ltx_border_b">13.97</td>
</tr>
</tbody>
</table>
</figure>
</section>
<section id="S4.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.3 </span>Block-wise versus Layer-wise </h3>

<div id="S4.SS3.p1" class="ltx_para ltx_noindent">
<p id="S4.SS3.p1.1" class="ltx_p">We examined the effects of layer-wise and block-wise tuning. As explained in Section <a href="#S3.SS1" title="3.1 SignRound ‣ 3 Methodology ‣ Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.1</span></a>, the term ”layer” refers to a linear/convolution layer, while ”block” specifically denotes a transformer block consisting of multiple linear layers. To simplify this evaluation, we set the sequence length to 256 and disable AMP. Based on the below results, block-wise tuning outperformed layer-wise tuning in the majority of scenarios.</p>
</div>
<figure id="S4.T6" class="ltx_table">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 6: </span>Comparing block-wise and layer-wise tuning for around 7B models, the models LLaMA7b, LLaMA7bv2, OPT6.7b, and BLOOM7b1 are denoted by 7b, 7bv2, 6.7b, and 7b1 respectively. The accuracy is the % average accuracy(<math id="S4.T6.3.3.m1.1" class="ltx_Math" alttext="\uparrow" display="inline"><semantics id="S4.T6.3.3.m1.1b"><mo stretchy="false" id="S4.T6.3.3.m1.1.1" xref="S4.T6.3.3.m1.1.1.cmml">↑</mo><annotation-xml encoding="MathML-Content" id="S4.T6.3.3.m1.1c"><ci id="S4.T6.3.3.m1.1.1.cmml" xref="S4.T6.3.3.m1.1.1">↑</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.T6.3.3.m1.1d">\uparrow</annotation></semantics></math>) of HellaSwag, WinoGrand, PIQA and LAMBADA . Perplexity (PPL) (<math id="S4.T6.4.4.m2.1" class="ltx_Math" alttext="\downarrow" display="inline"><semantics id="S4.T6.4.4.m2.1b"><mo stretchy="false" id="S4.T6.4.4.m2.1.1" xref="S4.T6.4.4.m2.1.1.cmml">↓</mo><annotation-xml encoding="MathML-Content" id="S4.T6.4.4.m2.1c"><ci id="S4.T6.4.4.m2.1.1.cmml" xref="S4.T6.4.4.m2.1.1">↓</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.T6.4.4.m2.1d">\downarrow</annotation></semantics></math>) is evaluated using the C4 dataset.</figcaption>
<table id="S4.T6.5" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S4.T6.5.1.1" class="ltx_tr">
<th id="S4.T6.5.1.1.1" class="ltx_td ltx_th ltx_th_column ltx_th_row ltx_border_r ltx_border_t"></th>
<th id="S4.T6.5.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_t" colspan="4">W4</th>
<th id="S4.T6.5.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" colspan="4">W3G128</th>
</tr>
<tr id="S4.T6.5.2.2" class="ltx_tr">
<th id="S4.T6.5.2.2.1" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_th_row ltx_border_r ltx_border_t">Size</th>
<th id="S4.T6.5.2.2.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">7b</th>
<th id="S4.T6.5.2.2.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">7bv2</th>
<th id="S4.T6.5.2.2.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">6.7b</th>
<th id="S4.T6.5.2.2.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_t">7b1</th>
<th id="S4.T6.5.2.2.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">7b</th>
<th id="S4.T6.5.2.2.7" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">7bv2</th>
<th id="S4.T6.5.2.2.8" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">6.7b</th>
<th id="S4.T6.5.2.2.9" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">7b1</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S4.T6.5.3.1" class="ltx_tr">
<th id="S4.T6.5.3.1.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r ltx_border_t">layer-acc-seq256</th>
<td id="S4.T6.5.3.1.2" class="ltx_td ltx_align_center ltx_border_t">67.50</td>
<td id="S4.T6.5.3.1.3" class="ltx_td ltx_align_center ltx_border_t">67.78</td>
<td id="S4.T6.5.3.1.4" class="ltx_td ltx_align_center ltx_border_t">63.46</td>
<td id="S4.T6.5.3.1.5" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">58.72</td>
<td id="S4.T6.5.3.1.6" class="ltx_td ltx_align_center ltx_border_t">65.96</td>
<td id="S4.T6.5.3.1.7" class="ltx_td ltx_align_center ltx_border_t">66.09</td>
<td id="S4.T6.5.3.1.8" class="ltx_td ltx_align_center ltx_border_t"><span id="S4.T6.5.3.1.8.1" class="ltx_text ltx_font_bold">61.60</span></td>
<td id="S4.T6.5.3.1.9" class="ltx_td ltx_align_center ltx_border_t">58.24</td>
</tr>
<tr id="S4.T6.5.4.2" class="ltx_tr">
<th id="S4.T6.5.4.2.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r">block-acc-seq256</th>
<td id="S4.T6.5.4.2.2" class="ltx_td ltx_align_center"><span id="S4.T6.5.4.2.2.1" class="ltx_text ltx_font_bold">67.64</span></td>
<td id="S4.T6.5.4.2.3" class="ltx_td ltx_align_center"><span id="S4.T6.5.4.2.3.1" class="ltx_text ltx_font_bold">67.96</span></td>
<td id="S4.T6.5.4.2.4" class="ltx_td ltx_align_center"><span id="S4.T6.5.4.2.4.1" class="ltx_text ltx_font_bold">64.55</span></td>
<td id="S4.T6.5.4.2.5" class="ltx_td ltx_align_center ltx_border_r"><span id="S4.T6.5.4.2.5.1" class="ltx_text ltx_font_bold">59.08</span></td>
<td id="S4.T6.5.4.2.6" class="ltx_td ltx_align_center"><span id="S4.T6.5.4.2.6.1" class="ltx_text ltx_font_bold">66.31</span></td>
<td id="S4.T6.5.4.2.7" class="ltx_td ltx_align_center"><span id="S4.T6.5.4.2.7.1" class="ltx_text ltx_font_bold">66.63</span></td>
<td id="S4.T6.5.4.2.8" class="ltx_td ltx_align_center">57.76</td>
<td id="S4.T6.5.4.2.9" class="ltx_td ltx_align_center"><span id="S4.T6.5.4.2.9.1" class="ltx_text ltx_font_bold">58.34</span></td>
</tr>
<tr id="S4.T6.5.5.3" class="ltx_tr">
<th id="S4.T6.5.5.3.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r ltx_border_t">layer-c4-ppl-seq256</th>
<td id="S4.T6.5.5.3.2" class="ltx_td ltx_align_center ltx_border_t">8.02</td>
<td id="S4.T6.5.5.3.3" class="ltx_td ltx_align_center ltx_border_t"><span id="S4.T6.5.5.3.3.1" class="ltx_text ltx_font_bold">7.92</span></td>
<td id="S4.T6.5.5.3.4" class="ltx_td ltx_align_center ltx_border_t">13.44</td>
<td id="S4.T6.5.5.3.5" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">15.73</td>
<td id="S4.T6.5.5.3.6" class="ltx_td ltx_align_center ltx_border_t">8.81</td>
<td id="S4.T6.5.5.3.7" class="ltx_td ltx_align_center ltx_border_t"><span id="S4.T6.5.5.3.7.1" class="ltx_text ltx_font_bold">8.69</span></td>
<td id="S4.T6.5.5.3.8" class="ltx_td ltx_align_center ltx_border_t"><span id="S4.T6.5.5.3.8.1" class="ltx_text ltx_font_bold">16.83</span></td>
<td id="S4.T6.5.5.3.9" class="ltx_td ltx_align_center ltx_border_t">16.15</td>
</tr>
<tr id="S4.T6.5.6.4" class="ltx_tr">
<th id="S4.T6.5.6.4.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_b ltx_border_r">block-c4-ppl-seq256</th>
<td id="S4.T6.5.6.4.2" class="ltx_td ltx_align_center ltx_border_b"><span id="S4.T6.5.6.4.2.1" class="ltx_text ltx_font_bold">7.81</span></td>
<td id="S4.T6.5.6.4.3" class="ltx_td ltx_align_center ltx_border_b">8.19</td>
<td id="S4.T6.5.6.4.4" class="ltx_td ltx_align_center ltx_border_b"><span id="S4.T6.5.6.4.4.1" class="ltx_text ltx_font_bold">13.10</span></td>
<td id="S4.T6.5.6.4.5" class="ltx_td ltx_align_center ltx_border_b ltx_border_r"><span id="S4.T6.5.6.4.5.1" class="ltx_text ltx_font_bold">15.71</span></td>
<td id="S4.T6.5.6.4.6" class="ltx_td ltx_align_center ltx_border_b"><span id="S4.T6.5.6.4.6.1" class="ltx_text ltx_font_bold">8.34</span></td>
<td id="S4.T6.5.6.4.7" class="ltx_td ltx_align_center ltx_border_b">10.84</td>
<td id="S4.T6.5.6.4.8" class="ltx_td ltx_align_center ltx_border_b">25.44</td>
<td id="S4.T6.5.6.4.9" class="ltx_td ltx_align_center ltx_border_b"><span id="S4.T6.5.6.4.9.1" class="ltx_text ltx_font_bold">16.05</span></td>
</tr>
</tbody>
</table>
</figure>
</section>
<section id="S4.SS4" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.4 </span>The analysis of hyperparameters sensitivity </h3>

<div id="S4.SS4.p1" class="ltx_para ltx_noindent">
<p id="S4.SS4.p1.1" class="ltx_p">We conducted a hyperparameters sensitivity analysis, the results of which are summarized in Table <a href="#S4.T7" title="Table 7 ‣ 4.4 The analysis of hyperparameters sensitivity ‣ 4 Experiments ‣ Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs" class="ltx_ref"><span class="ltx_text ltx_ref_tag">7</span></a>. In the ”steps100” configuration, we used 100 steps, and a learning rate of 1e-2.
In the ”lr4e-3” configuration, we set the learning rate to 4e-3. We also changed the sequence length of the calibration dataset from 512 to 2048, denoted by ”seq2048”. Please note that all other hyperparameters not mentioned in each configuration were kept the same as the default configurations, as detailed in Section <a href="#S4.SS1" title="4.1 Experimental Settings ‣ 4 Experiments ‣ Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4.1</span></a>. Overall, our method exhibits robustness to hyperparameters in common sense reasoning tasks, with the exception of the perplexity of LLaMA-7b-v2. However, we did discover that certain hyperparameters, such as the sequence length of the calibration dataset, can significantly impact performance in some scenarios, as demonstrated in Table <a href="#S4.T4" title="Table 4 ‣ 4.2 Comparing With Other Methods ‣ 4 Experiments ‣ Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a> and <a href="#S4.T5" title="Table 5 ‣ 4.2 Comparing With Other Methods ‣ 4 Experiments ‣ Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a>.</p>
</div>
<figure id="S4.T7" class="ltx_table">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 7: </span>Hyperparameter sensitivity analysis, the models LLaMA7b, LLaMA7bv2, OPT6.7b, and BLOOM7b1 are denoted by 7b, 7bv2, 6.7b, and 7b1 respectively. The accuracy is the % average accuracy(<math id="S4.T7.3.3.m1.1" class="ltx_Math" alttext="\uparrow" display="inline"><semantics id="S4.T7.3.3.m1.1b"><mo stretchy="false" id="S4.T7.3.3.m1.1.1" xref="S4.T7.3.3.m1.1.1.cmml">↑</mo><annotation-xml encoding="MathML-Content" id="S4.T7.3.3.m1.1c"><ci id="S4.T7.3.3.m1.1.1.cmml" xref="S4.T7.3.3.m1.1.1">↑</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.T7.3.3.m1.1d">\uparrow</annotation></semantics></math>) of HellaSwag, WinoGrand, PIQA and LAMBADA . Perplexity (PPL) (<math id="S4.T7.4.4.m2.1" class="ltx_Math" alttext="\downarrow" display="inline"><semantics id="S4.T7.4.4.m2.1b"><mo stretchy="false" id="S4.T7.4.4.m2.1.1" xref="S4.T7.4.4.m2.1.1.cmml">↓</mo><annotation-xml encoding="MathML-Content" id="S4.T7.4.4.m2.1c"><ci id="S4.T7.4.4.m2.1.1.cmml" xref="S4.T7.4.4.m2.1.1">↓</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.T7.4.4.m2.1d">\downarrow</annotation></semantics></math>) is evaluated using the C4 dataset.
</figcaption>
<table id="S4.T7.5" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S4.T7.5.1.1" class="ltx_tr">
<th id="S4.T7.5.1.1.1" class="ltx_td ltx_th ltx_th_column ltx_th_row ltx_border_r ltx_border_t"></th>
<th id="S4.T7.5.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_t" colspan="4">Accuracy</th>
<th id="S4.T7.5.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" colspan="4">PPL on C4</th>
</tr>
<tr id="S4.T7.5.2.2" class="ltx_tr">
<th id="S4.T7.5.2.2.1" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_th_row ltx_border_r ltx_border_t">Size</th>
<th id="S4.T7.5.2.2.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">7b</th>
<th id="S4.T7.5.2.2.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">7bv2</th>
<th id="S4.T7.5.2.2.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">6.7b</th>
<th id="S4.T7.5.2.2.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_t">7b1</th>
<th id="S4.T7.5.2.2.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">7b</th>
<th id="S4.T7.5.2.2.7" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">7bv2</th>
<th id="S4.T7.5.2.2.8" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">6.7b</th>
<th id="S4.T7.5.2.2.9" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">7b1</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S4.T7.5.3.1" class="ltx_tr">
<th id="S4.T7.5.3.1.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r ltx_border_t">steps100</th>
<td id="S4.T7.5.3.1.2" class="ltx_td ltx_align_center ltx_border_t">67.53</td>
<td id="S4.T7.5.3.1.3" class="ltx_td ltx_align_center ltx_border_t">67.76</td>
<td id="S4.T7.5.3.1.4" class="ltx_td ltx_align_center ltx_border_t"><span id="S4.T7.5.3.1.4.1" class="ltx_text ltx_font_bold">64.64</span></td>
<td id="S4.T7.5.3.1.5" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">58.76</td>
<td id="S4.T7.5.3.1.6" class="ltx_td ltx_align_center ltx_border_t">7.93</td>
<td id="S4.T7.5.3.1.7" class="ltx_td ltx_align_center ltx_border_t"><span id="S4.T7.5.3.1.7.1" class="ltx_text ltx_font_bold">7.83</span></td>
<td id="S4.T7.5.3.1.8" class="ltx_td ltx_align_center ltx_border_t">13.12</td>
<td id="S4.T7.5.3.1.9" class="ltx_td ltx_align_center ltx_border_t">15.71</td>
</tr>
<tr id="S4.T7.5.4.2" class="ltx_tr">
<th id="S4.T7.5.4.2.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r">lr4e-3</th>
<td id="S4.T7.5.4.2.2" class="ltx_td ltx_align_center">68.01</td>
<td id="S4.T7.5.4.2.3" class="ltx_td ltx_align_center">67.57</td>
<td id="S4.T7.5.4.2.4" class="ltx_td ltx_align_center">64.57</td>
<td id="S4.T7.5.4.2.5" class="ltx_td ltx_align_center ltx_border_r">59.47</td>
<td id="S4.T7.5.4.2.6" class="ltx_td ltx_align_center">7.81</td>
<td id="S4.T7.5.4.2.7" class="ltx_td ltx_align_center">10.29</td>
<td id="S4.T7.5.4.2.8" class="ltx_td ltx_align_center">13.09</td>
<td id="S4.T7.5.4.2.9" class="ltx_td ltx_align_center"><span id="S4.T7.5.4.2.9.1" class="ltx_text ltx_font_bold">15.66</span></td>
</tr>
<tr id="S4.T7.5.5.3" class="ltx_tr">
<th id="S4.T7.5.5.3.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r">seq2048</th>
<td id="S4.T7.5.5.3.2" class="ltx_td ltx_align_center"><span id="S4.T7.5.5.3.2.1" class="ltx_text ltx_font_bold">68.11</span></td>
<td id="S4.T7.5.5.3.3" class="ltx_td ltx_align_center"><span id="S4.T7.5.5.3.3.1" class="ltx_text ltx_font_bold">67.79</span></td>
<td id="S4.T7.5.5.3.4" class="ltx_td ltx_align_center">64.32</td>
<td id="S4.T7.5.5.3.5" class="ltx_td ltx_align_center ltx_border_r">59.39</td>
<td id="S4.T7.5.5.3.6" class="ltx_td ltx_align_center"><span id="S4.T7.5.5.3.6.1" class="ltx_text ltx_font_bold">7.76</span></td>
<td id="S4.T7.5.5.3.7" class="ltx_td ltx_align_center">9.97</td>
<td id="S4.T7.5.5.3.8" class="ltx_td ltx_align_center"><span id="S4.T7.5.5.3.8.1" class="ltx_text ltx_font_bold">13.06</span></td>
<td id="S4.T7.5.5.3.9" class="ltx_td ltx_align_center"><span id="S4.T7.5.5.3.9.1" class="ltx_text ltx_font_bold">15.66</span></td>
</tr>
<tr id="S4.T7.5.6.4" class="ltx_tr">
<th id="S4.T7.5.6.4.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_b ltx_border_r">default</th>
<td id="S4.T7.5.6.4.2" class="ltx_td ltx_align_center ltx_border_b">68.05</td>
<td id="S4.T7.5.6.4.3" class="ltx_td ltx_align_center ltx_border_b">67.74</td>
<td id="S4.T7.5.6.4.4" class="ltx_td ltx_align_center ltx_border_b">64.34</td>
<td id="S4.T7.5.6.4.5" class="ltx_td ltx_align_center ltx_border_b ltx_border_r"><span id="S4.T7.5.6.4.5.1" class="ltx_text ltx_font_bold">59.52</span></td>
<td id="S4.T7.5.6.4.6" class="ltx_td ltx_align_center ltx_border_b">7.84</td>
<td id="S4.T7.5.6.4.7" class="ltx_td ltx_align_center ltx_border_b">11.20</td>
<td id="S4.T7.5.6.4.8" class="ltx_td ltx_align_center ltx_border_b">13.08</td>
<td id="S4.T7.5.6.4.9" class="ltx_td ltx_align_center ltx_border_b">15.67</td>
</tr>
</tbody>
</table>
</figure>
</section>
<section id="S4.SS5" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.5 </span>The Analysis of Gradients and Their Effects on Rounding</h3>

<figure id="S4.F2" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="S4.F2.1" class="ltx_figure ltx_figure_panel ltx_minipage ltx_align_center ltx_align_middle" style="width:195.1pt;"><img src="/html/2309.05516/assets/figs-0925/llama-7b-hf-ratio-0925.png" id="S4.F2.1.g1" class="ltx_graphics ltx_img_landscape" width="598" height="370" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering">LLaMA-7B</figcaption>
</figure>
</div>
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="S4.F2.2" class="ltx_figure ltx_figure_panel ltx_minipage ltx_align_center ltx_align_middle" style="width:195.1pt;"><img src="/html/2309.05516/assets/figs-0925/Llama-2-7b-hf-ratio-0925.png" id="S4.F2.2.g1" class="ltx_graphics ltx_img_landscape" width="598" height="370" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering">LLaMA-7B-V2</figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="S4.F2.3" class="ltx_figure ltx_figure_panel ltx_minipage ltx_align_center ltx_align_middle" style="width:195.1pt;"><img src="/html/2309.05516/assets/figs-0925/opt-6.7b-ratio-0925.png" id="S4.F2.3.g1" class="ltx_graphics ltx_img_landscape" width="598" height="365" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering">OPT-6.7B</figcaption>
</figure>
</div>
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="S4.F2.4" class="ltx_figure ltx_figure_panel ltx_minipage ltx_align_center ltx_align_middle" style="width:195.1pt;"><img src="/html/2309.05516/assets/figs-0925/bloom-7b1-ratio-0925.png" id="S4.F2.4.g1" class="ltx_graphics ltx_img_landscape" width="598" height="370" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering">BLOOM-7B1</figcaption>
</figure>
</div>
</div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 2: </span>The impact of the rounding value introduced by the <math id="S4.F2.6.m1.1" class="ltx_Math" alttext="V" display="inline"><semantics id="S4.F2.6.m1.1b"><mi id="S4.F2.6.m1.1.1" xref="S4.F2.6.m1.1.1.cmml">V</mi><annotation-xml encoding="MathML-Content" id="S4.F2.6.m1.1c"><ci id="S4.F2.6.m1.1.1.cmml" xref="S4.F2.6.m1.1.1">𝑉</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.F2.6.m1.1d">V</annotation></semantics></math> in Eq. <a href="#S3.E2" title="In 3 Methodology ‣ Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a></figcaption>
</figure>
<div id="S4.SS5.p1" class="ltx_para ltx_noindent">
<p id="S4.SS5.p1.1" class="ltx_p">In this analysis, we dive into the distribution of the magnitude of <math id="S4.SS5.p1.1.m1.1" class="ltx_Math" alttext="V" display="inline"><semantics id="S4.SS5.p1.1.m1.1a"><mi id="S4.SS5.p1.1.m1.1.1" xref="S4.SS5.p1.1.m1.1.1.cmml">V</mi><annotation-xml encoding="MathML-Content" id="S4.SS5.p1.1.m1.1b"><ci id="S4.SS5.p1.1.m1.1.1.cmml" xref="S4.SS5.p1.1.m1.1.1">𝑉</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS5.p1.1.m1.1c">V</annotation></semantics></math> in Eq. <a href="#S3.E2" title="In 3 Methodology ‣ Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a> and its impact on rounding values across approximately 7 billion models at W4. The visual representations of these distributions are provided in Appendix <a href="#A2" title="Appendix B Visualization of V ‣ Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs" class="ltx_ref"><span class="ltx_text ltx_ref_tag">B</span></a>. Our investigation reveals that the majority of V values are concentrated within the range of [-0.3, 0.3]. Additionally, we observe an interesting pattern in the distribution of V across different layers. The middle layers exhibit a more tightly clustered distribution compared to the other layers. This observation aligns with the common understanding that the head and tail layers tend to be more sensitive to compression, while the middle layers are relatively more robust.</p>
</div>
<div id="S4.SS5.p2" class="ltx_para ltx_noindent">
<p id="S4.SS5.p2.1" class="ltx_p">Figure <a href="#S4.F2" title="Figure 2 ‣ 4.5 The Analysis of Gradients and Their Effects on Rounding ‣ 4 Experiments ‣ Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a> illustrates the impact of the rounding value introduced by the <math id="S4.SS5.p2.1.m1.1" class="ltx_Math" alttext="V" display="inline"><semantics id="S4.SS5.p2.1.m1.1a"><mi id="S4.SS5.p2.1.m1.1.1" xref="S4.SS5.p2.1.m1.1.1.cmml">V</mi><annotation-xml encoding="MathML-Content" id="S4.SS5.p2.1.m1.1b"><ci id="S4.SS5.p2.1.m1.1.1.cmml" xref="S4.SS5.p2.1.m1.1.1">𝑉</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS5.p2.1.m1.1c">V</annotation></semantics></math> in Eq. <a href="#S3.E2" title="In 3 Methodology ‣ Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a> for models around 7B at W4. The red line represents ”up rounding”, indicating that while RTN rounds the value to the floor, SignRound changes it to the ceiling. Conversely, the green line represents ”down rounding” indicating that while RTN rounds the value to the ceiling, SignRound changes it to the floor. It is worth noting that SignRound modifies only a small percentage of weight rounding values for each of the four models, namely 5.27%, 5.29%, 4.14%, and 4.10%.</p>
</div>
<div id="S4.SS5.p3" class="ltx_para ltx_noindent">
<p id="S4.SS5.p3.1" class="ltx_p">We were also intrigued by the possible correlation between rounding and activation, as previous research has shown that keeping only 0.1%-1% of the channels corresponding to larger activation can significantly improve the quantized performance in AWQ <cite class="ltx_cite ltx_citemacro_citep">(Lin et al., <a href="#bib.bib18" title="" class="ltx_ref">2023</a>)</cite>. We shown the result in Appendix <a href="#A3" title="Appendix C Correction between SignRound and salient activation channels ‣ Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs" class="ltx_ref"><span class="ltx_text ltx_ref_tag">C</span></a>.</p>
</div>
</section>
</section>
<section id="S5" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">5 </span>Conclusions and Limitations</h2>

<div id="S5.p1" class="ltx_para ltx_noindent">
<p id="S5.p1.1" class="ltx_p">In this paper, we present a highly effective and concise approach to optimize the weight rounding task. Our method, SignRound, leverages lightweight block-wise tuning using signed gradient descent, achieving remarkable results within a mere 400 steps. Extensive experiments demonstrate the superior performance of our approach. As part of our future work, we plan to apply our approach to more diverse LLM models (e.g., Code LLaMA <cite class="ltx_cite ltx_citemacro_citep">(Rozière et al., <a href="#bib.bib28" title="" class="ltx_ref">2023</a>)</cite>, LLaMA v2 Chat <cite class="ltx_cite ltx_citemacro_citep">(Touvron et al., <a href="#bib.bib33" title="" class="ltx_ref">2023b</a>)</cite>), and contribute our recipes and implementations to the open source community. On the other hand, although our method is generally effective, there are a few outliers in certain scenarios, where we plan to mitigate the issue by fine-tuning the hyperparameters.</p>
</div>
</section>
<section id="bib" class="ltx_bibliography">
<h2 class="ltx_title ltx_title_bibliography">References</h2>

<ul class="ltx_biblist">
<li id="bib.bib1" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Bengio et al. (2013)</span>
<span class="ltx_bibblock">
Yoshua Bengio, Nicholas Léonard, and Aaron Courville.

</span>
<span class="ltx_bibblock">Estimating or propagating gradients through stochastic neurons for
conditional computation.

</span>
<span class="ltx_bibblock"><em id="bib.bib1.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:1308.3432</em>, 2013.

</span>
</li>
<li id="bib.bib2" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Bisk et al. (2020)</span>
<span class="ltx_bibblock">
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al.

</span>
<span class="ltx_bibblock">Piqa: Reasoning about physical commonsense in natural language.

</span>
<span class="ltx_bibblock">In <em id="bib.bib2.1.1" class="ltx_emph ltx_font_italic">Proceedings of the AAAI conference on artificial
intelligence</em>, volume 34, pp.  7432–7439, 2020.

</span>
</li>
<li id="bib.bib3" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Bondarenko et al. (2023)</span>
<span class="ltx_bibblock">
Yelysei Bondarenko, Markus Nagel, and Tijmen Blankevoort.

</span>
<span class="ltx_bibblock">Quantizable transformers: Removing outliers by helping attention
heads do nothing.

</span>
<span class="ltx_bibblock"><em id="bib.bib3.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2306.12929</em>, 2023.

</span>
</li>
<li id="bib.bib4" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Dettmers et al. (2022)</span>
<span class="ltx_bibblock">
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer.

</span>
<span class="ltx_bibblock">Llm. int8 (): 8-bit matrix multiplication for transformers at scale.

</span>
<span class="ltx_bibblock"><em id="bib.bib4.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2208.07339</em>, 2022.

</span>
</li>
<li id="bib.bib5" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Dettmers et al. (2023)</span>
<span class="ltx_bibblock">
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer.

</span>
<span class="ltx_bibblock">Qlora: Efficient finetuning of quantized llms.

</span>
<span class="ltx_bibblock"><em id="bib.bib5.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2305.14314</em>, 2023.

</span>
</li>
<li id="bib.bib6" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Esser et al. (2019)</span>
<span class="ltx_bibblock">
Steven K Esser, Jeffrey L McKinstry, Deepika Bablani, Rathinakumar Appuswamy,
and Dharmendra S Modha.

</span>
<span class="ltx_bibblock">Learned step size quantization.

</span>
<span class="ltx_bibblock"><em id="bib.bib6.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:1902.08153</em>, 2019.

</span>
</li>
<li id="bib.bib7" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Frantar &amp; Alistarh (2022)</span>
<span class="ltx_bibblock">
Elias Frantar and Dan Alistarh.

</span>
<span class="ltx_bibblock">Optimal brain compression: A framework for accurate post-training
quantization and pruning.

</span>
<span class="ltx_bibblock"><em id="bib.bib7.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2208.11580</em>, 2022.

</span>
</li>
<li id="bib.bib8" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Frantar et al. (2022)</span>
<span class="ltx_bibblock">
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh.

</span>
<span class="ltx_bibblock">Gptq: Accurate post-training quantization for generative pre-trained
transformers.

</span>
<span class="ltx_bibblock"><em id="bib.bib8.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2210.17323</em>, 2022.

</span>
</li>
<li id="bib.bib9" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Gao et al. (2021)</span>
<span class="ltx_bibblock">
Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles
Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff,
Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang,
and Andy Zou.

</span>
<span class="ltx_bibblock">A framework for few-shot language model evaluation, September 2021.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.5281/zenodo.5371628" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.5281/zenodo.5371628</a>.

</span>
</li>
<li id="bib.bib10" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hassibi et al. (1993)</span>
<span class="ltx_bibblock">
Babak Hassibi, David G Stork, and Gregory J Wolff.

</span>
<span class="ltx_bibblock">Optimal brain surgeon and general network pruning.

</span>
<span class="ltx_bibblock">In <em id="bib.bib10.1.1" class="ltx_emph ltx_font_italic">IEEE international conference on neural networks</em>, pp. 293–299. IEEE, 1993.

</span>
</li>
<li id="bib.bib11" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hu et al. (2021)</span>
<span class="ltx_bibblock">
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean
Wang, Lu Wang, and Weizhu Chen.

</span>
<span class="ltx_bibblock">Lora: Low-rank adaptation of large language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib11.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2106.09685</em>, 2021.

</span>
</li>
<li id="bib.bib12" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Kim et al. (2023)</span>
<span class="ltx_bibblock">
Sehoon Kim, Coleman Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen,
Michael W Mahoney, and Kurt Keutzer.

</span>
<span class="ltx_bibblock">Squeezellm: Dense-and-sparse quantization.

</span>
<span class="ltx_bibblock"><em id="bib.bib12.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2306.07629</em>, 2023.

</span>
</li>
<li id="bib.bib13" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lee et al. (2021)</span>
<span class="ltx_bibblock">
Jung Hyun Lee, Jihun Yun, Sung Ju Hwang, and Eunho Yang.

</span>
<span class="ltx_bibblock">Cluster-promoting quantization with bit-drop for minimizing network
quantization loss.

</span>
<span class="ltx_bibblock">In <em id="bib.bib13.1.1" class="ltx_emph ltx_font_italic">Proceedings of the IEEE/CVF International Conference on
Computer Vision</em>, pp.  5370–5379, 2021.

</span>
</li>
<li id="bib.bib14" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lee et al. (2023)</span>
<span class="ltx_bibblock">
Jung Hyun Lee, Jeonghoon Kim, Se Jung Kwon, and Dongsoo Lee.

</span>
<span class="ltx_bibblock">Flexround: Learnable rounding based on element-wise division for
post-training quantization.

</span>
<span class="ltx_bibblock"><em id="bib.bib14.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2306.00317</em>, 2023.

</span>
</li>
<li id="bib.bib15" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Li et al. (2023)</span>
<span class="ltx_bibblock">
Xiuxian Li, Kuo-Yi Lin, Li Li, Yiguang Hong, and Jie Chen.

</span>
<span class="ltx_bibblock">On faster convergence of scaled sign gradient descent.

</span>
<span class="ltx_bibblock"><em id="bib.bib15.1.1" class="ltx_emph ltx_font_italic">IEEE Transactions on Industrial Informatics</em>, 2023.

</span>
</li>
<li id="bib.bib16" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Li et al. (2021)</span>
<span class="ltx_bibblock">
Yuhang Li, Ruihao Gong, Xu Tan, Yang Yang, Peng Hu, Qi Zhang, Fengwei Yu, Wei
Wang, and Shi Gu.

</span>
<span class="ltx_bibblock">Brecq: Pushing the limit of post-training quantization by block
reconstruction.

</span>
<span class="ltx_bibblock"><em id="bib.bib16.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2102.05426</em>, 2021.

</span>
</li>
<li id="bib.bib17" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Li et al. (2022)</span>
<span class="ltx_bibblock">
Zhengyi Li, Cong Guo, Zhanda Zhu, Yangjie Zhou, Yuxian Qiu, Xiaotian Gao,
Jingwen Leng, and Minyi Guo.

</span>
<span class="ltx_bibblock">Efficient activation quantization via adaptive rounding border for
post-training quantization.

</span>
<span class="ltx_bibblock"><em id="bib.bib17.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2208.11945</em>, 2022.

</span>
</li>
<li id="bib.bib18" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lin et al. (2023)</span>
<span class="ltx_bibblock">
Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Xingyu Dang, and Song Han.

</span>
<span class="ltx_bibblock">Awq: Activation-aware weight quantization for llm compression and
acceleration.

</span>
<span class="ltx_bibblock"><em id="bib.bib18.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2306.00978</em>, 2023.

</span>
</li>
<li id="bib.bib19" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Liu et al. (2023)</span>
<span class="ltx_bibblock">
Zechun Liu, Barlas Oguz, Changsheng Zhao, Ernie Chang, Pierre Stock, Yashar
Mehdad, Yangyang Shi, Raghuraman Krishnamoorthi, and Vikas Chandra.

</span>
<span class="ltx_bibblock">Llm-qat: Data-free quantization aware training for large language
models.

</span>
<span class="ltx_bibblock"><em id="bib.bib19.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2305.17888</em>, 2023.

</span>
</li>
<li id="bib.bib20" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Liu et al. (2021)</span>
<span class="ltx_bibblock">
Zhenhua Liu, Yunhe Wang, Kai Han, Wei Zhang, Siwei Ma, and Wen Gao.

</span>
<span class="ltx_bibblock">Post-training quantization for vision transformer.

</span>
<span class="ltx_bibblock"><em id="bib.bib20.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>,
34:28092–28103, 2021.

</span>
</li>
<li id="bib.bib21" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Merity et al. (2016)</span>
<span class="ltx_bibblock">
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher.

</span>
<span class="ltx_bibblock">Pointer sentinel mixture models.

</span>
<span class="ltx_bibblock"><em id="bib.bib21.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:1609.07843</em>, 2016.

</span>
</li>
<li id="bib.bib22" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Nagel et al. (2019)</span>
<span class="ltx_bibblock">
Markus Nagel, Mart van Baalen, Tijmen Blankevoort, and Max Welling.

</span>
<span class="ltx_bibblock">Data-free quantization through weight equalization and bias
correction.

</span>
<span class="ltx_bibblock">In <em id="bib.bib22.1.1" class="ltx_emph ltx_font_italic">Proceedings of the IEEE/CVF International Conference on
Computer Vision</em>, pp.  1325–1334, 2019.

</span>
</li>
<li id="bib.bib23" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Nagel et al. (2020)</span>
<span class="ltx_bibblock">
Markus Nagel, Rana Ali Amjad, Mart Van Baalen, Christos Louizos, and Tijmen
Blankevoort.

</span>
<span class="ltx_bibblock">Up or down? adaptive rounding for post-training quantization.

</span>
<span class="ltx_bibblock">In <em id="bib.bib23.1.1" class="ltx_emph ltx_font_italic">International Conference on Machine Learning</em>, pp. 7197–7206. PMLR, 2020.

</span>
</li>
<li id="bib.bib24" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">(24)</span>
<span class="ltx_bibblock">
OpenAI.

</span>
<span class="ltx_bibblock">Openai: Chatgpt.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://openai.com/blog/chatgpt" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://openai.com/blog/chatgpt</a>.

</span>
</li>
<li id="bib.bib25" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Paperno et al. (2016)</span>
<span class="ltx_bibblock">
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham,
Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel
Fernández.

</span>
<span class="ltx_bibblock">The lambada dataset: Word prediction requiring a broad discourse
context.

</span>
<span class="ltx_bibblock"><em id="bib.bib25.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:1606.06031</em>, 2016.

</span>
</li>
<li id="bib.bib26" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Park et al. (2022)</span>
<span class="ltx_bibblock">
Gunho Park, Baeseong Park, Se Jung Kwon, Byeongwook Kim, Youngjoo Lee, and
Dongsoo Lee.

</span>
<span class="ltx_bibblock">nuqmm: Quantized matmul for efficient inference of large-scale
generative language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib26.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2206.09557</em>, 2022.

</span>
</li>
<li id="bib.bib27" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Raffel et al. (2020)</span>
<span class="ltx_bibblock">
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael
Matena, Yanqi Zhou, Wei Li, and Peter J Liu.

</span>
<span class="ltx_bibblock">Exploring the limits of transfer learning with a unified text-to-text
transformer.

</span>
<span class="ltx_bibblock"><em id="bib.bib27.1.1" class="ltx_emph ltx_font_italic">The Journal of Machine Learning Research</em>, 21(1):5485–5551, 2020.

</span>
</li>
<li id="bib.bib28" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Rozière et al. (2023)</span>
<span class="ltx_bibblock">
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat,
Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin,
et al.

</span>
<span class="ltx_bibblock">Code llama: Open foundation models for code.

</span>
<span class="ltx_bibblock"><em id="bib.bib28.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2308.12950</em>, 2023.

</span>
</li>
<li id="bib.bib29" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Safaryan &amp; Richtárik (2021)</span>
<span class="ltx_bibblock">
Mher Safaryan and Peter Richtárik.

</span>
<span class="ltx_bibblock">Stochastic sign descent methods: New algorithms and better theory.

</span>
<span class="ltx_bibblock">In <em id="bib.bib29.1.1" class="ltx_emph ltx_font_italic">International Conference on Machine Learning</em>, pp. 9224–9234. PMLR, 2021.

</span>
</li>
<li id="bib.bib30" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Sakaguchi et al. (2021)</span>
<span class="ltx_bibblock">
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi.

</span>
<span class="ltx_bibblock">Winogrande: An adversarial winograd schema challenge at scale.

</span>
<span class="ltx_bibblock"><em id="bib.bib30.1.1" class="ltx_emph ltx_font_italic">Communications of the ACM</em>, 64(9):99–106,
2021.

</span>
</li>
<li id="bib.bib31" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Scao et al. (2022)</span>
<span class="ltx_bibblock">
Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić,
Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François
Yvon, Matthias Gallé, et al.

</span>
<span class="ltx_bibblock">Bloom: A 176b-parameter open-access multilingual language model.

</span>
<span class="ltx_bibblock"><em id="bib.bib31.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2211.05100</em>, 2022.

</span>
</li>
<li id="bib.bib32" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Touvron et al. (2023a)</span>
<span class="ltx_bibblock">
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne
Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric
Hambro, Faisal Azhar, et al.

</span>
<span class="ltx_bibblock">Llama: Open and efficient foundation language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib32.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2302.13971</em>, 2023a.

</span>
</li>
<li id="bib.bib33" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Touvron et al. (2023b)</span>
<span class="ltx_bibblock">
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine
Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale,
et al.

</span>
<span class="ltx_bibblock">Llama 2: Open foundation and fine-tuned chat models.

</span>
<span class="ltx_bibblock"><em id="bib.bib33.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2307.09288</em>, 2023b.

</span>
</li>
<li id="bib.bib34" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wang et al. (2019)</span>
<span class="ltx_bibblock">
Kuan Wang, Zhijian Liu, Yujun Lin, Ji Lin, and Song Han.

</span>
<span class="ltx_bibblock">Haq: Hardware-aware automated quantization with mixed precision.

</span>
<span class="ltx_bibblock">In <em id="bib.bib34.1.1" class="ltx_emph ltx_font_italic">Proceedings of the IEEE/CVF conference on computer vision
and pattern recognition</em>, pp.  8612–8620, 2019.

</span>
</li>
<li id="bib.bib35" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wei et al. (2022a)</span>
<span class="ltx_bibblock">
Xiuying Wei, Ruihao Gong, Yuhang Li, Xianglong Liu, and Fengwei Yu.

</span>
<span class="ltx_bibblock">Qdrop: randomly dropping quantization for extremely low-bit
post-training quantization.

</span>
<span class="ltx_bibblock"><em id="bib.bib35.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2203.05740</em>, 2022a.

</span>
</li>
<li id="bib.bib36" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wei et al. (2022b)</span>
<span class="ltx_bibblock">
Xiuying Wei, Yunchen Zhang, Xiangguo Zhang, Ruihao Gong, Shanghang Zhang,
Qi Zhang, Fengwei Yu, and Xianglong Liu.

</span>
<span class="ltx_bibblock">Outlier suppression: Pushing the limit of low-bit transformer
language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib36.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>,
35:17402–17414, 2022b.

</span>
</li>
<li id="bib.bib37" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wei et al. (2023)</span>
<span class="ltx_bibblock">
Xiuying Wei, Yunchen Zhang, Yuhang Li, Xiangguo Zhang, Ruihao Gong, Jinyang
Guo, and Xianglong Liu.

</span>
<span class="ltx_bibblock">Outlier suppression+: Accurate quantization of large language models
by equivalent and optimal shifting and scaling.

</span>
<span class="ltx_bibblock"><em id="bib.bib37.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2304.09145</em>, 2023.

</span>
</li>
<li id="bib.bib38" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Xiao et al. (2022)</span>
<span class="ltx_bibblock">
Guangxuan Xiao, Ji Lin, Mickael Seznec, Julien Demouth, and Song Han.

</span>
<span class="ltx_bibblock">Smoothquant: Accurate and efficient post-training quantization for
large language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib38.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2211.10438</em>, 2022.

</span>
</li>
<li id="bib.bib39" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Xiao et al. (2023)</span>
<span class="ltx_bibblock">
Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han.

</span>
<span class="ltx_bibblock">Smoothquant: Accurate and efficient post-training quantization for
large language models.

</span>
<span class="ltx_bibblock">In <em id="bib.bib39.1.1" class="ltx_emph ltx_font_italic">International Conference on Machine Learning</em>, pp. 38087–38099. PMLR, 2023.

</span>
</li>
<li id="bib.bib40" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Yao et al. (2021)</span>
<span class="ltx_bibblock">
Zhewei Yao, Zhen Dong, Zhangcheng Zheng, Amir Gholami, Jiali Yu, Eric Tan,
Leyuan Wang, Qijing Huang, Yida Wang, Michael Mahoney, et al.

</span>
<span class="ltx_bibblock">Hawq-v3: Dyadic neural network quantization.

</span>
<span class="ltx_bibblock">In <em id="bib.bib40.1.1" class="ltx_emph ltx_font_italic">International Conference on Machine Learning</em>, pp. 11875–11886. PMLR, 2021.

</span>
</li>
<li id="bib.bib41" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Yao et al. (2023)</span>
<span class="ltx_bibblock">
Zhewei Yao, Xiaoxia Wu, Cheng Li, Stephen Youn, and Yuxiong He.

</span>
<span class="ltx_bibblock">Zeroquant-v2: Exploring post-training quantization in llms from
comprehensive study to low rank compensation.

</span>
<span class="ltx_bibblock"><em id="bib.bib41.1.1" class="ltx_emph ltx_font_italic">arXiv:2303.08302</em>, 2023.

</span>
</li>
<li id="bib.bib42" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Yuan et al. (2023)</span>
<span class="ltx_bibblock">
Zhihang Yuan, Lin Niu, Jiawei Liu, Wenyu Liu, Xinggang Wang, Yuzhang Shang,
Guangyu Sun, Qiang Wu, Jiaxiang Wu, and Bingzhe Wu.

</span>
<span class="ltx_bibblock">Rptq: Reorder-based post-training quantization for large language
models.

</span>
<span class="ltx_bibblock"><em id="bib.bib42.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2304.01089</em>, 2023.

</span>
</li>
<li id="bib.bib43" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Yvinec et al. (2023)</span>
<span class="ltx_bibblock">
Edouard Yvinec, Arnaud Dapogny, Matthieu Cord, and Kevin Bailly.

</span>
<span class="ltx_bibblock">Spiq: Data-free per-channel static input quantization.

</span>
<span class="ltx_bibblock">In <em id="bib.bib43.1.1" class="ltx_emph ltx_font_italic">Proceedings of the IEEE/CVF Winter Conference on
Applications of Computer Vision</em>, pp.  3869–3878, 2023.

</span>
</li>
<li id="bib.bib44" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zellers et al. (2019)</span>
<span class="ltx_bibblock">
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi.

</span>
<span class="ltx_bibblock">Hellaswag: Can a machine really finish your sentence?

</span>
<span class="ltx_bibblock"><em id="bib.bib44.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:1905.07830</em>, 2019.

</span>
</li>
<li id="bib.bib45" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhang et al. (2022)</span>
<span class="ltx_bibblock">
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui
Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al.

</span>
<span class="ltx_bibblock">Opt: Open pre-trained transformer language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib45.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2205.01068</em>, 2022.

</span>
</li>
<li id="bib.bib46" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhuang et al. (2021)</span>
<span class="ltx_bibblock">
Bohan Zhuang, Mingkui Tan, Jing Liu, Lingqiao Liu, Ian Reid, and Chunhua Shen.

</span>
<span class="ltx_bibblock">Effective training of convolutional neural networks with low-bitwidth
weights and activations.

</span>
<span class="ltx_bibblock"><em id="bib.bib46.1.1" class="ltx_emph ltx_font_italic">IEEE Transactions on Pattern Analysis and Machine
Intelligence</em>, 44(10):6140–6152, 2021.

</span>
</li>
</ul>
</section>
<section id="A1" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix A </span>More results</h2>

<section id="A1.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">A.1 </span>Non-rigorous comparison with AWQ</h3>

<div id="A1.SS1.p1" class="ltx_para ltx_noindent">
<p id="A1.SS1.p1.1" class="ltx_p">We conducted a limited comparison between our approach and AWQ <cite class="ltx_cite ltx_citemacro_cite">Lin et al. (<a href="#bib.bib18" title="" class="ltx_ref">2023</a>)</cite>, considering that our evaluation methodology closely follows that of GPTQ and we only share a few common tasks with AWQ. It is important to acknowledge that this comparison inherently lacks rigor due to our reliance on referencing AWQ’s data alone. Consequently, this approach introduces the possibility of unfairness in the evaluation process, primarily stemming from the utilization of different calibration datasets and other potential factors that may influence the obtained results.</p>
</div>
<div id="A1.SS1.p2" class="ltx_para">
<p id="A1.SS1.p2.1" class="ltx_p">We present the results of our common tasks alongside AWQ in table <a href="#A1.T8" title="Table 8 ‣ A.1 Non-rigorous comparison with AWQ ‣ Appendix A More results ‣ Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs" class="ltx_ref"><span class="ltx_text ltx_ref_tag">8</span></a> and all the results of AWQ are from their paper. While we both report MMLU results, it is important to note that there was a bug fix <span id="footnote3" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">3</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">3</sup><span class="ltx_tag ltx_tag_note">3</span>https://github.com/EleutherAI/lm-evaluation-harness/pull/497</span></span></span> in lm-eval, resulting in significant changes to the baseline. So we have not included them in this report.</p>
</div>
<figure id="A1.T8" class="ltx_table">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 8: </span> Reported results of AWQ and Ours</figcaption>
<table id="A1.T8.1" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="A1.T8.1.1.1" class="ltx_tr">
<th id="A1.T8.1.1.1.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r ltx_border_t" colspan="2">LLaMA-7B</th>
<td id="A1.T8.1.1.1.2" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" colspan="3">AWQ</td>
<td id="A1.T8.1.1.1.3" class="ltx_td ltx_align_center ltx_border_t" colspan="3">Ours</td>
</tr>
<tr id="A1.T8.1.2.2" class="ltx_tr">
<th id="A1.T8.1.2.2.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r ltx_border_t"><span id="A1.T8.1.2.2.1.1" class="ltx_text">nbits</span></th>
<th id="A1.T8.1.2.2.2" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r ltx_border_t"><span id="A1.T8.1.2.2.2.1" class="ltx_text">Method</span></th>
<td id="A1.T8.1.2.2.3" class="ltx_td ltx_align_center ltx_border_t">PIQA</td>
<td id="A1.T8.1.2.2.4" class="ltx_td ltx_align_center ltx_border_t">Hella.</td>
<td id="A1.T8.1.2.2.5" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">Wino.</td>
<td id="A1.T8.1.2.2.6" class="ltx_td ltx_align_center ltx_border_t">PIQA</td>
<td id="A1.T8.1.2.2.7" class="ltx_td ltx_align_center ltx_border_t">Hella.</td>
<td id="A1.T8.1.2.2.8" class="ltx_td ltx_align_center ltx_border_t">Wino.</td>
</tr>
<tr id="A1.T8.1.3.3" class="ltx_tr">
<th id="A1.T8.1.3.3.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r ltx_border_t"><span id="A1.T8.1.3.3.1.1" class="ltx_text">16</span></th>
<th id="A1.T8.1.3.3.2" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r ltx_border_t">FP16</th>
<td id="A1.T8.1.3.3.3" class="ltx_td ltx_align_center ltx_border_t">78.35</td>
<td id="A1.T8.1.3.3.4" class="ltx_td ltx_align_center ltx_border_t">56.44</td>
<td id="A1.T8.1.3.3.5" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">67.09</td>
<td id="A1.T8.1.3.3.6" class="ltx_td ltx_align_center ltx_border_t">78.35</td>
<td id="A1.T8.1.3.3.7" class="ltx_td ltx_align_center ltx_border_t">56.42</td>
<td id="A1.T8.1.3.3.8" class="ltx_td ltx_align_center ltx_border_t">66.85</td>
</tr>
<tr id="A1.T8.1.4.4" class="ltx_tr">
<th id="A1.T8.1.4.4.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r ltx_border_t" rowspan="3"><span id="A1.T8.1.4.4.1.1" class="ltx_text">W3G128</span></th>
<th id="A1.T8.1.4.4.2" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r ltx_border_t">RTN</th>
<td id="A1.T8.1.4.4.3" class="ltx_td ltx_align_center ltx_border_t">75.84</td>
<td id="A1.T8.1.4.4.4" class="ltx_td ltx_align_center ltx_border_t">53.10</td>
<td id="A1.T8.1.4.4.5" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">63.22</td>
<td id="A1.T8.1.4.4.6" class="ltx_td ltx_align_center ltx_border_t">75.73</td>
<td id="A1.T8.1.4.4.7" class="ltx_td ltx_align_center ltx_border_t">53.17</td>
<td id="A1.T8.1.4.4.8" class="ltx_td ltx_align_center ltx_border_t">63.14</td>
</tr>
<tr id="A1.T8.1.5.5" class="ltx_tr">
<th id="A1.T8.1.5.5.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r">GPTQ</th>
<td id="A1.T8.1.5.5.2" class="ltx_td ltx_align_center">70.89</td>
<td id="A1.T8.1.5.5.3" class="ltx_td ltx_align_center">46.77</td>
<td id="A1.T8.1.5.5.4" class="ltx_td ltx_align_center ltx_border_r">60.93</td>
<td id="A1.T8.1.5.5.5" class="ltx_td ltx_align_center">72.58</td>
<td id="A1.T8.1.5.5.6" class="ltx_td ltx_align_center">47.10</td>
<td id="A1.T8.1.5.5.7" class="ltx_td ltx_align_center">59.91</td>
</tr>
<tr id="A1.T8.1.6.6" class="ltx_tr">
<th id="A1.T8.1.6.6.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r">Proposed</th>
<td id="A1.T8.1.6.6.2" class="ltx_td ltx_align_center">76.66</td>
<td id="A1.T8.1.6.6.3" class="ltx_td ltx_align_center">53.63</td>
<td id="A1.T8.1.6.6.4" class="ltx_td ltx_align_center ltx_border_r">66.14</td>
<td id="A1.T8.1.6.6.5" class="ltx_td ltx_align_center">76.61</td>
<td id="A1.T8.1.6.6.6" class="ltx_td ltx_align_center">53.98</td>
<td id="A1.T8.1.6.6.7" class="ltx_td ltx_align_center">66.06</td>
</tr>
<tr id="A1.T8.1.7.7" class="ltx_tr">
<th id="A1.T8.1.7.7.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_b ltx_border_r ltx_border_t" rowspan="3"><span id="A1.T8.1.7.7.1.1" class="ltx_text">W4G128</span></th>
<th id="A1.T8.1.7.7.2" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r ltx_border_t">RTN</th>
<td id="A1.T8.1.7.7.3" class="ltx_td ltx_align_center ltx_border_t">77.86</td>
<td id="A1.T8.1.7.7.4" class="ltx_td ltx_align_center ltx_border_t">55.81</td>
<td id="A1.T8.1.7.7.5" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">65.59</td>
<td id="A1.T8.1.7.7.6" class="ltx_td ltx_align_center ltx_border_t">77.58</td>
<td id="A1.T8.1.7.7.7" class="ltx_td ltx_align_center ltx_border_t">55.86</td>
<td id="A1.T8.1.7.7.8" class="ltx_td ltx_align_center ltx_border_t">65.75</td>
</tr>
<tr id="A1.T8.1.8.8" class="ltx_tr">
<th id="A1.T8.1.8.8.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r">GPTQ</th>
<td id="A1.T8.1.8.8.2" class="ltx_td ltx_align_center">77.20</td>
<td id="A1.T8.1.8.8.3" class="ltx_td ltx_align_center">53.98</td>
<td id="A1.T8.1.8.8.4" class="ltx_td ltx_align_center ltx_border_r">65.67</td>
<td id="A1.T8.1.8.8.5" class="ltx_td ltx_align_center">77.26</td>
<td id="A1.T8.1.8.8.6" class="ltx_td ltx_align_center">54.09</td>
<td id="A1.T8.1.8.8.7" class="ltx_td ltx_align_center">64.09</td>
</tr>
<tr id="A1.T8.1.9.9" class="ltx_tr">
<th id="A1.T8.1.9.9.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_b ltx_border_r">Proposed</th>
<td id="A1.T8.1.9.9.2" class="ltx_td ltx_align_center ltx_border_b">78.07</td>
<td id="A1.T8.1.9.9.3" class="ltx_td ltx_align_center ltx_border_b">55.76</td>
<td id="A1.T8.1.9.9.4" class="ltx_td ltx_align_center ltx_border_b ltx_border_r">65.82</td>
<td id="A1.T8.1.9.9.5" class="ltx_td ltx_align_center ltx_border_b">78.07</td>
<td id="A1.T8.1.9.9.6" class="ltx_td ltx_align_center ltx_border_b">55.92</td>
<td id="A1.T8.1.9.9.7" class="ltx_td ltx_align_center ltx_border_b">66.30</td>
</tr>
</tbody>
</table>
</figure>
</section>
<section id="A1.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">A.2 </span>Results of Wikitext2 ppl at W4</h3>

<div id="A1.SS2.p1" class="ltx_para ltx_noindent">
<p id="A1.SS2.p1.1" class="ltx_p">The perplexity results for Wikitext2 at W4 are shown in Table <a href="#A1.T9" title="Table 9 ‣ A.2 Results of Wikitext2 ppl at W4 ‣ Appendix A More results ‣ Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs" class="ltx_ref"><span class="ltx_text ltx_ref_tag">9</span></a>. In conclusion, our performance is comparable to that of GPTQ.</p>
</div>
<figure id="A1.T9" class="ltx_table">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 9: </span>Wikitext2 ppl ( <math id="A1.T9.2.2.m1.1" class="ltx_Math" alttext="\downarrow" display="inline"><semantics id="A1.T9.2.2.m1.1b"><mo stretchy="false" id="A1.T9.2.2.m1.1.1" xref="A1.T9.2.2.m1.1.1.cmml">↓</mo><annotation-xml encoding="MathML-Content" id="A1.T9.2.2.m1.1c"><ci id="A1.T9.2.2.m1.1.1.cmml" xref="A1.T9.2.2.m1.1.1">↓</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.T9.2.2.m1.1d">\downarrow</annotation></semantics></math>) at W4</figcaption>
<table id="A1.T9.3" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A1.T9.3.1.1" class="ltx_tr">
<th id="A1.T9.3.1.1.1" class="ltx_td ltx_th ltx_th_column ltx_th_row ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;"></th>
<th id="A1.T9.3.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;" colspan="4">LLaMA</th>
<th id="A1.T9.3.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;" colspan="4">OPT</th>
<th id="A1.T9.3.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;" colspan="4">BLOOM</th>
</tr>
<tr id="A1.T9.3.2.2" class="ltx_tr">
<th id="A1.T9.3.2.2.1" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_th_row ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">Size</th>
<th id="A1.T9.3.2.2.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">7b</th>
<th id="A1.T9.3.2.2.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">13b</th>
<th id="A1.T9.3.2.2.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">7bv2</th>
<th id="A1.T9.3.2.2.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">13bv2</th>
<th id="A1.T9.3.2.2.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">1.3b</th>
<th id="A1.T9.3.2.2.7" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">2.7b</th>
<th id="A1.T9.3.2.2.8" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">6.7b</th>
<th id="A1.T9.3.2.2.9" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">13b</th>
<th id="A1.T9.3.2.2.10" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">560m</th>
<th id="A1.T9.3.2.2.11" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">1b7</th>
<th id="A1.T9.3.2.2.12" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">3b</th>
<th id="A1.T9.3.2.2.13" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">7b1</th>
</tr>
<tr id="A1.T9.3.3.3" class="ltx_tr">
<th id="A1.T9.3.3.3.1" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_th_row ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">FP16</th>
<th id="A1.T9.3.3.3.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">5.67</th>
<th id="A1.T9.3.3.3.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">5.09</th>
<th id="A1.T9.3.3.3.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">5.47</th>
<th id="A1.T9.3.3.3.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">4.88</th>
<th id="A1.T9.3.3.3.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">14.62</th>
<th id="A1.T9.3.3.3.7" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">12.47</th>
<th id="A1.T9.3.3.3.8" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">10.86</th>
<th id="A1.T9.3.3.3.9" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">10.13</th>
<th id="A1.T9.3.3.3.10" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">22.41</th>
<th id="A1.T9.3.3.3.11" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">15.39</th>
<th id="A1.T9.3.3.3.12" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">13.48</th>
<th id="A1.T9.3.3.3.13" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">11.37</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A1.T9.3.4.1" class="ltx_tr">
<th id="A1.T9.3.4.1.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">RTN</th>
<td id="A1.T9.3.4.1.2" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">6.29</td>
<td id="A1.T9.3.4.1.3" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">5.53</td>
<td id="A1.T9.3.4.1.4" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">6.12</td>
<td id="A1.T9.3.4.1.5" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">5.20</td>
<td id="A1.T9.3.4.1.6" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">48.20</td>
<td id="A1.T9.3.4.1.7" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">16.92</td>
<td id="A1.T9.3.4.1.8" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">12.10</td>
<td id="A1.T9.3.4.1.9" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">11.32</td>
<td id="A1.T9.3.4.1.10" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">25.88</td>
<td id="A1.T9.3.4.1.11" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">16.97</td>
<td id="A1.T9.3.4.1.12" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">14.75</td>
<td id="A1.T9.3.4.1.13" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">12.10</td>
</tr>
<tr id="A1.T9.3.5.2" class="ltx_tr">
<th id="A1.T9.3.5.2.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;">GPTQ</th>
<td id="A1.T9.3.5.2.2" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">6.59</td>
<td id="A1.T9.3.5.2.3" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">5.33</td>
<td id="A1.T9.3.5.2.4" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A1.T9.3.5.2.4.1" class="ltx_text ltx_font_bold">6.09</span></td>
<td id="A1.T9.3.5.2.5" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A1.T9.3.5.2.5.1" class="ltx_text ltx_font_bold">5.16</span></td>
<td id="A1.T9.3.5.2.6" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">15.67</td>
<td id="A1.T9.3.5.2.7" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">13.30</td>
<td id="A1.T9.3.5.2.8" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">11.59</td>
<td id="A1.T9.3.5.2.9" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A1.T9.3.5.2.9.1" class="ltx_text ltx_font_bold">10.33</span></td>
<td id="A1.T9.3.5.2.10" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">23.95</td>
<td id="A1.T9.3.5.2.11" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">16.37</td>
<td id="A1.T9.3.5.2.12" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A1.T9.3.5.2.12.1" class="ltx_text ltx_font_bold">14.10</span></td>
<td id="A1.T9.3.5.2.13" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A1.T9.3.5.2.13.1" class="ltx_text ltx_font_bold">11.73</span></td>
</tr>
<tr id="A1.T9.3.6.3" class="ltx_tr">
<th id="A1.T9.3.6.3.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_b ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;"> Ours</th>
<td id="A1.T9.3.6.3.2" class="ltx_td ltx_align_center ltx_border_b" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A1.T9.3.6.3.2.1" class="ltx_text ltx_font_bold">6.12</span></td>
<td id="A1.T9.3.6.3.3" class="ltx_td ltx_align_center ltx_border_b" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A1.T9.3.6.3.3.1" class="ltx_text ltx_font_bold">5.32</span></td>
<td id="A1.T9.3.6.3.4" class="ltx_td ltx_align_center ltx_border_b" style="padding-left:5.0pt;padding-right:5.0pt;">298.42</td>
<td id="A1.T9.3.6.3.5" class="ltx_td ltx_align_center ltx_border_b ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;">9.15</td>
<td id="A1.T9.3.6.3.6" class="ltx_td ltx_align_center ltx_border_b" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A1.T9.3.6.3.6.1" class="ltx_text ltx_font_bold">15.65</span></td>
<td id="A1.T9.3.6.3.7" class="ltx_td ltx_align_center ltx_border_b" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A1.T9.3.6.3.7.1" class="ltx_text ltx_font_bold">13.05</span></td>
<td id="A1.T9.3.6.3.8" class="ltx_td ltx_align_center ltx_border_b" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A1.T9.3.6.3.8.1" class="ltx_text ltx_font_bold">11.18</span></td>
<td id="A1.T9.3.6.3.9" class="ltx_td ltx_align_center ltx_border_b ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;">10.66</td>
<td id="A1.T9.3.6.3.10" class="ltx_td ltx_align_center ltx_border_b" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A1.T9.3.6.3.10.1" class="ltx_text ltx_font_bold">23.80</span></td>
<td id="A1.T9.3.6.3.11" class="ltx_td ltx_align_center ltx_border_b" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A1.T9.3.6.3.11.1" class="ltx_text ltx_font_bold">16.22</span></td>
<td id="A1.T9.3.6.3.12" class="ltx_td ltx_align_center ltx_border_b" style="padding-left:5.0pt;padding-right:5.0pt;">14.13</td>
<td id="A1.T9.3.6.3.13" class="ltx_td ltx_align_center ltx_border_b" style="padding-left:5.0pt;padding-right:5.0pt;">11.80</td>
</tr>
</tbody>
</table>
</figure>
</section>
<section id="A1.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">A.3 </span>Other results for large models</h3>

<div id="A1.SS3.p1" class="ltx_para ltx_noindent">
<p id="A1.SS3.p1.1" class="ltx_p">We present the results for models with a capacity of 30B or higher at W4 in Table <a href="#A1.T10" title="Table 10 ‣ A.3 Other results for large models ‣ Appendix A More results ‣ Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs" class="ltx_ref"><span class="ltx_text ltx_ref_tag">10</span></a> and PPL on Wikitext2 in Table <a href="#A1.T11" title="Table 11 ‣ A.3 Other results for large models ‣ Appendix A More results ‣ Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs" class="ltx_ref"><span class="ltx_text ltx_ref_tag">11</span></a>. Furthermore, we observed that adjusting the sequence length of the calibration dataset led to improvements in specific scenarios, and we include these findings in our analysis. Overall, our approach demonstrates comparable accuracy performance to GPTQ for the given task. However, it is worth noting that we slightly fall behind GPTQ in terms of PPL tasks.</p>
</div>
<figure id="A1.T10" class="ltx_table">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 10: </span>Average % accuracy(<math id="A1.T10.4.4.m1.1" class="ltx_Math" alttext="\uparrow" display="inline"><semantics id="A1.T10.4.4.m1.1b"><mo stretchy="false" id="A1.T10.4.4.m1.1.1" xref="A1.T10.4.4.m1.1.1.cmml">↑</mo><annotation-xml encoding="MathML-Content" id="A1.T10.4.4.m1.1c"><ci id="A1.T10.4.4.m1.1.1.cmml" xref="A1.T10.4.4.m1.1.1">↑</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.T10.4.4.m1.1d">\uparrow</annotation></semantics></math>) of HellaSwag, WinoGrand, PIQA and LAMBADA and C4 ppl(<math id="A1.T10.5.5.m2.1" class="ltx_Math" alttext="\downarrow" display="inline"><semantics id="A1.T10.5.5.m2.1b"><mo stretchy="false" id="A1.T10.5.5.m2.1.1" xref="A1.T10.5.5.m2.1.1.cmml">↓</mo><annotation-xml encoding="MathML-Content" id="A1.T10.5.5.m2.1c"><ci id="A1.T10.5.5.m2.1.1.cmml" xref="A1.T10.5.5.m2.1.1">↓</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.T10.5.5.m2.1d">\downarrow</annotation></semantics></math>) for LLaMA &amp; OPT with size <math id="A1.T10.6.6.m3.1" class="ltx_Math" alttext="\geq" display="inline"><semantics id="A1.T10.6.6.m3.1b"><mo id="A1.T10.6.6.m3.1.1" xref="A1.T10.6.6.m3.1.1.cmml">≥</mo><annotation-xml encoding="MathML-Content" id="A1.T10.6.6.m3.1c"><geq id="A1.T10.6.6.m3.1.1.cmml" xref="A1.T10.6.6.m3.1.1"></geq></annotation-xml><annotation encoding="application/x-tex" id="A1.T10.6.6.m3.1d">\geq</annotation></semantics></math> 30B at W4. ”Ours-seq512” indicates that we have modified the sequence length of the calibration dataset from 256 to 512.</figcaption>
<table id="A1.T10.7" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="A1.T10.7.1.1" class="ltx_tr">
<th id="A1.T10.7.1.1.1" class="ltx_td ltx_th ltx_th_row ltx_border_r ltx_border_t"></th>
<td id="A1.T10.7.1.1.2" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" colspan="4">Accuracy</td>
<td id="A1.T10.7.1.1.3" class="ltx_td ltx_align_center ltx_border_t" colspan="4">PPL on C4</td>
</tr>
<tr id="A1.T10.7.2.2" class="ltx_tr">
<th id="A1.T10.7.2.2.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r ltx_border_t">
<span id="A1.T10.7.2.2.1.1" class="ltx_text"></span> Type</th>
<td id="A1.T10.7.2.2.2" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" colspan="2">LLaMA</td>
<td id="A1.T10.7.2.2.3" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" colspan="2">OPT</td>
<td id="A1.T10.7.2.2.4" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" colspan="2">LLaMA</td>
<td id="A1.T10.7.2.2.5" class="ltx_td ltx_align_center ltx_border_t" colspan="2">OPT</td>
</tr>
<tr id="A1.T10.7.3.3" class="ltx_tr">
<th id="A1.T10.7.3.3.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r ltx_border_t">Size</th>
<td id="A1.T10.7.3.3.2" class="ltx_td ltx_align_center ltx_border_t">30b</td>
<td id="A1.T10.7.3.3.3" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">65b</td>
<td id="A1.T10.7.3.3.4" class="ltx_td ltx_align_center ltx_border_t">30b</td>
<td id="A1.T10.7.3.3.5" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">66b</td>
<td id="A1.T10.7.3.3.6" class="ltx_td ltx_align_center ltx_border_t">30b</td>
<td id="A1.T10.7.3.3.7" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">65b</td>
<td id="A1.T10.7.3.3.8" class="ltx_td ltx_align_center ltx_border_t">30b</td>
<td id="A1.T10.7.3.3.9" class="ltx_td ltx_align_center ltx_border_t">66b</td>
</tr>
<tr id="A1.T10.7.4.4" class="ltx_tr">
<th id="A1.T10.7.4.4.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r ltx_border_t">FP16</th>
<td id="A1.T10.7.4.4.2" class="ltx_td ltx_align_center ltx_border_t">73.46</td>
<td id="A1.T10.7.4.4.3" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">75.48</td>
<td id="A1.T10.7.4.4.4" class="ltx_td ltx_align_center ltx_border_t">67.87</td>
<td id="A1.T10.7.4.4.5" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">69.54</td>
<td id="A1.T10.7.4.4.6" class="ltx_td ltx_align_center ltx_border_t">6.13</td>
<td id="A1.T10.7.4.4.7" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">5.98</td>
<td id="A1.T10.7.4.4.8" class="ltx_td ltx_align_center ltx_border_t">11.45</td>
<td id="A1.T10.7.4.4.9" class="ltx_td ltx_align_center ltx_border_t">10.99</td>
</tr>
<tr id="A1.T10.7.5.5" class="ltx_tr">
<th id="A1.T10.7.5.5.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r ltx_border_t">RTN</th>
<td id="A1.T10.7.5.5.2" class="ltx_td ltx_align_center ltx_border_t">72.33</td>
<td id="A1.T10.7.5.5.3" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">73.91</td>
<td id="A1.T10.7.5.5.4" class="ltx_td ltx_align_center ltx_border_t">65.94</td>
<td id="A1.T10.7.5.5.5" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">37.12</td>
<td id="A1.T10.7.5.5.6" class="ltx_td ltx_align_center ltx_border_t">6.54</td>
<td id="A1.T10.7.5.5.7" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">6.46</td>
<td id="A1.T10.7.5.5.8" class="ltx_td ltx_align_center ltx_border_t">13.56</td>
<td id="A1.T10.7.5.5.9" class="ltx_td ltx_align_center ltx_border_t">305.73</td>
</tr>
<tr id="A1.T10.7.6.6" class="ltx_tr">
<th id="A1.T10.7.6.6.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r">GPTQ</th>
<td id="A1.T10.7.6.6.2" class="ltx_td ltx_align_center">72.85</td>
<td id="A1.T10.7.6.6.3" class="ltx_td ltx_align_center ltx_border_r"><span id="A1.T10.7.6.6.3.1" class="ltx_text ltx_font_bold">74.45</span></td>
<td id="A1.T10.7.6.6.4" class="ltx_td ltx_align_center"><span id="A1.T10.7.6.6.4.1" class="ltx_text ltx_font_bold">67.55</span></td>
<td id="A1.T10.7.6.6.5" class="ltx_td ltx_align_center ltx_border_r">68.23</td>
<td id="A1.T10.7.6.6.6" class="ltx_td ltx_align_center"><span id="A1.T10.7.6.6.6.1" class="ltx_text ltx_font_bold">6.42</span></td>
<td id="A1.T10.7.6.6.7" class="ltx_td ltx_align_center ltx_border_r"><span id="A1.T10.7.6.6.7.1" class="ltx_text ltx_font_bold">6.23</span></td>
<td id="A1.T10.7.6.6.8" class="ltx_td ltx_align_center"><span id="A1.T10.7.6.6.8.1" class="ltx_text ltx_font_bold">11.59</span></td>
<td id="A1.T10.7.6.6.9" class="ltx_td ltx_align_center"><span id="A1.T10.7.6.6.9.1" class="ltx_text ltx_font_bold">11.24</span></td>
</tr>
<tr id="A1.T10.7.7.7" class="ltx_tr">
<th id="A1.T10.7.7.7.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r">Ours-seq256</th>
<td id="A1.T10.7.7.7.2" class="ltx_td ltx_align_center">72.69</td>
<td id="A1.T10.7.7.7.3" class="ltx_td ltx_align_center ltx_border_r">74.03</td>
<td id="A1.T10.7.7.7.4" class="ltx_td ltx_align_center">66.74</td>
<td id="A1.T10.7.7.7.5" class="ltx_td ltx_align_center ltx_border_r">68.80</td>
<td id="A1.T10.7.7.7.6" class="ltx_td ltx_align_center">6.47</td>
<td id="A1.T10.7.7.7.7" class="ltx_td ltx_align_center ltx_border_r">6.31</td>
<td id="A1.T10.7.7.7.8" class="ltx_td ltx_align_center">11.84</td>
<td id="A1.T10.7.7.7.9" class="ltx_td ltx_align_center">11.42</td>
</tr>
<tr id="A1.T10.7.8.8" class="ltx_tr">
<th id="A1.T10.7.8.8.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_b ltx_border_r">Ours-seq512</th>
<td id="A1.T10.7.8.8.2" class="ltx_td ltx_align_center ltx_border_b"><span id="A1.T10.7.8.8.2.1" class="ltx_text ltx_font_bold">72.86</span></td>
<td id="A1.T10.7.8.8.3" class="ltx_td ltx_align_center ltx_border_b ltx_border_r">73.91</td>
<td id="A1.T10.7.8.8.4" class="ltx_td ltx_align_center ltx_border_b">67.40</td>
<td id="A1.T10.7.8.8.5" class="ltx_td ltx_align_center ltx_border_b ltx_border_r"><span id="A1.T10.7.8.8.5.1" class="ltx_text ltx_font_bold">69.22</span></td>
<td id="A1.T10.7.8.8.6" class="ltx_td ltx_align_center ltx_border_b">6.47</td>
<td id="A1.T10.7.8.8.7" class="ltx_td ltx_align_center ltx_border_b ltx_border_r">6.34</td>
<td id="A1.T10.7.8.8.8" class="ltx_td ltx_align_center ltx_border_b">11.77</td>
<td id="A1.T10.7.8.8.9" class="ltx_td ltx_align_center ltx_border_b">11.45</td>
</tr>
</tbody>
</table>
</figure>
<figure id="A1.T11" class="ltx_table">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 11: </span>Wikitext ppl(<math id="A1.T11.3.3.m1.1" class="ltx_Math" alttext="\downarrow" display="inline"><semantics id="A1.T11.3.3.m1.1b"><mo stretchy="false" id="A1.T11.3.3.m1.1.1" xref="A1.T11.3.3.m1.1.1.cmml">↓</mo><annotation-xml encoding="MathML-Content" id="A1.T11.3.3.m1.1c"><ci id="A1.T11.3.3.m1.1.1.cmml" xref="A1.T11.3.3.m1.1.1">↓</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.T11.3.3.m1.1d">\downarrow</annotation></semantics></math>) for LLaMA &amp; OPT with size <math id="A1.T11.4.4.m2.1" class="ltx_Math" alttext="\geq" display="inline"><semantics id="A1.T11.4.4.m2.1b"><mo id="A1.T11.4.4.m2.1.1" xref="A1.T11.4.4.m2.1.1.cmml">≥</mo><annotation-xml encoding="MathML-Content" id="A1.T11.4.4.m2.1c"><geq id="A1.T11.4.4.m2.1.1.cmml" xref="A1.T11.4.4.m2.1.1"></geq></annotation-xml><annotation encoding="application/x-tex" id="A1.T11.4.4.m2.1d">\geq</annotation></semantics></math> 30B. ”Ours-seq512” indicates that we have modified the sequence length of the calibration dataset from 256 to 512.</figcaption>
<table id="A1.T11.5" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="A1.T11.5.1.1" class="ltx_tr">
<th id="A1.T11.5.1.1.1" class="ltx_td ltx_th ltx_th_row ltx_border_r ltx_border_t"></th>
<td id="A1.T11.5.1.1.2" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" colspan="4">W4</td>
<td id="A1.T11.5.1.1.3" class="ltx_td ltx_align_center ltx_border_t" colspan="4">W3G128</td>
</tr>
<tr id="A1.T11.5.2.2" class="ltx_tr">
<th id="A1.T11.5.2.2.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r ltx_border_t">
<span id="A1.T11.5.2.2.1.1" class="ltx_text"></span> Type</th>
<td id="A1.T11.5.2.2.2" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" colspan="2">LLaMA</td>
<td id="A1.T11.5.2.2.3" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" colspan="2">OPT</td>
<td id="A1.T11.5.2.2.4" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" colspan="2">LLaMA</td>
<td id="A1.T11.5.2.2.5" class="ltx_td ltx_align_center ltx_border_t" colspan="2">OPT</td>
</tr>
<tr id="A1.T11.5.3.3" class="ltx_tr">
<th id="A1.T11.5.3.3.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r ltx_border_t">Size</th>
<td id="A1.T11.5.3.3.2" class="ltx_td ltx_align_center ltx_border_t">30b</td>
<td id="A1.T11.5.3.3.3" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">65b</td>
<td id="A1.T11.5.3.3.4" class="ltx_td ltx_align_center ltx_border_t">30b</td>
<td id="A1.T11.5.3.3.5" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">66b</td>
<td id="A1.T11.5.3.3.6" class="ltx_td ltx_align_center ltx_border_t">30b</td>
<td id="A1.T11.5.3.3.7" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">65b</td>
<td id="A1.T11.5.3.3.8" class="ltx_td ltx_align_center ltx_border_t">30b</td>
<td id="A1.T11.5.3.3.9" class="ltx_td ltx_align_center ltx_border_t">66b</td>
</tr>
<tr id="A1.T11.5.4.4" class="ltx_tr">
<th id="A1.T11.5.4.4.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r ltx_border_t">FP16</th>
<td id="A1.T11.5.4.4.2" class="ltx_td ltx_align_center ltx_border_t">4.10</td>
<td id="A1.T11.5.4.4.3" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">3.56</td>
<td id="A1.T11.5.4.4.4" class="ltx_td ltx_align_center ltx_border_t">9.56</td>
<td id="A1.T11.5.4.4.5" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">9.34</td>
<td id="A1.T11.5.4.4.6" class="ltx_td ltx_align_center ltx_border_t">4.10</td>
<td id="A1.T11.5.4.4.7" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">3.56</td>
<td id="A1.T11.5.4.4.8" class="ltx_td ltx_align_center ltx_border_t">9.56</td>
<td id="A1.T11.5.4.4.9" class="ltx_td ltx_align_center ltx_border_t">9.34</td>
</tr>
<tr id="A1.T11.5.5.5" class="ltx_tr">
<th id="A1.T11.5.5.5.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r ltx_border_t">RTN</th>
<td id="A1.T11.5.5.5.2" class="ltx_td ltx_align_center ltx_border_t">4.54</td>
<td id="A1.T11.5.5.5.3" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">3.99</td>
<td id="A1.T11.5.5.5.4" class="ltx_td ltx_align_center ltx_border_t">10.98</td>
<td id="A1.T11.5.5.5.5" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">110.43</td>
<td id="A1.T11.5.5.5.6" class="ltx_td ltx_align_center ltx_border_t">4.87</td>
<td id="A1.T11.5.5.5.7" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">4.44</td>
<td id="A1.T11.5.5.5.8" class="ltx_td ltx_align_center ltx_border_t">23.05</td>
<td id="A1.T11.5.5.5.9" class="ltx_td ltx_align_center ltx_border_t">126.92</td>
</tr>
<tr id="A1.T11.5.6.6" class="ltx_tr">
<th id="A1.T11.5.6.6.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r">GPTQ</th>
<td id="A1.T11.5.6.6.2" class="ltx_td ltx_align_center"><span id="A1.T11.5.6.6.2.1" class="ltx_text ltx_font_bold">4.45</span></td>
<td id="A1.T11.5.6.6.3" class="ltx_td ltx_align_center ltx_border_r">4.16</td>
<td id="A1.T11.5.6.6.4" class="ltx_td ltx_align_center"><span id="A1.T11.5.6.6.4.1" class="ltx_text ltx_font_bold">9.66</span></td>
<td id="A1.T11.5.6.6.5" class="ltx_td ltx_align_center ltx_border_r">9.66</td>
<td id="A1.T11.5.6.6.6" class="ltx_td ltx_align_center">4.84</td>
<td id="A1.T11.5.6.6.7" class="ltx_td ltx_align_center ltx_border_r">4.17</td>
<td id="A1.T11.5.6.6.8" class="ltx_td ltx_align_center"><span id="A1.T11.5.6.6.8.1" class="ltx_text ltx_font_bold">9.75</span></td>
<td id="A1.T11.5.6.6.9" class="ltx_td ltx_align_center"><span id="A1.T11.5.6.6.9.1" class="ltx_text ltx_font_bold">10.58</span></td>
</tr>
<tr id="A1.T11.5.7.7" class="ltx_tr">
<th id="A1.T11.5.7.7.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r">Ours-seq256</th>
<td id="A1.T11.5.7.7.2" class="ltx_td ltx_align_center">4.51</td>
<td id="A1.T11.5.7.7.3" class="ltx_td ltx_align_center ltx_border_r">3.91</td>
<td id="A1.T11.5.7.7.4" class="ltx_td ltx_align_center">9.88</td>
<td id="A1.T11.5.7.7.5" class="ltx_td ltx_align_center ltx_border_r"><span id="A1.T11.5.7.7.5.1" class="ltx_text ltx_font_bold">9.56</span></td>
<td id="A1.T11.5.7.7.6" class="ltx_td ltx_align_center">4.85</td>
<td id="A1.T11.5.7.7.7" class="ltx_td ltx_align_center ltx_border_r"><span id="A1.T11.5.7.7.7.1" class="ltx_text ltx_font_bold">4.15</span></td>
<td id="A1.T11.5.7.7.8" class="ltx_td ltx_align_center">11.07</td>
<td id="A1.T11.5.7.7.9" class="ltx_td ltx_align_center">11.40</td>
</tr>
<tr id="A1.T11.5.8.8" class="ltx_tr">
<th id="A1.T11.5.8.8.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_b ltx_border_r">Ours-seq512</th>
<td id="A1.T11.5.8.8.2" class="ltx_td ltx_align_center ltx_border_b">4.52</td>
<td id="A1.T11.5.8.8.3" class="ltx_td ltx_align_center ltx_border_b ltx_border_r"><span id="A1.T11.5.8.8.3.1" class="ltx_text ltx_font_bold">3.90</span></td>
<td id="A1.T11.5.8.8.4" class="ltx_td ltx_align_center ltx_border_b">9.88</td>
<td id="A1.T11.5.8.8.5" class="ltx_td ltx_align_center ltx_border_b ltx_border_r">9.70</td>
<td id="A1.T11.5.8.8.6" class="ltx_td ltx_align_center ltx_border_b"><span id="A1.T11.5.8.8.6.1" class="ltx_text ltx_font_bold">4.81</span></td>
<td id="A1.T11.5.8.8.7" class="ltx_td ltx_align_center ltx_border_b ltx_border_r">4.17</td>
<td id="A1.T11.5.8.8.8" class="ltx_td ltx_align_center ltx_border_b">10.54</td>
<td id="A1.T11.5.8.8.9" class="ltx_td ltx_align_center ltx_border_b">10.87</td>
</tr>
</tbody>
</table>
</figure>
</section>
</section>
<section id="A2" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix B </span>Visualization of V</h2>

<div id="A2.p1" class="ltx_para ltx_noindent">
<p id="A2.p1.1" class="ltx_p">We provide an analysis of the magnitude distribution of V in Eq. <a href="#S3.E2" title="In 3 Methodology ‣ Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a> for approximately 7B models at W4 in Figure <a href="#A2.F3" title="Figure 3 ‣ Appendix B Visualization of V ‣ Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>. The findings reveal that the majority of V values are concentrated within the range of [-0.3, 0.3]. Notably, the middle layers demonstrate a narrower distribution in comparison to the other layers. This observation suggests that the head or tail layers may be more susceptible to the compression.</p>
</div>
<figure id="A2.F3" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="A2.F3.1" class="ltx_figure ltx_figure_panel ltx_minipage ltx_align_center ltx_align_middle" style="width:195.1pt;"><img src="/html/2309.05516/assets/figs-0925/llama-7b-hf-grad-0925.png" id="A2.F3.1.g1" class="ltx_graphics ltx_img_square" width="598" height="594" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering">LLaMA-7B</figcaption>
</figure>
</div>
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="A2.F3.2" class="ltx_figure ltx_figure_panel ltx_minipage ltx_align_center ltx_align_middle" style="width:195.1pt;"><img src="/html/2309.05516/assets/figs-0925/Llama-2-7b-hf-grad-0925.png" id="A2.F3.2.g1" class="ltx_graphics ltx_img_square" width="598" height="598" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering">LLaMA-7B-V2</figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="A2.F3.3" class="ltx_figure ltx_figure_panel ltx_minipage ltx_align_center ltx_align_middle" style="width:195.1pt;"><img src="/html/2309.05516/assets/figs-0925/opt-6.7b-grad-0925.png" id="A2.F3.3.g1" class="ltx_graphics ltx_centering ltx_img_square" width="598" height="595" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering">OPT-6.7B</figcaption>
</figure>
</div>
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="A2.F3.4" class="ltx_figure ltx_figure_panel ltx_minipage ltx_align_center ltx_align_middle" style="width:195.1pt;"><img src="/html/2309.05516/assets/figs-0925/bloom-7b1-grad-0925.png" id="A2.F3.4.g1" class="ltx_graphics ltx_img_square" width="598" height="594" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering">BLOOM-7B1</figcaption>
</figure>
</div>
</div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 3: </span>The distribution of the magnitude of V in Eq. <a href="#S3.E2" title="In 3 Methodology ‣ Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a> for different models, namely LLaMA-7B, LLaMA-7B-V2, OPT-6.7B, and BLOOM-7B1 at W4. Each color in the distribution represents a specific layer index in the models, with blue indicating shallow layers closer to the data layer, and red representing deeper layers.</figcaption>
</figure>
</section>
<section id="A3" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix C </span>Correction between SignRound and salient activation channels</h2>

<div id="A3.p1" class="ltx_para ltx_noindent">
<p id="A3.p1.1" class="ltx_p">We were also intrigued by the possible correlation between rounding and activation, as previous research has shown that keeping only 0.1%-1% of the channels corresponding to larger activation can significantly improve the quantized performance in AWQ <cite class="ltx_cite ltx_citemacro_citep">(Lin et al., <a href="#bib.bib18" title="" class="ltx_ref">2023</a>)</cite>. Therefore, we investigated whether the altered rounding values tend to fall more frequently in these salient channels. The results of our analysis, presented in Figure <a href="#A3.F4" title="Figure 4 ‣ Appendix C Correction between SignRound and salient activation channels ‣ Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>, reveal an interesting finding. The ratio, representing the percentage of altered rounding values falling within the top 1% salient activation channels out of all altered rounding values, is typically around 1%. This suggests that there is no strong correlation between rounding and activation. It is possible that rounding values of less significant channels need to be changed to compensate for the quantization error introduced by these salient channels.</p>
</div>
<figure id="A3.F4" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="A3.F4.1" class="ltx_figure ltx_figure_panel ltx_minipage ltx_align_center ltx_align_middle" style="width:195.1pt;"><img src="/html/2309.05516/assets/figs-0925/llama-7b-hf-gap-awq-0925.png" id="A3.F4.1.g1" class="ltx_graphics ltx_img_landscape" width="598" height="263" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering">LLaMA-7B</figcaption>
</figure>
</div>
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="A3.F4.2" class="ltx_figure ltx_figure_panel ltx_minipage ltx_align_center ltx_align_middle" style="width:195.1pt;"><img src="/html/2309.05516/assets/figs-0925/Llama-2-7b-hf-gap-awq-0925.png" id="A3.F4.2.g1" class="ltx_graphics ltx_img_landscape" width="598" height="263" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering">LLaMA-7B-V2</figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="A3.F4.3" class="ltx_figure ltx_figure_panel ltx_minipage ltx_align_center ltx_align_middle" style="width:195.1pt;"><img src="/html/2309.05516/assets/figs-0925/opt-6.7b-gap-awq-0925.png" id="A3.F4.3.g1" class="ltx_graphics ltx_img_landscape" width="598" height="260" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering">OPT-6.7B</figcaption>
</figure>
</div>
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="A3.F4.4" class="ltx_figure ltx_figure_panel ltx_minipage ltx_align_center ltx_align_middle" style="width:195.1pt;"><img src="/html/2309.05516/assets/figs-0925/bloom-7b1-gap-awq-0925.png" id="A3.F4.4.g1" class="ltx_graphics ltx_img_landscape" width="598" height="263" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering">BLOOM-7B1</figcaption>
</figure>
</div>
</div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 4: </span>The correction between SignRound and salient activation channels</figcaption>
</figure>
</section>
<section id="A4" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix D </span>Runtime</h2>

<div id="A4.p1" class="ltx_para ltx_noindent">
<p id="A4.p1.1" class="ltx_p">Table <a href="#A4.T12" title="Table 12 ‣ Appendix D Runtime ‣ Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs" class="ltx_ref"><span class="ltx_text ltx_ref_tag">12</span></a> provides a runtime comparison between GPTQ and our method. All measurements were conducted on a single NVIDIA A100 card with 80GB of memory. Although our method demonstrates slightly slower performance compared to GPTQ, it remains well within acceptable limits for real-world deployment.</p>
</div>
<figure id="A4.T12" class="ltx_table">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 12: </span> Runtime in seconds at W4 </figcaption>
<table id="A4.T12.1" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A4.T12.1.1.1" class="ltx_tr">
<th id="A4.T12.1.1.1.1" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_th_row ltx_border_r ltx_border_t"><span id="A4.T12.1.1.1.1.1" class="ltx_text">Type</span></th>
<th id="A4.T12.1.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_t" colspan="2">LLaMA</th>
<th id="A4.T12.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_t" colspan="2">OPT</th>
<th id="A4.T12.1.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" colspan="2">BLOOM</th>
</tr>
<tr id="A4.T12.1.2.2" class="ltx_tr">
<th id="A4.T12.1.2.2.1" class="ltx_td ltx_th ltx_th_column ltx_th_row ltx_border_r ltx_border_t"></th>
<th id="A4.T12.1.2.2.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">7B</th>
<th id="A4.T12.1.2.2.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_t">13B</th>
<th id="A4.T12.1.2.2.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">6.7B</th>
<th id="A4.T12.1.2.2.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_t">13B</th>
<th id="A4.T12.1.2.2.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">3B</th>
<th id="A4.T12.1.2.2.7" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">7B1</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A4.T12.1.3.1" class="ltx_tr">
<th id="A4.T12.1.3.1.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r ltx_border_t">GPTQ</th>
<td id="A4.T12.1.3.1.2" class="ltx_td ltx_align_center ltx_border_t"><span id="A4.T12.1.3.1.2.1" class="ltx_text ltx_font_bold">712</span></td>
<td id="A4.T12.1.3.1.3" class="ltx_td ltx_align_center ltx_border_r ltx_border_t"><span id="A4.T12.1.3.1.3.1" class="ltx_text ltx_font_bold">1240</span></td>
<td id="A4.T12.1.3.1.4" class="ltx_td ltx_align_center ltx_border_t">841</td>
<td id="A4.T12.1.3.1.5" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">1523</td>
<td id="A4.T12.1.3.1.6" class="ltx_td ltx_align_center ltx_border_t"><span id="A4.T12.1.3.1.6.1" class="ltx_text ltx_font_bold">345</span></td>
<td id="A4.T12.1.3.1.7" class="ltx_td ltx_align_center ltx_border_t"><span id="A4.T12.1.3.1.7.1" class="ltx_text ltx_font_bold">661</span></td>
</tr>
<tr id="A4.T12.1.4.2" class="ltx_tr">
<th id="A4.T12.1.4.2.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_b ltx_border_r ltx_border_t">Ours</th>
<td id="A4.T12.1.4.2.2" class="ltx_td ltx_align_center ltx_border_b ltx_border_t">899</td>
<td id="A4.T12.1.4.2.3" class="ltx_td ltx_align_center ltx_border_b ltx_border_r ltx_border_t">1590</td>
<td id="A4.T12.1.4.2.4" class="ltx_td ltx_align_center ltx_border_b ltx_border_t"><span id="A4.T12.1.4.2.4.1" class="ltx_text ltx_font_bold">819</span></td>
<td id="A4.T12.1.4.2.5" class="ltx_td ltx_align_center ltx_border_b ltx_border_r ltx_border_t"><span id="A4.T12.1.4.2.5.1" class="ltx_text ltx_font_bold">1429</span></td>
<td id="A4.T12.1.4.2.6" class="ltx_td ltx_align_center ltx_border_b ltx_border_t">467</td>
<td id="A4.T12.1.4.2.7" class="ltx_td ltx_align_center ltx_border_b ltx_border_t">843</td>
</tr>
</tbody>
</table>
</figure>
</section>
<section id="A5" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix E </span>Detailed Results of some LLaMa models</h2>

<div id="A5.p1" class="ltx_para">
<p id="A5.p1.1" class="ltx_p">Detailed results of LLaMA7B, LLaMA13B, LLAMA7B-V2, and LLAMA13B-V2 can be found in Table <a href="#A5.T13" title="Table 13 ‣ Appendix E Detailed Results of some LLaMa models ‣ Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs" class="ltx_ref"><span class="ltx_text ltx_ref_tag">13</span></a>, Table <a href="#A5.T14" title="Table 14 ‣ Appendix E Detailed Results of some LLaMa models ‣ Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs" class="ltx_ref"><span class="ltx_text ltx_ref_tag">14</span></a>, Table <a href="#A5.T15" title="Table 15 ‣ Appendix E Detailed Results of some LLaMa models ‣ Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs" class="ltx_ref"><span class="ltx_text ltx_ref_tag">15</span></a>, and Table <a href="#A5.T16" title="Table 16 ‣ Appendix E Detailed Results of some LLaMa models ‣ Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs" class="ltx_ref"><span class="ltx_text ltx_ref_tag">16</span></a> respectively.</p>
</div>
<figure id="A5.T13" class="ltx_table">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 13: </span>Accuracies(<math id="A5.T13.3.3.m1.1" class="ltx_Math" alttext="\uparrow" display="inline"><semantics id="A5.T13.3.3.m1.1b"><mo stretchy="false" id="A5.T13.3.3.m1.1.1" xref="A5.T13.3.3.m1.1.1.cmml">↑</mo><annotation-xml encoding="MathML-Content" id="A5.T13.3.3.m1.1c"><ci id="A5.T13.3.3.m1.1.1.cmml" xref="A5.T13.3.3.m1.1.1">↑</ci></annotation-xml><annotation encoding="application/x-tex" id="A5.T13.3.3.m1.1d">\uparrow</annotation></semantics></math>) of HellaSwag, WinoGrand, PIQA, LAMBADA and PPL(<math id="A5.T13.4.4.m2.1" class="ltx_Math" alttext="\downarrow" display="inline"><semantics id="A5.T13.4.4.m2.1b"><mo stretchy="false" id="A5.T13.4.4.m2.1.1" xref="A5.T13.4.4.m2.1.1.cmml">↓</mo><annotation-xml encoding="MathML-Content" id="A5.T13.4.4.m2.1c"><ci id="A5.T13.4.4.m2.1.1.cmml" xref="A5.T13.4.4.m2.1.1">↓</ci></annotation-xml><annotation encoding="application/x-tex" id="A5.T13.4.4.m2.1d">\downarrow</annotation></semantics></math>) of WikiText, PTB, C4 for <span id="A5.T13.6.1" class="ltx_text ltx_font_bold">LLaMA-7B</span>, ”Ours-2048” indicates that we have modified the sequence length of the calibration dataset from 512 to 2048.</figcaption>
<table id="A5.T13.7" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A5.T13.7.1.1" class="ltx_tr">
<th id="A5.T13.7.1.1.1" class="ltx_td ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;"></th>
<th id="A5.T13.7.1.1.2" class="ltx_td ltx_th ltx_th_column ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;"></th>
<th id="A5.T13.7.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">Hella.</th>
<th id="A5.T13.7.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">Wino.</th>
<th id="A5.T13.7.1.1.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">PIQA</th>
<th id="A5.T13.7.1.1.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">Lamb.</th>
<th id="A5.T13.7.1.1.7" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">Avg.</th>
<th id="A5.T13.7.1.1.8" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">Wiki.</th>
<th id="A5.T13.7.1.1.9" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">PTB</th>
<th id="A5.T13.7.1.1.10" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">C4</th>
</tr>
<tr id="A5.T13.7.2.2" class="ltx_tr">
<th id="A5.T13.7.2.2.1" class="ltx_td ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;"></th>
<th id="A5.T13.7.2.2.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">FP16</th>
<th id="A5.T13.7.2.2.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">56.42</th>
<th id="A5.T13.7.2.2.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">66.85</th>
<th id="A5.T13.7.2.2.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">78.35</th>
<th id="A5.T13.7.2.2.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">73.57</th>
<th id="A5.T13.7.2.2.7" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">68.80</th>
<th id="A5.T13.7.2.2.8" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">5.68</th>
<th id="A5.T13.7.2.2.9" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">10.12</th>
<th id="A5.T13.7.2.2.10" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">7.34</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A5.T13.7.3.1" class="ltx_tr">
<td id="A5.T13.7.3.1.1" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;" rowspan="4"><span id="A5.T13.7.3.1.1.1" class="ltx_text">W4G-1</span></td>
<td id="A5.T13.7.3.1.2" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">RTN</td>
<td id="A5.T13.7.3.1.3" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T13.7.3.1.3.1" class="ltx_text ltx_font_bold">54.96</span></td>
<td id="A5.T13.7.3.1.4" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">67.25</td>
<td id="A5.T13.7.3.1.5" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">77.31</td>
<td id="A5.T13.7.3.1.6" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">70.00</td>
<td id="A5.T13.7.3.1.7" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">67.38</td>
<td id="A5.T13.7.3.1.8" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">6.29</td>
<td id="A5.T13.7.3.1.9" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">11.25</td>
<td id="A5.T13.7.3.1.10" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">8.12</td>
</tr>
<tr id="A5.T13.7.4.2" class="ltx_tr">
<td id="A5.T13.7.4.2.1" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;">GPTQ</td>
<td id="A5.T13.7.4.2.2" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">52.25</td>
<td id="A5.T13.7.4.2.3" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">63.85</td>
<td id="A5.T13.7.4.2.4" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">74.59</td>
<td id="A5.T13.7.4.2.5" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">68.12</td>
<td id="A5.T13.7.4.2.6" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;">64.70</td>
<td id="A5.T13.7.4.2.7" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">6.59</td>
<td id="A5.T13.7.4.2.8" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">12.02</td>
<td id="A5.T13.7.4.2.9" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">8.64</td>
</tr>
<tr id="A5.T13.7.5.3" class="ltx_tr">
<td id="A5.T13.7.5.3.1" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;">Ours</td>
<td id="A5.T13.7.5.3.2" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">55.28</td>
<td id="A5.T13.7.5.3.3" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">66.14</td>
<td id="A5.T13.7.5.3.4" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">77.64</td>
<td id="A5.T13.7.5.3.5" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T13.7.5.3.5.1" class="ltx_text ltx_font_bold">73.14</span></td>
<td id="A5.T13.7.5.3.6" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;">68.05</td>
<td id="A5.T13.7.5.3.7" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">6.12</td>
<td id="A5.T13.7.5.3.8" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">10.88</td>
<td id="A5.T13.7.5.3.9" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">7.84</td>
</tr>
<tr id="A5.T13.7.6.4" class="ltx_tr">
<td id="A5.T13.7.6.4.1" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;">Ours-2048</td>
<td id="A5.T13.7.6.4.2" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T13.7.6.4.2.1" class="ltx_text ltx_font_bold">54.96</span></td>
<td id="A5.T13.7.6.4.3" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T13.7.6.4.3.1" class="ltx_text ltx_font_bold">67.56</span></td>
<td id="A5.T13.7.6.4.4" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T13.7.6.4.4.1" class="ltx_text ltx_font_bold">77.80</span></td>
<td id="A5.T13.7.6.4.5" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">72.13</td>
<td id="A5.T13.7.6.4.6" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T13.7.6.4.6.1" class="ltx_text ltx_font_bold">68.11</span></td>
<td id="A5.T13.7.6.4.7" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T13.7.6.4.7.1" class="ltx_text ltx_font_bold">6.05</span></td>
<td id="A5.T13.7.6.4.8" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T13.7.6.4.8.1" class="ltx_text ltx_font_bold">10.87</span></td>
<td id="A5.T13.7.6.4.9" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T13.7.6.4.9.1" class="ltx_text ltx_font_bold">7.76</span></td>
</tr>
<tr id="A5.T13.7.7.5" class="ltx_tr">
<td id="A5.T13.7.7.5.1" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;" rowspan="4"><span id="A5.T13.7.7.5.1.1" class="ltx_text">W4G128</span></td>
<td id="A5.T13.7.7.5.2" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">RTN</td>
<td id="A5.T13.7.7.5.3" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">55.86</td>
<td id="A5.T13.7.7.5.4" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">65.75</td>
<td id="A5.T13.7.7.5.5" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">77.58</td>
<td id="A5.T13.7.7.5.6" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T13.7.7.5.6.1" class="ltx_text ltx_font_bold">72.25</span></td>
<td id="A5.T13.7.7.5.7" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T13.7.7.5.7.1" class="ltx_text ltx_font_bold">67.86</span></td>
<td id="A5.T13.7.7.5.8" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">5.96</td>
<td id="A5.T13.7.7.5.9" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">10.54</td>
<td id="A5.T13.7.7.5.10" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">7.70</td>
</tr>
<tr id="A5.T13.7.8.6" class="ltx_tr">
<td id="A5.T13.7.8.6.1" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;">GPTQ</td>
<td id="A5.T13.7.8.6.2" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">54.09</td>
<td id="A5.T13.7.8.6.3" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">64.09</td>
<td id="A5.T13.7.8.6.4" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">77.26</td>
<td id="A5.T13.7.8.6.5" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">69.86</td>
<td id="A5.T13.7.8.6.6" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;">66.33</td>
<td id="A5.T13.7.8.6.7" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">6.29</td>
<td id="A5.T13.7.8.6.8" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">11.11</td>
<td id="A5.T13.7.8.6.9" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">8.02</td>
</tr>
<tr id="A5.T13.7.9.7" class="ltx_tr">
<td id="A5.T13.7.9.7.1" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;">Ours</td>
<td id="A5.T13.7.9.7.2" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">55.92</td>
<td id="A5.T13.7.9.7.3" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">66.30</td>
<td id="A5.T13.7.9.7.4" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">78.07</td>
<td id="A5.T13.7.9.7.5" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">72.07</td>
<td id="A5.T13.7.9.7.6" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;">68.09</td>
<td id="A5.T13.7.9.7.7" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T13.7.9.7.7.1" class="ltx_text ltx_font_bold">5.86</span></td>
<td id="A5.T13.7.9.7.8" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T13.7.9.7.8.1" class="ltx_text ltx_font_bold">10.49</span></td>
<td id="A5.T13.7.9.7.9" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T13.7.9.7.9.1" class="ltx_text ltx_font_bold">7.56</span></td>
</tr>
<tr id="A5.T13.7.10.8" class="ltx_tr">
<td id="A5.T13.7.10.8.1" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;">Ours-2048</td>
<td id="A5.T13.7.10.8.2" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T13.7.10.8.2.1" class="ltx_text ltx_font_bold">55.98</span></td>
<td id="A5.T13.7.10.8.3" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T13.7.10.8.3.1" class="ltx_text ltx_font_bold">66.77</span></td>
<td id="A5.T13.7.10.8.4" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T13.7.10.8.4.1" class="ltx_text ltx_font_bold">78.29</span></td>
<td id="A5.T13.7.10.8.5" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">71.78</td>
<td id="A5.T13.7.10.8.6" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T13.7.10.8.6.1" class="ltx_text ltx_font_bold">68.21</span></td>
<td id="A5.T13.7.10.8.7" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">5.88</td>
<td id="A5.T13.7.10.8.8" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">10.52</td>
<td id="A5.T13.7.10.8.9" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">7.58</td>
</tr>
<tr id="A5.T13.7.11.9" class="ltx_tr">
<td id="A5.T13.7.11.9.1" class="ltx_td ltx_align_center ltx_border_b ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;" rowspan="4"><span id="A5.T13.7.11.9.1.1" class="ltx_text">W3G128</span></td>
<td id="A5.T13.7.11.9.2" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">RTN</td>
<td id="A5.T13.7.11.9.3" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">53.17</td>
<td id="A5.T13.7.11.9.4" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">63.14</td>
<td id="A5.T13.7.11.9.5" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">75.73</td>
<td id="A5.T13.7.11.9.6" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">67.71</td>
<td id="A5.T13.7.11.9.7" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">64.94</td>
<td id="A5.T13.7.11.9.8" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">7.01</td>
<td id="A5.T13.7.11.9.9" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">12.83</td>
<td id="A5.T13.7.11.9.10" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">9.18</td>
</tr>
<tr id="A5.T13.7.12.10" class="ltx_tr">
<td id="A5.T13.7.12.10.1" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;">GPTQ</td>
<td id="A5.T13.7.12.10.2" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">47.10</td>
<td id="A5.T13.7.12.10.3" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">59.91</td>
<td id="A5.T13.7.12.10.4" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">72.58</td>
<td id="A5.T13.7.12.10.5" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">53.58</td>
<td id="A5.T13.7.12.10.6" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;">58.29</td>
<td id="A5.T13.7.12.10.7" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">8.28</td>
<td id="A5.T13.7.12.10.8" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">16.84</td>
<td id="A5.T13.7.12.10.9" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">10.45</td>
</tr>
<tr id="A5.T13.7.13.11" class="ltx_tr">
<td id="A5.T13.7.13.11.1" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;">Ours</td>
<td id="A5.T13.7.13.11.2" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T13.7.13.11.2.1" class="ltx_text ltx_font_bold">53.98</span></td>
<td id="A5.T13.7.13.11.3" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T13.7.13.11.3.1" class="ltx_text ltx_font_bold">66.06</span></td>
<td id="A5.T13.7.13.11.4" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T13.7.13.11.4.1" class="ltx_text ltx_font_bold">76.61</span></td>
<td id="A5.T13.7.13.11.5" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">69.82</td>
<td id="A5.T13.7.13.11.6" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;">66.62</td>
<td id="A5.T13.7.13.11.7" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">6.93</td>
<td id="A5.T13.7.13.11.8" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">11.67</td>
<td id="A5.T13.7.13.11.9" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">8.30</td>
</tr>
<tr id="A5.T13.7.14.12" class="ltx_tr">
<td id="A5.T13.7.14.12.1" class="ltx_td ltx_align_center ltx_border_b ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;">Ours-2048</td>
<td id="A5.T13.7.14.12.2" class="ltx_td ltx_align_center ltx_border_b" style="padding-left:5.0pt;padding-right:5.0pt;">53.45</td>
<td id="A5.T13.7.14.12.3" class="ltx_td ltx_align_center ltx_border_b" style="padding-left:5.0pt;padding-right:5.0pt;">65.67</td>
<td id="A5.T13.7.14.12.4" class="ltx_td ltx_align_center ltx_border_b" style="padding-left:5.0pt;padding-right:5.0pt;">76.55</td>
<td id="A5.T13.7.14.12.5" class="ltx_td ltx_align_center ltx_border_b" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T13.7.14.12.5.1" class="ltx_text ltx_font_bold">71.08</span></td>
<td id="A5.T13.7.14.12.6" class="ltx_td ltx_align_center ltx_border_b ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T13.7.14.12.6.1" class="ltx_text ltx_font_bold">66.69</span></td>
<td id="A5.T13.7.14.12.7" class="ltx_td ltx_align_center ltx_border_b" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T13.7.14.12.7.1" class="ltx_text ltx_font_bold">6.52</span></td>
<td id="A5.T13.7.14.12.8" class="ltx_td ltx_align_center ltx_border_b" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T13.7.14.12.8.1" class="ltx_text ltx_font_bold">11.60</span></td>
<td id="A5.T13.7.14.12.9" class="ltx_td ltx_align_center ltx_border_b" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T13.7.14.12.9.1" class="ltx_text ltx_font_bold">8.26</span></td>
</tr>
</tbody>
</table>
</figure>
<figure id="A5.T14" class="ltx_table">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 14: </span>Accuracies(<math id="A5.T14.3.3.m1.1" class="ltx_Math" alttext="\uparrow" display="inline"><semantics id="A5.T14.3.3.m1.1b"><mo stretchy="false" id="A5.T14.3.3.m1.1.1" xref="A5.T14.3.3.m1.1.1.cmml">↑</mo><annotation-xml encoding="MathML-Content" id="A5.T14.3.3.m1.1c"><ci id="A5.T14.3.3.m1.1.1.cmml" xref="A5.T14.3.3.m1.1.1">↑</ci></annotation-xml><annotation encoding="application/x-tex" id="A5.T14.3.3.m1.1d">\uparrow</annotation></semantics></math>) of HellaSwag, WinoGrand, PIQA, LAMBADA and PPL(<math id="A5.T14.4.4.m2.1" class="ltx_Math" alttext="\downarrow" display="inline"><semantics id="A5.T14.4.4.m2.1b"><mo stretchy="false" id="A5.T14.4.4.m2.1.1" xref="A5.T14.4.4.m2.1.1.cmml">↓</mo><annotation-xml encoding="MathML-Content" id="A5.T14.4.4.m2.1c"><ci id="A5.T14.4.4.m2.1.1.cmml" xref="A5.T14.4.4.m2.1.1">↓</ci></annotation-xml><annotation encoding="application/x-tex" id="A5.T14.4.4.m2.1d">\downarrow</annotation></semantics></math>) of WikiText, PTB, C4 for <span id="A5.T14.6.1" class="ltx_text ltx_font_bold">LLaMA-13B</span>, ”Ours-2048” indicates that we have modified the sequence length of the calibration dataset from 512 to 2048.</figcaption>
<table id="A5.T14.7" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A5.T14.7.1.1" class="ltx_tr">
<th id="A5.T14.7.1.1.1" class="ltx_td ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;"></th>
<th id="A5.T14.7.1.1.2" class="ltx_td ltx_th ltx_th_column ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;"></th>
<th id="A5.T14.7.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">Hella.</th>
<th id="A5.T14.7.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">Wino.</th>
<th id="A5.T14.7.1.1.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">PIQA</th>
<th id="A5.T14.7.1.1.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">Lamb.</th>
<th id="A5.T14.7.1.1.7" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">Avg.</th>
<th id="A5.T14.7.1.1.8" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">Wiki.</th>
<th id="A5.T14.7.1.1.9" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">PTB</th>
<th id="A5.T14.7.1.1.10" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">C4</th>
</tr>
<tr id="A5.T14.7.2.2" class="ltx_tr">
<th id="A5.T14.7.2.2.1" class="ltx_td ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;"></th>
<th id="A5.T14.7.2.2.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">FP16</th>
<th id="A5.T14.7.2.2.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">59.13</th>
<th id="A5.T14.7.2.2.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">70.32</th>
<th id="A5.T14.7.2.2.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">78.94</th>
<th id="A5.T14.7.2.2.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">76.17</th>
<th id="A5.T14.7.2.2.7" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">71.14</th>
<th id="A5.T14.7.2.2.8" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">5.09</th>
<th id="A5.T14.7.2.2.9" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">9.08</th>
<th id="A5.T14.7.2.2.10" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">6.80</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A5.T14.7.3.1" class="ltx_tr">
<td id="A5.T14.7.3.1.1" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;" rowspan="4"><span id="A5.T14.7.3.1.1.1" class="ltx_text">W4G-1</span></td>
<td id="A5.T14.7.3.1.2" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">RTN</td>
<td id="A5.T14.7.3.1.3" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">57.96</td>
<td id="A5.T14.7.3.1.4" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">68.19</td>
<td id="A5.T14.7.3.1.5" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">78.18</td>
<td id="A5.T14.7.3.1.6" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">70.95</td>
<td id="A5.T14.7.3.1.7" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">68.82</td>
<td id="A5.T14.7.3.1.8" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">5.53</td>
<td id="A5.T14.7.3.1.9" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">9.78</td>
<td id="A5.T14.7.3.1.10" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">7.23</td>
</tr>
<tr id="A5.T14.7.4.2" class="ltx_tr">
<td id="A5.T14.7.4.2.1" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;">GPTQ</td>
<td id="A5.T14.7.4.2.2" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">57.96</td>
<td id="A5.T14.7.4.2.3" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T14.7.4.2.3.1" class="ltx_text ltx_font_bold">70.24</span></td>
<td id="A5.T14.7.4.2.4" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">77.97</td>
<td id="A5.T14.7.4.2.5" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">73.84</td>
<td id="A5.T14.7.4.2.6" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;">70.00</td>
<td id="A5.T14.7.4.2.7" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">5.33</td>
<td id="A5.T14.7.4.2.8" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">9.48</td>
<td id="A5.T14.7.4.2.9" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">7.13</td>
</tr>
<tr id="A5.T14.7.5.3" class="ltx_tr">
<td id="A5.T14.7.5.3.1" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;">Ours</td>
<td id="A5.T14.7.5.3.2" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">58.02</td>
<td id="A5.T14.7.5.3.3" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">69.61</td>
<td id="A5.T14.7.5.3.4" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T14.7.5.3.4.1" class="ltx_text ltx_font_bold">78.94</span></td>
<td id="A5.T14.7.5.3.5" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T14.7.5.3.5.1" class="ltx_text ltx_font_bold">75.74</span></td>
<td id="A5.T14.7.5.3.6" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T14.7.5.3.6.1" class="ltx_text ltx_font_bold">70.58</span></td>
<td id="A5.T14.7.5.3.7" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T14.7.5.3.7.1" class="ltx_text ltx_font_bold">5.32</span></td>
<td id="A5.T14.7.5.3.8" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T14.7.5.3.8.1" class="ltx_text ltx_font_bold">9.37</span></td>
<td id="A5.T14.7.5.3.9" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T14.7.5.3.9.1" class="ltx_text ltx_font_bold">7.05</span></td>
</tr>
<tr id="A5.T14.7.6.4" class="ltx_tr">
<td id="A5.T14.7.6.4.1" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;">Ours-2048</td>
<td id="A5.T14.7.6.4.2" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T14.7.6.4.2.1" class="ltx_text ltx_font_bold">58.13</span></td>
<td id="A5.T14.7.6.4.3" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">69.69</td>
<td id="A5.T14.7.6.4.4" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">78.67</td>
<td id="A5.T14.7.6.4.5" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">74.95</td>
<td id="A5.T14.7.6.4.6" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;">70.36</td>
<td id="A5.T14.7.6.4.7" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">5.34</td>
<td id="A5.T14.7.6.4.8" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">9.49</td>
<td id="A5.T14.7.6.4.9" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T14.7.6.4.9.1" class="ltx_text ltx_font_bold">7.05</span></td>
</tr>
<tr id="A5.T14.7.7.5" class="ltx_tr">
<td id="A5.T14.7.7.5.1" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;" rowspan="4"><span id="A5.T14.7.7.5.1.1" class="ltx_text">W4G128</span></td>
<td id="A5.T14.7.7.5.2" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">RTN</td>
<td id="A5.T14.7.7.5.3" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">58.43</td>
<td id="A5.T14.7.7.5.4" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">70.32</td>
<td id="A5.T14.7.7.5.5" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">79.33</td>
<td id="A5.T14.7.7.5.6" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">75.32</td>
<td id="A5.T14.7.7.5.7" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">70.85</td>
<td id="A5.T14.7.7.5.8" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">5.26</td>
<td id="A5.T14.7.7.5.9" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">9.29</td>
<td id="A5.T14.7.7.5.10" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">6.94</td>
</tr>
<tr id="A5.T14.7.8.6" class="ltx_tr">
<td id="A5.T14.7.8.6.1" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;">GPTQ</td>
<td id="A5.T14.7.8.6.2" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T14.7.8.6.2.1" class="ltx_text ltx_font_bold">58.79</span></td>
<td id="A5.T14.7.8.6.3" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">70.56</td>
<td id="A5.T14.7.8.6.4" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">79.33</td>
<td id="A5.T14.7.8.6.5" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">75.00</td>
<td id="A5.T14.7.8.6.6" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;">70.92</td>
<td id="A5.T14.7.8.6.7" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">5.21</td>
<td id="A5.T14.7.8.6.8" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">9.28</td>
<td id="A5.T14.7.8.6.9" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">6.92</td>
</tr>
<tr id="A5.T14.7.9.7" class="ltx_tr">
<td id="A5.T14.7.9.7.1" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;">Ours</td>
<td id="A5.T14.7.9.7.2" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">58.62</td>
<td id="A5.T14.7.9.7.3" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T14.7.9.7.3.1" class="ltx_text ltx_font_bold">71.35</span></td>
<td id="A5.T14.7.9.7.4" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T14.7.9.7.4.1" class="ltx_text ltx_font_bold">79.76</span></td>
<td id="A5.T14.7.9.7.5" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">75.98</td>
<td id="A5.T14.7.9.7.6" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T14.7.9.7.6.1" class="ltx_text ltx_font_bold">71.43</span></td>
<td id="A5.T14.7.9.7.7" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T14.7.9.7.7.1" class="ltx_text ltx_font_bold">5.19</span></td>
<td id="A5.T14.7.9.7.8" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T14.7.9.7.8.1" class="ltx_text ltx_font_bold">9.18</span></td>
<td id="A5.T14.7.9.7.9" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T14.7.9.7.9.1" class="ltx_text ltx_font_bold">6.90</span></td>
</tr>
<tr id="A5.T14.7.10.8" class="ltx_tr">
<td id="A5.T14.7.10.8.1" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;">Ours-2048</td>
<td id="A5.T14.7.10.8.2" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">58.47</td>
<td id="A5.T14.7.10.8.3" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">70.56</td>
<td id="A5.T14.7.10.8.4" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">79.22</td>
<td id="A5.T14.7.10.8.5" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T14.7.10.8.5.1" class="ltx_text ltx_font_bold">76.23</span></td>
<td id="A5.T14.7.10.8.6" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;">71.12</td>
<td id="A5.T14.7.10.8.7" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T14.7.10.8.7.1" class="ltx_text ltx_font_bold">5.19</span></td>
<td id="A5.T14.7.10.8.8" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">9.19</td>
<td id="A5.T14.7.10.8.9" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T14.7.10.8.9.1" class="ltx_text ltx_font_bold">6.90</span></td>
</tr>
<tr id="A5.T14.7.11.9" class="ltx_tr">
<td id="A5.T14.7.11.9.1" class="ltx_td ltx_align_center ltx_border_b ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;" rowspan="4"><span id="A5.T14.7.11.9.1.1" class="ltx_text">W3G128</span></td>
<td id="A5.T14.7.11.9.2" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">RTN</td>
<td id="A5.T14.7.11.9.3" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">56.39</td>
<td id="A5.T14.7.11.9.4" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">67.56</td>
<td id="A5.T14.7.11.9.5" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">77.20</td>
<td id="A5.T14.7.11.9.6" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">69.63</td>
<td id="A5.T14.7.11.9.7" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">67.70</td>
<td id="A5.T14.7.11.9.8" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">5.88</td>
<td id="A5.T14.7.11.9.9" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">10.58</td>
<td id="A5.T14.7.11.9.10" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">7.86</td>
</tr>
<tr id="A5.T14.7.12.10" class="ltx_tr">
<td id="A5.T14.7.12.10.1" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;">GPTQ</td>
<td id="A5.T14.7.12.10.2" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">56.58</td>
<td id="A5.T14.7.12.10.3" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">67.96</td>
<td id="A5.T14.7.12.10.4" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">78.07</td>
<td id="A5.T14.7.12.10.5" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">72.31</td>
<td id="A5.T14.7.12.10.6" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;">68.73</td>
<td id="A5.T14.7.12.10.7" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">5.64</td>
<td id="A5.T14.7.12.10.8" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">9.95</td>
<td id="A5.T14.7.12.10.9" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">7.54</td>
</tr>
<tr id="A5.T14.7.13.11" class="ltx_tr">
<td id="A5.T14.7.13.11.1" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;">Ours</td>
<td id="A5.T14.7.13.11.2" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T14.7.13.11.2.1" class="ltx_text ltx_font_bold">57.04</span></td>
<td id="A5.T14.7.13.11.3" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T14.7.13.11.3.1" class="ltx_text ltx_font_bold">69.14</span></td>
<td id="A5.T14.7.13.11.4" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">77.86</td>
<td id="A5.T14.7.13.11.5" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">74.33</td>
<td id="A5.T14.7.13.11.6" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T14.7.13.11.6.1" class="ltx_text ltx_font_bold">69.59</span></td>
<td id="A5.T14.7.13.11.7" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T14.7.13.11.7.1" class="ltx_text ltx_font_bold">5.53</span></td>
<td id="A5.T14.7.13.11.8" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">9.81</td>
<td id="A5.T14.7.13.11.9" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">7.39</td>
</tr>
<tr id="A5.T14.7.14.12" class="ltx_tr">
<td id="A5.T14.7.14.12.1" class="ltx_td ltx_align_center ltx_border_b ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;">Ours-2048</td>
<td id="A5.T14.7.14.12.2" class="ltx_td ltx_align_center ltx_border_b" style="padding-left:5.0pt;padding-right:5.0pt;">56.62</td>
<td id="A5.T14.7.14.12.3" class="ltx_td ltx_align_center ltx_border_b" style="padding-left:5.0pt;padding-right:5.0pt;">68.82</td>
<td id="A5.T14.7.14.12.4" class="ltx_td ltx_align_center ltx_border_b" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T14.7.14.12.4.1" class="ltx_text ltx_font_bold">78.13</span></td>
<td id="A5.T14.7.14.12.5" class="ltx_td ltx_align_center ltx_border_b" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T14.7.14.12.5.1" class="ltx_text ltx_font_bold">74.42</span></td>
<td id="A5.T14.7.14.12.6" class="ltx_td ltx_align_center ltx_border_b ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;">69.50</td>
<td id="A5.T14.7.14.12.7" class="ltx_td ltx_align_center ltx_border_b" style="padding-left:5.0pt;padding-right:5.0pt;">5.57</td>
<td id="A5.T14.7.14.12.8" class="ltx_td ltx_align_center ltx_border_b" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T14.7.14.12.8.1" class="ltx_text ltx_font_bold">9.76</span></td>
<td id="A5.T14.7.14.12.9" class="ltx_td ltx_align_center ltx_border_b" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T14.7.14.12.9.1" class="ltx_text ltx_font_bold">7.37</span></td>
</tr>
</tbody>
</table>
</figure>
<figure id="A5.T15" class="ltx_table">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 15: </span>Accuracies(<math id="A5.T15.3.3.m1.1" class="ltx_Math" alttext="\uparrow" display="inline"><semantics id="A5.T15.3.3.m1.1b"><mo stretchy="false" id="A5.T15.3.3.m1.1.1" xref="A5.T15.3.3.m1.1.1.cmml">↑</mo><annotation-xml encoding="MathML-Content" id="A5.T15.3.3.m1.1c"><ci id="A5.T15.3.3.m1.1.1.cmml" xref="A5.T15.3.3.m1.1.1">↑</ci></annotation-xml><annotation encoding="application/x-tex" id="A5.T15.3.3.m1.1d">\uparrow</annotation></semantics></math>) of HellaSwag, WinoGrand, PIQA, LAMBADA and PPL(<math id="A5.T15.4.4.m2.1" class="ltx_Math" alttext="\downarrow" display="inline"><semantics id="A5.T15.4.4.m2.1b"><mo stretchy="false" id="A5.T15.4.4.m2.1.1" xref="A5.T15.4.4.m2.1.1.cmml">↓</mo><annotation-xml encoding="MathML-Content" id="A5.T15.4.4.m2.1c"><ci id="A5.T15.4.4.m2.1.1.cmml" xref="A5.T15.4.4.m2.1.1">↓</ci></annotation-xml><annotation encoding="application/x-tex" id="A5.T15.4.4.m2.1d">\downarrow</annotation></semantics></math>) of WikiText, PTB, C4 for<span id="A5.T15.6.1" class="ltx_text ltx_font_bold"> LLaMA-7B-V2</span>, ”Ours-2048” indicates that we have modified the sequence length of the calibration dataset from 512 to 2048.</figcaption>
<table id="A5.T15.7" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A5.T15.7.1.1" class="ltx_tr">
<th id="A5.T15.7.1.1.1" class="ltx_td ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;"></th>
<th id="A5.T15.7.1.1.2" class="ltx_td ltx_th ltx_th_column ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;"></th>
<th id="A5.T15.7.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">Hella.</th>
<th id="A5.T15.7.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">Wino.</th>
<th id="A5.T15.7.1.1.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">PIQA</th>
<th id="A5.T15.7.1.1.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">Lamb.</th>
<th id="A5.T15.7.1.1.7" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">Avg.</th>
<th id="A5.T15.7.1.1.8" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">Wiki.</th>
<th id="A5.T15.7.1.1.9" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">PTB</th>
<th id="A5.T15.7.1.1.10" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">C4</th>
</tr>
<tr id="A5.T15.7.2.2" class="ltx_tr">
<th id="A5.T15.7.2.2.1" class="ltx_td ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;"></th>
<th id="A5.T15.7.2.2.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">FP16</th>
<th id="A5.T15.7.2.2.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">56.69</th>
<th id="A5.T15.7.2.2.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">67.17</th>
<th id="A5.T15.7.2.2.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">78.35</th>
<th id="A5.T15.7.2.2.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">73.88</th>
<th id="A5.T15.7.2.2.7" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">69.02</th>
<th id="A5.T15.7.2.2.8" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">5.47</th>
<th id="A5.T15.7.2.2.9" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">32.91</th>
<th id="A5.T15.7.2.2.10" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">7.26</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A5.T15.7.3.1" class="ltx_tr">
<td id="A5.T15.7.3.1.1" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;" rowspan="4"><span id="A5.T15.7.3.1.1.1" class="ltx_text">W4G-1</span></td>
<td id="A5.T15.7.3.1.2" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">RTN</td>
<td id="A5.T15.7.3.1.3" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">55.51</td>
<td id="A5.T15.7.3.1.4" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">66.77</td>
<td id="A5.T15.7.3.1.5" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">77.58</td>
<td id="A5.T15.7.3.1.6" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">68.08</td>
<td id="A5.T15.7.3.1.7" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">66.98</td>
<td id="A5.T15.7.3.1.8" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">6.12</td>
<td id="A5.T15.7.3.1.9" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T15.7.3.1.9.1" class="ltx_text ltx_font_bold">61.61</span></td>
<td id="A5.T15.7.3.1.10" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">8.16</td>
</tr>
<tr id="A5.T15.7.4.2" class="ltx_tr">
<td id="A5.T15.7.4.2.1" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;">GPTQ</td>
<td id="A5.T15.7.4.2.2" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">54.74</td>
<td id="A5.T15.7.4.2.3" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">66.93</td>
<td id="A5.T15.7.4.2.4" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">76.17</td>
<td id="A5.T15.7.4.2.5" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">69.73</td>
<td id="A5.T15.7.4.2.6" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;">66.89</td>
<td id="A5.T15.7.4.2.7" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T15.7.4.2.7.1" class="ltx_text ltx_font_bold">6.09</span></td>
<td id="A5.T15.7.4.2.8" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">NAN</td>
<td id="A5.T15.7.4.2.9" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T15.7.4.2.9.1" class="ltx_text ltx_font_bold">7.90</span></td>
</tr>
<tr id="A5.T15.7.5.3" class="ltx_tr">
<td id="A5.T15.7.5.3.1" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;">Ours</td>
<td id="A5.T15.7.5.3.2" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">55.53</td>
<td id="A5.T15.7.5.3.3" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">67.09</td>
<td id="A5.T15.7.5.3.4" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">77.53</td>
<td id="A5.T15.7.5.3.5" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T15.7.5.3.5.1" class="ltx_text ltx_font_bold">70.81</span></td>
<td id="A5.T15.7.5.3.6" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;">67.74</td>
<td id="A5.T15.7.5.3.7" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">298.4</td>
<td id="A5.T15.7.5.3.8" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">2677</td>
<td id="A5.T15.7.5.3.9" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">11.20</td>
</tr>
<tr id="A5.T15.7.6.4" class="ltx_tr">
<td id="A5.T15.7.6.4.1" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;">Ours-2048</td>
<td id="A5.T15.7.6.4.2" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T15.7.6.4.2.1" class="ltx_text ltx_font_bold">55.63</span></td>
<td id="A5.T15.7.6.4.3" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T15.7.6.4.3.1" class="ltx_text ltx_font_bold">67.96</span></td>
<td id="A5.T15.7.6.4.4" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T15.7.6.4.4.1" class="ltx_text ltx_font_bold">77.64</span></td>
<td id="A5.T15.7.6.4.5" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">69.92</td>
<td id="A5.T15.7.6.4.6" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T15.7.6.4.6.1" class="ltx_text ltx_font_bold">67.79</span></td>
<td id="A5.T15.7.6.4.7" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">196.7</td>
<td id="A5.T15.7.6.4.8" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">2622</td>
<td id="A5.T15.7.6.4.9" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">9.97</td>
</tr>
<tr id="A5.T15.7.7.5" class="ltx_tr">
<td id="A5.T15.7.7.5.1" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;" rowspan="4"><span id="A5.T15.7.7.5.1.1" class="ltx_text">W4G128</span></td>
<td id="A5.T15.7.7.5.2" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">RTN</td>
<td id="A5.T15.7.7.5.3" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T15.7.7.5.3.1" class="ltx_text ltx_font_bold">56.55</span></td>
<td id="A5.T15.7.7.5.4" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">66.93</td>
<td id="A5.T15.7.7.5.5" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">77.37</td>
<td id="A5.T15.7.7.5.6" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">72.44</td>
<td id="A5.T15.7.7.5.7" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">68.32</td>
<td id="A5.T15.7.7.5.8" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T15.7.7.5.8.1" class="ltx_text ltx_font_bold">5.72</span></td>
<td id="A5.T15.7.7.5.9" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T15.7.7.5.9.1" class="ltx_text ltx_font_bold">50.25</span></td>
<td id="A5.T15.7.7.5.10" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">7.58</td>
</tr>
<tr id="A5.T15.7.8.6" class="ltx_tr">
<td id="A5.T15.7.8.6.1" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;">GPTQ</td>
<td id="A5.T15.7.8.6.2" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">56.16</td>
<td id="A5.T15.7.8.6.3" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T15.7.8.6.3.1" class="ltx_text ltx_font_bold">68.03</span></td>
<td id="A5.T15.7.8.6.4" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T15.7.8.6.4.1" class="ltx_text ltx_font_bold">78.56</span></td>
<td id="A5.T15.7.8.6.5" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">72.83</td>
<td id="A5.T15.7.8.6.6" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T15.7.8.6.6.1" class="ltx_text ltx_font_bold">68.90</span></td>
<td id="A5.T15.7.8.6.7" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">5.73</td>
<td id="A5.T15.7.8.6.8" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">NAN</td>
<td id="A5.T15.7.8.6.9" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T15.7.8.6.9.1" class="ltx_text ltx_font_bold">7.53</span></td>
</tr>
<tr id="A5.T15.7.9.7" class="ltx_tr">
<td id="A5.T15.7.9.7.1" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;">Ours</td>
<td id="A5.T15.7.9.7.2" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">56.21</td>
<td id="A5.T15.7.9.7.3" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">67.56</td>
<td id="A5.T15.7.9.7.4" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">77.64</td>
<td id="A5.T15.7.9.7.5" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">73.20</td>
<td id="A5.T15.7.9.7.6" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;">68.65</td>
<td id="A5.T15.7.9.7.7" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">60.03</td>
<td id="A5.T15.7.9.7.8" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">1786</td>
<td id="A5.T15.7.9.7.9" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">8.16</td>
</tr>
<tr id="A5.T15.7.10.8" class="ltx_tr">
<td id="A5.T15.7.10.8.1" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;">Ours-2048</td>
<td id="A5.T15.7.10.8.2" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">55.97</td>
<td id="A5.T15.7.10.8.3" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">67.09</td>
<td id="A5.T15.7.10.8.4" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">77.15</td>
<td id="A5.T15.7.10.8.5" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T15.7.10.8.5.1" class="ltx_text ltx_font_bold">73.57</span></td>
<td id="A5.T15.7.10.8.6" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;">68.45</td>
<td id="A5.T15.7.10.8.7" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">48.91</td>
<td id="A5.T15.7.10.8.8" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">1872</td>
<td id="A5.T15.7.10.8.9" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">8.05</td>
</tr>
<tr id="A5.T15.7.11.9" class="ltx_tr">
<td id="A5.T15.7.11.9.1" class="ltx_td ltx_align_center ltx_border_b ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;" rowspan="4"><span id="A5.T15.7.11.9.1.1" class="ltx_text">W3G128</span></td>
<td id="A5.T15.7.11.9.2" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">RTN</td>
<td id="A5.T15.7.11.9.3" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T15.7.11.9.3.1" class="ltx_text ltx_font_bold">54.65</span></td>
<td id="A5.T15.7.11.9.4" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">67.17</td>
<td id="A5.T15.7.11.9.5" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">75.90</td>
<td id="A5.T15.7.11.9.6" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">65.98</td>
<td id="A5.T15.7.11.9.7" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">65.92</td>
<td id="A5.T15.7.11.9.8" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">6.66</td>
<td id="A5.T15.7.11.9.9" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T15.7.11.9.9.1" class="ltx_text ltx_font_bold">44.89</span></td>
<td id="A5.T15.7.11.9.10" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">8.98</td>
</tr>
<tr id="A5.T15.7.12.10" class="ltx_tr">
<td id="A5.T15.7.12.10.1" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;">GPTQ</td>
<td id="A5.T15.7.12.10.2" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">52.93</td>
<td id="A5.T15.7.12.10.3" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">65.19</td>
<td id="A5.T15.7.12.10.4" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">76.44</td>
<td id="A5.T15.7.12.10.5" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">67.49</td>
<td id="A5.T15.7.12.10.6" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;">65.51</td>
<td id="A5.T15.7.12.10.7" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T15.7.12.10.7.1" class="ltx_text ltx_font_bold">6.57</span></td>
<td id="A5.T15.7.12.10.8" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">NAN</td>
<td id="A5.T15.7.12.10.9" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T15.7.12.10.9.1" class="ltx_text ltx_font_bold">8.61</span></td>
</tr>
<tr id="A5.T15.7.13.11" class="ltx_tr">
<td id="A5.T15.7.13.11.1" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;">Ours</td>
<td id="A5.T15.7.13.11.2" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">53.65</td>
<td id="A5.T15.7.13.11.3" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">66.14</td>
<td id="A5.T15.7.13.11.4" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T15.7.13.11.4.1" class="ltx_text ltx_font_bold">77.09</span></td>
<td id="A5.T15.7.13.11.5" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T15.7.13.11.5.1" class="ltx_text ltx_font_bold">70.64</span></td>
<td id="A5.T15.7.13.11.6" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;">66.88</td>
<td id="A5.T15.7.13.11.7" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">NAN</td>
<td id="A5.T15.7.13.11.8" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">1159</td>
<td id="A5.T15.7.13.11.9" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">9.88</td>
</tr>
<tr id="A5.T15.7.14.12" class="ltx_tr">
<td id="A5.T15.7.14.12.1" class="ltx_td ltx_align_center ltx_border_b ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;">Ours-2048</td>
<td id="A5.T15.7.14.12.2" class="ltx_td ltx_align_center ltx_border_b" style="padding-left:5.0pt;padding-right:5.0pt;">53.91</td>
<td id="A5.T15.7.14.12.3" class="ltx_td ltx_align_center ltx_border_b" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T15.7.14.12.3.1" class="ltx_text ltx_font_bold">67.32</span></td>
<td id="A5.T15.7.14.12.4" class="ltx_td ltx_align_center ltx_border_b" style="padding-left:5.0pt;padding-right:5.0pt;">76.33</td>
<td id="A5.T15.7.14.12.5" class="ltx_td ltx_align_center ltx_border_b" style="padding-left:5.0pt;padding-right:5.0pt;">71.12</td>
<td id="A5.T15.7.14.12.6" class="ltx_td ltx_align_center ltx_border_b ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T15.7.14.12.6.1" class="ltx_text ltx_font_bold">67.17</span></td>
<td id="A5.T15.7.14.12.7" class="ltx_td ltx_align_center ltx_border_b" style="padding-left:5.0pt;padding-right:5.0pt;">NAN</td>
<td id="A5.T15.7.14.12.8" class="ltx_td ltx_align_center ltx_border_b" style="padding-left:5.0pt;padding-right:5.0pt;">1739</td>
<td id="A5.T15.7.14.12.9" class="ltx_td ltx_align_center ltx_border_b" style="padding-left:5.0pt;padding-right:5.0pt;">10.11</td>
</tr>
</tbody>
</table>
</figure>
<figure id="A5.T16" class="ltx_table">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 16: </span>Accuracies(<math id="A5.T16.3.3.m1.1" class="ltx_Math" alttext="\uparrow" display="inline"><semantics id="A5.T16.3.3.m1.1b"><mo stretchy="false" id="A5.T16.3.3.m1.1.1" xref="A5.T16.3.3.m1.1.1.cmml">↑</mo><annotation-xml encoding="MathML-Content" id="A5.T16.3.3.m1.1c"><ci id="A5.T16.3.3.m1.1.1.cmml" xref="A5.T16.3.3.m1.1.1">↑</ci></annotation-xml><annotation encoding="application/x-tex" id="A5.T16.3.3.m1.1d">\uparrow</annotation></semantics></math>) of HellaSwag, WinoGrand, PIQA, LAMBADA and PPL(<math id="A5.T16.4.4.m2.1" class="ltx_Math" alttext="\downarrow" display="inline"><semantics id="A5.T16.4.4.m2.1b"><mo stretchy="false" id="A5.T16.4.4.m2.1.1" xref="A5.T16.4.4.m2.1.1.cmml">↓</mo><annotation-xml encoding="MathML-Content" id="A5.T16.4.4.m2.1c"><ci id="A5.T16.4.4.m2.1.1.cmml" xref="A5.T16.4.4.m2.1.1">↓</ci></annotation-xml><annotation encoding="application/x-tex" id="A5.T16.4.4.m2.1d">\downarrow</annotation></semantics></math>) of WikiText, PTB, C4 for <span id="A5.T16.6.1" class="ltx_text ltx_font_bold">LLaMA-13B-V2</span>, ”Ours-2048” indicates that we have modified the sequence length of the calibration dataset from 512 to 2048.</figcaption>
<table id="A5.T16.7" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A5.T16.7.1.1" class="ltx_tr">
<th id="A5.T16.7.1.1.1" class="ltx_td ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;"></th>
<th id="A5.T16.7.1.1.2" class="ltx_td ltx_th ltx_th_column ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;"></th>
<th id="A5.T16.7.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">Hella.</th>
<th id="A5.T16.7.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">Wino.</th>
<th id="A5.T16.7.1.1.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">PIQA</th>
<th id="A5.T16.7.1.1.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">Lamb.</th>
<th id="A5.T16.7.1.1.7" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">Avg.</th>
<th id="A5.T16.7.1.1.8" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">Wiki.</th>
<th id="A5.T16.7.1.1.9" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">PTB</th>
<th id="A5.T16.7.1.1.10" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">C4</th>
</tr>
<tr id="A5.T16.7.2.2" class="ltx_tr">
<th id="A5.T16.7.2.2.1" class="ltx_td ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;"></th>
<th id="A5.T16.7.2.2.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">FP16</th>
<th id="A5.T16.7.2.2.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">59.71</th>
<th id="A5.T16.7.2.2.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">69.61</th>
<th id="A5.T16.7.2.2.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">78.78</th>
<th id="A5.T16.7.2.2.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">76.71</th>
<th id="A5.T16.7.2.2.7" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">71.20</th>
<th id="A5.T16.7.2.2.8" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">4.88</th>
<th id="A5.T16.7.2.2.9" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">48.82</th>
<th id="A5.T16.7.2.2.10" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">6.73</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A5.T16.7.3.1" class="ltx_tr">
<td id="A5.T16.7.3.1.1" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;" rowspan="4"><span id="A5.T16.7.3.1.1.1" class="ltx_text">W4G-1</span></td>
<td id="A5.T16.7.3.1.2" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">RTN</td>
<td id="A5.T16.7.3.1.3" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">58.56</td>
<td id="A5.T16.7.3.1.4" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">69.30</td>
<td id="A5.T16.7.3.1.5" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T16.7.3.1.5.1" class="ltx_text ltx_font_bold">78.45</span></td>
<td id="A5.T16.7.3.1.6" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">74.36</td>
<td id="A5.T16.7.3.1.7" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">70.17</td>
<td id="A5.T16.7.3.1.8" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">5.20</td>
<td id="A5.T16.7.3.1.9" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">58.57</td>
<td id="A5.T16.7.3.1.10" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">7.14</td>
</tr>
<tr id="A5.T16.7.4.2" class="ltx_tr">
<td id="A5.T16.7.4.2.1" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;">GPTQ</td>
<td id="A5.T16.7.4.2.2" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">57.81</td>
<td id="A5.T16.7.4.2.3" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">67.48</td>
<td id="A5.T16.7.4.2.4" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">77.86</td>
<td id="A5.T16.7.4.2.5" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">73.84</td>
<td id="A5.T16.7.4.2.6" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;">69.25</td>
<td id="A5.T16.7.4.2.7" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T16.7.4.2.7.1" class="ltx_text ltx_font_bold">5.16</span></td>
<td id="A5.T16.7.4.2.8" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T16.7.4.2.8.1" class="ltx_text ltx_font_bold">52.46</span></td>
<td id="A5.T16.7.4.2.9" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T16.7.4.2.9.1" class="ltx_text ltx_font_bold">6.87</span></td>
</tr>
<tr id="A5.T16.7.5.3" class="ltx_tr">
<td id="A5.T16.7.5.3.1" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;">Ours</td>
<td id="A5.T16.7.5.3.2" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">58.63</td>
<td id="A5.T16.7.5.3.3" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T16.7.5.3.3.1" class="ltx_text ltx_font_bold">69.61</span></td>
<td id="A5.T16.7.5.3.4" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">77.91</td>
<td id="A5.T16.7.5.3.5" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">73.98</td>
<td id="A5.T16.7.5.3.6" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;">70.03</td>
<td id="A5.T16.7.5.3.7" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">9.15</td>
<td id="A5.T16.7.5.3.8" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">66.80</td>
<td id="A5.T16.7.5.3.9" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">7.72</td>
</tr>
<tr id="A5.T16.7.6.4" class="ltx_tr">
<td id="A5.T16.7.6.4.1" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;">Ours-2048</td>
<td id="A5.T16.7.6.4.2" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T16.7.6.4.2.1" class="ltx_text ltx_font_bold">58.87</span></td>
<td id="A5.T16.7.6.4.3" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">68.67</td>
<td id="A5.T16.7.6.4.4" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">78.07</td>
<td id="A5.T16.7.6.4.5" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T16.7.6.4.5.1" class="ltx_text ltx_font_bold">75.90</span></td>
<td id="A5.T16.7.6.4.6" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T16.7.6.4.6.1" class="ltx_text ltx_font_bold">70.38</span></td>
<td id="A5.T16.7.6.4.7" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">6.51</td>
<td id="A5.T16.7.6.4.8" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">60.35</td>
<td id="A5.T16.7.6.4.9" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">7.28</td>
</tr>
<tr id="A5.T16.7.7.5" class="ltx_tr">
<td id="A5.T16.7.7.5.1" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;" rowspan="4"><span id="A5.T16.7.7.5.1.1" class="ltx_text">W4G128</span></td>
<td id="A5.T16.7.7.5.2" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">RTN</td>
<td id="A5.T16.7.7.5.3" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">59.12</td>
<td id="A5.T16.7.7.5.4" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">69.46</td>
<td id="A5.T16.7.7.5.5" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">78.02</td>
<td id="A5.T16.7.7.5.6" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">76.29</td>
<td id="A5.T16.7.7.5.7" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">70.72</td>
<td id="A5.T16.7.7.5.8" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T16.7.7.5.8.1" class="ltx_text ltx_font_bold">4.98</span></td>
<td id="A5.T16.7.7.5.9" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">52.22</td>
<td id="A5.T16.7.7.5.10" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">6.87</td>
</tr>
<tr id="A5.T16.7.8.6" class="ltx_tr">
<td id="A5.T16.7.8.6.1" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;">GPTQ</td>
<td id="A5.T16.7.8.6.2" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">59.22</td>
<td id="A5.T16.7.8.6.3" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">68.51</td>
<td id="A5.T16.7.8.6.4" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T16.7.8.6.4.1" class="ltx_text ltx_font_bold">78.84</span></td>
<td id="A5.T16.7.8.6.5" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">76.13</td>
<td id="A5.T16.7.8.6.6" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;">70.68</td>
<td id="A5.T16.7.8.6.7" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">4.99</td>
<td id="A5.T16.7.8.6.8" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T16.7.8.6.8.1" class="ltx_text ltx_font_bold">51.59</span></td>
<td id="A5.T16.7.8.6.9" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">6.87</td>
</tr>
<tr id="A5.T16.7.9.7" class="ltx_tr">
<td id="A5.T16.7.9.7.1" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;">Ours</td>
<td id="A5.T16.7.9.7.2" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">59.20</td>
<td id="A5.T16.7.9.7.3" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">69.14</td>
<td id="A5.T16.7.9.7.4" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">78.35</td>
<td id="A5.T16.7.9.7.5" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">76.56</td>
<td id="A5.T16.7.9.7.6" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;">70.81</td>
<td id="A5.T16.7.9.7.7" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">5.80</td>
<td id="A5.T16.7.9.7.8" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">51.92</td>
<td id="A5.T16.7.9.7.9" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T16.7.9.7.9.1" class="ltx_text ltx_font_bold">6.84</span></td>
</tr>
<tr id="A5.T16.7.10.8" class="ltx_tr">
<td id="A5.T16.7.10.8.1" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;">Ours-2048</td>
<td id="A5.T16.7.10.8.2" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T16.7.10.8.2.1" class="ltx_text ltx_font_bold">59.25</span></td>
<td id="A5.T16.7.10.8.3" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T16.7.10.8.3.1" class="ltx_text ltx_font_bold">70.48</span></td>
<td id="A5.T16.7.10.8.4" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">78.29</td>
<td id="A5.T16.7.10.8.5" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T16.7.10.8.5.1" class="ltx_text ltx_font_bold">76.81</span></td>
<td id="A5.T16.7.10.8.6" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T16.7.10.8.6.1" class="ltx_text ltx_font_bold">71.21</span></td>
<td id="A5.T16.7.10.8.7" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">5.00</td>
<td id="A5.T16.7.10.8.8" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">51.78</td>
<td id="A5.T16.7.10.8.9" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T16.7.10.8.9.1" class="ltx_text ltx_font_bold">6.84</span></td>
</tr>
<tr id="A5.T16.7.11.9" class="ltx_tr">
<td id="A5.T16.7.11.9.1" class="ltx_td ltx_align_center ltx_border_b ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;" rowspan="4"><span id="A5.T16.7.11.9.1.1" class="ltx_text">W3G128</span></td>
<td id="A5.T16.7.11.9.2" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">RTN</td>
<td id="A5.T16.7.11.9.3" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">57.03</td>
<td id="A5.T16.7.11.9.4" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">67.56</td>
<td id="A5.T16.7.11.9.5" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">77.86</td>
<td id="A5.T16.7.11.9.6" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">72.37</td>
<td id="A5.T16.7.11.9.7" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">68.70</td>
<td id="A5.T16.7.11.9.8" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">5.52</td>
<td id="A5.T16.7.11.9.9" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">62.33</td>
<td id="A5.T16.7.11.9.10" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">7.58</td>
</tr>
<tr id="A5.T16.7.12.10" class="ltx_tr">
<td id="A5.T16.7.12.10.1" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;">GPTQ</td>
<td id="A5.T16.7.12.10.2" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">56.99</td>
<td id="A5.T16.7.12.10.3" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">66.69</td>
<td id="A5.T16.7.12.10.4" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T16.7.12.10.4.1" class="ltx_text ltx_font_bold">78.40</span></td>
<td id="A5.T16.7.12.10.5" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">72.85</td>
<td id="A5.T16.7.12.10.6" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;">68.73</td>
<td id="A5.T16.7.12.10.7" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">5.45</td>
<td id="A5.T16.7.12.10.8" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T16.7.12.10.8.1" class="ltx_text ltx_font_bold">55.09</span></td>
<td id="A5.T16.7.12.10.9" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">7.54</td>
</tr>
<tr id="A5.T16.7.13.11" class="ltx_tr">
<td id="A5.T16.7.13.11.1" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;">Ours</td>
<td id="A5.T16.7.13.11.2" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T16.7.13.11.2.1" class="ltx_text ltx_font_bold">57.29</span></td>
<td id="A5.T16.7.13.11.3" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">68.90</td>
<td id="A5.T16.7.13.11.4" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">77.37</td>
<td id="A5.T16.7.13.11.5" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">75.22</td>
<td id="A5.T16.7.13.11.6" class="ltx_td ltx_align_center ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;">69.70</td>
<td id="A5.T16.7.13.11.7" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T16.7.13.11.7.1" class="ltx_text ltx_font_bold">5.35</span></td>
<td id="A5.T16.7.13.11.8" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;">59.57</td>
<td id="A5.T16.7.13.11.9" class="ltx_td ltx_align_center" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T16.7.13.11.9.1" class="ltx_text ltx_font_bold">7.35</span></td>
</tr>
<tr id="A5.T16.7.14.12" class="ltx_tr">
<td id="A5.T16.7.14.12.1" class="ltx_td ltx_align_center ltx_border_b ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;">Ours-2048</td>
<td id="A5.T16.7.14.12.2" class="ltx_td ltx_align_center ltx_border_b" style="padding-left:5.0pt;padding-right:5.0pt;">57.20</td>
<td id="A5.T16.7.14.12.3" class="ltx_td ltx_align_center ltx_border_b" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T16.7.14.12.3.1" class="ltx_text ltx_font_bold">70.88</span></td>
<td id="A5.T16.7.14.12.4" class="ltx_td ltx_align_center ltx_border_b" style="padding-left:5.0pt;padding-right:5.0pt;">78.13</td>
<td id="A5.T16.7.14.12.5" class="ltx_td ltx_align_center ltx_border_b" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T16.7.14.12.5.1" class="ltx_text ltx_font_bold">75.35</span></td>
<td id="A5.T16.7.14.12.6" class="ltx_td ltx_align_center ltx_border_b ltx_border_r" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="A5.T16.7.14.12.6.1" class="ltx_text ltx_font_bold">70.39</span></td>
<td id="A5.T16.7.14.12.7" class="ltx_td ltx_align_center ltx_border_b" style="padding-left:5.0pt;padding-right:5.0pt;">10.38</td>
<td id="A5.T16.7.14.12.8" class="ltx_td ltx_align_center ltx_border_b" style="padding-left:5.0pt;padding-right:5.0pt;">66.22</td>
<td id="A5.T16.7.14.12.9" class="ltx_td ltx_align_center ltx_border_b" style="padding-left:5.0pt;padding-right:5.0pt;">7.92</td>
</tr>
</tbody>
</table>
</figure>
</section>
</article>
</div>
<div class="ar5iv-footer"><a href="/html/2309.05515" class="ar5iv-nav-button ar5iv-nav-button-prev">◄</a>
    <a class="ar5iv-home-button" href="/"><img height="40" alt="ar5iv homepage" src="/assets/ar5iv.png"></a>
    <a href="/feeling_lucky" class="ar5iv-text-button">Feeling<br>lucky?</a>
    <a href="/log/2309.05516" class="ar5iv-text-button ar5iv-severity-warning">Conversion<br>report</a>
    <a class="ar5iv-text-button" target="_blank" href="https://github.com/dginev/ar5iv/issues/new?template=improve-article--arxiv-id-.md&title=Improve+article+2309.05516">Report<br>an issue</a>
    <a href="https://arxiv.org/abs/2309.05516" class="ar5iv-text-button arxiv-ui-theme">View&nbsp;original<br>on&nbsp;arXiv</a><a href="/html/2309.05517" class="ar5iv-nav-button ar5iv-nav-button-next">►</a>
</div><footer class="ltx_page_footer">
<a class="ar5iv-toggle-color-scheme" href="javascript:toggleColorScheme()" title="Toggle ar5iv color scheme"><span class="color-scheme-icon"></span></a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/license" target="_blank">Copyright</a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/policies/privacy_policy" target="_blank">Privacy Policy</a>

<div class="ltx_page_logo">Generated  on Wed Feb 28 07:06:44 2024 by <a target="_blank" href="http://dlmf.nist.gov/LaTeXML/" class="ltx_LaTeXML_logo"><span style="letter-spacing:-0.2em; margin-right:0.1em;">L<span class="ltx_font_smallcaps" style="position:relative; bottom:2.2pt;">a</span>T<span class="ltx_font_smallcaps" style="font-size:120%;position:relative; bottom:-0.2ex;">e</span></span><span style="font-size:90%; position:relative; bottom:-0.2ex;">XML</span><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAsAAAAOCAYAAAD5YeaVAAAAAXNSR0IArs4c6QAAAAZiS0dEAP8A/wD/oL2nkwAAAAlwSFlzAAALEwAACxMBAJqcGAAAAAd0SU1FB9wKExQZLWTEaOUAAAAddEVYdENvbW1lbnQAQ3JlYXRlZCB3aXRoIFRoZSBHSU1Q72QlbgAAAdpJREFUKM9tkL+L2nAARz9fPZNCKFapUn8kyI0e4iRHSR1Kb8ng0lJw6FYHFwv2LwhOpcWxTjeUunYqOmqd6hEoRDhtDWdA8ApRYsSUCDHNt5ul13vz4w0vWCgUnnEc975arX6ORqN3VqtVZbfbTQC4uEHANM3jSqXymFI6yWazP2KxWAXAL9zCUa1Wy2tXVxheKA9YNoR8Pt+aTqe4FVVVvz05O6MBhqUIBGk8Hn8HAOVy+T+XLJfLS4ZhTiRJgqIoVBRFIoric47jPnmeB1mW/9rr9ZpSSn3Lsmir1fJZlqWlUonKsvwWwD8ymc/nXwVBeLjf7xEKhdBut9Hr9WgmkyGEkJwsy5eHG5vN5g0AKIoCAEgkEkin0wQAfN9/cXPdheu6P33fBwB4ngcAcByHJpPJl+fn54mD3Gg0NrquXxeLRQAAwzAYj8cwTZPwPH9/sVg8PXweDAauqqr2cDjEer1GJBLBZDJBs9mE4zjwfZ85lAGg2+06hmGgXq+j3+/DsixYlgVN03a9Xu8jgCNCyIegIAgx13Vfd7vdu+FweG8YRkjXdWy329+dTgeSJD3ieZ7RNO0VAXAPwDEAO5VKndi2fWrb9jWl9Esul6PZbDY9Go1OZ7PZ9z/lyuD3OozU2wAAAABJRU5ErkJggg==" alt="Mascot Sammy"></a>
</div></footer>
</div>

    <script>
      var canMathML = typeof(MathMLElement) == "function";
      if (!canMathML) {
        var body = document.querySelector("body");
        body.firstElementChild.setAttribute('style', 'opacity: 0;');
        var loading = document.createElement("div");
        loading.setAttribute("id", "mathjax-loading-spinner");
        var message = document.createElement("div");
        message.setAttribute("id", "mathjax-loading-message");
        message.innerText = "Typesetting Equations...";
        body.prepend(loading);
        body.prepend(message);

        var el = document.createElement("script");
        el.src = "https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js";
        document.querySelector("head").appendChild(el);

        window.MathJax = {
          startup: {
            pageReady: () => {
              return MathJax.startup.defaultPageReady().then(() => {
                body.removeChild(loading);
                body.removeChild(message);
                body.firstElementChild.removeAttribute('style');
              }); } } };
      }
    </script>
    <script>
    // Auxiliary function, building the preview feature when
    // an inline citation is clicked
    function clicked_cite(e) {
      e.preventDefault();
      let cite = this.closest('.ltx_cite');
      let next = cite.nextSibling;
      if (next && next.nodeType == Node.ELEMENT_NODE && next.getAttribute('class') == "ar5iv-bibitem-preview") {
        next.remove();
        return; }
      // Before adding a preview modal,
      // cleanup older previews, in case they're still open
      document.querySelectorAll('span.ar5iv-bibitem-preview').forEach(function(node) {
        node.remove();
      })

      // Create the preview
      preview = document.createElement('span');
      preview.setAttribute('class','ar5iv-bibitem-preview');
      let target = document.getElementById(this.getAttribute('href').slice(1));
      target.childNodes.forEach(function (child) {
        preview.append(child.cloneNode(true));
      });
      let close_x = document.createElement('button');
      close_x.setAttribute("aria-label","Close modal for bibliography item preview");
      close_x.textContent = "×";
      close_x.setAttribute('class', 'ar5iv-button-close-preview');
      close_x.setAttribute('onclick','this.parentNode.remove()');
      preview.append(close_x);
      preview.querySelectorAll('.ltx_tag_bibitem').forEach(function(node) {
        node.remove();
      });
      cite.parentNode.insertBefore(preview, cite.nextSibling);
      return;
    }
    // Global Document initialization:
    // - assign the preview feature to all inline citation links
    document.querySelectorAll(".ltx_cite .ltx_ref").forEach(function (link) {
      link.addEventListener("click", clicked_cite);
    });
    </script>
    </body>
</html>

2309.05516
Large Language Models (LLMs)  have proven their exceptional capabilities in performing language-related tasks. However, their deployment poses significant challenges due to their considerable memory and storage requirements. In response to this issue, weight-only quantization, particularly 3 and 4-bit weight-only quantization, has emerged as one of the most viable solutions. As the number of bits decreases, the quantization grid broadens, thus emphasizing the importance of up and down rounding. While previous studies have demonstrated that fine-tuning up and down rounding with the addition of perturbations can enhance accuracy in some scenarios, our study is driven by the precise and limited boundary of these perturbations, where only the threshold for altering the rounding value is of significance. Consequently, we propose a concise and highly effective approach for optimizing the weight rounding task. Our method, named SignRound, involves lightweight block-wise tuning using signed gradient descent, enabling us to achieve outstanding results within 400 steps. SignRound competes impressively against recent methods without introducing additional inference overhead.
The source code will be publicly available athttps://github.com/intel/neural-compressorsoon.
abstract
Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs
Abstract
2309.05516
1IntroductionLarge language models (LLMs)  have demonstrated exceptional proficiency on language-related tasks. Nevertheless, the deployment of LLMs presents notable hurdles due to their extensive memory and storage needs. Moreover, the computational demands of these models leads to the challenges for real-time applications. Consequently, it becomes imperative to explore techniques like quantization to facilitate the efficient deployment of LLMs.Quantization techniques can be broadly classified into two categories: quantization-aware training (QAT) and post-training quantization (PTQ) . QAT involves training the model with quantization in mind. During QAT, the model is trained using simulated lower-precision representations, allowing it to learn and adapt to the effects of quantization. This approach often yields better accuracy compared to PTQ. However, QAT comes with certain drawbacks, including increased training complexity, longer training times, and the need to tune hyperparameters. Applying QAT to LLMs can be particularly costly, despite recent effortsto improve the efficiency of fine-tuning LLMs. In contrast, PTQ directly quantizes the model without any simulated training or fine-tuning. While PTQ is a concise approach, it is susceptible to significant accuracy drops. This highlights the need for further advancements in PTQ methods to enhance their accuracy preservation capabilities.Two types of tensors could be quantized: activations and weights. Weight-only quantization has gained prominence in recent times as it offers a favorable tradeoff for LLMs. Quantizing activations for LLMs can be challenging, making weight-only quantization a more practical choice. Additionally, the primary bottleneck in generating new tokens for LLMs often lies in memory bandwidth, further emphasizing the significance of weight-only quantization. In this work, we only focus on weight only quantization.In order to quantize the weights, a rounding operation is necessary, with rounding-to-nearest (RTN)  being the predominant method. RTN quantizes each element independently by simply rounding it to the nearest integer. However, RTN fails to consider the relationships between weights and weights, as well as weights and activations. The potential of an advanced rounding strategy to improve accuracy has been initially demonstrated by Nagel et al.. They addressed the rounding task by formulating it as a quadratic unconstrained binary optimization problem and approximated the task loss by employing a Taylor series expansion. However, relying exclusively on the second-order term may not produce accurate results. This is because rounding can introduce considerable weight modifications that may make other order terms significant and non-negligible.We prefer the signed gradient descent method to effectively tackle the issue of sub-optimal rounding solutions. This approach is inspired by the well-defined boundaries of the solution space, which are confined to the range of [-0.5, 0.5], where only the threshold for altering the rounding value is of significance. Figure1provides an overview of our method, SignRound. It utilizes signed gradient descent to fine-tune the up and down rounding through block-wise output reconstruction, resulting in enhanced flexibility and faster convergence. Our contributions are primarily threefold:•We introduce a succinct and potent method for optimizing the weight-rounding task. Our approach utilizes a minimal amount of unlabeled data and executes quantization in a block-wise fashion. Moreover, it is worth noting that our method does not introduce any additional overhead during inference, further enhancing its general practicality.•Our findings demonstrate that a mere alteration of approximately 5% of the rounding values can significantly enhance the performance of some quantization models.•Our empirical results exhibit substantial performance enhancements over the established baseline of RTN, and our method contends favorably against recent techniques.
introduction
Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs
1Introduction
2309.05516
2Related WorkQuantization Aware Training.QAT methods have gained widespread popularity in model compression, as they enable the fine-tuning process, often leading to superior accuracy compared to the PTQ method. In their work,proposed a novel approach that estimates and scales the task loss gradient at each weight and activation layer’s quantizer step size, allowing for joint learning with other network parameters.put forward a progressive quantization scheme that involves quantizing activations after weights. Additionally, CPQeffectively identified the optimal quantization grids while naturally encouraging the underlying full-precision weights to gather around those quantization grids cohesively during training. While QAT methods are popular in relatively small-scale models, their application in LLMs is limited due to the high computational cost associated with training or fine-tuning.Post-training Quantization (PTQ) .PTQ methods simplify the quantization process without the needs of additional training.focused on minimizing quantization error through weight equalization and bias correction techniques.specifically addressed the quantization of vision transformers, introducing a ranking loss to preserve the relative order of self-attention results after quantization and exploring a mixed-precision quantization scheme.leveraged Optimal Brain Surgeonto tune weights during model compression. Both Hawqand HAQaimed to identify important layers and maintain higher precision for them. Given its low resource requirement, PTQ is particularly suitable for the quantization of Large Language Models (LLMs) . We will next focus on the quantization methods designed for LLMs, most of which fall under the category of PTQ.Large Language Models Quantization.Significant advancements have been made in addressing the pressing demand for quantizing large language models (LLMs) . LLM.int8() introduced a mixed precision approach to preserve essential channels in high precision. ZeroQuantV2employed low-rank matrices to enhance model quality recovery. RPTQmitigated the impact of range differences between channel by rearranging the channels and quantizing them in clusters. Other methods, such as SPIQ, SmoothQuant, Outlier Suppression+, utilized handcrafted equivalent transformations to mitigate quantization errors. While these approaches are effective, their applicability is limited due to the performance overhead involved during inference, because there is no chance to fuse the transformation scale to the model itself on certain model architectures. LLM-QATemploys QAT to enhance the performance of W4A8.
In the context of weight-only quantization, GPTQoptimized weights using the Optimal Brain Surgeontechnique, achieving low-bit quantization on LLMs with minimal computational overhead. AWQfollowed the equivalent transformation approach with additional tuning in a constrained space, and has the similar limitations as SmoothQuant. SqueezeLLMemployed sensitivity-based non-uniform quantization and dense-and-sparse decomposition to achieve lossless compression to ultra-low precision. While recent advancements in LLM quantization have made significant progress, there is still room for improvement in achieving minimal quantization loss without introducing inference overhead.Rounding Methods.Adaptive Roundinghas already showcased the potential of an advanced rounding strategy to enhance accuracy. They used the rounding task as a quadratic unconstrained binary optimization problem by approximating the task loss through a Taylor series expansion. However, considering only the second-order term may not yield accurate results. This is because the rounding value gets multiplied by a scaling coefficient during de-quantization, potentially introducing significant weight changes that make other order terms non-negligible. FlexRoundintroduces a more flexible approach to rounding by incorporating element-wise division. This allows for simultaneous learning of a shared quantization grid size and individual scales for each pre-trained weight. However, it’s not easily scalable to apply to LLMs due to the needs of specialized hyperparameters for each specific model and task. AQuantintroduced a dynamic approach where the border becomes a function dependent on the activation value to reduce the quantization error of activation. We specifically concentrate on the up and down rounding task for weight quantization in this work.Signed Gradient Descent.Signed gradient descent is not commonly utilized and is typically applied in specific scenarios, such as reducing communication costs. This is because signed gradient carries significantly less information compared to original gradient. Recent studies have shed light on the advantages of sign-based methods over gradient descent in certain conditions. Safaryan et al.found that sign-based methods are preferable when the Hessian matrix is concentrated on its diagonal and the maximal eigenvalue is much larger than the average eigenvalue. Li et al.investigated a variant of sign-based gradient descent that exhibits faster convergence. Additionally, Safaryan et al.proposed a stochastic sign descent with momentum, which converges under the standard bounded variance assumption with the optimal asymptotic rate. These findings contribute to a better understanding of the potential benefits and applications of signed gradient descent methods.
body
Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs
2Related Work
2309.05516
3MethodologyWe provide an overview of quantization before diving into the details of our approach. To quantize and de-quantize the weights, the following operation as shown in Eq.1is used (disregarding zero point for simplicity) .(1) whereis the quantization scale, which is a positive scalar value. However, it is important to mention that our method can be easily extended to cases whereis a vector or tensor. And the rounding operationis typically performed using the RTN method. While RTN is a concise approach, it quantizes each element independently, thereby losing the ability to model the correlation among different weights or activations.To introduce more flexibility into the rounding operation, a tensorwith the same shape ofis introduced. Each element offalls within the range of, in whichis set to 0.5 in all of experiments to ensure that the changes made only impact the rounding value.(2) This adjustment allows for a more adaptable and context-aware quantization process.
If we try to reconstruct the output of layers, the loss could be formulated as(3) whereis the input of the layer anddenotes the Frobenius norm. Then the final optimization task is described as the following(4) 3.1SignRoundFigure 1:An illustration of SignRound. Unlike the direct rounding in RTN, SignRound performs signed gradient descent to fine-tune the up and down rounding through block-wise output reconstruction. After lightweight forward and backward steps,WINT4 has been well optimized towards the minimal loss, therefore ready for the final inference deployment. Note that Quant and Dequant are two standard operations for quantization and dequantization respectively.Sincehas a clear boundary, i.e., and only the threshold for altering the rounding value is of significance, we prefer scaled signed gradient descent instead of normal gradient descent to optimize this task. Figure1shows an illustration of our method. More precisely, we follow the below optimization to approach the sub-optimal solution of Eq.4.(5) whereis the optimizing step, lr is the learning rate,is the absolute operation and B is the boundary we use, which is set to 0.5 in all our experiments.Further, by employing straight-through estimator (STE) , it can be easily demonstrated thatin Eq.5as following since elements ofare all positive.(6) (7) So our optimization could be simplified as(8) Moreover, as Eq3averages the loss of each element, which presumes that each one contributes equally to the network, that basically is not true. To alleviate this issue, we optimize the rounding task blockwise. To clarify, in our context, we use the term ’layer’ to refer to a linear/convolution layer, while ’block’ denotes a transformer block that typically consists of several linear layers.The above pseudocode1presents more details of SignRound.Algorithm 1SignRoundInput:Calibration Data, learning rate, total steps, Model, block modulewith weights, zero initialized, batch sizeOutput:1:,,2:fortodo3:4:get the inputs of m5:get the output of original module6:quantize and dequantize w via Eq.27:get the output of quantized module8:get the loss via Eq.39:ifthen10:11:12:endif13:14:update V via Eq.815:endfor
body
Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs
3Methodology
2309.05516
4ExperimentsIn this section, we conduct a comprehensive evaluation of SignRound from various perspectives. Firstly, we provide a brief overview of the LLM architectures and tasks that are included in our evaluation. Secondly, we present a detailed comparison between our method and some other existing approaches, highlighting the unique features and advantages of SignRound. Thirdly, we conduct additional experiments to further demonstrate the validity of our choices, assess the sensitivity of hyperparameters, and explore other relevant factors. Finally, the runtime is reported in AppendixDfor reference.4.1Experimental SettingsEvaluation and Datasets.We make assessments on several language tasks to satisfy the task-agnostic setting. Specifically, we report average accuracy results on four common sense reasoning tasks including HellaSwag, WinoGrande, PIQAand LAMBADA.Additionally, we benchmarked our models on MMLU, which encompasses 57 tasks spanning STEM, humanities, social science, and more. Evaluation for all these tasks was performed using the lm-eval-harness. Furthermore, we complement our evaluation with perplexity (ppl)  analysis on Wikitext2and C4, by following the source code111https://github.com/IST-DASLab/gptqof GPTQ.Quantization Configurations.In line with the approach taken in GPTQ, we specifically concentrate on weight-only quantization, targeting the linear layers within transformer blocks. Other layers, such as the embedding layer and typically the last layer like lm-head, are excluded from the quantization process. We initially intended to utilize the
piledataset for calibration, following AWQand SmoothQuant. However, due to its large size, we have opted to use the readily available pile-10k dataset222https://huggingface.co/datasets/NeelNanda/pile-10k, which consists of the first 10k samples from pile, for both GPTQ and our method. We employ standard uniform per-row asymmetric quantization on the min-max grid. Our evaluation primarily focuses on W4, W4G128, and W3G128, where W4 indicates quantizing weights with 4 bits and G represents finer-granularity grouping as described in.Large Language Models.Our experimental evaluation encompasses a range of widely adopted LLM architectures, such as LLaMAs, LLaMAs v2, BLOOMs, and OPTs. We cover a wide range of LLM parameters, ranging from millions to billions, to ensure comprehensive coverage and analysis.SignRound Hyperparameters.We selected 512 samples randomly from pile-10k and truncated each sample to a sequence length of 512. The tuning process involves adjusting each block for 400 steps using a learning rate of 2.5e-3, a batch size of 8, and employing a linear learning rate decay. We set the value of B in Eq.8to 0.5. Besides, we adopted automatic mixed precision(AMP)  to accelerate the tuning. It’s worth noting that adjusting the sequence length to 2048 yielded improvements in numerous scenarios. However, we did not adopt this as the default setting due to the associated runtime overhead.
For models30B, we made configuration adjustments to strike a balance between runtime and performance. Specifically, we reduced the sample count to 256, shorted the sequence length to 256, and disabled AMP.Table 1:Average % accuracy()  of HellaSwag, WinoGrand, PIQA and LAMBADA for LLaMA & OPT.nbitsmethodsLLaMAOPT7b13b7bv213bv2125m1.3b2.7b6.7b13b16FP1668.8071.1469.0271.2045.0957.6661.0464.9265.49W4RTN67.3868.8266.9870.1739.4147.2258.6162.9964.08GPTQ64.7070.0066.8969.2443.5856.1559.9263.0964.83Ours68.0570.5867.7470.0344.1356.1760.5864.3465.05W4G128RTN67.8570.8468.3270.7245.2756.4760.7064.0364.84GPTQ66.3270.9268.9070.6842.8856.9961.2364.7565.37Ours68.0971.4368.6570.8144.2357.3060.8664.7665.67W3G128RTN64.9467.7065.9268.7039.1142.6136.9956.0949.56GPTQ58.2968.7365.5168.7339.7854.4358.4762.9864.68Ours66.6269.5966.8869.7043.3155.4659.1253.4263.61Table 2:Average % accuracy()  of HellaSwag, WinoGrand, PIQA and LAMBADA for BLOOM.W4W4G128W3G128Size560m1b73b7b1560m1b73b7b1560m1b73b7b1FP1645.5052.3155.4860.2245.5052.3155.4860.2245.5052.3155.4860.22RTN43.1049.9753.1657.7344.2852.0854.8659.3140.8347.9852.5157.59GPTQ43.9550.9154.6558.2744.7952.0855.6859.5942.7448.8153.4158.12Ours45.0051.4754.6359.5245.4051.8555.4059.8344.0850.5253.6458.69Table 3:C4 ppl ()  at W4.LLaMAOPTBLOOMSize7b13b7bv213bv21.3b2.7b6.7b13b560m1b73b7b1FP167.346.807.266.7316.0714.3412.7112.0626.5919.4917.4815.20RTN8.127.238.167.1427.4918.8314.3713.3229.8721.2518.7616.05GPTQ8.647.137.906.8717.0415.0613.3912.2928.1520.7118.1815.67Ours7.847.0511.207.7216.9214.9713.0812.4828.1220.4118.1815.674.2Comparing With Other MethodsWe conducted a comprehensive benchmarking of our results against RTN and GPTQ. However, it is important to highlight that act-order was not enabled in GPTQ due to the kernel overhead it introduces, although it has the potential to improve accuracy for certain models. When evaluating perplexity (ppl) , we prioritize reporting the ppl on C4 dataset as our primary focus, taking into consideration the potential occurrence of NaN values when assessing perplexity for Wikitext2 and ptb datasets, both for SignRound and GPTQ. Furthermore, we conducted a limited and non-rigorous comparison between our approach and AWQin AppendixA.1.We begin by presenting the average accuracy results for the HellaSwag, WinoGrand, PIQA, and LAMBADA tasks across LLaMA, OPT, and BLOOM models with a size below 13B. These results are shown in Table1and2. In conclusion, our method outperforms RTN in 36 out of 39 scenarios, showcasing its effectiveness. Additionally, when comparing our approach to GPTQ, we surpass it in 32 out of 39 scenarios, further highlighting the strengths of our method. While our method showcases overall effectiveness, it is important to acknowledge the presence of outliers, such as OPT6.7B at W3G128. Although the root cause for this has not been identified yet, it could be mitigated by fine-tuning hyperparameters, as discussed in the following sections. For detailed results of LLaMA7B, LLaMA13B, LLAMA7B-V2, and LLAMA13B-V2, please refer to AppendixE. The results in AppendixEalso highlight that changing the sequence length to 2048 could bring noticeable improvement in many scenarios.We then present the perplexity (ppl)  results for C4 in Table3, along with the detailed results for Wikitext2 in AppendixA.2. In conclusion, we achieve better or comparable performance in 9 out of 12 models. In certain cases where the results may not be optimal, we can still fine-tune the hyperparameters to achieve better results, as demonstrated in the subsequent sections.Next, we present a comprehensive breakdown of the accuracies achieved by MMLU for LLaMA-7B and LLaMa-7B-V2 in Table4. By analyzing the average accuracies, we observe that SingRound outperforms RTN and GPTQ in 4 out of the 6 scenarios when the best model-wise setting is applied.Table 4:Accuracies()  of MMLU(5-shot)  for LLaMA-7B & LLaMA-7B-V2. ”Ours-2048” indicates that we have modified the sequence length of the calibration dataset from 512 to 2048.LLaMA-7BLLaMA-7B-V2Hums.STEMSocialOtherAvg.Hums.STEMSocialOtherAvg.FP1638.3231.1738.0536.8535.6451.4037.0052.2349.5146.56W4G-1RTN34.8429.5332.8736.2833.1044.0332.8344.9742.1940.24GPTQ33.3126.2929.8633.1130.3246.2134.2946.6844.8542.21Ours34.3031.0534.7436.6633.9547.2833.1446.9044.7042.10Ours204835.1030.6936.4336.8534.4247.4033.9249.6144.9143.00W4G128RTN36.3031.6737.4037.9935.4849.5436.5050.9547.8745.31GPTQ37.7729.6436.3837.4534.8350.3036.5150.9147.6945.43Ours36.0630.8635.9936.2134.4451.3937.8752.5649.6946.95Ours204835.6630.0536.1637.5734.4650.1236.7051.4448.2045.69W3G128RTN32.9730.2833.6632.6032.1741.1433.0640.9840.9438.51GPTQ30.7728.2930.7331.3330.1244.6637.5546.3643.4742.48Ours30.1228.2130.6430.3429.6844.5333.5344.6043.5240.82Ours204832.4328.6231.0332.1030.8542.7532.9842.8841.3039.34We also provide the results for models with a capacity of 30B or greater at W3G128 in Table5and W4 in AppendixA.3. Additionally, we discovered that recovering the sequence length to 512 of the calibration dataset yielded improvements in certain scenarios, and thus we include these results. In summary, our approach achieves comparable performance to GPTQ for the given accuracy task. However, we slightly lag behind GPTQ in terms of ppl tasks.Table 5:Average % accuracy()  of HellaSwag, WinoGrand, PIQA and LAMBADA and C4 ppl()  for LLaMA & OPT with30B at W3G128. ”Ours-seq512” indicates that we have modified the sequence length of the calibration dataset from 256 to 512.AccuracyPPL on C4TypeLLaMAOPTLLaMAOPTSize30b65b30b66b30b65b30b66bFP1673.4675.4867.8769.546.135.9811.4610.99RTN72.1773.6962.8338.006.856.5230.81285.41GPTQ72.0973.9766.7667.876.806.5211.7411.87Ours-seq25672.4573.7166.5168.006.836.5213.0013.34Ours-seq51271.9573.7866.7067.266.796.5312.5013.974.3Block-wise versus Layer-wiseWe examined the effects of layer-wise and block-wise tuning. As explained in Section3.1, the term ”layer” refers to a linear/convolution layer, while ”block” specifically denotes a transformer block consisting of multiple linear layers. To simplify this evaluation, we set the sequence length to 256 and disable AMP. Based on the below results, block-wise tuning outperformed layer-wise tuning in the majority of scenarios.Table 6:Comparing block-wise and layer-wise tuning for around 7B models, the models LLaMA7b, LLaMA7bv2, OPT6.7b, and BLOOM7b1 are denoted by 7b, 7bv2, 6.7b, and 7b1 respectively. The accuracy is the % average accuracy()  of HellaSwag, WinoGrand, PIQA and LAMBADA . Perplexity (PPL)  ()  is evaluated using the C4 dataset.W4W3G128Size7b7bv26.7b7b17b7bv26.7b7b1layer-acc-seq25667.5067.7863.4658.7265.9666.0961.6058.24block-acc-seq25667.6467.9664.5559.0866.3166.6357.7658.34layer-c4-ppl-seq2568.027.9213.4415.738.818.6916.8316.15block-c4-ppl-seq2567.818.1913.1015.718.3410.8425.4416.054.4The analysis of hyperparameters sensitivityWe conducted a hyperparameters sensitivity analysis, the results of which are summarized in Table7. In the ”steps100” configuration, we used 100 steps, and a learning rate of 1e-2.
In the ”lr4e-3” configuration, we set the learning rate to 4e-3. We also changed the sequence length of the calibration dataset from 512 to 2048, denoted by ”seq2048”. Please note that all other hyperparameters not mentioned in each configuration were kept the same as the default configurations, as detailed in Section4.1. Overall, our method exhibits robustness to hyperparameters in common sense reasoning tasks, with the exception of the perplexity of LLaMA-7b-v2. However, we did discover that certain hyperparameters, such as the sequence length of the calibration dataset, can significantly impact performance in some scenarios, as demonstrated in Table4and5.Table 7:Hyperparameter sensitivity analysis, the models LLaMA7b, LLaMA7bv2, OPT6.7b, and BLOOM7b1 are denoted by 7b, 7bv2, 6.7b, and 7b1 respectively. The accuracy is the % average accuracy()  of HellaSwag, WinoGrand, PIQA and LAMBADA . Perplexity (PPL)  ()  is evaluated using the C4 dataset.AccuracyPPL on C4Size7b7bv26.7b7b17b7bv26.7b7b1steps10067.5367.7664.6458.767.937.8313.1215.71lr4e-368.0167.5764.5759.477.8110.2913.0915.66seq204868.1167.7964.3259.397.769.9713.0615.66default68.0567.7464.3459.527.8411.2013.0815.674.5The Analysis of Gradients and Their Effects on RoundingLLaMA-7BLLaMA-7B-V2OPT-6.7BBLOOM-7B1Figure 2:The impact of the rounding value introduced by thein Eq.2In this analysis, we dive into the distribution of the magnitude ofin Eq.2and its impact on rounding values across approximately 7 billion models at W4. The visual representations of these distributions are provided in AppendixB. Our investigation reveals that the majority of V values are concentrated within the range of [-0.3, 0.3]. Additionally, we observe an interesting pattern in the distribution of V across different layers. The middle layers exhibit a more tightly clustered distribution compared to the other layers. This observation aligns with the common understanding that the head and tail layers tend to be more sensitive to compression, while the middle layers are relatively more robust.Figure2illustrates the impact of the rounding value introduced by thein Eq.2for models around 7B at W4. The red line represents ”up rounding”, indicating that while RTN rounds the value to the floor, SignRound changes it to the ceiling. Conversely, the green line represents ”down rounding” indicating that while RTN rounds the value to the ceiling, SignRound changes it to the floor. It is worth noting that SignRound modifies only a small percentage of weight rounding values for each of the four models, namely 5.27%, 5.29%, 4.14%, and 4.10%.We were also intrigued by the possible correlation between rounding and activation, as previous research has shown that keeping only 0.1%-1% of the channels corresponding to larger activation can significantly improve the quantized performance in AWQ. We shown the result in AppendixC.
body
Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs
4Experiments
2309.05516
5Conclusions and LimitationsIn this paper, we present a highly effective and concise approach to optimize the weight rounding task. Our method, SignRound, leverages lightweight block-wise tuning using signed gradient descent, achieving remarkable results within a mere 400 steps. Extensive experiments demonstrate the superior performance of our approach. As part of our future work, we plan to apply our approach to more diverse LLM models (e.g., Code LLaMA, LLaMA v2 Chat) , and contribute our recipes and implementations to the open source community. On the other hand, although our method is generally effective, there are a few outliers in certain scenarios, where we plan to mitigate the issue by fine-tuning the hyperparameters.
conclusion
Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs
5Conclusions and Limitations